[SOLVED] Input Parser size limit?

photogramdude

Hi, I'm wrangling a large amount of text from an API URL http://api.smartcitizen.me/devices/1616

via: Get URL text > input parser

I'm about halfway through and it has just given up on a virtually identical section to what was processed correctly above.

Is there a maximum size limit of field limit for the Input Parser? I've searched the manual and the forum...

Thanks,

Rob

bonemap

@robhblack said:

it has just given up on a virtually identical section

Hi Rob,

SmartCitizen looks very interesting thanks for introducing it here

If there is a limitation only @mark or @DusX will have the answer to enlighten us.

EDIT: I had a go at parsing the data but could not get any further than the same point indicated in your screen grab. Patch 3.07 attached: SmartcitizenParse.izz

Best Wishes

Russell

photogramdude

Hi,

Yeah Smart Citizen have sensors all over the world, and we want to tell the story of air pollution and biometrics (maybe with Cycling '74 MAX > Apple Watch) parsed input in a nice visual way.

Thanks for this huge contribution, I think logic would dictate Isadora Input Parser is setting an arbitrary KB limit on the length of the text field it parses. I can see that it all comes in, but then it falls over! I guess there could be a dev option to increase this?

In the meantime the workaround is to cleave the input in two somehow?

Cheers for this big help,

Rob

bonemap

Hi Rob,

I believe there is an alternative using json data and the Isadora javascript actor to parse the data stream into the patch. I am not able to advise you on the json method. unfortunately I am not proficient enough and am still learning.

When working with the data I did have n issue with parsing the degree symbol for a temperature field. I see from your screen grab that your field with the degree symbol has parsed OK. I am wondering why that is. Are you using a Mac or PC? I can’t think of why I couldn’t get that to work.

Best wishes

Russell

photogramdude

Hi Russell, yesterday I used the Hello OSC method from DusX which I got installed on my Mac OK, could see cabin pressure on my browser but Isadora itself steadfastly refused to pick up the input, rage quit after a few hours!

We're looking at Node Red.

I'm using a Mac, was wondering about making some of the fields more agnostic (was trying to strictly typecast so that variable data input eg null values using 2 bytes instead of 3 because no quotes doesn't make it fall over.

The documentation is simultaneously expansive but I must admit to struggling!

I'll look into the JSON method after trying to restore some sanity, but if I break the back of this it's massive since APIs have always been this monolithic terrifying concept for me!

Best,

R

photogramdude

Just checked, the input runs dry after 2048 characters or 2K. So it would be amazing if this limit could shift upwards in a future release ;)

bonemap

@robhblack

We will have to make a feature request for this.

Best wishes

Russell

mark

Dear All,

So I recompiled the Text Parser to have a buffer of 256K instead of 2k. It solves your problem. But the issue on MacOS is, if you install it, the "code signature" of the app will become invalid and -- depending on the version of macOS you're running -- it might not allow Isadora to run. On Windows, it should not be an issue.

So, attached you will find the Mac and Windows plugins in a .zip file. Follow the instructions about Manually Installing a Plugin which say Isadora 1 + 2, but are valid for Isadora 3 too.

Text Parser.zip

If the operating system refuses to launch Isadora after installing the plugin, re-install Isadora using the standard installer.

I hope it helps.

Best Wishes,

Mark

photogramdude

@mark said:

Manually Installing a Plugin

IT WORKS!!!

Much appreciated, I'll dive back into this tomorrow and share progress.

bonemap

@mark said:

I hope it helps.

Thanks @mark

I was able to get the new module to parse the data to the last line. I am using Mac OS Mojave and Isadora 3.07 stable.
I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

I have marked the data and string in red on this test patch: smartcitizenParse_degreeSymbol.izz

I have a Mac OS Catalina machine arriving in a couple of days, I might be onto other things by then otherwise I will try the new Text Parser there as well.

Best Wishes

Russell

photogramdude

@bonemap Thanks, this is what I came up with, the degree symbol is not posing a problem albeit my iMac has pretty much latest MacOS. Your version is more elegant with less redundancies. My plan was to torture test with a few different sensor networks (some have soil sensors etc so would need a javascript logic based interpreter to grab additional individual component sensors. currentcitizen.izz

bonemap

@robhblack said:

the degree symbol is not posing a problem

It could be the cause of my ° symbol issue is that my keyboard is set to 'Australian'? I tried your string but still not parsing it.

I have also put together a 'dynamic data' demonstration patch that uses open data from the World Bank: DynamicData.zip

best wishes

Russell

mark

@bonemap said:

I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

Yes, this is no going to work because this actor is not really unicode savvy. (It converts all strings to UTF-8, and the ° symbol ends up being more than one byte long in that reprepsentation. So you can't actually include it in the character set you're using inside of the square brackets (i.e., [-\" a-z _ A-Z 0-9:[]] )

But I feel in general that the technique you're using to skip a fixed number of bytes is maybe not the best way to do it. It would be better to parse for the label and then the data after it. Take a look at the simplified example file below:

This is the input string

{"description":"Temperature","unit":"ºC"}

Here is the parsing code with comments (which cannot of course be included in the real parsing string.)

"{" -- skip the starting bracket
"\"description\":" -- skip text "description":
"\"" -- skip opening double quote
desc : string=[^\"] -- accept all characters until the next double quote and output that string to the desc output
"\"" -- skip ending double quote
"," -- skip comma
"\"unit\":" -- skip text "unit:"
"\"" -- skip opening double quote
unit : string=[^\"] -- accept all characters until the next double quote and output that string to the unit output
"\"" -- skip ending double quote
"}" -- skip ending bracket

as you can see, this correctly grabs "°C" because it is not depending on matching particular characters like a-z, etc.

You could write this even more elegantly as follows

"{"
"\"" [^\"] "\":" -- skip any text in the form "xxx":
"\"" desc : string=[^\"] "\"" -- grab a parameter in the form "xxx" and output as desc
"," -- skip a comma
"\"" [^\"] "\":" -- skip any text in the form "xxx":
"\"" unit : string=[^\"] "\""-- grab a parameter in the form "xxx" and output as unit
"}"

Maybe this will allow you to improve you parsing and grab the stuff you need.

Best Wishes,
Mark

parsing-suggestion.izz

bonemap

@mark said:

this will allow you to improve you parsing

Hi @mark

Yes it will allow some improvement. That is a much appreciated insight. I did attempt something like that but will definitely give it another go.

best wishes

Russell

mark

@robhblack and @bonemap

I've just made your life a lot easier.

Get the new JSON Parser Actor

Best Wishes,
Mark

bonemap

@mark said:

Get the new JSON Parser Actor

Hi Mark,

That is remarkable - I am spinning!!

Best wishes

Russell