[SOLVED] Input Parser size limit?

[bonemap]

@robhblack said:

it has just given up on a virtually identical section

 Hi Rob,

SmartCitizen looks very interesting thanks for introducing it here

If there is a limitation only @mark or @DusX will have the answer to enlighten us.

EDIT: I had a go at parsing the data but could not get any further than the same point indicated in your screen grab. Patch 3.07 attached: SmartcitizenParse.izz

Best Wishes

Russell

[photogramdude]

Hi,

Yeah Smart Citizen have sensors all over the world, and we want to tell the story of air pollution and biometrics (maybe with Cycling '74 MAX > Apple Watch) parsed input in a nice visual way.

Thanks for this huge contribution, I think logic would dictate Isadora Input Parser is setting an arbitrary KB limit on the length of the text field it parses. I can see that it all comes in, but then it falls over! I guess there could be a dev option to increase this?

In the meantime the workaround is to cleave the input in two somehow?

Cheers for this big help,

Rob

[bonemap]

Hi Rob,

I believe there is an alternative using json data and the Isadora javascript actor to parse the data stream into the patch. I am not able to advise you on the json method. unfortunately I am not proficient enough and am still learning.

When working with the data I did have n issue with parsing the degree symbol for a temperature field. I see from your screen grab that your field with the degree symbol has parsed OK. I am wondering why that is. Are you using a Mac or PC? I can’t think of why I couldn’t get that to work.

Best wishes

Russell 

[photogramdude]

Hi Russell, yesterday I used the Hello OSC method from DusX which I got installed on my Mac OK, could see cabin pressure on my browser but Isadora itself steadfastly refused to pick up the input, rage quit after a few hours!

We're looking at Node Red.

I'm using a Mac, was wondering about making some of the fields more agnostic (was trying to strictly typecast so that variable data input eg null values using 2 bytes instead of 3 because no quotes doesn't make it fall over.

The documentation is simultaneously expansive but I must admit to struggling!

I'll look into the JSON method after trying to restore some sanity, but if I break the back of this it's massive since APIs have always been this monolithic terrifying concept for me!

Best,

R

[photogramdude]

Just checked, the input runs dry after 2048 characters or 2K. So it would be amazing if this limit could shift upwards in a future release ;)

[bonemap]

@robhblack

We will have to make a feature request for this.

Best wishes

Russell

[mark]

Dear All,

So I recompiled the Text Parser to have a buffer of 256K instead of 2k. It solves your problem. But the issue on MacOS is, if you install it, the "code signature" of the app will become invalid and -- depending on the version of macOS you're running -- it might not allow Isadora to run. On Windows, it should not be an issue.

So, attached you will find the Mac and Windows plugins in a .zip file. Follow the instructions about Manually Installing a Plugin which say Isadora 1 + 2, but are valid for Isadora 3 too. 

Text Parser.zip

If the operating system refuses to launch Isadora after installing the plugin, re-install Isadora using the standard installer.

I hope it helps.

Best Wishes,

Mark

[photogramdude]

@mark said:

Manually Installing a Plugin

 IT WORKS!!!

Much appreciated, I'll dive back into this tomorrow and share progress.

[bonemap]

@mark said:

I hope it helps.

 Thanks @mark 

I was able to get the new module to parse the data to the last line. I am using Mac OS Mojave and Isadora 3.07 stable.
I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

I have marked the data and string in red on this test patch: smartcitizenParse_degreeSymbol.izz

I have a Mac OS Catalina machine arriving in a couple of days, I might be onto other things by then otherwise I will try the new Text Parser there as well.

Best Wishes

Russell

[photogramdude]

@bonemap Thanks, this is what I came up with, the degree symbol is not posing a problem albeit my iMac has pretty much latest MacOS. Your version is more elegant with less redundancies. My plan was to torture test with a few different sensor networks (some have soil sensors etc so would need a javascript logic based interpreter to grab additional individual component sensors. currentcitizen.izz

[bonemap]

@robhblack said:

the degree symbol is not posing a problem

 It could be the cause of my ° symbol issue is that my keyboard is set to 'Australian'? I tried your string but still not parsing it. 

I have also put together a 'dynamic data' demonstration patch that uses open data from the World Bank: DynamicData.zip

best wishes

Russell

[mark]

@bonemap said:

I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

 Yes, this is no going to work because this actor is not really unicode savvy. (It converts all strings to UTF-8, and the ° symbol ends up being more than one byte long in that reprepsentation. So you can't actually include it in the character set you're using inside of the square brackets (i.e., [-\" a-z _ A-Z 0-9:[]] )

But I feel in general that the technique you're using to skip a fixed number of bytes is maybe not the best way to do it. It would be better to parse for the label and then the data after it. Take a look at the simplified example file below:

This is the input string

{"description":"Temperature","unit":"ºC"}

Here is the parsing code with comments (which cannot of course be included in the real parsing string.)

"{" -- skip the starting bracket
"\"description\":" -- skip text "description":
"\"" -- skip opening double quote
desc : string=[^\"] -- accept all characters until the next double quote and output that string to the desc output
"\"" -- skip ending double quote
"," -- skip comma
"\"unit\":" -- skip text "unit:"
"\"" -- skip opening double quote
unit : string=[^\"] -- accept all characters until the next double quote and output that string to the unit output
"\"" -- skip ending double quote
"}" -- skip ending bracket

as you can see, this correctly grabs "°C" because it is not depending on matching particular characters like a-z, etc.

You could write this even more elegantly as follows

"{"
"\"" [^\"] "\":" -- skip any text in the form "xxx":
"\"" desc : string=[^\"] "\"" -- grab a parameter in the form "xxx" and output as desc
"," -- skip a comma
"\"" [^\"] "\":" -- skip any text in the form "xxx":
"\"" unit : string=[^\"] "\""-- grab a parameter in the form "xxx" and output as unit
"}"

Maybe this will allow you to improve you parsing and grab the stuff you need.

Best Wishes,
Mark

parsing-suggestion.izz

[bonemap]

@mark said:

this will allow you to improve you parsing

 Hi @mark 

Yes it will allow some improvement. That is a much appreciated insight. I did attempt something like that but will definitely give it another go.

best wishes

Russell

[mark]

@robhblack and @bonemap

I've just made your life a lot easier.

Get the new JSON Parser Actor

Best Wishes,
Mark

[bonemap]

@mark said:

Get the new JSON Parser Actor

 Hi Mark,

That is remarkable - I am spinning!! 

Best wishes

Russell