assurance-tunnel
assurance-tunnel
assurance-tunnel
assurance-tunnel

[SOLVED] Input Parser size limit?



  • Hi, I'm wrangling a large amount of text from an API URL http://api.smartcitizen.me/devices/1616

    via: Get URL text > input parser

    I'm about halfway through and it has just given up on a virtually identical section to what was processed correctly above.

    Is there a maximum size limit of field limit for the Input Parser? I've searched the manual and the forum...

    Thanks,

    Rob


  • Beta Platinum

    @robhblack said:

    it has just given up on a virtually identical section

     Hi Rob,

    SmartCitizen looks very interesting thanks for introducing it here☺

    If there is a limitation only @mark or @DusX will have the answer to enlighten us.

    EDIT: I had a go at parsing the data but could not get any further than the same point indicated in your screen grab. Patch 3.07 attached: SmartcitizenParse.izz

    Best Wishes

    Russell



  • Hi,

    Yeah Smart Citizen have sensors all over the world, and we want to tell the story of air pollution and biometrics (maybe with Cycling '74 MAX > Apple Watch) parsed input in a nice visual way.

    Thanks for this huge contribution, I think logic would dictate Isadora Input Parser is setting an arbitrary KB limit on the length of the text field it parses. I can see that it all comes in, but then it falls over! I guess there could be a dev option to increase this?

    In the meantime the workaround is to cleave the input in two somehow?

    Cheers for this big help,

    Rob


  • Beta Platinum

    Hi Rob,

    I believe there is an alternative using json data and the Isadora javascript actor to parse the data stream into the patch. I am not able to advise you on the json method. unfortunately I am not proficient enough and am still learning.

    When working with the data I did have n issue with parsing the degree symbol for a temperature field. I see from your screen grab that your field with the degree symbol has parsed OK. I am wondering why that is. Are you using a Mac or PC? I can’t think of why I couldn’t get that to work.

    Best wishes

    Russell 



  • Hi Russell, yesterday I used the Hello OSC method from DusX which I got installed on my Mac OK, could see cabin pressure on my browser but Isadora itself steadfastly refused to pick up the input, rage quit after a few hours!

    We're looking at Node Red.

    I'm using a Mac, was wondering about making some of the fields more agnostic (was trying to strictly typecast so that variable data input eg null values using 2 bytes instead of 3 because no quotes doesn't make it fall over.

    The documentation is simultaneously expansive but I must admit to struggling!

    I'll look into the JSON method after trying to restore some sanity, but if I break the back of this it's massive since APIs have always been this monolithic terrifying concept for me!

    Best,

    R



  • Just checked, the input runs dry after 2048 characters or 2K. So it would be amazing if this limit could shift upwards in a future release ;)


  • Beta Platinum

    @robhblack

    We will have to make a feature request for this.

    Best wishes

    Russell



  • Dear All,

    So I recompiled the Text Parser to have a buffer of 256K instead of 2k. It solves your problem. But the issue on MacOS is, if you install it, the "code signature" of the app will become invalid and -- depending on the version of macOS you're running -- it might not allow Isadora to run. On Windows, it should not be an issue.

    So, attached you will find the Mac and Windows plugins in a .zip file. Follow the instructions about Manually Installing a Plugin which say Isadora 1 + 2, but are valid for Isadora 3 too. 

    Text Parser.zip

    If the operating system refuses to launch Isadora after installing the plugin, re-install Isadora using the standard installer.

    I hope it helps.

    Best Wishes,

    Mark



  • @mark said:

    Manually Installing a Plugin

     IT WORKS!!!

    Much appreciated, I'll dive back into this tomorrow and share progress.


  • Beta Platinum

    @mark said:

    I hope it helps.

     Thanks @mark 

    I was able to get the new module to parse the data to the last line. I am using Mac OS Mojave and Isadora 3.07 stable.
    I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

    I have marked the data and string in red on this test patch: smartcitizenParse_degreeSymbol.izz

    I have a Mac OS Catalina machine arriving in a couple of days, I might be onto other things by then otherwise I will try the new Text Parser there as well.

    Best Wishes

    Russell



  • @bonemap Thanks, this is what I came up with, the degree symbol is not posing a problem albeit my iMac has pretty much latest MacOS. Your version is more elegant with less redundancies. My plan was to torture test with a few different sensor networks (some have soil sensors etc so would need a javascript logic based interpreter to grab additional individual component sensors. currentcitizen.izz


  • Beta Platinum

    @robhblack said:

    the degree symbol is not posing a problem

     It could be the cause of my ° symbol issue is that my keyboard is set to 'Australian'? I tried your string but still not parsing it. 

    I have also put together a 'dynamic data' demonstration patch that uses open data from the World Bank: DynamicData.zip

    best wishes

    Russell



  • @bonemap said:

    I did have one issue: I was unable to parse the ° degree symbol (shift-option-8) Isadora hard crashed most times I tried to include the symbol in the character-set range.

     Yes, this is no going to work because this actor is not really unicode savvy. (It converts all strings to UTF-8, and the ° symbol ends up being more than one byte long in that reprepsentation. So you can't actually include it in the character set you're using inside of the square brackets (i.e., [-\" a-z _ A-Z 0-9:[\]] )

    But I feel in general that the technique you're using to skip a fixed number of bytes is maybe not the best way to do it. It would be better to parse for the label and then the data after it. Take a look at the simplified example file below:

    This is the input string

    {"description":"Temperature","unit":"ºC"}

    Here is the parsing code with comments (which cannot of course be included in the real parsing string.)

    "{" -- skip the starting bracket
    "\"description\":" -- skip text "description":
    "\"" -- skip opening double quote
    desc : string=[^\"] -- accept all characters until the next double quote and output that string to the desc output
    "\"" -- skip ending double quote
    "," -- skip comma
    "\"unit\":" -- skip text "unit:"
    "\"" -- skip opening double quote
    unit : string=[^\"] -- accept all characters until the next double quote and output that string to the unit output
    "\"" -- skip ending double quote
    "}" -- skip ending bracket

    as you can see, this correctly grabs "°C" because it is not depending on matching particular characters like a-z, etc.

    You could write this even more elegantly as follows

    "{"
    "\"" [^\"] "\":" -- skip any text in the form "xxx":
    "\"" desc : string=[^\"] "\"" -- grab a parameter in the form "xxx" and output as desc
    "," -- skip a comma
    "\"" [^\"] "\":" -- skip any text in the form "xxx":
    "\"" unit : string=[^\"] "\""-- grab a parameter in the form "xxx" and output as unit
    "}"
    

    Maybe this will allow you to improve you parsing and grab the stuff you need.

    Best Wishes,
    Mark

    parsing-suggestion.izz


  • Beta Platinum

    @mark said:

    this will allow you to improve you parsing

     Hi @mark 

    Yes it will allow some improvement. That is a much appreciated insight. I did attempt something like that but will definitely give it another go.

    best wishes

    Russell



  • @robhblack and @bonemap

    I've just made your life a lot easier.

    Get the new JSON Parser Actor

    Best Wishes,
    Mark


  • Beta Platinum

    @mark said:

    Get the new JSON Parser Actor

     Hi Mark,

    That is remarkable - I am spinning!! 

    Best wishes

    Russell