Arduino Firmata Actor

bonemap

I have not used the Firmata protocol before but have managed a few simple projects using Arduino boards. I have Uno and Micro boards that have been integrated into the Isadora workflow with simple sketches, things like triggering relays on/off, hacking remote controller to turn appliances on/off and simple sensor input like pulse sensor and accelerometers etc. For the Arduino sketches I have relied on modification of sketch code provided through Arduino forums etc. For more elaborate projects I have worked with experienced programmers to create Isadora based communications using OSC and wireless boards. I have plans for more ambitious projects for festivals in the August/September season this year, so the potential of what I understand from reading up about the Firmata protocol over the last couple of days has some appeal.

From what I interpret from the Firmata Actor Help document that has been provided, the Arduino Firmata actor will be somewhat similar to the existing Serial In Watcher and Send Serial Data group of actors but within a single actor integration. The Arduino Firmata actor would expand dependent on how many PIN lines are articulated within an edit text parser pop out. Is this correct? or does the integration suggest that active pins on the board would be auto detected and PIN lines auto generated?

Personally I would like to have the extended properties and parameters associated with the integration presented as a dialog interface when double clicking on the actor. Examples of that kind of control interface can be found in the AU audio actors. With addition of specifying in and out link ports on the actor itself. I don't think there is any example within the native Isadora actors. I guess this represents some kind of 'wizard' interface that would allow the display and modification of properties and parameters associated with the board identified with a serial port.

If needs be, as a non-programmer I would spend time learning the communication relationship between the Arduino board and an edit text parser style interface, because I have the need to undertake projects that would value the efficiency dividend provided by the Firmata integration. However it comes to pass, access to the board elements in a simplified but clearly articulated integration that works is going to be the bottom line.

The need to get data to and from performers has led to an investment in wireless boards and protocols and these systems we have developed with NodeMCU boards and OSC implementation. What would be the scope of working with wireless/bluetooth with the Firmata integration?

Best Wishes

Russell

liminal_andy

It would be an interesting plugin to have an official Isadora firmware to flash onto the Arduino that is essentially paired with a user actor that maps all TX pins to actor inputs and all RX pins to actor outputs. This would allow you to write voltages to pins directly from Isadora and read them back into Isadora as well, with all the communications handled for you to help out our non-programmer friends. This would limit the Arduino but would be a fast quickstart for those who just want to get data to and from their Isadora patch, and it should not require anything special as I think it can all be done with the existing serial communications tools.

Does this already exist? If not, could be a nice weekend project for me, if I ever reclaim a weekend!

mark

@liminal_andy said:

t would be an interesting plugin to have an official Isadora firmware to flash onto the Arduino that is essentially paired with a user actor that maps all TX pins to actor inputs and all RX pins to actor outputs.

My response here relates also to what @bonemap said above.

The problem with this plan is that many pins on the board can function as inputs or as outputs. Furthermore, how can we know if pin 5 is connected to an LED for which requires an on or off digital output, or to a servo motor output which requires a 14 bit value? The physical hardware connections would determine how you want the pins to function, and -- unless I'm missing something -- there is no way for Isadora (or any other software) to sense this hardware design.

[EDIT: we now have our own version of the Firmata Test App that is compatible with Big Sur. If you're on any version of macOS, you should be using this version. The latest version can always be found in the releases section of Firmata Test App on GitHub]

Downloading and trying the Firmata Test App might be informative. You'll see that you, as the user, need to choose what the function and direction of each pin is. It is not automatic.

The only way to implement Andy's plan is to force certain pins to be dedicated to certain modes and say to the user "pin 2 is digital input, pin 3 is digital output, pin 5 is for pulse width modulation output", etc. But wouldn't this be too limiting?

That said, the pin specification I've proposed above above could do exactly that by offering a pre-defined list of pin specifications as a default.

Thoughts?

Best Wishes,
Mark

liminal_andy

@mark Totally understand your point. We would be limiting the hardware dramatically by making these choices for the user. My thinking is that such a choice would allow me to hand a board to someone and say, "Here's the board, here's the actor, let's play." An early user may not need all of this functionality, they just need some PWM pins and some sense capabilities. When they are ready to grow out of the presets we created, they at least have the foundation. It's training wheels, but I feel strongly about getting more artists ready to use low level hardware because of the implications of physically reactive design.

mark

@liminal_andy said:

We would be limiting the hardware dramatically by making these choices for the user. My thinking is that such a choice would allow me to hand a board to someone and say, "Here's the board, here's the actor, let's play." An early user may not need all of this functionality, they just need some PWM pins and some sense capabilities.

Alright, I'm hearing you. Duly noted.

Best Wishes,
Mark

bonemap

@mark

Hi,

But the Firmata implementation does not replace the Serial interface group of actors -Serial In Watcher, Send Serial etc. does it? So in the overall scope of Isadora’s capability to interface with Arduino a simplified approach with Firmata will have no loss of extended functionality. As has been commented by a number of responses in this thread - the implementation is aimed at non-programmers and students/beginners. Some loss of extended functionality/limitation as a compromise towards simplicity and ease of use or as a quick prototyping, or introductory step to using a microcontroller with Isadora, is worth considering. Even as a non-programmer I know that I can turn to/take the next step towards serial functionality if I ever reach that level of sophistication with interfacing microcontrollers with Isadora.

So I think there is some merit in reflecting on a direct and albeit limited approach for the sake of plug and play simplicity.

Best wishes

Russell

mark

@bonemap said:

But the Firmata implementation does not replace the Serial interface group of actors -Serial In Watcher, Send Serial etc. does it?

No it does not replace them, it is an addition to the existing serial actors. This new actor will handle all the serial input and output internally.

We are always (painfully) mindful of backwards compatibility. We'd never take away an actor as fundamental as that.

Best Wishes,
Mark

bonemap

@mark said:

We'd never take away an actor as fundamental as that.

And I would say there is good, workable documentation and instruction for using Arduino with the Serial In Watcher, Send Serial Data actors. It is available through Isadora knowledge base. At least enough for a non-programmer like myself to succeed.

Best wishes

Russell

fubbi

What are the advantages of serial communication for the opportunistic prototyper?

I believe the future of accessible micro-controlling is ESP32 (web server, bluetooth low energy, processing power) and Teensy (Out of the box HID or Midi interface).

mark

@fubbi said:

What are the advantages of serial communication for the opportunistic prototyper?

That's a separate discussion, which you can most certainly start a thread about. Let's stick to the question at hand, OK? (Most Arduinos come with a USB serial interface connector that appears as a serial connection, as you know.)

Best Wishes,
Mar

mark

@bonemap said:

And I would say there is good, workable documentation and instruction for using Arduino with the Serial In Watcher, Send Serial Data actors. It is available through Isadora knowledge base. At least enough for a non-programmer like myself to succeed.

I'm mostly going on the multiple requests I've gotten on this over the years. On the Facebook post I made about this, several educators were immediately enthusiastic, so I would see merit in finishing this up. (It was basically two days of time while I waited for the team to test other stuff.)

Best Wishes,
Mark

bonemap

@mark

Super happy to get hands on with the Arduino Firmata implementation - if it is ready let’s try it.

Best wishes

Russell

tonyschuite

@fubbi

by knowing about ESP32 you already know about the Arduino more than the focus group for this actor and I would recommend just using the serial actors.

@marc

Reading the comments here, I think there is some good insights. as to the practical question of how you would implement an auto populated actor: after populating you would set the ins and outs to "none". If you then set one of the pins to be output, you disable the input field and vice versa, so it cannot be modified anymore.

Also, in regards to it being too many options: there are a fair few actors that have this. I don't think it is that much of a worry. however, you could go the route of only show the first 4 pins and add in the "help" section that you can enable more capabilities.

mark

@tonyschuite @bonemap @liminal_andy @juriaan

So, I've listened to all you have to say, and made substantial changes to the actor. I've explained how it's working now in this video, as it was easier to do this than to write it all out.

Please let me know if you think will help make this accessible to beginners and students who might use this actor.

Best Wishes,
Mark

Juriaan

@mark

Chapeau. That looks amazing and simple enough for students/people that never touched Arduino before. Just tell the system what you want, and be done with it :)

I could not wish for anything more, I love that it also labels the names of the Input / Output of the actor with the Pin / Type of pin that you generated. PWM, Analog, Digital, Encoder and Servo.

bonemap

@mark said:

I've explained how it's working now in this video

That appears very straight forward! nice solution with the innovation of the button to generate/automate connection ports, which I don't recall being apart of any other Isadora actor. Is it just Arduino's or are other boards suitable?

So the process would be grab the Firmata Standard Arduino Code and load it onto whatever flavour of board is at hand - using the Arduino IDE or similar. Connect the board through USB to the Serial Setup in Isadora as usual. Run the Firmata actor and check for a connection by observing the 'firmware' output. Determine what input and output ports are required for the project, enter these parameters and click the generate button - start linking data throughput with the rest of the patch.

Great !

What happens if I decide I need more ports after I have generated the initial ones can I go back and add through the same process or do I need to add through the edit text parser?

Best Wishes

Russell

mark

@bonemap said:

Are other boards suitable?

This actor relies the Firmata protocol, which allows things like asking the board what it's capabilities are (e.g. how many pins do you have? what can they do?) So if the board (whatever it is) runs Firmata, it would work.

So the process would be grab the Firmata Standard Arduino Code and load it onto whatever flavour of board is at hand - using the Arduino IDE or similar. Connect the board through USB to the Serial Setup in Isadora as usual. Run the Firmata actor and check for a connection by observing the 'firmware' output. Determine what input and output ports are required for the project, enter these parameters and click the generate button - start linking data throughput with the rest of the patch.

Yes, that's it.

What happens if I decide I need more ports after I have generated the initial ones can I go back and add through the same process or do I need to add through the edit text parser?

Well, right now it would overwrite whatever's there. If you've not modified it by hand, no biggie. But otherwise you'd lose your changes.

But if @tonyschuite agrees that this would make it easy for the students he teachers, I'll get this ready for distribution so people can try it.

Best Wishes,
Mark

jfg

@mark

Great and very useful. A lot of students will love it.

best

Jean-François

DusX

@mark this is awesome!!! Really fantastic!!

bonemap

@mark

Hi,

The Actor is amazing as is but Some additional reflecting on the procedure outlined in your video. There is an element of guess work in letting the actor report back on the availability of ports. I think I would want to know what a board is capable of before making the setting port parameters and generating them. If the board rejected a parameter setting I would want to know why.

A little online research into the specification of a particular board would probably be all that is required. Another approach might be to consider having an option to reveal the full capacity of the board through the actor- so there is some clear guideline to what the board is capable of before setting port assignments. Could the process be, using the existing actor as proposed, to trigger settings with an aim to reveal all of the the boards potential ports and then pair back/assign the ports on a second pass?

Or perhaps a dialog is populated with the boards port type and capacity when I select the port number... then I would be equiped with the information required to make appropriate port assignments for the board..

The demonstrated actor is great as is - so these are just additional thoughts.

Best wishes

Russell