Arduino Firmata Actor

mark

[EDIT: to all and @Woland and @jfg -- I copied these comments from Tony from the Isadora User Group Global on Facebook. He did not post them in this thread.]

@tonyschuite said:

I have been looking at the instructions. First off: is this necessary? I know Firmata can do a board capabilities request (sysex 0x6B, i believe). If you would add an active flag, you can have the actor send a BoardCapabilities Request when enabling. with the response you could populate the inputs and outputs of the actor. that way everything is selectable through options. Especially since this is an "Arduino for Non-Programmers"-actor.

Otherwise as a programmer, this solution also seems quite straightforward.

I see what you're saying, but the issue is that the "capabilities" information offers all the capabilities of the board -- that doesn't indicate what you want to do. For example, PIN 2 on my board offers all these capabilities:

Pin 2: digital input; digital output; servo; input_pullup

So, it can be either a digital input or an output, which means the Isadora actor would have a corresponding output property or an input property. But which one does the user want?

If you were to show every pin on the board in the actor, you'd get a long list that would look a little bit like the Firmata test app shown below.

If you were to auto-populate the actor, a way similar to the app shown above, there would be two inputs for each pin: the function and the value. That means 34 inputs on the actor given the list shown above. If you changed the function from output to input, you'd need to move the actual value from the input side of the actor to the output side.

I guess that just felt unwieldy. But I can also see that by auto populating the actor, it would be easier for the non-programmer.

What would you suggest Tony?

Best Wishes,
Mark

mark

@tonyschuite

I guess you could have a "wizard" that asked you what you want to do for each pin. But that then involves create a new Window and a bunch of other programming.

There is however the possibility of populating the specification with all inputs based on the board's capabilities, and letting the user edit that list once it has been generated.

Best Wishes,
Mark

Juriaan

@mark

Nice one! Really love the fact that Firmata support is coming to Isadora, makes it way easier for students to work with Arduino / microcontrollers in general.

Regarding your question;

Leave the link out from https://roboticsbackend.com/ar... it makes it more complicated than it actually is. A nice support article on the Troikatronix website makes more sense. The bottom part about the Pyfirmata / etc is just not necessary.
The format makes sense, but I have to say that perhaps my opinion is not as important. Ask a choreographer/dancer or it makes sense since those are the ones that should feel comfortable with actually using the Actor.
I will ask some of my students to reply to the docs / or they find it easy to understand when the actor is out / we can use it in the wild.

fubbi

A great idea Mark. Firmata also plays nice with Processing so lots of potential

F

bonemap

@mark

I have not used the Firmata protocol before but have managed a few simple projects using Arduino boards. I have Uno and Micro boards that have been integrated into the Isadora workflow with simple sketches, things like triggering relays on/off, hacking remote controller to turn appliances on/off and simple sensor input like pulse sensor and accelerometers etc. For the Arduino sketches I have relied on modification of sketch code provided through Arduino forums etc. For more elaborate projects I have worked with experienced programmers to create Isadora based communications using OSC and wireless boards. I have plans for more ambitious projects for festivals in the August/September season this year, so the potential of what I understand from reading up about the Firmata protocol over the last couple of days has some appeal.

From what I interpret from the Firmata Actor Help document that has been provided, the Arduino Firmata actor will be somewhat similar to the existing Serial In Watcher and Send Serial Data group of actors but within a single actor integration. The Arduino Firmata actor would expand dependent on how many PIN lines are articulated within an edit text parser pop out. Is this correct? or does the integration suggest that active pins on the board would be auto detected and PIN lines auto generated?

Personally I would like to have the extended properties and parameters associated with the integration presented as a dialog interface when double clicking on the actor. Examples of that kind of control interface can be found in the AU audio actors. With addition of specifying in and out link ports on the actor itself. I don't think there is any example within the native Isadora actors. I guess this represents some kind of 'wizard' interface that would allow the display and modification of properties and parameters associated with the board identified with a serial port.

If needs be, as a non-programmer I would spend time learning the communication relationship between the Arduino board and an edit text parser style interface, because I have the need to undertake projects that would value the efficiency dividend provided by the Firmata integration. However it comes to pass, access to the board elements in a simplified but clearly articulated integration that works is going to be the bottom line.

The need to get data to and from performers has led to an investment in wireless boards and protocols and these systems we have developed with NodeMCU boards and OSC implementation. What would be the scope of working with wireless/bluetooth with the Firmata integration?

Best Wishes

Russell

liminal_andy

It would be an interesting plugin to have an official Isadora firmware to flash onto the Arduino that is essentially paired with a user actor that maps all TX pins to actor inputs and all RX pins to actor outputs. This would allow you to write voltages to pins directly from Isadora and read them back into Isadora as well, with all the communications handled for you to help out our non-programmer friends. This would limit the Arduino but would be a fast quickstart for those who just want to get data to and from their Isadora patch, and it should not require anything special as I think it can all be done with the existing serial communications tools.

Does this already exist? If not, could be a nice weekend project for me, if I ever reclaim a weekend!

mark

@liminal_andy said:

t would be an interesting plugin to have an official Isadora firmware to flash onto the Arduino that is essentially paired with a user actor that maps all TX pins to actor inputs and all RX pins to actor outputs.

My response here relates also to what @bonemap said above.

The problem with this plan is that many pins on the board can function as inputs or as outputs. Furthermore, how can we know if pin 5 is connected to an LED for which requires an on or off digital output, or to a servo motor output which requires a 14 bit value? The physical hardware connections would determine how you want the pins to function, and -- unless I'm missing something -- there is no way for Isadora (or any other software) to sense this hardware design.

[EDIT: we now have our own version of the Firmata Test App that is compatible with Big Sur. If you're on any version of macOS, you should be using this version. The latest version can always be found in the releases section of Firmata Test App on GitHub]

Downloading and trying the Firmata Test App might be informative. You'll see that you, as the user, need to choose what the function and direction of each pin is. It is not automatic.

The only way to implement Andy's plan is to force certain pins to be dedicated to certain modes and say to the user "pin 2 is digital input, pin 3 is digital output, pin 5 is for pulse width modulation output", etc. But wouldn't this be too limiting?

That said, the pin specification I've proposed above above could do exactly that by offering a pre-defined list of pin specifications as a default.

Thoughts?

Best Wishes,
Mark

liminal_andy

@mark Totally understand your point. We would be limiting the hardware dramatically by making these choices for the user. My thinking is that such a choice would allow me to hand a board to someone and say, "Here's the board, here's the actor, let's play." An early user may not need all of this functionality, they just need some PWM pins and some sense capabilities. When they are ready to grow out of the presets we created, they at least have the foundation. It's training wheels, but I feel strongly about getting more artists ready to use low level hardware because of the implications of physically reactive design.

mark

@liminal_andy said:

We would be limiting the hardware dramatically by making these choices for the user. My thinking is that such a choice would allow me to hand a board to someone and say, "Here's the board, here's the actor, let's play." An early user may not need all of this functionality, they just need some PWM pins and some sense capabilities.

Alright, I'm hearing you. Duly noted.

Best Wishes,
Mark

bonemap

@mark

Hi,

But the Firmata implementation does not replace the Serial interface group of actors -Serial In Watcher, Send Serial etc. does it? So in the overall scope of Isadora’s capability to interface with Arduino a simplified approach with Firmata will have no loss of extended functionality. As has been commented by a number of responses in this thread - the implementation is aimed at non-programmers and students/beginners. Some loss of extended functionality/limitation as a compromise towards simplicity and ease of use or as a quick prototyping, or introductory step to using a microcontroller with Isadora, is worth considering. Even as a non-programmer I know that I can turn to/take the next step towards serial functionality if I ever reach that level of sophistication with interfacing microcontrollers with Isadora.

So I think there is some merit in reflecting on a direct and albeit limited approach for the sake of plug and play simplicity.

Best wishes

Russell

mark

@bonemap said:

But the Firmata implementation does not replace the Serial interface group of actors -Serial In Watcher, Send Serial etc. does it?

No it does not replace them, it is an addition to the existing serial actors. This new actor will handle all the serial input and output internally.

We are always (painfully) mindful of backwards compatibility. We'd never take away an actor as fundamental as that.

Best Wishes,
Mark

bonemap

@mark said:

We'd never take away an actor as fundamental as that.

And I would say there is good, workable documentation and instruction for using Arduino with the Serial In Watcher, Send Serial Data actors. It is available through Isadora knowledge base. At least enough for a non-programmer like myself to succeed.

Best wishes

Russell

fubbi

What are the advantages of serial communication for the opportunistic prototyper?

I believe the future of accessible micro-controlling is ESP32 (web server, bluetooth low energy, processing power) and Teensy (Out of the box HID or Midi interface).

mark

@fubbi said:

What are the advantages of serial communication for the opportunistic prototyper?

That's a separate discussion, which you can most certainly start a thread about. Let's stick to the question at hand, OK? (Most Arduinos come with a USB serial interface connector that appears as a serial connection, as you know.)

Best Wishes,
Mar

mark

@bonemap said:

And I would say there is good, workable documentation and instruction for using Arduino with the Serial In Watcher, Send Serial Data actors. It is available through Isadora knowledge base. At least enough for a non-programmer like myself to succeed.

I'm mostly going on the multiple requests I've gotten on this over the years. On the Facebook post I made about this, several educators were immediately enthusiastic, so I would see merit in finishing this up. (It was basically two days of time while I waited for the team to test other stuff.)

Best Wishes,
Mark

bonemap

@mark

Super happy to get hands on with the Arduino Firmata implementation - if it is ready let’s try it.

Best wishes

Russell

tonyschuite

@fubbi

by knowing about ESP32 you already know about the Arduino more than the focus group for this actor and I would recommend just using the serial actors.

@marc

Reading the comments here, I think there is some good insights. as to the practical question of how you would implement an auto populated actor: after populating you would set the ins and outs to "none". If you then set one of the pins to be output, you disable the input field and vice versa, so it cannot be modified anymore.

Also, in regards to it being too many options: there are a fair few actors that have this. I don't think it is that much of a worry. however, you could go the route of only show the first 4 pins and add in the "help" section that you can enable more capabilities.

mark

@tonyschuite @bonemap @liminal_andy @juriaan

So, I've listened to all you have to say, and made substantial changes to the actor. I've explained how it's working now in this video, as it was easier to do this than to write it all out.

Please let me know if you think will help make this accessible to beginners and students who might use this actor.

Best Wishes,
Mark

Juriaan

@mark

Chapeau. That looks amazing and simple enough for students/people that never touched Arduino before. Just tell the system what you want, and be done with it :)

I could not wish for anything more, I love that it also labels the names of the Input / Output of the actor with the Pin / Type of pin that you generated. PWM, Analog, Digital, Encoder and Servo.

bonemap

@mark said:

I've explained how it's working now in this video

That appears very straight forward! nice solution with the innovation of the button to generate/automate connection ports, which I don't recall being apart of any other Isadora actor. Is it just Arduino's or are other boards suitable?

So the process would be grab the Firmata Standard Arduino Code and load it onto whatever flavour of board is at hand - using the Arduino IDE or similar. Connect the board through USB to the Serial Setup in Isadora as usual. Run the Firmata actor and check for a connection by observing the 'firmware' output. Determine what input and output ports are required for the project, enter these parameters and click the generate button - start linking data throughput with the rest of the patch.

Great !

What happens if I decide I need more ports after I have generated the initial ones can I go back and add through the same process or do I need to add through the edit text parser?

Best Wishes

Russell