Controlling smart home devices with Isadora

declanjh

I'm at the beginning of a new theater project that will use AI to ‘direct’ the movements of stage elements (there will be no performers on stage). Just rising curtains, sliding panels, light changes, generated score, ventilators, smoke machines etc.

We want to use Isadora as the core of this automation - we have to figure out a way text input from an AI can be received by Isadora and used to trigger senes/moments/actions on stage. (any tips?)
But what we’re focusing on this first week of ‘rehersals’ is how Isadora can send commands to these different motors, lights, objects on stage.

For example we will use dmx and enttec for lighting. And also a mixture of osc and serial. This I have done before and should be fine.

But something that would make this project much easier is to be able to send commands to smart home devices - for example a smart curtain rail - from Isadora.

Is this is at all possible? Does anyone have any tips for how where I could start to research. I’m a bit clueless when it comes to the different smart home protocols….

I've looked into Homebridge and Homebridge OSC, but haven't had any luck. I dont have any coding experience and I fear this is beyond my current skillset

I'm running Isadora 3.2.6 on a Mac Studio (MacOS 15.0.1). So far we have smart devices that work with a mixture of Alexa, google home, and apple HomeKit.

Woland

@declanjh said:

Is this is at all possible? Does anyone have any tips for how where I could start to research. I’m a bit clueless when it comes to the different smart home protocols….

It's certainly possible, but interfacing with smart devices via APIs is generally not a simple endeavor unless one has some experience writing code.

APIs

Any smart device that has API documentation can likely be interacted with via Python and can therefore likely be triggered via Isadora's Pythoner actor.
The GET/POST URL actor may also be able to interact with any of these devices that have a REST API.
- You can right-click either of these two actors in the Scene Editor of Isadora and select the top option "Show Actor Help in Browser" to see more extensive documentation on how to use them.
As best I can tell from a quick Google search:
- Alexa APIs: https://developer.amazon.com/en-US/docs/alexa/rest-apis/rest-apis.html
- Google Home APIs: https://developers.home.google.com/apis
- Apple HomeKit APIs: https://developer.apple.com/documentation/homekit/
I've found that Claude and ChatGPT can be very useful for learning what specific parts of a code snippet do, asking questions about APIs/documentation, troubleshooting errors, and getting suggestions for how to structure my code based on a description of the functionality I want.
- It's good for workshopping straightforward, simple code but can often run into problems when you're working out the kinks in complex code that has lots of functionality and conditional statements.
- What's most useful about it to me is that I can write to it using the same sentence structure as I would to converse with a human, (e.g. " Can you help me troubleshoot? I am trying to write code that does X, Y, but I get this error [ERROR], when I run this code [CODE]. What does this error mean and how might I try to fix it?"), instead of trying to google the problem in a non-sentence syntax that I think will be most likely to yield useful results.
- You may even be able to find specific models that are trained to understand how to write code that interfaces with APIs, or even the specific APIs that you're working with.
- That being said, the more ethical route is hiring a human programmer to write code for you if your budget allows for it.

Arduino

A rather cludge-y (but viable) way to do things is to hook up these smart devices to physical buttons or remotes, then use Isadora to communicate with Arduinos connected to solenoids or servo motors that physically press the buttons or complete circuits to adjust/trigger things.

Philips Hue Light Integration

I do have a Python-based integration between Isadora and Philips Hue lights (commercial-grade color-changing bulbs and strips controllable via wifi) that I commissioned a few years ago as part of a project, but as I paid for its development and the client ultimately decided not to use my solution, I did not recoup what it cost me and, as such, unlike most of my resources and tools, it's not something that I give away for free. If it's of any interest to you, let me know.

Speech to Text to Logic

@declanjh said:

we have to figure out a way text input from an AI can be received by Isadora and used to trigger senes/moments/actions on stage. (any tips?)we have to figure out a way text input from an AI can be received by Isadora and used to trigger senes/moments/actions on stage. (any tips?)

Pythoner + Speech to Text will let you get text into Isadora, which you could then use machine learning in Pythoner to teach it what commands it can send and the context of what text would relate to controlling what devices.
I believe @mark was working on a ChatGPT patch at one point that could convert speech to text, so I know that it's possible to use Pythoner and speech-to-text to communicate back and forth with ChatGPT, meaning you could potentially create and train a GPT to understand the system you built and to control it based on voice commands/context clues by feeding it text created via speech to text.
A few years ago I worked informally with an artist who used speech-to-text with a chatbot to send text to Isadora over OSC for a showcase performance. Here are the files I still have from that: FILES
- This was done before the Pythoner actor existed, so the Python code was run in Pycharm.
I also worked on a project called Lucy AI, part of which involved talking to a trained bot, then the bot would formulate a response and I was sent text based on that response. Since this was contracted work, I cannot share the resources I have from that project, but the main point is that I can tell you that what you're trying to do is possible because I have been part of multiple projects that have done it in one form or another.

Juriaan

The issue with Smart home devices is that all devices are speaking a different language; ZigBee / Matter / Infrared / Bluetooth LE to just name a few. And all those devices have their own app / or their own way of 'working'.

What I did in my own house, is I used a 'in between' product that allows me to use a mixture of different devices. Some people are using HomeAssistant for this; others will utilize a Raspberry Pi or a Cloud based solution.

I use a Homey Pro for that purpose, which is the 'heart' of the operation. Within that I have flows, to control all the machinery / lights / blinds / etc in my home.

Since we now have one interface, and one machine to control it becomes 'easier' to trigger those operations, by using their SDK / Web API. Which can be a great solution for the idea that you have.. Just takes a hot sec to setup / verify before you enter the rehearsal space.

Oh and please, a theater is very different than a 'home', there can be much more signals in the air / or other factors causing things to fail. Smart Home products were never meant for the theater, and require a lot of testing. Please do not use them as 100% fail-safe operations for a practical effect in theater, or an important scene where something 100% has to work. If that is the case, use the products that were designed for that purpose / and the protocols that were designed for it.