[ANSWERED] Redundant Simultaneous Instances of Isadora acting as backup in case of failure ?

Bonami

Hi !

I'm exploring ways to have a second Isadora instance, on a second computer with the exact same specs, to run a show file simultaneously. 3x DisplayPorts Outputs per machine, 6x outputs total going into the Video Matrix. (This is in a critical live-show environment where redundancy is an absolute necessity) - If anything goes wrong with Computer "A", we switch to Computer "B" from the Video Matrix (this would be a manual operation from the operator.)

I tried searching on the forums but could not find any trails to follow..

I'm thinking OSC commands that send triggers from Computer A and receive the triggers on Computer B but that obviously means two different show files. I'm wondering if someone might have a better idea.

Thanks anyone for your help !!

Woland

@bonami said:

we switch to Computer "B" from the Video Matrix (this would be a manual operation from the operator.)

Doesn't have to be manual if you can control the Video Matrix through OSC, MIDI, TCP/IP, Serial, using the GET/POST URL Text actor, or with the device's own API via Python code using the beta Pythoner plugin (just send in a ticket using the link in my signature if you'd like access).

@bonami said:

I'm thinking OSC commands that send triggers from Computer A and receive the triggers on Computer B but that obviously means two different show files. I'm wondering if someone might have a better idea

I'm already working on an example file for you, but here's a basic explanation of what's coming:

Yes, OSC can be a way to go. You can also use Net Broadcaster actors and give each computer a separate Computer ID in Isadora > Preferences.../Settings... > Midi/Net > Net Setup > Computer ID.

If you want an automatic switch, you setup a "heartbeat", which is just:

On the "Leader" computer: A background Scene you keep active Wave Generator into an OSC Transmit or Net Broadcaster actor
On the "Follower" computer: An OSC Listener or Listener actor into a Trigger Delay actor into whatever it is that you want to trigger.

While the Wave Generator continues to function, it'll keep triggering the Trigger Delay actor and resetting its timer.

If the Wave Generator stops working (e.g. if Isadora crashes) then the timer on the Trigger Delay is allowed to run out and it triggers whatever is necessary to make the "Follower" become the new "Leader".

Now, you can make this as one file with a Selector that lets you tell the patch whether it's supposed to act as a leader or follower: Example: A Selector into the IP input of the OSC Transmit actor that lets you pick between the leader's IP or the Follower's IP and a Selector that closes a Gate to listen for communication from the "Leader" for the Leader computer and opens the Gate to listen for communication from the "Leader" on the Follower computer.

Woland

@bonami

Also, there's a professional AV community I'm part of called Office Hours Global, and I teach a free Isadora Lab (with no set curriculum) on Office Hours' 24/7 Zoom call every Thursday from 7pm to 8pm CET (so there will be one tonight). If you want to get your questions answered in person by me and play around with this concept in Isadora together you can register for the Zoom call here: https://zoom.us/w/96553474365?_x_zm_rtaid=BmS82XntQmSDwc5VH1xmhg.1707495640829.aa165f03228cff2c6df68f7116adcc28&_x_zm_rhtaid=189#success

I'll probably be working on this file in my lab tonight.

Best wishes,

Woland

jfg

if you can control your show from an osc program like Touch OSC this could be a solution:

https://osculator.net/forum/forum/support/extra-downloads-beta-releases/975-oscbroadcaster-send-osc-messages-to-two-computers-or-more

I haven't try it but it seems to be interssant

Best regards,

Jean-François

Bonami

@woland

Oh my god that is so extremely clever ! I would have never thought about this. This is a very creative stable and safe solution to my problem. Going to give it a go this afternoon on two separate computers to see if I can get a good grip on your excellent concept.

I'm deeply sad I missed the opportunity to join the Zoom as I look at your post a day too late. Still I registered on the website and will join on the next one :)

Thank you for your brilliant insights.

Bonami

@jfg

Oh I never thought about exploring this ! Definitely going to look into it for specific situations ! Merci beaucoup !

Trevor

I take the simple approach of the USB Go Box. Has two usb outputs, and will trigger cues on diferent machines in sync

https://www.thatlittlebox.co.uk/usb-go-box. Also, is designed as a cueing device to ensure no accidental cues.

CitizenJoe

@trevor That's clever.

Cheers,

Hugh

Woland

@woland said:

I'm already working on an example file for you, but here's a basic explanation of what's coming:

Sorry, I got swamped during the week. Here's what I have so far:

redundancy-heartbeat-switch-2024-02-15-3.2.6.izz

What's really neat is I figured out a way to just use the same patch on both the leader and follower computers with zero configuration in terms of telling which one you want to be the leader and the follower. Here's the basics of how it works:

Launch the patch on the Leader computer.
The patch checks whether or not there's a heartbeat from a Leader.
The patch doesn't see a heartbeat already, so it knows it's supposed to become the Leader.
The patch, running on the Leader computer, starts sending out a heartbeat.
Launch the patch on the Follower computer.
The patch checks whether or not there's a heartbeat from a Leader.
The patch sees a heartbeat, so it knows it's supposed to become a follower.
The patch, running on the Follower computer, starts operating as a follower.
If the heartbeat from the Leader computer stops, the patch, running on the Follower computer, will automatically become the Leader and start sending a heartbeat.
At this point, you could re-launch the patch on the original Leader computer and it would see the heartbeat and automatically start operating as a Follower.

It's totally up to you what to trigger when the switch from Leader to Follower happens, but I got pretty far in creating a mechanism for the heartbeat and triggering the passing of the baton if the Leader's heartbeat stops.

Bonami

@woland

Oh my god Woland ! Ahah this is absolutely above my expectations ! Your first answer was already very clever, I was not expecting such a detailed Izz file on top of it ! I'm navigating your patch at the moment. I had, in the meantime, created something along those lines after your reply, but nowhere near as automatic, complete, and conceptually mature as the file you provided. I'm glad I can learn from your reflexions throughout your patch. Thanks again, really.

Woland

@bonami said:

I'm navigating your patch at the moment.

My pleasure, and thank you for prompting me to make it. I hope it will be helpful to you and others. Let me know if you have any questions as I'd be happy to answer them. I also hope I'll be able to find the time to finish the file soon...

mark_m

@woland

I just want to say what valuable and important work this is: I recently lost an Isadora vs Disguise argument on a production, on the very basis that Isadora didn't have a built-in automatic fallover the way that Disguise systems do. I would rather program in Isadora than Disguise, and if I can demonstrate this kind of redundancy I wouldn't lost these arguments again :-)

Bonami

@mark_m Ahah same thing happening to me ! As I'm trying to push more and more of our live events over to Isadora (We mostly use Watchout and/or Disguise on larger productions) but for most corporate gigs, of all sizes, I would say 95% of them would be abundantly well-served with Isadora. But the last standing blockade is always the redundancy. Now @Woland created this ingenious way for me to solidly implement Isadora as a totally viable, relevant solution throughout our operations.

DillTheKraut

@bonami

I just like to second this! Coming from party/live visuals, going to commercial and musical productions, but also Opera and fine Art festivals and now at municipal theatre.
I have experience with all of them (disguise, Watchout, Wings, Pandoras, milllumin, Touchdesigner and Resolume (starting with 3.x , back in the days and moving to Izzy shortly after, because of limited layers :P ). Non of them have the combined balance of usability, lerning curve and 'give it to the client's hands' probability, as Isadora does! Call it scalability. The price/needed feature combination is far off anyways.
We now are running Wings successor Pixera and I so often miss the scene functionality and the logic possibiltys, beside alot of other issues :D.
The only real missing functionality, which is holding back the switch, are the frame synced multi server/device distribution and the somewhat oldschool GUI (which is just a psychological issue ;-) ).
For most commercial users, probably is the missing hardware complete system solutions + (official) service.

Woland

@woland said:

I also hope I'll be able to find the time to finish the file soon...

@Bonami I just finished the file in my After Hours Isadora Lab. Here's the file: redundancy-heartbeat-switch-2024-02-29-3.2.6.izz

The file and the link to a recording of me building this addition to the file will be posted in the Isadora channel of the Office Hours Discord soon.

This updated file includes an example Content Scene:

One of the things this Scene does is explain how to keep your Scene List in Synch across the two files (it's the section at the top right labeled "#Keeping the Active Scene in Synch Across Two Files").

Here's the guts of the Macro from "Method 1 - Next/Prev Scene" (I'd recommend the second method instead though):

Here's the guts of the Macro from "Method 2 - Scene Synch" (I'd recommend using this method):

It's possible I made a logic mistake somewhere, as I haven't tested extensively with two computers, but feel free to let me know if you encounter problems and I'd be happy to fix the file.

Best wishes,

Woland