[SOLVED] Ongoing Depth camera issues

Vidasonik

@vidasonik The hi res video is sent directly from the old mac with depth camera to the main projector. A replica Ableton sound set is used purely to interact with the Isadora file on the old mac that includes the depth camera /OpenNI tracker BETA in some of the scenes. Midi notes are sent from Ableton to trigger visuals etc. This means, of course, that each duplicate Ableton set must be started at the same time. So I position the two computers next to each other so I can press two keyboard keys (programmed to start the first track) at the same time to trigger the start of both audio sets. Not ideal! I use the new mac for audio. The same audio of the old mac is sent to Izzy via Blackhole because Izzy doesn't hear the internal audio without going through another audio device.

Woland

@vidasonik said:

So I position the two computers next to each other so I can press two keyboard keys (programmed to start the first track) at the same time to trigger the start of both audio sets. Not ideal!

You should be able to use a Keyboard Watcher actor on Computer 1 to:

Start Ableton on Computer 1 by sending MIDI to it
Send OSC from Computer 1 to Computer 2 to have Isadora on Computer 2 start Ableton on Computer 2

Woland

@vidasonik said:

The hi res video is sent directly from the old mac with depth camera to the main projector.

If you don't want to send video over the network, you can use a capture card to capture the video output of Computer 1 (old Mac) and bring it into Isadora via Live Capture Settings on Computer 2 (new Mac) so that you can use a Video In Watcher actor on Computer 2 (new Mac) to have all your video in the same Isadora patch.

Vidasonik

@woland a 'keyboard watcher' connected to a 'midi send' and 'OSC Transmit'? like this:

Vidasonik

@woland so a capture card is a piece of hardware? Surely using Video In watcher brings us to the same problem that you can't transmit large files over a local network?

Fred

@vidasonik

Hi @vidasonic, it sounds like you had a bad experience. Some of this thread sounds like an XY problem (https://xyproblem.info/), I definitely don’t mean offence to anyone in this thread but it is an interesting and common issue solving technical issues on forums.

It may seem like more trouble but it sounds like there may be a more robust solution to your issue that may be a little harder to implement but will give you some freedom later. It’s not exactly clear what your issues are or where they come from, but the main thing that seems evident is that you have 2 machines and only one is compatible with the depth camera. I would focus on working around that issue. From your description it seems like a good solution would be to implement the tracking with both your cameras on the older computer. Create a patch that can send any data that you need (the results of the tracking) and receive any required commands (switching scenes if need be). That tracking patch can likely run in a single scene and send and receive data via osc (use a wired connection). You can then use the tracking data as you had been on your newer computer and only have a single copy of ableton and a single machine that also outputs visuals. This is not an uncommon solution and is actually pretty sensible for distributing the workload. If you need all the actual video streams from the cameras because they are displayed on the projection then you can use NDI to send them from the older machine to the newer one. If you only need the webcam stream to send to the video projector you can consider moving that to the newer mac and only leaving the depth camera on the old mac. If none of the images from the cameras appear on the video projection then you can leave both on the older machine and just send the data.

Sometimes it can be tough to design the correct architecture for your technical solution. I would comment that continuing as you had been (trying to sync 2 machines and 2 sets of ableton) does not sound like the best solution (the y solution in an xy problem).

There are some great resources in isadora for doing this kind of multi machine setup and some excellent advice on the forum for achieving this too.

These kind of split systems can be automated and perfectly in sync with no need for manual intervention.

Let us know more details of what data needs to be where and the great brains trust of this forum will be able to guide you on a reliable solution.

Vidasonik

@fred Hi Fred thanks for your response. Yes I do need to be clear about what I am trying to do so hopefully the following will help:

Issues

It seems that owing to the impossibility of rewriting the OpenNI Tracker BETA software it will never be possible to use a depth camera with Izzy on any modern Apple Mac
As Izzy doesn’t see the sound that I output to the sound card on new Mac (it only accepts indirect sound through another device eg. Blackhole) I have to use old Mac also for sound into Izzy.
So I must use the old one and somehow hook the Macs together
I bought a router which was a total nightmare, local internet connection in theatres very unreliable. At this point I gave up trying to send live video from the old to the new. So have to connect the old Mac to projector.

For some inexplicable reason Izzy will not accept audio directly so I have to send from Ableton to Blackhole and then select Blackhole in live capture settings. I use frequency and level detectors to produce video effects. Therefore I can’t even use the new Mac for sound with Izzy. The live sound must come from the new Mac which has the Clarett+4Pre sound card connected for live guitars, and backing audio stems etc. Ableton preferences are setup with audio input and output device set to Clarett. Therefore, I also have to have the webcam (I use this also to track dancer movement with Difference actors etc) attached to the old Mac and play a duplicate Ableton file on the old Mac. I have files adjacent to the audio master stems with midi notes locked to the same beats and song length. Clearly the problem is to have them running in sync otherwise the various video FX triggered by midi notes, sound envelopes etc that have to be triggered by the Ableton file on the old Mac will not conform to the live audio coming from the new one! I also send midi notes to change scenes and so on.

This is all fine as long as I can get both setups running in sync automatically.

The idea was to have the whole set flowing without me having to press buttons to start each song so I can focus on guitar/vocals.

We will be doing more shows going forward so I need to somehow devise a robust and simple set-up if I am to carry on using the depth camera.

Thanks, David

Woland

@vidasonik said:

As Izzy doesn’t see the sound that I output to the sound card on new Mac (it only accepts indirect sound through another device eg. Blackhole) I have to use old Mac also for sound into Izzy.

You can create a Multi-Output device in your Mac's Audio MIDI Setup which outputs to your physical audio output device and Blackhole or iShowUCapture at the same time, which will let you pull the same audio back in through Isadora's Live Capture by selecting Blackhole or iShowUCapture. I've done this for multiple productions where I needed to output audio to a physical device but also needed to do virtual audio routing to get the same audio back into Isadora in order to perform audio analysis or create audio-reactive visuals.

@vidasonik said:

For some inexplicable reason Izzy will not accept audio directly

It's not that inexplicable. Virtual audio routing is easy enough to do and many tools exist for it (Loopback, iShowUCapture, Blackhole, etc) and have for a long time (I used SoundFlower for this 10+ years ago) so there's never really been a need for Isadora to be able to reinvent the wheel and make Isadora do the same thing these many purpose-built tools are designed to do.

@vidasonik said:

I bought a router which was a total nightmare, local internet connection in theatres very unreliable. At this point I gave up trying to send live video from the old to the new. So have to connect the old Mac to projector.

Getting a private network set up with a router is your best route forward for not only sending video between the computers, but also for sending the data that you'd need in order to keep your computers in synch. I know the process of getting into networking can be daunting at first, but once you can do it you'll be able to easily accomplish the task of creating systems that solve the exact problems you're having right now (e.g. multi-computer synch/multi-computer cueing systems and sending video between computers).

Vidasonik

@woland hi thanks for your reply

Points 1 and 2:

Yes, my first thought was to use Blackhole etc (I did have Loopback but that caused other issues so deleted it) however this introduced very noticeable latency so was not viable. Why won't Izzy receive audio from the Clarett soundcard?

Point 3:

I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work. Again, the depth camera should work on a modern mac.

jfg

@vidasonik said:

Point 3:I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work. Again, the depth camera should work on a modern mac.

you could try to connect both Mac with a ethernet crossover cable.

Best regards,

Jean-François

Vidasonik

@jfg sounds good! Thanks.

Woland

@vidasonik said:

a 'keyboard watcher' connected to a 'midi send' and 'OSC Transmit'? like this:

Not quite. You'd want the actors to be more like this (assuming you're using Computer 1 to trigger Ableton on Computer 2 with a Keyboard Watcher actor on Computer 1):

For more info on using Isadora to control Ableton, see this article: https://support.troikatronix.com/support/solutions/articles/13000081201-control-ableton-live-sessions-with-midi

You would also need to have the computers networked together so that you could send the OSC between them.

@vidasonik said:

@woland so a capture card is a piece of hardware?

Yes, it's a piece of hardware that, for example, could be plugged into an HDMI cable coming from computer 1 in order to get that video signal into Isadora's Live Capture as a video input device on computer 2.
Example: Computer 1 video output > computer 2's HDMI output port > HDMI cable > video capture device's HDMI input port > video capture device's USB cable > computer 2's USB port > Isadora's Live Capture Settings (selected as video input device for channel 1, for example) > Video In Watcher actor.

@vidasonik said:

Surely using Video In watcher brings us to the same problem that you can't transmit large files over a local network?

No, it wouldn't have anything to do with networking but, depending on the distance between the computers, it may not be viable to use a common video cable like HDMI/DVI and you might have to buy a repeater, extender, or use SDI, all of which is going to end up being more expensive than an ethernet switch and a couple of ethernet cables. Additionally, while it is possible to get the video signal from one computer to another using a video capture device such as a Blackmagic UltraStudio Mini Recorder or Elgato Cam Link 4k, it doesn't give you the capability to transmit any data so it doesn't solve your synch/cueing problem.

@vidasonik said:

Why won't Izzy receive audio from the Clarett soundcard?

The first thing to do is check all four of your Isadora Live Capture channels to make sure that you don't have the Clarett selected as the audio device for channels 2, 3, or 4, because if it is selected on one of those it won't show up as a selectable option in the list of audio devices in Live Capture Channel 1. If that isn't it, then my second guess would be that this might only show up as an audio output device to Isadora for some reason, but that would be very strange, as I have two Focusrite devices that work fine for audio capture into Isadora.

@vidasonik said:

I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work.

I am unclear why the recommended router wouldn't work, as it was specifically recommended because it's the exact router a member of our staff uses for precisely this purpose. I completely understand your frustration because both myself and another staff member went well outside the scope of what we officially offer in terms of technical support by providing multiple hours of consulting, video calls, programming, and troubleshooting free of charge in order to try to assist you in getting the router and networked computer setup running, as well as modifying your Isadora files to send and receive OSC and NDI. So, while I apologize that the time and energy spent on trying to get that specific router working for you didn't pay off, I would still like to stress that just because that one device wouldn't work for you doesn't mean you should abandon the entire approach. If your car breaks down and nobody can figure out what's wrong with it, that doesn't mean you should give up on driving forever and never get another car. You could pick up a 4/5-port gigabit ethernet switch for 16-25 euros and try again. You even saw that the setup worked as a proof of concept over your wifi when we were on a call together testing the specific Isadora files I made for each of your computers and at that point I said that all you'd need to do is get your own router or a switch and connect the computers to it with ethernet, and get a LAN up and running.

@vidasonik said:

local internet connection in theatres very unreliable

I also distinctly remember strongly recommending against trying to use wifi or a theatre's existing network for performances because you could easily run into any number of problems that would be outside of your control. You want your network to be locked down and private so that no outside factors can affect it. Additionally, traveling with your own router/switch means you already know that your network setup works, instead of trying to figure out something different at each new venue.

@jfg said:

you could try to connect both Mac with a ethernet crossover cable.

This is definitely worth a shot, but I'm not sure you'll be able to send NDI and OSC this way because two computers connected directly via a network cable don't necessarily always behave exactly the same as two computers both connected to a router/switch. I'm sure someone who knows more about networking than I do could explain it in better detail, but as far as I know, certain aspects of networking require a router or a switch in order to work properly.

@vidasonik said:

Again, the depth camera should work on a modern mac.

We didn't design, manufacture, nor do we sell the hardware and we are not able/allowed to alter the OpenNI code that is used to make it work because it's not ours, so we're neither able to, nor required to make it work on modern Macs. Despite having no obligation to do so, we have done our best to maintain the OpenNI plugins for years but, as far as I understand, the OpenNI code was written nearly a decade before the first M1 Mac computer even existed so there's simply not much we can do anymore. Writing our own library for this from scratch would require more time and resources than we have right now and trying to adapt a different library by somebody else would present its own problems, namely that 1) we'd have to find a library that is both good and allows for commercial use without an enormous licensing fee 2) we'd still have to invest more time and resources into adapting and testing it than we currently have, and 3) it very likely wouldn't work in the exact same way as the OpenNI library (might provide different data, differ in accuracy, not have one or more features we relied on, etc) so an existing Isadora file originally built using the OpenNI plugins would likely need significant adjustments to make it work with the theoretical new tracking plugin or, if the Isadora file required a feature that existed in OpenNI but didn't exist in the new library, it wouldn't work at all.

Best wishes,

Woland

Vidasonik

@woland Thanks so much for this comprehensive reply. Indeed you spent a lot of time with me (as you say, way beyond the call of duty, and I expressed my gratitude at the time) helping to set everything up and I remember feeling confident that a solution had been found which merely needed implementing.

I think I will need to step back from all this and reflect on everything to see if this is worth pursuing further, ie does what I am trying to do justify all this effort from yourselves as well as myself. I would hate to abandon it if I felt a solution was within reach, however. Thanks so much for bearing with me.

All the best, David

Woland

@vidasonik said:

I would hate to abandon it if I felt a solution was within reach, however.

For the record, routers tend to be trickier to set up than ethernet switches. Switches tend to be plug-and-play whereas routers can have more options to fiddle with.

Vidasonik

@woland ok can you recommend an ethernet switch?

DillTheKraut

@woland

Sorry, but as a network engineer, I have to correct your comment. There are complicated switches and simple switches, as well as simple routers/Access points and complicated setups, if used wrong. There are three common types of network connection devices: hubs, switches and routers. But there are different (sub-)types of switches, which can lead to issues, if not known.

- Hubs are very dumb devices, which blow all data from and to all ports, without any logic, which can cause them to slow down in data rich environments. There is only a few to buy nowadays, as switches became the standard.

- Switches are smarter hubs. They know which device is connected to which port and can therefore route data more efficient. But they come in two flavors: managed and UNmanaged. The later is kind of plug and play, where the managed ones try to optimize the data flow on a deeper level by default, which can be a struggle in some cases. In these situations, you need to know how to manage (pun intended ;) ) them, which needs deeper knowledge of network protocols and special network functionality.
Some managed switches as well come with very different types and numbers of extra functionalities and implemented services, which some of are like the ones routers usually provide (like DHCP). But usually these are deactivated by default.

The issue with MANAGED switches is especially true for protocols like NDI. The keyword here is multicast!

- Routers again are a special type of devices, with the main purpose of translating between two or more different networks, e.g. Internet and LAN. Some smaller wifi routers provide connectivity for two computers or even one only. The most bigger ones have an integrated hub or switch, which provides more ports to connect more devices. This should not be more complicated or different to use, like the stand alone hubs/switches, regarding their simple switching functionality.

But on top of this, they provide some extra services, like DHCP, which are active by default in most cases! Those usually do not interfere with the general network functionality. But though some CAN be used to simplify a setup, if used wrong, they can become an issue. Beside this there might be some routers with managing functionality as well as the managed switches and therefore have the same side effects. But they are usually upper class, upper price range.
Basic thing you have to be aware of, is the different types of ports they have (LAN, WAN, etc.). For pure local network functionality as a hub/switch, you would only use those ports with LAN functionality. Not the ones with WAN on it.

@Vidasonik
For a simple setup, have a look for UNmanaged gigabit switches with the number of ports needed. TP-Link, Netgear, Linksys are common brands in that class and don't have big differences in general functionality. Switches with 8 ports and less usually are unmanaged, if not otherwise stated.

If not already done, you should as well get a little bit into basic network knowledge, e.g. IP addresses and subnetmasks, as well as DHCP service.
A little hint: Manually set IPs and DHCP services (e.g. by default provided by most routers) in the same network, can clash, if not set up in a propper way.

DillTheKraut

@jfg

You nowadays don't need any crossover cables anymore as any gigabit device, be it the computers network port or the switch, are able to identify active and passive connections. You can just use any usual RJ45 Cat 5 and up cable.

These cables/adapters are from old 100mbit times, and where called uplink cables. Some hubs had a special uplink port to connect to other hubs without needing an uplink/ crossover cable. Modern switches, doesn't need those anymore.

DillTheKraut

@woland said:

This is definitely worth a shot, but I'm not sure you'll be able to send NDI and OSC this way because two computers connected directly via a network cable don't necessarily always behave exactly the same as two computers both connected to a router/switch. I'm sure someone who knows more about networking than I do could explain it in better detail, but as far as I know, certain aspects of networking require a router or a switch in order to work properly.

This is not true at all! Two computers connected by a standard network cable are just the same as with an unmanaged switch in between.

But you are right in the fact, that you need to do some basic settings on your own in both cases, if it isn't done by a specific device in the network, like a router with an DHCP Service.

DillTheKraut

I don't know what was already done in the support sessions @Woland is referring to. But while reading these comments it seems to me, that there might be some basic network setup issues.

I might state one important basic: If you don't have a DHCP server in your network, which is managing the IPs of all devices connected automatically (most routers provide a service like that), you have to set these IPs for every device your self. To get this right, you have to know a little bit about IP and network addresses and basic networking knowledge. This is especially true, if you add to an already existing network, like those in theaters (where often personnel with specific network knowledge is missing), without knowing its structure and setup.

Just putting devices in a network, without these basic settings might work (there is an auto IP set by default), but then is out of pure luck.

Woland

@dillthekraut said:

Sorry, but as a network engineer, I have to correct your comment.

Yayyyyy! I love learning more, especially when it leads to people getting better advice than I could give them. Thanks so much :)

(This is also why I put a bunch of "I think" and "As far as I know" language into the advice I was giving, as I wasn't entirely certain but was sure there was somebody out there who knew more than me and could correct me if needed.)

@dillthekraut said:

A little hint: Manually set IPs and DHCP services (e.g. by default provided by most routers) in the same network, can clash, if not set up in a propper way.
[...]
I might state one important basic: If you don't have a DHCP server in your network, which is managing the IPs of all devices connected automatically (most routers provide a service like that), you have to set these IPs for every device your self. To get this right, you have to know a little bit about IP and network addresses and basic networking knowledge. This is especially true, if you add to an already existing network, like those in theaters (where often personnel with specific network knowledge is missing), without knowing its structure and setup. Just putting devices in a network, without these basic settings might work (there is an auto IP set by default), but then is out of pure luck.

Using a separate network and static IPs is the way I've always worked on things like this, so everything on the network I'm going to be working with is mine, locked in, simple, and known.

@dillthekraut said:

This is not true at all! Two computers connected by a standard network cable are just the same as with an unmanaged switch in between.

But you are right in the fact, that you need to do some basic settings on your own in both cases, if it isn't done by a specific device in the network, like a router with an DHCP Service.

Aha! I knew there was something to it. Probably when I've tried this and seen people try it, the issue was that I or the person was being tripped up by settings we didn't know we needed to change. So two computers connected and configured correctly can send NDI and OSC back and forth? What setting would need to change? I'd love to know how to do this.