[SOLVED] Ongoing Depth camera issues
Vidasonik last edited by Woland
We did our postponed last show Saturday in Bradford for our Arts Council funded project, a mixed-media intercultural performance.
Unfortunately though the sound and lighting were a success we were let down by the visuals. I have spent much time over the last 6 months trying to set up a system which needs 2 macs because the depth camera will only work on older macs. I had carefully setup matching Isadora files and Ableton projects on each computer. I need the new one for sound. The set up of necessity needed data to be sent from one computer to the other to synchronous sound and vision. As well as the Orbbec I use a web cam for capturing movement of dancer. Basically it proved unreliable. There have always been issues but unfortunately on Saturday evening I had to resort to actually running from one mac to the next manually pressing keys to play the correct video with the correct backing track. Needless to say everything was out of sync. I am at the point of giving up using a depth camera as there seems to be no prospect of getting it to work reliably. This is disappointing as I have spent considerable time over the last six months trying to set up what is a very over-complicated system.
Does anyone have any advice!?
Your description doesn't actually say anything about the problem in detail beyond that it was "unreliable".
Is it an issue with getting the wired network setup between the two computers so that you can send OSC back and forth to trigger things?
Are you having trouble setting up a cueing system between the two computers?
Is the Orbecc not providing skeleton data?
Is Isadora crashing?
I spent a while with you on Zoom calls building Isadora files for you that are designed to send OSC back and forth over a wired network because that's the way I would have done it if I had to do this for a performance, but if something about that approach isn't working I'd need to know what's going wrong before I could give advice on how to fix it. On the topic of complexity, the system (when I left off) was no more complex than you needed it to be. You needed one computer for tracking data and one computer for sound because you didn't have a singular computer that could do both, and thus you needed the computers to communicate with each other. As long as they were doing so over a wired network instead of WiFi and you are sending the OSC data to the correct IP addresses, there should never be an issue with the OSC communication.
I would really like to help, but I'd need more information about what was going wrong in order to do so.
@woland Thanks a lot, yes you are right I didn't really provide any detail. I will try to put together a comprehensive description of the setup. One issue maybe is that I used the theatre internet for transmission whereas it may have worked better if i set up a local area network for the 2 macbook pros. Your help was fantastic earlier in the year. I abandoned the router solution as couldn't get it to work. So I only send data between the macs - no hi res video.
@vidasonik The hi res video is sent directly from the old mac with depth camera to the main projector. A replica Ableton sound set is used purely to interact with the Isadora file on the old mac that includes the depth camera /OpenNI tracker BETA in some of the scenes. Midi notes are sent from Ableton to trigger visuals etc. This means, of course, that each duplicate Ableton set must be started at the same time. So I position the two computers next to each other so I can press two keyboard keys (programmed to start the first track) at the same time to trigger the start of both audio sets. Not ideal! I use the new mac for audio. The same audio of the old mac is sent to Izzy via Blackhole because Izzy doesn't hear the internal audio without going through another audio device.
So I position the two computers next to each other so I can press two keyboard keys (programmed to start the first track) at the same time to trigger the start of both audio sets. Not ideal!
You should be able to use a Keyboard Watcher actor on Computer 1 to:
- Start Ableton on Computer 1 by sending MIDI to it
- Send OSC from Computer 1 to Computer 2 to have Isadora on Computer 2 start Ableton on Computer 2
The hi res video is sent directly from the old mac with depth camera to the main projector.
If you don't want to send video over the network, you can use a capture card to capture the video output of Computer 1 (old Mac) and bring it into Isadora via Live Capture Settings on Computer 2 (new Mac) so that you can use a Video In Watcher actor on Computer 2 (new Mac) to have all your video in the same Isadora patch.
@woland a 'keyboard watcher' connected to a 'midi send' and 'OSC Transmit'? like this:
@woland so a capture card is a piece of hardware? Surely using Video In watcher brings us to the same problem that you can't transmit large files over a local network?
Fred last edited by
Hi @vidasonic, it sounds like you had a bad experience. Some of this thread sounds like an XY problem (https://xyproblem.info/), I definitely don’t mean offence to anyone in this thread but it is an interesting and common issue solving technical issues on forums.
It may seem like more trouble but it sounds like there may be a more robust solution to your issue that may be a little harder to implement but will give you some freedom later. It’s not exactly clear what your issues are or where they come from, but the main thing that seems evident is that you have 2 machines and only one is compatible with the depth camera. I would focus on working around that issue. From your description it seems like a good solution would be to implement the tracking with both your cameras on the older computer. Create a patch that can send any data that you need (the results of the tracking) and receive any required commands (switching scenes if need be). That tracking patch can likely run in a single scene and send and receive data via osc (use a wired connection). You can then use the tracking data as you had been on your newer computer and only have a single copy of ableton and a single machine that also outputs visuals. This is not an uncommon solution and is actually pretty sensible for distributing the workload. If you need all the actual video streams from the cameras because they are displayed on the projection then you can use NDI to send them from the older machine to the newer one. If you only need the webcam stream to send to the video projector you can consider moving that to the newer mac and only leaving the depth camera on the old mac. If none of the images from the cameras appear on the video projection then you can leave both on the older machine and just send the data.
Sometimes it can be tough to design the correct architecture for your technical solution. I would comment that continuing as you had been (trying to sync 2 machines and 2 sets of ableton) does not sound like the best solution (the y solution in an xy problem).
There are some great resources in isadora for doing this kind of multi machine setup and some excellent advice on the forum for achieving this too.
These kind of split systems can be automated and perfectly in sync with no need for manual intervention.
Let us know more details of what data needs to be where and the great brains trust of this forum will be able to guide you on a reliable solution.
@fred Hi Fred thanks for your response. Yes I do need to be clear about what I am trying to do so hopefully the following will help:
- It seems that owing to the impossibility of rewriting the OpenNI Tracker BETA software it will never be possible to use a depth camera with Izzy on any modern Apple Mac
- As Izzy doesn’t see the sound that I output to the sound card on new Mac (it only accepts indirect sound through another device eg. Blackhole) I have to use old Mac also for sound into Izzy.
- So I must use the old one and somehow hook the Macs together
- I bought a router which was a total nightmare, local internet connection in theatres very unreliable. At this point I gave up trying to send live video from the old to the new. So have to connect the old Mac to projector.
For some inexplicable reason Izzy will not accept audio directly so I have to send from Ableton to Blackhole and then select Blackhole in live capture settings. I use frequency and level detectors to produce video effects. Therefore I can’t even use the new Mac for sound with Izzy. The live sound must come from the new Mac which has the Clarett+4Pre sound card connected for live guitars, and backing audio stems etc. Ableton preferences are setup with audio input and output device set to Clarett. Therefore, I also have to have the webcam (I use this also to track dancer movement with Difference actors etc) attached to the old Mac and play a duplicate Ableton file on the old Mac. I have files adjacent to the audio master stems with midi notes locked to the same beats and song length. Clearly the problem is to have them running in sync otherwise the various video FX triggered by midi notes, sound envelopes etc that have to be triggered by the Ableton file on the old Mac will not conform to the live audio coming from the new one! I also send midi notes to change scenes and so on.
This is all fine as long as I can get both setups running in sync automatically.
The idea was to have the whole set flowing without me having to press buttons to start each song so I can focus on guitar/vocals.
We will be doing more shows going forward so I need to somehow devise a robust and simple set-up if I am to carry on using the depth camera.
As Izzy doesn’t see the sound that I output to the sound card on new Mac (it only accepts indirect sound through another device eg. Blackhole) I have to use old Mac also for sound into Izzy.
You can create a Multi-Output device in your Mac's Audio MIDI Setup which outputs to your physical audio output device and Blackhole or iShowUCapture at the same time, which will let you pull the same audio back in through Isadora's Live Capture by selecting Blackhole or iShowUCapture. I've done this for multiple productions where I needed to output audio to a physical device but also needed to do virtual audio routing to get the same audio back into Isadora in order to perform audio analysis or create audio-reactive visuals.
For some inexplicable reason Izzy will not accept audio directly
It's not that inexplicable. Virtual audio routing is easy enough to do and many tools exist for it (Loopback, iShowUCapture, Blackhole, etc) and have for a long time (I used SoundFlower for this 10+ years ago) so there's never really been a need for Isadora to be able to reinvent the wheel and make Isadora do the same thing these many purpose-built tools are designed to do.
I bought a router which was a total nightmare, local internet connection in theatres very unreliable. At this point I gave up trying to send live video from the old to the new. So have to connect the old Mac to projector.
Getting a private network set up with a router is your best route forward for not only sending video between the computers, but also for sending the data that you'd need in order to keep your computers in synch. I know the process of getting into networking can be daunting at first, but once you can do it you'll be able to easily accomplish the task of creating systems that solve the exact problems you're having right now (e.g. multi-computer synch/multi-computer cueing systems and sending video between computers).
@woland hi thanks for your reply
Points 1 and 2:
Yes, my first thought was to use Blackhole etc (I did have Loopback but that caused other issues so deleted it) however this introduced very noticeable latency so was not viable. Why won't Izzy receive audio from the Clarett soundcard?
I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work. Again, the depth camera should work on a modern mac.
jfg last edited by jfg
Point 3:I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work. Again, the depth camera should work on a modern mac.
you could try to connect both Mac with a ethernet crossover cable.
@jfg sounds good! Thanks.
a 'keyboard watcher' connected to a 'midi send' and 'OSC Transmit'? like this:
Not quite. You'd want the actors to be more like this (assuming you're using Computer 1 to trigger Ableton on Computer 2 with a Keyboard Watcher actor on Computer 1):
For more info on using Isadora to control Ableton, see this article: https://support.troikatronix.com/support/solutions/articles/13000081201-control-ableton-live-sessions-with-midi
You would also need to have the computers networked together so that you could send the OSC between them.
@woland so a capture card is a piece of hardware?
Yes, it's a piece of hardware that, for example, could be plugged into an HDMI cable coming from computer 1 in order to get that video signal into Isadora's Live Capture as a video input device on computer 2.
Example: Computer 1 video output > computer 2's HDMI output port > HDMI cable > video capture device's HDMI input port > video capture device's USB cable > computer 2's USB port > Isadora's Live Capture Settings (selected as video input device for channel 1, for example) > Video In Watcher actor.
Surely using Video In watcher brings us to the same problem that you can't transmit large files over a local network?
No, it wouldn't have anything to do with networking but, depending on the distance between the computers, it may not be viable to use a common video cable like HDMI/DVI and you might have to buy a repeater, extender, or use SDI, all of which is going to end up being more expensive than an ethernet switch and a couple of ethernet cables. Additionally, while it is possible to get the video signal from one computer to another using a video capture device such as a Blackmagic UltraStudio Mini Recorder or Elgato Cam Link 4k, it doesn't give you the capability to transmit any data so it doesn't solve your synch/cueing problem.
Why won't Izzy receive audio from the Clarett soundcard?
The first thing to do is check all four of your Isadora Live Capture channels to make sure that you don't have the Clarett selected as the audio device for channels 2, 3, or 4, because if it is selected on one of those it won't show up as a selectable option in the list of audio devices in Live Capture Channel 1. If that isn't it, then my second guess would be that this might only show up as an audio output device to Isadora for some reason, but that would be very strange, as I have two Focusrite devices that work fine for audio capture into Isadora.
I bought the router on the advice of Izzy specialists but after spending many frustrating hours (on line manuals, helpdesks, customer servces etc) never managed to get it to work.
I am unclear why the recommended router wouldn't work, as it was specifically recommended because it's the exact router a member of our staff uses for precisely this purpose. I completely understand your frustration because both myself and another staff member went well outside the scope of what we officially offer in terms of technical support by providing multiple hours of consulting, video calls, programming, and troubleshooting free of charge in order to try to assist you in getting the router and networked computer setup running, as well as modifying your Isadora files to send and receive OSC and NDI. So, while I apologize that the time and energy spent on trying to get that specific router working for you didn't pay off, I would still like to stress that just because that one device wouldn't work for you doesn't mean you should abandon the entire approach. If your car breaks down and nobody can figure out what's wrong with it, that doesn't mean you should give up on driving forever and never get another car. You could pick up a 4/5-port gigabit ethernet switch for 16-25 euros and try again. You even saw that the setup worked as a proof of concept over your wifi when we were on a call together testing the specific Isadora files I made for each of your computers and at that point I said that all you'd need to do is get your own router or a switch and connect the computers to it with ethernet, and get a LAN up and running.
local internet connection in theatres very unreliable
I also distinctly remember strongly recommending against trying to use wifi or a theatre's existing network for performances because you could easily run into any number of problems that would be outside of your control. You want your network to be locked down and private so that no outside factors can affect it. Additionally, traveling with your own router/switch means you already know that your network setup works, instead of trying to figure out something different at each new venue.
you could try to connect both Mac with a ethernet crossover cable.
This is definitely worth a shot, but I'm not sure you'll be able to send NDI and OSC this way because two computers connected directly via a network cable don't necessarily always behave exactly the same as two computers both connected to a router/switch. I'm sure someone who knows more about networking than I do could explain it in better detail, but as far as I know, certain aspects of networking require a router or a switch in order to work properly.
Again, the depth camera should work on a modern mac.
We didn't design, manufacture, nor do we sell the hardware and we are not able/allowed to alter the OpenNI code that is used to make it work because it's not ours, so we're neither able to, nor required to make it work on modern Macs. Despite having no obligation to do so, we have done our best to maintain the OpenNI plugins for years but, as far as I understand, the OpenNI code was written nearly a decade before the first M1 Mac computer even existed so there's simply not much we can do anymore. Writing our own library for this from scratch would require more time and resources than we have right now and trying to adapt a different library by somebody else would present its own problems, namely that 1) we'd have to find a library that is both good and allows for commercial use without an enormous licensing fee 2) we'd still have to invest more time and resources into adapting and testing it than we currently have, and 3) it very likely wouldn't work in the exact same way as the OpenNI library (might provide different data, differ in accuracy, not have one or more features we relied on, etc) so an existing Isadora file originally built using the OpenNI plugins would likely need significant adjustments to make it work with the theoretical new tracking plugin or, if the Isadora file required a feature that existed in OpenNI but didn't exist in the new library, it wouldn't work at all.
@woland Thanks so much for this comprehensive reply. Indeed you spent a lot of time with me (as you say, way beyond the call of duty, and I expressed my gratitude at the time) helping to set everything up and I remember feeling confident that a solution had been found which merely needed implementing.
I think I will need to step back from all this and reflect on everything to see if this is worth pursuing further, ie does what I am trying to do justify all this effort from yourselves as well as myself. I would hate to abandon it if I felt a solution was within reach, however. Thanks so much for bearing with me.
All the best, David
I would hate to abandon it if I felt a solution was within reach, however.
For the record, routers tend to be trickier to set up than ethernet switches. Switches tend to be plug-and-play whereas routers can have more options to fiddle with.
@woland ok can you recommend an ethernet switch?
Sorry, but as a network engineer, I have to correct your comment. There are complicated switches and simple switches, as well as simple routers/Access points and complicated setups, if used wrong. There are three common types of network connection devices: hubs, switches and routers. But there are different (sub-)types of switches, which can lead to issues, if not known.
- Hubs are very dumb devices, which blow all data from and to all ports, without any logic, which can cause them to slow down in data rich environments. There is only a few to buy nowadays, as switches became the standard.
- Switches are smarter hubs. They know which device is connected to which port and can therefore route data more efficient. But they come in two flavors: managed and UNmanaged. The later is kind of plug and play, where the managed ones try to optimize the data flow on a deeper level by default, which can be a struggle in some cases. In these situations, you need to know how to manage (pun intended ;) ) them, which needs deeper knowledge of network protocols and special network functionality.
Some managed switches as well come with very different types and numbers of extra functionalities and implemented services, which some of are like the ones routers usually provide (like DHCP). But usually these are deactivated by default.
The issue with MANAGED switches is especially true for protocols like NDI. The keyword here is multicast!
- Routers again are a special type of devices, with the main purpose of translating between two or more different networks, e.g. Internet and LAN. Some smaller wifi routers provide connectivity for two computers or even one only. The most bigger ones have an integrated hub or switch, which provides more ports to connect more devices. This should not be more complicated or different to use, like the stand alone hubs/switches, regarding their simple switching functionality.
But on top of this, they provide some extra services, like DHCP, which are active by default in most cases! Those usually do not interfere with the general network functionality. But though some CAN be used to simplify a setup, if used wrong, they can become an issue. Beside this there might be some routers with managing functionality as well as the managed switches and therefore have the same side effects. But they are usually upper class, upper price range.
Basic thing you have to be aware of, is the different types of ports they have (LAN, WAN, etc.). For pure local network functionality as a hub/switch, you would only use those ports with LAN functionality. Not the ones with WAN on it.
For a simple setup, have a look for UNmanaged gigabit switches with the number of ports needed. TP-Link, Netgear, Linksys are common brands in that class and don't have big differences in general functionality. Switches with 8 ports and less usually are unmanaged, if not otherwise stated.
If not already done, you should as well get a little bit into basic network knowledge, e.g. IP addresses and subnetmasks, as well as DHCP service.
A little hint: Manually set IPs and DHCP services (e.g. by default provided by most routers) in the same network, can clash, if not set up in a propper way.
You nowadays don't need any crossover cables anymore as any gigabit device, be it the computers network port or the switch, are able to identify active and passive connections. You can just use any usual RJ45 Cat 5 and up cable.
These cables/adapters are from old 100mbit times, and where called uplink cables. Some hubs had a special uplink port to connect to other hubs without needing an uplink/ crossover cable. Modern switches, doesn't need those anymore.