Orbbec on Windows & Mac - help!

bonemap

Hi,

I have a newly arrived Astra direct from Orbbec. It is the third Astra device I am using with Isadora. However, this new device does not appear recognised by the OpenNITracker on Windows or Mac. The two previously purchased devices appear immediately and function great on Windows and Mac Isadora 3 patches (with Mac Isadora set to Rosetta).

To eliminate the issue of the device being faulty, I connected it to an Orbbec TOP (released Feb) in a Mac version of a TouchDesigner project, confirming that the device functions as expected with RGB, IR, Pointcloud, and Depth outputs accessible.

Therefore, a device firmware change may be the issue with compatibility in Isadora 3 Windows and Mac. As Orbbec are not overly transparent or makes working with firmware easy or accessible for non-programmers, I am now stuck with moving forward with this device in Isadora projects.

@mark Also, please note that Orbbec appears to have released new Mac SDKs last fortnight.

I am aware of these previous forum threads related to the use of Orbbec on Windows and Mac:

https://community.troikatronix...

Any assistance with this new Orbbec Astra functioning with OpenNi on Mac in Isadora would be greatly appreciated.

Best Wishes

Russell

DusX

@bonemap

What is the model you have, is it a 'astra pro'?

Have you installed the masquerade.txt file?

Perhaps re-installing the driver could help.

bonemap

@dusx

Hi Ryan,

Thanks for responding!

I am trying to get a new Astra Mini Pro to function with the Isadora OpenNi Tracker module.

It is a newly purchased Astra Mini Pro that does not work with Isadora. I have checked the functionality and connection to the device: it appears as a connected USB unit in the MacOS System Report, and in addition, its outputs are functioning in alternative Mac software, i.e., TouchDesigner. I have previously acquired an Astra Pro and Astra Mini Pro (some years ago) that both operate as expected with Isadora on WIN and Mac.

I have downloaded the current release of the OpenNi Tracker from the Add Ons, installed the masquerade.txt file as instructed and set Isadora to run in Rosetta mode. I have tried with Isadora 3.2.6 and the current Beta version.

As of now, I have yet to be able to locate any drivers for the Orbbec Astra cameras on Mac OS. However, I did come across this MacOS SDK https://www.orbbec.com/develop...

Best Wishes,

Russell

DusX

@bonemap

Perhaps the new camera doesn't provide the depth map in the same resolution as the previous cameras. This would be the specification that you need to look at. I am not sure the OpenNI actor will work with different resolutions than those supported by the Kinect Cameras.

bonemap

@dusx

Thanks for alerting me to this possibility. However, the Orbbec Astra Mini depth camera's default resolution is the same as Kinect at 640 x 480.

Could there be anything else to try with this?

Best Wishes,

Russell

DusX

@bonemap said:

Could there be anything else to try with this?

We don't know. This camera may differ in other ways. The OpenNi Tracker supports a few cameras from several manufacturers and therefore isn't specific to any one manufacturer's sdk, this can be both a strength and a weakness depending on your needs.

bonemap

@dusx

Hi Ryan,

We have received a reply from Orbbec about this, so it should be clear to any other Isadora user considering a new Orbbec camera sensor for use with the OpenNi Tracker.

This in from the Orbbec:
“I think Isadora software using the very old OpenNI Mac SDK support, it supports an older Astra Mini however, not your new Astra Mini pro. There is Orbbec SDK Mac OS version supports Astra Mini pro on Mac, but Isadora does not support this SDK yet.”

The frustrating issue here is that despite the two Orbbec camera sensors sharing the same name, “Astra Mini Pro”, they have different PIDs and are different cameras. The older one works with Isadora’s OpenNi, but the newer one does not. This is likely the case for the entire Astra line from Orbbec. I had considered looking for the old firmware version for the new camera sensor, but unfortunately, that strategy is not possible either. I also want to note that I have not been able to get the camera sensor functioning on a Windows version of Isadora either.

Best Wishes

Russell

dbini

@bonemap

...so you bought a Mini Pro Brick? or do you have other ways of using it?

bonemap

@dbini

All is not lost—the Orbbec camera sensor functions in the current Mac release of TouchDesigner software for Depth, RGB, and Point Cloud. Unfortunately, friends bought the Orbbec camera sensor on my recommendation for use with an Isadora and OpenNi-based component of a touring project. I will offer them my older Orbbec camera sensor so we can complete their project.

Besides the embarrassment of recommending a device that is no longer viable for use with Isadora. I will have to pay closer attention. At least I now know the state of play with this.

Best wishes

Russell

Juriaan

Hey @bonemap,

Thank you for bringing this to our attention. Will bring it up internally, sadly I think this will require some development work to make them compatible.

- Juriaan

DusX

@bonemap said:

There is Orbbec SDK Mac OS version supports Astra Mini pro on Mac, but Isadora does not support this SDK yet.

This would require an Orbbec specific actor, which is something we will consider (I had started work on this, but the SDK was under heavy development and body tracking was removed). As it stands an Orbbec only actor wouldn't include body tracking.

We are actively investigating the ML options for body tracking, however, this domain is changing quickly and we hope to settle on a solid solution.

bonemap

@dusx

Hi Ryan,

Thanks for the update. I get that about Orbbec SDK development but they appear to be leading this area of tech.

Something that is very critical for our work in media and performance is the ability to access skeleton tracking in low light, highly dynamic and no-light situations. That is why Kinect style tracking is still relevant despite its age. As you know, the structured light camera sensors (Kinect/astra pro) emit their own IR light source and are mostly agnostic to other light sources and projections. This is a critical quality for media and performance work, wouldn’t you agree? The additional parameters available with OpenNi - depth calibration, single or multiple body isolation - provide methodology in creating logic for real-time body tracking installations (Where I am at with this current project).

ML body tracking works with RGB video streams really well. The option to implement MediaPipe style ML body tracking is already an option by pairing Isadora with 3rd party apps that output body tracking data as OSC or json. But these solutions, while great, do not function well in low light and, moreover, will annoyingly track unwanted bodies appearing in projections or screen media in the real-time capture environment.

There might be some short term potential to pair depth camera sensor sources with a Python based ML body tracking solution in Isadora, but this would require access to depth cameras and a Python module in ARM version, both requiring development time and therefore prioritisation by yourself and @mark.

This brings us back to body tracking with the ancient OpenNi solution - particularly when it is the public audience/participant that we are tracking to create an interactive work in a darkened gallery setting.

In terms of a dedicated Orbbec module - using depth data with blob tracking is useful and worthwhile. I guess if we could do something with point cloud data in Isadora would be a game changer.

For now though, correct me if I am wrong, the OpenNi Tracking provides the most efficiency for real world application in media and performance in terms of working with Isadora. So we are thankful that it is still supported and functional in the current environments.

Best wishes

Russell