Guru Session #13: Body Tracking with Depth Cameras (Fri May 8th, 6pm CEST/5pm GMT/12pm EDT/9am PDT)

Juriaan

Hi there,

A few things that I noticed

1) Kinect v2 (Xbox One) is still crashing with the current version that was provided to us today. Sadly Isadora crashes so badly that I can't send the Crash logs to you since it doesnt create one with the normal workflow (So that you can send it from the Error prompt that pop-ups)

2) Load becomes massive like 70.6% on a Windows 10, MSI GE73VR 7RF Raider laptop (2000 euro gaming laptop with a Nvidia dedicated graphic card with just the Kinect v2 / OpenNi Tracker and a Projector attached to it's depth image in full screen mode)

3) When I say against Isadora that I only wish to receive the color map it still sends out the depth video signal as well.

4) I can't get the recording to work properly. What I did was I set the File input to tracking.oni and told the plugin to Record, do a silly dance, and press 'stop'. Then I try to play it back by pressing 'Play' and it doesn't show anything. It stops the active sensor flow and showcases a blank screen. When I'm looking in my file explorer or it even made a file called tracking.oni it doesn't show up.

DillTheKraut

@dillthekraut

Could solve it by removing the OpenNI drivers and installing the official Microsoft Kinect SDK.
Works perfectly now on the pc.

Demetri79

@jfg

Hardware: Mac Pro 2012 5,1 PCIe USB 3
MacOS Version: High Sierra 10.13.6
Depth Camera: Kinect v1 (Model 1414)
Isadora Version: 3.0.7

@mark

I dont have any homebrew installed. it came back with "-bash: brew: command not found"

So after much more testing I have more insight to my problem at least:

I tested all 3 versions now (0.9.2,0.9.5,0.9.6) , over 100 times shutting down my computer and following the suggested process. I think the problem for me is something related to the USB ports.. USB communication or USB power. I have tried all 9 (5 usb3 via pcie and 5 USB2 via 3 rear and 2 front). The USB3 ports on my PCIe card had better results working to provide power most of the time. i.e. sometimes the blinking green light on the kinect would stop blinking after a few blinks when connected to a USB2 port. This did happen a couple times on the USB3 ports as well though.

This led me to believe maybe an SMC reset might fix the problem which seems to be either USB communication or USB power. BUT I HAVE NO PROBLEMS WITH ANY OTHER USB DEVICE EVEN THOSE DRAWING POWER.. THEY ALL WORK PERFECTLY??

I performed an SMC reset (remove power cable from mac for 15 sec) and booted up and the Kinect was finally recognized for the first time since it stopped working thursday. But soon as I got the skeleton decoder and projector connected the skeleton data froze and the depth image froze. Deleting the Open NI tracker and placing in a new one did not work. The only way to get the kinect to be recognized by the Open NI is doing the whole process over again.. shutdown - SMC reset - boot up - place Open NI- kinect recognized - then freezes again within 5 sec.. even this process does not get the kinect recognized every time.. only about a about 25% of the time. I cannot get the kinect working properly like it was earlier this week.. it was working perfectly every time i connected it and never crashed once even with 4hour+ sessions.

All 3 versions of openNI have been tested over 30 times each .. I only got the kinect even recognized about 4 times total.. and the times it did get recognized it froze within seconds.. but the green light did not stop blinking when the data and image froze. I'm so bumed and confused

mark

@mark_m said:

The Windows link doesn't seem to be working.Ooops. Sorry about that. I fixed the link in the alpha builds thread. Thanks for the heads up.

Best Wishes,
Mark

mark

@dillthekraut said:

[EDIT: I see you figured this out and it worked. Glad to know that this installing the SDK solves the problem.]

But I do have an issue on a windows pc. With the OpenNI plugin in the plugin folder, I get the following error:

There was an omission in the original installation instructions. Please try the following:

1) Make a backup copy of the OpenNI Tracker.izzyplug folder
2) Delete the entire plugin OpenNI Tracker.izzyplug
3) Reinstall v0.9.6
4) But also also install the Kinect SDK v2.0. This is required to use the Kinect drivers on windows.

I'd like to know if that procedure gives you a working plugin. While the old drivers you downloaded may work, they do not support changes I have made in those drivers improve how they work. (Specifically, if you try the new record function with those old drivers, I'm pretty sure Isadora will crash as soon as you try to play the ONI file.)

Best Wishes,
Mark

mark

@juriaan said:

A few things that I noticed

I'll go through your list point by point

1) Kinect v2 (Xbox One) is still crashing with the current version that was provided to us today. Sadly Isadora crashes so badly that I can't send the Crash logs to you since it doesnt create one with the normal workflow (So that you can send it from the Error prompt that pop-ups)

Yeah, I don't really expect it to work well yet. It was an accident that it went out with the 0.9.5 build, but I figured at this point I'll just put it in and let people try. I think people on MacOS will have a better result since the entire driver was hacked/created from the ground up. They will have to deal with 512x424 image being embedded in a 640x480 frame right now.

2) Load becomes massive like 70.6% on a Windows 10, MSI GE73VR 7RF Raider laptop (2000 euro gaming laptop with a Nvidia dedicated graphic card with just the Kinect v2 / OpenNi Tracker and a Projector attached to it's depth image in full screen mode)

It's unclear to me if you're talking about a Kinect v1 or Kinect v2 here. I guess it's Kinect v1 since v2 is crashing, but please be specific. That said, loads that high are not my experience. I'll edit this post in a few minutes with my load figures from Windows 10. Keep in mind that the GPU is going have like zero impact on the efficiency of this plugin. The data from the camera comes in as CPU bitmaps, and everything is thus handled on the CPU not the GPU.

3) When I say against Isadora that I only wish to receive the color map it still sends out the depth video signal as well.

I think that "against Isadora" must be a typo... I guess what you're saying is that if you attempt to get the RGB camera only, you cannot. Yes??

4) I can't get the recording to work properly. What I did was I set the File input to tracking.oni and told the plugin to Record, do a silly dance, and press 'stop'. Then I try to play it back by pressing 'Play' and it doesn't show anything. It stops the active sensor flow and showcases a blank screen. When I'm looking in my file explorer or it even made a file called tracking.oni it doesn't show up.

Did you save the file? Unless you specify a full path name (e.g., C:\Users\USERFOLDER\Documents) the plugin creates a path name based on the path of the Isadora .izz file. If you haven't saved the file, maybe this fails?

Best Wishes,
Mark

mark

@demetri79 said:

I'm so bumed and confused

Well, I am not sure what to say to this. I mean, it seems like you had everything working for some time, and now it doesn't work. If you reboot the computer, there is no way that any data or code from the tracker plugin is lingering inside of your computer's memory. So I really don't think it's the OpenNI Tracker itself. Furthermore, you indicate the previous version (0.9.2), which was working well previouslyl, no longer works.

We need some way to prove the hardware itself is still working, because given the description of the frustrating process you describe, and the fact that the green light stops blinking, it is possible that your Kinect has failed.

Do you have some other software you could use to try to use that Kinect, e..g, the demo or release version of NI Mate perhaps? Or else the free Shapify.me from the App Store? (This successfully worked with my Kinect 1414, including moving it up and down using the internal motors. It never succeeded in scanning me, but I could see the image.)

Best Wishes,
Mark

mark

@Demetri79 and everyone else:

I've put together a simplified Kinect test app for MacOS using the examples from the open-source original OpenNI distribution.Inside this download you'll find two scripts: run-closest-point-viewer.command and run-simple-viewer.command. The first runs the "ClosestPointViewer" test app, the second runs the "SimpleViewer" test app.

Could people running High Sierra try running run-simple-viewer.command and verify it shows a depth map image? I need this to kind of prove that it is a good, universal test to show a working connection to the Kinect without any of my modified drivers in the loop.

These test apps work for me on High Sierra 10.13.6 (17G13021)

Note that these scripts and apps are not code signed. To run them under High Sierra, right click the script and choose "Open". You'll get a dialog like this:

Click the "Open" button to run the script. If everything is working, you should see this:

Note that it might take two attempts before it works. If you don't see an image the first time or the script otherwise fails, try it once more.

Diagnostic information will print in the Terminal window that opens. If this fails, posting the the info in the Terminal window might help me to debug your setup.

kinect-test-macos.zip

Best Wishes,
Mark

mark

Dear All,

As you can see, I'm trying to address the problems people have identified here. But, to give me a better picture of how this actor is working out, it would be good to hear from people for whom it is working too.

Please be specific about what sensor your using, and the operating system with version if it's not in your signature line. I assume you're all working with Isadora 3.0.7 or the current beta if you are a beta tester.

Best Wishes,
Mark

Juriaan

@mark

"Yeah, I don't really expect it to work well yet. It was an accident that it went out with the 0.9.5 build, but I figured at this point I'll just put it in and let people try. I think people on MacOS will have a better result since the entire driver was hacked/created from the ground up. They will have to deal with 512x424 image being embedded in a 640x480 frame right now."

Yep understood :) Still wanted to let you know. I sent you the crash reports earlier that basically have the same behavior. Isadora starts just fine with the OpenNI actor, after a while (5 minutes or so) the OpenNI actor crashes Isadora. After that the time duration that Izzy crashes after a restart is like 1.5 minutes or so

"It's unclear to me if you're talking about a Kinect v1 or Kinect v2 here. I guess it's Kinect v1 since v2 is crashing, but please be specific. That said, loads that high are not my experience. I'll edit this post in a few minutes with my load figures from Windows 10. Keep in mind that the GPU is going have like zero impact on the efficiency of this plugin. The data from the camera comes in as CPU bitmaps, and everything is thus handled on the CPU not the GPU."

Isadora 3.0.7 (Stable), Windows 10. Kinect Sensor v2. I added a reg key to actually boost my Intel CPU to 30 frames for the Kinect. When I didn't do that I only got 15 frames causing the Kinect Verifier application to yell ;) On the Kinect SDK forums you will find a forum post describing how to disable CPU sleeping.

"I think that "against Isadora" must be a typo... I guess what you're saying is that if you attempt to get the RGB camera only, you cannot. Yes??"

When I use the 'capture mode' input of the plugin and set it to Color I also receive the depth image. What I expected to happen is that I only get the color image and that the depth image was disabled / not active (So a red line instead of a green line to the projector)

Did you save the file? Unless you specify a full path name (e.g., C:\Users\USERFOLDER\Documents) the plugin creates a path name based on the path of the Isadora .izz file. If you haven't saved the file, maybe this fails?

Nope, I didn't save the file. Will test that out later tonight when I have some spare time.

mark

@juriaan said:

Isadora 3.0.7 (Stable), Windows 10. Kinect Sensor v2. I added a reg key to actually boost my Intel CPU to 30 frames for the Kinect. When I didn't do that I only got 15 frames causing the Kinect Verifier application to yell ;) On the Kinect SDK forums you will find a forum post describing how to disable CPU sleeping.

Hmmmm I didn't need to do any of this. My LOAD was 21% with Windows 10 running on Bootcamp on my MacBook Pro. (I know that's not _really_ a PC, but it's pretty much a PC.)

I'm investigating the Kinect v2 crash now. I just ran the Kinect v2 for 10+ minutes, dancing around all the while, and no crash. I'm now trying it again without the debugger running... it's been 7 minutes and all is well.

@Juriaan : Did you need to be in the frame to get it to crash? Or could you just let it sit with no one in the frame? Had you added anything to your patch aside from the OpenNI Tracker and a Projector? (I suppose heavier load/stuff in the patch could change the picture.)

Best Wishes,
Mark

JJHP3

Hi Mark

I tested the new files you made available. The simple player window did not open. Report included. Note: this is the same laptop (10.13.6) / 1414 that works so well with the new openNI plugins. Hope this helps you. John

JJHP3

@jjhp3 Trying to get a file uploaded: test instead:

Last login: Mon May 11 14:04:38 on ttys000

MacBook-Pro:~ jjhp3$ /Users/jjhp3/Library/Mobile\ Documents/com\~apple\~CloudDocs/JP\ Art/Isadora\ patches-web/Isadora\ guru/guru\ 13\ -\ kinect/kinect-test-macos/run-simple-viewer.command ; exit;

848 INFO New log started on 2020-05-11 14:04:57

882 INFO --- Filter Info --- Minimum Severity: VERBOSE

942 VERBOSE No override device in configuration file

969 VERBOSE Configuration has been read from '/Users/jjhp3/Library/Mobile Documents/com~apple~CloudDocs/JP Art/Isadora patches-web/Isadora guru/guru 13 - kinect/kinect-test-macos/OpenNI.ini'

973 VERBOSE OpenNI 2.2.0 (Build 33)-MacOSX (Nov 12 2013 16:14:14)

977 VERBOSE Using '/Users/jjhp3/Library/Mobile Documents/com~apple~CloudDocs/JP Art/Isadora patches-web/Isadora guru/guru 13 - kinect/kinect-test-macos/OpenNI2/Drivers' as driver path

981 VERBOSE Looking for drivers in drivers repository '/Users/jjhp3/Library/Mobile Documents/com~apple~CloudDocs/JP Art/Isadora patches-web/Isadora guru/guru 13 - kinect/kinect-test-macos/OpenNI2/Drivers'

OpenNI2-FreenectDriver: Using libfreenect v0.5.5

OpenNI2-FreenectDriver: Found device freenect://0

9534 INFO Device connected: Microsoft Kinect (freenect://0)

9569 INFO Device state changed: Microsoft Kinect (freenect://0) to 0

/Users/jjhp3/Library/Mobile Documents/com~apple~CloudDocs/JP Art/Isadora patches-web/Isadora guru/guru 13 - kinect/kinect-test-macos/run-simple-viewer.command: line 5: 14864 Illegal instruction: 4 "$SCRIPTPATH/SimpleViewer"

logout

Saving session...

...copying shared history...

...saving history...truncating history files...

...completed.

[Process completed]

mark

@jjhp3 said:

line 5: 14864 Illegal instruction: 4 "$SCRIPTPATH/SimpleViewer"

So you're saying that OpenNI Tracker works fine on your computer but the test application crashed? Did you try it twice as I asked?

I actually got a similar error just now after installing a High Sierra security updated 2020-3.

But the test app worked after I did the following:

1) Unplug Kinect USB connector
2) Disconnect power cable from Kinect
3) Reboot computer
3) Connect power cable to Kinect
4) Plug in Kinect USB connector

Could you please try that? I'd like to know that it actually works.

@Demetri79 -- what about you?

Best Wishes,
Mark

JJHP3

Mark.

1. Yes, OpenNI Tracker works well with the tutorials you supplied. Takes a couple of minutes max for the image to show up, but it always does appear.

2. I did as you suggested above and then ran it twice with the same result as above: ... "line 5: 704 Illegal instruction: 4 "$SCRIPTPATH/SimpleViewer"" Test #2 has a small variation "line 5: 734 Illegal instruction..."

John

Demetri79

@mark

the test application did not open a viewer window after 10 tries. And I did the boot up process you described exactly. Do you want all 10 terminal reports? I'm just providing 1 for now since they all looked essentially the same except for the numbers next to: INFO, VERBOSE, and LINE 5.. were different on each attempt. I even tried the closest point viewer test 4 times. no viewer window.

ALSO: today before I before i performed the simple viewer test.. first boot up of the day the kinect was recognized and output depth and skeleton data in an otherwise blank session containing only Open NI .0.9.2 and skeleton decoder. I let it run for 30 min with no problems. so then i shut the test patch and opened my patch which has the Open NI and skeleton decode already wired in and it was also recognized and passed image and data for about 2min until the image and data started glitching and freezing.. then froze completely. So I performed the complete shutdown booted back up and nothing.. could not get it working again after 10 shutdowns. So I switched to Open NI 0.9.6 .. could not get that working after 10 attempts. Then I performed the simple viewer test as mentioned above over 10 times.

there is no hardware or software issues with my kinect and I have varified this by connecting it to another computer running all the same versions of software and plugs: macbook pro 2013, mac os 10.13.6, isz 3.0.7.. and tested all 3 versions of Open NI and the kinect was recognized and worked flawlessly every time conected. i tested it 15 times and let it run for a couple hours.. no problems

Last login: Mon May 11 14:41:32 on ttys000
Demetris-Mac-Pro-4:~ demetrievdoxiadis$ /Users/demetrievdoxiadis/Desktop/kinect-test-macos/run-simple-viewer.command ; exit;
      542 INFO       New log started on 2020-05-11 14:43:18
      602 INFO       --- Filter Info --- Minimum Severity: VERBOSE
      659 VERBOSE    No override device in configuration file
      688 VERBOSE    Configuration has been read from '/Users/demetrievdoxiadis/Desktop/kinect-test-macos/OpenNI.ini'
      694 VERBOSE    OpenNI 2.2.0 (Build 33)-MacOSX (Nov 12 2013 16:14:14)
      698 VERBOSE    Using '/Users/demetrievdoxiadis/Desktop/kinect-test-macos/OpenNI2/Drivers' as driver path
      702 VERBOSE    Looking for drivers in drivers repository '/Users/demetrievdoxiadis/Desktop/kinect-test-macos/OpenNI2/Drivers'
/Users/demetrievdoxiadis/Desktop/kinect-test-macos/run-simple-viewer.command: line 5: 5097 Illegal instruction: 4 "$SCRIPTPATH/SimpleViewer"
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

DillTheKraut

@mark

Status report

Kinect 1414

MacBook Pro 2019 16"
i9 8C 2,4Ghz
64GB
Radeon HD 5500M 8GB
MacOS 10.15.4 Catalina

closest-point-viewer.command and run-simple-viewer.command running immediately without issues.

Isadora Plugins (0.9.2 and 0.9.6) running smooth and stable.

Zotac ZBOXNANO-AQ01
AMD A4-5000 APU with Radeon HD 8330
4GB RAM
Windows Kinect SDI v2.0

openni-tracker-tutorial works fluently.
But 3D Puppet is not working as three.min.js is missing (JS Quaternion Calc Angle actor).
The js file is located next to the openni-tracker-tutorial.izz patch.

Best
Dill

mark

@demetri79 and @jfg

So if I understand things correctly, the the only two people having problems are both on an older Mac Pro (aka "cheese grater" model).

@Demetri79 :

First, I don't think you have to try things ten times. Three is enough. If it doesn't work after three attempts, then it's not working.

That said: you have a Mac Pro 2012 5,1 running High Sierra 10.13.6, yes?

What processors/how many cores? (Can you please put your hardware configs in your signature line like @jfg? That save me having to go through this whole thread trying to find this information. Thanks. ;-) )

When you run the same plugins on your laptop, everything is fine, right?

Also, there was a recent update to High Sierra that broke, among other 32-bit apps, Abelton Live. Now, Isadora and OpenNI Tracker are all 64 bit, but I wonder if a software update could have caused your Mac Pro to stop recognizing the Kinect reliably. (Because -- based on your previous posts -- it worked well on the Mac Pro previously, right?)

@jfg :

You are also using a Mac Pro 2012 (3,33 GHz 6-Core Intel Xeon) running Mojave 10.14.6, yes?

When you run the same plugins on your laptop, everything is fine, right?

@Demetri79 & @jfg

My fear here is that the library "libusb" which handles communication to the Kinect is failing, especially since you both mention the PCI USB 3 cards you have installed. There is an NI Mate thread that details a user having problems with macOS Sierra on a Mac Pro that sounds really similar. (When I wrote this, the original NI Mate post wouldn't open; but you can read Google's cached version.) In that thread they instruct the user to try a different libusb included with NI Mate, but I don't think it ever solve the problem.

In the end, I may have you to install the homebrew package manager and compile libusb on your machine. When you do this, the library is compiled with specifically for the hardware and operating system on your computer. I don't know if this will solve the problem or not... but right now, I don't know what else to tell you two.

If you are willing to take the time to try compiling your own libusb with homebrew, let me know and I'll post the instructions.

Best Wishes,
Mark

mark

@juriaan (and all -- please read I want an opinion)

When I use the 'capture mode' input of the plugin and set it to Color I also receive the depth image. What I expected to happen is that I only get the color image and that the depth image was disabled / not active (So a red line instead of a green line to the projector)

I realize now why this is happening; it was actually by design. If you set the 'body tracking' to off, you will only get the color image. If 'body tracking' is on, then the plugin always enables the depth stream because you are telling the plugin you want to track bodies.

Do people find this confusing? Should I change it so that, if the mode is set to 'color', body/skeleton tracking is simply disabled?

Best Wishes,
Mark

mark

@juriaan said:

Nope, I didn't save the file. Will test that out later tonight when I have some spare time.

I have verified that if you have not saved the file, the current version of the plugin will not save the recording. I have fixed this for the next release.

Best Wishes,
Mark