Guru Session #13: Body Tracking with Depth Cameras (Fri May 8th, 6pm CEST/5pm GMT/12pm EDT/9am PDT)

Demetri79

@mark thanks for the help so far.

I dont have any homebrew installed. it came back with "-bash: brew: command not found"

i tried the process you outlined over and over and still cant get it working again regardless of which version of the OpenNItracker i use. I tried every usb port on my machine as some are 2.0 and some 3.0 on a pcie card. Sometimes when the OpenNI is placed in the blank scene it immediately reads: "resolving" in status and does nothing.. sometimes it flashes "init device" for a while and then stops flashing that and stays on init device.. sometimes the kinect itself stops flashing green once i place the OpenNI actor.. ive tired everything.. deleted prefs again, disconnected all non essential devices from computer.. deactivated all my qartz plugins, corea audio plugins not loaded, deleted my global user actor folder (since it had a full setup saved with tracker, skeleton, etc).. makes no sense as it worked all week.. the only software I installed since it was working is Newtek NDI tools.

the kinect itself is not the issue ..i tested the kinect on my macbook pro 2013 also running the same Mac os 10.13.6 and isadora 3.0.7.. and IT WORKS.. just can't get it working again on my tower. the laptop is too old and slow to use for my insatiable needs:(

jfg

@demetri79 said:

on my tower

can you tell us which tower? it is the same as mine: Mac Pro 2012 (5.1) USB 2.0 native+ Inatek PCIe USB 3.0 Card.

best

Jean-François

mark

Dear All,

v0.9.6 of OpenNI Tracker is available in the "alpha builds" forum thread.

The record feature is now enabled. It seems to work well on both Mac and Windows now. I'm sure this feature will be a boon for development, since you can now record and play the depth streams and have the tracking work as if there were a person in front of the camera. Let me know how it works for you.

@jfg This version may solve your problem with the Asus Xtion. Try it and let me know.

Best Wishes,
Mark

DanielPineiro

Hi!

The record feature works fine for me! That´s great. Thanks Mark!!!!

mark_m

@mark said:

v0.9.6 of OpenNI Tracker is available in the "alpha builds" forum thread.

The Windows link doesn't seem to be working, but it's available at:
https://troikatronix.com/files...

Cheers

Mark

jfg

@mark

Dear Mark,

I have to apologise because I gave you yesterday wrong information. I tested yet all different configurations new and the result is:

- 0.9.2 works fine on both computers (I have to use the masquerade file)

-0.9.5 and 6 won't recognise bodies on both (I haven't forget to put the masquerade file at the right place)

on the Macbook with USB 3.0 the Xtion is recognised.

on the MacPro (2012) the Xtion is only recognised over USB 2.0. If I connect it on the USB 3.0 Inateck PCIe card i get only device init error

The color video output give me only a frozen picture on the macBook (usb 3.0).(the first picture after switching). Work ± on MacPro (Jitter)

thank you for your help

best

Jean-François

JJHP3

Just sayin' the new openNI plugs are TERRIFIC! They work perfectly with my aging MBP laptop and kinect 1414. I used the kinect for a performance/installation back in 2012 and it was a pain... it then went on the shelf. I used DeliCode program in get the info into Isadora and then it was a bit of a crap shoot whether it would work or not. Congratulations to Mark and the team on getting this wonderful re-addition to Isadora up and running. The tutorial and the guru session were both most useful. Bravo ~

John

DillTheKraut

@mark

Hello and congratulations to this amazing peace of plugin. Same as @JJHP3, for the Guru Session I freed my 1414 from the years old dust and it ran on my MacBook in no Time!
I'm especially impressed by how fast the body recognition reacts and the accuracy of the skeleton tracking.

But I do have an issue on a windows pc. With the OpenNI plugin in the plugin folder, I get the following error:

This is true with Tracker plugin ver. 0.9.6 and 0.9.2

Isadora 3.0.7
Windows 10 Pro 1909 1836.592
AMD A-5000
4GB RAM

Best
Dill

DillTheKraut

@dillthekraut

I could solve the starting error by installing the OpenNI drivers from structur.io. https://structure.io/openni

But I do have a general issue now, as the pc isn't recognizing the kinect right. In the windows device manager list, only the XBOX NUI Monitor is appearing under other devices, without a proper driver associated.
At a Parallel Windows VM, it shows additional XBOX NUI Audio and XBOX NUI Camera. Which is missing on the pc. The actor in here keeps saying 'resolving uri'.
Does anyone have an idea?

Dill

Juriaan

Hi there,

A few things that I noticed

1) Kinect v2 (Xbox One) is still crashing with the current version that was provided to us today. Sadly Isadora crashes so badly that I can't send the Crash logs to you since it doesnt create one with the normal workflow (So that you can send it from the Error prompt that pop-ups)

2) Load becomes massive like 70.6% on a Windows 10, MSI GE73VR 7RF Raider laptop (2000 euro gaming laptop with a Nvidia dedicated graphic card with just the Kinect v2 / OpenNi Tracker and a Projector attached to it's depth image in full screen mode)

3) When I say against Isadora that I only wish to receive the color map it still sends out the depth video signal as well.

4) I can't get the recording to work properly. What I did was I set the File input to tracking.oni and told the plugin to Record, do a silly dance, and press 'stop'. Then I try to play it back by pressing 'Play' and it doesn't show anything. It stops the active sensor flow and showcases a blank screen. When I'm looking in my file explorer or it even made a file called tracking.oni it doesn't show up.

DillTheKraut

@dillthekraut

Could solve it by removing the OpenNI drivers and installing the official Microsoft Kinect SDK.
Works perfectly now on the pc.

Demetri79

@jfg

Hardware: Mac Pro 2012 5,1 PCIe USB 3
MacOS Version: High Sierra 10.13.6
Depth Camera: Kinect v1 (Model 1414)
Isadora Version: 3.0.7

@mark

I dont have any homebrew installed. it came back with "-bash: brew: command not found"

So after much more testing I have more insight to my problem at least:

I tested all 3 versions now (0.9.2,0.9.5,0.9.6) , over 100 times shutting down my computer and following the suggested process. I think the problem for me is something related to the USB ports.. USB communication or USB power. I have tried all 9 (5 usb3 via pcie and 5 USB2 via 3 rear and 2 front). The USB3 ports on my PCIe card had better results working to provide power most of the time. i.e. sometimes the blinking green light on the kinect would stop blinking after a few blinks when connected to a USB2 port. This did happen a couple times on the USB3 ports as well though.

This led me to believe maybe an SMC reset might fix the problem which seems to be either USB communication or USB power. BUT I HAVE NO PROBLEMS WITH ANY OTHER USB DEVICE EVEN THOSE DRAWING POWER.. THEY ALL WORK PERFECTLY??

I performed an SMC reset (remove power cable from mac for 15 sec) and booted up and the Kinect was finally recognized for the first time since it stopped working thursday. But soon as I got the skeleton decoder and projector connected the skeleton data froze and the depth image froze. Deleting the Open NI tracker and placing in a new one did not work. The only way to get the kinect to be recognized by the Open NI is doing the whole process over again.. shutdown - SMC reset - boot up - place Open NI- kinect recognized - then freezes again within 5 sec.. even this process does not get the kinect recognized every time.. only about a about 25% of the time. I cannot get the kinect working properly like it was earlier this week.. it was working perfectly every time i connected it and never crashed once even with 4hour+ sessions.

All 3 versions of openNI have been tested over 30 times each .. I only got the kinect even recognized about 4 times total.. and the times it did get recognized it froze within seconds.. but the green light did not stop blinking when the data and image froze. I'm so bumed and confused

mark

@mark_m said:

The Windows link doesn't seem to be working.Ooops. Sorry about that. I fixed the link in the alpha builds thread. Thanks for the heads up.

Best Wishes,
Mark

mark

@dillthekraut said:

[EDIT: I see you figured this out and it worked. Glad to know that this installing the SDK solves the problem.]

But I do have an issue on a windows pc. With the OpenNI plugin in the plugin folder, I get the following error:

There was an omission in the original installation instructions. Please try the following:

1) Make a backup copy of the OpenNI Tracker.izzyplug folder
2) Delete the entire plugin OpenNI Tracker.izzyplug
3) Reinstall v0.9.6
4) But also also install the Kinect SDK v2.0. This is required to use the Kinect drivers on windows.

I'd like to know if that procedure gives you a working plugin. While the old drivers you downloaded may work, they do not support changes I have made in those drivers improve how they work. (Specifically, if you try the new record function with those old drivers, I'm pretty sure Isadora will crash as soon as you try to play the ONI file.)

Best Wishes,
Mark

mark

@juriaan said:

A few things that I noticed

I'll go through your list point by point

1) Kinect v2 (Xbox One) is still crashing with the current version that was provided to us today. Sadly Isadora crashes so badly that I can't send the Crash logs to you since it doesnt create one with the normal workflow (So that you can send it from the Error prompt that pop-ups)

Yeah, I don't really expect it to work well yet. It was an accident that it went out with the 0.9.5 build, but I figured at this point I'll just put it in and let people try. I think people on MacOS will have a better result since the entire driver was hacked/created from the ground up. They will have to deal with 512x424 image being embedded in a 640x480 frame right now.

2) Load becomes massive like 70.6% on a Windows 10, MSI GE73VR 7RF Raider laptop (2000 euro gaming laptop with a Nvidia dedicated graphic card with just the Kinect v2 / OpenNi Tracker and a Projector attached to it's depth image in full screen mode)

It's unclear to me if you're talking about a Kinect v1 or Kinect v2 here. I guess it's Kinect v1 since v2 is crashing, but please be specific. That said, loads that high are not my experience. I'll edit this post in a few minutes with my load figures from Windows 10. Keep in mind that the GPU is going have like zero impact on the efficiency of this plugin. The data from the camera comes in as CPU bitmaps, and everything is thus handled on the CPU not the GPU.

3) When I say against Isadora that I only wish to receive the color map it still sends out the depth video signal as well.

I think that "against Isadora" must be a typo... I guess what you're saying is that if you attempt to get the RGB camera only, you cannot. Yes??

4) I can't get the recording to work properly. What I did was I set the File input to tracking.oni and told the plugin to Record, do a silly dance, and press 'stop'. Then I try to play it back by pressing 'Play' and it doesn't show anything. It stops the active sensor flow and showcases a blank screen. When I'm looking in my file explorer or it even made a file called tracking.oni it doesn't show up.

Did you save the file? Unless you specify a full path name (e.g., C:\Users\USERFOLDER\Documents) the plugin creates a path name based on the path of the Isadora .izz file. If you haven't saved the file, maybe this fails?

Best Wishes,
Mark

mark

@demetri79 said:

I'm so bumed and confused

Well, I am not sure what to say to this. I mean, it seems like you had everything working for some time, and now it doesn't work. If you reboot the computer, there is no way that any data or code from the tracker plugin is lingering inside of your computer's memory. So I really don't think it's the OpenNI Tracker itself. Furthermore, you indicate the previous version (0.9.2), which was working well previouslyl, no longer works.

We need some way to prove the hardware itself is still working, because given the description of the frustrating process you describe, and the fact that the green light stops blinking, it is possible that your Kinect has failed.

Do you have some other software you could use to try to use that Kinect, e..g, the demo or release version of NI Mate perhaps? Or else the free Shapify.me from the App Store? (This successfully worked with my Kinect 1414, including moving it up and down using the internal motors. It never succeeded in scanning me, but I could see the image.)

Best Wishes,
Mark

mark

@Demetri79 and everyone else:

I've put together a simplified Kinect test app for MacOS using the examples from the open-source original OpenNI distribution.Inside this download you'll find two scripts: run-closest-point-viewer.command and run-simple-viewer.command. The first runs the "ClosestPointViewer" test app, the second runs the "SimpleViewer" test app.

Could people running High Sierra try running run-simple-viewer.command and verify it shows a depth map image? I need this to kind of prove that it is a good, universal test to show a working connection to the Kinect without any of my modified drivers in the loop.

These test apps work for me on High Sierra 10.13.6 (17G13021)

Note that these scripts and apps are not code signed. To run them under High Sierra, right click the script and choose "Open". You'll get a dialog like this:

Click the "Open" button to run the script. If everything is working, you should see this:

Note that it might take two attempts before it works. If you don't see an image the first time or the script otherwise fails, try it once more.

Diagnostic information will print in the Terminal window that opens. If this fails, posting the the info in the Terminal window might help me to debug your setup.

kinect-test-macos.zip

Best Wishes,
Mark

mark

Dear All,

As you can see, I'm trying to address the problems people have identified here. But, to give me a better picture of how this actor is working out, it would be good to hear from people for whom it is working too.

Please be specific about what sensor your using, and the operating system with version if it's not in your signature line. I assume you're all working with Isadora 3.0.7 or the current beta if you are a beta tester.

Best Wishes,
Mark

Juriaan

@mark

"Yeah, I don't really expect it to work well yet. It was an accident that it went out with the 0.9.5 build, but I figured at this point I'll just put it in and let people try. I think people on MacOS will have a better result since the entire driver was hacked/created from the ground up. They will have to deal with 512x424 image being embedded in a 640x480 frame right now."

Yep understood :) Still wanted to let you know. I sent you the crash reports earlier that basically have the same behavior. Isadora starts just fine with the OpenNI actor, after a while (5 minutes or so) the OpenNI actor crashes Isadora. After that the time duration that Izzy crashes after a restart is like 1.5 minutes or so

"It's unclear to me if you're talking about a Kinect v1 or Kinect v2 here. I guess it's Kinect v1 since v2 is crashing, but please be specific. That said, loads that high are not my experience. I'll edit this post in a few minutes with my load figures from Windows 10. Keep in mind that the GPU is going have like zero impact on the efficiency of this plugin. The data from the camera comes in as CPU bitmaps, and everything is thus handled on the CPU not the GPU."

Isadora 3.0.7 (Stable), Windows 10. Kinect Sensor v2. I added a reg key to actually boost my Intel CPU to 30 frames for the Kinect. When I didn't do that I only got 15 frames causing the Kinect Verifier application to yell ;) On the Kinect SDK forums you will find a forum post describing how to disable CPU sleeping.

"I think that "against Isadora" must be a typo... I guess what you're saying is that if you attempt to get the RGB camera only, you cannot. Yes??"

When I use the 'capture mode' input of the plugin and set it to Color I also receive the depth image. What I expected to happen is that I only get the color image and that the depth image was disabled / not active (So a red line instead of a green line to the projector)

Did you save the file? Unless you specify a full path name (e.g., C:\Users\USERFOLDER\Documents) the plugin creates a path name based on the path of the Isadora .izz file. If you haven't saved the file, maybe this fails?

Nope, I didn't save the file. Will test that out later tonight when I have some spare time.

mark

@juriaan said:

Isadora 3.0.7 (Stable), Windows 10. Kinect Sensor v2. I added a reg key to actually boost my Intel CPU to 30 frames for the Kinect. When I didn't do that I only got 15 frames causing the Kinect Verifier application to yell ;) On the Kinect SDK forums you will find a forum post describing how to disable CPU sleeping.

Hmmmm I didn't need to do any of this. My LOAD was 21% with Windows 10 running on Bootcamp on my MacBook Pro. (I know that's not _really_ a PC, but it's pretty much a PC.)

I'm investigating the Kinect v2 crash now. I just ran the Kinect v2 for 10+ minutes, dancing around all the while, and no crash. I'm now trying it again without the debugger running... it's been 7 minutes and all is well.

@Juriaan : Did you need to be in the frame to get it to crash? Or could you just let it sit with no one in the frame? Had you added anything to your patch aside from the OpenNI Tracker and a Projector? (I suppose heavier load/stuff in the patch could change the picture.)

Best Wishes,
Mark