Guru Session #13: Body Tracking with Depth Cameras (Fri May 8th, 6pm CEST/5pm GMT/12pm EDT/9am PDT)

n.jones

@juriaan I was just about to start looking into doing the same thing . I’d love to know more , if that’s ok?

Juriaan

@n-jones

Sure ! Will tag you in the thread that I'm planning to write about it.

Light FX UK

Looking forward to this one.....been wrestling with my kinect1 for a couple of days ......

Demetri79

Looking forward to the Guru session.

Loaded up the OpenNItracker for kinect v1 and its way better than the old NI Mate from years ago. I adapted a 3D cube that I normally control with a wiimote to be controlled in 3D space by my right hand (x,y,z) and spin controlled in the left. (x,y). This patch is developed for a stationary performer using hands only so far and I noticed a couple of problem:

1- Noise- even when all limbs are completely still there is still a small float value changing causing unwanted movement. need to find a way to either gate it out with some small threshold in mind so the gate closes and stops the small values from passing or some other circuit that can ignore the tiny changing data when the performer is still. any ideas?

2- when you arm is fully extended (hand z), the xy range is very limited physically.. so need a way to mathematically progressively scale the usable range better for x and y based on the z value. I don't really know what math and actor combo would accomplish this. anyone one have nay ideas?

I have made a short demo video showing the patch in action.. and included the kinect and scaling section of my patch in case anyone is interested in adapting it or suggesting improvements to the concept. thanks:)

Cube kinect control 1.izz

https://www.youtube.com/watch?...

bonemap

@demetri79 said:

Noise- even when all limbs are completely still

Did you try the smoothing property for this?

In terms of the 3D rotation there is a tutorial file download that provides the quaternion rotation three.js Javascript module for that: https://troikatronix.com/plugi...

Kind Regards

Russell

crystalhorizon

@demetri79 did you try the smoother actor?

n.jones

@juriaan thanks

Demetri79

yes i have all the skeleton tracker outputs feeding limit scales, then into smoothers.. I should have included a screenshot of the actors for people who didn't want to open the patch and have a look.. this doesn't totally eliminate the jitteriness while allowing for a responsive reaction time.. i dont want to completely choke the output otherwise once I move my hand it will take a while before the corresponding object motion happens... just have to keep experimenting with different approaches.

the thing I am looking for is a math equation to progressively scale the hands x and y values based on its z value..

My scenario is not an actual 3D file (.3ds) mine is made from quartz composer (CI) so I can input different videos to each side of the cube.. it looks 3D but isn't actually so that javascript actor provided in the tutorial for the sphere doesn't apply to my situation. I would have tired to use a .3ds cube but you can only utilize 1 texture input in isadora correct? so that means i can only play the same video on all side as apposed to what i want is different videos on each side.. my quartz cube actually has 7 video inputs.. (6 sides + a background behind the cube). that is what I would want in the 3D word but seems unachievable currently unless i am mistaken. My version also has other added benefits like being able to have all of the dimensions manipulated in realtime with an essentially infinate Width, Height, and depth.. so just need to work out some better scaling math.

The .3ds files do have a distort parameter but it is very limited in how much it can stretch and reshape compared to what mine can do.. that is another aspect I would like to explore to see if it is possible do more extreme reshaping of .3ds files.

mark

@demetri79 said:

I would have tired to use a .3ds cube but you can only utilize 1 texture input in isadora correct? so that means i can only play the same video on all side as apposed to what i want is different videos on each side.. my quartz cube actually has 7 video inputs.. (6 sides + a background behind the cube). that is what I would want in the 3D word but seems unachievable currently unless i am mistaken.

Well, if you have a single cube wrapped with one texture, you can most definitely get a single image on each side. It's simply a matter of mapping the images into the right place in the texture you're feeding into the texture map. I would use a of Matte or Matte++ actors (one for each side) to get the individual images in the right place in the texture, and then feed the final output of that to the texture map input of the 3D Player actor.

I'm sorry I can't provide an example the moment. I'm busy preparing for today's session. But maybe that's enough to get you going.

Best Wishes,
Mark

mark

Dear Community,

Here's the links for Guru Session #13: Body Tracking with Depth Cameras

Watch the Live Stream (also good for viewing later)

Download the session materials. (This upload will not be complete until 5:50pm! Please don't try to download before that time!)

See you a few minutes!

Best Wishes,
Mark

primaldivine

@mark

Live streams says private only.Probably not started yet, but usually you had it up ten minutes before.

crystalhorizon

It says private session

nandada

@mark YouTube says the video is private!?

jfg

Video private

mark

@jfg

Sorry everyone for my mistake.

The video is public now.

Best Wishes,
Mark

Light FX UK

Thanks for trying to answer the question in the broadcast.

Sorry I couldn't get the kinect to be recognised, just a black screen. I have attached the Isadora screen for info. Appears to be an error on the status and on the sensor uri. I closed down Izzy and reconnected but unfortunately no joy.

Thank you

Gavin

Astrid

During the session in the chat I mentioned our research on OSC over internet for BH
Keen to see if it works will all this data from the skeleton decoder (bet it will be fine)

Here's our notes in dutch (hope translate works for you guys):

OSC transmit over internet (various locations)

credits to my HKU colleagues Tjerk Stoop, Tony Schuite & Simone van Dordrecht.

Astrid

Sadly, I still seem to have an incomplete OpenNI Tracker
(re-downloaded, got Jurriaans working one & robooted after throwing out previous versions, all to no avail.
My version of the plugins is from today 16.56)
I am connecting to the Kinect, but my numbers are in a quirky range, would love to adjust the m/cm/mm thing....

Below is what I see (checked but there a no hidden properties).
Any ideas or advice?

Demetri79

My Kinect V1 (1414) and OpenNITracker v 0.9.2 was working fine for days and then booted up toady and it doesnt work at all randomly. I dind't change anything from last night when it was working. In the OpenNI tracker BETA status output i see: init device .. when i hover over the stauts it shows a message : inactive device init err ?? Sometimes status says resolving, sometimes, no device. So i the deleted all 3 actor plugins: openni tracker, skeleton decoder, skeleton visualizer, and installed the new ones provided today with guru 13 materials and restarted computer.. still same results.. not working at all.. same error message: inactive, device init err?? i tried all the obvious, restarting, unplug kinect usb, unplug power to kinect, re plug in, remove opennitracker actor, deleted prefs,tried every possible order for trouble shooting…isadora crashes when i even try and delete the opennitracker actor from a scene with no other actors in it. also crashes isadora if i even try to close the patch or quit isadora. so i went back to the previous versions before the ones provided today.. same results.. its all not working and makes no sense.. nothing changed.

mac pro 2012 high sierra 10.13.6, isadora 3.0.7 OpenNI tracker v.0.9.2 and 0.9.5 skeleton decoder 0.8.3 and 0.9.1 skeleton visualizer 0.9.1 and 0.9.0

DanielPineiro

Hi!

Thanks for the todays guru session. OpenNI Tracker actor has a lot of possibilities!!!

Unfortunately this is not working too well for me. It works fine but just for maximum of 2 or 3 minutes, and then the video freezes. The reset input does not work, neither changing scene. Only If I delete the OpenNI Tracker actor and put it again, it works again but just for minutes.

I am using Isadora 3.0.7 and a Kinect 1414 on a MacBook Pro running Catalina 10.15.4.

Thanks for the help!!!!

Best regards!

Daniel