MediaPipe, Pythoner and Isadora for body tracking and segmentation . . .
-
I just wanted to share the success I’ve had using Isadora’s ‘Pythoner Actor’ with MediaPipe (segmentation and pose landmark detection) and any decent camera feed (in my case an iPhone) to track individual body parts (up to 33). Also, by using its segmentation output with the ‘alpha mask’ actor, I’m able to add textures to the segmentation, or place the alpha mask output to any background.
It’s been great fun and has founds its way into my interactive stage work.
William
Isadora: 4.1.3 (arm)
Chip: Mac Studio M2 Max
Mac OS: Sequoia 15.1.1 (24B91)
iPhone 13 Pro
iOS 18.1.1
-
Thanks for sharing. Love to see what artists are doing with Isadora features.
Like you, I have had a lot of fun with MediaPipe in Isadora. It is remarkable how accurate and stable the tracking is and it is not too bad at dealing with occlusion issues. My next step is using a thermal or infrared camera system so that the interactivity makes sense in a blackbox/low light performance setting. MediaPipe is really effective with available light cameras as long as there is lots of light - I am looking for a solution for this that works with theatrical/performance lighting that is a lot more subtle. My debut performance with this tracking was a 'cats cradle' game. During intermissions members of the audience at an outdoor film festival could play with virtual strings between their fingers while the Isadora stage was projected onto a giant mango tree next to a large cinema screen. The low latency and accuracy made for a fun audience experience. My only issue was having to light the participant enough to get good tracking and then this lighting interfering with the participant observing the projection - for the onlookers it worked really well. Projection always seems to work better when you can mitigate control and limit other light sources.
This screengrab matches Pythoner MediaPipe with Isadora's 3D Ropes actor module and a Motion Blur. I really appreciate the output effect of this with the physics of the 3D Ropes creating a very responsive live dynamic.
