Body tracking with Yolo

n.jones

@dusx I I've been using this for a while , using TD to send osc to izzy, probably a really unproductive and process intensive but it seems to work ? https://www.youtube.com/watch?v=Cx4Ellaj6kk

if there was some way of doing this in izzy that would be amazing !

DusX

@n-jones said:

if there was some way of doing this in izzy that would be amazing !

I is pretty easy to use MediaPipe with Pythoner. Which feature of MediaPipe do you use most... face, pose etc?
I can make an example file for this pretty quick. (YOLO runs faster [when GPU enhanced] and has better Pose tracking, that is why YOLO tracking has been worked on :)

n.jones

@dusx wow , that would awesome 🙌.
For me the things I find most impressive is the fact it’s runs locally without a need to install python and the ability to select or switch the models on and off 🤯 .
I’d love to keep my workflow in Izzy so any help in this would be amazing 🙏.

jfg

@dusx

Please :-)

Jean-François

DusX

@n-jones said:

without a need to install python

Well, this code will require some configuration of Pythoner.

By default Pythoner offers the full standard library included with Python, however, this doesn't include everything we need for this script.
So, a new Virtual Environment will need to be created, and then MediaPipe, OpenCV, and Numpy will need to be installed.
The module names for install are:

opencv-python
mediapipe
numpy

This is worth knowing, because the names for installation don't always match the names used for import into the python code. eg: opencv-python uses, import cv2, in the python code.
You can find available modules at: https://pypi.org/

If you haven't I recommend watching: Getting Started With Pythoner to get an understanding of how the configuration is handled. It really should only take 3-4 minutes to create the environment and install the modules needed, once you know how it works.

I will create a quick and dirty video tutorial to walk thru this for this specific script/project. ASAP

jfg

@dusx

1000 thanks

Best regards,

Jean-François

n.jones

@dusx thank you

Armando

@n-jones Have you guys seen movenet. Apparently is superfast !!!!

DusX

I have posted a 'rough cut' of a tutorial video on setting up Pythoner to work with MediaPipe to my personal YouTube channel.

Please let me know if you find any issues, or feedback regarding clarity etc.. I will do a round of updates to the video before making it available officially on the Troikatronix YT channel.

Attached here is an Example file (used in the video, but now updated).
Again, if you have suggestions to improve this let me know (it is meant as a starting point).

NOTE: Pythoner has the ability to use 3 different configurations for which python environment is used.
I will make a video to walk thru these and their pros and cons soon.
In this video, I focus on the global virtual environment setup Isadora supports.
The other options are a Local (to the project root) Virtual Environment, and the default environment included in the Pythoner actor (the most portable option, but limit in features).

Armando

@dusx Wow Ryan Bravo !!!!!! Fantastic !!!!!

jfg

@dusx

I have so much to do until the end of the month that I can't try. I'll get back to you as soon as I can. Thank you so much for your fantastic work

Best regards,

Jean-François

n.jones

@dusx

@dusx thanks so much for this The video makes it so much easier for me to understand now .

Definitely going to build it in the next project.

Pretty good for a rough cut by the way

Thank you !!

Armando

I just found a tool I think you'll love that was given me by the author. Colin Clark, an incredible person, a composer and software developer (and much more !) based in Toronto. He created a software called Movement OSC, (mac PC and linux) that does pose estimation and sends osc. It can also compare Movenet and MediaPipe so you can compare them ! The software formats the osc messages 4 ways Bundeled messages per axis, messages per axis, and bundled xyz array. I use just the latter then parse it in the Json parser

Here are the keypoints and the keypoints diagram

Enjoy !

0: nose
1: left_eye
2: right_eye
3: left_ear
4: right_ear
5: left_shoulder
6: right_shoulder
7: left_elbow
8: right_elbow
9: left_wrist
10: right_wrist
11: left_hip
12: right_hip
13: left_knee
14: right_knee
15: left_ankle
16: right_ankle
17: left_eye_inner
18: left_eye_outer
19: right_eye_inner
20: right_eye_outer
21: mouth_left
22: mouth_right
23: left_pinky
24: right_pinky
25: left_index
26: right_index
27: left_thumb
28: right_thumb
29: left_heel
30: right_heel
31: left_foot_index
32: right_foot_index

Keypoints diagram

GibsonMartelli

@armando

jfg

@armando

A thousand thanks. It works fantastic. Do you have an idea how to change the standard port and the format ? By every start it go back to 7500 and bundled Message per Axis and I cannot found how to change it.

Again a thousand thanks for this found.

Best regards,

Jean-François

Armando

@gibsonmartelli Hello Bruno and ruth !!!!

Armando

@jfg

Here is a test. In fact the software is even more interesting than what I see, apparently. It is a framework for testing different models that can live parallel lifestyles and be alternated and studied. Please enable from the menu the developer panel Astonishing.

movement-osc-test.izz

jfg

@armando

I have also made a test and more: I put it in a patch for an installation and it works very well. But for an installation there is two limitation:

- the window of movementOSC has to be active on a screen

- I haven't found the way to save the settings or to change the standard setting.

Anyway it is a very interesting software.

Best regards,

Jean-François

Armando

@jfg Yes you are right I also noted that. Give feedback to Colin, but have you seen the developer mode ? I am sure there is the possibility to correct that from within the software.

colinbdclark

Hi @jfg, I'm the author of MovementOSC. I'm really glad to hear it's been useful for you!

For the first issue, I haven't noticed this issue myself so I'm very interested to learn more and see if I can reproduce it. I just finished an residency at Jacob's Pillow recently where we used MovementOSC running in the background while another app (which transformed, visualized, and proxied OSC message to a robotic system and to Isadora) was running in the foreground. What operating system are running MovementOSC on, and what do you notice when it's not active on the screen?

The second issue, you're right, there's currently no persistence for settings. So it forgets any changes you make to the IP address, port, or model settings. I work on MovementOSC in my spare time, so I'm not exactly sure when I'll be able to add this feature, but I've filed an issue for it and will try to implement something when I can.

I'm excited to see what kinds of things you end up doing with it!

Colin