Interactive Wings

GoddessMoon2017

Thank you @Monty for your youtube video about how to use Kinect and NiMate. I got a little lost towards the end with all the Kinect point actors. I am still working on a learning curve.
I am wanting to create a set of wings that is

interactive and move up and down with movement of your arms. ( see attached photo)
1) is this too ambitious? Is there a more simple way to make this and get the wings to move? Should I make the basic model in Adobe ?
2) The other obstacle is the I am a poor artist that can't afford an Isadora license. I use the demo version.

I have worked though the Kinect tutorial. Now I need to see specifically how do you make the Kinect actor? How you do you hook it up so it will move with a video or person? Does this make sense?

dbini

- changing the width and/or height of the wings can get them to look like they are moving in a semi-flappy style.

- If you use NIMate to send OSC control data to Isadora, you could lock the centre point of your wings picture to the body x and y. i did this in a show a few years ago - there's something similar here where i attach some antlers to a dancer's shadow.

GoddessMoon2017

@dbini Your video is absolutely beautiful. The problem is I have no idea how you got there. In my frustration sometimes I feel Isadora is a big secret that i am not privy to. I need to know specifically how to make patches, Kinect actors, etc that will move wings when my arms move.

Woland

@goddessmoon2017

It's a very specific use case so I'm not sure anyone can give you step-by-step instruction on how to do it, but here is some advice and a possible workflow that I just brainstormed. None of it is tested, I was just programming in my head, so you'll have to build it and experiment and see how/if it works for you. Hopefully it will serve as a good starting point for you. My advice is to break your large project down into its most basic building blocks, build your small proof of concepts for each component, and then finally put the pieces together. This is how I approach projects.

List out what you're trying to do, then break that down. You want to make moving wings that track a person and flap when you move your arms, therefore your components would be:

Learn about value scaling and tracking.

Start out with a Shapes actor (as a stand in for the wings) and work on programming it to follow a Mouse Watcher actor (as a stand-in for the person) using the x/y horizontal/vertical data.
- The goal is to get the shape to move left and right with your mouse movements.
These tutorials on Value Scaling are old, but still relevant, as value scaling will play an important part (Part 1) (Part 2)
Pay attention to the Hold Range and Limit Scale Value value actors. They will be important.

Get Kinect Tracking working
- Go through the Isadora Knowledgebase articles on how to use the Kinect. (Part 1) (Part 2) (Part 3) Also make sure to download the relevant files.

Find out your range of movement by connecting the Kinect Data points for your torso to Hold Range actors. This will give you the max and min values recorded when you move. This gives you numerical values for your range of movement.
Use the kinect torso point and your x and y values to get a Shapes actor to move with you. Once you can move left and right and have the shapes actor move the way you want, you can later use this method and substitute in your wing media. (Limit Scale Value actors are helpful to convert one range to another, e.x. converting a range 0-100 to a range of 0-360)
Use the kinect torso point z value to control the size of the shapes actor.

The higher your z value, the closer you are to the camera, the larger you'll want your wings to be.
Run the Shapes actor through a Zoomer actor and find the right scaling to make your Kinect Z data increase and decrease the size of your wings

Find an image, video, or 3ds file of wings
- You'll want to legally source media that's either public domain, licensed for commercial use without attribution, or make your own from scratch. (Using random images from google is illegal as it violates intellectual property laws.)
  - Try Pexels, or Pixabay. You can also go to google images, select Tools > Usage Rights to sort by license type. Don't actually go by this or download based on what google says the license is, when you find an image, go to the actual site, make sure it is licensed for use in the manner you want to use it in, then download it from the site itself.
  - I searched around for "angel wing png" and found this image which I'll use as an example.

Put your wing media in Isadora and figure out the way you want it to look when you manipulate it

Use Envelope Generators, Envelope Generator++, Ease In-Out, or Ease In-Out 2D actors to change values over time on a FFGLPanSpinZoom, Zoomer, or 3D Quad Distort actor until you like the "animated" look of your wings flapping. Tweak until your happy. (as @dbini said, changing the width and/or height of the wings can make them look like they are moving in a semi-flappy-style).
In the end, you want to trigger this by moving your arms, but first start out just making it animate the way you want using a Keyboard Watcher instead of motion tracking to trigger it. (Hook it up so that a keypress triggers the flapping.)

Figure out how you want to use the kinect data from your arms. What value will trigger the wing flapping?
- Note: To do this part (and any part where you need to repeatedly test something with the kinect, you'll either want a second body to move around in front of the kinect while you program and watch the values, or you'll want to run the patch on your primary computer and then use TeamViewer on a laptop next to you to control the main computer for programming purposes so that you can be your own moving body and not have to run back and forth between being in front of the computer and being in front of the kinect.
  - One idea: Triggering the flapping based on how fast you move your arm(s)
    - Take x and y value from a kinect arm data point, feed it into a 2d Velocity actor. This will give you the velocity of your arms. (You'll want to experiment to find a velocity of arm movement you like by looking at the values while you or somebody else is in front of the kinect moving)
    - Pick "trigger" velocity and say, okay, I want the wings to flap when I move my arms with velocity "x".
    - Use a Comparator actor set to "ge" (greater than or equal to). Connect the 2d Velocity to one input and set your chosen trigger velocity as the number that you are comparing the incoming velocity to.
    - Connect the "true" trigger of the Comparator actor to a Speak Text actor set to say "Flap". (This is a stand in for the actual series of actors that will trigger your wings flapping. Speak Text actors are great to use for tests because the give you auditory feedback on whether something you are trying worked or not and they do not need to be reset so you can keep testing over and over.)
    - Once you can trigger the Speak Text actor by moving your arms at the speed you'd want to cause the wings to flap, you're ready to connect it to the series of actors you made to "animate" the wing flaps.

There are other steps you will need to take, and again none of this is tested, I was just doing a theoretical build in my head based on how I would approach this. What you are trying to do is not an easy thing. There are a lot of moving parts and I suggest you take this approach of breaking it all down into prototype building blocks and then only connect the blocks when you have each block working. Hopefully what I have laid out here will serve as a good starting point for you. There are just so many different ways to accomplish things that I don't think anybody can tell you exactly how to do exactly what you want.

Here's an example file to get you started.

Best wishes,

Woland

Edit: In combining everything, I'd try to have the performer in front of a rear projection screen so that the wings could be projected from the back (once you figure out the scaling to make them follow the performer properly with the Kinect stationed in front of the performer). A trickier way to do it would be to use IR Tracking in combination with body mapping to project the wings from the front.

Skulpture

Yes this is a very specific case. If I was doing this project i'd set aside about 4 weeks worth of experimentation. I can think of a few avenues to explore which are all very similar to @Woland methods. I will see if I can think of any other ways to do this over the coming weeks.

Woland

@goddessmoon2017 said:

2) The other obstacle is the I am a poor artist that can't afford an Isadora license. I use the demo version.

As @Skulpture said, this is going to be complex build that will likely take weeks. Not being able to save your work will be a huge obstacle, but I may have a solution. Please check your messages.

Best wishes,

Woland

GoddessMoon2017

Thanks to @Woland and @Skulpture for this awesome tutorial. I am still reading over it and I"m amazed at the depth and time you took to make this. I cannot thank you enough! This will help me grow and learn more. I'l come back if I have any questions. Thanks guys

Skulpture

@goddessmoon2017 said:

Thanks to @Woland and @Skulpture for this awesome tutorial. I am still reading over it and I"m amazed at the depth and time you took to make this. I cannot thank you enough! This will help me grow and learn more. I'l come back if I have any questions. Thanks guys

That was all @Woland work I can't take credit for it. But me and him have been talking about your project and the best way to approach it. I look forward to seeing your R&D into this.

GoddessMoon2017

When I started this project I had no idea it would be so hard to do. There has got to be another way. When I use the mouse watcher the image will move but when I use Kinect point it doesn't move.

Woland

@goddessmoon2017

It certainly is an ambitious project but it sounds like you're almost there so don't lose heart! I think the final hurdle might be mastering scaling.

Kinect Point should give you x, y, and z data for the torso. Use that x and y data to try controlling the horizontal and vertical position of a Shapes actor first. In the screenshot you attached, I don't see the x and y data of the Kinect point connected to anything. Try connecting it to a hold Range actor first (to find your range for the input of the kinect data by moving around), then connect it to a Limit Scale value to scale the kinect data (the min and max show by the Hold Range actor when you moved around to get the range data) to the same range as the Shapes actor. If you can do that, you can make the wings move, it's the same principle, but hooked up to the "horz center" and "vert center" of the Zoomer actor for the wings. The Isadora manual entries for the Hold Range and Limit Scale Value actors will be helpful.

Best wishes,

Woland

ppp586

@woland

Hi woland, could you help me? how to install syphon for processing 2.2.1 ?

I get some trouble like the fellowing screenshot.

Thank you!

Jingping

Woland

@ppp586 said:

I get some trouble like the fellowing screenshot.

I don't work with Processing outside of the Kinect setup, and it's not our software, so I'm afraid I don't know what that Processing error means or how to fix it.

The workaround is that the following link contains everything you need for the Kinect setup, including that correct version of the Syphon library for Processing 2.2.1: https://www.dropbox.com/sh/1ga...

You should be able to just drop the Syphon library in the folder where Processing stores its libraries.

ppp586

@woland

Hi woland,

I got the dropbox files. Thank you so much!