Interactive projections through kinect?

  • Hi everyone,

    I just recently am getting involved in trying to create interactive projections through isadora, and hoping to use a kinect for it as well. I've used an IR camera before and motion tracked based on that contrast, but am looking for something a little less resource consuming than setting up a bunch of IR lights to create the back wall of light and have the actor dark. i was hoping the kinect would help for this. there is a project where the artist is picking up a painting and showing it around to the audience. This would be choreographed much slower of course, but it would be cool to have the projection appear on their 3'x4' painting surface and be able to follow it in real time. I've done this through after effects compositing (of course) but what would be the best method through isadora? or should i look into a separate software completely? 

  • using kinect for this would be possible, but fairly complicated, as it is programmed to look for human skeletons, and what you really want to look for is the rectangle of the painting.

    as you have experience of IR tracking, maybe a good solution would be to install some IR LEDs into the painting and track those using an IR camera.
    somewhere online there's video of an illusionist who uses this method to create entertaining projected magic stuff with a blank canvas. i'll have a hunt around and see if i can find it for you.

  • aha! found him - Marco Tempest:

    here's where he shows how it works:

  • wow this is exactly what i'm looking for. I suppose I could create the little LED lights on the corners and use an IR camera, then track it through eyes in isadora? Think it'll be able to handle that kind of latency? That looks pretty quick!

    I suppose I'll leave my kinect ideas for masking just on the performers, which i suppose that's the benefit. 

  • actually it sounds like I would need some sort of corner pin to tack to each of the 4 LEDs, and that way the image can be tracked and manipulated as the camera saw them. I'll purchase some and put them through a poster board to see what results I can get. Has anyone else attempted to try this within Isadora?

  • Beta Tester

    I don't have a video to show you but I have done this with big moving scenery "tagged" with IR LEDs on the corners and tracked with an PS3 Camera with the IR mod and an aftermarket lens. Use the 3D Quad Distort actor and some clever math to make it work

  • Awesome! I'll find some of these IR LEDs online and pin them up. I'm sure I'll be back asking advice when it doesn't go quite according to plan. Thanks for the leads though, all help is greatly appreciated.

  • Izzy Guru

    Yeah you don't really need a kinect to be honest. Even a basic camera with colour tracking of 4 points could work BUT for the 'illusion' IR is used; as quite rightly described by Dbini and the Marco Tempest examples.

    A standard CCTV camera or a hacked PS3 with IR mod trick will work for sure.
  • Izzy Guru


    The actor that will help you turn the image would be the "calc rectangle" actor. For each point eyes++ is seeing you forward the position into the x and y inputs.


  • cool, just ordered the ps3 eye yesterday so that should come in, can't wait to hack it and try it out. I'm interested to see the speed at which the tracking can handle and those limitations, but it should still give an interesting effect. I'm going to start researching the LED hook up now, I'm sure there's plenty on izzy forum to read on.

  • Also, just to chime in here, the Wii Controller has an IR camera that will track four infra-red light sources. OSCulator will pass the data into Isadora via OSC. Michel attempted to use this system for a show, maybe he can give a short report about the Wii and how it worked for him in this situation.

    The actor Michel mentions (Calc Rectangle) was created for him when he ran into trouble with that show. The Calc Rectangle is important because it keeps the "sort order" of the four points consistent, so that you can feed them into the 3D Quad Distort actor and it will also affect the right corner.
    Best Wishes, Mark
  • Izzy Guru

    Yes, as mark said we where using a Wiimote for this, at the end it didn't make it into the show because next to all other things we where doing the "real" actors never had time to play around with it to get comfortable on how they had to handle the projection surface. As you can see in Marco Tempest's video, fast movements let the projection always be a bit behind. The distance the wiimote worked ok was up to 8 meters, but depends extremly on the brightness of the LED's. For expanding the up - down and left - right tilt range we used 3 IR LED's on every corner tilted a bit away from each other, but it depends on what angle the LED's will emit, the larger the angle the less bright they are.


  • Ir leds is definitely a way to go

    Instead of tilt range you might modify the leds by wrapping them

    check out [this thread](;search_string=smoother%20tracking;guest=21876349&t=search_engine#8669)
    If the frame can stay in a certain range with little or nothing else on the same distance line line you could try trapping (tracking and mapping) it with a kinect.
    I wrote a processing app for this purpose but it involves using syphon and processing so let me know if you are into it.

  • This thread has really helped me out - thanks all! Just one thing - I now have four LEDs ona  board being tracked through the wiimote camera, in turn feeding the Calc Rectangle actor and the quad distort actor. The rectangle moves and rotates with the movement of the LEDs, but because the position of the LEDs are constantly recalculated, the video image mapped onto the rectangle constantly maintains its aspect. It rotates so far and then snaps back to its original vertical state.

    Am I missing something simple here?

  • If you are on OSX you can also try AR toolkit, there is a version for quartz. You have to mark your target with a fiducial (can be done with IR reflective material (you can train your own fiducials). The advantage of this is that is gives a homography matrix back that allows you to easily get a 3D rotation of the target on which to re-project. I have used this a few times and it works well.