• Products
    • Isadora
    • Get It
    • ADD-ONS
    • IzzyCast
    • Get It
  • Forum
  • Help
  • Werkstatt
  • Newsletter
  • Impressum
  • Dsgvo
  • Press
  • Isadora
  • Get It
  • ADD-ONS
  • IzzyCast
  • Get It
  • Press
  • Dsgvo
  • Impressum

Navigation

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Popular
    • Tags

    Help - motion tracking for moving surfaces

    How To... ?
    8
    17
    5559
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Bodo
      Bodo last edited by

      hi folks

      i´am working with an artist on a peace this days - first time i realy work with the new mappingfeatures - it is realy great and works perfectly - we map different surfaces like triangle, circles (1mx1m) and so on - the idea (for the next rehearsels in may 2016) is to have a motion tracking system for three of these 7 surfaces that the actors can take them and move with them and the mapping is following them in each axis....how you guys would realise this - with ir-led on the corners of the surfaces + ir camera and ir light, kinect or is there a better system, do i need a nother software infront of izzy - would this be very difficult - especialy because they will cross each other? - i never worked with motion tracking before...

      thanks a lot for any help!
      ciao bodo

      MacPro Late 2013 OSX 10.13., Izzy 2.6.1

      http://studio6.berlin

      1 Reply Last reply Reply Quote 0
      • Skulpture
        Skulpture Izzy Guru last edited by

        Hi,

        Quite a big task and arguably one of the hardest things to achieve. It is possible - but with limitations/adaptations. There is only one artist I know who has achieved this who is:
        Marco Tempest
        http://www.marcotempest.com/
        One of the many ways he has done this is using the method you suggest in your post. 
        I would start with a basic rectangle, IR LED's and webcam with IR filter and use the publishing feature of mapper, or even the 3D quad distort actor at first to get the hardware and method working. Then work towards other shapes and multiple instances. 
        Kinects are good at working out depth and people; not so much tracking exact shapes. I hope this helps for now. 

        Graham Thorne | www.grahamthorne.co.uk
        RIG 1: Custom-built PC: Windows 11. Ryzen 7 7700X, RTX3080, 32G DDR5 RAM. 2 x m.2.
        RIG 2: Laptop Dell G15: Windows 11, Intel i9 12th Gen. RTX3070ti, 16G RAM (DDR5), 2 x NVME M.2 SSD.
        RIG 3: Apple Laptop: rMBP i7, 8gig RAM 256 SSD, HD, OS X 10.12.12

        1 Reply Last reply Reply Quote 0
        • Bodo
          Bodo last edited by

          Hi Skulpture

          thanx for your thoughts - but how to get the signal fom the camera into data which i can use in isadora? i remember (10 years ago) Mark used Eyesweb infront of isadora for it - is it possible only with isadora this days...???

          ciao Bodo

          MacPro Late 2013 OSX 10.13., Izzy 2.6.1

          http://studio6.berlin

          1 Reply Last reply Reply Quote 0
          • crystalhorizon
            crystalhorizon Beta Platinum last edited by

            I asked this task myself a lot...

            Alexander Nantschev | http://www.crystalhorizon.at | located in Vienna Austria

            1 Reply Last reply Reply Quote 0
            • Skulpture
              Skulpture Izzy Guru last edited by

              Personally I would use another computer; mac or PC with isadora or max/msp or similar doing the tracking and then another computer doing the mapping (isadora probably).

              The first computer would need to do the tracking using IR and then send the XY and Z (depth) co-coordinates of each corner over OSC. 
              OBJECT ONE (rectangle) > Corner One X Y Z > Corner Two X Y Z , etc etc, 
              OBJECT TWO (square) > Corner One X Y Z > Corner Two X Y Z , etc etc, 
              The only thing I can't think how to solve is how to distinguish between Object ONE and Two? Because all the camera will see is a load of IR lights... know what I mean?

              Graham Thorne | www.grahamthorne.co.uk
              RIG 1: Custom-built PC: Windows 11. Ryzen 7 7700X, RTX3080, 32G DDR5 RAM. 2 x m.2.
              RIG 2: Laptop Dell G15: Windows 11, Intel i9 12th Gen. RTX3070ti, 16G RAM (DDR5), 2 x NVME M.2 SSD.
              RIG 3: Apple Laptop: rMBP i7, 8gig RAM 256 SSD, HD, OS X 10.12.12

              1 Reply Last reply Reply Quote 0
              • Bodo
                Bodo last edited by

                thanks a lot! - how you would track in izzy - can eyes actor handle those ir led`s?

                MacPro Late 2013 OSX 10.13., Izzy 2.6.1

                http://studio6.berlin

                1 Reply Last reply Reply Quote 0
                • Fred
                  Fred last edited by

                  Isadora may be able to do moving projection control but it will probably not be accurate. A few problems you will encounter is that Isadoras 3D control is quite limited (you are moving these objects in 3d space and should move the projection in a virtual 3d space as well to match propoerly). The biggest problem is that to avoid gimbal lock when tracking rotation you will need to describe the rotations with a matrix or quaternion numbers neither of which can be used in Isadora. Any good tracking system will recognise the object and and give you proper rotations (you will have a lot of trouble just using corner points and trying to give the idea of rotation to your projection).

                  You can achieve this a few ways, one would be to use an IR reflective coating to paint a marker only visible to the IR camera and use a tracking system with orientation to track the object (something like AR toolkit or some other marker recognition). Other options include some kind of rigid body motion tracking system such as optitrak arena. These solutions will easily know the difference between any objects.
                  Also whatever tracking and projection system you use will need to be fast if you want it to match - very fast. This is possible but requires some hardcore gear and skills.

                  http://www.fredrodrigues.net/
                  https://github.com/fred-dev
                  OSX 13.6.4 (22G513) MBP 2019 16" 2.3 GHz 8-Core i9, Radeon Pro 5500M 8 GB, 32g RAM
                  Windows 10 7700K, GTX 1080ti, 32g RAM, 2tb raided SSD

                  1 Reply Last reply Reply Quote 0
                  • Bodo
                    Bodo last edited by

                    @Fred - thank you for all those detailed informations! - to let you know: rotations are not realy nessesary there are just a 2d triangle, circle and rectangle - which will be moved by an actor mainly from one side of the stage to the other - maybe a little in the hight and depth - do you think this make it easyer and make itpossible to do it just with izzy???

                    MacPro Late 2013 OSX 10.13., Izzy 2.6.1

                    http://studio6.berlin

                    1 Reply Last reply Reply Quote 0
                    • Fred
                      Fred last edited by

                      It is up to you and your clients- how clean they want the tracking. In a theoretical scenario when everything else on the stage is very well lit so if the tracking is off it is burned out by lights and the actor never rotates the object at all - in any vector (keeps parallel to the projector at all times and does not rotate the shapes). Also depends on what you are projecting on them and how much you care if they are off. Marco Tempest stuff is super slick and does not allow for this kind of inaccuracy(well it does but a much lower margin of error than izzy can do. 

                      If you can calibrate your projector and camera (a difficult task but that means you know where each pixel in the camera relates to each pixel in the projector this means compensating for lens distortion if you do it properly) then I would cover the boards in a material that is highly IR reflective, I would make sure that nothing else in the view of the camera is IR reflective and then you have a mask and a blob to track . It will be slow, there will be a lag when you move the item the video will follow several frames later. If that is ok then you are on the way and you can use Izzy kind of. Getting your board to be the only IR reflective source on the stage will be a difficult task and negotiation with costume and set. You would be surprised what reflects IR light and you need a laptop, IR light and small IR camera to test all the materials that will be on stage in that scene.
                      In the end it depends if you are willing to settle for a kind of OK solution, but the illusion works best by taking care of all the little details, this requires other hardware and software.

                      http://www.fredrodrigues.net/
                      https://github.com/fred-dev
                      OSX 13.6.4 (22G513) MBP 2019 16" 2.3 GHz 8-Core i9, Radeon Pro 5500M 8 GB, 32g RAM
                      Windows 10 7700K, GTX 1080ti, 32g RAM, 2tb raided SSD

                      1 Reply Last reply Reply Quote 0
                      • Bodo
                        Bodo last edited by

                        thank you so much Fred for taking your time to give this detailed information - i will do a test with izzy and maybe we have to hire an expert...

                        MacPro Late 2013 OSX 10.13., Izzy 2.6.1

                        http://studio6.berlin

                        1 Reply Last reply Reply Quote 0
                        • C
                          clmuller last edited by

                          Hi Bobo,

                          I've a similar project, how is yours ?
                          Thanks for the feedback
                          Claudio
                          1 Reply Last reply Reply Quote 0
                          • vanakaru
                            vanakaru last edited by

                            To keep different shapes IR pointers not mixed up you may want to use different IR wavelength LEDs - like 850nm; 750nm - with appropriate filters and separate cameras. The other possible help could be a GPS or even Wii attached to the shape. This would give you additional clue about specific objects location in space.

                            I have not done any of this - so it is just a thought.

                            MBP 4.1 & MBP (Retina, Mid 2012) MBP Retina 2017

                            1 Reply Last reply Reply Quote 0
                            • Marci
                              Marci last edited by

                              OpenCV can handle 2D primitives tracking from any mjpg (jpg) src... there are Python examples out there: http://stackoverflow.com/questions/11424002/how-to-detect-simple-geometric-shapes-using-opencv Might be some sketches on the processing forums covering it using the Java OpenCV libs that could cast a mask into Izzy via syphon... will have a ratch n' see what I can find. As per @Fred: calibration of camera to Kinect to projector will be crucial for quality results, and will need recalibration for every rig. RGBD Toolkit / DepthKit provides this for Kinect & KinectONE respectively... see http://www.depthkit.tv/rgbdtoolkit/

                              rMBP 11,3 (mOS 10.13) / rMBP 11,4 (mOS 10.14) / 3x Kinect + Leap / TH2Go
                              Warning: autistic - may come across rather blunt and lacking in humour!

                              1 Reply Last reply Reply Quote 0
                              • Bodo
                                Bodo last edited by

                                Hi clmuller,

                                because of budget we decided to map the stuff manually (i realised that there is a bunch off fast and expensive hardware nessasary and of course a ot of skills) to realised a good working trackingsystem  - so no motiontracking right know - but would be happy to get some news if you find a good and inexpensiv way - we are going to start the project next week...

                                ciao Bodo

                                MacPro Late 2013 OSX 10.13., Izzy 2.6.1

                                http://studio6.berlin

                                1 Reply Last reply Reply Quote 0
                                • crystalhorizon
                                  crystalhorizon Beta Platinum last edited by

                                  Mark mentioned in the Werkstatt Berlin at his IR tracking workshop that he would love something like eyesweb for Mac and that he probably will write it on his own. That would be really great.

                                  Alexander Nantschev | http://www.crystalhorizon.at | located in Vienna Austria

                                  1 Reply Last reply Reply Quote 0
                                  • fubbi
                                    fubbi last edited by

                                    I did this quite a lot a few years ago and I used IR leds in the middle of the surface for the version where things stayed on their side of the stage (no risk for accidental swapping). And geometry was not important (no clean mapping, just roughing it in)

                                    Having a good camera and well prepared LED was key. I had to upgrade from using the hacked PS3 camera from peau productions to using a CCD camera as recommended on frieder weiss homepage: http://frieder-weiss.de/eyecon/infrared.html, to get enough resolution for a smooth track and the width of a whole stage.
                                    I also prepared the LED with some tubing to concentrate the emitted light, which stabilized the track a lot. I posted images of how that was made in [this post](http://forum.troikatronix.com/cgi-bin/forum/gforum.cgi?post=8697;search_string=fubbi%20IR%20led;guest=22417115&t=search_engine#8697) but they don't load for me, I hope they load for you.
                                    I also tried a show where we had a processing application looking for corner leds and trapezing the image between the points, Now you can probably do the same by publishing inputs for your isadora map and tracking individual LED. 
                                    My two cents on keeping multiple LED identified would be to blink them in different frequencies and count the blinks. Either in isadora or processing
                                    F

                                    Mac M2 Ultra, 64gb — Berlin

                                    1 Reply Last reply Reply Quote 0
                                    • Skulpture
                                      Skulpture Izzy Guru last edited by

                                      @fubbi RE: "My two cents on keeping multiple LED identified would be to blink them in different frequencies and count the blinks."

                                      This is how the [blacktrax](http://blacktrax.cast-soft.com/) system work in conjunction with the d3 media servers.

                                      Graham Thorne | www.grahamthorne.co.uk
                                      RIG 1: Custom-built PC: Windows 11. Ryzen 7 7700X, RTX3080, 32G DDR5 RAM. 2 x m.2.
                                      RIG 2: Laptop Dell G15: Windows 11, Intel i9 12th Gen. RTX3070ti, 16G RAM (DDR5), 2 x NVME M.2 SSD.
                                      RIG 3: Apple Laptop: rMBP i7, 8gig RAM 256 SSD, HD, OS X 10.12.12

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post