[ANSWERED] Webcam background removal

photogramdude

Webcam background removal, I've been looking at GLSL options (but would need to set a pretty hefty 'webcam' input variable I imagine).

There's a few methods online using Python or TensorFlow etc.

Anyone done this or an approximation? I've obviously done a shape key with gaussian blur etc but not quite the same as segmenting head and shoulders.

Thanks as ever, geniuses!

Rob

DusX

do you have control over the background image?

mark

@robhblack said:

Anyone done this or an approximation?

Well, this is pretty rough approximation. Load the attached patch, start up your webcam in the Live Capture Settings, and try the following:

Simple Background Subtraction Scene:

1) Make sure you are completely out of the frame and press letter 'g' -- this captures the background without you in it.
2) Now move into the frame. You should see your face/body with a black background.

The background subtraction algorithm will make white wherever there is a difference between the background and the person in the frame. This mask is then used with the Add Alpha Channel actor to mask out everything except the body/face. You can adjust the 'threshold' input of the "Threshold" actor to adjust the mask.

Background Subtraction Tracking Scene:

1) Make sure you are completely out of the frame and press letter 'g' -- this captures the background without you in it.
2) Now move into the frame. You should see an oval tracking your movements.

The white area you saw before is now being used by the Eyes++ actor to track your movements. A Shapes actor creates the oval that follows your movements, and this oval is used as a mask with the Add Alpha Channel actor. As before, you can adjust the 'threshold' input of the "Threshold" actor to perhaps improve the tracking. (Look in the monitor shown in the Eyes++ actor to see how it's doing with the tracking.)

This setup does not have any way to find out the size of your face and adjust the oval.

Caveats:

Both of these patches work best if your camera's auto-exposure is off, if it's possible to turn it off. If the auto exposure adjusts the image too much, it won't the background subtraction algorithm won't work well. It also requires that the camera not move at all once you've snapped the background. If the webcam moves, you'll need to snap the background again.

This is not by any means a perfect solution. But maybe it spurs some ideas.

Best Wishes,
Mark

background-removal.izz

photogramdude

Hi, those are great solutions and I didn't realise Isadora could do that. I have some ideas regarding linking multiple eyes together to lock on to the eye shape and then put a cluedo cutout over with a gaussian mask. Have attempted a couple of versions attached sans eyes prior to this. Obv foreground faces not always more luminant than background and a variety of other problems inherent in a non machine vision approach. ndi fakebg.izz ndi fakebg2.izz

Ultimately for our application we're going for zero user requirements so trying to avoid asking the user to do anything. However, it could be possible to gamify this step in the interim as an onboarding... Thanks!