COMMUNITY POLL: Most Important Video Effects? (Quick Response Please!)

[mark]

Thank you all. I'll be passing this information on the programmer.

I want to do a bit of "thinking out loud" regarding some of the plugins you're requesting.

I'll break them into two groups:

**Group A: Requires GPU Readback -- Eyes, Calc Brightness**

I'm going to need to do some research to see how much of a performance hit something like Eyes and Calc Brightness are going to impose. In those cases, you will need to read from the GPU to the GPU, which is just the absolute worst thing to do when what your goal is performance. GPUs are very good at accepting data _from_ the CPU, but _giving it back_ is not what they're designed to do. (I'm not saying it can't be done, just wondering how much it will affect performance.)

**Group B: Requires Loads of GPU Memory – Buffer, Shimmer**

These actors concern me a little because of the limited memory on a graphics card – at least when compared to a CPU. The worst case scenario is if you're exclusively processing full HD video (1920x1080) -- each frame is about 8MB. The GPU memory is going to disappear pretty fast if you start using the Buffer actor to keep those massive textures in memory. 

The card in my recently bought retina display is 2048MB. That means a total of 256 frames of video. Each Stage output uses 2 to 6 frames depending on what you're doing. Every FFGL effect you use eats one more frame of GPU memory.

I mean, I quire certain it can be done. But that GPU VRAM is going to dry up really quickly. At the very most, the number of frames you could put in the buffer (again, full HD) would hover somewhere just over 225 (given a few stages and playing a few movies, etc.). That's 9 seconds of video at 25fps, and 7.5 seconds at 30fps.

Not a lot. But maybe that's all you're expecting?

Best Wishes,

Mark

[ioio]

That" what I expected and for me 9 seconds is enough.

[crystalhorizon]

Interesting thoughts and information Mark.

[Skulpture]

Sorry I missed the chance to comment here - but all the main actors I use have been listed. Mad busy a work yesterday.

[Fred]

@Mark Buffer should always be CPU (RAM)  but upload frames as needed to the GPU (GPU only needs what we see in a frame and buffer only shows one frame at a time) I have a buffer style video recorder and looper that lets me load 10 gig of frames into ram that works great and uploads a frame at a time to the GPU as needed)

Any chance you would switch to using the openCV library for eyes++, it also has a lot lot lot of other great functions and would be an incredible addition to Isadora (Izzy would need a few more types to deal with it but nothing it does not deserve and could not make use of). Although much of the work is CPU, there is a lot of stuff that is now GPU accelerated, it is all cross platform and BSD license. 

Alternately there are always compute shaders for the GPU or openCL/CUDA acceleration that are widely adopted and work well.

Fred

[ioio]

@Fred I used GPU Buffer with a "live feedBack program" with glsl in openframeworks with 4k resolution, and a 2GB buffer (not very long) gave the ability (by blending buffer and live input) to slow down and stabilize the effect which is very fast and hard to pilot. And the same effect with CPU buffer slowed down a lot the process which was no more real time.

ioio

[Fred]

@ioio  The dev branch of OF and the nighlty builds are now 64bit, giving you access to pretty much all the ram in your computer. The system I made has 1 frame latency to input and 1 to output. 4k GPU buffer will eat all your gfx memory for no reason or need and give you less memory to play with. As for making this on GPU, any latency can be minimised with efficient code and threads to handle capture decompression (especially copying or passing a reference of the image to the destination) . I used the blackmagic SDK to get the frames (some OF uses qtkit or older versions quicktime for capture). These libraries can be very slow getting frames and unwrapping them. That is the joy of coding, the efficiency and performance are scalable if you have the time and skills. I think to replace the functionality of the buffer in GPU would be very inefficient especially considering how much RAM machines have as standard vs GFX cards, especially on laptops.

Fred

[ioio]

@Fred you're certainly right. In my case i only use video capture to inject a signal (it can be poor in latency and resolution) after it works in a closed loop in the gpu with two buffers which are treated with shaders effect (zoom shift contrast rotate, threshold hue etc... ) and mixed together. So for me with a 2 gb GPU, a few seconds of video buffer to slow down at some times, it runs really smooth on 4K with 5% CPU usage on my laptop.

[Fred]

@ioio this kind of buffer is pretty interesting though, usually it is coded into things like a feedback buffer which could be a cool effect, or multi pass blurs use the same tool to store the image for multi-pass processing.

[eight]

Here is my Cinder based multithreaded buffer implementation, in which the frames are kept in CPU memory and are consumed by the GPU as needed: https://github.com/eighteight/CinderEight/blob/master/PS3EyeSlowMo/src/PS3EyeSlowMoApp.cpp

This approach will be limited by the size of the RAM (there is no limit check for that though at the moment).

In addition, the frames could be jpeg–encoded to conserve CPU memory, as shown in this example:

https://github.com/eighteight/Cinder-VideoStream/blob/master/samples/CinderVideoStreamServer/src/CinderVideoStreamServerApp.cpp#L118

These examples are using PS3 Eye as input, but can be generalized.

--8

[videovoce]

Hi Mark,

I hope it isn't too late to add my wish list for updated FFGL plugins. Here are the ones I am using in my current projects. Many thanks!

dots

slit scan

interlacer

tile

solarize

contrast adjust

threshold

video inverter

difference

effect mixer

motion blur

directional blur

time blur

lionel blur

gaussian blur

[DusX]

@videovoce 
'effect mixer', Yes.. thats a great thought

[leben]

Yes, effect mixer, please!

[Pascal]

difference

motionblur

timeblur

zoomer

effectmixer

[Ubik]

Scaler!!!

effectmixer

zoomer

chopper

autofade

In FFGLGradient I would like to be able to set Brightness and Saturation for each color. Also, to have circular, square or reflected gradient would also be nice. 

[mark]

Dear All,

Thank you for all the suggestions. The programmer is actively working on these now, and I'm going to prioritize the list in response to what you've said above. You can look forward to more news on this by the end of August.

Best Wishes,

Mark

[crystalhorizon]

Bravo Mark! Looking forward for the new effects!