COMMUNITY POLL: Most Important Video Effects? (Quick Response Please!)

crystalhorizon

Here is my Christmas list for Santa ;-)

Motion blur

Video Delay

Freeze

Chopper

Difference

Zoomer

Multimix

Gaussian blur

The top would be:

Interactive Grid like in eMotion where like the Motion brush it changes interactively:

See minute 0:27

[EDIT: Some of the items were not effects and are already rendering on the GPU, so I edited those out. -- MC]

ioio

1. "anti fisheye effect" for correcting wide angles camera...very important for good camera tracking.

2\. Homography transformation effect... to correct perspective.

DusX

Additionally I like using 'Color Measure' and 'brightness'.

ioio

I forgot a video gpu buffer.

LPmode

Difference

Buffer - (there is a video FFGL delay RGB but it's not as flexible)

Brightness Calculator

Eyes- even if this is not an effect

eight

GPU Motion blur

GPU Accumulator with arbitrary amount of frames

GPU PhotoShop-like blending modes

--8

vanakaru

In some form Eyes GPU would be awesome, but the ones I use the most are

Freeze

Video Delay

Blur(any kind)

artoo

thanks mark for this first package of ffgl plug ins my wish: -deinterlace and maybe -refraction

primaldivine

Luminance Key (I have installed a FFGLLumaKey, but need to test it)

Calc Brightness

Difference

Explode

Shimmer

Posterise

Threshold

TimeBlur (FF)

Wave (FF)

TimeWarp and/or TimeSlice (FF)

Bloom (FF)

Burn (FF)

Glow (FF)

Kaleidoscope (FF)

QC Filmstock is also great... for adjusting video to look like film with parameters Saturation, Contrast, Grain and Vignette.

Kinect...?

thanks Mark!

primaldivine

Pixel Mapping (without needing a Lanbox)

mark

Thank you all. I'll be passing this information on the programmer.

I want to do a bit of "thinking out loud" regarding some of the plugins you're requesting.

I'll break them into two groups:

**Group A: Requires GPU Readback -- Eyes, Calc Brightness**

I'm going to need to do some research to see how much of a performance hit something like Eyes and Calc Brightness are going to impose. In those cases, you will need to read from the GPU to the GPU, which is just the absolute worst thing to do when what your goal is performance. GPUs are very good at accepting data _from_ the CPU, but _giving it back_ is not what they're designed to do. (I'm not saying it can't be done, just wondering how much it will affect performance.)

**Group B: Requires Loads of GPU Memory – Buffer, Shimmer**

These actors concern me a little because of the limited memory on a graphics card – at least when compared to a CPU. The worst case scenario is if you're exclusively processing full HD video (1920x1080) -- each frame is about 8MB. The GPU memory is going to disappear pretty fast if you start using the Buffer actor to keep those massive textures in memory.

The card in my recently bought retina display is 2048MB. That means a total of 256 frames of video. Each Stage output uses 2 to 6 frames depending on what you're doing. Every FFGL effect you use eats one more frame of GPU memory.

I mean, I quire certain it can be done. But that GPU VRAM is going to dry up really quickly. At the very most, the number of frames you could put in the buffer (again, full HD) would hover somewhere just over 225 (given a few stages and playing a few movies, etc.). That's 9 seconds of video at 25fps, and 7.5 seconds at 30fps.

Not a lot. But maybe that's all you're expecting?

Best Wishes,

Mark

ioio

That" what I expected and for me 9 seconds is enough.

crystalhorizon

Interesting thoughts and information Mark.

Skulpture

Sorry I missed the chance to comment here - but all the main actors I use have been listed. Mad busy a work yesterday.

Fred

@Mark Buffer should always be CPU (RAM) but upload frames as needed to the GPU (GPU only needs what we see in a frame and buffer only shows one frame at a time) I have a buffer style video recorder and looper that lets me load 10 gig of frames into ram that works great and uploads a frame at a time to the GPU as needed)

Any chance you would switch to using the openCV library for eyes++, it also has a lot lot lot of other great functions and would be an incredible addition to Isadora (Izzy would need a few more types to deal with it but nothing it does not deserve and could not make use of). Although much of the work is CPU, there is a lot of stuff that is now GPU accelerated, it is all cross platform and BSD license.

Alternately there are always compute shaders for the GPU or openCL/CUDA acceleration that are widely adopted and work well.

Fred

ioio

@Fred I used GPU Buffer with a "live feedBack program" with glsl in openframeworks with 4k resolution, and a 2GB buffer (not very long) gave the ability (by blending buffer and live input) to slow down and stabilize the effect which is very fast and hard to pilot. And the same effect with CPU buffer slowed down a lot the process which was no more real time.

ioio

Fred

@ioio The dev branch of OF and the nighlty builds are now 64bit, giving you access to pretty much all the ram in your computer. The system I made has 1 frame latency to input and 1 to output. 4k GPU buffer will eat all your gfx memory for no reason or need and give you less memory to play with. As for making this on GPU, any latency can be minimised with efficient code and threads to handle capture decompression (especially copying or passing a reference of the image to the destination) . I used the blackmagic SDK to get the frames (some OF uses qtkit or older versions quicktime for capture). These libraries can be very slow getting frames and unwrapping them. That is the joy of coding, the efficiency and performance are scalable if you have the time and skills. I think to replace the functionality of the buffer in GPU would be very inefficient especially considering how much RAM machines have as standard vs GFX cards, especially on laptops.

Fred

ioio

@Fred you're certainly right. In my case i only use video capture to inject a signal (it can be poor in latency and resolution) after it works in a closed loop in the gpu with two buffers which are treated with shaders effect (zoom shift contrast rotate, threshold hue etc... ) and mixed together. So for me with a 2 gb GPU, a few seconds of video buffer to slow down at some times, it runs really smooth on 4K with 5% CPU usage on my laptop.

Fred

@ioio this kind of buffer is pretty interesting though, usually it is coded into things like a feedback buffer which could be a cool effect, or multi pass blurs use the same tool to store the image for multi-pass processing.

eight

Here is my Cinder based multithreaded buffer implementation, in which the frames are kept in CPU memory and are consumed by the GPU as needed: https://github.com/eighteight/CinderEight/blob/master/PS3EyeSlowMo/src/PS3EyeSlowMoApp.cpp

This approach will be limited by the size of the RAM (there is no limit check for that though at the moment).

In addition, the frames could be jpeg–encoded to conserve CPU memory, as shown in this example:

https://github.com/eighteight/Cinder-VideoStream/blob/master/samples/CinderVideoStreamServer/src/CinderVideoStreamServerApp.cpp#L118

These examples are using PS3 Eye as input, but can be generalized.

--8