What do these do?

CitizenJoe

The new FFGL plug ins are great, but what do these do?

FFGLPremultiply = Premultiplies the video stream

FFGLUnpremultiply = Unpremultiples the video stream

Yes, I can read it, but I don't understand what premultiply means/does.

Thanks,
Hugh

eight

https://goo.gl/6Q64hb

mark

Dear Citizen Joe,

I must say, I had to educate myself on this too, because in CPU based version of Isadora everything was non-premultiplied alpha (which still make the most sense to me.)

**SHORT ANSWER**

The short answer is this: Most (but not all) FFGL plugins expect pre-multiplied alpha input and give pre-multiplied alpha output. Isadora currently outputs non-premultiplied alpha when it plays a movie and when you get live-input from a camera. Those non-premultiplied images are _automatically_ premultiplied before they are given to FreeFrame GL plugins. So in this case you don't have to think about it.

But some expect non-premultiplied alpha. And some output non-premultiplied alpha. The problem is that the FFGL spec does not say which flavor (premultiplied or non-premultiplied) is the standard. When I asked the FFGL developers list about this, one developer said "non-premultiplied" and another (a fellow from Resolume) said pre-multiplied. So, it's a bit of a frustrating situation because -- in fact -- we don't know which system a given FFGL might use.

At this point, I am heading in the direction of having all GPU images be pre-multiplied in Isadora. There are various reason for this which are explained in the Gory Details below.

Bottom line: unless you're trying to do masking with blurs, and using FFGL plugins to do it, you won't need these actors.

**GORY DETAILS**

Now, here come the extensive gory details. I hope it's useful (or at least entertaining.)

This whole business began when I started to encounter weird rendering problems as I added the GPU processing to Isadora. There would be these weird, dark halos around blurs that I couldn't figure out. As it turns out, it was because I was mixing non-premultiplied alpha and pre-multiplied alpha.

I saw this problem first when I tried some third party FreeFrame GL effects. I went to the FreeFrame GL developers mailing list and asked, "Is there a standard for the video processed by FFGL plugins? Is it supposed to be non-premultiplied or premultiplied?" The surprising answer was, "there is no standard." D'oh!!!! Well... at least it spurred me on to start figuring this out.

Here is how the two versions work.

Imagine a 50% grey image. In the non-premuliplied world the ARGB values look like this:

R = 1.0, G = 1.0\. B = 1.0, A = 0.5

In other words, the RGB part is actually white (all 1.0) and the alpha is 50% (0.5). To get the actual values you need to render to the screen, the computer must multiply the RGB part by the A part, ending up with RGB = 0.5, 0.5, 0.5\. The good thing about non-premultiplied is that the color fidelity is as good as it can be. The bad thing is the extra multiply, because it takes time. (Especially in the CPU world.) And, a problem that comes up later with blending images.

Here's the same 50% grey pixel in the pre-multiplied world.

R = 0.5, G = 0.5, B = 0.5, A = 0.5

The difference here is that the multiply operation has already been done for you. That speeds things up because, to blending two images requires one less multiply. But, you lose color fidelity. For instance, imagine a fully transparent red pixel.

R = 1.0, G = 0.0, B = 0.0, A = 0.0 (non-premultiplied)

R = 0.0, G = 0.0, B = 0.0, A = 0.0 (premultiplied)

As you can see, you could in fact replace the alpha of the first pixel (e.g., your own alpha mask) and recover the original color. That's not possible with the premultiplied version because the all three colors are set to 0.0 because of the pre-multiplication.

But the really important bit is when it comes to blending images. The math on is still tough for me to understand. But some pictures in a StackOverflow question I posted when I needed help may explain things

http://stackoverflow.com/questions/28649647/opengl-dark-halos-around-blurred-text/28655239#28655239

In the end, the _only_ way to blend two images that have alpha channels and a blur is by using premultiplied alpha. That's why I'm heading in that direction for Isadora.

Hopefully that sheds a little light. This is something I want to insulate the users from, so I need to continue to think about it and get all my alpha ducks in a row inside the code. But hopefully the above gave you a bit of insight.

Best Wishes,

Mark

CitizenJoe

Thank you, @mark, for the detailed explanation.

Hugh