GLSL - baby steps (Or "How I Added Alpha Support to Some of TT's Shaders")

[Kathmandale]

Hi All,

since the GLSL guru session I've been determined to not just drop the 'new' actors into my patches but to learn to understand them better and begin to modify them and eventually (maybe, perhaps, one day) write my own. A lot of the tutorials I've found and the writing online about GLSL is really dense and seems to be aimed at people wanting to 'create something from nothing' drawing the textures in the shader. For now I'm more interested in using them to effect an existing video, as I guess a lot of Isadora users will be. I'd love there to be a bit more discussion on here about the subject and I'm sure it would help people get started working with shaders. So, I thought I'd take the plunge and post this.

I'm not a coder. Or at least, I don't consider myself one. I've tinkered around with arduinos and understood enough about the code I've used in certain projects to re-write things I've found online, or combined bits of other peoples work to make the thing I want. So I'm not a complete beginner either, but I certainly don't have extensive knowledge or experience with code. You've been warned!

I'm working on a project at the moment that, because of the way it's being created, really relies on having an alpha channel on lots of the video layers. We were also really keen to use some of the effects from the TT GLSL shader library and some of the wider world of 3rd party shaders. We quickly found however that many of the shaders we were interested in didn't preserve the alpha channel, we'd put RGBA video in and seemingly get RGB out. So I set myself the task of seeing if I could fix this and modify the code in the shaders we liked to keep the alpha channel. It's taken a while but I'm getting a better understanding of how the shaders work, how the code is written, and I'm starting to have some success. I thought I'd share what I've done so far here. Hopefully it will be helpful to anyone a couple of steps behind me and hopefully those who are a couple (or a couple of thousand) steps in front of me will be able to offer some more thoughts and advice.

Before I get into the specific code around here's a few basic things I learnt that I found really useful, some of which Mark covered in the guru session, and the rest mostly from this tutorial:

Swizzling

Where you see a vec3 or vec4 in the code this means a vector containing 3 or 4 values respectively. So, an RGBA value could be represented as:

vec4 colorName = (1.0, 0.7, 0.2, 1.0);

with the red value 1.0, the green 0.7, the blue 0.2 and the alpha 1.0

The useful trick in GLSL code is that once we've got that vector called 'colorName' we can recall different parts of it in different ways. There's the standard array-style way, that I think is familiar in all C++, where, if we just wanted the red bit of colorName we could write colorName[0] to get the first value from the vector. If we only wanted the alpha value, or forth part of the vector, we could write colorName[3] which would return 1.0. But, GLSL has a feature called 'Swizzling' that gives us another method to do this and now I've understood this a lot of the code becomes clearer. Swizzling gives a different syntax to the array-style way of writing the code with some handy naming conventions. You could write colorName.x for example to return the first value or colorName.xy to return the first two values. You can use this technique to return the values in any order so colourName.yx is also valid, returning the second value and then the first. And, as GLSL is designed for graphics there's a handy RGBA syntax as well as follows:

Array-style: [0] [1] [2] [3]

Co-ord style: .x .y .z .w

Colour style .r .g .b .a

Texture style .s .t .p .q

So, if we just wanted the alpha channel we could write any of colorName[3] colorName.w colorName.a or colorName.q or perhaps if we wanted to swap the values around we could make a new vec4 by writing vec4 colorNew = (colorName.brga); swapping the positions of the RGB values. You could also write colorName.xxxy to return the first value 3 times followed by the second value.

gl_FragColor

Next, as I understand it, at the end of the code you'll find a line with gl_FragColor in it. This is what 'draws' the final texture or output of the code with 4 values representing Red, Green, Blue and Alpha. So, by looking at the code it's often quite clear to see if the Alpha channel was being preserved by looking at this last line. If it looks something like this then you can see the alpha channel is being universally set to 1.0 as the last value in the gl_FragColor vector is 1.0:

gl_FragColor = vec4(mix(textureColor.rgb, colorBlend, luminance),1.0);

Putting Alpha back in:

The line above was taken from the TT Sliding Gradient Shader. We can also see that it's returning three values from a vector called textureColor using the .rgb swizzle. Looking further back up the code we can see that when textureColor was first declared it was a vec4 from the line:

vec4 textureColor = texture2D(tex0, gl_TexCoord[0].xy);

texture2D means a standard 2D texture and tex0 means the first texture being input to the shader, in Isadora this would be the first gpu video input. So we can see that textureColor includes the alpha channel. So, in this instance of the TT Sliding Gradient Shader, all we need to do to put the alpha back in, unmodified, is change the last line to:

gl_FragColor = vec4(mix(textureColor.rgb, colorBlend, luminance),textureColor.a);

In a lot of cases I found that it was often this simple, look for the code that first captures the texture2D and return it to the gl_FragColor line at the end. Using that method you can quickly add alpha back in to a lot of shaders, of the TT ones that includes Luma Gradient, Sliding Gradient and Sorbel Edge Detection. Those three shaders all use the same naming convention of 'textureColor' so you can use textureColor.a in each one.

Some of the TT shaders are a bit different. TT Psycho Colors is almost the same but calls the texture2D c0 rather than textureColor. So in this case we can put the alpha back in by replacing that final alpha value (currently 1.0 in the final line) with c0.a

Finally, the TT Color Bands shader is a bit different again. There are multiple fixes but this is the one I settled on. When c0 is declared it is a vec3 and only contains the .rgb values from the tex0 input, currently the code is ignoring the alpha channel altogether. The line looks like this:

vec3 c0 = texture2D(tex0, gl_TexCoord[0].xy).rgb;

I changed that and added a line as follows:

vec4 textureColor = texture2D(tex0, gl_TexCoord[0].xy);
vec3 c0 = (textureColor.rgb);

So now we have a vector called textureColor containing the complete input including the alpha channel. c0 contains the rgb part of that as before, so we don't need to change any of the rest of the code as in effect c0 is the same as before we changed anything. Now in the final gl_FragColor line we can replace the alpha value with textureColor.a as below, and we've got our alpha back.

gl_FragColor = vec4((c0 * luminance) * (1.0 - mono_mod) + vec3(luminance, luminance, luminance) * mono_mod, textureColor.a);

This approach seems to work for most of the shaders I've looked at, some are more complicated, the CMYK halftones for example where the trick was to include the alpha in the earlier calculations moving between CMYK and RGBA values. I guess if there isn't an obvious way to put it back in at the end, you can always look for where it gets lost in the first place and put it back in there.

[mark]

@kathmandale

It is so great that you put so much effort into writing this out. I'm sure that others will benefit from reading this. If you want to share the updated code and have us integrate it into our plugins, I would be happy to receive it and credit you in the source code.

I've taken the liberty of modifying the title so that more users might know be drawn in to read what you've written.

Best Wishes,
Mark

[gapworks]

@kathmandale Excellent Explanation ! Thank a lot

[Kathmandale]

@mark

Thanks Mark. I've attached some text files with the amended code here.

Looking forward to seeing what other people are doing with shaders.

All the best,

Andrew

TT Shaders with Alpha.zip