In hopes of keeping you abreast of what's going on, I'm going to start this thread, which is for me to informally discuss what I'm working on. My hope is to post here every 7 to 10 days. You may comment if you like, but please only to ask questions about what I'm talking about here. Please don't put feature requests, bug reports and that sort of thing in this thread – post them instead to the appropriate topics in this forum.
If you don't know alpha channels, here's a super brief introduction. Each pixel in an image has four components, RGB (the color components) and A (the alpha component.) The alpha component determines how transparent the pixel is. When it is full (255 when the values range from 0 to 255) then the pixel is opaque; nothing on a lower layer shows through. But as that alpha number gets lower, the pixel becomes more and more transparent.
Most codecs (Apple Pro Res 422, HAP, H264, MP4, etc.) don't support an alpha channel, and will automatically set that value to 255. If you're using those codecs, you will never have encountered this problem. So the alpha channel only becomes important if you're using Apple Pro Res 4444, HAP Alpha or HAP Q, or, for the old timers, the Animation codec.
The confusion comes when you have to choose between straight alpha, or premultiplied alpha. For a lot of folks – even advanced users – the difference between the two is an area of total voodoo. If you don't know what I'm talking about, or if you know but don't understand the difference, this excellent post has an introduction and a deeper explanation and several examples of how alpha handling works in several popular software programs.
The shortest way I can explain it is: straight alpha is better because it doesn't manipulate the color information. But anytime you combine/mix two images, you're going to need to premultiply them first, which means that the output must also now be premultiplied. (For example, the Video Mixer actor must make this conversion.) Premultiplication is a lossy conversion, because once the resulting texture is stored on the GPU, attempts to "unpremultiply" the image to get the original "straight" version back will always be inaccurate.
The key thing I needed to fix, as pointed out by @dbengali, was that Isadora wasn't handling straight alpha properly. That's because I assumed that most people would be outputting premultiplied alpha, given that a lot of programs like Final Cut Pro X and DaVinci Resolve don't even offer straight alpha output as an option, and so the Movie Player always assumed premultiplied. I've addressed that by adding a new input to the Movie Player that allows you to specify what kind of alpha the movie uses, which is pretty much the strategy of all softwares.
To let you see why any of this matters, take a look at a snapshot of a video @dbengali gave me as a test. Here's the video composited over a grey background with the current release:
Note the white "halos" around the figures where the edges are blurred. That's wrong. But When I set the Movie Player's new 'alpha mode' input to 'straight' I see this
which is how it should look. That's because Isadora know knows it must premultiply the image before it composites it with the grey background.
Here's another test, with a test video which is rendered as premultiplied. The result in the current version is correct.
Here's the same video rendered with the new 'alpha mode' input set to 'straight'
which is wrong.
Here's the important takeaway, even if you're not an alpha expert:
- If you see light 'halos' around blurred edges of your video, then the video is straight alpha but is being rendered as if it was premultiplied.
- If you see dark 'halos' around blurred edges of your video, then the video is premultiplied but is being rendered as if it was straight alpha.
Since v2.0, Isadora has kept track of whether a video stream was premultiplied or not, but this information wasn't visible to you. I've now exposed this in the Video Monitor pane that appears when you hover over a link that is carrying video – note the alpha information at the bottom of the
Then I spent the rest of the time fixing some actors that didn't handle one of the two forms of alpha handling properly, including the Text Draw, Matte, Matte++, and Chroma Key actors. I believe all of this is handled now, and will hand off a build to the team so they can test what I've done and ensure I haven't opened some terrible can of worms by adding the 'alpha mode' input on the Movie Player.
Next up is HAP playback issues on Windows. We've got some bugs in their that @Fred discovered, and @DusX has given me repros for these so I can easily replicate the problem. Given that HAP performs amazingly well on a PC with a good graphics card (even at very high resolutions), we're going to put a lot of energy into this and ensure it works beautifully. That's going to take up all of next week for sure.
That's my entry for this week. More next week.
Thank you for this! Very interesting read.
Love the idea of this blog Mark. Better than emails and easily searchable, etc.
A HAP(py) Week + Cue Numbers
It's been a dense couple of weeks. Xenia Leydel – producer of the Werkstatt and director of the TroikaTronix HQ – was on holiday for over two weeks. (Many thanks to @crystalhorizon who took on most of the order processing and inquires during that time.) In addition, I was back in my hometown of Omaha, Nebraska for a week to help out my mother and siblings after my father had an operation on his knee. (He's doing well, thank you.) Still, while jetting around the globe and managing non-programming tasks, I managed to get a good amount of work done. (I actually love 8 hour plane rides, because they are one most productive times. No interruptions!)
With that as the background, the majority of my energy since my last post has been spent on HAP. First, there were some bugs with the current HAP implementation in Windows – notably, if you attempted to play more than 75 clips (even if they are one after the other) Isadora would crash. Additionally, high resolution movies sometimes suffered from flickering artifact. So, about a week of my time was solving all of this, and it now seems fixed. This effort is really important. I've gotten some incredibly positive reports from @Fred who did a massive show, with high resolution clips, and performance was through the roof on his souped up PC. In the end, Isadora performed perfectly for 20+ shows. It's clear that, for the absolute best performance, HAP is the way to go, especially when running under Windows.
The next few days were spent on another important HAP development: I now have HAP playback working natively in AVFoundation. Previously Isadora would only play HAP movies using QuickTime. The selection of the playback engine of this happens "behind your back," as long as the 'optimize' input is set to 'performance. Isadora will generally use AVFoundation when the 'optimize' mode is 'performance. But, if you try to play a clip that AVFoundation doesn't support (the Animation, for instance) it falls back to QuickTime. You can tell which "engine" is being used by looking at the 'pb engine' output of the Movie Player. AV = AVFoundation, and QT = QuickTime.
In the past, Isadora fell back to QuickTime for HAP movies. But no longer. Now they play in AVFoundation and it works great.
Enhancements for Movies with Alpha Channels
I also put quite a bit of effort finishing up issues brought up by @dbengali and in this post and @Woland in this post. You can read about the alpha issues in my previous entry. I 've added a new 'alpha mode' input on the Movie Player that allows you to specify whether the movie is 'straight alpha' (unpremultiplied) or premultiplied so that Isadora can handle either one. The team is now going through all the actors to ensure that they work as expected when the alpha is in either format.
A Long Requested Feature: Cue Numbers
Finally, the last couple of days were spent on a long requested feature: Cue Numbers in Isadora. Here's a little preview:
We have some questions about how this feature should work, given that light boards (the usual model for the cue number feature requests) don't have cut/copy/paste/drag. How do you handle the cue numbers when you start doing stuff like this? (NOTE: please don't comment on that here... if necessary, we'll start a thread on the forum for your feedback.)
That's it for now. We're still checking out Isadora's performance with High Sierra. I've got one thing to investigate there, but generally it seems OK.
I am at the amazing Schmiede for the coming week, enjoying being surrounded by nearly 300 open hearted, generous media artists.
That said, I'm still doing my thing. The latest addition is a new Background actor. This will allow all of you to create colored area that is below all the actors, helping you to visually organize your patches. In the past, users have often resorted to using the Comment actor to accomplish something similar. But this actor works much better than the Comment actor because 1) it will never be drawn above other actors in the scene, even if it is selected, and 2) if you click on it when making a segmented patch cord, it will not cancel the operation. You can change the color of the Background actor by double clicking the body of the actor.
@Woland offered a very thoughtful list of features that should with the new Cue Number feature I discussed in my previous post. While we probably won't implement everything he mentioned (to avoid the dreaded "feature creep") but we will take everything he said – and anything you all have to add – into consideration as we complete this feature. Please offer your insights and ideas in the thread that @Woland started.