In hopes of keeping you abreast of what's going on, I'm going to start this thread, which is for me to informally discuss what I'm working on. My hope is to post here every 7 to 10 days. You may comment if you like, but please only to ask questions about what I'm talking about here. Please don't put feature requests, bug reports and that sort of thing in this thread – post them instead to the appropriate topics in this forum.
If you don't know alpha channels, here's a super brief introduction. Each pixel in an image has four components, RGB (the color components) and A (the alpha component.) The alpha component determines how transparent the pixel is. When it is full (255 when the values range from 0 to 255) then the pixel is opaque; nothing on a lower layer shows through. But as that alpha number gets lower, the pixel becomes more and more transparent.
Most codecs (Apple Pro Res 422, HAP, H264, MP4, etc.) don't support an alpha channel, and will automatically set that value to 255. If you're using those codecs, you will never have encountered this problem. So the alpha channel only becomes important if you're using Apple Pro Res 4444, HAP Alpha or HAP Q, or, for the old timers, the Animation codec.
The confusion comes when you have to choose between straight alpha, or premultiplied alpha. For a lot of folks – even advanced users – the difference between the two is an area of total voodoo. If you don't know what I'm talking about, or if you know but don't understand the difference, this excellent post has an introduction and a deeper explanation and several examples of how alpha handling works in several popular software programs.
The shortest way I can explain it is: straight alpha is better because it doesn't manipulate the color information. But anytime you combine/mix two images, you're going to need to premultiply them first, which means that the output must also now be premultiplied. (For example, the Video Mixer actor must make this conversion.) Premultiplication is a lossy conversion, because once the resulting texture is stored on the GPU, attempts to "unpremultiply" the image to get the original "straight" version back will always be inaccurate.
The key thing I needed to fix, as pointed out by @dbengali, was that Isadora wasn't handling straight alpha properly. That's because I assumed that most people would be outputting premultiplied alpha, given that a lot of programs like Final Cut Pro X and DaVinci Resolve don't even offer straight alpha output as an option, and so the Movie Player always assumed premultiplied. I've addressed that by adding a new input to the Movie Player that allows you to specify what kind of alpha the movie uses, which is pretty much the strategy of all softwares.
To let you see why any of this matters, take a look at a snapshot of a video @dbengali gave me as a test. Here's the video composited over a grey background with the current release:
Note the white "halos" around the figures where the edges are blurred. That's wrong. But When I set the Movie Player's new 'alpha mode' input to 'straight' I see this
which is how it should look. That's because Isadora know knows it must premultiply the image before it composites it with the grey background.
Here's another test, with a test video which is rendered as premultiplied. The result in the current version is correct.
Here's the same video rendered with the new 'alpha mode' input set to 'straight'
which is wrong.
Here's the important takeaway, even if you're not an alpha expert:
- If you see light 'halos' around blurred edges of your video, then the video is straight alpha but is being rendered as if it was premultiplied.
- If you see dark 'halos' around blurred edges of your video, then the video is premultiplied but is being rendered as if it was straight alpha.
Since v2.0, Isadora has kept track of whether a video stream was premultiplied or not, but this information wasn't visible to you. I've now exposed this in the Video Monitor pane that appears when you hover over a link that is carrying video – note the alpha information at the bottom of the
Then I spent the rest of the time fixing some actors that didn't handle one of the two forms of alpha handling properly, including the Text Draw, Matte, Matte++, and Chroma Key actors. I believe all of this is handled now, and will hand off a build to the team so they can test what I've done and ensure I haven't opened some terrible can of worms by adding the 'alpha mode' input on the Movie Player.
Next up is HAP playback issues on Windows. We've got some bugs in their that @Fred discovered, and @DusX has given me repros for these so I can easily replicate the problem. Given that HAP performs amazingly well on a PC with a good graphics card (even at very high resolutions), we're going to put a lot of energy into this and ensure it works beautifully. That's going to take up all of next week for sure.
That's my entry for this week. More next week.
Thank you for this! Very interesting read.
Love the idea of this blog Mark. Better than emails and easily searchable, etc.
A HAP(py) Week + Cue Numbers
It's been a dense couple of weeks. Xenia Leydel – producer of the Werkstatt and director of the TroikaTronix HQ – was on holiday for over two weeks. (Many thanks to @crystalhorizon who took on most of the order processing and inquires during that time.) In addition, I was back in my hometown of Omaha, Nebraska for a week to help out my mother and siblings after my father had an operation on his knee. (He's doing well, thank you.) Still, while jetting around the globe and managing non-programming tasks, I managed to get a good amount of work done. (I actually love 8 hour plane rides, because they are one most productive times. No interruptions!)
With that as the background, the majority of my energy since my last post has been spent on HAP. First, there were some bugs with the current HAP implementation in Windows – notably, if you attempted to play more than 75 clips (even if they are one after the other) Isadora would crash. Additionally, high resolution movies sometimes suffered from flickering artifact. So, about a week of my time was solving all of this, and it now seems fixed. This effort is really important. I've gotten some incredibly positive reports from @Fred who did a massive show, with high resolution clips, and performance was through the roof on his souped up PC. In the end, Isadora performed perfectly for 20+ shows. It's clear that, for the absolute best performance, HAP is the way to go, especially when running under Windows.
The next few days were spent on another important HAP development: I now have HAP playback working natively in AVFoundation. Previously Isadora would only play HAP movies using QuickTime. The selection of the playback engine of this happens "behind your back," as long as the 'optimize' input is set to 'performance. Isadora will generally use AVFoundation when the 'optimize' mode is 'performance. But, if you try to play a clip that AVFoundation doesn't support (the Animation, for instance) it falls back to QuickTime. You can tell which "engine" is being used by looking at the 'pb engine' output of the Movie Player. AV = AVFoundation, and QT = QuickTime.
In the past, Isadora fell back to QuickTime for HAP movies. But no longer. Now they play in AVFoundation and it works great.
Enhancements for Movies with Alpha Channels
I also put quite a bit of effort finishing up issues brought up by @dbengali and in this post and @Woland in this post. You can read about the alpha issues in my previous entry. I 've added a new 'alpha mode' input on the Movie Player that allows you to specify whether the movie is 'straight alpha' (unpremultiplied) or premultiplied so that Isadora can handle either one. The team is now going through all the actors to ensure that they work as expected when the alpha is in either format.
A Long Requested Feature: Cue Numbers
Finally, the last couple of days were spent on a long requested feature: Cue Numbers in Isadora. Here's a little preview:
We have some questions about how this feature should work, given that light boards (the usual model for the cue number feature requests) don't have cut/copy/paste/drag. How do you handle the cue numbers when you start doing stuff like this? (NOTE: please don't comment on that here... if necessary, we'll start a thread on the forum for your feedback.)
That's it for now. We're still checking out Isadora's performance with High Sierra. I've got one thing to investigate there, but generally it seems OK.
I am at the amazing Schmiede for the coming week, enjoying being surrounded by nearly 300 open hearted, generous media artists.
That said, I'm still doing my thing. The latest addition is a new Background actor. This will allow all of you to create colored area that is below all the actors, helping you to visually organize your patches. In the past, users have often resorted to using the Comment actor to accomplish something similar. But this actor works much better than the Comment actor because 1) it will never be drawn above other actors in the scene, even if it is selected, and 2) if you click on it when making a segmented patch cord, it will not cancel the operation. You can change the color of the Background actor by double clicking the body of the actor.
@Woland offered a very thoughtful list of features that should with the new Cue Number feature I discussed in my previous post. While we probably won't implement everything he mentioned (to avoid the dreaded "feature creep") but we will take everything he said – and anything you all have to add – into consideration as we complete this feature. Please offer your insights and ideas in the thread that @Woland started.
Cue Numbers + a handy new Control for the Control Panel
So last week I completed remaining tasks with Cue Number feature. That included adding a system to allow you to either let Isadora number the cues automatically, or to force you to number them manually whenever you inserted a new scene or pasted a group of scenes. This was all a bit tricky because the whole concept of scenes in Isadora is really different than a light board because you can't paste multiple scenes or drag scenes around. The team is now reviewing those additions. We also added a MIDI Show Control feature: you can send MSC commands to get Isadora to jump to a specific cue, or have Isadora send MSC "Go" commands whenever a new scene becomes active.
Then over the past two days, I finished this new Control for the Control Panel:
It automatically updates to show list of scenes, shows the cue numbers (if they are also visible in the scene list), and optionally allows you to go trigger a scene by clicking on it.
- Click Activate Scene: If this is enabled, then clicking on a scene performs a jump to that scene from the current scene
- Color for the current scene, next scene, and inactive scenes. (Defaults to the colors used in the Scene List)
- Scene Name Color: the color of the text for the Scene Name
- Vertical Spacing: the gap between scenes. (Height of the actualy items in the list is based on the font size)
- Current Scene Line: if this is non zero, then the current scene will always be drawn on the specified line of the control. E.g., if you say 3, then there will be then the two scenes previous to the current scene will be visible above the current scene. (I don't love this name, but space is limited... ideas?) If this is zero, then there the control will scroll as necessary (and as possible) to show the current scene.
- Current Scene Indicator: If checked, this will draw a triangle arrow to indicate which scene is current. This is just to give the operator some extra feedback about which scene is current.
- Indicator Color: the color of the above indicator.
I'm thinking about adding options to allow you to add a "Go" and a "Prev" button to this control at the bottom, which you could click to go to the next scene or to go back one scene.
That's what's happened this week.... more to come.
Over and Out,
Well, it's been quite the hectic week here, culminating with a new build for the team at the end of a 16 hour day of mashing on code. The work of the previous week wasn't so much taken by adding new features, but instead 1) ensuring that the new features mentioned here previously are crash free, and 2) improving and modifying behaviors of the cue number features based on discussions we're having among the team, and 3) fixing a few old, outstanding bugs. That said, there are a few new actors as listed below.
In terms of the older bugs, one showed up if you dragged some scenes, and then chose "Undo" and then "Redo," the scenes would end up in the wrong place. (Doh!) No one, users or team or me, ever noticed this before; it has to be years old. But there was some other stuff with dragging scenes, and just managing this took a couple of days.
Another longstanding issue were some hard limitations in OSC: only 200 available packets could be in memory at one time, and there was a 1000 byte limit to the amount of data each packet could hold. Our dear friend @DusX, who likes to pull copious amounts of data from the internet, managed to break these limits. But mainly these kind of hard limitations are never nice for people who like to do weird things (e.g,. most of you) so we've added a feature to allow you to customize this in the Isadora Preferences:
Now you can set the number of packets that can be in memory at the same moment, the default size of an input buffer, and the absolute biggest buffer that will ever be allocated. The latter property reflects a new feature: previously, the amount of data that each packet could hold was fixed – the "Default Data Size" in the above window. But now, you can allow larger packets to be accepted if they arrive, with the "Maximum Data Size" being the biggest that would ever be allowed. (It's good not to have an infinite limit on this, because a packet with bad formatting might look like it was 4GB long or whatever... that would lead to a crash simply because the memory chunk would be too big.) I reckon that 256K is a pretty healthy maximum size for an OSC packet.
There is a GPU version of the Dots actor that replaces the old one. (The old CPU version is now called "Classic Dots.") New features include the ability to crossfade between a specified dot color and the source color, the ability to specify the maximum dot size, and the fact that the CPU dot version is now anti-aliased. (Thus the removal of the troublesome 'scale' input which really was a workaround for the fact that the dots weren't anti-aliased when drawn by our dear, but now ancient, QuickTime.)
Likewise, there is a GPU version of the Lines actor that replaces the old one. (Old, CPU version = "Classic Lines".) The new features include a new 'end cap' input that allows flat, square, or round endcaps, a background color input, and a 'classic' input that will cause this actor to replicate the behavior of the old CPU based Lines actor that used QuickDraw for drawing. (We might remove this input since the old, classic CPU version still exists.)
Finally, I also created a brand new actor called OSC Multi Listener. This allows you to capture a contiguous range of OSC inputs. You tell the actor the first channel and the number of channels to copy, and the matching outputs will appear, labeled with the corresponding channel numbers to make it clear which channel is which.
In relation to this, I made a kind of nifty OSC Recorder user actor to which @DusX is making some improvements to ensure timing accuracy. We'll get that out to you as soon as we can.
That's it for now folks. Hope you all have a nice weekend approaching.