Mark's Blog

[mark]

Dear All,

The past week has been devoted to embedded alpha channel issues, based on two separate threads that appeared simultaneously. This one started by @Woland and this one started by @dbengali.

If you don't know alpha channels, here's a super brief introduction. Each pixel in an image has four components, RGB (the color components) and A (the alpha component.) The alpha component determines how transparent the pixel is. When it is full (255 when the values range from 0 to 255) then the pixel is opaque; nothing on a lower layer shows through. But as that alpha number gets lower, the pixel becomes more and more transparent.

Most codecs (Apple Pro Res 422, HAP, H264, MP4, etc.) don't support an alpha channel, and will automatically set that value to 255. If you're using those codecs, you will never have encountered this problem. So the alpha channel only becomes important if you're using Apple Pro Res 4444, HAP Alpha or HAP Q, or, for the old timers, the Animation codec.

The confusion comes when you have to choose between straight alpha, or premultiplied alpha. For a lot of folks – even advanced users – the difference between the two is an area of total voodoo. If you don't know what I'm talking about, or if you know but don't understand the difference, this excellent post has an introduction and a deeper explanation and several examples of how alpha handling works in several popular software programs.

The shortest way I can explain it is: straight alpha is better because it doesn't manipulate the color information. But anytime you combine/mix two images, you're going to need to premultiply them first, which means that the output must also now be premultiplied. (For example, the Video Mixer actor must make this conversion.) Premultiplication is a lossy conversion, because once the resulting texture is stored on the GPU, attempts to "unpremultiply" the image to get the original "straight" version back will always be inaccurate.

The key thing I needed to fix, as pointed out by @dbengali, was that Isadora wasn't handling straight alpha properly. That's because I assumed that most people would be outputting premultiplied alpha, given that a lot of programs like Final Cut Pro X and DaVinci Resolve don't even offer straight alpha output as an option, and so the Movie Player always assumed premultiplied. I've addressed that by adding a new input to the Movie Player that allows you to specify what kind of alpha the movie uses, which is pretty much the strategy of all softwares.

To let you see why any of this matters, take a look at a snapshot of a video @dbengali gave me as a test. Here's the video composited over a grey background with the current release:

Note the white "halos" around the figures where the edges are blurred. That's wrong. But When I set the Movie Player's new 'alpha mode' input to 'straight' I see this

which is how it should look. That's because Isadora know knows it must premultiply the image before it composites it with the grey background.

Here's another test, with a test video which is rendered as premultiplied. The result in the current version is correct.

Here's the same video rendered with the new 'alpha mode' input set to 'straight'

which is wrong.

Here's the important takeaway, even if you're not an alpha expert:

• If you see light 'halos' around blurred edges of your video, then the video is straight alpha but is being rendered as if it was premultiplied.
• If you see dark 'halos' around blurred edges of your video, then the video is premultiplied but is being rendered as if it was straight alpha.

Since v2.0, Isadora has kept track of whether a video stream was premultiplied or not, but this information wasn't visible to you. I've now exposed this in the Video Monitor pane that appears when you hover over a link that is carrying video – note the alpha information at the bottom of the

Then I spent the rest of the time fixing some actors that didn't handle one of the two forms of alpha handling properly, including the Text Draw, Matte, Matte++, and Chroma Key actors. I believe all of this is handled now, and will hand off a build to the team so they can test what I've done and ensure I haven't opened some terrible can of worms by adding the 'alpha mode' input on the Movie Player.

Next up is HAP playback issues on Windows. We've got some bugs in their that @Fred discovered, and @DusX has given me repros for these so I can easily replicate the problem. Given that HAP performs amazingly well on a PC with a good graphics card (even at very high resolutions), we're going to put a lot of energy into this and ensure it works beautifully. That's going to take up all of next week for sure.

That's my entry for this week. More next week. ;-)

-- Mark

[crystalhorizon]

Thank you for this! Very interesting read.

[Skulpture]

Love the idea of this blog Mark. Better than emails and easily searchable, etc.

Thanks!

[mark]

A HAP(py) Week + Cue Numbers

Hello All,

It's been a dense couple of weeks. Xenia Leydel – producer of the Werkstatt and director of the TroikaTronix HQ – was on holiday for over two weeks. (Many thanks to @crystalhorizon who took on most of the order processing and inquires during that time.) In addition, I was back in my hometown of Omaha, Nebraska for a week to help out my mother and siblings after my father had an operation on his knee. (He's doing well, thank you.) Still, while jetting around the globe and managing non-programming tasks, I managed to get a good amount of work done. (I actually love 8 hour plane rides, because they are one most productive times. No interruptions!)

HAP Fixes/Improvements

With that as the background, the majority of my energy since my last post has been spent on HAP. First, there were some bugs with the current HAP implementation in Windows – notably, if you attempted to play more than 75 clips (even if they are one after the other) Isadora would crash. Additionally, high resolution movies sometimes suffered from flickering artifact. So, about a week of my time was solving all of this, and it now seems fixed. This effort is really important. I've gotten some incredibly positive reports from @Fred who did a massive show, with high resolution clips, and performance was through the roof on his souped up PC. In the end, Isadora performed perfectly for 20+ shows. It's clear that, for the absolute best performance, HAP is the way to go, especially when running under Windows.

The next few days were spent on another important HAP development: I now have HAP playback working natively in AVFoundation. Previously Isadora would only play HAP movies using QuickTime. The selection of the playback engine of this happens "behind your back," as long as the 'optimize' input is set to 'performance. Isadora will generally use AVFoundation when the 'optimize' mode is 'performance. But, if you try to play a clip that AVFoundation doesn't support (the Animation, for instance) it falls back to QuickTime. You can tell which "engine" is being used by looking at the 'pb engine' output of the Movie Player. AV = AVFoundation, and QT = QuickTime. 

In the past, Isadora fell back to QuickTime for HAP movies. But no longer. Now they play in AVFoundation and it works great.

Enhancements for Movies with Alpha Channels

I also put quite a bit of effort finishing up issues brought up by @dbengali and in this post and @Woland in this post. You can read about the alpha issues in my previous entry. I 've added a new 'alpha mode' input on the Movie Player that allows you to specify whether the movie is 'straight alpha' (unpremultiplied) or premultiplied so that Isadora can handle either one. The team is now going through all the actors to ensure that they work as expected when the alpha is in either format.

A Long Requested Feature: Cue Numbers

Finally, the last couple of days were spent on a long requested feature: Cue Numbers in Isadora. Here's a little preview:

We have some questions about how this feature should work, given that light boards (the usual model for the cue number feature requests) don't have cut/copy/paste/drag. How do you handle the cue numbers when you start doing stuff like this? (NOTE: please don't comment on that here... if necessary, we'll start a thread on the forum for your feedback.)

That's it for now. We're still checking out Isadora's performance with High Sierra. I've got one thing to investigate there, but generally it seems OK.

Best Wishes,
Mark

[mark]

Hey there,

I am at the amazing Schmiede for the coming week, enjoying being surrounded by nearly 300 open hearted, generous media artists. ;-)

That said, I'm still doing my thing. The latest addition is a new Background actor. This will allow all of you to create colored area that is below all the actors, helping you to visually organize your patches. In the past, users have often resorted to using the Comment actor to accomplish something similar. But this actor works much better than the Comment actor because 1) it will never be drawn above other actors in the scene, even if it is selected, and 2) if you click on it when making a segmented patch cord, it will not cancel the operation. You can change the color of the Background actor by double clicking the body of the actor.

@Woland offered a very thoughtful list of features that should with the new Cue Number feature I discussed in my previous post. While we probably won't implement everything he mentioned (to avoid the dreaded "feature creep") but we will take everything he said – and anything you all have to add – into consideration as we complete this feature. Please offer your insights and ideas in the thread that @Woland started.

Best Wishes,
Mark

[mark]

Cue Numbers + a handy new Control for the Control Panel

So last week I completed remaining tasks with Cue Number feature. That included adding a system to allow you to either let Isadora number the cues automatically, or to force you to number them manually whenever you inserted a new scene or pasted a group of scenes. This was all a bit tricky because the whole concept of scenes in Isadora is really different than a light board because you can't paste multiple scenes or drag scenes around. The team is now reviewing those additions. We also added a MIDI Show Control feature: you can send MSC commands to get Isadora to jump to a specific cue, or have Isadora send MSC "Go" commands whenever a new scene becomes active.

Then over the past two days, I finished this new Control for the Control Panel:

It automatically updates to show list of scenes, shows the cue numbers (if they are also visible in the scene list), and optionally allows you to go trigger a scene by clicking on it.

Options include:

• Click Activate Scene: If this is enabled, then clicking on a scene performs a jump to that scene from the current scene
• Color for the current scene, next scene, and inactive scenes. (Defaults to the colors used in the Scene List)
• Scene Name Color: the color of the text for the Scene Name
• Vertical Spacing: the gap between scenes. (Height of the actualy items in the list is based on the font size)
• Current Scene Line: if this is non zero, then the current scene will always be drawn on the specified line of the control. E.g., if you say 3, then there will be then the two scenes previous to the current scene will be visible above the current scene. (I don't love this name, but space is limited... ideas?) If this is zero, then there the control will scroll as necessary (and as possible) to show the current scene.
• Current Scene Indicator: If checked, this will draw a triangle arrow to indicate which scene is current. This is just to give the operator some extra feedback about which scene is current.
• Indicator Color: the color of the above indicator.

I'm thinking about adding options to allow you to add a "Go" and a "Prev" button to this control at the bottom, which you could click to go to the next scene or to go back one scene.

That's what's happened this week.... more to come.

Over and Out,
Mark

[mark]

Dear All,

Well, it's been quite the hectic week here, culminating with a new build for the team at the end of a 16 hour day of mashing on code. The work of the previous week wasn't so much taken by adding new features, but instead 1) ensuring that the new features mentioned here previously are crash free, and 2) improving and modifying behaviors of the cue number features based on discussions we're having among the team, and 3) fixing a few old, outstanding bugs. That said, there are a few new actors as listed below.

In terms of the older bugs, one showed up if you dragged some scenes, and then chose "Undo" and then "Redo," the scenes would end up in the wrong place. (Doh!) No one, users or team or me, ever noticed this before; it has to be years old. But there was some other stuff with dragging scenes, and just managing this took a couple of days.

Another longstanding issue were some hard limitations in OSC: only 200 available packets could be in memory at one time, and there was a 1000 byte limit to the amount of data each packet could hold. Our dear friend @DusX, who likes to pull copious amounts of data from the internet, managed to break these limits. But mainly these kind of hard limitations are never nice for people who like to do weird things (e.g,. most of you) so we've added a feature to allow you to customize this in the Isadora Preferences:

Now you can set the number of packets that can be in memory at the same moment, the default size of an input buffer, and the absolute biggest buffer that will ever be allocated. The latter property reflects a new feature: previously, the amount of data that each packet could hold was fixed – the "Default Data Size" in the above window. But now, you can allow larger packets to be accepted if they arrive, with the "Maximum Data Size" being the biggest that would ever be allowed. (It's good not to have an infinite limit on this, because a packet with bad formatting might look like it was 4GB long or whatever... that would lead to a crash simply because the memory chunk would be too big.) I reckon that 256K is a pretty healthy maximum size for an OSC packet.

NEW/UPDATED ACTORS

There is a GPU version of the Dots actor that replaces the old one. (The old CPU version is now called "Classic Dots.") New features include the ability to crossfade between a specified dot color and the source color, the ability to specify the maximum dot size, and the fact that the CPU dot version is now anti-aliased. (Thus the removal of the troublesome 'scale' input which really was a workaround for the fact that the dots weren't anti-aliased when drawn by our dear, but now ancient, QuickTime.)

Likewise, there is a GPU version of the Lines actor that replaces the old one. (Old, CPU version = "Classic Lines".) The new features include a new 'end cap' input that allows flat, square, or round endcaps, a background color input, and a 'classic' input that will cause this actor to replicate the behavior of the old CPU based Lines actor that used QuickDraw for drawing. (We might remove this input since the old, classic CPU version still exists.)

Finally, I also created a brand new actor called OSC Multi Listener. This allows you to capture a contiguous range of OSC inputs. You tell the actor the first channel and the number of channels to copy, and the matching outputs will appear, labeled with the corresponding channel numbers to make it clear which channel is which.

In relation to this, I made a kind of nifty OSC Recorder user actor to which @DusX is making some improvements to ensure timing accuracy. We'll get that out to you as soon as we can.

That's it for now folks. Hope you all have a nice weekend approaching. 

Best Wishes,
Mark

[mark]

Dear All,

Well, it's been over three weeks since my last post. I've been too busy to write. (Not just with programming, but dealing with those nasty f'ing forum spammers that are annoying us all, and trying to get Isadora running fine for a large university installation where they're running Windows 7 on their lab computers. The fun never ends ;-) )

In any case, we're getting very close to the final feature set for v2.6 and we'll be opening it up for selected beta testers first thing next week.

One long-requested feature is finished: the new "Capture Stage to Picture" actor.

It complements the existing "Capture Stage to Disk" (renamed now to "Capture Stage to Movie") and "Capture Camera to Disk." I redid the icons for all three of these actors as you ca see above. You can capture the pictures as JPG, PNG, BMP, TIF, and (single frame) GIF files. I'm sure many of you will put this to instant use.

Also, I spent some time on this:

(Designers note: this is the Comic Sans of styled Comment actor examples . It was designed to show off the new feature, so please excuse the kitsch. ;-) )

You might be of the opinion that styled text in a comment actor might not be worth spending time on. But as we move forward to create Isadora patches that help users learn, we want to be able to make really clear tutorial files inside of Isadora itself – something that the team will be working on in the coming weeks and months.

Moreover, because of this effort, I see a clear path to offer a "Styled Text Draw" actor, because I learned how I could render RTF files in these text boxes on both Mac and Windows. So this effort has broader implications than just showing different fonts. It would be a great addition to be able to not only render text, but to style it as well.

To go with this, is our own custom "Font Panel." Mostly I had to do this because Windows doesn't have a standard tool for this like Mac OS. (They have this ribbon thing, but I really hate the whole ribbon notion; it's pretty much the definition of clutter.)

You select the fonts just like you do with the popup Actor Toolbox. If you want to change the color, you click on the swatch at the bottom of the font list. Pretty standard stuff but very Isadora in it's look and feel.

Finally, we have a nice addition specifically for Windows users: integrated Spout support.

In fact, these actors are exactly the same as the Syphon Receive and Syphon Stage Output actors which we've had for a while on Mac OS. If you save a patch with these Syphon actors on Mac OS, they'll open up on Windows as Spout actors instead and vice-versa.

So that's where we're at at the moment. We're going to stop creating and start testing next week, and get this out to you as soon as we can manage.

Best Wishes,
Mark

[mark]

Dear All,

Just quickly: as we head for the release of v2.6, there was one feature the team had requested that I implemented on my long plane ride to Mexico City yesterday.

There is a new "Show Tagged Actors" button is at the bottom of the scene editor:

What does it do? Well, when you click it, any any actor whose name starts with a hashtag will appear in a popup menu. Choose that item and the Scene Editor will scroll directly to that actor, placing it in the top-left corner.

(In case you didn't know it, you can rename any actor by choosing Actors > Rename Actor.) The obvious use here is to tag your Background actors with a hashtag, but any actor you tag in this way will show up in the popup list.

Another quick fix: you can not only change the color of the new Background actor, but also now the Comment actor as well.

We also made it so that the only way to select Background actor is by clicking on it's title. That way, dragging a selection rectangle won't select the Background as well. Instead, if you want to move the Background and the actors on top of it, you'll need to select the actors first and then shift-click the title of the Background actor. I want to avoid a "Lock/Unlock" feature as many users won't notice it in the menu. Given that this behavior is explained in the Help view, hopefully this is a good solution.

Best Wishes,
Mark   

[mark]

Hey all,

I hope that the warmth of summer has headed your way as it has here in Berlin. It's been a while since I've posted because I've been deep in the land of coding for the last three months as I've reworked the entire (ancient) framework upon Isadora was built. While it may not look like something special, the image below is the first patch created on a fully 64-bit Isadora.

Now, there's still a lot to do before we start extending the capabilities of Isadora 2.6 in this new build. (For instance, now I have to go an rebuild all the plugins, though I don't anticipate this taking very long.) And then of course, there will be lots and lots of testing to do.

Still, it was pretty nice after working so hard to see it boot up and basically work. ;-)

There is a lot more to come before the late September release of Isadora 3. But being able to post the image above marks a major milestone in our efforts to make you love Ms. Izzy even more.

Warmest Wishes from the Coding Desk,
Mark

[RIL]

Thanks Mark for all the incredible work !!

Best,

Maxi -Isadora Latin Network

[bonemap]

Thanks for the update.

@mark said:

September release of Isadora 3

 Is there an early indication of the upgrade cost per license? Will it be similar to the v1 > v2 or perhaps that + %10? Or is it still undecided?

Best Wishes 

Bonemap 

[mark]

@bonemap said:

Is there an early indication of the upgrade cost per license? Will it be similar to the v1 > v2 or perhaps that + %10? Or is it still undecided?

 I am working on the pricing structure for an internal review by the team. We should have news for you on this in the next couple of weeks.

Best Wishes,
Mark

[mark]

Dear Community,

So, I've got all the plugins running native under 64 bit and they seem to be working pretty well. I also got FreeFrame working under 64 bit as well.

Here's a question for anyone that cares to comment: what about numeric precision? 

In the 32 bit version of Isadora, both floating point numbers and integer numbers are 32 bit as well. That gives you the following range:

32-bit Integers: -2147483648 to +214748364
32-bit Floating Point: ±1.18×10⁻³⁸ to ±3.4×10³⁸ (that's 10 to the 38th power)

In the new version, we could have it either way. I could keep the ranges the same as they are, or I could move to 64 bit versions

64-bit Integers: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
64-bit Floating Point: ±2.23×10⁻³⁰⁸ to ±1.80×10³⁰⁸

(Now the ranges for the floating point values might not mean much to you, but suffice to say that many numbers cannot be represented exactly using floating point numbers. Even with the the 64 bit floating point number, the true representation of 0.1 internally is 0.1000000000000000055511151231257827021181583404541015625. It's just that Isadora only displays four numbers after the decimal point. Suffice to say that 64 bit floats are more accurate than 32 bit ones in terms of this error.)

There is only one use case I have encountered where the 32 bit version held me up: I was trying to import timestamps using the Data Array actor that were expressed as microseconds since Jan 1st, 1900. (I think this is some kind of standard.) The 32 bit float cannot represent this number, and so I had to write a program use a reference date that was Jan 1st, 1980 instead.

For most of what we do, the 32 bit versions have been adequate. The upside of 64 bit is more accuracy. The downside is that it will significantly increase the size of your .izz files if you have a complicated patch, because ever single number (i.e., an input or output  to an actor) will now take 8 bytes instead of 4.

Does anyone have thoughts on this?

Best Wishes,
Mark

[bonemap]

@mark said:

thoughts on this

 If I understand this correctly, the question is about potential increase in file size vs. numerical accuracy? What about file efficiency, cycles and FPS - are there any other anticipated impacts that differentiates the two options?

If the only impact is in the difference in project files size, I have reviewed my patch collection and the average and maximum sizes of my Isadora project files? I would rarely exceed 3-4mb file size and would imagine an increase of up to 10mb file size would be acceptable in favour of more numerical accuracy. But do we have enough information in @Mark's post to make an informed comment about this? I don't mean to doubt a master - just making an enquiring query.

Best Wishes,

Bonemap

[LPmode]

I would say make it for the future so make it 64bit. We want to be able to explore any new possibilities it has to offer even though we may not know what they are. 

The file size should not matter but I do have one issue with bigger file sizes, - currently Isadora freezes while saving. and the bigger the patch, the longer the freeze. I submitted it as a bug maybe over a year ago. I had 10mb+ patcehs that would hang Isadora for over 10 seconds while it was saving. 

[mark]

@bonemap said:

What about file efficiency, cycles and FPS

Good point... and it is something that I have already looked into. My research so tells me that there won't be measurable difference on performance, because – internally on all modern Intel processors – the size of the floating point value doesn't affect performance. Here's quote from one Stackoverflow article on the topic:

The classic x86 architecture uses floating-point unit (FPU) to perform floating-point calculations. The FPU performs all calculations in its internal registers, which have 80-bit precision each. Every time you attempt to work with float or double, the variable is first loaded from memory into the internal register of the FPU. This means that there is absolutely no difference in the speed of the actual calculations, since in any case the calculations are carried out with full 80-bit precision. The only thing that might be different is the speed of loading the value from memory and storing the result back to memory. Naturally, on a 32-bit platform it might take longer to load/store a double as compared to float. On a 64-bit platform there shouldn't be any difference.

In the end, the video processing (even on the GPU) and the disk throughput are the two factors that affect Isadora's performance.

Best Wishes,
Mark