Metal for Mac

[Skulpture]

Thanks for the summary Mark.

[Unfenswinger]

The announcement of Metal has put to bed the mystery of why Apple wasn't progressing quickly with its OpenCL drivers.

I'm pretty sure that OpenGL and OpenCL are pretty much dead ducks on OS X and Apple will push forward with Metal. Metal apparently works on all Mac shipped in 2012 onwards. I agree OpenGL won't die immediately but Apple won't be going beyond 4.1 or OpenCL 1.2\. I suppose we can be grateful they didn't kill QC yet! :)

There seems to have been a bit of a seismic shift in graphics APIs and with DX12, Vulcan and now Metal all stripping themselves literally back to the bare metal. Reading what Jack Greasley Mari developer says on the subject I'm quite hopeful. He made the point all thee new APIs are very similar so it won't create a huge duplication of effort to support them all and will be better in the long term for the developer as there's much less platform specific code needed to be written. I'm no developer so no doubt there's devil in the details of these broad generalisations.

The market will decide if Metal is a success, I'm always going to buy the best bang for buck so if software X is going to support my hardware better than software Y I'll buy X. Adobe have shown some amazing speed improvements in quick demos so hopefully software that I own will make benefit of the compute side of Metal for similar performance gains. Personally, I can't wait to be able to produce massive live particle systems rendered in realtime being affected by sound or HID. I think the computational side of Metal will be a big win for us.

[Maximortal]

Gaming is on windows, windows is going to dx12 (i'm not so sure but mantle have an unclear future).

Graphic performaces of a typical mac is far far distant to a typical windows gaming machine, this's a fact, not an opinion. 

What normally had leaded adoption of an API sets until now is gaiming world. IMHO the future will be on DX 12, more if rumors about adoptions of them by google become real. 

It takes quite long time before scenario become more clear (could Vulkan become a open and good alternative and so on), until that moment just big company as Adobe can make this kind of leap (i guess apple's money should had a part on this decision ;-)  )

[Unfenswinger]

DX12, vulkan and Mantle are similar in approach to Metal with much overlap.

I haven't read one negative article about Metal from developers yet and particularly developers working on multiple OSes as it ties up with plans they have to adopt the other APIs on other platforms.

Metal is used throughout El Capitan so it looks like it'll more compatible with older Macs than first thought. More life in the old 2010 MP yet maybe!

OpenGL on Windows benefitted from the arms race between GPU hardware developers who were producing high performance drivers to come top of gaming benchmarks, the Mac never benefitted from that work because it was Windows specific optimisation.

Just heard Unity is going to support Metal for their editor not just player which could mean some exciting developments for realtime 3D performance work.

[mark]

Just a couple of other points

In the end, Isadora does not rely heavily on 3D operations. Generally speaking, you've got a texture, and you render it. Sometimes with a simple quad, and -- for things like the mapper -- with more complex 2D poiygons. Now, I cannot profess to be an expert on these new systems, but in such cases @Unfenswinger do you imagine that there will be some kind of massive performance improvement?

Of course, shader support is important. And this will become more important in Isadora in the future. And perhaps that's where Metal/DX12/etc. will give us some important benefits.

Another implication of these developers is that all FreeFrameGL plugins stop working if only Metal, DX12, etc, are available. They are currently based on OpenGL 2 and so would be incompatible with these systems. That's a pretty major headache for all of the developers.

Best Wishes,

Mark

[Unfenswinger]

I don't know Mark but the new APIs seem to be trying to take out a lot of legacy or computational expensive code out of the driver and allowing the developer to schedule those tasks so even a simple task may have some inefficiency to it that would benefit from the new APIs and developer intervention.

I would certainly love see Isadora make use of the GPU compute side of Metal, imagine if the all the current CPU actors could be executed on the GPU? I would think the performance would be dramatically improved. Most of my framerate issues are down to the CPU running out of steam not the GPU. I think Moore's law is dead for the CPU but appears to be alive and kicking in the GPU arms race.

Good point about FreeframeGL, I'm sure Syphon would also need to be recoded to support Metal too so I expect/hope many developers will support it if the demand is there.

My feeling about live performance is that it really is about the spectacle and any technology that can improve that then I'm very excited by it. I see a movement to fully live interactive 3D projection mapping with no 3D content needed to be rendered before hand. We should be heading for 3D video game level of fidelity of graphics and moving away from wireframe cones and phong shaded spheres.

I'm just a consumer that has over ambitious views of what developers can do but I'm not after a free lunch and I'm very happy to pay and donate for developers work in updating their software as it's unlikely to be straight forward progress.

[mark]

Dear All,

I am always interested in new technologies that offer us creative possibilities. But I also get a bit miffed when marketing-hype leads people to think that the "latest and greatest" is always superior to previous technologies. Determining how much one will benefit (and what it will cost in terms of time and money during development) requires far more careful consideration that what is offered during a product launch demonstration.

So, in an effort to learn more about this myself, I had a long conversation with an expert on the topic of GPU programming last night so that I might better understand how Isadora would benefit from these new technologies.

Here are a few things I learned.

1) DX12, Metal and Vulcan all do essentially the same thing. They bypass what we now know as the OpenGL driver, and give you direct access to the GPU. This means that you can bypass time consuming tasks that the typical OpenGL driver would impose upon you. (For instance, validating a very large vertex buffer object [a list of points] in which only a few vertexes have actually been changed.) However, it also means that there the programming becomes far more complicated than it was previously, because all of the details that were previously handled by the driver are now up to the programmer.

One way of saying it goes like this: if OpenGL is javascript, then Vulkan/DX12/Metal is assembler. But to make this whole thing more down to earth, let's try a football metaphor: when you use OpenGL, the coach (the application programmer) tells the team: "get the ball in the goal." Each member of the team (the OpenGL driver) has been individually trained to take into account their own position, the position of the other players on the field, whether those players are their side or not, etc., etc., and their individual skills coordinate to get the the ball in the goal. With Vulkan, DX12, Metal the coach gives specific instructions to each individual player, who then must obey those instructions. It is up to the coach to do the work that all the individual players used to do. If you're a really good coach, you're going to benefit because you are the mastermind of everything that happens on the field. But if your coaching skills are not so great, not only might you not benefit, you might inferior performance.

To close point one here, I asked my friend, "would you say that this means the complexity of programming (and thus the likelihood of bugs) goes up at least one order of magnitude?" He replied "At least one, maybe more. For sure, these new systems offer you a much bigger chance to 'shoot yourself in the foot.' " He finished by saying that really, really smart people programmed the OpenGL driver. To get big benefits with the new systems, you need programmers that are just as smart.

2) As pointed out in this article, the new systems are "aimed specifically at video games." Visual processing programs like Isadora, VDMX, Resolume, vvvv, Jittter, etc., etc., have very different goals than a video game. The notion that they will receive massive speed boots by using Vulkan/DX12/Metal is a question, not a foregone conclusion.

A GPU has many different pipelines. Two important ones are the vertex shader pipeline, which processes geometry, and the fragment shader pipeline, which processes color.

In gaming, the goal is to feed hundreds of thousands of triangles specified in 3D space to those the vertex pipeline so that the shader pipeline can apply a unchanging texture map image and all the lighting effects to those triangles, and finally convert the result into a 2D image on your screen.

The typical goal of a visual processing programs is to feed those pipelines a relatively small number_ of 2D triangles_ with a constantly changingtexture map – a very different specification from the gaming mandate above.

Specifically, visual processing applications are very often rendering a simple quad (i.e., a rectangle) that consists of two triangles. This means that the vertex shader pipeline is nearly idle because it is designed to handle far more triangles than that. Furthermore, such programs are constantly be uploading video images from the CPU into the GPU – whether it happens within the playback API (e.g., AVFoundation, DirectShow) or is handled by the application itself, this step has to happen at some point. This can be very problematic, because it is very prone to cause stalls on the GPU while it waits for the CPU to finish getting the data where it needs to be. (Because of Apple's early support of video processing/editing tools, you can find many pages on this topic in the Apple documentation.)

Again, it comes down to the skill of the programmer. If you know what you are doing, you can probably benefit. But visual programming languages may not benefit nearly as much as the game engines for which these new technologies are the target.

[CONTINUED ON NEXT POST]

[mark]

3) As soon as you add third-party plugins, you can kiss all those benefits goodbye. So, let's say you've put in a lot of time and spent a lot of money to get all your internal GPU related programming converted to Metal, etc. Now, you add in the possibility of using third party plugins. All the careful work you did to ensure that the vertex shader pipeline was humming along at top speed is now moot, because these plugins either do not use the new systems at all (i.e., FreeFrameGL plugins that use the old OpenGL 2 API) or were created by programmers less skilled than the ones you hired to improve your visual processing application. (My friend went on quite a rant here about Quartz Composer, which he claims is a terrible offender in this regard.)

Almost all visual processing applications offer plugins because such programs are focused on creativity. But whole point of the Vulkan/DX12/Metal systems is to bypass the OpenGL driver so that you can be in total and absolute control of every aspect of the rendering cycle. As soon as code from someone else's plugin enters that pipeline, you have now lost control, and your program is only as fast as the slowest plugin you happen to be using.

4) Finally, Metal and DX 12 are not cross platform. For this reason, I hope that Vulkan is the ultimate winner. As a developer with limited resources of money and time, OpenGL offers a great benefit: you write the code once, and it runs the same on Mac and Windows. Given a choice between a platform specific API and a cross platform API, the decision regarding which I will use is a no-brainer.

5) OpenGL is not dead. From the article I previously cited, take note of this quote from Neil Trevett, president of the Khronos Group, that maintains and develops OpenGL standard:

"This is one of the key messages here," says Trevett. "Vulkan is nice and new and shiny, but of course it's just at the beginning of its development cycle. OpenGL, they're in their prime. They're enabling access to billions of devices. There's going to be business imperatives that we not just maintain those APIs—we evolve those APIs. For years, probably, to come. We're not abandoning OpenGL.”

------------

So that's quite a bit of detail. I hope that this little report offers everyone some insight. But let me offer these final thoughts.

As the creator and designer of Isadora, i've got to balance a lot of factors. Video performance is just one of them. We get lots and lots of features requests from users. People want to do things like reading motion capture data directly from the second-generation Kinect, or to have integrated support for DMX and/or Artnet lighting control, or a larger palette of video effects. Oh yeah, and really basic stuff like Unicode support for our Russian, Japanese, and Israeli friends. The list goes on and on.

In the end, Isadora is a tool for creativity. Performance is important, but adding features that give users more creative options and/or make Isadora easy to use is a far higher priority. It is the guiding principle by which I will decide where to allocate time, money and other resources. If it becomes clear that the Vulkan/DX12/Metal triumvirate will give Isadora users those options, I will invest in those technologies. In the meantime, I will focus my energy, and the energy of the team, to improve stability and to offer as many creative possibilities as possible.

Sincerely,

Mark

[mark]

One final note specifically related to this comment by @Unfenswinger: imagine if the all the current CPU actors could be executed on the GPU?

Rest assured that there will be significant news on this topic in the not-too-distant future.

Best,

Mark

[Bill Cottman]

Thanks for your ability and willingness to investigate the cutting edges and to explain in understandable language. Isadora is far closer to the cutting edge than my needs demand, but it is comforting to know I'm using a modern tool.

[Unfenswinger]

Thanks for the all the detail Mark.

Looking forward to the future announcement also. That's great news, no, amazing news!