Audio Features for Isadora: What Do You Want?

mark_m

Another vote for vamping.

And aside from that I would like first and foremost a way of playing back an audio file or a movie file and for Isadora to be able to analyse its frequencies, so that I can create images that respond to the beat.

I would also like there to be an energy level monitor, so that I can get Isadora to 'listen' to and respond to live sounds of a certain volume level and / or pitch.

I really really want to understand better how to integrate Isadora with, say, Ableton Live. People keep threatening to write tutorials, but they haven't appeared yet.

And finally, I'd like Isadora to be able to respond to voice commands, like my GoPro does. Or my iPhone... "Hey, Isadora, go to next scene"

I'm not serious about that last one.

Michel

@mark

Please make the sound level setting in movie player and sound player switchable to dB instead of percentage. Its sometimes embarrassing if someone asks to put up the level a few dB's and I say I can't I must do it in percentage.

eight

@mark Buffer and Delay actors, similar to those in Video department.

Michel

@mark

It would be great if Isadora could read 24bit audio files. It does work when I embed them into a movie. I guess not many will ever hear the difference between 16bit and 24bit be it because of the hardware or their ears. But we have sound designer students and they would probably swear they hear it. The main difference is explained in this youtube video, starting at 6min 45 sec:

kdobbe

@mark

Yes on both... I should have been a bit clearer.

bonemap

Hi,

From what I have learnt about audio input and routing in Isadora (I could have easily missed something) there appears to be currently the following options for the Mac version:

Audio sources:
1. Live Capture: Microphone/ Line in (4)
2. Audio File
3. Video File
4. Sound Input plug-in (device selectable)

Audio matrix routing is currently configurable in three places within Isadora Mac version:
1. Sound Output Setup: Sound Output Routing: Channel 1 - 16 /Ext. 1-32 (in pairs)
2. MoviePlayer: audio trks /trk vol /trk pan /trk out (1-32 in pairs)
3. AUMatrixMixer: /inputs (64) /in channels (64) /out channels (16)

Really it would be shame to lose any of these options from the Mac version, but it would be great to see them ported to the Windows version in some way.

I don't know how possible it is to have a cross-platform audio plug-ins, but access to a suite of 3rd party plugins including audio generators would also be a good outcome.

best wishes

Russell

mark

@bonemap said:

Really it would be shame to lose any of these options from the Mac version, but it would be great to see them ported to the Windows version in some way.

Dear Russel,

None of the Core Audio features will be removed from MacOS. So nothing you mention related to that will change. (Sound Input plug-in, AUMatrixMixer.)

The Movie Player routing will change, so this affects #1 and #2 in your second list. But it's going to be be better (in my opinion) because the pair limitation will be removed -- you'll simply be able to send any movie audio channel to any output on the device you desire.

And while I hear several of you mentioning cross-platform audio processing via plugins, remember that's something we will have to consider for a future release. As I said at the outset such a feature will not be part of Isadora 3.1.

Best Wishes,
Mark

Fred

@mark

I would love to see audio control and routing inside Isadoras node system. So the same way we have a projector we could have an audio output, this would be a hardware device attached to the system, each audio signal channel by channel or as a group could be connected in the same way we connect other Isadora data, from output to input. We could use the same tools like the router, switcher and transmit and receive as we do for other data.

This means we could route audio from a video file, input device or audio file player either for all channels or channel by channel to any output device, or to multiple devices or dynamically changing outputs. On OSX this would mean those signals could be interrupted by AU plugins and if VST was implemented a lot of the infrastructure would be there already.

It would be great if frequency analysis was switched to an actor as well. I think in general it would be great to avoid having another layer of interface in Isadora.

A few actors, like a mixer for combining audio and a splitter and an audio in watcher would be great to keep the consistency as well. If these could transparently integrate with au actors (or need conversion as we do with vid CPU/vud GPU formats) it would be very cool, maybe AU matrices and au audio input or au file players could be phased out slowly.

Also a revamped audio file player with the same behaviour as the movie player, but supporting wav, aiff MP3, Ogg and whatever else is easy to add. But for sure full support of standards, particularly wave files, so all bit and sample rates and multichannel wave support as well.

Fred

Fred

@fred oh and also an updated audio player, that has the same kind of interface/input parameters as the movie player currently has, I find myself making movies of audio almost all the time so I can use the movie player instead of the sound player.

tomthebom

@juriaan

As far as I know, Isadora offers routing MIDI in and out of Isadora. When it comes to manipulate MIDI, we are using DAWs like Ableton LIVE or others.
And maybe that is the most efficient way when it comes to making musik, since Mark and the crew has to priortise their efforts to improve Isadora.

And it is just a thought: What would happen, if we had actors like MIDI chord, MIDI Arpegiator, MIDI Transpose and some others to be used with Isadora actors like 3d Particles?
I am pretty sure, this can be done in Isadora already including some heavy math.
But -since I am working a bit with MIDI and know, what fireworks might start with these MIDI-features, it could very well be worth it, couldn’t it?

Best regards, Tom

mark_m

@fred said:

@fred oh and also an updated audio player, that has the same kind of interface/input parameters as the movie player currently has, I find myself making movies of audio almost all the time so I can use the movie player instead of the sound player.

Yes, this!

Juriaan

Make them first class citizens of the Isadora program

Audio in my opinion should not be included as an parameter in the Movie player / Sound player where you just choose what Channels the Audio track should go, instead I propose that audio becomes a signal that we can route / alter through out the program in future releases of Isadora. (After Izzy 3.1 it opens the posibilities of adding VST plugins or certain audio plugins / allows and enpowers the community to create tools in the plugin section of the website regarding audio)

How I personally would love to see it :

Audio becomes a signal (just like video is / text / floats / int)
Audio has to be routed to an output device using an Audio out node. (So we hide the parameters of the Movie player / Sound player, they are only there to play the file)
Audio in can be a Movie Player / Sound Player / Live Capture device / NDI (with Audio)
Audio channels can be linked with the Audio Output > Setup window. There you can say that Output 1 for example should be Output 5 on your Sound card

The workflow above already exist in the program for Video, so it is not a new practice that you introduce and makes sense if we compare it to how the stages work for video (Generators / inputs that are sent to a Projector that has a stage connected to it)

Input matters

Video signals in Isadora are coming mainly from Media files / and or the occasional live camera / sensor live feed / etc.
What I propose is that we get an dedicated way of getting the inputs of our sound card inside Isadora. This can be done through the Live Capture settings

bonemap

Hi,

Unlike @Juriaan, I am not adverse to integrated audio channels (or audio frequency bands) integrated into the Movie Player. This is a preference for the kind of synaesthetic and generative work that I focus on when using Isadora, but perhaps I am the in the minority here.

I do agree that a module architecture dedicated to audio is going to be a most useful and stimulating development. However, I don’t see an issue with having a diversity of multichannel audio solutions associated with flavours of media input and playback.

Since the discussion is prefaced with a specific roadmap for multichannel audio it is unclear how the more revolutionary ideas suggested by participants in this thread might be considered.

Isadora is not a timeline based interface, but in my mind a waveform over timeline module for audio playback, looping, de-vamping, live channel mapping, live effects routing would provide an invaluable addition to the software.

Best wishes

Russell

bonemap

hi,

perhaps the Sound Player and Movie Player could have an option to display the audio waveform in the progress bar. That way the module could be expanded to see more waveform detail and create more accurate loops etc.

kirschkematthias

@juriaan

very good idea! And timecode control, start, stop, pause, jump, loop function, input and output routing, output devices, sound output of videos for manipulation etc

greetings Matthias

kirschkematthias

@fred

Yes! yes! yes!

mark

So for all of you passionate about, as Juriaan said it, "making audio a first class citizen": as I said, implementing any kind of plugin structure where audio is routed through the program is a major undertaking. Are all of you saying you'd prefer to not have the new audio routing features (which I believe we can give you in relatively short order) but instead that you'd prefer to wait the until the fall of 2020 to get fully patchable audio added to Isadora?

On macOS adding this kind of funcionality would require far less effort because Core Audio gives gives it to you as what is essentially a built-in feature. I have searched for a open-source ASIO based VST host for Windows that emulates the behavior of what macOS offers -- but I have never come across one. That means we have to build the whole thing from nothing, and then to test it and make sure it is reliable in mission critical situations. This effort would pretty much consume the resources of the company, and aside from bug fixes, I don't think you'd see many other major improvements if we were to take on such a project.

Why do I say fall 2020? I've become far more cautious about estimating how long it takes for us to do something this big. I am guessing it would take roughly four to five months to implement the core features we need. I would then add two to three months of internal testing plus at least two more months of beta testing. This comes out to a development time of approximately 9 to 10 months.

Thoughts?

Best Wishes,
Mark

P.S. I am not saying I'm going to take this on. There are a constellation of concerns that will determine what features we add when. I am trying to:
1) hear what you want most, and
2) give you a real-world time frame for accomplishing the features your requesting.

jfg

@mark

I prefer to have the new audio routing features as soon as possible and that you keep time to correct bugs or optimise existing actors.

Thanks

best

Jean-François

bonemap

Hi,

Perhaps you could release the OpenNi tracker suite of plugins push for increased sales based on its awesomeness, fill the coffers/get cashed up and outsource the audio module architecture for major release in 12-18 months as ‘Isadora 4 Audio’. All the while in the short term providing a multichannel MoviePlayer for PC.

Sounds like a plan - but ?

Fred

@mark For me VST is something that can wait. I would really like audio treated the same way as any other signals, so node based routing and connecting outputs to inputs for all audio channels of audio and audio from videos, audio inputs and audio playback, and having audio outputs as an actor. But I understand this is a big task and will need to be cut into chunks and worked on progressively. I am sure you have checked out a all of these opensource projects but Jack, openAL, dr_wav, dr_mp3, dr_flac and stb_vorbis are some useful libraries I have come across , but my requirements for licensing are much less strict than yours.

I am not sure about timing, IMHO I think it is better to wait for a complete interface (if the node based approach is what will slow you down), rather than expand on the current audio routing setup or make another intermediate interface.