[ANSWERED] Convert sound frequencies to rgb colours

bonemap

Hi,

Here are some of my explorations of this topic. You are welcome to modify for your use.

https://troikatronix.com/add-ons/bonemap-audio-frequency-7-band-analysis/

https://www.facebook.com/10000...

There are significant limitations when using audio frequency analysis in Isadora. The big one (that is not documented) when working with multichannel audio input i.e. from your DAW, Isadora’s audio frequency analysis will only interpret channel 1. There is no option to analyse frequencies of additional channels in a multichannel audio input stream. So if you have a standard stereo track as an audio input, only one (the top channel) of the two channels that make up the stereo track are able to be transcribed as audio frequencies. Similarly, if you have a 4 channel audio track with content only encoded to channels 3&4, Isadora’s audio frequency analysis will not work because there is nothing on channel 1.

However, you can use audio levels (Sound Level Watcher ++) to achieve independent level analysis of each channel in a multichannel audio input stream.

Another issue is audio input routing. If your multichannel audio is embedded in an audio or video file and placed in Isadora as a Movie Player or Sound Player module a third party routing software is required to Loopback the audio as an input that can then be accessed for audio frequency analysis. This is further complicated because at the time of writing, the multichannel functionality of the Movie Player remains an unfinished project and as a result working with multichannel sound files that are either audio files or video files requires different workflows and software/hardware setups (We might anticipate a future fix for this situation in the Isadora development roadmap).

Best wishes

Russell

Notdoc

@vidasonik

may i suggest you do the analysis in Ableton and use OSC to bring the data into Izzy? Playing to the relevant strengths of each software... Both have very good OSC devices/actors available.

bonemap

@notdoc

“do the analysis in Ableton and use OSC to bring the data into Izzy“

Hi,

That sounds great! Would love to learn more about that method.

Best wishes

Russell

Armando

@bonemap Eventually analysis can be done in max (even better than in Live, although Cycling 74 belongs to Ableton) and be sent from Max to Isatura via OSC....

bonemap

@armando

Hey! Thanks for that. Yes, I see that OSC frequency analysis out of Ableton requires a Max patch. For my purposes, the added latency would be an issue even using the AUSampleDelay module, and it would likely be unusable with live ambient sound input.

Best Wishes,

Russell

Fred

@bonemap Ableton and max should add negligible latency, especially if you set the sound card buffer low. Most modern sound cards can work at 64 sample latency or lower. At 44.1khz this is pretty negligible (1.4 milliseconds). FFT, no matter how you slice it, on Isadora, Ableton or any platform, will incur a delay as it has to average frequency over time. Isadoras implementation in the sound frequency watcher has no control over the window length, whereas you can control this in Ableton, meaning you could reduce delay further than in isadora.

Signal transmission with OSC, especially on the same machine will have negligible or zero latency.

Your biggest delay is likely to be video output, projectors and screens can add whole frames of delay, DMX signals will not have this delay. If you want really tight sync Ableton may help you out in other ways. If you want super tight light and video sync, running audio through Ableton and using 2 sets of analysis (same settings) for the same signal and adding a delay to one, will let you delay (using an audio delay in Ableton) the OSC that gets sent to the lights so they match up perfectly with the video with very fine control.

If you don't want to use Ableton you can also do this in Chataigne, which is free and has quite a lot more control than Isadora.

bonemap

@fred

Thanks that’s all solid advice. The commitment propositions of the various software- Ableton Live, Max for Live etc. mean I am not in a position to invest in these options at this point. My work around has been to use sound levels with multichannel audio rather than frequency analysis. At any rate the frequency analysis in Isadora appears to be partly driven by audio level.

Best wishes

Russell

dbini

I've been working with a musician recently, generating some visuals from him playing the kantele (a traditional Finnish zither/harp-type instrument)
Using frequency analysis in Isadora has been a bit rough and ready and hit and miss. @bonemap - I'm going to try your user actor - thanks for sharing, Russell. I will also check out the possibilities of analysis in Ableton and OSCing through to Izzy, and report back next week...

Notdoc

@bonemap

my experience has been actually the other way around, using Izzy's amazing blob tracking to modulate audio stuff in Ableton using OSC. So not colour data, although that is also great in Izzy.

But it's easily fast enough. I've mainly been using light sources to move things around in the sound picture so it relates spatially to what's going on in the image.

It's not complex, but it's perceptible, which is important for me. I'm not really interested in creating a supersubtle relationship between gradations of colour and sound if it's not directly perceptible and meaningful to an audience. The human emotional spectrum relating to colour and sound is very diverse, whereas spatially we are all pretty much on the same page.

DusX

@vidasonik said:

Ableton to rgb colours in Izzy

This tutorial could be helpful if you don't have Ableton Studio. Control Isadora From Ableton Live with MIDI : TroikaTronix

Vidasonik

@notdoc thanks for all this wonderful help and advice! I've not got into OSC yet what would be a simple way forward? Thanks, David

Notdoc

@vidasonik no problem. my knowledge has tight limits, however..:-)

you should be able to use the Ableton Connection kit for OSC devices to talk to the onboard ones in Izzy.

But even better for what you are trying (if i understand it...) is the free 'LiveGrabber' OSC Ableton tools from Showsync. The 'Analysis Grabber' can 'grab' audio frequency data which you can send via OSC to Izzy.

I haven't tried it, but it should work. https://www.showsync.com/tools...