If you follow the instructions on pg 90 of the manual, you'll see that you can use the CueMix FX software to create groups of the individual channels - which you can match to the channels you're outputting from Isadora - which you can then control using the CueMix FX software OR the Master Volume knob on the MOTU.
That would solve your problem of outputting too high a level to the speaker, as you can then use the Master Volume knob on the MOTU (or the CueMix FX software) to control the master volume of your chosen channels. You can then refine the levels in Isadora, and not have to worry about setting the level of the sound player actor. Which you should not have to do anyhow, as there ought to be an overall level control from the Mac to the MOTU. As @JJHP3 suggests, not having control over volume is most unusual.