After a bit of research it does seem like the Mavericks codec or container problem at the bottom of this.
If I set the compression codec to Hap, Hap Alpha, H.264 or ProRes4444 the stage records, and captured files can be read directly in Isadora, but as above they'll all need to be imported in QT7Pro and recompressed before they'll show up in the Finder quickview, or used by some other apps. Other codecs might work too, but after a bit of success with these ones creating files that could be converted later I didn't check them all to see if one creates a file that can be read directly by QTX or the Finder immediately.
I'd mistaken the Finder's difficulties for "failed" recordings, but once reprocessed these are OK. Just a pain to keep having to work on them twice.
I wanted to confirm that a captured file from a channel could be replayed within the same patch without having to reprocess mid-show, but this seems fine.