Higher-resolution video output will tighten things up. Each pixel is very multiplied by the nature of projection, so squeezing more into the same space by using higher resolution images/projectors will make things much cleaner looking.
But if you really need to soften the edge inside Isadora you can create a mask using Izzy map with a white video feed, sent to a virtual stage that you then grab and apply as an alpha channel to your actual output. In this way you can apply a blur to the black and white image before using it as the alpha mask.. this will give you soft edges.