Extended Video (and Audio) Delay

joshuawg

Thanks so much for teasing out the less expensive option. This seems very doable! (Also you were right to catch my bad simple addition, sorry!)

Just for comparison's sake, what sort of load are you getting on your Mac Pro when running the 9 video delays? Any recommendations for minimum system requirements?

Thanks again!

Fred

@joshuawg you can also do this in another way. I’m not sure if it’s easily possible Isadora but if you can precisely record and playback the right sized sections of video this should be able to be done on a single computer.

The issues to solve would be precise starting and stopping of the record process with no gaps and the seamless loading and playback of the chunks of video. With a modern computer these limitations will not be from hardware but from how the software lets you control these processes.

I would use another platform with more prefixes control to do the recording and playback of smaller chunks- gstreamer is a good option because it can do this, it’s free, and it supports NDI so you can get the streams back in isadora

gstreamer is an open source library that lets you do command line controlled video processing. It has a lot of features like fork and branch of video streams as well as recording and joining. It is very fast with low overhead and many functions are threaded.

You can ask it to record a stream into chunks and also to rebuild those chunks into streams. You can add chunks to a stream in real time. You can also stream playback with NDI.

Here is a bash script that outlines how this might work. You can use chatgpt or copilot to help you debug this on hardware.

#!/bin/bash

# Parameters
CAMERA_SOURCE="/dev/video0" # Adjust according to your system
SEGMENT_DURATION=10 # Duration in seconds for each segment
MAX_DELAY=780 # Maximum delay in seconds (13 minutes)
SEGMENT_DIR="/path/to/segments" # Directory to store video segments
NDI_OUTPUT_BASE_NAME="NDI_Delayed_Stream" # Base name for NDI output streams
DELAYS=(60 120 180 240 300 360 420 480 540) # Delays in seconds for NDI streams

# Ensure the segment directory exists
mkdir -p $SEGMENT_DIR

# Function to segment live video
segment_live_video() {
gst-launch-1.0 -e \
v4l2src device=$CAMERA_SOURCE ! videoconvert ! x264enc ! h264parse ! splitmuxsink location="$SEGMENT_DIR/segment_%05d.mp4" max-size-time=$(($SEGMENT_DURATION * 1000000000))
}

# Function to clean up old segments
cleanup_old_segments() {
while true; do
current_time=$(date +%s)
for segment in "$SEGMENT_DIR"/segment_*.mp4; do
segment_time=$(stat -c %Y "$segment")
segment_age=$((current_time - segment_time))
if ((segment_age > MAX_DELAY)); then
rm -f "$segment"
fi
done
sleep 10 # Check every 10 seconds
done
}

# Function to create and manage delayed streams
create_delayed_stream() {
local delay_seconds=$1
local ndi_output_name=$2
local segment_pattern="$SEGMENT_DIR/segment_%05d.mp4"

gst-launch-1.0 -e \
multifilesrc location="$segment_pattern" index=0 ! decodebin ! videoconvert ! queue ! ndisink name=$ndi_output_name &
}

# Start segmenting live video in the background
segment_live_video &

# Start cleanup of old segments in the background
cleanup_old_segments &

# Create delayed streams
for delay in "${DELAYS[@]}"; do
ndi_output_name="${NDI_OUTPUT_BASE_NAME}_${delay}s"
create_delayed_stream $delay $ndi_output_name &
done

# Wait for all background processes to finish
wait

joshuawg

@fred I hadn't heard of gstreamer — really excited to look into it! Will let you know if I can get it working!

Woland

@fred said:

gstreamer is an open source library that lets you do command line controlled video processing. It has a lot of features like fork and branch of video streams as well as recording and joining. It is very fast with low overhead and many functions are threaded.

You always know about the coolest things <3

joshuawg

Just wanted to send a quick update, since I seem to have gotten this to work! Sadly I wasn't able to go the gstreamer route with my limited coding abilities, but having a vague idea of the underlying theory, I was able to get it working in Isadora. As @fred suggested, it's a rolling capture of videos that are then played back on a trigger delay to create the illusion of a video delay. Of course the blip between videos using a single capture-camera-to-movie actor was a bit of a giveaway, so I ended up using an hdmi splitter to split my camera signal across two camera inputs (using loopback to split the audio signal as well), so I can alternately capture the two feeds with a bit of an overlap and crossfade between them — which has the benefit of letting me capture consistent audio! Anyway, thanks again for all of your suggestions. The load is super low and I'm excited to start playing with it.

Woland

@joshuawg said:

Just wanted to send a quick update, since I seem to have gotten this to work! Sadly I wasn't able to go the gstreamer route with my limited coding abilities, but having a vague idea of the underlying theory, I was able to get it working in Isadora. As @fred suggested, it's a rolling capture of videos that are then played back on a trigger delay to create the illusion of a video delay. Of course the blip between videos using a single capture-camera-to-movie actor was a bit of a giveaway, so I ended up using an hdmi splitter to split my camera signal across two camera inputs (using loopback to split the audio signal as well), so I can alternately capture the two feeds with a bit of an overlap and crossfade between them — which has the benefit of letting me capture consistent audio! Anyway, thanks again for all of your suggestions. The load is super low and I'm excited to start playing with it.

Wow that sounds fantastic! Did the performance already happen, if so, is there a recording available to watch? If not, where and when is it?

joshuawg

@woland

Doesn't happen until April actually — but I'm about to head out on a retreat tomorrow so I'm going to be working with the new patch then!

Fred

@joshuawg Wow, I would not have tried to do all this in Isadora, that amazing and so much more efficient that saving frames in memory.

antc

@joshuawg

Very nice patch! as an minimalist alternative for that "probably expensive hardware" way, this one seems a wonderful solution!

Hope you don't mind if I take some ideias from there.. ;)

joshuawg

@antc

Absolutely! Go for it.

Getting the crossfade between videos takes a little finessing — and sometimes there's a little ghosting/misalignment but in general I'm finding it really effective!