severe video sync issue

Lauri

Hi,

For a long time I’ve noticed some video sync/stutter problems in different situations in Isadora – whlie playing a stage, while recording a stage,…

I came across this problem again and made a test project to investigate it. (Please see the image below.)

The results:

A) on the stage (while recording and while not recording) there occurs stutter periodically but different times on each camera signal – the stutter started in the first camera about one minute after launching the patch. Let's say cam1 stutters but cam2 doesn't stutter, and when cam1 stops stutter then after a while cam2 starts to stutter, and when cam2 stops stutter then cam1 starts to stutter… and this goes on periodically (at least in this test on MacPro this seems to be about 7mins cycle).

B) when investigating the captured stage video file (for instance in QuickTime7) the stutter is clearly similar to what was seen on Isadora stage. And playing it frame by frame, I can see that there are dropped/duplicate frames, almost every second frame, but only one camera at a time (like I explained above).

C) investigating with QT7 the captured camera files (Capture Camera to Disk), the files are playing 50fps with no dropped/duplicate frames. (Well, only in one capture cam file 5 frames dropped during 12mins, the other totally without dropped frames.) This proves that there’s no fault in the capture dongle or the signal entering Isadora.

My conclusion is that there’s a problem how Isadora (or Mac OSX) handles the video sync on screen. But as I said above, I’ve noticed this before in different situations while using Isadora for a project, even without any capture actors/processes.

In the testing setup Isadora prefs were set to 50fps (target Frame Rate); default resolution and stage preview settings 1280x720; Isadora capture settings 720p50; capturing 720p50. But the issue happens with other frame rates (50, 59,94, 60,…) and resolutions (720p, 1080p,…) too – the cameras, capture devices, Isadora settings adjusted accordingly. I tested the issue with two macs:

1. MacPro (2013)

- 3.0GHz 8-Core Intel Xeon E5 processor

- 64GB of 1866MHz DDR3 ECC memory

- Dual AMD FirePro D700 graphics processors

- OSX10.12.6

2. MacBookPro (2018)

- 2.6GHz 6-core Intel Core i7

- 32GB of 2400MHz DDR4 memory

- Radeon Pro Vega 20 with 4GB of HBM2 memory

- OSX 10.14.3

And the issue occurs similarly on both of them.

Additionally, in this test, I’ve used two Sony a6300 cameras and two Magewell USB Capture HDMI Gen2 capture devices.

You can find also a .zip-folder containing 5sec samples of the three video files to illustrate the problem from the following link:

https://www.dropbox.com/s/6yol...

You can see the dropped/duplicate files in capt_stage_to_movie.mov on the left side of the video image, if you go frame by frame.

Please could anybody help solving this annoying problem!

Thanks in advance,

Lauri

Lauri

…nobody? I’m surprised.
Has anybody experimented with the patch to try to recreate the issue? I would be interested if I’m the only one. Also interested if it happens with PC (I’ve no access to windows machines). I’m convinced it will happen also with 25fps, 30fps,… and will test it soon. I will test it soon myself also with BM UltraStudio Mini Recorders (which I have, but have very poor functionality with Sony cams).

However, it would be valuable to know if others get the same results.

By the way, there was a slip in the image of the patch: the Trigger Value actor does not do anything, it’s unnecessary. So here’s a new image of a corrected patch.

Woland

@lauri

Sorry, I'm not sure how I missed this. Did you also submit a support ticket, that's generally the best way to bring potential bugs to our attention.

Could you also post your test patch, it wasn't included in the link and in order to repro the issue I need the patch.

Woland

Also, capturing two live feeds, outputting a stage, capturing two live feeds to disk, and capturing the stage to disk all simultaneously could be too demanding a task for the computers; you might be hitting a hardware bottleneck somewhere. What is your use case? Do you need the recorded movies to be played back later as part of the project or are you recording things for archival purposes? If you're recording for archival purposes, my suggestion is to use a signal splitter to split the signal coming from the two live feed sources and send the signal to your playback computer for playback and send the same signal to a separate computer setup to do the archival capture camera/stage to disk. You could also put a signal splitter between the playback computer and the projector it is outputting to so that you can send one feed of the stage output to the projector and the same feed of the stage output to the capture computer for capture.

barneybroomer

I would suggest to take a look at the video input settings of the Magewell 2 input devices is the captured frame rate indeed what is should be? I wonder also why the BM devices are not match for the sony cams. The only thing I can think of the Sony cams don't fit the fixed video profiles of the BM. You tested on Mojave and High Sierra, but with what version of Isadora? Since Mojave is also fully 64 bit, and the latest Izzy not, I would only focus on the High Sierra system. The Magewells are USB3, did you use maybe some hubs in-between? If so, check in the system report if the whole flow is USB3 and not USB2.

According to your findings:

A - Maybe the capture drivers are not optimised to run 2 units on the same machine. I know with the BM's it is no problem.

B- Capture frame rates, I really suspect there is something wrong there.

C- Did you record 2 streams at the same time with 2 instances of QT7 here? And if so, maybe it proofs capture works, but without Izzy.

In previous patches you encountered the same stuttering issues without recording, so in my opinion the problem is really capturing 2 video streams at the same time. I wonder what activity monitor will say if you record just one camera, and then 2. And, what Woland said before, it is a lot, capturing 2 streams, record them and record the stage at the same time.

Cheers, Barney

Lauri

@Woland @barneybroomer

Thousand thanks for your replies!

I haven’t done a support ticket yet, but will do soon. I put the image of the patch to explain the patch (and reproduce the issue), but please find the test patch here.

Lauri's capture test.izz

About the patch, there are four scenes, first (Q1) is empty, second (Q2) streams two camera sources and records both cameras and the stage, third (Q3) streams two camera sources and records only the stage, fourth (Q4) only streams two camera sources.

Now, I’m seeing the stutter (dropped/duplicate frames, see explanation in my first post) even with Q4 which is only streaming simultaneously the two cam sources. That should not overload Isadora, right?

I’m seeing the stutter also in Q3 that streams two camera sources and records only the stage. And that should not overload Isadora either, right? And investigating the recorded file, I can see that the stutter is real (going frame by frame for instance in QT7, and I used QT7 because there you can easily “play” the file frame by frame with arrow keys and see the time-code numbers).

And as told before, the stutter occurs (obviously) also with Q4. However, this scene was made only to exactly prove that the fault is NOT in the capture cards, because the captured camera streams are not stuttering (don’t have dropped/duplicate frames) and play perfectly the recorded 720p50 for instance in QT. So, my goal is not to use this (Q4) to anything else than testing.

However, my goal is to stream simultaneously two live camera feeds, and simultaneously record the stage, and… – but not recording the individual cameras. But if the starting point of just streaming two simultaneous live cam feeds does not produce stutterless stage, it will ruin the goal. (In my test patch, you can also replace the live video streams with movie players, and the same issue occurs eventually – maybe not with the same length of cycle I described initially).

Why not BM UltraStudio Mini Recorders? Well, with Sony cameras, you can only get 1080i50 signal through BM Mini Recorder. Nothing else works. And I was explained couple of years ago that Sony and BM use “different kind of HDMI protocol”. And that’s why, for example, you cannot get even any true 1080p25 or 1080p50 from Sony through BM UltraStudio Mini Recorder. I can get a signal through BM Mini Recorder, when Sony cam video recording is set to 1080p25 or 1080p50, and in camera HDMI out set to interlaced (1080i50). BM does not accept any other signal from Sony HDMI. And from these signals BM delivers only 1080PsF25 or 1080i50. But then every second line missing – resulting to unsharp image (half vertical resolution) and therefore also edges not smooth. And this is unacceptable for me.

Yes, I know, Isadora 2.x is not compatible with Mojave, but I was forced to Mojave, because Apple replaced my previous MBP to a new one. However, because the combination seems to work in my machine, I wanted to test with that too. My tests were made with Isadora 2.6.2b3 on both MacPro and MBP.

Because the issue occurs in cycles, I would advice to make the test at least 10min long. So if possible, please test yourselves and let me know if you are able to reproduce the issue.

Thousand thanks in advance!

Lauri

DusX

Have you tried to disable vsync?

Woland

@lauri said:

if the starting point of just streaming two simultaneous live cam feeds does not produce stutterless stage, it will ruin the goal.

Try this method as a workaround to see if it results in a stutterless stage for you: http://recordit.co/2pevGpGdAl (The stuttering you see in this is because the gif has a low FPS.)

Basically, if starting the live feed and beginning to record the stage at the same instant is causing a stutter, does starting the live feed earlier solve the problem?

>>EXAMPLE FILE (AND RECORDING OF ME WITH A FAKE CAT ON MY HEAD)<<

Best wishes,

Woland

Lauri

Thanks for your replies!

@DusX Will try as soon as getting to my studio.

@Woland The patch does not start live feeds and recording simultaneously, there's a "Key Table Watcher" actor to start recording. However, as I pointed out, the stutter occurs even without any recording. Please see/try the patch and the last scene (Q4).

Thanks,

Lauri

m_theater

@lauri

https://forums.macrumors.com/t...

This is a link showing underlying architecture of the nMP

Limitations of the newMacPro
Looking just at raw data without handling overhead 2 Full HD Streams won‘t go through the connection between the usb 3 —> PCH chip (Limited to 500 MB/s)

24-bit, 1080p @ 50 fps: 24 × 1920×1080 × 50 = 2.3 Gbit/s.—> 289 MB/s x 2 Streams —> 578 MB/s

with 720p50 or 1080p25 streams it shall work:
24-bit, 720p @ 50 fps: 24 × 1280×720 × 50 = 1 Gbit/s.—> 128 MB/s x 2 Streams —> 256 MB/s
24-bit, 1080p @ 25 fps: 24 × 1920×1080 × 25 = 1.2 Gbit/s.—> 148 MB/s x 2 Streams —> 296 MB/s

——————————

Handling in Izzy

Is there any particular reason you are working with PAL 50hz frequency ?
Your internal Laptop Screen will only refresh @ 59.9 or 60 hz, it is probably the same with the Screen you have attached to your nMP.

I made the observation that it is hard for Izzy to keep up with interpolating between 60 Frames/s of your Screen and the 50 or 25 Target fps

In other words the video processing pipeline will always be adjusted in its speed.
which can lead to missed frames / double frames

I could imagine a solution to oversample your inputs
1080p30 video input
1080p30 video files

Target frame rate 60 hz
Even if the fps drops you have at least all frames shown ones (see below)
Frame 1 Frame 1
Frame 2 Frame 2
//drop//
Frame 2 Frame 3
Frame 4 Frame 4
…

greetings m_theater

barneybroomer

@m_theater oooooow, this is a good one!

Lauri

@m_theater Thanks for your reply!

720p is enough for me, and (as proved, see original post) Isadora can record two live feeds 720p50 without dropping frames. So, USB bus should not be the culprit. And I have the nMP screens adjusted to 50Hz refresh rate (monitor and projector), so that should not be the issue either. (And while using MBP, the external display was also set to 50Hz, although I understand the conflicting refresh rates between laptop internal and external projector can be a problem.)

Why PAL 50Hz, well, I live and work mainly in Europe and the electricity is 50Hz, and that affects the lighting too. Due to that, there can be irritating flicker in camera. And if using 60Hz frequency in camera, the stutter speed should be 1/100s to avoid the flicker, and that's too short (low light installation environment). With 50Hz frequency, I can have the shutter 1/50 (double).

However, I can assure you, the dropped/duplicate frame issue happens also when everything is set to 60Hz (project, camera, capture device, monitor, projector,...). Interestingly enough, if I adjust the capture device output to 59,94fps, the stutter cycle becomes about 15secs, but that's evidently due to conflicting frame rates.

I tried also setting the capture device output to 30fps, but it does not make it look smoother. Smoothness is the key issue here, right? The stutter is visible also with cam/capture 30fps. When Isadora project frame rate is set to 60, it records the stage in 60fps, not 30fps even if Compression Settings/Motion/Frames per second is set to 30 (this setting seems to affect only captured camera files).

So, to preserve smoothness, I would still prefer using 50fps (or 60fps if I was forced to use 60Hz displays). But again, the stutter is evident whether 30,50 or 60fps.

But I would be interested if anybody can reproduce this stutter issue. Anybody tried? Please find the patch and instructions in the posts above.

Thanks again!

Lauri

barneybroomer

@lauri unfortunately I am stucked in Frankfurt at this moment, in the studio I have Mac Pro bin that I could test, but I use BMI UltraStudio HD Mini's with 1080P as input

Lauri

@barneybroomer Thanks for your reply. If you have a chance after returning from Frankfurt to test, it would be great! I would expect it does not make a difference if it's BM or Magewell.

Fred

@lauri I have used the Blackmagic stuff for years and recently got a magewel device. They are ok, but I did notice a few things, they use more resources than Blackmagic, and although they are lower latency they end up with lower quality colour encoding. The bandwidth use means they need to do more compression for the signal onboard, which means more decompression to get actual pixels.

I have a 4 input capture card from blackmagic on my PC and Isadora can deal with it perfectly well. I have pushed it to 4* 1080p50 signals with high resource use but still a smooth image.

The magewel devices have a hardware configuration tool that you can use to set incoming video formats and even scale the image. I would make sure that the signal you are inputting from the cameras is the same as what you want to capture, otherwise you are performing unnecessary conversion.

As for your issues with Blackmagic and Sony cameras, they are both industry standards (although the consumer cameras you are using have little control over the HDMI output, Sony will adhere to standards for this) You do need to do a bit of work, ie make sure that your cameras are outputting exactly the resolution you are trying to capture at, so if you want 720p make sure your camera is outputting a 720p signal. Although magewel devices will (depending on the driver and software), let you capture cross format (camera outputs format X and you capture at format y) this is going to give you overhead problems. The suggestion you had trouble with the Blackmagic capture cards makes me a think maybe this is the case.

Can you verify the output from your cameras is 720p how you are connecting to them (HDMI or SDI), and how you verify they are outputting a 720p signal over that connection. You can verify this usually by plugging the signal directly into a tv, most TVs will then display the input format for a few seconds, if it is not 720p this is the first problem). As far as I can see the cameras only output 1080 signals so before Isadora gets the signal magewel and Mac os are converting it. This overhead is likely to give you trouble.

Then as far as resource use, the macpro trashcans are pretty lacking in USB 3, the performance is worse than suggested above and as far as I know the resources are shared, so the single PCI e 2.0 lane, also shares resources with the networking gear (I think could not find the document easily). Here is some info https://macperformanceguide.co... in this case using thunderbolt attached capture cards will give you a lot better performance.

I know a lot of people use multiple cameras in isadora and this part of the software is a bit of a no brainer, it either uses the apple built in driver's or Blackmagic drivers and then just shows the frame. It works pretty well for me, but I carefully put together a machine that is designed to handle this kind of thing.

Can you test with another software that can do 2 channel simultaneous capture and record? OBS is free and can do this, check if OBS has any issues with this as it may well be a hardware limitation.

Armando

@lauri said

Sorry to maybe say a stupid thing. But is your Isadora version 64 bits? The 2.6 you are using is not optimized for OS X 10.14. I had trouble in Mojave with the 2.6 n.

Lauri

@Fred Thank you for your reply. And sorry for not responding earlier.

Sony a6300 HDMI out options are ”auto”, 2160p/1080p, 1080p and 1080i. There’s no 720p HDMI out.

So, with Magewell, I’m forced to down-res the signal to 720p. The incoming HDMI signal is 1080p50 and outgoing 720p50 that are both shown in Magewell USB Capture Utility. (So, Magewell accepts the incoming 1080p50, while BM UltraStudio Mini doesn’t.) I can verify the down-ressed signal both from Magewell USB Capture Utility (Capture format: 1280x720, 50fps, YUY2) and Isadora Live Capture Settings (Video Input: Resoluton Native, Format 1280x720).

Yes, it’s true that 2x1080p50 is too much for USB lane.

(With BM UltraStudio down-ressing is not possible, it does not accept 720p setting, black screen. And, as told before, BM UltraStudio Mini does not accept 1080p from Sony HDMI. BM UltraStudio Mini accepts only 1080i50 (or i60) from Sony HDMI, keeps it as that or converts to 1080PsF25 (or PsF30), but you will see the interlacing on both. So, there’s no real 1080p25 or 1080p30.)

However, as mentioned before, with Magewell down-ressed signals, Isadora captures both simultaneous camera signals flawless, no dropped/duplicate frames. So, if it can do that, why it cannot draw the streams properly? Even when only streaming the live cams on stage, no recording involved.

--------

When using BM UltraStudio Minis for capturing live cams (1080i50 or 1080PsF25), there’s also some sort of stuttering that happens cyclically. It looks different, but that’s probably because the framerate is not 50fps.

(P.S. Interestingly, when using BM UltraStudio Minis and cam HDMI out 1080i50 (the only setting that works with BM UltraStudio Minis), I can get cam record to file only if using Apple driver. If using Blackmagic driver, the recording starts but there are no files.)

--------

When replacing the Video In Watcher actors with Movie Player actors and playing the files through the Video Mixer actor, the result is very stuttery. This should not be heavy task at all. The live cam recorded files are Photo-JPEG (720p50, of course), no dropped/duplicate frames. VLC shows Codec details: Motion JPEG video, Planar 4:2:2 YUV full scale, Color Space ITU-R BT.601 Range.

When playing the recorded live cam file outside Isadora, I get strange results (I have checked that the file does not have any dropped/duplicate frames):
- playing the file with QuickTimeX, the file stutters a lot, there’s some kind of cycle that the file doesn’t stutter, but mostly stuttering - playing the file with QuickTime7, the file stutters in cycles, but has longer periods when not stuttering and the cycles are more regular
- when playing the file with VLC, the file doesn’t stutter at all, except that there’s one dropped frame every 72secs
This brings me back to think about a sync issue.

----------

Also, usually when starting the patch (Q4, only playing the superimposed live cam streams, no recording), there’s no stutter for about 2mins and then the stutter starts first in one cam – and then the stutter changes to the other cam and stuttering alternately in a cycle of about 7mins. So, why would it play well for a while and then start stutter later, if the problem is the ”cross format processing”.

----------

I downloaded OBS but didn't really get it work in a similar way to record (or just show) two superimposed live cam feeds simultaneously.

Lauri

@Armando

No, I use Isadora 2.6.2b3 (or also 2.6.1), so it's not 64bit. However, as stated before, I made the tests both with MacPro OSX10.12.6 and MBP OSX10.14.4. I know Isadora 2.6.x is not compatible with OSX10.14.x, but combination works with no problems in my MBP(2018). So, in this context, forget about the MBP/Mojave, the issue is prevalent in MacPro/Sierra.

Fred

@lauri well the mini recorder is only for 1.5 GHz signals, clearly stated in all the specs, on the website and instructions and even in the setup utility. 1080p50 is 3ghz, so if course it will "not work", it adheres to all specs but you are trying to get it to do something it does not, there are other products from black magic for capturing 1080p 50, best to do the proper research before you purchase. The mini recorder is functioning as it should and has no "problems" with the sony output, it is not designed to capture that format and never was.

So even though you are capturing at "720p", your input is 1080p to the magewel and you are using that bandwidth on your usb3 bus so that for sure is part of your problem .There is no magical conversion, your capture device is running at 1080p50 and then running a conversion on every frame, if you were not trying to record and further push the limited resources you have I would suggest just running at 1080p50.

The artefacts you see from the 1080i and the 1080psf25 capture are more to do with the sony output from this consumer camera than any of your capture devices. If the camera outputs a proper deinterlaced signal it will look fine, however many of these cheaper souped up DSLRs cannot really do on sensor 1080i. Instead it is converted afterwards.

As your cameras do not output 720p you should really run at 1080p, but your system for USB based capture is under spec (as it is for this setup @ 720p, as you are still capturing at 1080p on the magewel and those frames are uploaded to the CPU for processing and then delivered to Isadora at 720p).

Running a whole system at 1080p is not that easy (which even though you bump down to 720p after capturing, you are actually capturing at 1080p). The Gear you are using is going to be pushed trying to do this, in general apple don't make anything ready for this, no good gfx cards and getting a system equipped with proper 1080p capture is very expensive.

OBS is simple to use and can preview and record 2 streams, I think it is worth reading the manual and checking your system can do what you want it to before looking for faults in isadora, and as well you are running an unsupported system. I know this sucks, Isadora is not ready for Mojave yet and every new computer can run Mojave.

As media artists using this kind of gear with high expectations there is a lot of research involved, we are not running some simple software and it takes good planning and piece by piece testing to make things rub smoothly and reliabily.

Lauri

@Fred Thank you for your reply!

First, I'm not running on Mojave! I've said that many times in my posts. I've tried the patch in my Mojave-laptop, BUT I'm running the tests in my MacPro 10.12.6.

Ok, I understand that I'm at/over the limits of the MacPro. And I do understand there's a lot research involved. I would appreciate all the help in finding a solution. It feels just a bit frustrating, I do not know where to head forward – if it's the case for only needing to change the computer, operational system, cameras and capture devices.