severe video sync issue

barneybroomer

I would suggest to take a look at the video input settings of the Magewell 2 input devices is the captured frame rate indeed what is should be? I wonder also why the BM devices are not match for the sony cams. The only thing I can think of the Sony cams don't fit the fixed video profiles of the BM. You tested on Mojave and High Sierra, but with what version of Isadora? Since Mojave is also fully 64 bit, and the latest Izzy not, I would only focus on the High Sierra system. The Magewells are USB3, did you use maybe some hubs in-between? If so, check in the system report if the whole flow is USB3 and not USB2.

According to your findings:

A - Maybe the capture drivers are not optimised to run 2 units on the same machine. I know with the BM's it is no problem.

B- Capture frame rates, I really suspect there is something wrong there.

C- Did you record 2 streams at the same time with 2 instances of QT7 here? And if so, maybe it proofs capture works, but without Izzy.

In previous patches you encountered the same stuttering issues without recording, so in my opinion the problem is really capturing 2 video streams at the same time. I wonder what activity monitor will say if you record just one camera, and then 2. And, what Woland said before, it is a lot, capturing 2 streams, record them and record the stage at the same time.

Cheers, Barney

Lauri

@Woland @barneybroomer

Thousand thanks for your replies!

I haven’t done a support ticket yet, but will do soon. I put the image of the patch to explain the patch (and reproduce the issue), but please find the test patch here.

Lauri's capture test.izz

About the patch, there are four scenes, first (Q1) is empty, second (Q2) streams two camera sources and records both cameras and the stage, third (Q3) streams two camera sources and records only the stage, fourth (Q4) only streams two camera sources.

Now, I’m seeing the stutter (dropped/duplicate frames, see explanation in my first post) even with Q4 which is only streaming simultaneously the two cam sources. That should not overload Isadora, right?

I’m seeing the stutter also in Q3 that streams two camera sources and records only the stage. And that should not overload Isadora either, right? And investigating the recorded file, I can see that the stutter is real (going frame by frame for instance in QT7, and I used QT7 because there you can easily “play” the file frame by frame with arrow keys and see the time-code numbers).

And as told before, the stutter occurs (obviously) also with Q4. However, this scene was made only to exactly prove that the fault is NOT in the capture cards, because the captured camera streams are not stuttering (don’t have dropped/duplicate frames) and play perfectly the recorded 720p50 for instance in QT. So, my goal is not to use this (Q4) to anything else than testing.

However, my goal is to stream simultaneously two live camera feeds, and simultaneously record the stage, and… – but not recording the individual cameras. But if the starting point of just streaming two simultaneous live cam feeds does not produce stutterless stage, it will ruin the goal. (In my test patch, you can also replace the live video streams with movie players, and the same issue occurs eventually – maybe not with the same length of cycle I described initially).

Why not BM UltraStudio Mini Recorders? Well, with Sony cameras, you can only get 1080i50 signal through BM Mini Recorder. Nothing else works. And I was explained couple of years ago that Sony and BM use “different kind of HDMI protocol”. And that’s why, for example, you cannot get even any true 1080p25 or 1080p50 from Sony through BM UltraStudio Mini Recorder. I can get a signal through BM Mini Recorder, when Sony cam video recording is set to 1080p25 or 1080p50, and in camera HDMI out set to interlaced (1080i50). BM does not accept any other signal from Sony HDMI. And from these signals BM delivers only 1080PsF25 or 1080i50. But then every second line missing – resulting to unsharp image (half vertical resolution) and therefore also edges not smooth. And this is unacceptable for me.

Yes, I know, Isadora 2.x is not compatible with Mojave, but I was forced to Mojave, because Apple replaced my previous MBP to a new one. However, because the combination seems to work in my machine, I wanted to test with that too. My tests were made with Isadora 2.6.2b3 on both MacPro and MBP.

Because the issue occurs in cycles, I would advice to make the test at least 10min long. So if possible, please test yourselves and let me know if you are able to reproduce the issue.

Thousand thanks in advance!

Lauri

DusX

Have you tried to disable vsync?

Woland

@lauri said:

if the starting point of just streaming two simultaneous live cam feeds does not produce stutterless stage, it will ruin the goal.

Try this method as a workaround to see if it results in a stutterless stage for you: http://recordit.co/2pevGpGdAl (The stuttering you see in this is because the gif has a low FPS.)

Basically, if starting the live feed and beginning to record the stage at the same instant is causing a stutter, does starting the live feed earlier solve the problem?

>>EXAMPLE FILE (AND RECORDING OF ME WITH A FAKE CAT ON MY HEAD)<<

Best wishes,

Woland

Lauri

Thanks for your replies!

@DusX Will try as soon as getting to my studio.

@Woland The patch does not start live feeds and recording simultaneously, there's a "Key Table Watcher" actor to start recording. However, as I pointed out, the stutter occurs even without any recording. Please see/try the patch and the last scene (Q4).

Thanks,

Lauri

m_theater

@lauri

https://forums.macrumors.com/t...

This is a link showing underlying architecture of the nMP

Limitations of the newMacPro
Looking just at raw data without handling overhead 2 Full HD Streams won‘t go through the connection between the usb 3 —> PCH chip (Limited to 500 MB/s)

24-bit, 1080p @ 50 fps: 24 × 1920×1080 × 50 = 2.3 Gbit/s.—> 289 MB/s x 2 Streams —> 578 MB/s

with 720p50 or 1080p25 streams it shall work:
24-bit, 720p @ 50 fps: 24 × 1280×720 × 50 = 1 Gbit/s.—> 128 MB/s x 2 Streams —> 256 MB/s
24-bit, 1080p @ 25 fps: 24 × 1920×1080 × 25 = 1.2 Gbit/s.—> 148 MB/s x 2 Streams —> 296 MB/s

——————————

Handling in Izzy

Is there any particular reason you are working with PAL 50hz frequency ?
Your internal Laptop Screen will only refresh @ 59.9 or 60 hz, it is probably the same with the Screen you have attached to your nMP.

I made the observation that it is hard for Izzy to keep up with interpolating between 60 Frames/s of your Screen and the 50 or 25 Target fps

In other words the video processing pipeline will always be adjusted in its speed.
which can lead to missed frames / double frames

I could imagine a solution to oversample your inputs
1080p30 video input
1080p30 video files

Target frame rate 60 hz
Even if the fps drops you have at least all frames shown ones (see below)
Frame 1 Frame 1
Frame 2 Frame 2
//drop//
Frame 2 Frame 3
Frame 4 Frame 4
…

greetings m_theater

barneybroomer

@m_theater oooooow, this is a good one!

Lauri

@m_theater Thanks for your reply!

720p is enough for me, and (as proved, see original post) Isadora can record two live feeds 720p50 without dropping frames. So, USB bus should not be the culprit. And I have the nMP screens adjusted to 50Hz refresh rate (monitor and projector), so that should not be the issue either. (And while using MBP, the external display was also set to 50Hz, although I understand the conflicting refresh rates between laptop internal and external projector can be a problem.)

Why PAL 50Hz, well, I live and work mainly in Europe and the electricity is 50Hz, and that affects the lighting too. Due to that, there can be irritating flicker in camera. And if using 60Hz frequency in camera, the stutter speed should be 1/100s to avoid the flicker, and that's too short (low light installation environment). With 50Hz frequency, I can have the shutter 1/50 (double).

However, I can assure you, the dropped/duplicate frame issue happens also when everything is set to 60Hz (project, camera, capture device, monitor, projector,...). Interestingly enough, if I adjust the capture device output to 59,94fps, the stutter cycle becomes about 15secs, but that's evidently due to conflicting frame rates.

I tried also setting the capture device output to 30fps, but it does not make it look smoother. Smoothness is the key issue here, right? The stutter is visible also with cam/capture 30fps. When Isadora project frame rate is set to 60, it records the stage in 60fps, not 30fps even if Compression Settings/Motion/Frames per second is set to 30 (this setting seems to affect only captured camera files).

So, to preserve smoothness, I would still prefer using 50fps (or 60fps if I was forced to use 60Hz displays). But again, the stutter is evident whether 30,50 or 60fps.

But I would be interested if anybody can reproduce this stutter issue. Anybody tried? Please find the patch and instructions in the posts above.

Thanks again!

Lauri

barneybroomer

@lauri unfortunately I am stucked in Frankfurt at this moment, in the studio I have Mac Pro bin that I could test, but I use BMI UltraStudio HD Mini's with 1080P as input

Lauri

@barneybroomer Thanks for your reply. If you have a chance after returning from Frankfurt to test, it would be great! I would expect it does not make a difference if it's BM or Magewell.

Fred

@lauri I have used the Blackmagic stuff for years and recently got a magewel device. They are ok, but I did notice a few things, they use more resources than Blackmagic, and although they are lower latency they end up with lower quality colour encoding. The bandwidth use means they need to do more compression for the signal onboard, which means more decompression to get actual pixels.

I have a 4 input capture card from blackmagic on my PC and Isadora can deal with it perfectly well. I have pushed it to 4* 1080p50 signals with high resource use but still a smooth image.

The magewel devices have a hardware configuration tool that you can use to set incoming video formats and even scale the image. I would make sure that the signal you are inputting from the cameras is the same as what you want to capture, otherwise you are performing unnecessary conversion.

As for your issues with Blackmagic and Sony cameras, they are both industry standards (although the consumer cameras you are using have little control over the HDMI output, Sony will adhere to standards for this) You do need to do a bit of work, ie make sure that your cameras are outputting exactly the resolution you are trying to capture at, so if you want 720p make sure your camera is outputting a 720p signal. Although magewel devices will (depending on the driver and software), let you capture cross format (camera outputs format X and you capture at format y) this is going to give you overhead problems. The suggestion you had trouble with the Blackmagic capture cards makes me a think maybe this is the case.

Can you verify the output from your cameras is 720p how you are connecting to them (HDMI or SDI), and how you verify they are outputting a 720p signal over that connection. You can verify this usually by plugging the signal directly into a tv, most TVs will then display the input format for a few seconds, if it is not 720p this is the first problem). As far as I can see the cameras only output 1080 signals so before Isadora gets the signal magewel and Mac os are converting it. This overhead is likely to give you trouble.

Then as far as resource use, the macpro trashcans are pretty lacking in USB 3, the performance is worse than suggested above and as far as I know the resources are shared, so the single PCI e 2.0 lane, also shares resources with the networking gear (I think could not find the document easily). Here is some info https://macperformanceguide.co... in this case using thunderbolt attached capture cards will give you a lot better performance.

I know a lot of people use multiple cameras in isadora and this part of the software is a bit of a no brainer, it either uses the apple built in driver's or Blackmagic drivers and then just shows the frame. It works pretty well for me, but I carefully put together a machine that is designed to handle this kind of thing.

Can you test with another software that can do 2 channel simultaneous capture and record? OBS is free and can do this, check if OBS has any issues with this as it may well be a hardware limitation.

Armando

@lauri said

Sorry to maybe say a stupid thing. But is your Isadora version 64 bits? The 2.6 you are using is not optimized for OS X 10.14. I had trouble in Mojave with the 2.6 n.

Lauri

@Fred Thank you for your reply. And sorry for not responding earlier.

Sony a6300 HDMI out options are ”auto”, 2160p/1080p, 1080p and 1080i. There’s no 720p HDMI out.

So, with Magewell, I’m forced to down-res the signal to 720p. The incoming HDMI signal is 1080p50 and outgoing 720p50 that are both shown in Magewell USB Capture Utility. (So, Magewell accepts the incoming 1080p50, while BM UltraStudio Mini doesn’t.) I can verify the down-ressed signal both from Magewell USB Capture Utility (Capture format: 1280x720, 50fps, YUY2) and Isadora Live Capture Settings (Video Input: Resoluton Native, Format 1280x720).

Yes, it’s true that 2x1080p50 is too much for USB lane.

(With BM UltraStudio down-ressing is not possible, it does not accept 720p setting, black screen. And, as told before, BM UltraStudio Mini does not accept 1080p from Sony HDMI. BM UltraStudio Mini accepts only 1080i50 (or i60) from Sony HDMI, keeps it as that or converts to 1080PsF25 (or PsF30), but you will see the interlacing on both. So, there’s no real 1080p25 or 1080p30.)

However, as mentioned before, with Magewell down-ressed signals, Isadora captures both simultaneous camera signals flawless, no dropped/duplicate frames. So, if it can do that, why it cannot draw the streams properly? Even when only streaming the live cams on stage, no recording involved.

--------

When using BM UltraStudio Minis for capturing live cams (1080i50 or 1080PsF25), there’s also some sort of stuttering that happens cyclically. It looks different, but that’s probably because the framerate is not 50fps.

(P.S. Interestingly, when using BM UltraStudio Minis and cam HDMI out 1080i50 (the only setting that works with BM UltraStudio Minis), I can get cam record to file only if using Apple driver. If using Blackmagic driver, the recording starts but there are no files.)

--------

When replacing the Video In Watcher actors with Movie Player actors and playing the files through the Video Mixer actor, the result is very stuttery. This should not be heavy task at all. The live cam recorded files are Photo-JPEG (720p50, of course), no dropped/duplicate frames. VLC shows Codec details: Motion JPEG video, Planar 4:2:2 YUV full scale, Color Space ITU-R BT.601 Range.

When playing the recorded live cam file outside Isadora, I get strange results (I have checked that the file does not have any dropped/duplicate frames):
- playing the file with QuickTimeX, the file stutters a lot, there’s some kind of cycle that the file doesn’t stutter, but mostly stuttering - playing the file with QuickTime7, the file stutters in cycles, but has longer periods when not stuttering and the cycles are more regular
- when playing the file with VLC, the file doesn’t stutter at all, except that there’s one dropped frame every 72secs
This brings me back to think about a sync issue.

----------

Also, usually when starting the patch (Q4, only playing the superimposed live cam streams, no recording), there’s no stutter for about 2mins and then the stutter starts first in one cam – and then the stutter changes to the other cam and stuttering alternately in a cycle of about 7mins. So, why would it play well for a while and then start stutter later, if the problem is the ”cross format processing”.

----------

I downloaded OBS but didn't really get it work in a similar way to record (or just show) two superimposed live cam feeds simultaneously.

Lauri

@Armando

No, I use Isadora 2.6.2b3 (or also 2.6.1), so it's not 64bit. However, as stated before, I made the tests both with MacPro OSX10.12.6 and MBP OSX10.14.4. I know Isadora 2.6.x is not compatible with OSX10.14.x, but combination works with no problems in my MBP(2018). So, in this context, forget about the MBP/Mojave, the issue is prevalent in MacPro/Sierra.

Fred

@lauri well the mini recorder is only for 1.5 GHz signals, clearly stated in all the specs, on the website and instructions and even in the setup utility. 1080p50 is 3ghz, so if course it will "not work", it adheres to all specs but you are trying to get it to do something it does not, there are other products from black magic for capturing 1080p 50, best to do the proper research before you purchase. The mini recorder is functioning as it should and has no "problems" with the sony output, it is not designed to capture that format and never was.

So even though you are capturing at "720p", your input is 1080p to the magewel and you are using that bandwidth on your usb3 bus so that for sure is part of your problem .There is no magical conversion, your capture device is running at 1080p50 and then running a conversion on every frame, if you were not trying to record and further push the limited resources you have I would suggest just running at 1080p50.

The artefacts you see from the 1080i and the 1080psf25 capture are more to do with the sony output from this consumer camera than any of your capture devices. If the camera outputs a proper deinterlaced signal it will look fine, however many of these cheaper souped up DSLRs cannot really do on sensor 1080i. Instead it is converted afterwards.

As your cameras do not output 720p you should really run at 1080p, but your system for USB based capture is under spec (as it is for this setup @ 720p, as you are still capturing at 1080p on the magewel and those frames are uploaded to the CPU for processing and then delivered to Isadora at 720p).

Running a whole system at 1080p is not that easy (which even though you bump down to 720p after capturing, you are actually capturing at 1080p). The Gear you are using is going to be pushed trying to do this, in general apple don't make anything ready for this, no good gfx cards and getting a system equipped with proper 1080p capture is very expensive.

OBS is simple to use and can preview and record 2 streams, I think it is worth reading the manual and checking your system can do what you want it to before looking for faults in isadora, and as well you are running an unsupported system. I know this sucks, Isadora is not ready for Mojave yet and every new computer can run Mojave.

As media artists using this kind of gear with high expectations there is a lot of research involved, we are not running some simple software and it takes good planning and piece by piece testing to make things rub smoothly and reliabily.

Lauri

@Fred Thank you for your reply!

First, I'm not running on Mojave! I've said that many times in my posts. I've tried the patch in my Mojave-laptop, BUT I'm running the tests in my MacPro 10.12.6.

Ok, I understand that I'm at/over the limits of the MacPro. And I do understand there's a lot research involved. I would appreciate all the help in finding a solution. It feels just a bit frustrating, I do not know where to head forward – if it's the case for only needing to change the computer, operational system, cameras and capture devices.

Fred

@lauri Ok, sorry, I saw you had removed the mojave system from the setup for now.

Lets try to make a few tests to see what the system can handle, sorry if some of these are repeats of what you have already done, or what you have mentioned, but we can try and thorough.

First a few questions about your setup, how clean is your install? Do you have a lot of other software running (especially adobe)? Was the current os 10.12.16 a clean install or upgrade from previous OS?

For testing lets go with 2 screens, one for the interface and one for the output, set them up to both be 1080p50.

For the following tests leave the system running a while and take some notes while running activity monitor of what Isadora and quicktime are using resource wise (so CPU and ram). Whenever you report with activity monitor also sort by CPU usage and RAM usage and take note if there is anything using more resources than the app we are investigating.

The following tests should be run for 15 minutes or so each, if you see any of the skipping you mention please record it with a phone so we can see it without it causing extra load on the system.

1. Use the utility to reset all the settings on your magewel devices, plug in just one and capture at 1080p50 making sure the output of the camera is 1080p50 (by connecting to a screen briefly) as well and the system is not doing any conversion. Use Isadora with a blank patch and just a video in watcher attached to a projector showing the image full screen on the second monitor. Report what you see on activity monitor for ram and CPU and what you see visually, as well as your Isadora target frame rate and also take note of the cycles.

2. Use the same test but grab at 720p and report what you see on activity monitor for ram and CPU and what you see visually, as well as your Isadora target frame rate and also take note of the cycles.

3. Close isadora and open quicktimeX go to new movie recording and select the magewel capture device and capture at 1080p50 and report what you see on activity monitor for ram and CPU and what you see visually

4. Open quicktimeX go to new movie recording and select the magewel capture device and capture at 720p50 and report what you see on activity monitor for ram and CPU and what you see visually

5. Return to Isadora, use the magewel utility to reset all hardware settings to default for the second capture device, make a new isadora patch with 2 video in watchers set to grab at 1080p50 and 2 projectors and use the mapper to show the inputs side by side. Report what you see on activity monitor for ram and CPU and what you see visually, as well as your Isadora target frame rate and also take note of the cycles.

6. Perform the above test at 720p50. Report what you see on activity monitor for ram and CPU and what you see visually, as well as your Isadora target frame rate and also take note of the cycles.

7. Go to your applications folder and duplicate the quicktime Player application so there is 2 copies. Open one at a time and select new movie recording and choose a magewel device, repeat for the second copy of the application and choose the other device. Report what you see on activity monitor for ram and CPU and what you see visually with the previews running one on each screen, you should be able to select different resolutions so try this at 2x720p50 and 2x 1080p50

7b. Try to record from both of these at the same time, trying for both 1080p50 and 720p50

8. Please run blackmagic disk speed test (it comes when you install the blackmagic driver) on the drive you are using as your target to record video from the stage to and take a screen shot and post it.

8b. Please run blackmagic disk speed test (it comes when you install the blackmagic driver) on the drive you are using as your target to record video from the stage while you have Isadora open and showing one live camera full screen, try this at 1080p50 and 720p50 to and take a screen shot and post it.

9. Make a stage recording test, make a new Isadora patch and with some moving text using text draw or some other moving shapes and make it full screen on the second monitor (running 1080p50), report what you see on activity monitor for ram and CPU and disk usage. Try with proress, h264 and photojpeg.

These are the tests I would do to find the bottleneck. Ideally it would be good to do all this with a clean install and no other software running.

If you have the time to do this and let us know the results we can try to see what is getting you bogged down and if there is a way to tune your system to get what you want out of it.

As a last possibility, I am curious if you switch all your system over to 1080i50, all cameras, all capturing and all screens (some screens will not do this, so if possible try find ones that do) I am curious what results you get. I did some checking and the 1080i output of your cameras should be OK, I am guessing you see interlacing as maybe the field order is reversed (I think you can change this in the magewel settings) or because when you switched over your output screens were still progressive, they should also be 1080i for this to work. This is a less desirable pathway for you I know but it would mean you could stop the system doing a real time format conversion from 1080p50 to 720p50 and keep a low data rate that your USB bus should be able to keep up with.

Getting more out of the computer is going to be tough, it has no PCIe expansion and only thunderbolt 2, so your choice of capture cards is pretty limited, even via expansion chassis via thunderbolt.

Have a look at performing these tests and reporting the results, I cannot guarantee that it will reveal the problem. I will re-iterate that there are definitely people that can do multiple channels of capture without issue.

If you can get everything working without the stage recording maybe there are other ways to get the job done. I dont really trust stage recording and generally hand this off to a hyperdeck hard disk recorder that i can remote control and feed back through a video mixer/router. It is for me a much more peaceful workflow. You would be surprised how much work recording a video is for a computer.

Lauri

@Fred

Thank you for your support and kindly providing steps for tests!

Unfortunately, I’m travelling to France tomorrow and returning in the beginning of May, and cannot go back to my studio before that. So, I need to postpone the testing, my apologies for any inconvenience.

A bit about my MacPro system, I have all my apps in the machine, including Adobe, but Creative Cloud app not running and auto launch always dis-activated. And always when using Isadora, I quit/force quit all adobe processes from Activity Monitor, and other apps are not running.
I believe Sierra was not the original OS installed and was installed over the previous OS. As you recommend, I suppose, I will do a clean install of either 10.12 or 10.13, install Isadora and capture drivers to a clean machine, and make the tests – although in the beginnig of May. Sorry for the delay, hope it's ok.

Thanks again!

Lauri

DusX

I have been trying to follow this thread. Unfortunately I do not have the hardware to do a fair test myself.
If Fred is correct in how the Magwell capture works (sends video out the Computers CPU for scaling) then yes, it seems 2 streams is likely at least a peice of the problem. This could be solved by using hardware scalers, before capture. If the feeds are scaled to 720 before reaching the capture device, that will open up your bandwidth.

Lauri

@Fred Thousand thanks again for your test instructions! And my deepest apologies for this long long delay in replying – my schedule has gone up and down during these almost 2 months, truly sorry! I hope you would still be willing to go through my test results and give me feedback, if possible. And if needed, propose some further tests to be done.

There are some things I find a bit peculiar, for instance that the cycles usually stay up for a long time and suddenly start to sink and rise back again for long time. But maybe there's a simple explanation... Anyway, the test results became a bit extensive in length, so I thought to put them into a pdf, instead of posting it straight here.

IsadoraTests.pdf

Please, anybody, have a look on these.

@DusX Thank you for your comment too. It would be nice if a hardware scaler would be a solution. But again, while I used 2 Magewells, there was no dropped frames in the captured video files, only on the stage, and even on the stage the channels didn't mostly stutter simultaneously...

Any advice welcome, of course.

Thanks again in advance,

Lauri

P.S. also eagerly waiting for the new version!