GPU vs CPU how do they divide the work?

The basic components of an NLE are:

1. Media files stored on an external HD
-Media access speed is determined by: HD speed, Raid level and connection speed to workstation.
2. CPU> decode, encode and calculate non-accelerated effects.
3. GPU>accelerates:
Alpha Adjust, Basic 3D, Black & White, Brightness & Contrast
Color Balance (RGB), Color Pass (Windows only)
Color Replace, Crop Drop Shadow, Edge Feather, Eight&Four-Point Garbage Matte
Extract, Fast Color Corrector, Gamma Correction, Garbage Matte (4, 8, 16)
Gaussian Blur, Horizontal Flip, Levels, Luma Corrector, Luma Curve, Noise
Proc Amp, RGB Curves, RGB Color Corrector, Sharpen
Sixteen-Point Garbage Matte, Three-way Color Corrector, Timecode
Tint, Track Matte Key, Ultra Keyer, Video Limiter, Vertical Flip
Cross Dissolve, Dip to Black, Dip to White, Directional Blur, Fast Blur, Invert, Additive Dissolve, Film Dissolve, Warp Stabilizer

7 steps to render/playback a frame:
1. Frame is fetched from disk.
2. CPU decodes the frame (CODEC has to be properly installed)
3. Frame is modified to an intermediate format and put into memory.
4. If GPU acceleration can be used, the intermediate format is uploaded to the video card for processing.
5. The intermediate results are then downloaded from the video card back into memory.
6. The CPU starts encoding to the final delivery format.
7. The final results are written to disk or played on screen.

A video file gets decoded on the CPU. Performance is determined by how well it is optimized and if it can offload tasks to GPU or not. A lot of codecs on PC call the QT32 process which runs in a 4GB memory space.
ProRes playback is not GPU accelerated. More cores are more important than multiple GPUs, as not all instructions have been written to run outside
the CPU. It’s a common misconception that adding a GPU will
automatically take and run all the regular instructions from
the CPU. Software engineers must write applications to
send instructions to a specific processor before they can
expect speed improvements.
A typical system with eight cores needs 3 GB per core, which equals 24 GB of RAM required by After Effects to run multiprocessing properly.
Although that fits into a 32 GB computer, After Effects may still hold as
much as 6 GB of RAM for other applications, bringing the full
amount of RAM left for After Effects to 24, which is already

close to the limit.

This applies to Windows/PC only:
Below Playback the way it should be: Low CPU load and Disk I/O at the bit rate of the codec. Note the files on the timeline being read in the Disk I/O list.
1AVID_AVID-MXF_PLAY_NO_FILTER
Now look what happens when playing back a ProRes QuickTime in Premiere on a PC
1. CPU cycles taken up by Premiere
2. CPU cycles taken up by the QT32 process
3-8 The files on disk being read by the QT32 process including:
9 pagefile.sys the Page File it has to keep writing and reading off the drive to compensate for the 4GB memory limit.
PR_PRO_PLAY_NO_FILTER_pagefile

Performance Grid: Codecs, Platforms, Wrappers, Quicktime, MXF

I had an aha-moment recently after switching from a QuickTime based project on Premiere, to C300>native MXF. The system was all of a sudden snappy, no stalls, a lot more realtime performance and super fast renders and exports.
My system must have been struggling with something…
A codec that works with low CPU load on a Mac might have a high CPU load on PC.‬ On Mac QuickTime uses the 64-bit AV Foundations framework but on PC Premiere still uses the deprecated 32-bit QTkit framework. Adobe finally wrote an unofficial white paper March 2016 about performance. ProRes is not supported for export on Windows.
I try to remove QuickTime as much as possible from my PC/Windows workflows. But don’t take my word for it, do your own performance tests!
I think MXF combined with PC friendly codec is the authoring format of the future.
Please chime in using the Google form below.
mi-smile
Mike

My thoughts:
Below is my (imperfect, incomplete) performance grid that shows what platform and software I try to use with each codec. I usually work on a standalone system. On any kind of collaborative project I recommend transcoding to native Avid media and using their ISIS storage. There is still no match for Avid’s media management especially when collaborating in realtime on a deadline.

QuickTime MOV most codecs ProRes

Arri Alexa

RED DSLR AVCHD H264       H265 in MP4 container. Like GH4 Collaborate with Native Avid Media DNxHD DNxHR Native Sony XDCAM-EX XDCAM-HD XDCAM-HD422 Blackmagic Canon Native C500 C300 C100 5DM2 5DM3 7D
ICO_apple ICO_premiere ICO_resolveICO_fcpx ICO_apple ICO_premiere ICO_resolveICO_fcpx ICO_appleICO_winICO_fcpx ICO_resolve ICO_premiere ICO_win ICO_resolve ICO_premiere ICO_winICO_avid ICO_win ICO_resolve ICO_premiere ICO_appleICO_fcpx ICO_resolve ICO_premiere ICO_appleICO_winICO_fcpx ICO_resolve ICO_premiere

Most cameras record in non-Quicktime formats like MP4 or MXF, some are more PC friendly, others work better on Mac. The Alexa can record in two compression types/file formats: either you can record as a ProRes compressed Quicktime *.mov or as an MXF-wrapped DNxHD compressed file. Hopefully, the next generation of external recorders will follow suit. One MXF recorder already exists: A laptop with BM-mini Recorder running Avid Or Premiere!

Apple research:
-I love ProRes on Mac, I think it’s a great looking, very easy too use codec.
It works in 64 bit. If a production uses ProRes, I fight for editing on Mac.
-Most people seem to agree that Macs are build for Apple’s own software, not the competition’s.
QuickTime is deprecated? What does that mean in practice?
What is Apple doing with QuickTime?
-The big loser on the Mac platform is Premiere. Adobe has made strides in improving the performance of their applications, but it still doesn’t compare to what Apple has achieved with FCPX.
There is an incompatibility between Premiere and Open-GL and AMD.
-The winner is FCPX. Software and hardware are tuned to each other for maximum performance. It takes advantage of modern hardware, as it’s 64-bit (uses all your RAM), OpenCL GPU-accelerated (uses your graphics card for playback, render and export), and supports Grand Central Dispatch (uses all of your CPUs and cores).

PC/Windows research:
When I work with QuickTime and most codecs on PC, I’m constantly experiencing mini crashes (white screen lockup) and slow export times. I’m forced to restart every hour because pagefile.sys and QT32 processes running in the background bog the system down. I think the white screen lockup is the pagefile.sys being written or read by Windows from the slow internal HD! (takes about 30sec)
There is no 64-bit version of QuickTime, primarily because Apple has ceased development of QuickTime on Windows (except for security and minor updates). QuickTime has been 7.x for 10 years now, and 8.0 does not seem to be anywhere in the distance. Avid, Adobe and Resolve work great with non-Quicktime media or Avid’s own media (MXF).
The QT-32 process on PC runs in a 4 GB memory space, even when lots of memory is installed. To keep up with playback, the system has to keep swapping banks of memory (pagefile.sys) to the hard-drive instead of using all installed memory causing severe performance degradation. The other drawback is the very limited threading capability, causing a far from efficient use of all the logical cores in the system.
MXF is a much better container format, as it is just a container, unlike QuickTime, which is a full multimedia framework.
MXF requires no installation (and hence never needs updating); doesn’t launch a process when used, so 32- and 64-bit matter not; and it has no bugs or odd quirks – it just holds the video, audio and metadata.
-I avoid Quicktime on PC. Any codec! > I use native camera media, MXF, MP4 or native Avid Media Files (which are MXF).
-I don’t use ProRes on PC, with any of the NLEs, even though they say it works. It is just too much trouble, glitchy and slow.
-ProRes inside MXF on Avid is a joke. Their glossy brochures talk about compatibility, but I can’t make it work. I only use AMA to transcode to Avid media, not for editing.
-In Premiere> QuickTime for Windows cannot export H.264 on computers that have more than 16 CPU cores due to a problem in the Apple H.264 compressor component.

 

Multi-Cam:
-If possible I do Multi-Cam on Avid with transcoded DNxHD36 media if more than 4 angles for the offline. For online relink to native media.
-Next best is Premiere with the same DNxHD36 media if more than 4 angles and native media with <4 angles.

Delivery:
You can edit without QuickTime but deliver in Quicktime. When a client asks for QuickTime & certain codec, I give them what they want. This means keeping a Mac around somehow for ProRes delivery.
Saving MXF clips to MXF container in Premiere
AVID – Fast Import Apple ProRes .mov to AVID Pro Res .mxf Workflow
Arri Alexa bypassing Quicktime using MXF instead

Interoperability with DAM systems:
Dalet supports import/ingest/export in MXF
-you have to test and update your workflow company wide to make use of this.
Delivery of MXF with Closed captioning

Uninstall QuickTime to avoid all kinds of conflicts with your NLE.
It forces your NLE to use their workarounds for play back Quicktime codecs.
Don’t worry, you can always put it back, the NLE usually has it’s own process to playback QuickTime sources. Premiere works fine on Windows without QuickTime installed.
It is a great trouble shooting step to try to improve the realtime capabilities of your system.

Uninstall QuickTime on Mac:
mac-qt-library

1. Internal HD>Library>Quicktime
2. Remove all or just the components you are trouble shooting, out of this folder.
3. restart apps using QuickTime.

Uninstall Quicktime on PC/Windows:
1. Control Panel>Add/Remove Software
2. Restart Computer.
3. click here to download latest version from Apple.

Resources:
Security Alert uninstall Quicktime
-Avid’s take on Quicktime
Do your own performance tests.
How do I speed up rendering, exporting, or encoding?
PREMIERE PRO AND MULTIPLE GPUS
-Adobe’s smart rendering workflow
Good article about how video codecs work
QuickTime is deprecated? What does that mean in practice?
What is Apple doing with QuickTime?
-Adobe says it buried on a support page: ProRes is not supported on Windows!
A Guide to Common Video Formats: Containers, Compression, and Codecs
Build a balanced system
GPU vs CPU how is the work shared?
Free Resolve-12: Great basic editor, fantastic color corrector > the standard for most TV & Film. Also has one feature no other NLE’S have: Export files from timeline as individual files!
Free Red CINE-X: Transcodes any Red file into an editable format: even Quicktime ProRes on Mac and PC!

Performance test: Codecs and QuickTime on PC.

I had an aha-moment recently after switching from a QuickTime based project on Premiere, to C300>native MXF. The system was all of a sudden snappy, no stalls, a lot more realtime performance and super fast renders and exports.
My system must have been struggling with something…
A codec that works with low CPU load on a Mac might have a high CPU load on PC.‬ On Mac QuickTime uses the 64-bit AV Foundations framework but on PC Premiere still uses the deprecated 32-bit QTkit framework. Adobe finally wrote an unofficial white paper March 2016 about performance. ProRes is not supported for export on Windows.
QuickTime on Mac works in 64-bit (use all your RAM), OpenCL GPU-accelerated (use your graphics card for playback, render and export), and support Grand Central Dispatch (use all of your CPUs and cores).
With QuickTime and most codecs on PC, I’m experiencing “mini crashes” > white screen lock ups, plus stuttery, slow performance and slow renders & exports. Why?
The QT-32 process on PC runs in a 4 GB memory space, even when lots of memory is installed. To keep up with playback, the system has to keep swapping banks of memory (pagefile.sys) to the hard-drive instead of using all installed memory. I think the white screen lock up is the pagefile.sys being written or read by Windows from the slow internal HD! (freezes about 30sec)
There is no 64-bit version of QuickTime, primarily because Apple has ceased development of QuickTime on Windows (except for security and minor updates). QuickTime has been 7.x for 10 years now, and 8.0 does not seem to be anywhere in the distance.
I urge you to spend one hour to do your own tests, on your system with your own media. Figure out what formats and codecs give you the best realtime performance. Each of the NLE’s has it’s own workarounds to bypass QuickTime depending on codec used. Save yourself and your clients thousands of hours not waiting anymore for renders, exports and restarts.
mi-smileMike

Resources
Security Alert uninstall QuickTime on PC.
-Avid’s advice about Quicktime
QuickTime is deprecated? What does that mean in practice?
What is Apple doing with QuickTime?
How do I speed up rendering, exporting, or encoding?
CPU vs GPU
PREMIERE PRO AND MULTIPLE GPUS
-Adobe’s smart rendering workflow
A Guide to Common Video Formats: Containers, Compression, and Codecs
-In Premiere> QuickTime for Windows cannot export H.264 on computers that have more than 16 CPU cores due to a problem in the Apple H.264 compressor component.

Delivery in MXF
You can edit without QuickTime but deliver in Quicktime. When a client asks for QuickTime & certain codec, I give them what they want. This means keeping a Mac around somehow for ProRes delivery.
Saving MXF clips to MXF container in Premiere
AVID – Fast Import Apple ProRes .mov to AVID Pro Res .mxf Workflow
Arri Alexa bypassing Quicktime using MXF instead
About smart rendering in Premiere

Here is what to do
1. Set yourself up for testing> press Ctrl+Alt+Delete and start Task manager> Resource Monitor >Overview tab.
Make visible
-CPU
-Disk
2. Observe CPU Usage & Disk I/O with different codecs.
With an optimized codec like AVID, Sony EX422 or Cineform your CPU load should be low 5-10% and your Disk I/O should match whatever the bit rate is of the codec. My conclusion is that the QT32 process doubles CPU load during playback, render and export. An Intra-frame, non-QuickTime codec plays at 5-10% CPU, as opposed to 50-60% CPU as QuickTime, while also adding to Disk I/O. It means that a timeline with filters will take twice as long to export and playback always stutters.
The most astonishing thing is that a Premiere timeline with an QuickTime codec, rendered green, will still stutter playing back on both Mac and PC! I think that is embarrassing and unacceptable!
The culprit is the  QT32 process multiplying itself for no reason, taking up more and more CPU cycles. As your project grows and your timeline becomes more complicated this keeps dragging your realtime capabilities down until it literally stops… Restart!

Below Playback the way it should be: Low CPU load and Disk I/O at the bit rate of the codec. Note the files on the timeline being read in the Disk I/O list.
1AVID_AVID-MXF_PLAY_NO_FILTER
Now look what happens when playing back a ProRes QuickTime in Premiere.
1. CPU cycles taken up by Premiere
2. CPU cycles taken up by the QT32 process
3-8. The files on disk being read by the QT32 process including:
9. pagefile.sys the Page File it has to keep writing and reading off the drive to compensate for the 4GB memory limit. >Causes “white screen lock up” (about 30 sec.)
PR_PRO_PLAY_NO_FILTER_pagefile