3 Takeaways From Moscow State University's 2019 Codec Comparison
From my perspective, one of the most useful and credible codec analyses comes from Moscow State University, which just published its 2019 Video Codec Comparison. This essential resource includes multiple HEVC codecs, plus VP9 and AV1. As always, you’ll have to pay to get all the data ($760 through 11/30/2109, $950 thereafter), but the free version provides useful data points. There are two versions of the report; the Main report which tested 12 codecs with 100 videos using objective metrics, and the Subjective report, which tested 5 clips and 11 encoders subjectively rated by 732 viewers.
The Main report tests three use cases—fast encoding (60 fps), universal encoding (25 fps), and ripping encoding (1 fps)—and if the codec doesn’t meet that minimum performance threshold it’s not considered for that test. The Subjective report only considered codecs that could produce 1 fps. In January 2020, MSU plans to release a report considering high-quality codecs that can produce a frame every 2 minutes.
Here are three takeaways from the 2019 MSU report.
x.265 May Not Be Your Best Option
x265 is free with FFmpeg so it's the most accessible HEVC codec, but it has come up short in multiple MSU reports and this one is no exception. By way of explanation, all three charts below use x264 performance as the benchmark, which is why it’s always at 100%. Lower numbers indicate the ability to produce the same quality as x264 at the percentage shown, so the highest-performing HEVC codec in Figure 1, HW265, can deliver the same quality as x264 at 57% the data rate. x265 requires 76%, which is 33% higher than HW265. So, if you’re considering encoding with HEVC, it’s worth exploring the other higher-performing HEVC codecs shown in the report.
Figure 1. Objective performance as measured by SSIM.
AV1 Looks Better in Real-World Testing
Like HEVC, AV1 is a standard so there will be multiple codec versions. The WZAurora AV1 Encoder (from Visionular) tested by MSU not only met the 1 fps speed requirements but also produced dazzling quality (as shown in Figure 2), outperforming the best HEVC encoder by about 28%.
Figure 2. Subjective performance of multiple codecs as measured by Subjectify.us.
But, There’s One Significant Caveat
The sharp-eyed reader will have noted WZ Aurora in the middle of the pack in Figure 1 but at the front in Figure 2. What’s the difference? Check out Figure 3, which shows performance as measured by subjective testing (in red) and with SSIM (in blue). In all relevant cases save the WZAurora AV1 Encoder the results are fairly consistent. With WZAurora, the differential was nearly 50%.
Figure 3. Objective vs. subjective ratings for the tested codecs.
I asked MSU group leader Dr. Dmitriy Vatolin about this scoring discrepancy. He responded, “Currently, we can’t give the exact answer, but we can guess some possible reasons. For example, it can be contrast or brightness increase of encoded stream, which looks better for the observers. We did such a study when we analyzed the applicability of the VMAF metric to objective comparisons (see our paper 'Hacking VMAF With Video Color and Contrast Distortion.'"
I’m hoping to test the WZAurora codec in early 2020, and this is an issue I can look for. In the meantime, the MSU report provides lots of valuable data about the real-world performance of multiple HEVC codecs as well as AV1 and VP9.
Related Articles
The head of the Moscow State University Graphics and Media Lab—the people behind VQMT and Subjectify.us—offers his insights into objective and subjective metrics, as well as VMAF hacking and AV1.
13 Dec 2021
The good news: As always, Moscow State's codec studies are some of the most comprehensive available. The bad news: Unless you're TikTok or Tencent, you won't have access to some of the best performers.
11 Jun 2021
If you're serious about experimenting with different codecs and/or encoding parameters, MSU's Video Quality Measurement Tool is an essential tool, and version 13 brings some welcome improvements.
01 Jun 2021
Moscow State University's Video Quality Measurement Tool was already good. Enhancements in the new version, including new metrics and the ability to run multiple analyses simultaneously, make it even better.
23 Mar 2018
Latest codec quality comparison study finds AV1 tops on quality, but far behind on speed, and that VP9 beats out HEVC. Moscow State also launched Subjectify.us, a new service for subjectively comparing video quality.
30 Jan 2018
In addition to comparing HEVC codecs, report also compares HEVC to VP9 and x264, with potentially controversial results
09 Nov 2015
Never heard of it? Learn why we call this video encoding analysis tool invaluable, and a must-have for anyone serious about encoding or compression.
30 Dec 2014