August 16, 2019
By Jan Ozer Contributing Editor
Featured Articles

HEVC, AV1, VVC: How to Make Sense of 2019's World of Codecs

On March 27, 2019, Luxembourg-based Sisvel announced two new patent pools offering licenses on patents “relevant to the VP9 and AV1 specifications.” The pools apply solely to consumer display devices, like smartphones, computers, and TVs, and consumer non-display devices, like set-top boxes, dongles, and graphics cards. The standard rates for VP9 are €0.24 for display devices and €0.08 for non-display devices, while the rates for AV1 are €0.32 and €0.11, respectively. (The rates in U.S. cents are similar.)

The pools represent IP from JVCKENWOOD Corp., Koninklijke Philips, Nippon Telegraph and Telephone Corp., Orange, and Toshiba IPR Solutions. According to a Q&A with the Sisvel CEO, the Sisvel pools won’t charge for content, but haven’t ruled out charges for software-based playback. (Author’s note: As detailed at Streaming Learning Center, the author is providing some editorial consulting services to Sisvel.)

When AOMedia announced the release of AV1, it also announced the formation of a legal defense fund for any patent-related litigation, so the Sisvel pool is hardly surprising. No one has sued anyone yet, so the defense fund hasn’t kicked in. The next major development will be when Sisvel files a patent list, which should occur by late summer or early fall 2019. Once the list is filed, other IP professionals can start to review both the veracity of the patents and whether they apply to AVI and VP9. Until then, the only real effect of the Sisvel announcement is that all potential users are on notice that AV1 may not be royalty-free.

AV1 BROWSER SUPPORT

Since Google and Mozilla are both AOMedia members, it’s not surprising that Chrome and Firefox support AV1 playback. Throw in support from the Opera browser, and AV1 now plays in 35.28% of all browsers, according to Can I use (Figure 3), with more coming soon. That is, in March, Google announced that Android Q, now available in beta, “introduces support for the open source video codec AV1. This allows media providers to stream high quality video content to Android devices using less bandwidth.” This seems to indicate that support will be playback-only, not for recording.

Figure 3. As of May 2019, AV1 played in Chrome, Firefox, and Opera, accounting for 35.28% of all browsers.

In November 2018, Microsoft released the beta AV1 Video Extension, which allows AV1 playback on Windows 10 devices. No word on when this will be added to a Windows 10 update for broader distribution or if or when AOMedia member Apple will add AV1 to its computers or devices.

Hardware support for AV1 encode/decode wasn’t scheduled to appear until mid-2020, about 2 years after the bitstream freeze, and it appears to be on track. Note that browser-based playback has proven to be more efficient than many pundits predicted, which bodes well for AV1 adoption for browser-based playback. In October 2018, StreamingMedia.com reported that 1080p playback of an AV1-encoded YouTube video on Chrome consumed about 20% of CPU on an HP ZBook notebook powered by a 2.8 GHz Intel Xeon E3-1505M v5 CPU. A few days later, I reported on my blog that playback of the same video on the same notebook on Firefox consumed about 10% to 15% of CPU resources (Figure 4).

Figure 4. AV1 playback on an HP ZBook powered by a 2.8 GHz Intel Xeon E3-1505M v5 CPU

On May 23, 2019, Mozilla announced the integration of the dav1d decoder into Firefox, which should drop the required CPU resources even further. Mozilla also reported that 11.8% of video playback in Firefox Beta used AV1, up from 3% in March and 0.85% in February. Much of this traffic likely comes from YouTube, which posted an AV1 playlist in September 2018. In addition, Google software engineer Steven Robertson promised at Demuxed 2018 that YouTube would distribute 1TB per second of AV1-encoded video by the end of October 2018. Netflix also released some AV1-encoded video for public consumption in September 2018.

As we reported in 2018, however, Netflix encoded its 1080p video to 6.7Mbps, while YouTube was at 5.1Mbps, so neither company was pushing the quality envelope. Although Facebook claims that AV1 saves 51% when compared to x264, and 32.5% when compared to VP9, the company hasn’t publicly stated that it is shipping AV1-encoded video. At this point, a year after the bitstream freeze, we don’t know if any company is achieving the bitrate savings reported by Facebook at any kind of scale.

We do know that encoding times have dropped dramatically. When StreamingMedia.com first looked at AV1 encoding back in August 2018, encoding times were 45,216 times longer than real time. This dropped to 147 times longer than real time in tests from March 2019. Just before the 2019 NAB Show, Intel and Netflix announced SVT-AV1, which they claim is capable of real-time 4K/60p 10-bit encoding when running on Intel Xeon Scalable processors and Intel Xeon D processors (Figure 5).

Figure 5. SVT-AV1 is reportedly capable of real-time 4K/60p 10-bit encoding on certain Intel CPUs.

What’s the quality hit? At this point, it’s unknown, but the qualitative difference between SVT-AV1 and the AV1 version shipped by AOMedia should diminish over time. Here’s why.

According to the BBC, AV1 was built with Google’s VP9 specification as a base. To this, different “tools” are proposed by various AOMedia members for inclusion in the codec. Necessarily, at least at first, the successful tools are bolted into the codec architecture as opposed to integrating for maximum encoding efficiency. This is why AV1’s encoding times skyrocketed during the codec development process.

Intel’s SVT was designed to enable “the encoder core to be split into independently operating threads, each thread processing a different segment of the input picture, that run in parallel on different processor cores, without introducing any loss in fidelity.” This speeds encoding on multiple-core CPUs like Intel Xeons. However, the porting of AV1 to the SVT architecture starts with the core functions and adds different tools over time. For this reason, version 1 of any SVT codec won’t produce the same quality as a mature version of the same codec, but should catch up over time.

So, over the last 12 months, AV1 has demonstrated encoding and playback efficiency and substantial browser support, but also the prospect of royalties on the horizon.

VVC

VVC is the next-generation (after HEVC) standards-based codec developed jointly by MPEG and the ITU. The quality target is 30% bitrate reduction over HEVC at the same perpetual quality. Although the codec isn’t scheduled to ship until the end of 2020, some quality comparisons have already started to arrive.

The findings from the aforementioned BBC study are shown in Figure 6, with JEM standing for the VVC Joint Exploration Model and HM the reference HEVC model. As described in the 2018 International Broadcasting Convention paper titled, “An Overview of Recent Video Coding Developments in MPEG and AOMedia,” the study included both objective and subjective tests.

Figure 6. The BBC found HEVC (HM) and AV1 to be about the same, with VVC (JEM) substantially better than both.

At HD resolutions, the BBC found AV1 to be only 7% more efficient than HEVC, with VVC 33% more efficient than HEVC. At UltraHD resolutions, AV1 was only 2% better than HEVC, while VVC was 27% better. For those who care about such things, the BBC is both a member of AOMedia and a licensor member of the MPEG LA HEVC patent pool.

When reconciling these findings with Facebook’s, note that the BBC used reference encoders for its VVC and HEVC testing, which are encoders that use every tool in the codec. As such, encoding times are too lengthy for commercial use. In addition, the BBC used standard test clips that promote comparability among different test generations, in essence, testing encoders no one will ever use for production encoding with clips no consumer will ever watch. In contrast, Facebook analyzed 100 of its most popular test clips using actual production encoders, which makes it a completely different test.

On the licensing front, one would assume that contributors to VVC include many of the same companies in the HEVC and H.264 pools. To help avoid the issues experienced with HEVC, a group of companies formed the Media Coding Industry Forum (MC-IF). As stated in a press release, “MC-IF will initially focus on establishing VVC and related standards as well-accepted and widely-used standards, for the benefit of consumers and industry.” One key goal is to “Provide a forum for, and encourage, the discussion of issues related to the licensing of intellectual property rights relevant to the deployment and use of these Standards, in the furtherance of the Purpose.” While laudable, MC-IF has no teeth, so the impact of the organization remains to be seen.

A Future Decided by ... Patent Attorneys?

To a great degree, the codec future will be dictated by AV1’s royalty-free status, which AOMedia could secure by proving that member companies truly reinvented the wheel with AV1 or by licensing technology as needed. Remember that Google used this latter tactic to cease the formation of a VP8 patent pool by MPEG LA back in 2013. Or perhaps AOMedia can garner enough control over AV1’s IP to distribute AV1 with a fair and reasonable royalty.

If not, it’s likely that many of the same IP owners behind H.264, HEVC, and VVC own technology used by AV1. This suggests a truly dystopian future in which IP owners attempt to impose HEVC-like business terms on AV1, VVC, and future codecs. You’d like to think they learned their lesson, but 6 years, 4 months, and 4 days after the release of HEVC, they haven’t seemed to.

[This article appears in the July/August 2019 issue of Streaming Media Magazine as "2019: A Brave New (Codec) World."]

Free

for qualified subscribers

Subscribe Now Current Issue Past Issues

Via LA's Heath Hoglund Talks MPEG LA/Via Licensing Patent Pool Merger

Following on the news that Via Licensing has acquired codec patent pool administrators MPEG LA to form Via LA as a "unified entity," Via LA President Heath Hoglund discusses how the pool will operate, as well as resulting synergies including streamlined IP distribution and economies of scale in this interview with Streaming Learning Center's Jan Ozer.

08 May 2023

How to Produce VVC With FFmpeg

Anytime you start working with a new codec, there are some fundamental tests that you should run to achieve optimal performance/quality optimization. In this article, I'll take you through those tests while encoding VVC using a version of FFmpeg that includes the Fraunhofer VVC codec.

21 Mar 2023