October 26, 2018
By Jan Ozer Contributing Editor
Featured Articles

At the Battle of the Codecs, Answers on AV1, HEVC, and VP9

On October 17, I moderated an event entitled the Battle of the Codecs: AV1 vs VVC, presented by Bitmovin as an adjunct to the Demuxed conference in San Francisco. I learned a lot about how many companies were using or planning to use codecs like VP9, AV1, and HEVC, and their experiences with these technologies to date, which I’ll share with you in this article.

[Editor's Note: Those interested in the next generation of codecs will want to attend the Video Engineering Summit, part of Streaming Media West.]

The structure of the event was as follows: The audience of about a hundred was divided into five groups representing the H.264, VP9, HEVC, AV1, and VVC (Versatile Video Coding) codecs. These groups were led by Thierry Fautier, vice president of video strategy at Harmonic (H.264), Colleen Henry from Facebook Video (VP9, presenting her views, not presenting on behalf of Facebook), Tom Vaughan, vice president of strategy at Beamr (HEVC), Christian Feldmann, codec engineer at Bitmovin (AV1), and Oliver Gunasekara, CEO at NGCodec (VVC). Note that the group leaders were somewhat randomly assigned and do not necessary imply stronger support for a particular codec.

Initially, the groups met to rank each codec against the other codecs in the characteristics defined in Figure 1.

Figure 1. We ranked the codecs on these characteristics.

Then we reconvened into one large group to discuss and agree on these rankings, which we accomplished by physically placing the experts along a line marked from low to high (Figure 2). This exercise was more an entertaining voicing of opinions than a research- and fact-based comparison, and there was seldom unanimous agreement on the final ranking.

Figure 2. Codec experts try to line up in order of compression efficiency.

During this group discussion, users could share their experiences with the different technologies and ask questions. At the end of the discussion, each group moderator identified the markets they felt represented to strongest business case for codec adoption.

Figure 3. The codec scorecard

Figure 3 presents the scorecard for the event, and though there were few surprises, many of the categories elicited new information from the moderators or attendees. For example, in the first category, encoding performance (which really should be called encoding time, not performance), one encoding engineer from YouTube reported that encoding times for AV1 were now only about 16X those of VP9, a significant drop from the 1000x+ encoding times reported just a few short weeks ago. This was a nice real-word verification for the encoding time improvement that we reported here, though AV1 still ranked last in this criteria.

In terms of decode performance, one attendee reported that a major social media company was already distributing AV1 streams to mobile viewers with efficient playback, using decoders included in the company’s iOS and Android apps. I shared my finding that Chrome and Firefox were playing 1080p video on a single-CPU HP ZBook notebook using between 15 and 20 percent of CPU resources. At least from the perspectives of encoding time and decode performance, these discussions alleviated two of the initial concerns about deploying AV1.

However, the next category raised significant issues regarding AV1’s compression efficiency, with Fautier revealing an Harmonic-sponsored codec evaluation that found that AV1 produced similar compression efficiency to HEVC, which he said was similar to findings from the BBC and encoding vendor ATEME.

There was substantial animated discussion regarding the comparative efficiency of HEVC and AV1; you can see the disparity between the line-up in Figure 2, which had HEVC well ahead of VP9, and the conclusion in Figure 3. Though the group good-naturedly settled on equal quality, most (but not all) third-party analyses rate HEVC higher in this regard. While some in the group believed that AV1 was superior to HEVC and VP9, they also agreed that it trailed VVC, though this technology is still at least two years out.

TCO was a complicated factor that balanced bandwidth savings against encoding, licensing, and other deployment costs when producing a given compression quality target. Whatever the formula, the results shown in Figure 3 make little sense that none of the moderators could explain, so please ignore.

IP Framework relates to the licensing structure where open-source royalty-free codecs like VP9 and AV1 enjoy a distinct advantage. While some in the audience felt that AV1 may be challenged by IP claims, these concerns were offset by the IP defense fund that the Alliance for Open Media has established, plus the sheer size and technology savviness of the member companies. Most in the group seemed to agree that companies that implemented VP9 had little risk of any IP-related lawsuit or challenge.

While the HEVC licensing structure has been widely criticized, several in the audience felt that the criticism was overblown, with one attendee commenting that HEVC royalty rates were comparable to other technologies like Dolby audio. Many agreed that while the HEVC licensing model was challenging for freely distributed software like browsers, it was workable for hardware devices where the costs could be passed along to the ultimate buyer.

Meet the Media Coding Industry Forum (MC-IF)

Though VVC ranked as the most complex from a licensing perspective, there was some hope that VVC will be handled more effectively than HEVC. As an attendee explained, these hopes relate to the formation of a new industry group called the Media Coding Industry Forum (MC-IF). According to the group’s website, “MC-IF will work on the non-technical aspects of deployment of media standards, notably including licensing, by facilitating cross-industry discussion.”

Multiple companies that are members of the MC-IF were represented at the event, and we heard that achieving a rational licensing policy for VVC was a primary goal. However, these attendees also clarified that the MC-IF would have no ability to dictate policy to the VVC IP owners so that its success was far from guaranteed. Notably, a few days after the event, several outlets reported that Apple had joined the MC-IF.

Next up on the scorecard is the installed base of decoders, which H.264 handily won since it plays almost everywhere. VP9 was next, with playback on all major browsers other than Safari, and most major platforms including an increasing share of connected TV, set-top boxes, and OTT devices (like Roku 4). Though VP9 doesn’t play natively on iOS or Apple TV devices, software-based playback (as opposed to hardware accelerated playback) may be supplied via apps. Of course, while this works well when you have a “must-have” app like YouTube or Netflix, it may not be effective for smaller companies with less ubiquitous apps. Though there have been some third-party attempts to deploy VP9 decoders on AppleTV devices, the 4K AppleTV can reportedly only display YouTube in 1080P or lower resolutions, which likely means no VP9 playback on what likely is a very well distributed Apple TV app.

Still, the platforms available for VP9 far outnumber those available for HEVC, which lacks playback support in Chrome and Firefox, but does play on Android and iOS devices, and virtually all STBs, connected TVs, and current models of OTT devices like the Roku 4, as well as the native browsers on most current Windows computers and all Macs. As one participant pointed out, nearly all HEVC playback is accomplished in hardware, which is much more CPU/battery efficient than software-based playback and produces a more consistently high-quality experience.

AV1 is barely out of the gate so enjoys no hardware-based playback, though it is supported by the most current versions of Firefox and Chrome on some platforms. As mentioned above, one prominent social media company is already distributing AV1 streams to mobile users for playback via its iOS or Android applications. If AV1 produces the promised 20 to 30 percent encoding efficiencies, you would expect many similarly large companies to do the same. Obviously, since it’s not yet released in any form, VVC ranked last.

The final characteristic was ecosystem adoption, which measured the range of available hardware and software encode/decode implementations, including contribution, live encoding, and support for non-streaming applications like security, low-latency, and military. Here, H264 still dominated, with VP9 ranked second primarily because of the range of decode options. However, most panelists expected existing non-streaming applications to migrate from H.264 to HEVC, pointing to multiple devices like HEVC live streaming encoders and on-camera encoders, and the total dearth of those devices for VP9. Both AV1 and VVC ranked last as hardware options weren’t available for either technology.

Looking forward, the H.264 advocate predicted that while H.264 would lose share in streaming markets, it still had room to grown in non-streaming markets that needed high quality, low latency video at reasonable bitrates, with reasonable decoding requirements, and reasonable costs. The HEVC advocate felt that HEVC usage would dominate in distribution to OTT and connected TV devices, with a significant pickup in HEVC distributed via HLS to Apple desktops, mobile devices, and the AppleTV 4K.

VP9 seemed destined for high-volume, long-tail content for OTT and UGC markets, succeeded by AV1 in the short to mid-term. VVC was too far out to predict.

The Major Takeaways

For me, the most significant takeaways related to AV1, which seems a lot closer to deployment than it did six weeks ago. There are doubts as to encoding efficiency, though we’ll know a lot more in the next few months as real-world deployments happen. It was also interesting to see that in addition to the connected TV and similar markets, HEVC is destined to succeed H264 as the codec of choice for a broad range of non-streaming markets like on-camera encoders, security, and contribution, where VP9 appears unlikely to play any significant role.

The big question with VVC is whether the IP owners can come together to offer an all-inclusive royalty package at a competitive rate. While MC-IF offers some hope, it clearly doesn’t have the authority to dictate this result.