A Progress Report: The Alliance for Open Media and the AV1 Codec
On April 5, 2016, the Alliance for Open Media announced three new members; AMD, ARM, and NVIDIA, and that the AOMedia Video codec, also called AV1, would be developed as an open source project.
As you may recall, the Alliance formed in September 2015, with Amazon, Cisco, Google, Intel, Microsoft, Mozilla, and Netflix as founding members. Its goal was creating an open-source codec for the internet and other markets, and the makeup of the membership presaged fast codec acceptance by hardware and software developers; fast deployment in Chrome, Internet Explorer, and Firefox; and usage by at least two top tier premium content distributors.
Last week's announcement seemed liked a good opportunity to check the group’s progress, so I spoke with Gabe Frost, executive director of the Alliance for Open Media, whose day job is principal engineering manager on the Windows Devices Group. Also on the line was Matt Frost (no relation), head of strategy and partnerships for Chrome media.
We started by talking about the new members. According to Gabe Frost, all three either produce CPUs and/or graphics chips, or license IP that enables other companies to produce chips for computers, graphics cards, mobile devices, and set-top boxes. The benefits of involving these companies were ensuring that the codec was hardware-friendly, and facilitating and accelerating AV1 hardware support. This is especially true for mobile devices, where CPU-based video playback can shorten the battery life of devices without hardware support, and for STBs, where hardware acceleration is essential for compatibility.
“It’s crucial that hardware manufacturers are engaged in the designs so there are no barriers to transitioning hardware support into the hardware roadmap," Gabe Frost said. "Chip vendors have very fixed deadlines, and we want to ensure that codec development intersects our partner tape outs.”
What are the benefits of pursing codec development as an open source project?
“We’re driving engineering across multiple companies," Gabe Frost said. "Doing it behind closed doors with different company policies slows things down.” Note that unlike other open-source codec projects, AV1 users wouldn’t have to share their own source code. In this regard, Matt Frost said, “Developers can get access to the source codec, improve on it, integrate it, and never give anything back.” While the group is hoping that most users contribute to the project, that isn't a requirement.
Performance and Ship Date
What are the main goals for AV1 for both performance and timing? Roughly speaking, the group is targeting improvement of 50 percent over VP9/HEVC with reasonable increases in encoding and playback complexity. However, the major focus is on driving improvements with an eye towards producing a codec that member companies including Amazon, Netflix, and YouTube can quickly implement and use.
With Microsoft, Google, and Firefox in the group, the optimizations necessary to play 4K 60fps in a browser are a priority. One focus is clearly UHD video, including high bitrate, wider color gamut, and increased frame rates. The base version of the codec will support 10-bit and 12-bit encoding, as well as the BT.2020 color space. Another focus is providing a codec for WebRTC (Real-Time Communications), an initiative supported by Alliance members Google and Mozilla, and similar applications including Microsoft’s Skype.
When will the codec ship? The group is aiming to freeze the bitstream sometime between the end of the 2016 and March 2017. Expect to see browser-based support soon thereafter, with the first hardware support within 12 months after that.
Recounting YouTube’s experience with VP9, Matt Frost said, “With VP9, we froze the bitstream in June 2013, and were streaming VP9 from YouTube in August or September of that year.” Regarding AV1, Frost expects YouTube to transition to the codec as quickly as possible, particularly for video configurations such as UHD, HDR, and high frame rate videos where the codec is expected to deliver significant bandwidth savings over VP9. That said, Frost expects YouTube to encode videos to VP9 and H.264 formats for backwards compatibility for at least several years.
So how will contributions from Google’s VP10, Mozilla’s Daala, or Cisco’s Thor be represented in AV1? Gabe Frost explained, “The baseline code in the AV1 project represents where the lion’s share of investment came from before the investment,” essentially indicating that code from VP10, by far the most mature of the three, will dominate. Frost indicated that AV1 will contain some of the best ideas and features of both Daala and Thor.
GPU Acceleration From the Start
Speaking for Microsoft, Gabe Frost noted that the company is directing significant energy for next-generation experiences to AV1, and that the work done integrating VP9 into Edge will give them a great head start, particularly regarding GPU acceleration on Windows and Windows devices such as Xbox One. “We’re deploying significant resources to make sure the codec works well on PCs and Windows devices, which really means prioritizing GPU acceleration," he explained.
Having hardware vendors like NVIDIA in the Alliance means robust GPU acceleration out of the gate, Matt Frost noted. “Most codecs are released by small groups of developers that get the codec out first, then think about optimizations like GPU acceleration,” he said. “Working as the Alliance, we’re optimizing for power efficiency and GPU acceleration from the start, which should provide a real advantage with overall adoption.”
Considering that VP9 had strong browser support and broad GPU acceleration, but few large users other than YouTube, why will the broader market embrace the new codec? While third-party adoption has been slow, the amount of VP9 encoded content was orders of magnitude larger than that encoded with HEVC, Matt Frost was quick to point out. While HEVC is used primarily for specialty applications such as 4K or HDR—places where H.264 doesn’t perform well—VP9 is replacing H.264 in YouTube’s entire encoding ladder, from 240p to 4K.
Several trends will contribute to a much faster acceptance rate for AV1 than VP9, Matt Frost said. First, VP9 launched while Flash still dominated as a playback platform, and Adobe never added VP9 or VP8 playback to Flash. This meant video producers couldn’t use VP9 without converting to HTML5, which clearly wasn’t ready. With the market rapidly shifting to HTML5, more developers will have access to VP9 and AV1. The trend toward UHD and HDR video will also push premium producers towards a new codec.
Playback in the Browser
H.264 has enjoyed a long life, initially because Flash made playback ubiquitous, and ultimately because it turned out to be a highly effective codec. H.264’s longevity is also related to the fact that most producers still shoot and distribute in HD and smaller resolutions. For those considering UHD, HEVC playback is still essentially nowhere outside of connected TVs and the latest Android devices.
While many analysts, myself included, predicted that HEVC playback would be available in Chrome and IE/Edge by now, Google and Microsoft are clearly prioritizing work on VP9 and AV1 over HEVC. Though Microsoft initially pledged to include HEVC playback in Windows 10, it later removed support, opting only to support HEVC-enabled hardware implementations. The bottom line is no software-based HEVC decode in Windows or Edge.
What about Chrome? While software-based HEVC support is available in Android 5.0, Google will only pay that royalty on hardware devices that it ships directly to customers. This means Android shipments might not be sufficient to reach the caps set by MPEG LA and HEVC Advance, so including HEVC playback in Chrome would incur a charge. Beyond these two patent pools, there’s also Technicolor’s license to deal with, plus the risk of other HEVC IP owners attempting to impose their own licenses. All these, plus Google’s obvious focus on VP9/AV1, seem like negative indicators for HEVC playback in Chrome anytime soon.
It seems unlikely that Mozilla, which never licensed H.264, will ever license HEVC. Apple, a member of the MPEG LA HEVC licensing group, is seemingly backing away from HEVC support, removing a mention of the technology from iPhone spec sheets. Between its desktop, mobile, and OTT business lines, Apple would pay far more in HEVC royalties than it would ever earn, which might make the Alliance and AV1 more attractive than it would seem at first glance.
While Apple and Microsoft might seem like strange bedfellows, the two companies recently coauthored a requirements document for a new Common Media Application Format (CMAF) as MPEG-A, Part 20. According to Gabe Frost (speaking for Microsoft and not the Alliance), “CMAF defines the encoding and decoding of segmented media. CMAF segments can be used in environments that support adaptive bitrate streaming of the media segments using HTTP(S) and any presentation description, such as DASH MPD, Smooth Streaming Manifest, and Apple HTTP Live Streaming Manifest (m3u8).”
While this collaboration falls far short of Apple joining the Alliance for Open Media, it does signal that the two companies can work together effectively. It also indicates that Apple, long and publicly intransigent on MPEG DASH, is willing to contribute to new standards that benefit the common good. Will Apple join the Alliance? “We’ve been talking to a lot of companies, and recruiting definitely isn’t our challenge," Gabe Frost said. "The hard part is the engineering effort, taking the next step, and driving contributions through the open source project.”
The bottom line is that HEVC playback within the browser is nowhere, and has little visible momentum, which makes VP9 and AV1 the only games in town for browser-based, UHD playback.
What About Broadcast?
Ten years ago, broadcast was broadcast and streaming was streaming. TVs and cable boxes supported the codecs pumped through cable and satellite, and little video from the internet was watched in the living room. A relatively small number of large cable and satellite operators chose the codec or codecs for their systems.
Today, that situation is reversing. Companies such as Netflix, Amazon, and YouTube are pumping billions of hours of content into living rooms as consumers unplug, and more and more content producers are going direct via IP video. YouTube has indicated that it will stop distributing UHD content in H.264, relying solely on VP9 and then AV1. This, plus anticipated AV1 use by Netflix and Amazon, should make AV1 support a checklist item for future connected TVs and set-top boxes. Once this happens, other publishers are free to choose any codec they would like, and why not choose the one that delivers browser-based playback, as well as TV? Certainly HEVC will dominate in the short term as the infrastructure codec for large cable and satellite systems, but its hold over IP delivery to the living room is clearly in doubt.
What This Means for Producers
How does this impact those producing at 1080p and smaller resolutions? YouTube distributes VP9 rather than H.264 when possible because it translates to lower bitrates at the same quality level, or HD video at the same data rates as SD. Depending on your goals, deploying VP9 or AV1 could mean reduced bandwidth costs, improved quality of service, or both. Someday soon, H.264-encoded video will look as bad as Windows Media or VP6 encoded video looks today, and even HD and SD producers will be compelled to switch.
Fortunately, when you do, industry support for VP9 and AV1 should be significantly improved. As an example, JW Player has announced that it will add VP9 as a supported format for its OVP service, and there have been rumors about several prominent encoding vendors supporting VP9 as well, perhaps as soon as this year's NAB. Don’t expect a clean cutover to VP9 or AV1, however, as the need for mobile and backwards compatibility means that we’ll all be encoding in H.264 for many years to come.
Related Articles
Today the Alliance for Open Media froze the AV1 bitstream and released an unoptimized software encoder and decoder; AV1 decode should arrive in several browsers and some content from member companies over the next few months, with hardware implementations in about a year.
28 Mar 2018
Netflix and YouTube could start using AV1 as soon as early 2018, while hardware implementations will take much longer.
30 Aug 2017
Some of the best minds in the streaming industry say AV1 will be challenged by infringement claims, but some of the best minds could be wrong.
31 Jul 2017
For educational video, some choose to shoot in 4k so they can zoom and enhance in post, but stream in mobile-friendly 720p. However, there's no reason to rush to 4K just yet.
01 May 2017
Once counted out, VP9 is on the rise, with support from Netflix, JW Player, Brightcove, and more. In this interview, David Ronca of Netflix talks about VP9 savings, encoding, and testing.
24 May 2016
In this Q&A with two video IP and technology investment banking professionals, we ask about patent risk, VP 8/9, and the possibility of a third HEVC patent group forming.
16 Sep 2015
With Google, Amazon, Cisco, Microsoft, and others joining forces in the Alliance for Open Video to create a new royalty-free codec, an alternative to HEVC (and its controversial royalties) is on the way. Does this spell the beginning of the end for HEVC?
01 Sep 2015
Leaders from all parts of the online video ecosystem come together to create an industry group that will work for the betterment of streaming interoperability and quality.
14 Nov 2014