Apple Joins Alliance for Open Media: What Does it Mean?

On January 5 , roughly 27 months after the inception of the Alliance for Open Media (AOM), Apple appeared on the organization's website as a "founding member." Like everyone else, we first read about the story on CNET, and CNET likely found out when the Alliance simply updated its website. As you probably know, AOM was formed in September 2015, during the height of the HEVC royalty insanity, to consolidate the development of multiple open-source codecs into one. That codec is called AV1, and the bitstream is scheduled to be frozen in early 2018.

The obvious questions are why did Apple join the group, and how do we expect Apple to deploy AV1 and possibly other codecs like VP9? We'll tackle those in order after a pause for reflection.

What Just Happened to Video on the Web

When Adobe added H.264 to Flash in 2007, Adobe developer Tinic Uro wrote an iconic blog post, since taken down, called "What Just Happened to Video on the Web." Within 2 or 3 months or so, computer-based streaming, which was almost all there was back then, pivoted from On2's VP6, the ironic predecessor to AV1, to H.264, from which HEVC was born.

So I'll hijack Uro's line and ask "what just happened?" From a philosophical perspective, Apple's move is tectonic. Regarding impact on the codec market, that remains to be seen.

The philosophical backstory is this. Google purchased On2 for $120 million or so and released VP8 as an open-source codec in May 2010. Apple had fully committed to H.264, with support in QuickTime and various devices, and was a licensor in the MPEG LA H.264 (and HEVC) patent pools.

One of the first reviews of VP8's source code was by Jason Garrett-Glaser (then a third-year college student also known as Dark Shikari) who was (is) one of the developers of the x264 codec. His review, also since taken down, was extremely critical from both technical and intellectual property (IP) perspectives. For example, he stated "VP8, as a spec, should be a bit better than H.264 Baseline Profile and VC-1. It's not even close to competitive with H.264 Main or High Profile." Regarding IP, he also stated "VP8 is simply way too similar to H.264: a pithy, if slightly inaccurate, description of VP8 would be 'H.264 Baseline Profile with a better entropy coder.' Though I am not a lawyer, I simply cannot believe that they will be able to get away with this, especially in today's overly litigious day and age."

When asked what he thought about VP8, Apple's Steve Jobs, responding in a 4:30 a.m. email from his iPhone, simply provided a link to Garrett-Glaser's review, in essence a digital "what he said." To paraphrase: bad codec, worse IP risk.

Steve Jobs' initial thoughts on VP8/WebM, a key technical antecedent to AV1.

Not surprisingly, Apple never deployed VP8 or VP9, which became problematic when YouTube started deploying 4K videos solely in VP9. Since then, Apple has supported only standards-based codecs, most recently adding HEVC to HLS in June, 2017. On the other hand, Apple did collaborate with Microsoft on the Common Media Application Format (CMAF). As we reported back then,

While this collaboration falls far short of Apple joining the Alliance for Open Media, it does signal that the two companies can work together effectively. It also indicates that Apple, long and publicly intransigent on MPEG DASH, is willing to contribute to new standards that benefit the common good. Will Apple join the Alliance? "We've been talking to a lot of companies, and recruiting definitely isn't our challenge," [AOM executive director and Microsoft employee] Gabe Frost said. "The hard part is the engineering effort, taking the next step, and driving contributions through the open source project."

So, while Apple's joining AOM is a dramatic turnaround, it's not completely unexpected.

Why Did Apple Join AOM?

Many articles have attributed Apple's move to Mozilla's claim that AV1 will be 25-35% more efficient than HEVC. While this is certainly worth chasing, these numbers are speculative, and I think Apple's reasons are more strategic than financial.

Mozilla predicted at Demuxed that AV1 will be 25% more efficient than HEVC.

One useful analysis on this front is Jeremy Horwitz's VentureBeat article called "3 Reasons Apple Joined the Alliance for Open Media." The first reason Horwitz cited was Apple wants to add AV1 to its own devices. While true, understand that most of the mobile devices that Apple addressed by adding HEVC to HLS have HEVC decoding hardware built in, minimizing the impact of the new codec on battery life. We don't yet have a clear picture of how CPU-intensive AV1 decoding will be, but if it's substantially more complex than HEVC, publishers would likely prefer HEVC to preserve their viewer's battery life. The best guess is that it will take about 2 years for hardware decode to reach any consumer product.

The second reason proffered by Horwitz is more compelling: "Apple needs AV1 to stream videos to non-Apple devices." Specifically, it appears that Apple is serious about building a content business akin to Netflix and Amazon. While as a hardware developer, Apple can blissfully ignore the importance of Android devices, it can't as a content service. Netflix, in particular, has embraced VP9 as a delivery codec, initially for mobile downloads to Android devices, and then in a more expanded role. Of course, Android started supporting HEVC in software with version 5, and many Android devices already have HEVC hardware. Still, VP9 is much more of a universal format than HEVC, which still doesn't play in Chrome, Firefox, Internet Explorer, or Opera. In the short term, it will be very interesting to see if and when Apple includes VP9 playback in Safari and in iOS updates. The bottom line is that if Apple is serious about becoming a player in content distribution, they had to have AV1.

The third reason Horwitz cites is Apple wanting to help steer AV2. While undoubtedly true, that seems so far off that I won't explore this any further.

What Do We Expect Apple to Do?

Apple is Apple, and they aren't talking. Our best guess is what other companies did when they joined the group, or what we expected them to do.

Regarding VP9, soon after AOM was formed, founding member Microsoft added VP9 to Edge. In for a penny, in for a pound; it makes sense that Apple would do the same for Safari, MacOS, iOS, and tvOS as soon as possible. This would deliver the immediate benefit of allowing Safari viewers to play UHD videos from YouTube.

Regarding AV1, we expect Google (Chrome), Mozilla (Firefox), and Microsoft (Edge) to add AV1 decode to their browsers soon after the ink is dry on the specification. Our best guess is that Apple will do the same.

On the hardware front, we expect Intel, NVIDIA, ARM, and other members to add AV1 encode/decode to their respective CPUs, systems on a chip (SO), graphics cards, and other hardware devices as soon as possible. Again, our best guess is that Apple will add AV1 hardware encode/decode to their devices as soon as possible, though, as stated, this means two years or so.

We expect YouTube to start encoding videos into AV1 format within weeks after the spec is released, with Netflix trailing a few months later. Of course, both of these services currently support VP9, which is much more efficient than H.264 with similar decode requirements, and plays nearly everywhere. Apple would get the most immediate bang for their buck with VP9 support, but we would expect some AV1 encoded content by mid-2018.

What about adding AV1 or VP9 to HLS? Again, depending upon decode requirements, AV1 might not be useful on devices without hardware support, limiting the short term benefit. If Apple adds VP9 support to its iOS devices, and VP9 in HLS is technically feasible, Apple may take this step as well.

What's this Mean for General Codec Usage?

When Apple added HEVC to HLS, the reaction in the HEVC encoding community was akin to finding out the Pope embraced Buddhism. You have to add HEVC today, if not sooner, to avoid falling behind the avalanche of competitors who are rushing to do so. HEVC or bust, baby! As we've seen, the reaction by actual publishers has been a bit slower.

In truth, because of the decode issue discussed multiple times above, AV1 likely won't play a significant short-term role for playback to Apple devices, though VP9 might. Still, many have predicted that AV1 will run afoul of IP challenges, as if members like Cisco, Google, Microsoft, and Intel were technically unsophisticated and would buy an IP-related pig in a poke. Apple's joining the group throws another nail in the coffin of that theory and adds to the war chest that could be used to defend those claims.

What about the impact on Apple's decision upon consumers? It could dramatically change which codec we use to watch our videos, though it won't in the short term, and even when it does we likely won't be able to tell.

If you're a content producer or distributor, you now have another very strong reason to consider AV1. If you're not in the business of selling content, the impact will take much longer; it may be several years before you should start thinking about encoding into AV1.

Perhaps most of all, Apple's joining AOM sends a clear message to HEVC IP owners that unlike what they believed after Apple added HEVC to HLS, HEVC isn't the only game in town. What this incredibly tone-deaf group does with this message remains to be seen.