July 17, 2018
By Jan Ozer Contributing Editor
Featured Articles

Return of the Codec Wars: A New Hope—a Streaming Summer Sequel

Legacy Apple platforms incompatible with HEVC seamlessly played the H.264 streams in the ladder, no muss, no fuss. HEVC playback was at full frame rate on all tested devices, including older devices like an iPhone 6. The only potential issue was that 4K-capable devices with more than adequate bandwidth seldom stepped up beyond the highest quality 1080p stream, an issue that we reported to Apple and that it is currently analyzing.

Overall, at least as it relates to delivering hybrid streams to Apple devices, adding HEVC to HLS should involve little player development expense, at least for those playing HLS natively rather than via an app. Adding HEVC to HLS also involves minimal development risk, as playback compatibility and quality was very good.

Hopefully, this data will help break the HEVC-in-HLS logjam and we’ll see many streaming producers chasing the improved QoE and bandwidth savings that deploying HEVC in HLS should provide. Recent survey data published by Streaming Media indicates that many producers do plan to implement HEVC in HLS in the coming months. Specifically, the survey was sponsored by Harmonic and published in January 2018 as a report entitled “The Impact of Apple’s HEVC Adoption: A Survey-Based Report.” More than 600 StreamingMedia.com readers responded, with more than 70 percent considering adding HEVC to their HLS streams by the end of 2019, though potential royalties were listed as a significant concern by many respondents.

Whether this proves accurate remains to be seen. To date, most HEVC deployments have been to Smart TVs, OTT, and set-top-box (STB) platforms, large companies chasing new markets, and services rather than bandwidth efficiencies. In these cases, the companies either assume the HEVC royalty risks or simply don’t participate. In many cases, if royalties are imposed, the service providers can increase prices to recoup the expense.

Although it seems unlikely that Velos Media or other HEVC IP owners would attempt to charge a royalty on free internet video (e.g., non-subscription or pay-per-view), cautious CFOs simply can’t rule this out. Since it’s impossible to run a break-even analysis when costs aren’t known, and challenging to recoup a royalty expense when you’re not charging for your videos, it will be interesting to see how many publishers assume the royalty risk to chase the bandwidth savings.

VP9

VP9 is an open-source codec owned by Google, and the successor to VP8, which Google acquired from codec vendor On2 in 2009. In terms of reach, VP9’s strength is in browser-based computer playback, with support in Chrome, Firefox, Edge, and Opera, but not currently Safari. Android started supporting VP9 in Version 4.4, which means that about 95 percent of all Android devices can play VP9. VP9 is well supported on most Smart TVs and OTT devices, though not as universally as HEVC.

From a features perspective, VP9 is weak in several key areas. There are few, if any, commercially available real-time VP9 encoders, reducing VP9’s appeal as a contribution format, though multiple transcoding engines, including some from Wowza and Nimble Streamer, can transcode into VP9 format for distribution. VP9 was not featured in any of the low-latency streaming offerings that I could find, though it’s very widely used in communications applications like WebRTC. While VP9 has the bit depth to support HDR video, today, it’s only currently supported in one of the three primary HDR formats, HLG (Hybrid Log Gamma), and not Dolby Vision or HDR10 or 10+.

Basically, the primary motivation for deploying VP9 is for playback on computers and Android devices. Like HEVC, VP9 should deliver about the same quality as H.264 at about 60 percent the bitrate, though generally VP9 encodes more quickly than HEVC, so should be less expensive to encode. Producers considering this approach should run the same analysis suggested in Figure 4 to identify the bandwidth savings, and then run a break-even analysis from there.

VP9 is a royalty-free technology, although as with HEVC, there is some risk of some third party claiming IP infringement. Note that Nokia actually brought suit claiming that VP9 predecessor VP8 infringed upon its technology, but lost in a German court. Beyond Nokia, in 2013, Google licensed “technologies that ‘may’ be essential to VP8” from 11 patent holders in the MPEG LA H.264 patent pool, and the agreement also extended to VP9. This agreement, plus the general momentum toward AV1, makes the risk of any IP-related suit seem modest, but this certainly can’t be ruled out.

VP9 CURRENT STATUS

VP9 has enjoyed several high-profile use cases, including YouTube for streaming and Netflix for downloadable movies. In terms of service provider deployments, JW Player uses VP9 in the online video platform (OVP) of its business. Otherwise, VP9 usage has been modest, with Encoding.com reporting that only 6 percent of streams produced in 2017 were VP9, and Bitmovin reporting that only 10 percent of its respondents were using VP9. Conversely, in the aforementioned Streaming Media survey, 27.3 percent of respondents said they planned to implement VP9 in 2017 or beyond.

This brings us to the AV1 codec.

AV1

As I mentioned earlier, the AV1 is a codec produced by a group called the Alliance for Open Media. Founded in 2015, AOM consolidated the open-source codec development efforts from Google (VP10), Mozilla (Daala), and Cisco (Thor), along with compression expertise from other founding members like Microsoft and Intel. Other members include major players like Amazon, Netflix, Facebook, Apple, Hulu, BBC, Nvidia, ARM, Broadcom, and Bitmovin. The AV1 codec was originally scheduled to ship no later than March 2017, and was released in draft form about a year late, in March 2018.

According to AOM, the Alliance anticipates a four-phase rollout for AV1 (Figure 5). Phase 1 involves deploying specifications, reference software, and reference streams so products and services that would deploy the codec can begin integration. Phase 2 involves optimization of the reference software for more efficient encode/decode, playback in browsers, and the distribution of AV1 encoded content. According to other slides in the AOM presentation, Phases 1 and 2 should occur in 2018.

Figure 5. The four-phase rollout for AV1

Phase 3, which should roll out during 2019, involves support for AV1 playback in gaming consoles as well as improved encode/decode performance via hybrid software and hardware implementations like GPUs or FPGAs. Finally, 2 years after the launch, the first consumer devices with AV1 encode and decode should appear in phones, tablets, computers, OTT players, and set-top boxes.

AOM’s star-studded membership suggests that this schedule should be met. For example, at NAB 2018, both Google and Mozilla showed AV1 playback in test versions of their browsers. YouTube and Netflix indicated that AV1 encode was already integrated into their encoding pipelines, to be deployed soon after browser-based playback was released. The 2-year release to commercial hardware schedule is pretty standard, and with members like Intel, Nvidia, ARM, and Broadcom, seems achievable.

Still, although AV1 has a marvelous pedigree and promise, too little is known to predict how quickly it will become relevant. For example, as indicated in Figure 1, although AV1 playback will likely be included in Chrome, Firefox, Edge, and perhaps Safari over the next few months, we don’t know the CPU load required to play AV1-encoded video. If the CPU load is too high, playback will be slow and balky on older computers.

Figure 1 also reflects that hardware-assisted playback on mobile platforms and support in Smart TVs, OTT boxes, and set-top boxes won’t appear for 2 years. Even then, of course, the installed base of AV1-enabled devices will be insignificant compared to HEVC, VP9, and H.264.

Early quality trials from AOM members like Facebook are promising, with AV1 outperforming VP9 by about 30 percent and H.264 by about 50 percent (Figure 6). However, encoding times were glacial, with some AV1 encodes taking more than 10,000 times longer than H.264, which should directly affect encoding cost.

Figure 6. In these tests by Facebook, AV1 proved about 30 percent more efficient than VP9.

These are very preliminary trials, and most AOM members expect encoding times to drop precipitously. For example, at NAB Netflix’s director of encoding algorithms, Anne Aaron, said “Encoding time, right now, is very, very slow. ... But I’m hopeful and ... I expect the complexity to go down. Probably around five to 10 times more complex than VP9, that’s what we’re expecting. Or if it’s around that range, that’s OK with us.”

These comments highlight several key points. First, when your videos are viewed millions or tens of millions of times, as are Netflix’s and Facebook’s, you can spend a lot more on encoding if it delivers meaningful bandwidth savings. So, what’s reasonable for the largest scale publishers may not be right for smaller publishers.

Second, we’re in the very early days of AV1 encoding, and running competitive trials now, before encoders are at least reasonably optimized, is an almost-useless exercise. Over the next 12 months or so, we’ll see a raft of comparisons; some that confirm Facebook’s findings and some that dispute them. None of these matter until tests are performed on actual production software producing usable output in a reasonable encoding time.

Basically, over the next 12 months, AV1 will achieve the same reach as VP9, but may bog down on older computers, and will cost orders of magnitude more to encode, making it too expensive for all but the absolute largest video distributors. It clearly won’t be viable for live production or transcoding, or for any kind of low-latency transmission, and without support from Smart TVs, STBs, or OTT devices, high dynamic range video.

Moving down the cost factors in Figure 1, AV1 is also a royalty-free technology, though this doesn’t mean there won’t be challenges from companies like Nokia or others that own HEVC- or H.264-related IP rights. That said, suing AOM would certainly be a formidable task, as its members include very many well-funded and IP-savvy technology firms, and the group established a defense fund in case a smaller member or licensee is sued.

What did our survey respondents say? As shown in Figure 7, interest in AV1 was the highest of all listed codecs (HEVC was addressed in other questions). Whether this translates to actual adoption, of course, remains to be seen, but clearly for the vast majority of potential users, AV1 is not ready for deployment.

Figure 7. Planned codec adoption according to a Streaming Mediasurvey sponsored by Harmonic

Summary

Where does that leave us? Over the next 12 months, adding HEVC to HLS seems like the most obvious move for large content producers. In many, but not all, regions, this should deliver both improved QoE and significant bandwidth savings. As mentioned, if you’re already producing HEVC for other platforms, adding HEVC to HLS makes a lot of sense.

I’d like to believe that a significant number of companies will deploy VP9 for browser-based and Android viewing, but this opportunity has been available for years with few takers, and the coming availability of AV1 may stall VP9 deployments.

As for H.264, it’s clearly not going anywhere. Companies seeking to reduce their bandwidth costs here should explore the savings potential of per-title encoding technologies, which are now available from almost all encoding vendors. Remember that while H.264 is tried, true, and familiar, it likely is both reducing your QoE and increasing your bandwidth costs.

[This article appears in the July/August 2018 issue of Streaming Media magazine as "Return of the Codec Wars?"]