The State of Video Codecs 2024
It’s been 3-plus years since the MPEG codec explosion that brought us VVC, LCEVC, and EVC. Rather than breathlessly trumpet every single-digit quality improvement or design win, I’ll quickly get you up-to-speed on quality, playability, and usage of the most commonly utilized video codecs and then explore new codec-related advancements in business and technology.
A warning for the casual reader: The most significant codec-related occurrence in 2023 was the increased likelihood that you may have to pay to use them. I don’t mean royalties on hardware or software decode; I mean content royalties. If the thought of that might keep you up at night, perhaps you should skip this article.
How Does It Perform?
We adopt new codecs because they enable entry into new markets like HEVC and HDR and because they save us bandwidth costs. In May 2023, I gave a talk at Streaming Media East titled Choosing a Codec in 2023 that included the chart shown in Figure 1. I covered mostly open source codecs like x264, x265, and libaom, along with LCEVC and the Fraunhofer version of VVC. Different versions of these codecs will deliver different results, as I’ll discuss in a bit. All codecs are rated in comparison to x265, which is normalized at 100%, a useful presentation technique I borrowed from Moscow State University (MSU).
Figure 1. Relative codec performance
There are many codec comparisons out there, and all present different results. At a high level, however, most show that if you abandon H.264 for any other codec, you can significantly reduce the bitrates of your videos while delivering the same or better quality. As it relates to HEVC and VP9, Bitmovin found these two codecs neck and neck in a study from 2020, particularly over a full bitrate ladder. My tests generally agree.
Figure 1 also shows that AV1 delivers meaningful savings over either HEVC or VP9, and VVC is more efficient than AV1. LCEVC performance depends on the base layer; with an HEVC base, it delivers about 22% greater quality than x265.
Where Codecs Play
While we love to compare codec quality, the most important adoption-related consideration is where the codec plays. Obviously, you only achieve bandwidth savings on devices that can actually decode and play the video encoded with that codec.
Typically, when analyzing playability, we consider three markets: browser, mobile, and living room. Their compatibility estimates are shown in Figure 2. Some of these estimates come from reputable sources, like Can I Use, which lists technologies that are compatible with other technologies, and ScientiaMobile, which tracks hardware support among mobile phones, at least for AV1 and HEVC.
Figure 2. Estimates of hardware support by device
For datapoints not covered, I very politely asked Bard for the estimated overall percentage of support for each codec and market. If you’re reading this in Streaming Media, my editors agreed it was useful information with all of the obvious caveats, and I hope you do as well.
Probably the biggest codec adoption news in 2023 was Apple adding AV1 decode to the iPhone 15 Pro and iPhone 15 Pro Max. While undoubtedly a positive step, it will take 1–2 years before this boosts the installed base of hardware AV1 support to a level that most publishers would find enticing.
As noted, when it comes to mobile, the table in Figure 2 covers hardware playback. But what about software? That’s a fair question, particularly given that software playback can impact playback frame rate and battery life. As an example, Meta deployed AV1 in Reels, although it took considerable development effort. YouTube has been shipping AV1-encoded video for years but doesn’t appear to stream AV1 to phones without hardware support.
One vendor, India-based MX Player, is already deploying VVC to mobile devices with software decode, while several other vendors have released studies showing very high playback frame rates on iOS and Android devices (download a summary here). Still, most mainstream publishers tend to avoid deploying new codecs to mobile devices until hardware playback exists in a sufficiently large number of the installed base. The major exception here is LCEVC, which is an enhancement codec that plays efficiently on most platforms without hardware support.
Who’s Using What?
It’s always interesting to learn how each codec is currently being used and how publishers plan to use them in the future. In 2023, Telestream resurrected Encoding.com’s always useful “Global Media Format” report, which details the codecs that Encoding.com produced for its clients in 2022. These numbers add up to 100%, representing 100% of the company’s 2022 production (Figure 3).
Figure 3. Codec distribution for 2022 Telestream/Encoding.com encodes
Note that WebM includes both VP8 and VP9, but predominantly VP9. As expected, AVC (H.264) dominates, followed by HEVC, which isn’t surprising given premium content customers like TNT, Fox, Tubi, BBC, CNN, NBC, Peacock, and NBCUniversal. Note also that Encoding.com is now called Vantage Gateway.
Figure 4 shows the estimated codec usage for VOD from The 7th Annual Bitmovin Video Developer Report. This features the results of an industry survey as opposed to the “Global Media Format” report, which details actual production statistics.
Figure 4. Current (in blue) and planned codec usage from “The 7th Annual Bitmovin Video Developer Report”
The numbers here add up to more than 100% because respondents used more than a single codec. Numbers in blue are for codecs currently in production, while numbers in red are for codecs that are planned to be implemented within the next 24 months. Again, H.264 and H.265’s related results are not surprising.
AV1 usage is an enigma. In The 6th Annual Bitmovin Video Developer Report, 14% of respondents claimed they were using AV1, and 42% were planning to implement it within 12–24 months. In the current report, these numbers dropped to 8% and 32%. Absent a dramatic shift in the composition of the survey respondents, these numbers are tough to reconcile.
That said, AV1 isn’t the only codec for which usage declined: VVC dropped from 15% currently in use and 29% planned to 7% in use and 12% planned. It may be that the streaming industry’s ardor for new codec deployments is waning.
How Much Does It Cost?
After quality and compatibility, most producers consider cost. This is where things get interesting. For years, the status quo has been this:
- All of the standards-based codecs, including H.264, HEVC, VVC, and LCEVC, are royalty bearing, with one or more patent pools and clear rate structures. However, the overwhelming majority of the royalties are applied to hardware and software playback devices, not content.
- Google has maintained that VP9 is open source and royalty-free, and the Alliance for Open Media has done the same for AV1. However, in March 2020, patent pool administrator Sisvel announced two patent pools, one each for VP9 and AV1. Under the license terms, however, Sisvel is only charging royalties on playback devices, not content.
On Oct. 18, 2023, Avanci launched its Avanci Video platform, which is targeting video publishers for content-related royalties. Here’s what we know from the FAQ:
Who should obtain a license from Avanci Video?
Internet video streaming companies that are providing Internet video streaming services to its users using any of the five standards—H.265 (HEVC), H.266 (VVC), VP9, AV1, and MPEG-DASH—have the option to take a license from Avanci Video. Our licensing program is open to all internet video streaming companies including subscription-based entertainment services, ad-based video sharing sites, social media and video messaging platforms, and video conferencing providers.
You can see the codecs covered in the pool, which excludes H.264, likely because the 20-year patent protection on many of the related patents is starting to expire. There are 26 patent owners in the platform that own various patents in each of the listed technologies, including several from the Sisvel pools. Avanci does not provide the royalty rate.
Even before the Avanci announcement, it appeared as if codec-related patent owners were moving toward an effort to start charging content owners for codec usage. (See What Your Codec Will Cost You: Robert J.L. Moore Talks Avanci Video Codec Patent Pool Launch for a discussion with attorney Robert J.L. Moore about the Avanci pool and previous efforts toward content royalties.) The bottom line is that prior to 2023, it was relatively safe to assume that publishers could use advanced codecs with little concern over the potential for codec royalties. Unfortunately, that is no longer the case.
AI Is Coming
The last 12–14 months have been absolutely seismic with regard to the impact AI has had on our day-to-day lives, particularly at work. When will AI impact the codec space?
Well, it appears that it already has. Figure 5 shows subjective results from the MSU Video Codecs Comparison 2022, which was released in November 2023. As with Figure 1, x265 is the reference codec at 100%, with others shown as more or less efficient. Smaller is better here, as you probably noticed, and the smallest goes to Tencent TVC, which is about 10% more efficient than Tencent’s VVC codec (VVC is also H.266).
Figure 5. Tencent TVC, the highest-performing codec in a recent Moscow State University study, has an AI component.
What is TVC? Here’s what we know from a Tencent press release: “TVC continues to overcome compression bottlenecks caused by long standard iteration, optimize high encoding/decoding complexity introduced by new AI coding tools, and meet customized encoding/decoding demand by thoroughly optimizing coding tools, fast algorithms, engineering and bitrate control.” It’s hard to discern the signal from the noise regarding how much AI is actually involved, but this isn’t the only instance of AI popping up relating to codecs.
Deep Render is an AI startup focused on developing advanced video compression technologies. Founded in 2018 by two students from Imperial College London, Deep Render uses AI to intelligently analyze and reconstruct visual data to minimize artifacts at high compression rates. It claims that its codec already offers 5x smaller video file sizes, with a goal of up to 50x improvements. Deep Render unveiled its technology in 2022 and in 2023 raised $9 million in Series A funding. In addition, it was chosen as the winner of the 2023 Intel Startup Innovator Award. You can see a video about the company.
Also worth tracking is the Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) initiative, which was founded by Leonardo Chiariglione, who also founded MPEG. MPAI is focused on enhancing video-coding technologies through AI, particularly respecting the EVC codec.
What’s impressive about MPAI is not the nuts and bolts of the target areas for enhancements, which are intra prediction, super resolution, and in-loop filtering. Rather, it’s the big picture focuses of its research, which includes AI for health applications, AI for connected autonomous vehicles, and AI for the metaverse and human-machine communication (Figure 6).
Figure 6. One MPAI standard involves connected autonomous vehicles.
If you’re a broadcast or media type, you’re absolutely right to scoff at the idea that these AI-based development efforts will immediately impact your work, as it takes years for codecs to percolate through the cumbersome process of standards adoption and deployment in operating systems, browsers, and devices, not to mention encoders and transcoders. However, for closed applications like conferencing, autonomous vehicles, or even medical devices, where one vendor controls both the encode and decode sides of the equation, these codecs could be (like the mirror says) closer than they appear.
Related Articles
In this wide-ranging interview, Streaming Media's Jan Ozer and intellectual property attorney Robert Moore of Moore IP solutions delve into a number of topics related to video codec intellectual property issues and patent monetization, two constantly moving targets.
06 May 2024
Welcome to your codec update for 2023. I'm here to help you decide whether it's time to go all in on AV1, VVC, LCEVC, or EVC or whether it's better to stick with H.264, VP9, and HEVC. Along the way, I'll include 2022-2023 updates to identify significant changes that have occurred since last year.
12 Apr 2023
With H.264 usage finally beginning to decline and several newer codecs ascendant, this 2022 codec update reports on the most significant announcements from the last year relating to H.264, VP9, HEVC, AV1, VVC, LCEVC, and EVC.
25 Mar 2022
AVC led to format consolidation, but we've seen more fragmentation in recent years. Fortunately, the huge critical mass around AOMedia might lead to a simpler future.
04 Mar 2021
Companies and Suppliers Mentioned