MPEG: What Happened?
Since its establishment by Leonardo Chiariglione and Hiroshi Yasuda (from Nippon Telegraph and Telephone) in 1988, the Moving Pictures Experts Group (MPEG) has made an indelible mark on the transition from analog to digital video. According to Chiariglione, "globally, the annual value of products and services that rely on MPEG standards is ~1.5 trillion USD, or ~2% of the world gross product." Certainly, no one could possibly doubt the incredible value that video codecs created by MPEG has played during the COVID-19 crisis in entertainment and commerce.
On June 6, 2020, the message shown in Figure 1 appeared on Chiariglione's MPEG site announcing that MPEG is "closed." A little digging revealed that while Chiariglione had, in fact, separated from MPEG, MPEG was in fact not closed, though it was significantly reorganized. Still, if there was a Mount Rushmore for those who contributed to the success of digital video, Chiariglione would clearly have a bust, making his parting with MPEG significant enough to memorialize.
Figure 1. MPEG is closed? Emphasis supplied and image edited for space.
So, we'll do three things in this story. First, we'll tell you what happened. Then we'll share our thoughts on the impact Chiariglione's parting with MPEG will have going forward. Third, once we learned of Chiariglione's parting, we asked readers to contribute some thoughts about his and MPEG's contributions to digital video. Only a few readers contributed, but we'll share their thoughts below.
What Happened?
By way of background, MPEG is the standard-setting organization that brought us MPEG-1, MPEG-2, H.264, and HEVC, with Versatile Video Coding (VVC), Essential Video Coding (EVC), and Low Complexity Enhancement Video Coding (LCEVC) set to launch in 2020. Technically, MPEG operates under the International Organization for Standardization or ISO. From an organizational perspective, SC 29 (shown at the top of Figure 2) is managed by the ISO and IEC Joint Technical Committee (ISO/IEC JTC 1), which develops international standards for ICT. JTC 1 has 22 subcommittees, including SC 29.
There are multiple working groups under SC 29, including Working Group 1, for JPEG (not shown for space reasons), and Working Group 11, for MPEG, which until June 6 was run by Chiariglione. Beneath each working group are subgroups, including those shown in Figure 2 with their individual functions.
Figure 2. What MPEG was before the reorganization, from descriptions here.
After the June 6 reorganization, the top two spots stay the same, but Working Group 11 is seemingly eliminated and many of the former subgroups are pushed up to Working Group (WG) status, as you can see in Figure 3. I derived this new structure from the Resolutions of the SC29th Plenary Meeting held online on June 4, 2020. According to a note received from Gary Sullivan, "Note that the MPEG coordination group will likely have the primary role of coordinating MPEG work rather than SC29 itself."
Note that all the working groups and advisory groups in the Resolution are prefaced with MPEG which I removed from the organization chart to save space. So, the document calls the Audio Coding Working Group the MPEG Audio Coding Working Group, preserving the MPEG name. In an announcement dated June 24, ISO stated that the reorganization would become effective on July 17, 2020.
Figure 3. The SC 29 and MPEG after the reorganization.
As you can see in Figure 3, SC 29 also chose the aforementioned Dr. Gary Sullivan, a Microsoft employee who was formerly co-chair of the JCT-VC subgroup, to succeed Teruhiko Suzuki in 2020 as chair of the group. According to an article in Rethink Research (subscription required), "We understand Chiariglione was offered a range of "honorary" roles but declined, clearly considering this offer an insult to his integrity given that these positions are virtually powerless. So, Chiariglione abruptly resigned from the ISO and declared the death of MPEG as his mic-drop moment."
What Does it All Mean?
So, we have a change in organizational structure and a change of leadership. What does that mean? Regarding structure, Chiariglione shared his thoughts in multiple articles both before and after the restructuring. Particularly on point were comments from his February 2018 article "Can MPEG overcome its Video 'Crisis'?" After describing MPEG's organization structure detailed in Figure 2 above, Chiariglione stated:
[MPEG]is a probably unique organization in ISO, but it exists because standards for media systems require a strong interaction of its components. Because of this MPEG holds several joint meetings where relevant subgroups discuss and agree on matters of common interests. The ability to develop complex digital media standards is one of the reasons of the success of MPEG standards in the market. Breaking up MPEG would deal a fatal blow to the ability to create valid standards.
In other words, having multiple subgroups that have to work together under a single manager makes a lot of sense, though in essence this is what we see after the reorganization as well. Other articles Chiariglione wrote after the restructuring make it clear that he also feels that the ISO structure is hypocritical, chaotic, and feudal, and that standards are better set from the bottom up (by members), as MPEG used to work, than from the top down (by national standards bodies), as he claims ISO works. I honestly don't know enough about how ISO and MPEG functions to evaluate any of these claims, but they make for interesting reading.
What does this mean for the MPEG video codecs to be released in 2020, VVC, EVC, and LCEVC? As stated in the Rethink Research article, "Not much is expected to change on a technical basis for the three new MPEG standards coming out this year—JVET VVC, MPEG-5 Part 1 EVC, and MPEG-5 Part 2 LCEVC. All three are ISO/IEC standards, and all three are already technically complete, so the organizational changes of ISO SC29 should only impact future ISO/MPEG standards, rather than on the MPEG standards that have already been completed. Of course, industry adoption of these three emerging standards will depend on technical merit, ease of implementation and business cases, more than what the new-look ISO SC29 says or does."
So the codecs clearly will be launched, but will they thrive?
The World According to Chiariglione
Chiariglione wouldn't comment directly for this article, but has made his thoughts clear in his many writings. Reflecting upon MPEG's successes in "A future without MPEG," he writes, "MPEG has gambled for 30 years, sometimes winning, sometimes losing." Clearly, he considered MPEG-2 and AVC/H.264 wins, writing that "MPEG LA developed the MPEG-2 Video license. MPEG-2 was widely used" and "MPEG-4 AVC is a very successful standard that can proudly bear the "generic" attribute because it is used for broadcasting and online streaming as well."
On the other hand, HEVC was a loss, at least as it relates to streaming. In the same article, Chiariglione writes, "After 7 years, MPEG-H HEVC patent holders could not get their acts together and propose a decently unified license. HEVC is used in broadcasting, however, use for streaming is limited at best." Also regarding HEVC, in "A crisis, the causes and a solution," he wrote: "At long last everybody realises that the old MPEG business model is now broke, all the investments (collectively hundreds of millions USD) made by the industry for the new video codec will go up in smoke and AOM's royalty-free model will spread to other business segments as well."
It's clear that Chiariglione considers AOM a significant competitor to MPEG, and he predicts a dystopian future for video codecs and video codec developers if AV1 succeeds as a royalty-free codec. In "A future without MPEG," he writes that "the void left by HEVC has been filled by AOM with their AV1 specification which is widely used for streaming."
In "A crisis, the causes and a solution," he continues "AOM will certainly give much needed stability to the video codec market but this will come at the cost of reduced if not entirely halted technical progress. There will simply be no incentive for companies to develop new video compression technologies, at very significant cost because of the sophistication of the field, knowing that their assets will be thankfully—and nothing more—accepted and used by AOM in their video codecs."
He continues in "What to do with a jammed machine?," "The new future is one where thousands of well-paid jobs in media compression and academic research that feeds it will go because there is no longer a pressing demand for new compression technologies."
Of course, since Chiariglione wrote this article Sisvel launched patent pools for AV1 and VP9, so AV1's royalty-free status is certainly in doubt. Still, this does nothing to diminish AOM's role as major competitor to MPEG and the MPEG way of creating codecs. (Full disclosure: the author is consulting with Sisvel regarding these pools.)
To summarize, after the wildly successful MPEG-2 and AVC, Chiariglione feels that MPEG stumbled badly with HEVC and now faces a major competitor in AOM, partially as a result of HEVC's licensing issues. With this as background, it's not hard to see why Chiariglione was rankled by SC 29's decision to essentially replace him with Gary Sullivan, who's employed by Microsoft, one of the founding members of the AOM, a major competitor to MPEG whose members have suppressed the deployment of HEVC.
As an example, both Mozilla and Google are founding members of AOM, and HEVC still doesn't play in their respective browsers, even on systems with HEVC hardware decoders (Figure 4). Both companies could have easily piggybacked off existing hardware or software HEVC decode available on supported platforms likely without incurring a royalty since that's how Mozilla supports H.264 playback, first using hardware-based decoders available on the host system and later the Cisco OpenH.264 binary modules. Chrome doesn't support HEVC playback on Android even though the Android operating system supports HEVC playback in software.
Note that though the CanIUse data shown in Figure 4 indicates that Edge won't play HEVC, I tried HEVC playback in Edge on my HP ZBook running Windows 10 and it worked just fine. Not so, however, for either Chrome or Firefox.
Figure 4. AOM members Mozilla (Firefox) and Google (Chrome) still don't support HEVC playback in their browsers.
To be sure, Microsoft is a huge corporation, and standards and codec development are rife with frenemies, coopetition, and strange bedfellows. For example, Apple owns patents in the MPEG LA HEVC patent pool yet became a founding member of AOM (27 months after AOM launched). BBC owns patents in the same pool and is also a (non-founding) member of AOM.
Still, somewhere in Microsoft's Redmond campus there are clearly employees who wish MPEG would simply disappear, and Sullivan's ascension, despite his impressive and impeccable credentials, looks from the outside like putting the fox in charge of the henhouse.
I asked Chiariglione about this apparent conflict of interest, and he responded, "no comment." But he did share that "the request for candidates to SC 29 stated 'Candidates for this senior and 'neutral' international standards position must avoid bias and even the appearance of conflict of interest." As an objective outsider, it doesn't appear that Sullivan meets this standard, and perhaps this explains, in part, Chiariglione's feeling that despite obviously living on in form, MPEG is dead in fact.
Predictions for the Crop of 2020
So, we've perhaps explained Chiariglione's enmity, but what does it mean for the success of the three codecs MPEG will launch in 2020—Versatile Video Coding (VVC), Essential Video Coding (EVC), and Low Complexity Enhancement Video Coding (LCEVC)? As I explained in "Inside MPEG's Ambitious Plan to Launch 3 Codecs in 2020," MPEG seems to have had three goals for these codecs. First, to simplify the IP picture (and avoid another HEVC imbroglio). Second, to offer a royalty-free alternative to combat AOM and AV1. Third, to offer a CPU-friendly alternative to stop the burdensome computational complexity required by new codecs.
VVC was almost fully birthed before these goals were formed and is the natural successor to HEVC. According to Chiriaglione, "Most likely the number of VVC patent holders is much larger than HEVC's," making VVC, in essence, another HEVC waiting to happen. However, outside of ISO and MPEG, the Media Coding Industry Forum was formed in 2018, in part to "Provide a forum for, and encourage, the discussion of issues related to the licensing of intellectual property rights relevant to the deployment and use of these Standards, in the furtherance of the Purpose."
While the jury on VVC and the MC-IF is still out, Chiriaglione is not optimistic. Discussing VVC and the MC-IF in "A future without MPEG," he states, "So far, however, no concrete results have been made known outside, if we leave aside the events organized at conferences. MC-IF has 31 members, 7 of which are licensing entities (i.e. a little less than ¼ of all members). The 'industry' members account for just ½ of the HEVC patent holders. In these conditions, it is hard to believe that VVC will fare better than HEVC. It could very well fare worse, because VVC adoption in broadcasting will take years, if ever."
Moving on to codec number two, EVC has two profiles: the first purportedly royalty-free, the second encumbered by patents from four companies, which should simplify the licensing picture significantly. Regarding EVC, Chiariglione states "EVC is promising because it provides a quality that is comparable with or better than AV1, although less than VVC. EVC may have a chance if a licence will be published. However, this has not happened yet."
Finally, LCEVC is the low complexity alternative that can be implemented entirely in software, so is the only one of the three codecs that could possibly have an impact in 2020. LCEVC's IP is reportedly almost exclusively owned by V-NOVA, again significantly simplifying the licensing picture. Regarding LCEVC, Chiariglione comments, "A decent license for LCEVC could be a game changer and another threat to VVC that could mean additional pressure on VVC patent holders to provide a single and decent license."
Circling back to VVC, Chiariglione concludes, "The success of EVC could help VVC succeed. Seeing the threat of EVC (and AV1), VVC patent holders could get their act together and provide a decent licence. While I would welcome such a development, I consider it as having a low likelihood."
What's SC 29's take on all this? I asked Sullivan two questions, one about the potential conflict of interest and the second on whether MPEG plans to take future steps to improve the licensing posture of its codecs. He didn't reply to either, though this is a pre-holiday week (July 4) and he said that MPEG was "in the middle of meetings." If he does reply, we'll post his responses to the end of the article.
So, all we have is ISO's statement from this press release that "SC 29 regrets that it has been informed of the resignation of Dr. Leonardo Chiariglione, the former Convenor of SC 29/WG 11 (MPEG), and expresses its appreciation for his service in that capacity over a period of more than three decades. During this time, MPEG has become the pre-eminent standards group worldwide for the coding of audio and video and the specification of systems for media synchronization, presentation, storage and transport, as well as for other aspects of media coding and delivery."
Analysis
To repeat the Rethink Research quote from above, "industry adoption of these three emerging standards will depend on technical merit, ease of implementation, and business cases, more than what the new-look ISO SC29 says or does." There's so much that we don't know about performance and licensing that it's impossible to predict the success of any of the three, though it's hard to argue with any of Chiariglione's conclusions.
Had this reorganization happened ten years ago, at the height of AVC's prominence, a smooth succession would have been assured. But the HEVC licensing mess cast a pall on MPEG video standards while AV1 offered a bright shiny alternative, even if Sisvel's AV1 pool may prove that all that glitters is not gold. At least as it relates to video codecs, MPEG is no longer the unassailable monolith that it once was. On the other hand, Chiariglione may not have been able to lead MPEG out of the woods, as he has been railing against the ISO machine for so long that his voice was seemingly no longer being heard.
Clearly, Chiariglione has been a Jobs-like impresario who coaxed groundbreaking achievements from the video codec development community and he's parting at a time of great organizational need. The big question is whether Gary Sullivan turns out to be Tim Cook or John Sculley.
Industry Thoughts
We asked industry experts for their insights and opinions on Chiariglione's departure and the changes at MPEG, and we present them below in slightly edited form.
Thierry Fautier, Vice President, Video Strategy, Harmonic
What does it mean that MPEG was folded into SC29?
Not much; this is to me ISO politics
What was MPEG's impact in the audio/video codec space?
Huge, as this is the only standard that covers the compression of audio, video, and associated metadata for broadcast and OTT streaming. This is the source of trillions of [dollars in] revenues per years and probably employs millions of people.
What were MPEG's greatest failures?
Licensing terms, not included in the ISO framework, was a pain point. In AVC, the industry got their act together with the AVC Alliance (commercial group) and the MPEG Industry Forum (tech group) to fight Microsoft VC-1. In HEVC nothing was done, hence the issues. For VVC, MC-IF [the Media Coding Industry Forum] was created to facilitate VVC licensing, so there is some hope.
Larry Horn, President and CEO, MPEG LA
MPEG LA salutes the Moving Picture Experts Group (MPEG) and its founders Leonardo Chiariglione and Hiroshi Yasuda for leading the world into the digital media age and beyond. MPEG mobilized and inspired the world's brightest engineers with an uncompromising focus on quality and independence as the pathway to market acceptance, and leaving the market free to find its way, was determined to achieve it.
As high as MPEG's bar was, its achievements are higher. From this vision came digital video, audio, data, and systems empowerment delivering the world's information to home and phone.
MPEG LA is proud to have played an important role. Balancing reasonable access to MPEG users with reasonable return and incentive to MPEG inventors helped make the most widely used standards in consumer electronics history available to a world market. The market's creative energies unleashed commercially competitive, interoperable ecosystems of new industries, previously unimaginable products, and countless jobs that have generated trillions of dollars in product sales, connected countries, cultures, and individuals as one, and enhanced quality of life for billions.
Don't believe what you read. MPEG's doors are not closing. Its foundation is strong, its data flows, and its work will continue to leave a timeless mark.
Tariq Malik, Senior Product Manager, Global Eagle
Sad news about MPEG … Working in the IFE industry the requirement for MPEG video has been very prevalent. We have been dependent on the development of codecs from the early days of MPEG-1. The airlines utilize seatback VOD systems from the early 90s. The OEM manufacturers for these systems requested MPEG deliverables for their platforms and mandated a specific muxed transport stream file. Fast-forward to today and the support for these legacy systems is still required, but technology is moving on. Thanks to the MPEG team for all the hard work and contributions.
Chris Henderson, Manager, Digital Asset Technology at Warner Bros.
First off, I don't think I would have a career right now without several of the MPEG standards out there. MPEG-2 and MPEG-4 are amazing, far-reaching achievements. Without 2, digital broadcasting would have been proprietary and financial mess with an uneven quality in the industry. VOD would likely not exist or be very different. Without 4, OTT would be just as bad or worse. Real, Microsoft, Apple, and Duck would likely be our video overlords today, if not Sorenson and others.
Some say MPEG made piracy easier by making decoding standards. All tools can be used for bad and good, I say.
Yuriy Reznik, Technology Fellow and Head of Research, Brightcove, Inc.
No one has contributed to the success of digital media standards more than Dr. Leonardo Chiariglione. He created MPEG standards organization over 30 years ago, and lead it to define today's most essential media codecs and formats: MPEG-2, MP3, AAC, AVC, HEVC, DASH, etc. "
In my view, HEVC has already been broadly deployed. It is used in cable/broadcast, for delivery to Apple devices, and for delivery of UHD/HDR content to UHDTVs and other UHD-capable devices. Granted, its reach is not as broad and ubiquitous as the one of MPEG-2 or AVC, but it is very significant.
I do believe that AV1 will be used for significant amount of streaming traffic.
MPEG just got "absorbed" by the parent committee SC29, which most likely will continue to call itself "MPEG," but the original founder of MPEG – Dr. Leonardo Chariglione is out, and different sub-groups of MPEG – such as audio, video, JVET, etc., have become new independent working groups (WGs) under SC29, with their current chairs becoming conveners.
In my view, this restructure marks the end of an era for MPEG, the way we know it, where Leonardo was a clear visionary and driving force for most things that MPEG did. It started with the promise to end "format wars," and it delivered on itmostly—with clear successes of MPEG-1, MPEG-2, and MPEG-4/AVC (aka H.264). But MPEG's record was not spotless, as there've been times when proprietary codecs were taking over—such as RealVideo and Windows Media in late 1990s in streaming, and most recently a challenge from AV1, or Opus and Dolby's codecs in audio space. The recent array of MPEG video codecs—VVC, EVC, LCEVC—all being released in 2020 and with considerable overlap in capability also brings questions about the focus of this organization, and it … listening at all to needs of the industry. But hopefully with new structure and management new bright days and era of MPEG is still ahead.
[Editor's Notes: An earlier version of this story misidentified the current chair of SC 29. On 7/7/20, the article was edited to clarify AOM’s competitive stance vis a vis HEVC.]
Related Articles
Jan Ozer put EVC, VVC, and LCEVC through the paces, checking each for not only encoding quality, encoding complexity, and playback efficiency but also power consumption. Each one has its pros and cons; read on to find out how they all performed.
29 Dec 2021
The Moving Picture Experts Group, which had been responsible for developing standards and executing royalties for video and audio codecs, shut down last week. We'll be writing up an article on the implications for the industry, and we need your help.
09 Jun 2020
The pace of innovation is getting faster and the demands on video codecs are getting greater. MPEG's three-part plan answers questions of royalties, licensing, and computational efficiency. Meet VVC, MPEG-5 Part 1 (EVC), and MPEG-5 Part 2 (LCEVC).
15 Oct 2019
Thanks to a fractured HEVC licensing system companies no longer have the financial incentive to innovate, but Leonardo Chiariglione suggests steps to reverse the damage.
31 Jan 2018
Companies and Suppliers Mentioned