What Is HEVC (H.265)?
This is another installment in our series of "What Is...?" articles, designed to offer definitions, history, and context around significant terms and issues in the online video industry.
HEVC (H.265) Executive Summary
H.265/High Efficiency Video Coding (HEVC) is the successor codec to H.264, which, like H.264, is jointly developed by the ISO/IEC Moving Picture Experts Group and ITU-T Video Coding Experts Group (VCEG). The primary goal of the new codec is 50 percent better compression efficiency than H.264 and support for resolutions up to 8192x4320.
HEVC Technology Background
By way of background, the ITU-T began development of a successor to H.264 in 2004, while ISO/IEC began working in 2007. In January 2010, the groups collaborated on a joint Call for Proposals, which culminated in a meeting of the MPEG & VCEG Joint Collaborative Team on Video Coding (JCT-VC) in April 2010, at which the name High Efficiency Video Coding (HEVC) was adopted for the codec.
In October, 2010, the JCT-VC produced the first working draft specification, with the Draft Standard -- based upon the eight working draft specifications -- approved in July, 2012. On January 25, 2013, the ITU announced that HEVC had received first stage approval (consent) in the ITU-T Alternative Approval Process, while MPEG announced that HEVC had been promoted to Final Draft International Standard (FDIS) status in the MPEG standardization process.
In essence, this means that the initial versions of the specification were frozen so that multiple vendors could finalize their first HEVC products. The current implementation includes a Main profile supporting 8-bit 4:2:0 video, a Main 10 profile with 10-bit support, and a Main Still Picture profile for still digital pictures that uses the same coding tools as a video "intra" picture.
HEVC will continue to advance, with work already starting on extensions for 12-bit video and 4:2:2 and 4:4:4 chroma formats, as well as incorporating scalable video coding and 3D video into the spec.
How It HEVC (H.265) Works
Like H.264 and MPEG-2, HEVC uses three frame types, I-, B- and P-frames within a group of pictures, incorporating elements of both inter-frame and intraframe compression. HEVC incorporates numerous advances, including:
Coding Tree Blocks: Where H.264 used macroblocks with a maximum size of 16x16, HEVC uses coding tree blocks, or CTBs, with a maximum size of 64x64 pixels. Larger block sizes are more efficient when encoding larger frame sizes, like 4K resolution. This is shown in Figure 1.
Figure 1. Larger blocks sizes enhance encoding efficiency. Images from HEVC webinar by Elemental Technologies, viewable on-demand.
More intra-prediction directions: Where H.264 used 9 intra prediction directions, HEVC can use over 35, adding more potential reference pixel blocks that fuel more efficient intra-frame compression (see Figure 2, from an Ateme presentation). The obvious cost is the additional encoding time required to search in the additional directions.
Figure 2. Searching is expanded to find more reference pixel blocks.
Other advances include.
- Adaptive Motion Vector Prediction, which allows the codec to find more inter-frame redundancies
- Superior parallelization tools, including Wavefront parallel processing, for more efficient encoding in a multi-core environment
- Entropy coding is CABAC only, no more CAVLC
- Improvements to the deblocking filter and the creation of a second filter called Sample Adaptive Offset that further limits artifacts along block edges
The HEVC (H.265) Results Please
There are a number of published white papers and presentation that focus on comparing the quality of HEVC vs. H.264 and MPEG-2. One of the most oft-cited sources is an article entitled, "Comparison of the Coding Efficiency of Video Coding Standards-Including High Efficiency Video Coding (HEVC)", which reports the results of both PSNR (Peak Signal to Noise Ratio) comparisons and subjective evaluations. The report looked at multiple scenarios, including interactive and entertainment applications.
For the entertainment-related comparisons, the study encoded multiple clips ranging in resolution from 832x480 (480p) to 1920x1080 (1080p). For the PSNR-related tests, the study encoded files using four different technologies, HEVC, H.264, MPEG-4 and H.263, until all files had the same PSNR value.
Table 1. HEVC efficiency compared to H.264, H.263 and MPEG-4 using PSNR values in entertainment applications.
The study then showed viewers multiple files encoded at multiple data rates with H.264 and HEVC and asked them to grade the results. From these tests, the researchers concluded, "the test sequences encoded with HEVC at an average of 53 percent lower bit rate than the H.264/MPEG-4 AVC HP encodings achieved approximately the same subjective quality."
Another article entitled "Subjective Quality Evaluation of the Upcoming HEVC Video Compression Standard", compared H.264 and HEVC using both PSNR and subjective comparisons. The study concluded:
The test results clearly exhibited a substantial improvement in compression performance, as compared to AVC... For the natural contents considered in this study, a bit rate reduction ranging from 51 to 74 percent can be achieved based on subjective results while the predicted reduction based on PSNR values was only between 28 and 38 percent. This difference is mostly due to the fact that PSNR doesn't take into account the saturation effect of the human visual system. PSNR also doesn't capture the full nature of the artifacts: AVC compressed sequences exhibit blockiness while HEVC compression tends to smooth out the content, which is less annoying. For the synthetic content considered in this study, a 75 percent bitrate reduction can be achieved based on subjective results while the predicted reduction based on PSNR values was 68 percent.
It's reasonable to be at least a little skeptical of such results, since the comparisons were largely produced by technologists contributing to HEVC effort, using encoders that have not been released for sale or (in most cases) even for general-purpose beta testing. Speaking on the condition of anonymity, one CTO of a major encoding vendor estimated that HEVC would enable a 30 percent reduction in file size at the same quality level at 1080p resolution, with further increases at higher resolutions.
Where will HEVC (H.265) Play?
Playback statistics are harder to come by. However, multiple companies have demonstrated HEVC playback on a tablet computer, including Qualcomm on an Android tablet powered by a 1.5 GHz Qualcomm Snapdragon S4 dual-core CPU. Note, however, that the video was only 480p in resolution, which makes sense for tablet display but is far from the 4K video HEVC is designed to enable. At the 2012 PBS Technology Conference, an Ericsson presentation estimated that encoding HEVC could require up to 10x more computational complexity with 2x-3x the complexity upon decode.
According to a report entitled "HEVC Decoding in Consumer Devices," senior analyst Michelle Abraham from the Multimedia Research Group estimated that the number of consumer devices that shipped in 2011 and 2012 that would be capable of HEVC playback with a software upgrade totaled around 1.4 billion, with over a billion more expected to be sold in 2013.
Figure 3. Unit shipment so HEVC-capable consumer devices (after software upgrades).
According to Abraham, in compiling these statistics she assumed that all PCs shipped in each year would be HEVC capable. The report also includes tables summarizing shipments of HEVC decode-capable tablets, portable media players, streaming media players, video game consoles, Blu-ray players, digital TV sets, and set-top boxes.
Despite this significant installed base, analyst Frost & Sullivan believes that HEVC adaption is at least five years away for consumer content services. Delaying HEVC adaption includes forces like the recent significant investment in AVC gear made by many pay TV operators, lack of broad-based support for HEVC in the OTT encoding and deployment ecosystem, and slow HEVC encode and decode chip deployment to preserve the profitability of existing AVC chipsets.
According to Frost & Sullivan vice president Dan Rayburn, the initial HEVC rollouts will occur in closed-loop solutions like enterprise video conferencing, ultra HD services in the far east, and low-bandwidth video-on-demand services worldwide, due to the potential cost savings associated with deploying lower bandwidth HEVC video. Rayburn expects satellite direct to home (DTH) service providers to start rolling out HEVC gear in the 2014-2015 timeframe, with some pilot digital terrestrial television (DTT) channels expected in 2015. Overall, however, he concludes:
While certain applications will embrace HEVC much sooner than the norm and HEVC encoding and decoding cores should mature by 2014, we expect it will be around 2017 before a comprehensive ecosystem of first-generation HEVC-enabled products will come to market. Furthermore, we expect AVC to remain in widespread use even in 2018, although it will definitely be considered a commodity technology at that point -- much as MPEG-2 is today.
HEVC (H.265) Royalties Will Apply
One factor that may slow HEVC adaption is uncertainty surrounding royalties. Like H.264, many of the technologies contributing to HEVC are patented, and patent owners will want compensation for the use of their intellectual property. In June 2012, MPEG LA, a leading packager of patent pools, and the organizer of the H.264 patent pool, announced a call for patents essential to HEVC, and a third meeting of the 25 respondents occurred in February, 2013.
However, according to MPEG LA officials, there is no set timeframe for the issuance of royalty guidelines or even assurance that a patent group will coalesce, since there are other packagers, or the respective owners could decide to assert their rights individually. Some market segments, most notably chip, encoding, and other infrastructure vendors, will likely press on with their HEVC-related efforts in the face of this uncertainty and simply reserve for potential royalties. However, other segments, particularly the distributors of free streaming content chasing the bandwidth savings afforded by HEVC, will almost certainly wait until royalties are known.
Competition In Sight for HEVC (H.265)?
These royalties make competitive technologies like Google's VP9 worthy of mention, particularly since Google added VP9 decode to beta versions of Chrome in December, 2012, along with a new decoder for audio streams encoded with the Opus codec. According to a Google presentation made at the Internet Engineering Task Force meeting in Atlanta in November, 2012, the goal of VP9 was similar quality as HEVC at lower data rates. In a requirements document available on the WebM website, Google stated that VP9 would not be shipped unless it improved quality over VP8 by 50 percent at a cost of only 40 percent higher decoding complexity, compared to 200 to 300 percent for HEVC.
As a codec, VP8 compared quite favorably to H.264, producing virtually the same quality at all tested data rates. However, multiple factors doomed VP8 to failure, including the fact that Apple refused to enable VP8 playback on iOS devices or in Safari, that Microsoft refused to include playback in Internet Explorer 9, that H.264 was a joint ITU/MPEG standard, and the fact that it came to market much later than H.264.
Many of these same factors are still in play: while Apple is expected to embrace HEVC, it's unlikely that it will support VP9 for the same reasons the company refused to support VP8; also, there are potential intellectual property issues. In addition, HEVC is a joint standard, so it already has a vast head start in terms of silicon support for encoding and playback. Both VP9 and HEVC should appear on the market around the same time, however, which may give VP9 a better chance than VP8.
Regarding the intellectual property issues: on February, 2011, MPEG LA released a call for patents for VP8, and as of July, 2011, 12 parties have stepped forward. Still, there's no further progress reported on the MPEG LA website, and our MPEG LA contact confirmed that there was nothing new to report.
Going Forward with HEVC (H.265)
Of course, as Rayburn from Frost & Sullivan points out, HEVC can't be deployed until the full encode/decode/transport infrastructure is in place. Some encoding vendors, like Elemental Technologies, have announced that all current encoders will be retrofitted for HEVC support via software upgrades sometime in the future. Before purchasing an enterprise-class or even desktop encoder going forward, ask whether or not the encoder will support HEVC going forward, and how much that support will cost.
Beyond this, precursors to significant availability of HEVC are market dependent. For example, in video conferencing, it's the availability of real-time HEVC encoders and decoders. In the streaming media space, the playback side is always the driving force, since few producers will encode to a new format until it's clear that it can play reliably for a meaningful group of viewers.
For general-purpose streaming, it's hard to get excited about HEVC without:
- A set royalty policy from MPEG LA
- Ubiquitous playback via HEVC in a player like the Flash Player or Silverlight Player (neither Microsoft nor Adobe responded to requests for information regarding if or when this might happen)
- The incorporation of HEVC playback into the iOS or Android platforms, either via an app or OS upgrade, and a clear indication about which of the installed base of devices will be HEVC-capable with these upgrades (Given the long-standing policy of both companies not to comment on future technologies, and the black ink-like opaqueness of their general-purpose dealings with the press, we didn't even ask)
- The availability of inexpensive decode silicon that can be incorporated into OTT set top boxes, or the announcement that some current OTT boxes can be retrofitted for HEVC playback via firmware or software upgrades
Conclusion
Between NAB (April, 2013) and IBC (September, 2013) expect a flurry of announcements of HEVC-related technologies and products. During that time, the first wave of HEVC capable encoders and decoders will also come to market, making it possible to gauge the technology's real world performance, benefits, costs, and the interoperability of the encoded HEVC streams.
Until then, as the Frost & Sullivan report suggests, it's most meaningful to consider HEVC adaption in a micro rather than macro sense, since the precursors and economic drivers are different in each market. While the inevitable general-purpose buzz around HEVC is bound to reach Kaepernick-like proportions, what matters is how these announcements impact the markets that you serve.
Related Articles
A high-level view of state of streaming media in 2019, and the concepts and and technologies that make it work.
12 Feb 2019
To celebrate Streaming Media's 20th year, here's an overview of 20 major patents and standards that have impacted the growth of the streaming media industry.
18 Apr 2018
Scheduled to be the first codec released by the Alliance for Open Media, AV1 is positioned to replace VP9 and compete with HEVC. While we don't know many details yet, the backing of the Alliance should give AV1 a significant competitive advantage.
03 Jun 2016
Streaming Ultra HD video to the home will require serious compression. Look for the HEVC codec to make the next jump in video resolution possible.
30 Dec 2013
AVC is a mature and proven choice, while HEVC is still in its infancy. Dan Rayburn challenges the common wisdom on HEVC.
02 Dec 2013
Everyone agrees that HEVC is the codec of the future, but an unclear royalty picture and a paucity of compatible devices means widespread adoption is a long way off.
16 Apr 2013
While previous jumps in compression technology have taken years to find adoption, we may see PCs and other devices that can play HEVC files on the market by the end of 2013
14 Mar 2013
MPEG DASH is the latest hot topic in the online video space. Here we break down what it is, and what its implications might be for video delivery in the future.
22 Nov 2011
Apple's HTTP Live Streaming (HLS) protocol is the technology used to deliver video to Apple devices like the iPad and iPhone. Here's a primer on what HLS is and how to use it.
14 Oct 2011
A look behind H.264, the world's most popular video codec, including encoding parameters and royalty concerns
04 Apr 2011
A high-level view of streaming media technology, history, and the online video market landscape
26 Feb 2011
Companies and Suppliers Mentioned