Testing HEVC: Please Take a Moment to Help Measure Performance
When Apple added HEVC to HLS in June 2017, many predicted it would open the floodgates of HEVC adoption. This hasn’t happened. You can find some clues as to why in the Harmonic-sponsored report entitled “The Impact of Apple’s HEVC Adoption: A Survey-Based Report.” Specifically, when asked about potential concerns regarding implementing HEVC in HLS, the majority of respondents mentioned content royalties, compatibility issues, and the ability to create one set of HEVC-encoded content to deliver to all HLS end points.
We can’t help on the royalty issue; unlike the MPEG LA and HEVC Advance patent pools, which won’t charge royalties for HEVC-encoded streaming videos, the “brain trust” at Velos Media, the third HEVC royalty pool, has obstinately resisted providing similar clarification. On the other two concerns, we can provide a bit of guidance.
Here’s the back story: To prepare for a workshop at Streaming Media East on producing HLS with HEVC, I created two encoding ladders that combined HEVC and H.264 content. One implemented Apple’s complete recommendation, which calls for complete duplicate H.264 and HEVC ladders. The other ladder used H.264 for low-resolution files (through 540p) and HEVC for high-res (720p through 4K).
I created the files using a combination of FFmpeg and the open source tool Bento4. Using a feature in FFmpeg, I burned the codec, resolution, and data rate into each file so you could tell exactly which file was playing within the adaptive group at all times. Initially, I hosted the files on a normal web server, but later Akamai agreed to host the files on its content delivery network to ensure that delivery issues weren’t hampering playback.
I posted links to the two HLS presentations in a LinkedIn article entitled “Please Help Me Test HEVC Playback in HLS.” In the article, I asked readers to test the streams on a variety of Apple devices and report their results to me. As an incentive, I promised to send all testers a summary report and instructions on how they could duplicate my test files using FFmpeg and Bento4. All told, readers sent reports from tests on more than 60 test devices—not as many as I would have liked, but certainly more than I could have tested myself.
What did we learn? First, performance and compatibility were excellent. There were no reports of any disruption when switching from H.264 to HEVC streams, or vice versa. Older devices that were incompatible with HEVC simply played the H.264 streams.
Another interesting finding was that, in most cases, the Apple endpoints wouldn’t retrieve a file larger than the vertical resolution of the display device. So the largest resolution I could play on my MacBook Air was 720p, even though I had the bandwidth to play the 4K streams. This makes perfect sense, and probably existed all along, but without the burned-in file descriptions, it was just difficult to tell.
One anomaly was that 2K/4K files were very seldom retrieved, even on devices that had both the resolution to display the video and the bandwidth to retrieve them. This is one reason we asked Akamai to host the videos. An Apple representative was one of our testers, and Apple is looking into this.
What did we learn about the two ladders? There were several instances where a device retrieved a different maximum quality file from the two encoding ladders, like 1080p HEVC from the complete ladder, and 360p from the split ladder. But these findings weren’t consistent, so I’m hesitant to make any firm conclusions without additional work.
This is where you come in. The files are still up in the LinkedIn article, along with instructions on how to test and what information to send. Send me test results and I’ll send you the instructions on how to reproduce my test files. One thing I think everyone involved learned is that burning the codec, resolution, and data rate into the files provided a clarity simply unavailable without this data. FFmpeg provides a simple way to do this, and my article will help you use it in your own testing.
[This article appears in the July/August 2018 issue of Streaming Media magazine as "School Supply List for Teaching With Video."]
Related Articles
Demos and conversations at Streaming Media East showed that HEVC is an excellent technology, and there's real anger around its disfunctional royalty policies.
22 May 2019
A new research report highlights how H.264/H.265 workflows have become integrated, as well as discussing VP9 and AV1 as possible AVC replacements
28 Jun 2018
H.264 still leads, HEVC is starting to gain traction, and AV1 had its coming-out party. To add to the confusion, other codecs offer alternatives to all three. Jan Ozer makes sense of all the codec news from this year's NAB.
17 Apr 2018
It's not easy, but it can be done, as long as you have the right tool and know the tricks. Here are eight steps to success using Softvelum Nimble Streamer.
22 Mar 2018
Companies and Suppliers Mentioned