Low Latency HLS Spec Nears Finalization
The need for a consistent low latency approach to HTTP-based segment or chunk "streaming" delivery has been discussed for more than a decade. As far back as 2011, with the ratification of Dynamic Adaptive Streaming via HTTP (DASH for short) by the Motion Picture Experts Group, a number of red flags went up warning of a potential "painting ourselves into a delayed delivery corner" with the 6- to 30-second delays that HTTP-based delivery entailed. These warnings were raised in Streaming Media magazine, on online forums, at Streaming Media shows, and at other industry events.
A decade on, it looks like those concerns have been heard. Starting in 2019, Apple—which had been part of the MPEG DASH ratification but had not implemented any parts of the international standard, relying instead on its MPEG-2 Transport Stream (m2ts) approach before shifting to byte-ranging addressing of fragmented MP4 (fMP4) files—began talking about the possibility of a low-latency version of its HTTP Live Streaming (HLS) delivery specification.
Apple wasn’t the first to discuss the low-latency approach to HTTP delivery, and several attempts have been made using DASH and the common media format (CMAF). But since it’s Apple, which dominates HTTP-based delivery with HLS, many in the industry waited to hear what Apple’s Roger Pantos had to say on the topic.
Pantos has had an Internet Engineering Task Force (IETF) specification on HLS in draft form for well over a decade. The first HLS spec, dated May 1, 2009, was authored by Pantos and has ever since affectionately been called "the Pantos spec" as shorthand for HLS recommendations and best practices on how to deliver HLS streams to Apple, Android, and other over-the-top or set-top box devices.
In 2017, after 23 different versions, each lasting about 5 months, the informational specification moved to the next step and became RFC 8216. The move to an RFC meant that requests for comments were the last step finalize the specification into a more formal IETF standard. Along with Pantos, the MLB Advanced Media group signed on as the only non-Apple authors of RFC 8216.
In 2019, though, Pantos had seen how other segment-based HTTP delivery approaches had begun to address low latency, and set about to redefine HLS in a low-latency version. He did so in a presentation at Apple’s World Wide Developers' Conference (WWDC 2019) where he introduced the low-latency HLS concept and laid out the requirements to consistently deliver at almost one-third the typical delay that HLS was known for. And those recommendations were spelled out on the Apple developer site for both the initial and now "almost final" version of low-latency HLS (LL-HLS).
The industry was swift to give feedback—both positive and, to a larger extent, negative—about the initial Pantos LL-HLS specification. As such, the latest version of the Pantos spec, from April 2020, takes a bit of a sharp turn from Pantos’ earlier presentations at WWDC and our own Streaming Media West 2019.
One big change is that the new spec is intended to completely supersede all non low-latency HLS specifications. In fact, Pantos notes in the April 2020 edition that "This document obsoletes RFC 8216."
Another big change is the newest Pantos spec drops the requirement that all HTTP servers used to deliver LL-HLS use HTTP/2 PUSH.
At Streaming Media West 2019, Pantos sang the praises of HTTP/2 as a way to send and receive multiple requests over a single HTTP connection, which had the potential to reduce the notoriously chatty HTTP / TCP thereby reducing overheads and request pipelining.
Yet industry feedback was that the existing infrastructure at many HLS delivery shops was based around the original HTTP server version, and that there were impractical assumptions on Apple’s part about the ease fo transition to HTTP/2.
Which leads us to the current state, as of September 2020. While the April 2020 version of the Pantos spec still recommends HTTP/2 PUSH, it’s moved the recommendation to Appendix B. In its place, there’s an explicit, required "hinting" extension, called appropriately EXT-X-PRELOAD-HINT.
Pieter-Jan Speelmans of THEOplayer explained this shift, and how it is actually beneficial in removing an additional roundtrip between servers and end-user players, but also noted Apple’s specification are always subject to change.
"Instead of using HTTP/2 push, Apple is instead adding a new tag called #EXT-X-PRELOAD-HINT," writes Speelmans. "With this tag, a server publishing a low latency HLS stream is required to announce the most likely location of the next media data which will become available. This allows a player client to perform a request, allowing the data to flow in as soon as the next part of a segment becomes available. This process can then be repeated, allowing the removal of additional round-trip time when loading new media data (which was the main reason to use HTTP/2 push)."
As part of the ongoing discussion about LL-HLS and how it potentially might tie into other work that had been done by DASH and the industry’s attempt at a common media format (CMAF), Speelmans will be part of a roundtable discussion on September 23. Along with Wowza and Fastly representatives, Speelmans will further explain Apple’s shift away from HTTP/2 and what it means for adoption of the overall LL-HLS ecosystem.
[Photo of Roger Pantos speaking at Streaming Media West 2019, by Eric Schumacher-Rasmussen.]
Related Articles
Rather than focusing on random tasks, this tutorial will walk you through the fundamentals of encoding for low latency HLS with FFmpeg, OBS, Mux Video, and THEOplayer
02 Mar 2022
The goal is to create, store, and distribute only one version of each piece of media. HTTP Live Streaming is the key to that kind of efficiency, Apple says.
14 Nov 2018
Companies and Suppliers Mentioned