Know Your Tech for Low-Latency Streaming
What are some of the key enabling technologies of low-latency streaming? Robert Reinhardt, CTO, videoRx, goes in-depth with this topic during a presentation from Streaming Media West 2022, covering areas such as server ingest and client delivery protocols like WebRTC, NDI, RTMP, and HLS.
“The technologies that we have, when it comes to low latency include ingest protocols, or muxers--they're not necessarily synonymous, but WebRTC is more of a transport layer,” Reinhardt says. “It's not necessarily a specific way to mux audio and video, although, like HLS, it's a wrapper that muxes audio and video together. But when it comes to server ingest, we've got low-latency protocols. NDI, of course, had a huge boost during COVID because a lot of production workflows went to full NDI--people using NDI within their own private LANs on the cloud, and using NDI, of course, on location, or coming out of Teams or Zoom. NDI is a very popular--license-free, for the most part--ultra-low-latency way to get video around that can be uncompressed or compressed, depending on what flavor of NDI you're using.
“RTSP/UDP, that's used more with the kind of security cameras, traffic cams out there. Again, we get all sorts of audiences here. I'm not presuming you all come from media and entertainment. So when it comes to municipality live streaming, I do work for various municipalities in British Columbia, and across the United States I've worked with the city of Colorado Springs, a few municipalities in California, working specifically with their traffic cams. And they want low latency on that too. Most of those cameras are IP cameras that are Axis or another vendor like Axis, and those are all RTSP pulls from those cameras into an infrastructure that hopefully won't add much more latency on top of it. And of course, RTMP Flash has been around for a long time and is gone, but RTMP is its legacy that even Facebook and YouTube today still use for ingest.
“So if you're doing a live stream on those platforms, you probably already know the latency is pretty high because we can't do RTMP out anymore. We could do RTMP in. Gradually, that's gonna be phased out, but because there's such an infrastructure investment on top of RTMP, I don't think it's gonna go away next year or the year after that. RTMP is probably gonna be a legacy protocol that sticks around, and it can actually be pretty low latency--even ultra-low latency, and I'll go back to that slide in just a second that I skipped to from Wowza that basically refers to tuning all of these different protocols and muxers that might be out there.
“For server ingest, I put the popular ones on the left-hand side. It's missing SRT, which should be there. SRT, of course, is pushed by Haivision. It's open source, and SRT is quickly becoming a popular replacement to RTMP, particularly if you're using a codec that's not H.264. If you want to start using modern codecs like HEVC or AV1, you're not gonna be able to use RTMP very easily to do that. Yury Udovichenko of Softvelum has adapted an RTMP variant that will work with other codecs, but that's very specific to the infrastructure that his company's been working on.
“Client delivery of course is how we're consuming these streams, not just how a server might be talking to a point of origin or a remote location for client delivery. We've got our standard HTTP delivery with HLS-DASH and CMAF variant of that. WebRTC, of course, is there for client delivery as well.
“We still have web socket services out there. nanocosmos, whose tagline was ‘around the world in about a second,’ used web sockets for that playback mechanism. So web sockets wasn't necessarily an end-to-end delivery for them. It was a client delivery that was easier to scale than WebRTC. So you still see some web socket implementations. Web sockets have been around in browsers for a long, long time, and it's just a generic socket. You could send whatever you want over a web socket: data, audio, video. It's not necessarily an easy protocol to work with because, again, it was not necessarily designed for sending video and audio and video around the web like WebRTC was.
“Apple HLS has around 30 seconds of latency. Typically they put it down in the 18+ seconds of latency column. I would say your average HLS latency is 30 seconds, mainly because people are using ten-second chunk sizes and a three-chunk playlist. So you multiply 3 chunks times 10--not including any kind of delays between those chunks being delivered across CDNs--and you're looking at 30 seconds. It's not too hard to reduce that latency down to six seconds. You'll see under HLS Tuned, Wowza put it just after five seconds and you could get a two-second chunk size and keyframe interval times three. Again, if you start minimizing your chunk size, just multiply it by the number of chunks that are listed in the manifest, and that gives you your average latency. You'll need to add some time to that too, just because of transports between edges and your origin potentially. But I would say if you have a two-second segment size and it's on a playlist that's repeated three times, then you're probably looking anywhere from 6-10 seconds of latency in a tuned playlists like that.
“And that's not hard to do. Anyone who's got a media server can tune their packaging to that and not have to go through many jumps in not too many hoops to do that. As we get closer though to this sub-one-second latency, we start to get into more different technologies like WebRTC, you can see is first and foremost in this near real time. We're closer to 250 milliseconds. Generally speaking, I think most people are looking to achieve under 500 milliseconds, if not under 300 milliseconds of latency when they're using WebRTC. And of course, there's a cost associated with that. WebRTC doesn't scale as easily as any of these http methods of delivery, so you're gonna have to budget accordingly, whether it's building out your own WebRTC infrastructure using someone else's.
“We now have Low-Latency HLS (LL-HLS) that's been out for a number of years. I remember when Roger Pantos came to Streaming Media West 2019. He was talking about Low-Latency HLS for the very first time at the conference. And so we've had some time over COVID to see how that's gonna evolve. The original Low-Latency HLS spec had some HTTP version 2-specific PUT commands in it, that they've since removed it so that CDNs don't have to use it. And instead they're using this preload hint that you can put into manifests that are specifically Low-Latency HLS. And of course you could tune any of the others, like RTMP.
“Back in the days when HQ Trivia was a massively popular trivia game, I had a couple of clients that were trying to get on that same bandwagon, and we were using RTMP libraries in native apps to play RTMP in a smartphone app. So if you're building your own customized playback technology, then you have a lot more options still if it's not gonna be strictly within the domain of the browser. So RTMP could be an option for playback if you were building a custom environment for it these days.
“I wouldn't put too much weight into RTMP playback just because we've got options that are a lot more mature now, like WebRTC. Even just a few years ago, WebRTC didn't have the kind of cross-platform and cross-browser acceptance or standards that we have now. Now you don't have to worry so much about H.264 vs. VP8. Those are still codecs that are in play that might need some transcoding back and forth, depending on your workflows. But it's come a long way, and again, COVID accelerated that."
Learn more about low-latency streaming at Streaming Media East 2023.
Related Articles
What are some of the best practices for evaluating streaming tech vendors? Nadine Krefetz of Reality Software discusses the various factors at play in making these decisions with LaShawn McGhee of Revry, Rob Collins of Starz, and Anil Malhotra of Bango in this clip from Streaming Media Connect 2023.
11 Sep 2023
CDN77's Juraj Kacaba sits down with Tim Siglin to discuss low-latency streaming and the edge in this interview from Streaming Media East 2023.
16 Jun 2023
Tim Siglin of Help Me Stream Research Foundation sits down with Daniel Alinder of Vindral to discuss latency, sync, 8K, and Vindral in this Streaming Media East 2023 interview.
07 Jun 2023
How much does consumer feedback impact streaming app and platform innovation and evolution? Is there an influencer niche for everyday streaming consumers? Fandango's Rema Morgan-Aluko, LG's Matthew Durgin, and Vizio's Greg Barnard discuss how seriously they take user feedback and how they apply it in this clip from Streaming Media Connect 2023.
24 Apr 2023
End-to-end workflows for live streaming at scale are complex affairs, with troubleshooting challenges at each stage when delivery breaks down. An expert panel from Paramount, Amagi, TAG VS, and Nomad Technologies discusses key best practices for troubleshooting large-scale streams when the pressure is on.
22 Feb 2023
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi's Brian Ring, Dolby,io's David Hassoun, Nomad Technologies' Adam Miller, Paramount's Corey Smith, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from Streaming Media West 2022.
30 Jan 2023
The best chunk size for low-latency streaming is dependent on a number of factors based on different use cases, and there is often a need for some compromise and tradeoffs in quality or speed. Nadine Krefetz, Consultant, Reality Software, Contributing Editor, Streaming Media, asks three industry experts what their chunk size preferences are for their requirements.
26 Sep 2022
Interactive streaming is the future for high-quality ultra-low latency applications that will unlock unique and unprecedented experiences for users
19 Aug 2022
CNN+ Live Operations Manager Ben Ratner discusses how even "ultra-low latency" complicates hybrid (cloud and on-prem) workflows in this clip from Streaming Media Connect 2022.
18 Apr 2022
Andy Howard and Tim Siglin discuss the importance of low latency in video conferencing and how popular video conferencing tools like Zoom, Teams, and WebEx use scalable video encoding to guarantee access and lower latency in this clip from their panel at Streaming Media Connect 2022.
04 Apr 2022
Millicast Chief Revenue Officer Ryan Jespersen discusses how WebRTC reduces streaming latency in this clip from Streaming Media Connect 2022.
16 Mar 2022
Companies and Suppliers Mentioned