How to Build Unbreakable Live Streams at Scale
What are the best workflows to develop in order to deliver reliable live streams at scale? Adam Miller, CEO, Nomad Technologies, David Hassoun, Chief Technologist, Dolby Cloud Media Solutions, Dolby.io, Peter Wharton, Chief Strategy & Cloud Officer, TAG Video Systems, and Corey Smith, Sr. Director, Advanced Production Technology, CBS Sports Digital, Paramount discuss how their organizations work to provide seamless live scaled streaming in this panel from Streaming Media West 2022.
“Live event streams can break down anywhere along the line,” says the moderator, Eric Schumacher-Rasmussen, Chair, Streaming Media Conferences, and CMO, id3as. He asks Adam Miller of Nomad Technologies to kick off the discussion. “What do you see as the key touch points to focus on when building a workflow for reliability?”
Miller emphasizes that there are countless possible failure touch points with live streaming at scale. He says that at Nomad they try to break these possibilities down in an organized chart that narrows the categories to focus on into three parts. “What is the cost to make this redundancy better?” he says. “What is the effort level to make it better? [And] where do you put your budget and time and energy? And when you do that, you're going to find that out of those hundred breakage points, probably about six really become most prominent. And that's going to be where you can put your energy to help increase the reliability rather than someone saying, ‘Hey, I need multiple encoders.’ Well, put the energy towards where it's going to matter the most. So that's what we normally do when we start looking at redundancy.”
Peter Wharton of TAG VS notes that most people outside of the industry have little idea of the vast complexity of these live streaming systems. “You need to monitor the whole thing, especially if you're doing ad hoc building of systems or you're setting up a live system on demand,” he says. “Because then you have all these moving parts and now, you're just building something instantaneously. So you have to make sure all those moving parts are working. It's not something you've had running for months, and you know how it works. But at the same time, every point in that workflow has a different value in the workflow.”
Wharton says that it’s a matter of adjusting for priorities. “As you go further down that food chain, every touch point which is going to impact the viewer is impacting fewer and fewer viewers,” he says. “So, therefore, you have to also make sure that you are monitoring in a way that you're actually adjusting the cost of your monitoring to the value of the content at each point of the workflow. And that's a challenge to get all that right but still make sure it works everywhere. Because you can't spend the same amount of money for some edge of a CDN that affects one region that you do at the core.”
David Hassoun of Dolby.io highlights the crucial need for validation testing of systems. “Every one of those key breakpoints, you really need to be able to hammer that, and you're going to have assumptions of how it can break, and those are the ones that you can at least test,” he says. “But also try to look beyond what you didn't expect and understand what your different failure scenarios are, how you're going to transition, and make sure that that is smooth. It needs to be really worked through. It becomes muscle memory into all the teams that this is going to affect and what's going to happen in those situations.”
Miller mentions what he calls the “golden rule” of leaving what is working alone and not over-tinkering. “When it comes close to time to deploy, don't touch anything!” he says. “People forget that, and they think, “Oh, let's just change out this encoder at the last minute.’ If you're trying to reliably distribute something, touch as little as possible. And if you're going to do it ten times, don't touch it at all. Build it once, and then just leave it and reuse it ten times. I find that a lot of people forget that golden rule of ‘Don't be changing things two minutes before.’”
Corey Smith of CBS Sports and Paramount stresses that “There’s a real art to figuring out both the telemetry coming in from the outside, [and] also how the customers experience the event.” He points out that CDNs are not always the most accommodating when stress-testing all possible failure points.
“When I was at Xbox,” he says, “we were doing a lot of things to scale large customer events, whether it's the E3 Keynote or other Mars Moon Lander or whatever that happened to be on the console that particular day. But we took a lot of pride in actually testing to failure. But when you go to an Akamai or you go to a Limelight or some other CDN provider and say, ‘Hey, I'm going to stress test my network out. Can you help support two and a half to three terabytes per second of traffic because I want to scale to two and a half million concurrence? How do you tell a CDN provider that's where you want to test to? They're going to laugh you out the door and say we can't absorb that on our network. But a lot of it is knowing where your traffic's going, being multi-CDN agnostic or being agnostic to a single CDN provider, but also getting feedback and telemetry from your clients so now you can make actual intelligent traffic decisions in near real time to what your customers are actually seeing. So if you are pushing a bit rate of 10 megabits plus, so you're doing 1080p Plus video at high quality, you don't exactly own all of the edge ecosystem that you're actually deploying to. You just own the video player tech, so to speak. You kind of want to get that feedback so you can say, ‘Hey, this city and provider in this particular part of the world is not doing so well and this is the CDN provider we need to start shedding traffic to. You have to build those kind of telemetry systems into the actual base of the application because you can't just flip it off and flip it back on during a live event. You have to be able to ebb and flow your traffic across the global internet and be able to do it seamlessly.”
Learn more about live streaming at scale at Streaming Media East 2023.
Related Articles
How do you maintain five-nines uptime for tentpole event streams reaching hundreds of thousands or millions of users? CBS Sports Digital's Corey Smith, Bulldog DM's John Petrocelli, and EdgeNext's Joshua Johnson offer tips and best practices for successfully scaling up your streams in this clip from Streaming Media Connect 2023.
10 May 2023
How can live video help advertisers meet the challenges of a media consumption ecosystem perpetually in flux, such as cord-cutting, brand safety, ad-blocking, and more? BulldogDM's John Petrocelli, who works with many of the biggest brands in the business, and CBS Sports Digital's Corey Smith discuss how to make streams work for brands so the streams thrive and the brands and viewers keep coming back in this clip from Streaming Media Connect 2023.
31 Mar 2023
End-to-end workflows for live streaming at scale are complex affairs, with troubleshooting challenges at each stage when delivery breaks down. An expert panel from Paramount, Amagi, TAG VS, and Nomad Technologies discusses key best practices for troubleshooting large-scale streams when the pressure is on.
22 Feb 2023
How do you meet the challenges of monitoring stream performance and quality for dozens or thousands of live streams at once? Experts from Paramount, Dolby, TAG Video Systems, and Nomad Technologies discuss multistream monitoring tech and strategies in this panel from Streaming Media West 2022.
15 Feb 2023
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi's Brian Ring, Dolby,io's David Hassoun, Nomad Technologies' Adam Miller, Paramount's Corey Smith, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from Streaming Media West 2022.
30 Jan 2023
What are the most dynamic approaches for localizing live event streams? Dan Turow of Evertz and Marisa Elizondo of fuboTV talk about the ways their organizations are working to best integrate local and user-preference-based content and experiences into live event streams.
15 Nov 2022
Companies and Suppliers Mentioned