Best Practices for High-Availability Streaming at Scale
How do you maintain five-nines uptime for tentpole event streams reaching hundreds of thousands or millions of users? Steve Nathans-Kelly, VP and Editor-in-Chief of Streaming Media and Video Publishing Director at Information Today Inc., discusses this topic with Corey Smith, Sr. Director, Advanced Production Technology, CBS Sports Digital, Paramount, Joshua Johnson, Sr. Director, Solution Architects, EdgeNext, and John Petrocelli, CEO/Founder, Bulldog DM, in this clip from Streaming Media Connect 2023.
Nathans-Kelly begins by asking Corey Smith, “What would you say in your experience are the keys to creating high availability service with large-scale streams and tentpole events?”
Smith emphasizes that developing detailed customer analytics is a major key for providing seamless high-availability streaming at scale. “You have to evolve your analytics to figure out the customer experience,” he says. “Because if you don't understand what the customer's going through, you can never improve, right? If you never analyze the data, your customer base is constantly in this turmoil of, ‘Should I even bother with this thing anymore?’ Like if I go to a streaming service for live concerts and it's constantly falling over, why would I go back? Why would I spend money to go watch a streaming concert when I could easily go down the street and watch it at Five Point Amphitheater here in Orange County or go somewhere else.”
“I agree,” Johnson says. He points out that pulling real-time data is especially valuable. “It used to be that the data was so slow to get to us that acting on it was always detrimental. But now, actually, it is literally being fed in, and we're being able to make dynamic changes and choices as the situation goes on. But at the same time, if you look at the foundation of where you're at and the people…actually behind the scenes, that's key to it. Because if you don't have that foundational piece and they're not solid enough to actually provide the infrastructure or the inbound, then it doesn't matter how good the end is. You're not getting the initial piece in a qualified manner in any type of quantity [or] any type of quality.”
Smith says, “Oh, a hundred percent.” He also highlights that a wide view of audience location is essential to choosing the correct providers in the first place for large-scale streaming events. For the continental US and North America, he says, “You're looking at the typical providers, Akamai Limelight, etc. if you're going to do your CDN distribution. “But what do you do for a global international audience? What partnerships do you make there? How is the client telemetry influencing making decisions on constantly connecting the customer to the best stream possible? It's really the things around streaming delivery and load balancing across multiple CDM providers in different areas of the world that really hasn't changed much, from the three-tier architecture model of how an application is built in a data center somewhere…you have a load balancer. You have to spread out your application servers, you have to spread your middle-tier business logic, and your databases. It's basically the same thing: origin, origin, shield, edge…”
Johnson concurs with Smith about the importance of knowing the dominant players in a specific geographic region. “Which provider is the dominant player in that general area, who's got the infrastructure?” he says. “And are you actually relying upon them, or are you relying upon them and their partners, and do you actually know who their partners are? And then it becomes a question [of] do you see that partner as a value?”
Smith notes that, “A lot of those partnerships are only as good as their peering agreements, so at some point, it's still a business conversation on paper. Theoretically, you can get bits to anybody in the world. It's how you best optimize those routes.”
John Petrocelli says that while all of these aforementioned factors are crucial, many players who initially swept into the live music streaming realm during the pandemic were ill-prepared for the various technical hurdles that come with international-scale live streaming events. “I think about 100 to 200 companies went out and got venture funding and kind of ran into that market thinking, ‘Oh, this is no problem, I can pull this off,’” he says. “But what we saw was this is not an easy thing to do. And because you got some funding, I don't think you're going to figure it out in 90 days or even six months. And people weren't contemplating things like redundancy and serving to, first of all a big domestic audience, but potentially given who the talent was, it could be a pretty massive international audience. And it was a rude awakening for a lot of these companies.”
Petrocelli says that this reckoning ultimately was a test for who could be adaptable and wide-viewed enough to learn from mistakes and adjust quickly, and that this period served as a positive catalyst for improved tech and growth in the live streaming sphere. “I think that's consolidated back to reality, now that the people remaining in that space kind of understand the undertaking and what a best practice approach is to pull the experience off, but then to do it in real-time, and to scale it to a potentially pretty significant simultaneous audience.”
Learn more about high-availability streaming at scale at Streaming Media East 2023.
Related Articles
Any streaming workflow is only as strong as its weakest (or least-tested) link. The more massive the stream, unfortunately, the larger the opportunity and the smaller the margin for error. So, what do the experts say about the architectural demands and challenges of maintaining five-nines uptime and broadcast quality when the stakes are too high to let either suffer? And what solutions do they recommend?
19 May 2023
Delivering reliable streams is both more challenging and more critical when streaming at scale. So how do you develop workflows to ensure your large-scale streams won't fail? Experts from Dolby.io, TAG VS, Paramount, and Nomad Technologies weigh in on network stress-testing, monitoring, and more best practices at Streaming Media West 2022.
24 Feb 2023
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi's Brian Ring, Dolby,io's David Hassoun, Nomad Technologies' Adam Miller, Paramount's Corey Smith, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from Streaming Media West 2022.
30 Jan 2023
What are some of the biggest issues encountered when handling interactive streaming at scale? The widespread increase of highly interactive streaming such as auctions, sports betting, and gaming has put pressure on networks while increasing the need to integrate interactivity into more traditional methods of OTT content delivery.
01 Nov 2022
What are some of the biggest adtech challenges for live streaming at scale? Jarred Wilichinsky of Paramount talks about the ways his team works to mitigate technical issues, such as minimizing latency, load testing, and correcting audio levels, along with ensuring that the ads themselves meet acceptable legal standards and practices.
18 Oct 2022
Streaming Media's Tim Siglin and Bulldog DM's John Petrocelli discuss how tentpole live streaming events have changed over the last three years for the streaming pros who deliver them, the audiences that engage with them, and the brands that invest in them in this interview from Streaming Media East 2022.
06 Jun 2022
Companies and Suppliers Mentioned