The Last 24: Preparing and Troubleshooting for Successful Live-Streamed Events

The last 24 hours before any live event are probably more stressful than the event itsel for the streaming and production crew. Hopefully you’re aware of everything that needs to get done, and you’re making it so, but there’s always a nagging feeling that you forgot one critical item, or that something preventable will go wrong before or during the event.

So we decided to ask a range of experienced live event producers what they focused on during the last 24 hours, and to share some practices and techniques that allowed them to avoid, or workaround, problems that would otherwise derail an event. Here are their answers; some presented in question-and-answer format, others as a narrative.

Micah Gordon, nugs.net
Last 24 hours: Checking Bandwidth

nugs.net is the leading source for official live music from some of the largest touring artists in the world, including Metallica, Bruce Springsteen, Pearl Jam, Phish, and the Red Hot Chili Peppers. nugs.net’s platform includes downloads, CDs, webcasts, and subscription streaming services, delivering exclusive live content to millions of fans on a daily basis. We got answers from Micah Gordon, Director of Production (Figure 1, below).

Figure 1. Nugs.net’s Brad Serling and Micah Gordon at a Phish webcast in Mexico in January 2016

As an overview, nugs.net handles the encode, ingest, and distribution of the live streams, not the actual production. nugs.net takes HD-SDI feeds from the video mixer into two custom encoders using the vMix live production software, and sends two 1080p streams to separate Wowza Streaming Engine instances running in the Amazon cloud. There, the incoming streams are translated into seven streams, including one audio-only stream, which are sent to third-party CDNs like Akamai for adaptive delivery.

SMP: What you typically focus on in the last 24 hours?

Gordon: Depending on the artist that we’re working with, we are usually focused on making sure that the bandwidth at the venue is sufficient. We’ll run several test streams on ideally two separate circuits. If we’re using satellite internet, we’ll get that set up to make sure we have clear line of site so that we can maximize our upstream bandwidth.

SMP: What’s the biggest disaster you can share that has occurred during a live event?

Gordon: A rainstorm took out the server closet at a venue and we lost our internet connection when we had thousands of people streaming. I ended up tethering my phone to our encoder and was able to stream the last hour of the show through that. We now carry satellite backup with us after that rainstorm incident so that we always have two outbound circuits for our primary and backup streams.

Robert Feldman, Total Webcasting
Last 24 hours: Worrying about Traffic

Total Webcasting provides cost effective, high-quality, full-service and self-service webcasting solutions to businesses, nonprofits, and educational institutions worldwide. Robert Feldman is president and founder.

SMP: Describe your typical webcast.

Feldman: Our typical webcast is held in a conference center or university, though occasionally we will webcast from a corporate or law office. Many of these are pay-per-view for continuing education credit so the viewer is not very tolerant when something goes wrong.

SMP: What’s your connectivity strategy?

Feldman: Most of the time, we rely on whatever internet we can get from the venue. In typical law or corporate offices, the biggest challenge is getting around firewalls. Flash streaming sets off lots of alarms. In hotels we push back when asked to use wireless, and insist on a hardwire. In the last couple of years, we have seen few issues related to internet connectivity. Also, since our outgoing stream is 1.5Mbps, that isn’t a heavy lift anymore.

SMP: What are your production and encoding tools?

Feldman: We built our own portable production system that incorporates multiple HD robotic cameras, switching, and lower-third graphics (Figure 2, below). We use the Open Broadcaster Software (OBS) running on a real quad-core laptop (Gigabyte or some other gamer type) and when streaming and recording at 1.5Mbps at 720p the CPU does not go above 40% (the same setup using Flash Media Live Encoder would be closer to 80%). The newest version of OBS is the best encoding software we have ever tried. We also use an Epiphan Pearl as a backup encoder and recorder.

Figure 2. The Mediacart at the heart of the Total Webcasting solution.