The Last 24: Preparing and Troubleshooting for Successful Live-Streamed Events

Shawn Lam, Shawn Lam Video, Inc
Last 24 Hours: Debugging Platform Issues and Client-Related Problems

Shawn Lam Video, Inc, a Vancouver, BC video production company, provides corporate video communications and high-definition webcasting services. Shawn shared a recent webcasting experience for a corporate client.

SMP: Give us an overview of the event.

Lam: The event was a hybrid webinar with 300 external viewers and just over 100 live viewers, using Adobe Connect as our webcast platform.

SMP: Describe the technical setup.

Lam: We had two Sony HD video cameras, one a medium follow and the other a wide shot, and the video switcher was the Blackmagic ATEM 1 M/E. Being an Adobe Connect webinar, our video pod was limited to 240 P and we selected the highest quality for both the computer presentation and video pods. I believe the total resolution of Adobe Connect webinars is 1024x768 at 20fps, with a bitrate of under 1Mbps. Compared to an HD webcast, resolution and bitrate isn't as important a consideration.

The presenters were using PowerPoint slides, which you can upload into Adobe Connect, where the presenter or webcaster can advance the slides within the control panel. Or, you can input a feed from the presenter’s computer, but this gets tricky if they have video or audio to playback because you have to play that for the live audience, without feeding back the audio into the Adobe Connect stream.

SMP: So, what went wrong?

Lam: From the moment we arrived the day before the event, I had a feeling we were going to have issues. The Internet at the venue was down for hours, a remote presenter was travelling and we could not rehearse their remote connection or audio and video workflow. One of the presentations that we had wouldn’t open in the latest version of Microsoft PowerPoint on our notebooks, while another presenter didn’t want to send their presentation in advance. This was in addition to us experiencing frequent video drop-outs in the Adobe Flash-based Adobe Connect.

SMP: What a nightmare. How did you spend the last 24 hours?

Lam: We addressed each problem as best we could. Our backup for connectivity was to tether each computer to a cellphone or LTE stick, but the venue got back online after rebooting their main modem and all routers. Unfortunately, this delayed our testing and almost completely eliminated our rehearsal time.

We solved the PowerPoint compatibility issue by uploading the presentation to Adobe Connect and then manually advancing the slides in Connect to follow along with the local projector connection. The remote presenter’s flight was delayed, so we didn’t have to pull off the remote video and slide connection with them, after all, instead, we had a local presenter present to the remote presenter’s slides and later had the remote presenter join for only the Q&A audio.

The one issue we couldn’t solve was the intermittent video dropout issue in Connect, which we later discovered was caused by out-of-date Adobe Flash Players on both my primary and backup systems. Intermittent failures are the hardest to diagnose, and had we not lost that rehearsal time due to failed Internet connection, I’m sure we could have isolated and fixed this problem before the event.

In the end we had a massive debrief with the client to go over all that went wrong, how we fixed it on the fly, how we could prevent the issue the next time, and to reinforce why we have deadlines and guidelines that have to be followed. That said, the reality is that you often only get the opportunity for a short rehearsal with stand-ins, the presentations always get changed at the last minute, and some piece of technology crashes. It’s how you handle these situations that differentiates producers who can roll with the punches and still stay on their feet and those that are down for the count (Figure 4, below).

Figure 4. Shawn Lam during happier times than the scene described above

Alden Fertig, Vice President, Product, Ustream, an IBM Company
Last 24 hours: Bandwidth, and then everything else

Ustream, now an IBM company, is a premier live event service provider. Alden Fertig, now VP Product (Figure 5, below), came up through the ranks, with stops as product manager and director of customer success. Fertig has produced many of Ustream’s live event training materials, and too many events to count. He shares his experience with a recent event that Ustream was involved with, IBM’s Interconnect Conference in Las Vegas, in February 2016.

Figure 5. Alden Fertig of Ustream

SMP: Tell us about the event.

Fertig: IBM’s InterConnect 2016 is IBM’s annual conference with dozens of sessions and over 25,000 attendees.

SMP: What was the scope of the production?

Fertig: There were three different live channels covering the conference over the course of 5 days. Each session was also recorded and available later as on-demand content.

SMP: Describe the encoding setup.

Fertig: We used a custom 8-core PC box equipped with a Blackmagic Design DeckLink Quad for capturing the SDI signal from the video mixer. We used Telestream Wirecast to encode the stream to 720p at 1.5Mbps, which we transmitted to Ustream for transmuxing and distribution. Since we were on the expo floor, we used a leased, wired connection with at least 20Mbps, which is our standard configuration.

SMP: Why 720p at 1.5 with so much bandwidth available?

Fertig: It's a fair question. We could have gone higher, but our experience with these connections is that you don’t want to push the limits. Having lots of headroom is easily the difference between a stream staying solid or disconnecting or having any buffering. Also, the nature of the content (talking head, low motion) didn’t require a significant amount of bitrate to look great and then you can keep even the highest bitrate within reach of those on slower connections vs. pushing it.

SMP: What did you focus on the last 24 hours?

Fertig: The last 24 hours (and this is typical for any good event) is all about testing, testing, testing. Checking everything to make sure the end-to-end experience is flawless.

Constantly checking on-site bandwidth at the venue is extremely important. If the bandwidth connection is strong, you’ll want to test audio, frame rates, and deinterlacing (if applicable). Check all locations where the player is embedded, and ensure that it works well across all devices (iOS, Android, etc). If possible, make sure that adaptive bitrate delivery is working as expected.

SMP: What’s the biggest disaster you can share that has occurred during a live event?

Fertig: Totally unrelated to this event, we did a stream many years ago of an awards show. There was a televised portion and a live-streamed portion. The live stream was supposed to continue with post-event content, but went black shortly after the conclusion of the main event. After much scrambling, we learned that the house A/V people just started striking everything once the main event was over and someone had unplugged a lot of cables.

We've seen similar situations with satellite feeds running out of time. But, as in life, we turned this into a valuable learning experience. Now we insist on coordinating with the teams on the ground early in the planning to make sure they know about the web stream and the need to support it post-live event. Sometimes this requires budgeting for extra time for those other teams not directly related to the web stream.