Behind the Scenes at Facebook Live: Architecting a Success
Facebook Live is one of the most significant streaming media success stories to date, a platform that almost immediately became essential to product marketers large and small and has shown no sign of slowing down.
Seeking the story behind the product, we spoke via email with Dave Capra, engineering manager for Facebook Live, and Abhishek Mathur, director of engineering for video and live infrastructure. Here’s the Q&A.
STREAMING MEDIA: What were the key business reasons for creating Facebook Live?
CAPRA (right): The key focus for us was really about finding new ways to bring people together. We launched Facebook Mentions in July 2014, an app for public figures and celebrities to connect authentically with their fans from their phones. We saw good usage and retention, but we were missing the truly authentic connection with fans. So we started prototyping interactive video ideas that could help here, and Live was the clear favorite.
SM: How did Facebook perceive the market prior to the start of product development? There was YouTube Live, and lots of paid services (Livestream, Ustream). What was the market gap that Facebook perceived?
CAPRA: We knew mobile video was becoming a huge thing, particularly after the ALS Ice Bucket Challenge went viral on Facebook, but we didn’t look at it from a market perspective too much. We were mostly focused on what the best thing we could build for our users was, and knew Live could get a celebrity “face to face” with their fans. So video was just another way to connect people in an interactive, authentic way.
SM: When Live started, was it perceived as a make-or-break service feature, or a nice-to-have feature? Has that evolved?
MATHUR (right): Facebook has always been a social place, and through Facebook Live, we learned how truly social the video experience on Facebook can be. For example, you can talk about the video with friends and the community in real time, interact with the broadcaster, or even interact with celebrities.
When it started, it was a feature within a suite of features that lived inside of the Facebook Mentions app. Early on, it became clear that Live would stretch Facebook’s infrastructure in unprecedented ways. It’s since then expanded across a growing number of really meaningful use cases and become a core way of sharing important moments on Facebook.
As an example, while broadcasts from public figures and publishers have generated some of our largest audiences, the vast majority of live videos come from regular people. We started to see people forming communities around video genres or creators. Video behavior on Facebook is, above all, community-driven. On Facebook, people discover video through community. They watch video as a community. And they discuss video as a community.
Live has really taken off, becoming a significant feature within the overall Facebook platform. There have been more than 3.5 billion broadcasts—and 150 billion reactions to Live broadcasts. In fact, what we have found is that live videos generate an average of six times as many interactions as regular videos.
SM: Was there a key whiteboard session where the product was conceived?
MATHUR: The origin story was actually from a hackathon, where a couple engineers prototyped what having “Live Profile Pictures” would feel like, using a webcam. When we became serious about enabling Live at scale, a group of engineers worked together in a hackathon to build a high-scale, low-latency infrastructure prototype that powers the current Live system.
SM: What were the key technical challenges, and how did Facebook overcome them?
MATHUR: We worked on three big challenges. First was scale. We correctly anticipated a large number of concurrent broadcasters and viewers (aggregate and on a single stream). We did a lot of work to handle the spike-y viewer/broadcaster load of Live where unexpected events could become big. The article “Under the Hood: Broadcasting Live Video to Millions” (go2sm .com/fblive) describes the work we did to handle these “thundering herd” problems. With Live growing significantly in volume/quality and duration and expanding to products like Instagram, we are investing deeply in our compute infrastructure to scale and be efficient.
The second major challenge was reach. Our broadcasters and viewers are on varied connectivity and devices. Interestingly, poor connectivity is not just an emerging market issue; it can happen anywhere. Live moments can happen any time, and the connectivity in those places may not be ideal (weather events, extreme sports, stadiums, concerts, etc.). Our team has done a lot of work to make broadcast and playback successful in low/variable connectivity scenarios. We experimented with different protocols like RTMP, RTC, and low-latency DASH to provide the best experience to our users. This is not a solved problem yet; we are still working on some deep investments to make Live truly available to all users in all connectivity situations.
The third challenge was latency. We realized early on that reducing broadcast latency is better for user engagement. Ever since, we have worked tirelessly to reduce the broadcast latency for Live. We work across our stack (client, processing, service, and playback) to shave latency where possible. We care about milliseconds we can save in operations, as it all adds up. As you can imagine, this requires us to make complicated choices—latency, quality, and stalls almost form a triangle of compromise with one change affecting others. For example, we can reduce the broadcast latency by changing the playback buffer, but it might cause more stalls. We are continually evaluating the right optima for our broadcasters and viewers.
SM: What was the development environment like? Structured and pressured? Were all key milestones met, or did the product get pushed back at any point?
MATHUR: It was quite organic and collaborative in nature, and involved engineers from different parts of the company—Live Product, Traffic, Storage, Video Infrastructure, Security, Hardware, Capacity, etc., coming together to make this successful. We organized war rooms and weekly scrums to keep everyone in sync. We extensively tested the product constantly and got early feedback from our partners to make sure we were getting it right.
SM: Describe the key evolutionary stages of the product from start to now?
CAPRA: There were quite a few key milestones that chart our evolution since we introduced Facebook Live. These include:
- Mentions: We launched Mentions in July 2014 as a new Facebook Creative Labs app that makes it easy for verified users and public figures to talk with their fans and each other on the go, and gives creators a more authentic connection with their audiences.
- iOS for Profiles: In January 2016, we announced that we would be rolling out Live to anyone in the United States with an iPhone. This move came after December 2015, when we began testing the ability for people to share live video on Facebook, which quickly gained popularity among users.
- Android for Profiles: We rolled out Live to Android devices in February 2016 to unlock the broadcasting ability for a huge number of new broadcasters. We also found that more than 50 percent of Live viewers have Facebook installed on their Android devices. In addition to announcing the rollout to Android phones, we also brought the ability to go Live in multiple countries outside the United States.
- Live API: We then developed the Live API in 2016 with a goal of introducing a new way for developers and publishers to join forces to build immersive and interactive live video experiences on Facebook. The Live API allows sophisticated broadcasters to seamlessly incorporate Live into their existing broadcast setup and create new ways to interact with their viewers. With the Live API, Pages and profiles can use multiple cameras, add graphic overlays and other special effects, and build engaging live video experiences on Facebook.
- Live Likes and Interaction Improvements: As Facebook Live really started to take off, we noticed that people were communicating with Live more than ever before. From our research, we realized that people were commenting more than 10 times more on Facebook Live videos than on regular videos. In April 2016, we introduced new ways to interact with Live videos on the platform. To create a more fun and engaging way to interact with friends through Live, we introduced Live Reactions, which makes it easy for viewers to express their feelings in real time during a live broadcast. Using the same reactions we launched in News Feed, viewers can select Love, Haha, Wow, Sad or Angry, and the reactions animate right on top of the video—we saw some really great feedback from this update!
Related Articles
In what could be a high profile failure for the social network, it's reportedly in talks to sell subscriptions to cable networks to its members.
14 Dec 2018
The Ocean Cleanup has begun the final phase of plan to clear away the Great Pacific Garbage Patch, and many saw it live thanks to a collaboration with LiveU.
05 Dec 2018
While Facebook signaled that it would change Live API permission rules several months ago, some producers were likely still caught off guard.
02 Aug 2018
In an exclusive, Streaming Media East attendees got a first-look at Facebook Live persistent stream keys, live cross-posting, and live rewind.
08 May 2018
Skip the off-the-cuff behind the scenes videos. Connecting with an audience on Facebook Live means planning the event ahead of time and bringing viewers into the action.
19 Apr 2017