Netflix Goes to the Other Side of the Mirror With Bandersnatch
This article will describe some of the technology and techniques behind interactive experiences on Netflix. I lead an engineering team focused on source specifications and encoding technologies, but the work I’ll discuss here was done by my colleagues on the user interface (UI) player and streaming client technology teams.
At Netflix, we focus on bringing great stories to life with emphasis on quality and seamless experiences. To that end, we constantly develop new encoding strategies, adaptive streaming algorithms, playback technology, and user experiences. We’re eager to experiment with innovative storytelling techniques as well, building a framework for interactive experiences on existing infrastructure and leveraging the scale of our platform.
The success of these experiments ultimately depends on Netflix members and their willingness to engage with the story, to accept responsibility of control. This requires a little bit of retraining given current viewing habits. Each of our interactive titles to date has included a series of tutorials, one for each platform, designed to encourage viewers to pick up their remote control, game controller, or even just their pointer finger and propel the story forward.
I particularly enjoyed the tutorial for “Bandersnatch,” an interactive episode of the anthology series Black Mirror that premiered in December 2018. “Bandersnatch” viewers can alter the storyline seamlessly in real time based on choices they make through answers to successive prompts at the bottom of the screen. You can see the tutorial. As you watch it, please note the way that the UI interacts with the video.
Figure 1 shows Charlie Brooker, producer of Black Mirror and the writer of “Bandersnatch,” reviewing an early prototype of the interactive UI. As engineers, we may focus on one definition of seamlessness, but another really important component of the success of all these projects has been a tight feedback loop between engineering and creative, which gives a lot of control to storytellers to craft the overall viewer experience. In this way, the design and the execution of both the Netflix UI and the media assets delivered by an external production integrate to complete the director’s final vision within the product.
Figure 1
Figure 2 charts the default path for this article. Excluding any unforeseen interruptions in our current space-time continuum or any non-linearity bugs, this is what the rest of the article should look like, with a TRT (Total Running Time) of about 2,500 words. We’ll begin by reviewing the adaptive streaming strategies at play here, then talk a little bit about the framework and vocabulary for producing one of these branching narratives—as we call them internally—using “Bandersnatch” as a case study. Then, working backwards, we’ll have a look at how these media assets were actually delivered to us and then how we’d like to use Society of Motion Picture and Television Engineers (SMPTE)’s IMF (Interoperable Master Format) for this use case into the future.
Figure 2
Interactive Storytelling at Scale
Netflix may not be the first to experiment with this type of storytelling, but we are the first to do it at this scale. Nearly everything we do at Netflix we do across the globe in dozens of languages on families of devices installed over the course of 10 years.
Streaming Media readers who distribute or deliver content understand the challenge in deploying a consistent experience across an ecosystem of varying capabilities and implementations. Seamless playback, for example, is a primary objective for the streaming client teams, and it is especially tricky with interactive content when you don’t know what to play ahead of time.
Whether you’re delivering interactive content or not, I think we can all agree that buffering is the enemy.
Adaptive Bitrate Encoding and Seamless Streaming Experiences
I know that this is a review for many readers, but I’d like to provide a brief primer on adaptive streaming and some specific encoding improvements that lead to a seamless streaming experience (Figure 3). A large part of our job on the encoding team is to produce multiple derivatives of the same content so that they can be delivered under a range of network conditions.
Figure 3
Figure 4 shows two hypothetical examples of H.264 AVC encodes packaged as ISO BMFF (Base Media File Format) MP4. If you’re a “Bandersnatch” fan, think of these as alternate versions of the same constructed reality.
Figure 4
While the data rates of these streams necessarily differ, the structure of each must be consistent such that all encoded segments align. This allows a player to switch from one stream to another without negatively impacting playback. An adaptive streaming algorithm may choose to use different segments from each of these streams to match varying network throughput.
This is a simplified example of what is an incredibly challenging and complex problem, and until very recently, we guaranteed conformance among different streams by restricting segment durations to 2 seconds. In the graphic shown in Figure 5, each one of those little boxes represents a 2-second segment. This is not an uncommon practice in our industry, and most streaming algorithms can switch as needed on a 2-second cadence to adapt to their network throughput.
Figure 5
In 2018, we re-evaluated this contract between encoding and playback, instead aligning segment boundaries with shot boundaries, and ultimately optimized the encoding parameters on a per-shot basis. Now, adaptations during playback are largely hidden by edit decisions inherent in the content itself. The contract between encoding and streaming also paved the way for arbitrary keyframe insertion, providing granular control over the structure of an entire encoding ladder within our distributed transcoding workflow.
Additionally, quantization is reduced. This enables the data rate of our streams to more closely match the actual throughput of a playback session. Where supported, we see up to a 50% reduction in the overall size of our encodes for the same quality when compared to our per-title encoding optimizations.
Enter Interactive Content
Now let’s take a look at the content within these streams. First, a standard title on Netflix where there is only a default path is laid out as shown in Figure 6, with a simple instruction to the player to start at frame zero and play every subsequent frame until the frames run out, with some exceptions around introduction and credit sequences.
Figure 6
Puss and Book: Trapped in an Epic Tale was our first official branching narrative title. It launched in May 2017 and played a lot like a fractured fairy tale. Young viewers helped Puss navigate his way through a series of conflicts, ultimately helping him escape from an enchanted storybook, where he had been trapped by a despotic narrator. We used Puss as a model to begin working out our acquisition, encoding, and deployment strategy.
In 2018, we launched Minecraft: Story Mode. That pushed the limits of everything we had built. Then came “Bandersnatch.” In preparation for “Bandersnatch,” Netflix developers built the Branch Manager tool (Figure 7) to help Black Mirror producers visualize their timelines, evolve the story from script to screen, and ultimately bridge the gap between Charlie Brooker’s imagination and our nascent engineering solutions. The tool appears to have been well received. In fact, if you watched “Bandersnatch,” and if you made the right choices, you may have seen this story line where Branch Manager makes a cameo.
Figure 7
There are some basic rules by which we assume all current branching narratives abide, and we’ve also come up with some vocabulary to talk about it. Each title is constructed of a series of interactive segments in which each segment contains synchronized media, video, audio, and text. It may contain a choice point prompting the user to make a decision and also defines which segments will be played after a user has made his or her selection.
How ‘Bandersnatch’ Uses Narrative Branching
“Bandersnatch” is a standalone episode of the Black Mirror series. It’s about Stefan, an aspiring game developer in the U.K. in the 1980s. “Bandersnatch” viewers help Stefan navigate a series of challenges as he attempts to complete his colossal vision for a new video game, even as he begins to question his own reality.
Related Articles
Process enables direct IMF package delivery via API entirely in the cloud, saving Technicolor time because there is no need to download IMF locally and re-upload it to deliver to a platform
04 Dec 2020
Why did Netflix's stock fall in Q2? Because the OTT leader's price increase lead to service cancellations. That's a warning to all SVODs: If it can happen to Netflix, it can happen to any service.
28 Oct 2019
SVOD leader Netflix had a weak quarter, but the researchers at eMarketer see it bouncing back with strong subscriber numbers for the rest of the year.
21 Aug 2019
Netflix's Andy Schuler will reveal how the company makes interactive content work when he presents a Streaming Media East keynote in May.
01 Feb 2019
It's important for YouTube publishers to create a great user experience for our viewers, and interactivity can help. Here's a walk-through of how to use YouTube's interactive features.
26 Mar 2018
While startup Beam Interactive only launched in January, it's already made a big impression with online gamers and the team at Xbox.
12 Aug 2016