Walking with Daemons: BBC Imagines Interactive His Dark Materials
Fans of BBC and HBO hit drama His Dark Materials may soon be able to enter its fantasy world and participate in the action alongside their very own daemon.
The latest developments in personalized and interactive object-based media by the BBC have brought this exciting possibility a step closer.
"Productions like His Dark Materials are already starting to be adequate to be able to produce an object-based media experience," says Juliette Carter, BBC R&D Engineer. "There are already some scenes in [His Dark Materials] which have been created in CGI, rendered using a game engine, and composited out in video. The content is already there."
The next step is to stream the media objects for localized rendering on the client’s device.
"Viewers will be able to press a button and navigate through a scene," says Rajiv Ramdhany, BBC Project R&D Engineer. "You could have your own personal daemon rendered next to you as you explore the world of Trollesund."
For the uninitiated, a "daemon" in the TV fiction is a personal representation of the human soul and appears as a talking animal.
The BBC has already developed an interactive online game in which fans can "discover" their own daemon. The suggestion is that in future scenarios, this information can be used to insert the user's daemon into a personalized narrative of the drama.
"Given a fully digital set of media assets it will be possible for different parts of a scene to be rendered and for the viewer to find themselves in the middle of a battle scene, able to see all around you," says Ramdhany. "This is the holy grail in terms of interactive narrative."
The concept is not unique to His Dark Materials, of course. All content including other narrative shows such as episodic drama EastEnders, entertainment (e.g RuPaul’s Drag Race UK), children’s programming, news, documentary, and sports—live or recorded—could become personalized media experiences created and delivered more like packages of interactive software than traditional file-based broadcast media.
The BBC is at the forefront of researching ways to continue broadcasting to millions, whilst tailoring the experience to be unique to individuals. For a publicly funded broadcaster (more than for SVODs like Netflix, which has also broken ground in this area) mass scaling interactive experiences is a central challenge.
"BBC R&D have been pursuing object-based media as a research priority," says Carter. "We believe low-latency streaming has the potential to address some of the challenges presented by the delivery of complex experiences to a range of devices."
The BBC is also trying to make object-based media creation simple and inexpensive for production teams. It says that content creation—be that a drama or a live event—already generates a wealth of media objects at the point of capture ranging from different audio source, telemetry and stats to camera angles.
"We can identify which of these we can defer to the point of rendering and how all objects can be orchestrated," explains Ramdhany. "The aim is to defer computation as late as possible since this delivers the greatest flexibility (in real-time rendering) and the best possible experience."
Ideally this means computation on the device itself, and that raises the question about whether all devices are capable of this.
Taking its cue from low-latency game streaming services Google Stadia and Microsoft Project xCloud, the BBC’s approach is to stream a remote experience on any device, regardless of its computational ability.
Its target platforms are the major browsers (Chrome, Firefox, Safari, Edge) and native platforms (Mac OS, Android, Raspberry Pi). In tests, a HD 60 frames per second display was rendered on Mac and Ubuntu devices with just 40ms delay. Lower-end devices, such as Raspberry Pi Zero, displayed more latency, but BBC R&D says it is still able to provide a quality experience.
That’s partly because it has adopted a "write once, run anywhere" approach so that the same software codebase is used to create the "executables" (content) of its target platforms. Playback is via a redesigned "multi-format" or Single Service version of iPlayer.
"We want to reduce the amount of software that has to be written to target all these devices and adapt to users and context responsively—so we are looking at new approaches like WebAssembly to write once and run everywhere," explains Carter.
The Single Service Player has been demonstrated to switch "seamlessly" between video and interactive experiences that are written in game engines or the BBC's own object-based media StoryKit, which was used to make an interactive episode of BBC technology show Click earlier this year. The Player can also switch between remotely streamed and locally rendered content according to the capability of the device.
Remote game streaming, in itself, is not enough. BBC R&D is looking into services that adapt to the computing capability in the user's devices and supplement this with computing power from remote servers.
"Relying on high-bandwidth video streaming for interactivity is not a future-fit solution," says Ramdhany. "We’re starting to explore flexible computation—the idea being to harness the compute capability of all client devices available to the user as well as those resources in the cloud and at the edge. We’re looking at what distribution of computational capability offers the best performance to playback a particular experience."
All this work is at the experimental stage. Further work is needed on, among other areas, storage and transfer of data and reducing latency. Opening sourcing the technology is an option.
"Using our research, we can make informed decisions on what the BBC can use this technology for and whether the BBC should invest in it or buy it," he says. "We can also assess its impact on the network and on devices."
BBC/HBO have already shot the second series of His Dark Materials, likely due for release this time next year. It is likely that interactive object-based media experiences will be produced to complement it and future seasons.
Below: A scene from His Dark Materials showing a puppet used on set (top) and the CG rendering created later (bottom).
Related Articles
The BBC is preparing to shut down its traditional television and radio broadcasts and shift everything online, but will do so only if the principal of universal affordable access is maintained.
12 Dec 2022