Learn Why Mixed Reality Studios Are the Future of Live Streaming
As live events and presentations become more sophisticated, content creators will need sets or studios equipped to produce immersive content that merges the real world with the virtual. The demand for studios that can accommodate Mixed Reality (MR) video capture and live streaming, which combines Virtual Reality (VR) and Augmented Reality (AR), will grow rapidly in the near future.
These advanced studios will be either independent off-site locations that are available for rent by the day or hour, or custom system integrations designed to be installed on-site at the office. MR studios are proliferating at a steady pace across the world. These studios are geared for gaming or eSports, corporate communications, presentations, product launches, demos, training, education, architecture, science, medical, government, and virtual commerce.
Both MR and VR blend the real and virtual worlds, but each allows for a different level of interactivity. Currently, VR-based MR shows the presenter in a fully computer-generated virtual environment. Microsoft HoloLens MR is based on holographic/3D objects being displayed in real rooms and environments.
Today, live and recorded MR content is viewed on 2D screens such as flat screens, smartphones, and desktops to help viewers better visualize how the presenter or player is merged in a virtual world. Viewing streams on devices such as Oculus’ Gear VR or Android’s Daydream, VR headsets, and HoloLens-like headsets will become more common over time.
VR/MR studios will need pre-lit greenscreens that support room-scale VR. Solutions for chromakeying/compositing live video with Unity 3D and VR headsets will allow subjects to be composited with graphics, overlays, virtual sets, and 3D models.
These studios will also be equipped with cameras that are synced and calibrated to allow multiple live, real-world camera feeds that are mounted with trackers that match the virtual camera Field of View (FOV) in 3D space while switching between first-person Point of View (POV) and third person when needed. Floating cameras will be configured for motion tracking on gimbals, allowing for fluid camera moves to get different perspectives of the subject or presenter composited consistently within the virtual space and interacting with virtual objects.
When using the Microsoft HoloLens, Spatial Mapping is required to allow the developer or interface designer to identify where to place objects and reference points within the stage, or on surfaces such as walls, ceilings, or floors, with accurate scale and physics. This involves scanning the set for real objects and converting them into meshes to determine how presenters can interact with them. Real cameras will be configured to capture directly from a POV that shows viewers what the HoloLens view looks like.
Unity and Unreal Engine are the current standards for VR, AR, and MR development. Unity is a powerful 3D engines and game development platform that actively facilitates development through partnerships with all major VR companies and Microsoft for the HoloLens. Unity 3D will be critical component in the studio since it allows for the creation of 3D graphics, animations, and customized physics.
Unity 3D and Unreal can trigger events with physical-location or object hotspots. A presenter will be able touch an object that activates another sequence of events or animations, or even walk to another location on the stage and be transported to another virtual set.
Widely used in the gaming community, Open Broadcaster Software (OBS) is an open source encoding and streaming solution optimized for MR video. Other software encoders such as Telestream Wirecast can do the job if you composite everything before it hits the encoder.
MR will enhance user experiences by providing a new degree of visualization and simulation, incorporating video, audio, data, graphics, holographs, and environments. It will deeply impact viewers by drawing them deeper into engaging, unforgettable experiences.
[This article appears in the June 2017 issue of Streaming Media Magazine as "Mixed Reality Studios Are the Future of Live Streaming."]
Related Articles
Step into the future of immersive storytelling (literally) and subscribe to content bundles based on individual preferences. That's where streaming TV is headed, experts suggest.
09 Jan 2018
The rise of virtual spaces could dramatically change how people work, play, and interact. Before that happens, VR needs stable and fast wireless connectivity.
22 Jun 2017
Intel and Microsoft are among those building tools for a merged reality video experience that could be streamed directly to the home.
23 Mar 2017
Mixed reality, a combination of augmented reality and virtual reality, offers new possibilities for connecting with audiences
08 Sep 2016