VR? AR? Today, It's All About XR

VR is dead, long live VR! That about sums up the reality check the industry is having to spin about next-gen immersive video. The GSMA, which represents the interests of mobile network operators worldwide, admits that virtual reality (VR) continues to suffer from "post-hype realities," a hype that the GSMA's annual Mobile World Congress (MWC) played a role in building. At the same time, the GSMA points to the current mobile ecosystem as a "laboratory" for AI and immersive reality: "Whereas the smartphone wars centred on the app economy, the new battleground is in AI development and a push towards immersive reality." 

Analysts predict the global augmented reality (AR) and VR market will grow. Prescient & Strategic Intelligence charts a yearly 42.9% increase, topping $1.274 billion in 2030, largely based on an uptick in the sales of tablets and smartphones. 

Futuresource Consulting expects that the market will have remained relatively flat in 2020, with just fewer than 18 million VR and AR headsets estimated to have shipped. It says this is mainly driven by smartphone VR viewers, which account for more than half of all VR shipments. The market mix is expected to rapidly develop, however, with Futuresource Consulting forecasting that console VR headsets, PC headsets, and all-in-ones (such as Oculus' Quest 2) will enjoy solid growth through to 2024.

In short, VR has failed to take flight, but its time will come again. This article reviews what happened, what's going on now, and what's next for immersive video.

VR 1.0 Stalls

Facebook lit the fuse for VR in 2014 when it acquired Oculus for $3 billion. VR tech and production companies multiplied, with 2017 being the pinnacle. Jaunt raised $100 million from Disney, Google, and Sky, among others, before being bought by Verizon in 2019. Since then, though, things fizzled. Nokia sidelined its OZO division. Google shuttered its VR platform, Daydream. NextVR hit a market value of $800 million before selling to Apple in 2020.

"The VR market was overhyped and did not deliver meaningful results besides CG-based VR games," says Thierry Fautier, president of the Ultra HD Forum and VP of video strategy at Harmonic. "VR failed to take off because the technology was not convincing, due to severe issues like motion sickness and the ‘screen door effect,' and the quality was not good enough (4K vs. the 16K probably needed). We now see a second wave of VR enabled by 5G and new HMDs [head-mounted displays] with 8K capability."

The hype cycle began for VR and AR linked to 5G deployment in 2017. In April that year, Korea Telecom and Verizon claimed the world's first end-to-end 5G network in a demonstration in which a Verizon employee appeared as a hologram on a monitor at Korea Telecom's headquarters. 

"If the 5G network is commercialized, 3D hologram video calls will be available as one of the representative 5G-based services," a Korea Telecom spokesperson told The Korea Herald. "Through a complete hologram video call, users will be able to meet a person in a remote area in a real size in real time."

At MWC 2018, South Korea's SK Telecom debuted 8K "social virtual reality" and "hologram AI." In September that year, Vodafone demonstrated a live holographic appearance of the England women's soccer captain over 5G. In February 2019, U.K. mobile operator EE showcased its 5G network by putting a hologram of "digital supermodel" Shudu on the red carpet at the BAFTA Awards. At MWC 2019, ZTE and China Telecom teamed up to show 5G 8K VR panoramic live streaming from a traveling bus. 

For years, the industry has been building the hype around 5G and, in particular, its "game-changing" capability to stream video games and AR/VR experiences in real time. After all, 5G is the first network infrastructure that can deliver the speed (100Mbps) and latency (10 ms) required by VR and AR mobile applications. But despite heavy investment from Apple, Microsoft, Google, and Facebook, VR/AR is far from mainstream. 

"The lack of a suitable supporting network is one of the factors that has contributed to a slower than expected uptake in mainstream adoption of VR," says Sol Rogers, CEO and founder of immersive content creator REWIND, in an article he wrote for Forbes. "5G will usher in the next era of immersive and cloud-connected experiences. Faster, more uniform data rates, lower latency and lower cost per bit will ensure it."

VR headset ownership has dipped from peaks in 2017 in Australia, the U.K., and the U.S. Whereas 13% of U.S households owned one in 2017, by 2019 that number had fallen to 8%. According to the GSMA, "Expensive, clunky hardware and continued challenges with dizziness have not helped, but limited content libraries beyond gaming and lack of edge infrastructure are also to blame." 

Sports leagues and broadcasters are among those that continue to see VR as a means of boosting viewer interest. The challenge will be to convince pay-TV customers to spend more for VR broadcasts when they are already being pulled in the opposite direction toward cheaper and more-flexible streaming packages. 

Even the $1.2 billion market size predicted for VR/AR in 2030 is dwarfed beside the existing $1 billion-plus market value of esports or the $1.52 trillion in total global mobile revenues anticipated by the GSMA by 2025.

"XR [extended reality, the catchall term for VR, AR, and mixed reality, or MR] has been a disappointment insofar as it's not met the hype that has often surrounded it, but these expectations were, in hindsight, largely misplaced," says Stephen Mears, a research analyst at Futuresource Consulting. "Too many technological developments needed to be made and coincide with one another for XR to really take off, and there's still a few steps to go. Notably, improved visuals must be combined with spatial audio technologies, and likewise the SoCs [systems on a chip] must be further developed for specific XR use cases."

AR is a little different because it is inherently more suitable for on-the-go mobile computing. Google, Microsoft, and Magic Leap quickly realized that AR's biggest early-days potential lies in B2B. Even Lenovo has launched AR platform ThinkReality for manufacturing.

"[Consumer AR] is hard, as the technology cannot deliver the consumer expectation at a price consumers find agreeable," Fautier says. "Google Glass was a failure because the experience was flawed and [the] price was high. The Magic Leap experience was not great, the ecosystem incomplete, as it required a special piece of hardware, and the price was high."

Earlier this year, Gartner removed AR from its list of emerging technologies, considering the tech to have matured and ready to move into the enterprise space.

Magic Leap was one of the early frontrunners in the augmented reality (AR) field, but the company soon discovered that the future lay not in consumer experiences but in B2B.

VR 2.0 = XR

The term XR is now vogue. You can't get a clearer description of its components than that provided by futurist and strategic advisor Bernard Marr: 

• VR: A fully immersive experience where a user leaves the real-world environment behind to enter a fully digital environment via VR headsets.
• AR: An experience where virtual objects are superimposed onto the real-world environment via smartphones, tablets, heads-up displays, or AR glasses.
• MR: A step beyond augmented reality where the virtual objects placed in the real world can be interacted with and respond as if they were real objects.

Qualcomm is a leading promoter of an XR future. The chipmaker is explicitly linking XR with 5G and has corralled a number of mobile operators to commercialize a head-worn "XR viewer" that can be connected to a 5G smartphone and powered by Qualcomm silicon. "Mobile XR has the potential to become one of the world's most ubiquitous and disruptive computing platforms of the next decade, just as the smartphone of this decade has become," says a Qualcomm blog post. "At some point in the future, we see the convergence of the smartphone, mobile VR headset, and AR glasses into a single XR wearable. … XR glasses could replace many other screens in your life—even big ones like the TV in your living room." 

China Mobile—the world's largest wireless network operator—Deutsche Telekom, EE, KDDI, Orange, SoftBank, Verizon, Vodafone, Tele­fón­ica, and NTT DOCOMO are also on board. "We believe that the next decade will see a transition into a heads up society where eventually glasses will be an alternative to smartphones," says Sean Seaton, SVP of group partnering and devices at Deutsche Telekom, in a press release. 

Qulcomm is working on extended reality (XR) glasses, which will connect to a 5G smartphone. As indicated in this image, such glasses will not only help first responders see their environments more effectively, but also could monitor those responders' activity and health.

Qualcomm supports Nreal, an XR viewer that connects via USB-C to an Android device, with all of the processing, storage, and wireless network connectivity needed for XR taking place on the phone instead of the headset. Brands including iQIYI, 3Glasses, and Panasonic are expected to follow next year with gadgets bearing a "Qualcomm XR Optimized" badge. Qualcomm mentions enterprise applications such as how "workplace meetings can be revolutionized through holographic tele­presence with virtual collaboration platforms." 

Exploring XR

Meanwhile, Deutsche Telekom is teaming up with Orange to develop consumer XR content. The first partnership is with Montreal-based Felix & Paul Studios to provide "the ISS experience." Morgan Bouchet, director of digital content innovation and head of XR at Orange, says, "[We] have been moving forwards with projects to develop platforms that deliver XR/5G, and have been confronted with a common challenge: finding high-quality interactive and immersive content. Supply is still limited in this market, with designs being few, far between, and expensive. Together, we are better equipped to compete with larger players, especially [Google, Apple, Facebook, and Ama­zon], in order to purchase premium content."

The two companies are also looking into XR in sports and TV. Daniel Aslam, Deutsche Telekom's global partnering and XR business development manager, states, "We are exploring XR [Immersive] Media use cases where for example the TV interacts with mixed reality glasses. This we will continue in 2021 in a co-development project with Orange." 

"Growth in XR in the short-term will be driven by gaming and B2B, particularly in education and training," says Futuresource Consulting's Morris Garrard. "Improvements in productivity in the workplace will help bring XR into consumers' homes."

Perhaps the biggest (non-Qualcomm) XR product will be Apple's glasses, rumored to launch in mid-2021, possibly with an 8K spec.

"Ultimately, there will be compelling use cases for standalone (untethered) XR headsets and wearables, most likely in enterprise AR initially," says Mears. "However, the industry must ensure a seamless consumer experience and, crucially, software and content developers need to be brought on board to make sure that there is something actually worth doing with these devices when they're developed."

VR and AR require bandwidth between 20­Mbps and 30Mbps and can therefore be consumed in a stadium without an HMD, which Harmonic believes will be a popular use case. Multi-view, multicam, and multi-game HD video, which will also require 20Mbps, has been demonstrated at the RG Lab with France Télévisions.

"Before 2025, we will see in-stadium applications on mobile devices," predicts Fautier. "For VR or 2D, the applications are multicam, wide-angle camera for live and catch-up action. For AR, we envision score and statistics overlay and point cloud for volumetric capture. We believe multi-view and replay of action in the stadium can now be deployed at scale with 5G."

Watching a boxing match with a 3D version on your coffee table could become a reality with innovation from BT Sport.

XR Applications

Nor does XR need to be considered a purely visual opportunity. For example, there is capacity for audible AR, in which the world is described to the wearer through a voice assistant. Simon Forrest, principal analyst at Futuresource Consulting, posits that this may take the form of an advanced hearable product that's able to use location-based sensors and spatial awareness to deliver precisely timed, contextually aware information that's relevant to what the user is doing. "This has consequences on the advancement of AI in voice assistants and how internet search operates because all results must be spoken as the top answer, each and every time," he says.

Beyond headline speeds, the ultra-low latency in 5G effectively brings edge devices and cloud services far closer topologically, blurring the boundaries over where a compute resource can be placed. Furthermore, the opportunity for network slicing—effectively reserving guaranteed bandwidth for specific applications—will present mobile networks with advanced capability. 

"This opens up new applications that demand real-time response from cloud services, with XR one vertical that is able to take full advantage," says Forrest. "Nevertheless, we're still around 5 to 7 years away from blanket 5G service coverage in most developed regions, and certainly, devices won't be ubiquitously connecting at gigabit speeds, since this would necessitate a massive densification of 5G network infrastructure."

Forrest concludes that XR products cannot simply rely on high-bandwidth mmWave. Centered on 26 GHz, mmWave adds the "super data layer," providing ultra-high bandwidth service of up to 10Gbps over very short distances. "Therefore, 5G should be considered as one part of the overall solution for XR," he says. "In tandem, SoCs are being developed with neural network accelerators on board. This innovation is affording opportunity for edge-AI, massively improving the compute capabilities available on XR devices themselves. Combining this with low-latency 5G connectivity then allows cloud-based compute to run alongside. Ultimately, 5G enables a balance of compute between edge and cloud, with capacity to execute in near real-time."

For in-home XR experiences, 5G will be "essentially meaningless," says Forrest, with Wi-Fi 6 far more important in ensuring low-
latency, high-quality XR experiences if the content being consumed or used is being directly streamed, such as sports. Moreover, the Wi-Fi 7 (802.11be) standard, presently under specification and scheduled for commercial launch in 2024, is expected to offer peak connectivity speeds of 30Gbps and lower latencies, competing directly with 5G mmWave. Forrest points out that this will necessitate fiber-to-the-premises in order to connect that massive local bandwidth to the internet backbone.

New codecs such as Versatile Video Coding (VVC) and Essential Video Coding (EVC; MPEG-5) will help reduce video bandwidth for broadcast/push VR applications, and these codecs will be used in tandem with 5G and Wi-Fi 6 connectivity. But video frames aren't necessarily the best technology for XR, and there are alternatives under development. One such innovation is in volumetric capture techniques: Video can be captured using multiple cameras, and then a "point cloud" is generated, which describes the 3D [x, y, z] coordinates and color of each pixel in the scene. 

"From thousands of points, a set of 3D objects are created, and these can be rendered in the headset; coupled with the increase in local compute performance, this offers potential for new XR products based around graphical rendering techniques, rather than just video frames," says Forrest.

BT Sport is exploring how to capture volumetric video and deliver interactive immersive experiences over 5G, both within sports stadiums and to augment the live broadcast at home. Ideas include streaming a real-time virtual volumetric hologram of a boxing match onto a viewer's coffee table, simultaneously with the live feed. It is all at the proof-of-concept stage, but is part of a nearly $40 million U.K. government-funded program to develop applications that will drive 5G take up. 

6DoF Free Viewpoint

Companies within MPEG are actively defining algorithms for the compression of point clouds, such that capture and transmission are standardized. These are MPEG V-PCC and MPEG G-PCC for video and graphics respectively, both of which seek to capture 3D volumetric data and reduce bandwidth significantly.

"Beyond 2030, we anticipate there will be 6DoF [6 degrees of freedom] light field applications that will require high computation on the client side, as well as new display technology," Fautier says. "Those instances will [be suitable for] location-based environments. A second application is the free viewport application to render a game. While that application is not in real time today, we expect it to be."

In 2019, Canon used its Free Viewpoint Video System to provide highlight videos at the Rugby World Cup. The company says that the system "makes use of multiple cameras positioned around a stadium … and the high-resolution feeds are combined and converted into 3D spatial data, generating a comprehensive three-dimensional map of the action moment by moment."

Another application, which requires specific software in the 5G network, is FeMBMS (Further evolved Multimedia Broadcast Multicast Service). This new mobile broadcast mode in 3GPP Release 17 addresses distribution of popular content up to a radius of 60 km. Harmonic expects to see the first deployment at the Paris Olympics in 2024. If that is conclusive, it could be widely deployed after 2025. 

"Mobile broadcast is the wild card, as it has failed in 4G," Fautier says. "However, if the QoE is much better in situations where streaming is congested, then FeMBMS will be successful. MNOs will be able to monetize their network to content providers, broadcasters, or event organizers (like the IOC or FIFA)."

If XR is to fly, it needs a disruption in display technology, according to Qualcomm. Attributes of these "glasses" include delivering field of view for both immersive VR and useful AR (completely opaque for VR, yet at least about 85% transparent for AR), driving high dynamic range (HDR) of at least Rec. 2020 gamut, and refreshing at a minimum of about 120Hz (per eye). It also has to cost less than $100. 

Qualcomm also says that 5G-enhanced mobile broadband is required for XR mass adoption. "XR video will be the killer use case for 5G," it states. Target attributes include latency down to 1 ms and a uniform experience—even at the cell edge. Qualcomm predicts that over the next decade, 5G XR should advance to attain speeds of 200Mpbs to 5,000Mbps, with interactive, real-time, 3D Free-Viewpoint, 6DoF, 8K/90 fps–120 fps HDR video.

Spatial Computing Frontier

XR devices are the gateway toward the real prize targeted by tech giants like Magic Leap, Microsoft, Apple, Google, and Nvidia: the next-gen internet, which is theorized as 3D and tactile, requiring a new human-machine inter­action designed around voice and gesture.

Spatial computing, as defined by Magic Leap, "allows digital content to interact with its physical surroundings and people to interact with content seamlessly, without the limits of a screen." The company is bringing this forward by running spatial computing on the 5G infrastructure of partners such as Japan's NTT DOCOMO.

Nvidia's bid for a controlling share of this online future is called Omniverse. Targeted initially at the enterprise, Omniverse "fuses the physical and virtual worlds to simulate reality in real time with photorealistic detail," according to president and CEO Jensen Huang in a recent keynote. "Cloud native and photo­real, with path tracing and material simulation, the Omniverse allows designers and artists and even AIs to connect in a common world. This is the beginning of the Star Trek Holo­deck, realized at last." 

ARwall and Webster University Bridge the Gap Between Education and Virtual Production Technologies With New Virtual Production Facility

Webster University has become the first university in the Midwest to have a state-of-the-art virtual production facility thanks to a unique partnership with ARwall, a Metaverse 100 XR technology company.

22 Feb 2023