SME '17: 4 Challenges of Virtual Reality

VR may be immersive and engaging, but when it’s your job to deliver a seamless VR experience, it’s also extremely challenging. Satender Saroha, director of engineering/architect at Yahoo, talked to an audience at Streaming Media East about “Challenges & Opportunities in Delivering 360° VR/3D Experiences at Yahoo”—going past the sleek, sexy end-user experience, and diving into the nitty-gritty of producing and delivering these experiences.

Capturing the video is the easy part. Yahoo uses rigs of GoPro cameras. Then comes the stitching process, which consists of three steps:

• Raw frame image processing
• Calibration (to avoid the vertical parallax while aligning the overlapping portions of the images)
• Optical flow (this process fills in the gaps between cameras virtually)

But none of this really compares to challenges presented by delivering the video and providing a seamless end-user experience—especially on mobile devices. Saroha presented four main challenges to the audience, and listed the solutions (or temporary workarounds) that Yahoo has adopted.

Challenge 1: Motion-to-photon latency on mobile

Solution: Single rendering and scanline racing help Yahoo cope with the latency inherent in mobile VR experiences.

Challenge 2: 360° support in Safari on mobile web

Solution: Yahoo embeds its player in an iframe on the same domain as the CDN, and also employs a reverse proxy for mobile web traffic.

Challenge 3: Enormous Data

Solution: Cube mapping can achieve 30% bandwith reduction over equirectangular. Alternatively, you can achieve a 40% reduction with HEVC, VP9 vs H.264

Challenge 4: Achieving 6 DoF (Degrees of Freedom)

Solution: There currently is no solution to this problem, but Saroha points to emerging Adobe solutions that say they can enhances existing monoscopic 360 videos with 6 DoF by inferring the camera path and 3D scene geometry using structure-from-motion techniques. Additionally, it uses a novel warping algorithm to synthesize views on the fly by warping original content.

Any creators looking to get into the VR business is going to have to solve these challenges on their own, but learning from the experiences of companies like Yahoo can help pave the way.