The VR experience is fun for the person who is wearing the VR headset, but it disconnects you from your real surroundings. The audience of the ‘show’ laughs because you scream, jump or laugh, but they actually have no idea where you ‘are’ or what you see.

First to pick up this problem were the Google Machine Perception researchers; in collaboration with Daydream Labs and YouTube Spaces. They have been working on solutions that address the issue. They came up with virtually “removing” the headset and create a realistic see-through effect to show the facial expressions of the user, and then there was Mixed Reality.
https://www.youtube.com/embed/MW7un0_s_Oc

How mixed reality works?

Mixed Reality (MR) is a medium that shares the virtual experience of a VR user in a two dimensional video format allowing other viewers to get a feel for it.

To achieve the goal they mix three components:

• Dynamic face model capture
• Calibration and Alignment
• Compositing and Rendering

The captured 3d model of your face allows to generate any desired eye-gaze, thus making the synthesised face look natural and alive. Calibration and alignment consist of the camera capturing a video stream of the VR user in front of a green screen and then composites a cutout of the user with the virtual world to create the final MR video. The 3D face model is rendered onto the headset which gives the impression of desired ‘removed’ headset. Producing a suitable rendering of the 3D face model is the last step, consistent with the content in the camera stream.

VR user captured in front of a green-screen is blended with the virtual environment to generate the MR output: Traditional MR output has the user face occluded, while our result reveals the face. Note how the headset is modified with a marker to aid tracking. //Copyrights of the image: research.googleblog.com

SUMMARY

The VR headset removal technology enhances Mixed Reality, allowing the medium to show their face in a natural and convincing fashion.

While the technology still remains unnatural and distracting; it’s a first step into the future of communication and social interaction. From now on it’s very likely that diverse applications like VR video conference meetings, multiplayer VR gaming, and exploration with friends and family will follow soon.

WHAT’S NEXT?

Currently the technology is evolving but I think we still need to wait a few years to be able to experience this in our home environments. Nevertheless, I am excited to see where and how it is going to expand. Are we gonna sit one day in different rooms and have a meeting as if we were in one room? Is it going to be a long distance face-to-face conversation? Will we still be disconnected from our surroundings? Finally, are we going to need any headsets or is it going to be an implant or some kind of unnoticeable wearable? I think it’s just matter of when, not if.

This blogpost was written by our UX designer, Kate Gornas.