A way to show not just holograms, but people interacting with them
Virtual reality and augmented reality headsets can offer mind-blowing digital experiences, but they are, by design, singular experiences. Trying to describe it to someone without a headset — who sees nothing but real-life objects and a nerd wearing a computer on her face — is an exercise in futility.
Microsoft knows this is the case with its HoloLens “mixed reality” headset. So the company has come up with a new way for HoloLens users — specifically HoloLens app makers — to capture video from a HoloLens. During a recent visit to Microsoft’s HoloLens lab in Redmond, Washington, my Verge colleague Vjeran Pavic and I participated in a HoloLens demo with a special camera rig set up in the room; moments later, were able to watch video of ourselves interacting with little digital robots.
The rig, which Microsoft calls “spectator view,” requires a combination of hardware and software and was just published as an open-source solution on the HoloLens GitHub webpage. This is all fairly deep in the tech weeds, but the TL;DR is that it’s now a lot easier to make a true-to-life video that shows not only a HoloLens app, but also a person interacting with a HoloLens app.
It’s a small step, but it’s one headed in the right direction for head-mounted displays. And it’s coming at a time when VR and AR companies aren’t just dealing with technical challenges around sharing their apps, but running into ethical boundaries as well, sometimes relying on special effects rather than “real” video to get their points across.
To back up a little bit: HoloLens is a Windows 10 PC that sits on your head. It’s a pretty hefty headset, but it’s untethered, which means you can move freely around the room while you play with apps. It has lightly tinted holographic lenses, four different cameras, and a special holographic processing unit in addition to its CPU and GPU.
Unlike virtual reality, which envelops you in a completely virtual world, HoloLens’ mixed reality means you still see the real world around you, but you also see digital objects — apps literally float in front of your eyes. The biggest criticism to date of HoloLens, and it’s one that still stands, is that the field of view is small: for such a big lens, the stuff floating in front of your eyes feels tiny. In some cases apps even appear to get cut off.
Still, with HoloLens you might see a three-dimensional motorcycle in the 3D-modeling app Maya (courtesy of Autodesk), Buzz Aldrin standing in the room talking about Mars (courtesy of NASA), a friend’s face in a Skype chat box (Skype is owned by Microsoft). The Volvo app lets you pick the color and trim for your new S90 and see how the car will look, true to scale, right before your eyes. These are just a few examples.
But for app makers, showing off these apps to people has become a challenge. The HoloLens does have a built-in camera that will record video of the point of view of the headset wearer, but it’s low-quality video. And since the camera is outward-facing it can’t provide visuals of the person actually wearing the HoloLens. Microsoft has had its own in-house rig for making promotional HoloLens videos, but it involves a Red Dragon camera, which can range in price from $15,000 to $50,000. Not exactly accessible.
“Mixed reality capture is kind of the equivalent of the camera on your cell phone,” says Ben Reed, Microsoft’s head of business strategy for HoloLens, about the built-in camera. “It’s handy, it’s convenient, it does a good job of what it’s meant to do, but it’s not designed to be broadcast quality, what you’d see on TV. We realized we needed another way show other people what the wearer is seeing.”
MICROSOFT IS CALLING THIS “SPECTATOR VIEW,” AND IT REQUIRES A CAMERA WITH HDMI OUTPUT, A CUSTOM MOUNT, AND A PC
The “spectator view” hack, on the other hand, requires a DSLR, or really any camera with an HDMI output. (It could even work with a newer GoPro.) First you’re supposed to mount the HoloLens to the camera, stabilized with a tripod. This requires a custom-made mount. Then you wirelessly connect the HoloLens to a PC, but not to share video; just to share positioning data. In the demo we saw, Microsoft was running Unity gaming development software on said PC.
The next step is to output video from the camera to a PC capture card via an HDMI cable. Now the PC is receiving both positioning data and video. All of the rendering and processing of the data is done inside of the Unity software, and voila! You have video of people walking around the room wearing HoloLens headsets, but you also see the apps and the digital objects and the games they’re playing around them.
Okay, so this is not really a voila! thing. That custom mount still needs to be machined or 3D printed. There’s special calibration software involved, which Microsoft is publishing the code for. The PC has has to meet the minimum requirements for running Unity. Also: this won’t capture audio.
But for people who have already invested $3,000 in a HoloLens headset and are serious about making mixed-reality apps, the setup might not be as daunting. Other VR and AR capture techniques are equally as involved. To capture the video feed from VR headsets like the Oculus Rift or HTC Vive, you can use a screen-recording app like Bandicam. But showing a real person in a virtual world gets complicated. In some cases, you might play a VR game in front of a green screen, attach a motion-sensing controller to a video camera, screen-record the game, and then run a software program that places that person in the virtual environment.
Microsoft’s Reed said multiple times that this new tool is something that its existing HoloLens customers have been requesting. Often the question becomes: how much can you splice together until the video you’re showing simply becomes special effects, and not representative of what the app actually does? It’s a line that Magic Leap, the buzzed-about and highly secretive AR startup out of Florida, seems to have crossed, with reports emerging that at least one of its AR demo videos was actually produced by New Zealand’s top visual effects studio.
For a company like Trimble, which uses its positioning and scanning technology to make mining apps, this kind of setup is something the company has been looking for “since day one.”
“For our mining customers, we were looking for something that would let them visualize the mines,” says Aviad Almagor, who runs Trimble’s mixed-reality program. “In the past, with the kinds of tools we were using to clarify the idea, we’d have to say, ‘Hey, what we are going to present is not really what you’ll see in the HoloLens; there are some special effects.’ We had to set expectations among the users. What we can do now with spectator view is show a live stream of the user.”
It’s still unclear exactly how many HoloLens users are out there in the world; Microsoft has only said that sales are “in thousands.” So this new videography tool is one that only applies to a relatively small number of people, unlike, say, a PC update or a new smartphone camera. But in rapidly advancing world of augmented reality, virtual reality, and mixed reality, it at least offers a hack for making videos that feel, well, real.