Updates:
We are using the Magic Window system in SIGGRAPH 2017, in project Holodoodle.
——-
Have you felt confused watching friends in VR/AR experiences? Have you felt lonely exploring VR/AR environments even when others are nearby? By turning the phones of spectators into magic windows, we can now put spectators together with VR headset users and provide a shared mixed reality experience for them both.
The thesis presentation is here:
As a developer in the VR/AR industry, I’ve found that the current interactive mechanism behind VR/AR is somehow broken in the sense that most VR/AR experiences are designed for one person alone. This often leads to a feeling of isolation and separation between the real and virtual worlds, for both the people wearing headsets and the spectators watching them.
Our first attempt to solve this problem was to make a multiplayer VR experience. We had success. People enjoyed co-located, multiplayer VR, and the oral communications among them came back, just as friends talking together at a party. Thanks to the shared gameplay, the feeling of isolation for people was diminished despite the fact that they were wearing headsets.
But another form of isolation remained. Spectators watching the people wearing headsets still didn’t know what was happening in the virtual worlds of the VR users, and a barrier still existed between them. I began to wonder if there was a way to join spectators together with VR users. This would require:
A multiplayer network to link people together
A tracking solution for everyone in the event space
A solution for everyone to see his or her actions in the virtual world
I discovered that if we could use the spectators’ mobile phones, everything could be done in a single application.
This is how Magic Window was born.
A typical scenario is below:
Alice, Bob, Claire, and David go together to a VR event showcasing an awesome, multiplayer game, but since the HMD is limited, only two people can play at a time. Alice and Bob go first, which makes Claire and David a little bit sad since they don’t know what Alice and Bob are experiencing. Luckily, Claire and David find a mysterious QR code on the wall, and, after scanning, their mobile phones turn into Magic Windows. They can see what Alice and Bob are experiencing, exactly at the same location. Their mobile phones seem to have turned into another set of eyes for them, eyes that can see the virtual world. The magical thing is that Alice and Bob, who are in the virtual world, find that they are seeing something new, too. They have been joined in their virtual world by two new virtual people, the avatars of Clare and David. All four say hello to each other, orally, just as they do in the physical world in their daily life. Clare is interested in the shooting game Alice and Bob are playing. She joins their game with her mobile phone, by clicking the screen. David is not interested in the game and decides to show his friends on Facebook the game the others are playing. He, as a spectator, starts recording the game with his phone, at which point, Alice, Bob, and Claire notice a camera icon appearing on David’s avatars’ head.
After my implementation, I did some user testing and discovered some interesting things. First, as with the co-located, multiplayer VR games, oral communications came back between the players. They could see the same virtual environments and no longer felt as isolated. Second, the people without VR headsets were interested in joining their friends in the virtual world, even if only with their phones. The most important finding was that Magic Window gives people outside a virtual world the chance to join in. It brings people who would otherwise be separated onto the same stage, sharing the same world again.
Magic Window is still under development. In addition to enhancing the spectator experience, Magic Window can also be used as a live streaming and documentation tool for VR/AR. MY final delivery will be an open-source tool for anyone developing VR/AR applications to use.
