Augmented Reality With Microsoft HoloLens And Project Tango

If you’ve been reading Owner’s Magazine lately, you’ve probably read my article about virtual reality. There I detailed information about the latest VR headsets to make a proper buying decision. However, I intentionally left out augmented reality. Augmented reality, or AR, is different from VR in which VR fully immerses the user inside of another world, AR (also known as mixed reality) superimposes virtual objects onto the real world. Holograms are an example of augmented reality. Popular AR games such as Pokémon Go are also examples of AR where your smartphone camera superimposes the virtual Pokémon on a live video feed making it seem as if the Pokémon were standing in the real world. The technology behind AR is nothing new. Even in today’s world of smartphones, many augmented reality apps were released on iPhone and Android devices back in 2008 and 2009. However, with the seemingly popularity of VR, more attention is also being given to AR solutions. Two of the biggest players in this space are Microsoft and Google.

Microsoft HoloLens

One of the more interesting AR devices comes from Microsoft. The HoloLens is a head mounted device that allows the user to view “holograms” through the lenses (hence the term “HoloLens”). The HoloLens features a bevy of sensors such as an accelerometer, gyroscope, magnetometer, four IR sensors for mapping the environment, a depth camera, ambient light sensors, 2.4-megapixel front camera, and microphone array. Along with an Intel CPU, Microsoft developed a custom chip called the “Holographic Processing Unit” or HPU. The HPU processes the sensor data as well as hand gestures and voice recognition. The HoloLens runs full Windows 10 and can run Universal Windows Programs. Regarding the hand gestures, there are two main gestures: bloom and tap. Bloom is simply balling your fist up palm facing up, and opening the fingers up like a flower (hence “bloom”). The bloom gesture activates the shell which allows you to access other programs. This is akin to pressing the Windows key on a PC or the Xbox button on the Xbox One. The second gesture is the tap. You aren’t literally tapping the air but it’s more like holding your index finger up then touching your thumb. This is used to select objects and windows. Windows can even be dragged around and pinned to walls or tables. Microsoft isn’t technically marketing this towards consumers yet. This is primarily for developers to create applications although anyone can buy the development kit for a wallet busting $3000.

Project Tango

Project Tango is the augmented reality platform developed by Google. While the Microsoft HoloLens is an actual headset, Tango goes a more practical route and utilizes a smartphone or tablet form factor for AR. The devices provide the hardware necessities such as various sensors and RGB-IR camera for depth sensing. The Tango software uses the hardware to enable motion tracking and visual recognition of the environment. The IR camera allows Tango to map the environment including distance, sizes, and depth. That information is then stored for future applications to use. This has many practical uses such as navigating through stores, mapping, and visualizing data. For example, you could use Tango to see exactly how a piece of furniture would look in your home before buying it. Google partnered with Lenovo to release the Phab 2 Pro, the first commercially available smartphone with Tango built in. At CES 2017, ASUS announced that it will also be releasing a Tango enabled phone along with Daydream VR.

Which Is The Future? VR vs AR?

There has been some debate over which platform is more viable for the future. Virtual reality completely immerses the user in a new world. These kinds of experiences are better suited to content consumption and entertainment. All three of the major VR platforms (HTC Vive, Oculus Rift, PlayStation VR) all focus on games as their primary use case although some of them do allow you to watch movies in your own personal theater. At Facebook’s Oculus Connect event back in October, Mark Zuckerburg unveiled a vision for new ways to use VR for social interactions. He used the Oculus Rift to interact with virtual representations of himself and two other colleagues in a totally virtual space. Imagine being able to debate politics and religion with your friends in real time in a virtual world…actually…that may not go so well. However, being able to interact with another person in virtual reality would dramatically change and revolutionize how we connect with our friends and family. The primary drawback with VR is the total disconnect from the real world. VR often involves putting on a somewhat bulky headset and tuning out the world around you. The advantage of AR is that it allows you to interact with virtual objects while still being cognizant of the real world around you. AR also allows more practical, everyday uses. Imagine having a digital map displayed on your windshield on your car that helps you navigate and alerts you to any hazards or speed limits. Imagine virtually placing a couch in your actual living room before buying the couch. Construction workers and architects could use AR to precisely build structures while seeing a virtual representation of the building. Google’s Translate app can already translate text into another language in real time. So which one is the future? I think one of two things will happen. Either AR will eventually become dominant and heavily used in everyday life or there will be a mixture of AR and VR depending on the medium. VR will likely dominate the entertainment space with movies and gaming while AR is what most people will interact with daily. Project Tango will further accelerate the adoption of AR into Android phones. Apple is rumored to working with Carl Zeiss for their own augmented reality solution. History has shown us that once Apple implements a feature, the market generally follows suit. Regardless of what happens, the dream of the Star Trek HoloDeck is slowly coming to fruition.