ERNI Technology Post No. 56: Reality beyond the screen with Microsoft Hololens

Improving reality with Microsoft HoloLens

Microsoft HoloLens is a brand new kind of device that allows us to create Mixed Reality experiences. While Virtual Reality completely occludes our real world with a computer-generated one and Augmented Reality adds computer information on top of our reality, Mixed Reality is able to “know” the environment and create interactions between real objects and computer-generated ones, in fact “mixing” both worlds together.

A brief introduction to HoloLens

Device capabilities and specs

What makes HoloLens a really unique device are the untethered capabilities. While other devices like HTC Vive, Oculus Rift or Meta2 rely on a connected computer, HoloLens is a standalone device, having all the needed onboard capabilities to:

  • process and render computer-generated objects
  • detect the environment
  • «understand» user interaction through gestures and voice.

Computer hardware for HoloLens is composed of an Intel Atom processor, 2GB of RAM and 64GB of flash storage. But the real brain of the beast is the HPU, the Holographic Processor Unit, a SoC created by Microsoft to work with the hologram calculations.

The HoloLens HPU is a 24-core DSP processor with 1GB of DDR3 RAM capable of working on complex calculations 200 times faster than the Intel Atom processor, all while drawing less than 10W of power from the device battery. This leaves the GPU and CPU free to concentrate on running Windows 10 and additional apps.

Source: Microsoft

And as noted above, HoloLens runs a custom-made 32-bit version of Windows 10, much like the Windows 10 IoT Core version, but with a new shell called Windows Holographic on top of it.

In the sensors field, HoloLens is fully packed with four environment-understanding cameras, one depth camera, one HD photo and video camera, four microphones and one light sensor.

Source: Microsoft

These sensors make HoloLens capable of understanding and knowing the user's surroundings, but how can a developer use all these sensors and data?

Developing options

Microsoft is well-known for creating state-of-the-art development tools and easy-to-use SDKs so developers can focus on content and experiences without friction from the tools they use.

To develop for HoloLens, Microsoft has worked together with the developer community to introduce some interesting options.

First of all, as HoloLens uses Windows 10, every Windows 10 UWP (Universal Windows Platform) app out there can be executed on HoloLens without changing a single line of code. HoloLens automatically creates a Window for your app, enabling:

  • 3D space positioning
  • Movement
  • Rotation
  • Scaling
  • Gestures

But this simply shows a 2D surface with your app, not a fully holographic experience. This could be a good choice for Obeya rooms or information apps. Think about using HoloLens to put a video player on a wall and resizing the screen to 100” … Your very own cinema with HoloLens!

For truly immersive apps that make use of advanced gestures, voice recognition and spatial mapping, you need to recruit a 3D holographic developer, using well-known 3D engines. Currently, you can create 3D experiences in HoloLens using:

  • Unity 3D
  • Plain Concepts Wave Engine
  • Xamarin URHOSharp

Unity 3D is the official tool proposed by Microsoft to create 3D mixed reality experiences for HoloLens. So it is the one we will cover in this article.

HoloLens simulator

As of today, only developers in the USA and Canada can access a physical HoloLens device. To aid HoloLens development, Microsoft created a device simulator that can be installed in Visual Studio to help test the apps while in development.

This simulator loads a default room and presents you with the basic Windows Holographic UI so you can deploy your apps, both from Unity or UWP, and run them. Also, it includes some extra tools:

Simulation information, where you can enable/disable hand detection and see the detection simulation:

Or room configuration, where you can load rooms saved from a physical HoloLens device:

It can’t replace a real device for professional apps, but it can speed up the development for large teams with access to one physical device.

Developing for HoloLens

Creating a basic Unity3D experience

Microsoft has integrated the experience of developing HoloLens with Unity3D very well; you only need to take one thing into account: «one size unit in Unity3D equals one meter in the real world»

With this rule, you can start designing your objects and placing your camera. The start position of the camera would be (0,0,0) as the user's position in the room will define the camera position in runtime. In Unity3D, you only define the object’s position initially with the user position.

When you are ready to create your HoloLens app, go to the Build scene menu and select the Windows Store platform with the Universal SDK and D3D type of SDKs.

Then, in the Player Settings, you need to check Virtual Reality Supported and the Windows Holographic SDK.

And you are ready to go! Build your scene and open the project in Visual Studio to deploy to simulator or device.

Head tracking

Selecting objects in HoloLens is pretty easy. HoloLens tracks your head movement and traces a ray from it to infinity. If any object collides with it, HoloLens understands you are looking at it and you can select it.

To simplify user interaction, it is a good idea to add a 3D pointer in your scene. This could be any mesh you want with a little C# Script associated to it that tests the ray casting interactions between the head position and the gaze direction:

public class WorldCursor : MonoBehaviour

{

        private MeshRenderer meshRenderer;

        // Use this for initialization

        void Start ()

        {

                meshRenderer = this.gameObject.GetComponentInChildren<MeshRenderer>();

        }

       

        // Update is called once per frame

        void Update ()

        {

                var headPosition = Camera.main.transform.position;

                var gazeDirection = Camera.main.transform.forward;

                RaycastHit hitInfo;

                if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))

                {

                        // If the raycast hit a hologram...

                        // Display the cursor mesh.

                        meshRenderer.enabled = true;

                        // Move thecursor to the point where the raycast hit.

                        this.transform.position = hitInfo.point;

                        // Rotate the cursor to hug the surface of the hologram.

                        this.transform.rotation = Quaternion.FromToRotation(Vector3.up, hitInfo.normal);

                }

                else

                {

                        // If the raycast did not hit a hologram, hide the cursor mesh.

                        meshRenderer.enabled = false;

                }

        }

}

As you can see, HoloLens SDK is really easy to use – as for the above example, it gives the developer all the needed pre-processed information so you only need to focus on using the values from the current objects.

Why HoloLens?

HoloLens opens a new world of possibilities for creating human experience. Such a personal device, totally untethered, is able to know where you are and the exact shape and nature of your environment. It can open new methods for helping people and corporations in a way no other tool can.

How will HoloLens change our lives?

A wide range of possibilities is opened now in multiple areas, either as a professional tool or as a personal device.

It will change the way people communicate, allowing holoportation, bringing us the possibility to attend meetings « in person », only by the use of holograms. This will entail very significant cost savings while allowing a considerable increase in productivity.

Through holoportation, we will be able to attend meetings « in person » only by using holograms. Source: Microsoft

It will also change the academic world. For example, there will be the possibility of performing remote training for surgeons; nowadays, there is limited space in operation rooms and there is not always the possibility of attending surgeries; with the HoloLens, an unlimited number of people may attend a surgery.

Another academic use of the HoloLens is designed for students: they will be able to learn anatomy without using any books or computer screens; they will simply be able to view the organs and body systems just as they are in «reality».

HoloLens will change the world of academics: e.g., it will enable performing remote training for surgeons or learning anatomy without books or computer screens – just by viewing organs as they are in reality. Source: Microsoft

Another industry which will experience a major change will be the gaming industry. HoloLens is able to modify the environment in which a user plays. Both users and the real world will become part of the game, with them being able to interact with real objects and change their «state».

The gaming industry: HoloLens is able to modify the environment in which the user plays.

posted on 28.09.2016
Categories: 
by: Alfons Llopart Martinez