Building the HoloToolkit.Sharing Library

Collaboration using AR is a fascinating area with many potential applications. The HoloToolkit is a very handy resource in general and includes the HoloToolkit.Sharing library to assist with collaboration. The HoloToolkit-Unity actually contains built versions of the Server, SessionManager and Profiler but it seemed like a good idea to build from scratch.

There are a few pre-requisites:

  • Windows SDK 10.0.10240
  • Windows SDK 10.0.10586
  • Common Tools for Visual C++
  • Windows 8.1 SDK and Universal CRT SDK
  • Java 8 SDK

An easy way to get the Windows SDKs is to run the BuildAll.bat script which will exit with an error if something is missing. Then use the solution file for the element that failed to start VS2015. VS2015 will then install the missing components. The Java SDK needs to be installed manually and requires environment variables JAVA_BIN that points to the JDK bin directory and JAVA_INCLUDE that points to the JDK include directory. The BuildAll.bat script should then complete successfully.

The Server is run using SharingService.exe and the user needs administrator permission to install as a service. This can be done by opening a command window in administrator mode and running the command from that for example. It’s actually useful to run the server using the -local flag (as a command line program) as then it’s easy to see status and error messages. The SessionManager displays current server state including connected clients.

ER – Enhanced Reality

I was intrigued by the Vrvana VR + stereo camera pass-through headset. This is a practical version of something that I have been thinking about for a while. HoloLens does a fantastic job of AR/MR but it does have a limited field of view, something that may be inevitable with the waveguide type design. The other limitation is that it can only overlay on reality, not selectively replace it (at least not in all lighting conditions).

Enhanced Reality (ER), by using stereo cameras whose feeds are displayed as in a conventional VR headset can solve the field of view problem and allow all aspects of the field to be enhanced, replaced or overlaid as required. Take the case of using the headset for driving a car (although people will probably not be doing that much longer). For a start, the car would no longer need a dashboard or any instrumentation – everything would be virtual, including big touchscreen displays. Looking out of the windscreen, the image seen could be augmented by data from radar, IR cameras or anything else that enhances the experience. Objects of interest could be enhanced in brightness perhaps. It could incorporate Google Translate style street sign translation and replacement. Obviously any other useful heads-up data could be displayed, such as navigation, speed, temperature etc.

Complex airplane cockpits could potentially be a thing of the past also. Most of the cockpit consists of devices that give information to the pilots or are simple switches and levers. All of these could be virtual. Maybe you keep a joystick and a couple of rudder pedals but that would be it – it’d just be two chairs in a room :-). Meanwhile, the view out of the windscreen could be enhanced in the same way as described earlier.

I am sure there are many applications where the ability to enhance, modify and replace any part of the field of view would be of value. An important aspect of a true ER headset is that even very bright features in the field can be replaced, something that is difficult to do with AR headsets.

The HoloLens is absolute magic

hololens2I really don’t know how this guy got into my office. Spent the last couple of hours trying to grab the HoloLens back from people in the house playing Fragments. Of course now everyone wants me to buy a HoloLens. The quality is actually much better than the screen shot suggests – the rats running around on the office floor were especially amusing. And who are these people standing around?hololens3I used the HoloLens companion app to get these screen captures – it allows others to see what the wearer is seeing.

The nice thing is that nobody has complained of any motion effects. I am usually the most sensitive and can only last about five minutes with an Oculus DK2 but even I survived just fine. Being untethered with no wires or other boxes to worry about is a major plus. Just put it on and play.

HoloLens: best new toy in the office…ever

hololensManaged to get hold of a HoloLens for a while to try out some ideas. I think that Microsoft’s latest 3D effort is really exciting and it’s a very active time for AR/MR in general as the Lenovo Phab 2 Pro Tango enabled phone is due to be available in the near future. Now looking forward to exploring some of Microsoft’s demo apps for the HoloLens along with using the HoloLens version of Unity for some custom work.

Augmented reality

ARGlassesI think this photo is hilarious – I look more demented than augmented. I broke out some old Silicon Micro Display AR glasses from a previous existence and I might revive some old code that works with them. Since the display just plugs in as a standard HDMI monitor, it’s pretty easy to work with. I am thinking about using a Raspberry Pi 3 with an OpenGL app to overlay the real world. Not Magic Leap for sure but it’s the easiest way to get going with some AR ideas.