WebRTC

Using ARKit with ExpoKit, React Native, three.js and soon (hopefully) WebRTC

This rather unimpressive test scene, captured from an iPhone, is actually quite interesting. It is derived from a simple test app using Expo that makes it easy to use React Native, ARKit and three.js to generate native iOS (and Android although not with ARKit of course) apps. Expo provides a nice environment where a standard app supports rapid development of javascript apps on top of the underlying native support. This test app worked will in Expo within a very short space of time.

The only problem is that I also want to support WebRTC in the app. There is a React Native WebRTC implementation but as far as I can tell it requires that the app be detached from Expo to ExpoKit so that it can be included in Xcode. Unfortunately, that didn’t work as AR support didn’t seem to be included in the automatically generated project.

To include ARKit support requires that the Podfile in the project’s ios directory be modified to add AR support. The first section should look like this:

source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '9.0'
target 'test' do
  pod 'ExpoKit',
   :git => "http://github.com/expo/expo.git",
   :tag => "ios/2.0.3",
   :subspecs => [
     "Core",
     "CPP”,
     "AR"
   ]
...

Basically “AR” is added as an extra subspec. Then ARKit seems to work quite happily with ExpoKit.

HoloLens Spectator View…without the HoloLens

I’ll explain the photo above in a moment. Microsoft’s Spectator View is a great device but not that practical in the general case. For example, the original requires modifications to the HoloLens itself and a fairly costly camera capable of outputting clean 1080p, 2k or 4k video on an HDMI port. Total cost can be more than $6000 depending on the camera used. My goal is to do much the same thing but without requiring a HoloLens and at a much lower cost – just using a standard camera with fairly simple calibration. Not only that, but I want to stream the mixed reality video across the internet using WebRTC for both conventional and stereo headsets (such as VR headsets).

So, why is there a HoloLens in the photo? This is the calibration setup. The camera that I am using for this Mixed Reality streaming system is a Stereolabs ZED. I have been working with this quite a bit lately and it seems to work extremely well. Notably it can produce a 2K stereo 3D output, a depth map and a 6 DoF pose, all available via a USB 3 interface and a very easy to use SDK.

Unlike Spectator View, the Unity Editor is not used on the desktop. Instead, a standard HoloLens UWP app is run on a Windows 10 desktop, along with a separate capture, compositor and WebRTC streamer program. There is some special code in the HoloLens app that talks to the rest of the streaming system. This can be present in the HoloLens app even when run on the HoloLens without problems (it just remains inactive in this case).

The calibration process determines, amongst other things, the actual field of view of the ZED and its orientation and position in the Unity scene used to create the virtual part of the mixed reality scene. This is essential in order to correctly render the virtual scene in a form that can be composited with the video coming from the ZED. This is why the HoloLens is placed in this prototype rig in the photo. It puts the HoloLens camera roughly in the same vertical plane as the ZED camera with a small (known) vertical offset. It’s not critical to get the orientation exactly right when fitting the HoloLens to the rig – this can be calibrated out very easily. The important thing is that the cameras see roughly the same field. That’s the because the next step matches features in each view and, from the positions of the matches, can derive the field of view of the ZED and its pose offset from the HoloLens. This then makes it possible to set the Unity camera in the desktop in exactly the right position and orientation so that the scene it streams is correctly composed.

Once the calibration step has completed, the HoloLens can be removed and used as required. The prototype version looks very ungainly like this! The real version will have a nice 3D printed bracket system that will also have the advantage of reducing the vertical separation and limit the possible offsets.

In operation, it is required that the HoloLens apps running on both the HoloLens(es) and the desktop are sharing data about the Unity scene that allows each device to compute exactly the same scene. In this way, everyone sees the same thing. I am actually using Arvizio‘s own sharing system but any sharing system could be used. The Unity scene generated on the desktop is then composited with the ZED camera’s video feed and streamed over WebRTC. The nice thing about using WebRTC is that almost anyone with a Chrome or Firefox browser can display the mixed reality stream without having to install any plugins or extensions. It is also worth mentioning that the ZED does not have to remain fixed in place after calibration. Because it is able to measure its pose with respect to its surroundings, the ZED could potentially pan, tilt and dolly if that is required.

Using C++ REST SDK as a static library

cpprestsdk is a handy library for implementing REST interfaces in C++ code. On Windows it is conveniently available as a NuGet package. However, it is a dynamic link library (DLL). Normally that’s not a problem except for the fact that I am trying to combine it with WebRTC which builds as a static library and mixing the two causes strange crashes. I don’t know if it is possible to compile WebRTC as a DLL – the ninja build is very complicated :-(.

I found this very helpful post which provides life-saving instructions on how to build and use cpprestsdk as a static linked library. In fact the cpprestsdk Visual Studio solution now includes a project to build a static library so that part is mostly taken care of. It is still necessary to change the Runtime Library option to /MTd (debug) and /MT (release) in the casablanca140.static project however.

To use the library, the preprocessor symbol _NO_ASYNCRTIMP needs to be defined in the project as described in the post. You also have to add winhttp.lib, crypt32.lib, bcrypt.lib and zlibstatic.lib as additional linker dependencies. The library for the latter needs to copied somewhere that the project can find – it can be obtained from the packages in the cpprestsdk folder for zlib.

Another thing is that the static version of cpprestsdk needs to be built in Debug and Release modes if the new project is to run in both modes. With some things you can use Release for both – this doesn’t seem to work for cpprestsdk.

The result now is that I can use REST for signaling in my WebRTC project – not as easy as I expected!

Developing Electron apps with Visual Studio Code

I have been trying out Electron as a way of developing some WebRTC apps to work with the Janus gateway. In the end I decided that Visual Studio Code was a good route to take for Javascript code development. One thing that wasn’t at all obvious though was how to get breakpoints to work. I found this blog entry that had the answer – no way I would have been able to work it out myself so go to that link for the original source (reproduced here for my convenience).

First thing is to install the Debugger for Chrome extension for VS Code – instructions are here. Then, the .vscode/launch.json file should look something like this:

{
  // Use IntelliSense to learn about possible Node.js debug attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Main Debug",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron",
      "windows": {
        "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron.cmd"
      },
      "program": "${workspaceRoot}/main.js",
      "protocol": "legacy"
    },
    {
      "name": "Renderer Debug",
      "type": "chrome",
      "request": "launch",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron",
      "runtimeArgs": [
        "${workspaceRoot}",
        "--enable-logging",
        "--remote-debugging-port=9222"
      ],
      "sourceMaps": false
    }
  ]
}

Using this launch file, to debug in the main process use Main Debug, to debug in the renderer process use Renderer Debug.

WebRTC on the Moverio BT-300

Since the BT-300 is an Android device, I thought it would be fun to try out WebRTC on it using the Android WebRTC sample here. Right now it’s WebRTC everything chez richard so this was a natural thing to test on the BT-300.

It worked just fine. The screen capture above shows the browser display from a desktop running Chrome. I am looking at the display hence the funky effect. The photo below is an attempt to show what it looks like in the BT-300.

In this case, the desktop camera is watching me desperately trying to juggle my camera and the BT-300 to get this image which is the left eye display from the BT-300. I must stress that the real quality of the image on the BT-300 is vastly superior to how it looks here – it’s really tricky to get any sort of photo in fact.

Building the Android sample takes a long time – in fact downloading the code takes for ever as it is about 16G of download! One extra point is that, after the gsync line, it is necessary to enter:

build/install-build-deps-android.sh

in order to get all the prerequisites installed. Apart from that, everything was straightforward.