WebRTC

Creating a new plugin for the Janus WebRTC server

I am working on a system to support multi-site podcasting using WebRTC and the Janus Server seemed like a good place to start. None of the example plugins does exactly what I want so, rather than modify an existing plugin, I decided to create a new one based on an existing one (videoroom). The screen capture shows the result. At this stage, it is identical to the video room plugin hence the identical look of the test. There are a few steps to doing this such that it is integrated into the configuration and build system and there’s no way I will remember them, hence this aide-memoire!

One thing I noticed which has nothing to do with a new plugin is that I needed to install gtk-doc-tools before I could compile libnice as described in the dependency section of the readme.

Anyway, the janus-gateway repo has a plugins directory that contains c source (amongst other things) of the various plugins. I decided to base my new plugin on the videoroom plugin so I copied janus_videoroom.c into rt_podcall.c for the new plugin. Then, using a text editor, I changed all forms of text involving “videoroom” into “podcall”.

Once the source is created, it can be added into the configure.ac file which is in the root of the repo. Basically, I copied anything involving “videoroom” and changed the text from “videoroom” to “podcall”. The same also needs to be done for Makefile.am.

It is also necessary to create a configuration file for the new plugin. The repo root has a directory called conf which is where all of the configurations are held. I copied the janus.plugin.videoroom.jcfg.sample into janus.plugin.podcall.jcfg.sample to satisfy that requirement.

In order to test the plugin, it’s useful to add code into the existing demo system. The repo root has a directory called html that contains the test code. I copied videoroomtest.html and videoroomtest.js into podcall.html and podcall.js and edited the files to fix the references (such as plugin name) from videoroom to podcall.

To make the test available in the Demos dropdown, edit navbar.html and add the appropriate line in the dropdown menu.

Once all that’s done, it should be possible to build and install the modified Janus server:

sh autogen.sh
./configure --prefix=/opt/janus
make
sudo make install
sudo make configs

The Janus server needs a webserver in order to run these tests. I used a very simple Python server to do this:

from http.server import HTTPServer, SimpleHTTPRequestHandler
import ssl

server_address = ('localhost', 8080)
httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket,
                               certfile='../certs/mycert.pem',
                               keyfile='../certs/mycert.key',
                               server_side=True)
httpd.serve_forever()

This is run with Python3 in the html directory and borrows the sample Janus certificates to support ssl. Replace localhost with a real IP address to allow access this server outside of the local machine.

Using ARKit with ExpoKit, React Native, three.js and soon (hopefully) WebRTC

This rather unimpressive test scene, captured from an iPhone, is actually quite interesting. It is derived from a simple test app using Expo that makes it easy to use React Native, ARKit and three.js to generate native iOS (and Android although not with ARKit of course) apps. Expo provides a nice environment where a standard app supports rapid development of javascript apps on top of the underlying native support. This test app worked will in Expo within a very short space of time.

The only problem is that I also want to support WebRTC in the app. There is a React Native WebRTC implementation but as far as I can tell it requires that the app be detached from Expo to ExpoKit so that it can be included in Xcode. Unfortunately, that didn’t work as AR support didn’t seem to be included in the automatically generated project.

To include ARKit support requires that the Podfile in the project’s ios directory be modified to add AR support. The first section should look like this:

source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '9.0'
target 'test' do
  pod 'ExpoKit',
   :git => "http://github.com/expo/expo.git",
   :tag => "ios/2.0.3",
   :subspecs => [
     "Core",
     "CPP”,
     "AR"
   ]
...

Basically “AR” is added as an extra subspec. Then ARKit seems to work quite happily with ExpoKit.

HoloLens Spectator View…without the HoloLens

I’ll explain the photo above in a moment. Microsoft’s Spectator View is a great device but not that practical in the general case. For example, the original requires modifications to the HoloLens itself and a fairly costly camera capable of outputting clean 1080p, 2k or 4k video on an HDMI port. Total cost can be more than $6000 depending on the camera used. My goal is to do much the same thing but without requiring a HoloLens and at a much lower cost – just using a standard camera with fairly simple calibration. Not only that, but I want to stream the mixed reality video across the internet using WebRTC for both conventional and stereo headsets (such as VR headsets).

So, why is there a HoloLens in the photo? This is the calibration setup. The camera that I am using for this Mixed Reality streaming system is a Stereolabs ZED. I have been working with this quite a bit lately and it seems to work extremely well. Notably it can produce a 2K stereo 3D output, a depth map and a 6 DoF pose, all available via a USB 3 interface and a very easy to use SDK.

Unlike Spectator View, the Unity Editor is not used on the desktop. Instead, a standard HoloLens UWP app is run on a Windows 10 desktop, along with a separate capture, compositor and WebRTC streamer program. There is some special code in the HoloLens app that talks to the rest of the streaming system. This can be present in the HoloLens app even when run on the HoloLens without problems (it just remains inactive in this case).

The calibration process determines, amongst other things, the actual field of view of the ZED and its orientation and position in the Unity scene used to create the virtual part of the mixed reality scene. This is essential in order to correctly render the virtual scene in a form that can be composited with the video coming from the ZED. This is why the HoloLens is placed in this prototype rig in the photo. It puts the HoloLens camera roughly in the same vertical plane as the ZED camera with a small (known) vertical offset. It’s not critical to get the orientation exactly right when fitting the HoloLens to the rig – this can be calibrated out very easily. The important thing is that the cameras see roughly the same field. That’s the because the next step matches features in each view and, from the positions of the matches, can derive the field of view of the ZED and its pose offset from the HoloLens. This then makes it possible to set the Unity camera in the desktop in exactly the right position and orientation so that the scene it streams is correctly composed.

Once the calibration step has completed, the HoloLens can be removed and used as required. The prototype version looks very ungainly like this! The real version will have a nice 3D printed bracket system that will also have the advantage of reducing the vertical separation and limit the possible offsets.

In operation, it is required that the HoloLens apps running on both the HoloLens(es) and the desktop are sharing data about the Unity scene that allows each device to compute exactly the same scene. In this way, everyone sees the same thing. I am actually using Arvizio‘s own sharing system but any sharing system could be used. The Unity scene generated on the desktop is then composited with the ZED camera’s video feed and streamed over WebRTC. The nice thing about using WebRTC is that almost anyone with a Chrome or Firefox browser can display the mixed reality stream without having to install any plugins or extensions. It is also worth mentioning that the ZED does not have to remain fixed in place after calibration. Because it is able to measure its pose with respect to its surroundings, the ZED could potentially pan, tilt and dolly if that is required.

Using C++ REST SDK as a static library

cpprestsdk is a handy library for implementing REST interfaces in C++ code. On Windows it is conveniently available as a NuGet package. However, it is a dynamic link library (DLL). Normally that’s not a problem except for the fact that I am trying to combine it with WebRTC which builds as a static library and mixing the two causes strange crashes. I don’t know if it is possible to compile WebRTC as a DLL – the ninja build is very complicated :-(.

I found this very helpful post which provides life-saving instructions on how to build and use cpprestsdk as a static linked library. In fact the cpprestsdk Visual Studio solution now includes a project to build a static library so that part is mostly taken care of. It is still necessary to change the Runtime Library option to /MTd (debug) and /MT (release) in the casablanca140.static project however.

To use the library, the preprocessor symbol _NO_ASYNCRTIMP needs to be defined in the project as described in the post. You also have to add winhttp.lib, crypt32.lib, bcrypt.lib and zlibstatic.lib as additional linker dependencies. The library for the latter needs to copied somewhere that the project can find – it can be obtained from the packages in the cpprestsdk folder for zlib.

Another thing is that the static version of cpprestsdk needs to be built in Debug and Release modes if the new project is to run in both modes. With some things you can use Release for both – this doesn’t seem to work for cpprestsdk.

The result now is that I can use REST for signaling in my WebRTC project – not as easy as I expected!

Developing Electron apps with Visual Studio Code

I have been trying out Electron as a way of developing some WebRTC apps to work with the Janus gateway. In the end I decided that Visual Studio Code was a good route to take for Javascript code development. One thing that wasn’t at all obvious though was how to get breakpoints to work. I found this blog entry that had the answer – no way I would have been able to work it out myself so go to that link for the original source (reproduced here for my convenience).

First thing is to install the Debugger for Chrome extension for VS Code – instructions are here. Then, the .vscode/launch.json file should look something like this:

{
  // Use IntelliSense to learn about possible Node.js debug attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Main Debug",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron",
      "windows": {
        "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron.cmd"
      },
      "program": "${workspaceRoot}/main.js",
      "protocol": "legacy"
    },
    {
      "name": "Renderer Debug",
      "type": "chrome",
      "request": "launch",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/electron",
      "runtimeArgs": [
        "${workspaceRoot}",
        "--enable-logging",
        "--remote-debugging-port=9222"
      ],
      "sourceMaps": false
    }
  ]
}

Using this launch file, to debug in the main process use Main Debug, to debug in the renderer process use Renderer Debug.

WebRTC on the Moverio BT-300

Since the BT-300 is an Android device, I thought it would be fun to try out WebRTC on it using the Android WebRTC sample here. Right now it’s WebRTC everything chez richard so this was a natural thing to test on the BT-300.

It worked just fine. The screen capture above shows the browser display from a desktop running Chrome. I am looking at the display hence the funky effect. The photo below is an attempt to show what it looks like in the BT-300.

In this case, the desktop camera is watching me desperately trying to juggle my camera and the BT-300 to get this image which is the left eye display from the BT-300. I must stress that the real quality of the image on the BT-300 is vastly superior to how it looks here – it’s really tricky to get any sort of photo in fact.

Building the Android sample takes a long time – in fact downloading the code takes for ever as it is about 16G of download! One extra point is that, after the gsync line, it is necessary to enter:

build/install-build-deps-android.sh

in order to get all the prerequisites installed. Apart from that, everything was straightforward.