SHAPE AssetTags: a different way to create virtual augmentations for XR spaces

The earlier work with UWB tags generated an idea for something I am calling a SHAPE AssetTag. Essentially, this is a tag that is associated with a virtual SHAPE augmentation. The augmentation follows the position and orientation of the tag, making for a very simple way to implement augmented spaces. If engineered properly, it could be an extremely simple piece of hardware that would be essentially the UWB hardware along with a MEMS IMU and a battery. Instead of WiFi as in this prototype, pose updates could be sent over the UWB infrastructure to make things really simple. Ideally, these would be extremely cheap and could be placed anywhere in a space as a simple way of adding augmentations. These augmentations can be proxy objects (all aspects of a proxy object augmentation can be modified by remote servers) and really can be as simple or complex as desired.

Note that the SHAPE AssetTag doesn’t need to contain the actual asset data (although it could if desired). All it needs to do is to provide a URL of a repository where the asset (either Unity assetbundle or glTF blob) can be found. The asset is then streamed dynamically when it needs to be instantiated. It also provides information about where to find function servers in the case of a proxy object. The SHAPE device app (in this case an iOS app running on an iPad Pro) doesn’t need to know anything about SHAPE AssetTags – they just inject normal looking (but transient) augmentation updates into the SHAPE system so that augmentations magically appear. Obviously, this kind of flexibility could easily be abused and, in real life, a proper security strategy would need to be implemented in most cases. For development, though, it’s nice for things just to work!

One application that I like is a shared space where people can bring along their virtual creations in the form of some SHAPE AssetTags and just place them in the SHAPE-enhanced space so that any user in the space with XR devices could see them.

Another idea is that items in stores could have SHAPE AssetTags attached to them (like security tags today) so that looking at the item with an XR device would perhaps demonstrate some kind of feature. Manufacturers could supply the asset and functions servers, freeing the retail store from having to implement something for every stocked item. This could of course be done with QR codes but then the augmentations would not be physically locked to the item, something that could enable some very interesting augmentations. The item could be picked up and moved but the augmentation would retain the correct physical pose with respect to the item.

For now, the hardware is a complete hack with multiple components but it does prove that the concept is viable. In the photo above, the UWB tag (the white box on the floor under the figure’s right foot) controls the location of the augmentation in the physical space. A Raspberry Pi fitted with an IMU provides orientation information and drives the resulting pose via WiFi to the SHAPE servers. The augmentation is the glTF sample CesiumMan and includes animation data. Here are a couple of videos showing how the augmentation tracks the UWB tag around and that the IMU controls the augmentation’s orientation.

By the way, the software didn’t quite work first time…

So, is there any point to this? I am not sure. There are obviously many ways of doing the same thing without any physical hardware. However, the use of UWB makes it easy to achieve consistent results across multiple platforms with different spatial mapping as it provides absolute physical coordinates. Plus, there’s something fun about throwing a small tag on a surface and watching the augmentation appear!

imu and imuview – adding IMU sensing to rtndf data flow pipelines

imuviewUp to now the only data sources in rtndf were video and audio. imu is a new Python PPE that can be used to stream IMU data (fused pose, sensor readings etc) into an rtndf data flow pipeline. Another new PPE is imuview, this time a C++ PPE, that can display the resulting stream. The screen capture above shows the data being streamed from a Raspberry Pi SenseHat which is a full 11-dof sensor.

One of the nice things about using a pub/sub system like MQTT is that it is possible to hook into any of the pipeline links to see what data is flowing. To this end, a future PPE will be a generic viewer. The user just gives it the topic and it determines the type of data and displays it appropriately. A very handy debugging tool!