SHAPE AssetTags: a different way to create virtual augmentations for XR spaces

The earlier work with UWB tags generated an idea for something I am calling a SHAPE AssetTag. Essentially, this is a tag that is associated with a virtual SHAPE augmentation. The augmentation follows the position and orientation of the tag, making for a very simple way to implement augmented spaces. If engineered properly, it could be an extremely simple piece of hardware that would be essentially the UWB hardware along with a MEMS IMU and a battery. Instead of WiFi as in this prototype, pose updates could be sent over the UWB infrastructure to make things really simple. Ideally, these would be extremely cheap and could be placed anywhere in a space as a simple way of adding augmentations. These augmentations can be proxy objects (all aspects of a proxy object augmentation can be modified by remote servers) and really can be as simple or complex as desired.

Note that the SHAPE AssetTag doesn’t need to contain the actual asset data (although it could if desired). All it needs to do is to provide a URL of a repository where the asset (either Unity assetbundle or glTF blob) can be found. The asset is then streamed dynamically when it needs to be instantiated. It also provides information about where to find function servers in the case of a proxy object. The SHAPE device app (in this case an iOS app running on an iPad Pro) doesn’t need to know anything about SHAPE AssetTags – they just inject normal looking (but transient) augmentation updates into the SHAPE system so that augmentations magically appear. Obviously, this kind of flexibility could easily be abused and, in real life, a proper security strategy would need to be implemented in most cases. For development, though, it’s nice for things just to work!

One application that I like is a shared space where people can bring along their virtual creations in the form of some SHAPE AssetTags and just place them in the SHAPE-enhanced space so that any user in the space with XR devices could see them.

Another idea is that items in stores could have SHAPE AssetTags attached to them (like security tags today) so that looking at the item with an XR device would perhaps demonstrate some kind of feature. Manufacturers could supply the asset and functions servers, freeing the retail store from having to implement something for every stocked item. This could of course be done with QR codes but then the augmentations would not be physically locked to the item, something that could enable some very interesting augmentations. The item could be picked up and moved but the augmentation would retain the correct physical pose with respect to the item.

For now, the hardware is a complete hack with multiple components but it does prove that the concept is viable. In the photo above, the UWB tag (the white box on the floor under the figure’s right foot) controls the location of the augmentation in the physical space. A Raspberry Pi fitted with an IMU provides orientation information and drives the resulting pose via WiFi to the SHAPE servers. The augmentation is the glTF sample CesiumMan and includes animation data. Here are a couple of videos showing how the augmentation tracks the UWB tag around and that the IMU controls the augmentation’s orientation.

By the way, the software didn’t quite work first time…

So, is there any point to this? I am not sure. There are obviously many ways of doing the same thing without any physical hardware. However, the use of UWB makes it easy to achieve consistent results across multiple platforms with different spatial mapping as it provides absolute physical coordinates. Plus, there’s something fun about throwing a small tag on a surface and watching the augmentation appear!