Mixed Reality and the missing fourth dimension

The screen capture above is a scene from a HoloLens via mixed reality capture (MRC) showing four virtual rings with different levels or brightness. The top left is 100% red, the bottom right black and the other two are intermediate levels of brightness.

The photograph above was shot through a HoloLens and is a reasonable representation of what the wearer actually sees. Unsurprisingly, since all see-through MR headsets work by overlaying light on the real scene, the black ring has vanished and the intermediate brightness rings become transparent to some degree based on the relative brightness to the real world scene.

This is a considerable obstacle for inserting realistic virtual objects into the real world – if they are dark, they will be almost transparent. And while indoors it is possible to control ambient lighting, the same is certainly not true outdoors.

What is needed is not just support for RGB but RGBA where A is the fourth dimension of color in this case. The A (alpha) value specifies the required transparency. The Unity app running on the HoloLens does of course understand transparency and can generate the required data but the HoloLens has no way to enforce it. One way to do this would be to supplement the display with an LCD that acts as a controllable matte. The LCD controls the extent to which the real world is visible at each display pixel while the existing display controls the color and intensity of the virtual object. No doubt there are significant challenges to implementation but this may be the only way to make see-through MR headsets work properly outdoors.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.