HoloLens used as wayfinding device to guide blind people through complex buildings

Large, complex buildings can be difficult for anyone to navigate, but take away the benefit of sight and they can be downright impossible. But a solution may be on the horizon.

Recent prosthetic approaches to restore vision for the blind have attempted to convey raw images to the brain. These attempts have all suffered from the same problems, however: a lack of bandwidth and the extensive training required to interpret unusual stimuli.

This new approach, created by Researchers at the California Institute of Technology, takes an alternate approach that uses the Microsoft HoloLens mixed reality headset to restore vision at the cognitive level, bypassing the need to convey sensory data. The HoloLens is used to capture video and other data, extract the important scene knowledge, and convey that knowledge through auditory augmented reality.

The researchers say their design principle was to give sounds to all relevant objects in the environment. Each object in the scene can “talk” to the user with a voice that comes from the object’s location. As the object gets closer to the user, it’s pitch increases. The user has several modes of control to select which objects speak: Scan, Spotlight, and Target.

See Also: Altered reality showdown

In Scan mode the objects call out their names in sequence from left to right to provide a quick overview of the scene. In Spotlight mode, the object directly in front of the user speaks, allowing the user to explore the scene by moving their head. In Target mode, the user selects one object that calls repeatedly at the press of a clicker. Obstacles and walls emit a hissing sound as the user gets too close.

For long range navigation, the system becomes a “virtual guide.” The guide follows a precomputed path through a building that has been pre-scanned with the HoloLens. The guide will repeatedly call out “follow me,” staying one meter ahead of the user, and will offer warnings about any upcoming turns or flights of stairs.

The researchers say neither training nor modification of the physical environment are required to use any of the system’s modes and blind subjects will be able to navigate an unfamiliar multi-story building on their first attempt.

You can watch a video of one of the test subjects using the system here.