Sensory Substitution Technology

Technology Will Change How We Perceive Reality

Neil Harbisson
Photo licensed under CC by Flickr user Campus Party Europe in Berlin.

Our senses are a window to our reality. They are fundamental, and inescapable. But even our most basic interface with the world is susceptible to the influences of technology. One of the ways that technology can shape our perception is via sensory substitution.

What is Sensory Substitution?

Sensory substitution is the act of using technology to convert one sensory stimuli into another. A traditional example of this is Braille. Braille lettering converts the visual stimuli of print into raised bumps, sensed by touch.

It can take some time for the brain to adjust to substituting one sense for another, but after the adjustment period, it begins to interpret the stimuli using the other sense. Many blind people can read using braille with the same ease and effortlessness as someone reading print. 

It Works Because the Brain is Adaptable

This flexibility of the brain is not just limited to reading using touch. Researchers have identified a visual cortex in the brain dedicated to sight. Yet in blind people, this region gets used for other tasks. 

This adaptability of the mind allows researchers to push sensory substitution beyond braille. More sophisticated forms of sensory substitution are being developed, and are now emerging.

Modern Examples and Advocates

Sonic glasses are a more recent example of sensory substitution. These glasses use a camera mounted in the sight-line of the user. The camera converts what the user is seeing into sound, varying the pitch and volume based on what's seen. Given time to adapt, this technology can restore a sense of sight to the user. 

Neil Harbisson, an advocate of this tech, had an antenna permanently attached to his skull. The antenna translates color into sound. Harbisson, who is colorblind, reported that after some time with the antenna, he began to perceive colors. He even began to dream in color where before he couldn’t. His decision to fix the antenna to his skull gained him publicity as an advocate for cyborgs in society.

Another proponent of sensory substitution is David Eagleman. A researcher at Baylor University, Dr. Eagleman has developed a vest with a series of vibrating motors. The vest can translate many different types of sensory input into patterns of vibration on the user’s back. An early test showed a profoundly deaf person able to perceive spoken words after 4 sessions wearing the vest.

Creating New Senses

An interesting further application of this vest is that it can extend beyond the traditional senses. We perceive only a thin slice of the information that is available to us as part of our reality. For example, the vest can link to sensors that offer perception in other modalities, beyond hearing, such as sight. It could allow the user to "see" beyond visible light, into infrared, ultraviolet, or radio waves.

In fact, Dr. Eagleman has put forward the idea of perceiving things beyond what we understand as reality. One experiment had the vest present the user with tactile information about the state of the stock market. This allowed the user to potentially perceive the economic system as though it were any other sense, like sight. The user was then asked to make stock transaction decisions based on how they felt. Dr. Eagleman’s lab is still determining whether a human can develop an intuitive “sense” of the stock market.

Tech Will Shape Our Understanding of Reality

The ability to perceive systems like the stock market is an early research topic. But, if the brain can adapt to perceive sight or sound via touch, there may be no end to its ability to perceive complex things. Once the brain becomes acclimatized to perceiving the entire market, it could act instinctively. This could allow users to make trading decisions below the level of conscious awareness. Eagleman calls this a “new brain” receiving input far beyond the traditional 5 senses.

This seems far from reality, but the technology to potentially make this happen already exists. The idea is complex, but the principles have proven sound since the creation of Braille. 

Technology will become a layer between the world and our minds. It will mediate our entire perception of the world, making the invisible things in our reality visible.