Virtual Audio Helps Pilots Hear Through the Noise

Test Pilot Dana Purifoy participates in research

It is difficult to imagine a more pressure-intensive environment than the mission control room at NASA's Dryden Flight Research Center. Simultaneously bombarded by a cacophony of mostly male voices speaking rapidly and at times unclearly, the mission control officer must quickly determine their level of importance, then decide whether to modify or stop the test in progress or tell the pilot and copilot to eject.

But by employing some of the same techniques used in arcade games, researchers led by Professor Walter Karplus, a faculty member of the Computer Science Department, are using "Virtual Audio" to help sort out some of the confusion. Eventually, they hope their research will lead to applications in other information-intense environments such as operating rooms and nuclear power plants.

In addition to the information coming in from the pilot and copilot flying each mission, the mission control officer receives continual updates from chase planes and down-range radar along with information from 20-30 engineers and technicians inside the mission control center. In the face of uncertainty, the mission controller, known as "NASA-1," must make correct decisions rapidly on the basis of many simultaneous inputs. "If the wrong action is taken, disasters can occur," Karplus says.

On the other end of the spectrum, the project addresses the boredom and tiredness that accompanies the need to maintain constant vigilance when there is little or no activity, which one researcher described as "like watching paint dry."

These conditions are similar to those familiar to air traffic controllers, who must keep track of dozens of planes and those experienced by surgeons, who must coordinate activities in an often-crowded operating room where, as Karplus points out "if the surgeon is misled, he may kill the patient."

Karplus is no stranger to either the operating room or virtual reality. He has successfully employed virtual reality to give medical doctors a preoperative method of observing how blood flows through aneurysms deep inside the human brain.

Part of the impetus for this research emanates from something called the "cocktail party effect." Standing amid a large gathering of people, the sound of voices intermingle into a low rumble or hum. Occasional words and phrases are intelligible, but for the most part, you cannot hear any of the multiple conversations. However, if someone at the far end of the room were to mention your name, chances are you would recognize it. In a similar sense, Karplus says, he and his researchers set out to determine "how much can be done with directional audio in an environment which normally would be complete babble."

In an effort to mimic these conditions, they developed a virtual or three-dimensional audio system which creates the "illusion that each voice channel comes from a different point in space."

From a purely technical point of view, Karplus says, they have developed a test-bed simulation of what NASA-1 has to do. The results of their research will allow them to optimize the human/computer interface to help him do his job better. Preliminary analysis of data collected during a six-month study, conducted primarily by computer science Ph.D student Michael D. Orosz, suggests three-dimensional audio is effective in optimizing information transfer, minimizing incorrect decisions and minimizing response time.

To create the three-dimensional effect, researchers relied upon earlier findings that indicate the manner in which sounds coming from various directions are perceived. This information was obtained by implanting tiny microphones inside subjects' ears. Data from the microphones was then used to produce algorithms which in turn were used to synthetically create the effect of sounds coming from various directions.

Until now, the results of this research were used primarily in arcade games. This project represents one of the first potentially useful applications of this technology.

To determine the optimum position for various sounds, Karplus and his researchers also drew upon the work of Caltech neurologist and Nobel Prize winner Roger W. Sperry in the area of cerebral specialization. Sperry's "split-brain" research determined which hemisphere of the brain is better suited to handle different types of information.

For example, in people who are right-dominant (usually right-handed), the left hemisphere has a greater capacity for understanding and processing information of a sequential nature, such as language and math, as well as performing analytic functions. The right hemisphere is better suited to pattern recognition, identifying pictures, faces, music and performing intuitive functions. So it is important that each type of information is directed toward the hemisphere of the brain which is better suited to process it.

The English language relies heavily upon the use of consonants to convey meaning. In fact, a page of text with all the vowels removed would still communicate much of the intended content. Recognizing consonants is a Left Brain function. Vowels - which convey mostly emotion - are better understood by the Right Brain. Therefore, messages of mainly an informational nature would be better routed to the Left Brain, while those of an emotional nature - such as an urgent call for help - would be more easily recognized by the Right Brain. This is only true, however, for people who are right-dominant. The outputs must be reversed for people who are left-dominant. This can be accomplished with a simple switch.

Orosz points out that sound need not be speech. The sound of an engine changing pitch can sometimes convey as much - if not more - information than a verbal description.

In the course of their study, researchers also developed a rule-based multimedia language. Previously, there was no well-defined method of mapping inputs and outputs. So they created a simple, easy to use language with if-then rules. Now, by using a simple text editor, anyone can incorporate data into a multimedia interface.

In the future, researchers hope to tackle the problems caused by "cognitive illusions." Cognitive illusions are "tricks" the senses sometimes play on the brain.

In an effort to eliminate these problems, researchers plan to attach what Karplus calls "earcons" - icons for the ears - to some sound channels. These would be small, specific, identifying sounds such as whistles and changes in volume or pitch which would help distinguish one voice from another. Earcons would also help alleviate the problem of vigilance by maintaining the listener's attention during times when there was no information on certain channels.

Additionally, Karplus plans to experiment with automatic speech recognition and voice recognition. Key words such as "Mayday" could be identified and given higher priority by changing their direction, raising their volume or coupling them with earcons or a visual cue. This would help eliminate the possibility of them going undetected.

All these measures improve the ratio of signal to noise and that results in better communication.

-David Brown