What you see affects what you hear
By Ruth SoRelle, M.P.H.
Can you read lips? You probably think you cannot. However, researchers at Baylor College of Medicine and City College of New York report that what you see can improve your understanding what you hear – as much as sixfold. In fact, you lip-read more than you realize.
It all comes together in your brain, which marries the visual information you derive from the person's face and lip movements to help you interpret what you hear, said Wei Ji Ma, Ph.D., assistant professor of neuroscience at BCM and the lead author, in a report that appears online in the open access journal PLoS One, (Public Library of Science One).
"Most people with normal hearing lip-read very well, even though they don't think so," said Ma. In fact, lip reading is most helpful when there is a moderate level of noise, he said.
Only good in minimum sound
It does not significantly improve understanding in extremely noisy rooms or if the person speaking has a very faint voice.
"We find that a minimum sound level is needed for lip-reading to be most effective," said Ma.
Previous research had studied the effect of lip reading using only a specific, limited list of words from which people chose. Ma's work was the first to study word recognition in setting where people could freely report what they thought they heard.
The lip-reading data help scientists understand how the brain integrates two different kinds of stimuli – visual and auditory.
Mathematical model
Ma and his colleagues constructed a mathematical model that allowed them to predict how successful a person will be at integrating the visual and auditory information.
People actually combine the two stimuli close to optimally, Ma said. What they perceive depends on the reliability of the stimuli.
"Suppose you are a detective," he said. "You have two witnesses to a crime. One is very precise and believable. The other one is not as believable. You take information from both and weigh the believability of each in your determination of what happened."
In a way, lip-reading involves the same kind of integration of information in the brain, he said.
Full face vs. cartoon
In some situations, the person's full face is shown and the person speaks normally with the visual matching what the person hears. In that case, the visual information – the lip reading cues – actually help with understanding when there is moderate background noise.
In others, the person's face is cropped down to almost a "cartoon" that does not truly mouth the word. Surprisingly, the visual information can provide some help, but not as much as when the full face is shown.
Different stimuli, wrong word
In another study, the person mouths one word but the audio projects another, and often the brain integrates the two stimuli into a totally different word.
"The mathematical model can predict how often the person will understand the word correctly in all these contexts," Ma said.
An example of the visual and audio stimuli used in the experiment can be found at http://bme.engr.ccny.cuny.edu/faculty/parra/bayes-speech.
For another video of this experiment and press release on it, go to http://www.bcm.edu/news/lipreading.cfm
Others who took part in this research include Xiang Zhou, Lars A. Ross, John J. Foxe and Lucas C. Parra of The City College of New York in New York City.
