Marcel Proust told us years ago that an image can summon up remembrances of things past, in his case thousands of words triggered by a single madeleine. Now an international team of neuroscientists has shown that, just as Proust’s gateau led to a string of novels, a few spoken words can summon up an image of the speaker’s face.
Here’s how: Test subjects watched a video of a stranger identified only by profession talking for about two minutes. Later, when the subjects were hooked up to a functional MRI and heard an audiotape of the same speaker saying something entirely different, the part of the brain which stores the faces we have seen lit up like the Fourth of July.
In Proust’s A la recherche du temps perdu, seeing, smelling, and tasting a madeleine brought back a flood of memories for the narrator. Perhaps if madeleines could talk, their sweet murmurings would have started Proust off on a very long series of pictures instead.
It turns out that speech associated with a real, live face, or a video of one, creates a visual memory in the brain. When a control group listened to a voice while viewing not the speaker’s face, but a picture illustrating that person’s occupation, their brains didn’t light up the same way.
With an fMRI, subjects who heard strangers talking while watching videotapes of their faces showed more brain activity in the area of the brain that processes images than those who listened to the same voices but saw slides of inanimate objects instead of the speakers’ faces.
The researchers have concluded from their fMRI data that the brain stores facial signatures which enable us to predict speech content when we hear the sound of a familiar voice. They note that we all use facial cues to help interpret conversation in a noisy room, and that displaying a talking face helps the hearing impaired to better understand a phone conversation, even if it’s not the face of the caller, and even if the face is not that of a real person, but is computer generated. So they suggest that simulated faces might be deployed to increase comprehension of the spoken word in many other situations, too.
But the general public has been cool to the idea of the videophone – webcam addicts excepted, no one really likes the idea of callers reading our lips in the privacy of our homes. Nor are talking faces likely to clarify some of the most common instances of hard-to-decipher speech:
· Hello, this is Capt. XBCHKJL&*. Right now we anticipate a delay of %%%^&% with an estimated wheels up time of *&%^$(*.
· Take the first right, then left about a mile down, east, I think it is, on Jones, or Stewart – does Stewart cross Jones there? – and then it’s about five minutes – you can’t miss it, there a gas station on the corner, or maybe it’s a bank now?
· Welcome to customer support . . . .
As Pres. George H. W. Bush learned, even when the message is clear, it may not be best to have people read your lips. Bush uttered his famous phrase, “Read my lips. No new taxes,” at the 1988 Republican National Convention. He won the presidential race that year but wound up eating his words, raising taxes, and making fewer television appearances after he was voted out of office.
Presidential gaffes notwithstanding, the investigators don’t actually recommend television for visual speech reinforcement, because, while adults may benefit from watching tapes of people talking, it’s not clear that either recorded or simulated talking heads help children to understand their native language or to learn a foreign one.
This is bad news for the many parents buying foreign-language DVDs to improve their children’s chances of getting into a selective nursery school. Another team of researchers has shown that 9-month-old infants listening to as little as 5 hours of Mandarin spoken to them, in person, by a native speaker, could distinguish the sounds of Chinese, while another group watching a DVD of the same material could not. Apparently, despite the fact that familiar voices can trigger brain waves of test subjects trained to recognize a videotaped face, it’s human interaction, not staring at a screen, that stimulates real language learning.
Children have an amazing ability to soak up language, but plunking them down in front of the TV to watch Sesame Street in Chinese won’t give kids a head start in the college race. Parents intent on building their children’s résumés will just have to forget about programs like Baby Einstein or the Rosetta Stone and wheel their strollers down to the local Berlitz – or they could just serve fewer burgers and more tofu and hire a Chinese nanny.
Zhima Jie is the Mandarin version of Sesame Street that aired in Shanghai. Big Bird is saying, “I’m Da Niao, Big Bird’s cousin. I am from Zhima Jie in China. My favorite color is yellow. What is your favorite color?” We all know that Chairman Mao’s favorite color was red.
