What we see shapes what we hear

"In face-to-face communication, language entails much more than just speech," explains senior investigator Hans Rutger Bosker. "Speakers make use of different channels (mouth, hands, and face) to get a message across. We want to understand how listeners make use of these different streams of information when they are listening to someone." In a well-known illusion called the 'McGurk effect', people hear a sound (like the 'b' in 'ba') as a different sound (for instance 'pa' or 'fa'), depending on the lip movements they see. But is there also a manual McGurk effect? Does what we hear depend on the gestures we see?

Plato or plateau?

To investigate this question, the researchers chose a set of Dutch words that differed only in stress pattern. For instance, the word "PLAto" -- with stress on the first syllable -- refers to the philosopher from ancient Greece. However, "plaTO," pronounced with stress on the second syllable, refers to a plateau. Participants watched a video of Bosker producing the words (with ambiguous stress) while making beat gestures ("Now I say the word ... plato"). Participants then had to decide which word they heard (PLAto or plaTEAU?). Would it matter whether beat gestures occurred at the first or second syllable?

Listeners were more likely to hear stress on a syllable if there was a beat gesture on that syllable. This 'manual McGurk effect' occurred for both words and non-words ("BAAGpif" or "baagPIF"?). Even more surprisingly, beat gestures influenced what vowel people heard (long or short 'a' in 'baagpif / bagpif'), as vowel length is typically associated with the stress pattern of a word.

"Listeners listen not only with their ears, but also with their eyes," says Bosker. "These findings are the first to show that beat gestures influence which speech sounds you hear." Bosker and Peeters think that the effect of beat gestures may be even bigger in real life, when speech is less clear than in the lab. In noisy listening conditions, visual beat gestures might be even more important for successful communication. "So wash your hands, and use them," Bosker adds jokingly.

"Our findings also have the potential to enrich human-computer interaction and improve multimodal speech recognition systems. It seems clear that such systems should take into account more than just speech," Bosker concludes. "We will follow up on the study by using virtual reality to test how specific these effects are -- are they induced by beat gestures only or also by other types of communicative cues, such as head nods and eyebrow movements?