Ventriloquism In Motion: How Sound Can Move Light

The study, conducted in conjunction with Prof. Jon Driver at University College London, revealed that the perceived direction of motion from a given visual object (in this case, red bars across a screen), depends on minute variations in the timing of an accompanying sound (a sequence of beeps, for example). This provides evidence that the brain’s integration of these visual and audio cues occurs at a very early stage of processing.

Every day examples of audio-visual integration include our ability to identify who is saying what in a noisy crowd and the illusion that sound comes directly from the an actor’s lips seen on a television, rather than from the loudspeakers; the latter is the well-known ‘Ventriloquist Effect’, where seeing influences the location of sounds.

The audiovisual illusion revealed by this new research could be dubbed ‘reverse ventriloquism in motion’, as it shows that sound affects what we see. This might explain why if we watch dancing without sound, the dancers appear to have no rhythm; and why the sound of a ball hitting a racket can help us to determine the direction of the ball in a game of tennis even though the ball moves faster that the camera or eye can track.

Dr. Freeman believes that his research could have profound implications for the understanding of the neural processes that underlie multisensory perception. This knowledge could be applied in a number of industries: “The illusion could be applied to novel displays that change their appearance depending on sound, which may be of use in advertising or providing an eye-catching multisensory warning or alert in safety-critical applications. It may also eventually be useful in detecting and diagnosing subtle perceptual differences thought to be characteristic of certain clinical conditions such as dyslexia and autistic spectrum.”