Does throwing my voice make you want to shop here?

Humans locate sound by combining visual and auditory cues. Because the visual cues are generally less variable, they can override audio cues, leading to the well-known ventriloquism effect, which occurs when a human perceives the location of a sound to be different from its actual source. It is also well known that humans have personal space, which varies according to social, personal, and environmental factors. Although both phenomena have long been studied individually, until the development of virtual reality, it has not been possible to study how the ventriloquism effect alters personal space.

"In particular, we wanted to know how it affects the rapport between the user and shop assistant," says Professor Zempo Keiichi, lead author of the study. Rapport, or the quality of interpersonal service, strongly affects loyalty and satisfaction, and skilled salespeople use several techniques to build rapport with customers.

In their experiments, the researchers asked 16 people in the virtual shop environment to define their personal space and record their impression when approached by shop assistants. Some assistants had both a voice and an image at the same position, and others had a voice that was located at different distances between the user and assistant.

"We found that rapport was not affected when the deviation between the sound and visual positions could not be tolerated; however, when it could be tolerated, we found two distinct phenomena," explains Professor Zempo Keiichi. The first was similar to the "uncanny valley," which occurs when an imperfect human representation invokes feelings of uneasiness in a real human. This decreased rapport with the virtual assistant. But when the sound moved even closer to the human, the rapport increased.

The authors call this phenomenon the "mouth-in-the-door" phenomenon because it is similar to the "foot-in-the-door" phenomenon, in which a small, unconscious consent, such as not moving away when someone starts to speak, causes a person to improve their evaluation of the other person. Without these virtual experiments, this phenomenon would have likely remained undiscovered. But now that it is known, the authors believe it can be used to improve the user experience, especially in virtual shop scenarios.