What are you looking at? 'Virtual' communication in the age of social distancing

In a first-of-its-kind study, neuroscientists from Florida Atlantic University demonstrate that a person's gaze is altered during tele-communication if they think that the person on the other end of the conversation can see them. People are very sensitive to the gaze direction of others and even 2-day-old infants prefer faces where the eyes are looking directly back at them. The phenomenon known as "gaze cueing," a powerful signal for orienting attention, is a mechanism that likely plays a role in the developmentally and socially important wonder of "shared" or "joint" attention where a number of people attend to the same object or location. The ability to do this is what makes humans unique among primates.

Throughout almost all of human history, conversations were generally conducted face-to-face, so people knew where their conversational partner was looking and vice versa. Now, with virtual communication, that assumption no longer holds -- sometimes people communicate with both cameras on while other times only the speaker may be visible. The researchers set out to determine whether being observed affects people's behavior during online communication.

For the study, published in the journal Attention, Perception & Psychophysics, co-authors Elan Barenholtz, Ph.D., an associate professor of psychology, a member of the Center for Complex Systems and Brain Sciences in FAU's Charles E. Schmidt College of Science and a member of FAU's Brain Institute (I-BRAIN), and Michael H. Kleiman, Ph.D., a postdoctoral researcher at FAU, compared fixation behavior in 173 participants under two conditions: one in which the participants believed they were engaging in a real-time interaction and one in which they knew they were watching a pre-recorded video.

The researchers wanted to know if face fixation would increase in the real-time condition based on the social expectation of facing one's speaker in order to get attention or if it would lead to greater face avoidance, based on social norms as well as the cognitive demands of encoding the conversation.

Similarly, they wanted to know where participants would fixate on the face. Would it be the eyes more in the real-time condition because of social demands to make eye contact with one's speaker? Or, in the pre-recorded condition, where the social demands to make eye contact are eliminated, would participants spend more time looking at the mouth in order to encode the conversation, which is consistent with previous studies showing greater mouth fixations during an encoding task.

Results of the study showed that participants fixated on the whole face in the real-time condition and significantly less in the pre-recorded condition. In the pre-recorded condition, time spent fixating on the mouth was significantly greater compared to the real-time condition.

There were no significant differences in time spent fixating on the eyes between the real-time and the pre-recorded conditions. These findings may suggest that participants are more comfortable looking directly at the mouth of a speaker -- which has previously been found to be optimal for encoding speech -- when they think that no one is watching them.

To simulate a live interaction, the researchers convinced participants that they were engaging in a real-time, two-way video interaction (it was actually pre-recorded) where they could been seen and heard by the speaker, as well as a pre-recorded interaction where they knew the video was previously recorded and therefore the speaker could not see their behavior.

"Because gaze direction conveys so much socially relevant information, one's own gaze behavior is likely to be affected by whether one's eyes are visible to a speaker," said Barenholtz. "For example, people may intend to signal that they are paying more attention to a speaker by fixating their face or eyes during a conversation. Conversely, extended eye contact also can be perceived as aggressive and therefore noticing one's eyes could lead to reduced direct fixation of another's face or eyes. Indeed, people engage in avoidant eye movements by periodically breaking and reforming eye contact during conversations."

There was a highly significant tendency for participants engaging in perceived real-time interaction to display greater avoidant fixation behavior, which supports the idea that social contexts draw fixations away from the face compared to when social context is not a factor. When the face was fixated, attention was directed toward the mouth for the greater percentage of time in the pre-recorded condition versus the real-time condition. The lack of difference in time spent fixating the eyes suggests that the additional mouth fixations in the pre-recorded condition did not come at the cost of reduced eye fixation and must have derived from reduced fixations elsewhere on the face.

Comparisons between total fixation durations of the eyes versus the mouth were calculated for both the real-time and pre-recorded conditions, with the eyes of both conditions being significantly more fixated than the mouth. Gender, age, cultural background, and native language did not have an influence on fixation behavior across conditions.

"Regardless of the specific mechanisms underlying the observed differences in fixation patterns, results from our study suggest participants were taking social and attentional considerations into account in the real-time condition," said Barenholtz. "Given that encoding and memory have been found to be optimized by fixating the mouth, which was reduced overall in the real-time condition, this suggests that people do not fully optimize for speech encoding in a live interaction."