How we see others' emotions depends on our pre-conceived beliefs

"Perceiving other people's facial emotion expressions often feels as if we are directly reading them out from a face, but these visual perceptions may differ across people depending on the unique conceptual beliefs we bring to the table," explains Jonathan Freeman, the paper's senior author and an associate professor in NYU's Department of Psychology and Center for Neural Science. "Our findings suggest that people vary in the specific facial cues they utilize for perceiving facial emotion expressions."

The study, conducted with Jeffrey Brooks, an NYU doctoral student, involved a series of experiments in which subjects were asked about their conceptualizations of different emotions. This was used to estimate how closely related different emotions were in a subject's mind. For instance, some people might think anger and sadness are more similar emotions if they conceptually associate both these emotions with actions such as crying and slamming your fist on a table; other people might think they're entirely different emotions because they associate the two emotions as feeling completely different and resulting in different actions.

Specifically, subjects were assessed in how similarly they held different pairs of the following emotions in their mind: Anger, Disgust, Happiness, Fear, Sadness, and Surprise. These six emotions have long been argued by some scientists to be universal across cultures and genetically hard-wired in humans.

Freeman and Brooks then set out to test whether the different ways subjects conceptually held the six emotions in their mind may warp how subjects visually perceive these emotions on others' faces.

The subjects viewed a series of images of human facial expressions of the emotions and made judgments about the emotion these faces were expressing. To gauge subjects' perceptions, the researchers deployed an innovative mouse-tracking technology, developed by Freeman, that uses an individual's hand movements to reveal unconscious cognitive processes -- in this case, which emotion categories became activated in subjects' minds during their visual perception of a facial expression. Unlike surveys, in which subjects can consciously alter their responses, this technique requires them to make split-second decisions, thereby uncovering less conscious tendencies through their hand-motion trajectory.

Overall, the experiments showed that when individuals believed any two emotions were conceptually more similar, faces they saw from those categories of emotions were visually perceived with a corresponding similarity.

Specifically, when subjects held any two emotions, such as anger and disgust, as conceptually more similar, their hand attempted to simultaneously indicate that they saw both "anger" and "disgust" when viewing one of those facial expressions -- even though presumably each expression only depicted a single emotion at a time.

In a final experiment, a technique known as 'reverse correlation' was used to visualize the six different emotions in the mind's eye of a subject. The researchers started with a single neutral face and created hundreds of different versions of this face that were overlaid with different patterns of random noise. The noise patterns create random variations in the face's cues; for example, one version might look more like it is smiling rather than frowning. On each trial of the experiment, subjects were presented with two different versions of this face and decided which of the two appeared more like a specific emotion (e.g., anger) -- even though in reality it was only the noise pattern creating any difference in the two versions' appearance. On the basis of the noise patterns a subject chose, an average facial "prototype" for each of the six emotions could be visualized -- serving as a kind of window into the mind's eye of a subject.

Converging with the mouse-tracking results, when any two emotions were conceptually more similar in a subject's mind, the images of those two visualized facial prototypes physically resembled one another to a greater extent. For instance, if a subject viewed anger and disgust to be conceptually more similar, the visualized images of what an angry face and a disgusted face look like to that subject had a greater physical resemblance.

"The findings suggest that how we perceive facial expressions may not just reflect what's in the face itself, but also our own conceptual understanding of what the emotion means," explains Freeman, who notes interest in facial expressions has intrigued scientists dating back to Charles Darwin in the 19th century. "For any given pair of emotions, such as fear and anger, the more a subject believes these emotions are more similar, the more these two emotions visually resemble one another on a person's face. The results suggest that we may all slightly differ in the facial cues we use to understand others' emotions, because they depend on how we conceptually understand these emotions."

The authors note that the study's results contrast with classic scientific theories of emotion that assume each emotion has its own specific facial expression that humans universally recognize. Based on this view, the same exact facial expression, such as a scowling face for anger, should always elicit a perception of anger, and our personally-held beliefs about what constitutes "anger" should not affect the process.

The findings, Freeman observes, may have implications for artificial intelligence and machine learning. Automated algorithms for facial emotion recognition, and other computer-vision and security applications, are geared to spot emotions, which could potentially be enhanced by incorporating conceptual processes.

The research was supported, in part, by a grant from the National Institutes of Health (R01-MH112640).