New technique sheds light on body language in job interviews, sales calls, team projects

The system, named Protractor by the Cambridge-Dartmouth team, uses invisible light to record how people employ body language by measuring body angles and distances between individuals.

Prior studies have revealed that body language can influence many aspects of everyday life including job interviews, doctor-patient conversations and team projects. Each setting includes a specific set of interaction details such as eye contact and hand gestures for which an accurate monitoring of distance and relative orientation is crucial.

"The ability to use invisible light to determine someone's role and attitude in social settings has powerful implications for individuals and organizations that are concerned about how they communicate," said Cecilia Mascolo, professor of mobile systems at the University of Cambridge.

Body language is already commonly studied through video sessions, audio recordings and paper questionnaires. Compared to the new, light-based system, these approaches can require invasive cameras, necessitate complex infrastructure support, and impose high burdens on users.

"Our system is a key departure from existing approaches," said Xia Zhou, assistant professor of computer science at Dartmouth. "The ability to sense both body distance and relative angle with fine accuracy using only infrared light offers huge advantages and can deepen the understanding on how body language plays a role in social interactions."

Protractor is a lightweight, wearable tag resembling an access badge worn with a lanyard or clip. The device measures non-verbal behavior with fine granularity by using near-infrared light from photodiodes. The light technology operates at a wavelength commonly used in television remote controls.

Before settling on infrared light for the unit, the research team also considered ultrasound and radio frequency. In addition to the overall accuracy, infrared was favorable because light cannot penetrate human bodies, ensuring the accurate sensing of face-to-face interaction. Near infrared light is also imperceptible to human eyes and keeps the sensing unobtrusive.

Although well-suited for measuring body language, the research team needed to correct for when a user's hand or clothing could temporary block the light channel. They did so by designing algorithms that exploit inertial sensors to work around the absence of light tracking results.

In proving the system, researchers also had to devise a way for the sensors to accurately identify participants and to limit power consumption.

"By modulating the light from each Protractor tag to encode the tag ID, each tag can then figure out which individuals are participating. To increase energy efficiency, we also adapt the frequency of emitting light signals based on the specific context," said Zhao Tian, a PhD candidate at Dartmouth that worked on the research team.

To study the technique's effectiveness, the team used the Protractor tags to track non-verbal behaviors during a problem-solving group task known as "The Marshmallow Challenge." In this task, teams of four members were given 18 minutes to build a structure that could support a marshmallow using tape, string and a handful of spaghetti.

"Beyond simply observing body language with the tags, we identified the task role each group member was performing and delineated each stage in the building process through the recorded body angle and distance measurements," said Alessandro Montanari, a researcher at the University of Cambridge.

In the study of 64 participants, Protractor achieved 1- to 2-inch mean error in estimating interaction distance and less than 6 degrees error 95 percent of the time for measuring relative body orientation. The system also allowed researchers to assess an individual's task role within the challenge with close to 85 percent accuracy while identifying stages in the building process with over 93 percent accuracy.

According to the research team, the system will not only support social research, but it can also potentially provide real-time feedback during interviews and other interactions. Trainers, supervisors and team facilitators can use these findings to better understand team dynamics and intervene during intense problem-focused discussions to achieve higher creativity.

Protractor can also help study the impact of culture on body language in light of research that shows that cultural backgrounds can impact the way people think, feel, and act while working with others -- an important feature in today's highly-internationalized workplaces.