20 million tweets used to understand people and real-world situations

Researchers from Florida Atlantic University used more than 20 million Tweets to study the psychological characteristics of real-world situations that people actually experienced over the course of two weeks. David Serfass, corresponding author and a Ph.D. psychology student at FAU, and Ryne Sherman, Ph.D., study co-author and professor of psychology in FAU's Charles E. Schmidt College of Science, wanted to learn about the kinds of situations people experience across time, and how gender and population density might affect situation experiences. Findings from this study showed large gender differences and significant differences between weekdays and weekends. However, they also showed that people in urban and rural areas experience situations that are, for the most part, psychologically similar.

Twitter has approximately 271 million users who are responsible for more than 500 million Tweets every day. People frequently Tweet about their locations, what they are doing, how they are feeling, or things they find interesting in the present moment. In other words, people tend to Tweet about the situations they experience.

"Twitter is a digital stream of consciousness of its users and we wondered if we could determine the psychological characteristics of situations people were experiencing based on their Tweets," said Serfass. "There are few compilations of data on human thought, behavior, and emotions this vast, making Twitter an excellent medium for understanding human experience."

This new FAU research addresses two questions: (1) Is it possible to automatically and accurately extract situation characteristics from Tweets? (2) What can we learn about the situations people experience from their Tweets? In this first-of-its-kind study, Serfass and Sherman were able to develop a method for automatically extracting meaningful information about the situations people experience in their daily lives from Tweets.

In the study, they gathered 5,000 Tweets and had research assistants in their lab rate each of these Tweets on eight core dimensions of situations, dimensions which they helped uncover in previous research. Next, they used a computer program called the Linguistic Inquiry Word Count (LIWC) to quantify the words used in Tweets into distinct psychological and lexical groupings, such as self-references, positive words, negative words, and personal pronouns.

Serfass and Sherman then used machine learning techniques to determine which word categories tended to co-occur with which psychological characteristics. For example, they found that people who were in situations characterized by "duty" were more likely to use words like "work" and "job." People who were in situations characterized by adversity were more likely to use swear words.

The scoring methods used in this study represent the "tip of the iceberg" in terms of what can be learned about the situations people create, encounter, and imagine using the automated scoring methods developed by Serfass and Sherman. Because these researchers now know the links between what words people use to describe their situations and the psychological characteristics of those situations, they can use those links to predict what other situations are like.

"That is just what we did. We applied our scoring algorithms to more than 20 million Tweets gathered from Twitter," said Sherman. "Thus, we were able to map out the kinds of situations that people experience across time and day, and in urban versus rural areas of the U.S."

Serfass notes that some of the findings that were both interesting and intuitive are that people experienced on average more positivity on the weekend and more negativity during the work week. People also experienced higher levels of duty during the "9 to 5" workday and more sociality in the evenings. In terms of gender differences, females experienced higher levels of mating and more emotional situations -- both positive and negative -- than males.

"This research has implications for how we can use social media to understand human experience," said Sherman. "Think about what we can learn from situations surrounding holidays, festivals, sporting events, political upheavals, and even natural disasters, which could be examined using these methods. In that sense, we are really just getting started."