Crowdsourcing with mobile apps brings 'big data' to psychological research

"Mobile devices offer researchers an exciting new means to crowdsource an experiment using games that are actually tests of cognition or other brain functions," said Stephen R. Mitroff, PhD, of Duke University, lead author of the report, published in APA's Journal of Experimental Psychology: Human Perception and Performance. "Questions that could have taken decades to answer in a laboratory setting, or that could not be realistically answered in a lab, can be examined using big data gathered in a relatively short time."

To demonstrate the potential of mobile technology to gather data, the researchers partnered with Kedlin Co., the developer of the popular mobile app game Airport Scanner, which challenges players to identify illegal items in luggage passing through an airport x-ray scanner. Players view one bag at a time and tap their touchscreen to identify banned items. The game includes a logbook of illegal items (e.g., guns, hand grenades, switchblades) and legal items (e.g., ear phones, clothing), and the list expands from a handful of possible target items to hundreds as players progress through the game.

Between January 2013 and November 2014, the game provided anonymous data from more than 2 billion trials from over 7 million mobile devices that Mitroff and his research team at the Duke Center for Cognitive Neuroscience have been analyzing, the report said. Players consent to the data collection when installing the game on their smartphone or other hand-held device.

One research question explored through the Airport Scanner game was examining players' ability to spot items that occurred on screen very rarely, or in less than 0.1 percent of all bags.

"When a target appears only 0.1 percent of the time, you need 1,000 trials to get just a single case of that target. Too many trials would be needed to realistically assess detection of that particular target in a laboratory setting," Mitroff said. "With the large dataset, we were able to look at hundreds of cases for each of the nearly 30 rarest targets."

Another experiment found that when two banned items both appeared in a bag, players were more likely to miss the second to-be-found item when the two were different than if they were identical. For example, players were more likely to miss a dynamite stick after finding a gasoline can than they were to miss a dynamite stick after first finding a dynamite stick.

"While using a game interface to assess cognitive abilities is not new to psychological research, mobile technology offers a phenomenal opportunity to examine cognitive processes on a large scale," Mitroff said. "In 1998, 1 million trials were analyzed by Jeremy M. Wolfe, PhD, of Harvard Medical School, to understand visual search. This was a mind-blowing amount of data that took 10 years to collect. Today, we can collect over a million trials a day through Airport Scanner."

Other advantages of crowdsourcing include that it can be a less-expensive means of collecting data automatically and continuously and that it has the ability to mimic real-world situations that are difficult to address in a laboratory, according to the report.

Crowdsourcing research data does have disadvantages, the authors noted. Researchers must have the skills to create a fun game app or they need to partner with a developer. Collecting large amounts of data may not necessarily result in high-quality data, which requires carefully developing research questions to be addressed through the game. And collecting data through crowdsourcing means researchers have no control over who is playing, they said.