Screeners miss the really rare stuff: Commonly found objects may be crowding out identification of the unusual items

Duke University researchers analyzed data from searches of 20 million virtual suitcases in the game Airport Scanner created by Kedlin Co. and found that users failed in most cases to identify objects that occurred only rarely.

"We're seeing that people are really bad at finding items that are not likely to appear," said Stephen Mitroff, Ph.D., associate professor of psychology and neuroscience and member of the Duke Institute for Brain Sciences.

In the game, players scan images that look like X-rays of carry-on luggage, trying to find hundreds of possible items, including guns and dynamite sticks, as well as over-sized bottles and scissors.

For this study -- which appears online in Psychological Science -- researchers evaluated gameplay data from December 2012 to March 2013 to determine how often the players found 78 different illegal items in light of how often the items appeared. Investigators used target frequencies (the appearance rate of a specific illegal object) to understand a player's success at identifying targets when they appear.

Thirty items were "ultra-rare," appearing in the game less than 0.15 percent of the time -- a rate that is comparable to the presence of cancerous markers in real-life radiological screenings, according to Mitroff. Yet the study results revealed players correctly identified those 30 targets only 27 percent of the time. Targets that appeared with more than 1 percent frequency were pinpointed 92 percent of the time.

"This isn't a matter of overall vigilance or how frequently players responded, since half of the searches had a to-be-found item present," Mitroff said. "This effect is about being able to detect specific items and how likely you are to miss them when they occur infrequently."

For example, airport screeners are more likely to encounter pocket knives or incorrectly-packed prescription medications than they are to see a gun in a carry-on bag. So, they might become conditioned to catch the illegal, though relatively harmless, items that appear more frequently than a rarely-seen weapon.

"With the very large array of potential targets, the searchers seem to be highly sensitive to how frequently each individual target appeared, and they adjusted the focus of their searches so that targets that appeared rarely were a low priority," Mitroff said. "This situation may be what occurs in many real-world searches."

This research is important, Mitroff said, because it explores how rare objects slip past us no matter how hard we're looking for them or how critical finding them might be, something he and co-author Adam Biggs, a post-doctoral researcher from the psychology and neuroscience department in the Center for Cognitive Neuroscience, have termed the "ultra-rare item effect."

The strong relationship between the frequencies and associated detection rates surprised Mitroff's team even though they anticipated a connection. Based on these results, Mitroff said, the door is open to further research into how behaviors can be modified to catch more ultra-rare items.

The potential is also there, Mitroff said, for the Airport Scanner game to play a role in Transportation Security Administration training and standard operating procedures. The research team now has access to more than 1.5 billion trials from the smartphone app for analysis.

"We want to better understand why this happens." he said. "Are you attuned to more frequent items at the cost of others? If you don't see something very often, it makes sense that you don't want to waste energy looking for it, but at the end of the day you still want to detect some of these ultra-rare items."

The anonymous game data were shared in accordance with the Standard Apple User agreement, and Duke University approved the analysis of the data. This research received no external funding support.