New method of harvesting crowd wisdom

Their method, explained in a newly published paper, uses a technique the researchers call the "surprisingly popular" algorithm to better extract correct answers from large groups of people. As such, it could refine wisdom-of-crowds surveys, which are used in political and economic forecasting, as well as many other collective activities, from pricing artworks to grading scientific research proposals.

The new method is simple. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer.

"In situations where there is enough information in the crowd to determine the correct answer to a question, that answer will be the one [that] most outperforms expectations," says paper co-author Drazen Prelec, a professor at the MIT Sloan School of Management as well as the Department of Economics and the Department of Brain and Cognitive Sciences.

The paper is built on both theoretical and empirical work. The researchers first derived their result mathematically, then assessed how it works in practice, through surveys spanning a range of subjects, including U.S. state capitols, general knowledge, medical diagnoses by dermatologists, and art auction estimates.

Across all these areas, the researchers found that the "surprisingly popular" algorithm reduced errors by 21.3 percent compared to simple majority votes, and by 24.2 percent compared to basic confidence-weighted votes (where people express how confident they are in their answers). And it reduced errors by 22.2 percent compared to another kind of confidence-weighted votes, those taking the answers with the highest average confidence levels.

The paper, "A solution to the single-question crowd wisdom problem," is being published today in Nature. The authors are Prelec; John McCoy, a doctoral student in the MIT Department of Brain and Cognitive Sciences; and H. Sebastian Seung, a professor of neuroscience and computer science at Princeton University and a former MIT faculty member. Prelec and McCoy are also researchers in the MIT Neuroeconomics Laboratory, where Prelec is the principal investigator.

A capital idea

To see how the algorithm works in practice, consider a case the researchers tested. A group of people were asked a yes-or-no question: Is Philadelphia the capital of Pennsylvania? They were also asked to predict the prevalence of "yes" votes.

Philadelphia is not the capital of Pennsylvania; the correct answer is Harrisburg. But most people believe Philadelphia is the capital because it is a "large, historically significant city." Moreover, the people who mistakenly thought Philadelphia is the state capital largely thought other people would answer the same way. So they predicted that a very high percentage of people would answer "yes."

Meanwhile, a certain number of respondents knew that Harrisburg is the correct answer. However, a large portion of those people also anticipated that many other people would incorrectly think the capital is Philadelphia. So the people who themselves answered "no" still expected a very high percentage of "yes" answers.

That means the answer to the two questions -- Is Philadelphia the capital? Will other people think so? -- diverged. Almost everyone expected other people to answer "yes." But the actual percentage of people who answered "yes" was significantly lower. For this reason, the "no" answer was the "surprisingly popular" one, since it deviated from expectations of what the answer would be.

And since the "surprisingly popular" answer differed in the "no" direction, that tells us the correct answer: No, Philadelphia is not the capital.

The same principle applies no matter which direction responses deviate from expectations. When people were asked if Columbia is the capital of South Carolina, the opposite happened: More people answered "yes," compared to their expectations of how many people would say "yes." So the surprisingly popular answer was, correctly: Yes, Columbia is the capital.

The wisdom of subsets of crowds

In this sense, the "surprisingly popular" principle is not simply derived from the wisdom of crowds. Instead, it uses the knowledge of a well-informed subgroup of people within the larger crowd as a diagnostically powerful tool that points to the right answer.

"A lot of crowd wisdom weights people equally," McCoy explains. "But some people have more specialized knowledge." And those people -- if they have both correct information and a correct sense of public perception -- make a big difference.

This is the case across scenarios that the researchers tested. Consider art. The researchers asked art professionals to guess the price range for different contemporary artworks. Individually, art experts selected price ranges that were typically too low, perhaps because selecting a lower range is a reasonable, safe answer for an artwork that the expert does not recognize. Collectively, this makes the majority opinion of an expert panel even more biased in the direction of low prices.

And this is where the "surprisingly popular" principle makes a difference, since it does not depend on an absolute majority of expert opinion. Instead, suppose a relatively small number of experts believe a piece sold for $100,000, while anticipating that most other people will think it sold for less. In that case, the evaluations of those experts will lead the "surprisingly popular" answer to be that the artwork was more expensive than most people thought.

"The argument in this paper, in a very rough sense, is that people who expect to be in the minority deserve some extra attention," Prelec says.

Recovering truth

The scholars recognize that the "surprisingly popular" algorithm is not theoretically foolproof in practice. It is at least conceivable that people could anticipate a "surprisingly popular" opinion and try to subvert it, although that would be very hard to execute. It is also the case, as they write in the Nature paper, that "These claims are theoretical and do not guarantee success in practice, as actual respondents will fall short of ideal."

Still, the researchers themselves hope their work will be tested in a variety of settings. In the paper they express confidence that the "surprisingly popular" principle will prove durable, asserting: "Such knowledge can be exploited to recover truth even when traditional voting methods fail."