In surveys, people say they'll pay twice what they're actually willing to spend

But when researchers compared what study participants reported they were willing to spend on goods with what they actually shelled out in experiments designed to mimic a real-world shopping experience, there was a big gap.

"People said they'd spend about twice as much, on average, as they actually spent when faced with more realistic circumstances," said co-lead researcher Wuyang Hu of The Ohio State University.

Asking people to estimate how much they'd spend on a good or service is a key part of economics research and the results can drive decision making by business, government leaders and others, said Hu, a professor of agricultural, environmental, and development economics at Ohio State.

"Unfortunately, in many circumstances, it's extremely difficult to observe people's actual behavior, so we have to use these hypothetical, made-up scenarios," Hu said. "The problem is, we know that what people say they will spend and what they actually spend don't always line up."

The difference between these two amounts is called "hypothetical bias" and it stands in the way of research that can shape important decisions, like evaluating the worth of cleaning up after an environmental catastrophe, Hu said. In fact, the whole notion of this phenomenon came about after the 1989 Exxon-Valdez oil spill, when lawyers grappled with arriving at the cost of work to soak up the oil and protect animals, plants and natural resources in Alaska.

Hu and his co-author, Jerrod Penn of Louisiana State University, wanted to better understand the discrepancy between what people report they'll pay and what they'll actually pay. To do that, they reviewed and summarized results found in previous studies, either published or not. They looked at more than 500 articles and chose 132 studies that included hypothetical questions and an attempt to create a real-life spending experience -- such as a classroom set up to look like a store in which study participants had actual money to spend. The study appears in the American Journal of Agricultural Economics.

Overall, the difference between hypothetical spending and real spending was almost twofold. Hu stressed that even this could be an underestimate, given that the "real" spending observations are done under circumstances that attempt to mimic real life.

"Sometimes people are aware they're being observed. Sometimes they have no idea. But it's not really a grocery store or a conservation group's office where you're making an actual donation. It's an experiment, so we take those so-called 'real' spending observations with a grain of salt as well," Hu said.

Somewhat surprisingly, the researchers found that student study participants were not less reliable than others -- a trend that has emerged in other research, and led to concerns about the validity of economics research that relies largely on college students' feedback about spending.

"Some people have reached the conclusion that because students don't have much money to spend in real life, they tend to overstate what they'd spend, but we didn't find that," Hu said.

The researchers also found evidence that three common techniques designed to mitigate bias were effective: "cheap talk," certainty follow-up and consequentiality.

Researchers who use the cheap talk approach explicitly address the issue to study subjects, acknowledging that people often claim on surveys that they'll spend more than they would actually spend in real life. Researchers then tell them not to do that. Cheap talk techniques reduced differences by 41 percent, the researchers found.

Certainty follow-up involves asking the study participant to rate his or her level of certainty about a given survey response and this method reduced differences by 99 percent.

Consequentiality is when researchers emphasize the importance and value of the survey participants' honest responses. This approach reduced differences between hypothetical and real spending by 95 percent.

Penn, an assistant professor of resource economics at Louisiana State, said the evidence in favor of cheap talk and other efforts to mitigate hypothetical bias is important -- especially because some previous studies have cast doubt on the usefulness of such techniques.

"This means that exploring and testing new mitigation methods and how methods can complement each other continues to be a valuable avenue of future research," Penn said.

Hu said he hopes this robust meta-analysis of prior research will lead to improved techniques for factoring in this hypothetical bias or designing studies that do a better job of reducing it.

"Researchers will benefit from having more information about adjusting their results upward or downward to come closer to the truth about how much people will spend," Hu said.

He said he'd like to develop an app that researchers could use to estimate the impact of hypothetical bias based on the nature of their study, the participants and whether the good or service being studied is public or private.

"Hypothetical bias is really the tip of the iceberg in the entire human behavior puzzle that everyone's trying to crack in economics. Hopefully this brings us a little bit closer to the truth," Hu said.