Flaw in common approach of public opinion surveys about science

"We set out to determine whether we can accurately assess public opinion on complex science issues with one question, or if we need to break the issue down into questions on each of the issue's constituent parts," says Dr. Andrew Binder, an assistant professor of communication at NC State and lead author of the study. "We found that, to varying degrees, accuracy really depends on breaking it down into multiple questions for people."

To assess the problematic nature of a single-question surveys, the researchers developed two surveys; one focused on nanotechnology and the other on biofuels. In each survey, respondents were asked an overarching question: do the risks associated with nanotechnology/biofuels outweigh the benefits; do the benefits outweigh the risks; or are the risks and benefits approximately the same? The researchers then asked survey participants a series of questions about specific risks and benefits associated with nanotechnology or biofuels.

The researchers then compared a participant's response to the overarching question with his or her responses to the specific questions in order to see whether the overarching question accurately captured the opinion of the individual respondent.

They found a problem.

"There was a significant discrepancy among people who responded to the overarching question that the risks of emerging science outweighed the benefits when compared to their responses to the questions about the specific risks and benefits," says Binder. "Namely, those same people really perceived more benefits than risks when given the opportunity to evaluate these attributes separately.

"For example, in the nanotechnology survey, 50 percent of respondents who said risks outweighed benefits actually evaluated nanotechnology positively in the other portion of the survey," Binder says. "In fact, only 35.4 percent of respondents who thought risks outweighed benefits actually calculated more risks than benefits in the specific section of the survey." The researchers found similar, though less pronounced, results in the biofuels survey.

The study also showed that people who said that benefits outweighed risks in response to the overarching question consistently perceived more benefits than risks in the specific question section of the surveys.

"The bottom line is that social scientists and journalists need to be very careful when relying on data from a single, overarching survey question," Binder says. "These oversimplified questions can result in misleading poll data and create problems for policymakers who base their decisions on those findings. They can also be problematic because they may contribute to different polls showing widely different results, which weakens the public's faith in surveys generally."

NC State's Department of Communication is part of the university's College of Humanities and Social Sciences.