Policy statements on the effects of media overlook scientific complexity

"Although there certainly are some pretty good media policy statements out there, many of the policy statements were not very accurate and where there were inaccuracies, these tended to lean in the direction of conclusions that were generally scarier than could be defended by the actual data," says psychology researcher Christopher J. Ferguson of Stetson University, who coauthored the paper with fellow media researchers. "There's no assumption of bad faith, of course, but it seems many professional organizations are struggling to develop policy statements that effectively communicate the complicated, messy and nuanced nature of many media effects fields."

Ferguson and his coauthors are all researchers with expertise in some aspect of media effects, although they don't always draw the same conclusions about the impact of different forms of media. They consider ongoing discussion and debate to be an important part of the scientific process, but they noticed that many organizations' policy statements about media effects didn't acknowledge that any such debate was taking place.

"We were curious to know how often this was happening and, if this was happening a lot, point out directions that could lead to more accurate statements in the future," says Ferguson.

Using Google Scholar and targeted web searches, the team of researchers identified media effects policy statements produced by professional advocacy organizations that represent scholars or clinicians in relevant fields (e.g., American Academy of Pediatrics, American Psychological Association). These searches produced 24 public statements, with the earliest issued in the early 1990s. The statements covered impacts resulting from media violence, screen time, sexual content, and more "general" effects.

The research team broke into subgroups to evaluate each type of statement, using a standardized rubric focused on specific characteristics: citation bias, false consistency, lack of clarity of transparency, overgeneralization, exaggeration, insulation, and noncredible sources.

In general, the researchers found a noticeable increase in the frequency of media effects statements in the last 30 years. Most of the 24 policy statements came from organization-based committees and were produced by scholars who had interest and expertise in the field.

The research team found that the majority of statements, 19 out of 24, showed citation bias, citing evidence that supported a specific conclusion without mentioning existing evidence that did not support the conclusion. Similarly, 22 out of 24 statements were characterized by false consistency, implying that the evidence on media effects was more consistent than it was in actuality. And only one statement made any reference to the existence of diverse viewpoints among scholars in that area.

The team concluded that 15 out of the 24 statements overgeneralized results, applying media effects findings to contexts far beyond the scope of the original research. And 19 out of 24 statements made exaggerated claims about media effects, suggesting public health or other societal impacts without noting the small or trivial size of the effects found in many research studies.

Most of the statements did not provide detailed information about who produced the statement or how they were selected. They also did not indicate how the data that informed the statements were selected.

Ferguson and colleagues suggest that these findings have important implications for both policymakers and parents.

"Since these are 'policy statements,' presumably they are staking out policy positions the organizations would like to see policymakers move on. But policymakers may need to be cautious not to mistake these policy positions for a fair summary of current research," Ferguson says. "The other group of concern is parents, since many parents may become needlessly worried about media effects when policy statements proclaim the evidence to be stronger, more consistent, or more applicable to real life behaviors than it actually is."

Based on their findings, the research team devised a checklist for best practices that, if followed, would substantively improve the accuracy and quality of such policy statements:

-Acknowledge disconfirmatory data

-Focus on the magnitude of effects

-Acknowledge limitations of research methods

-Solicit balanced views

-Avoid secondary sources

-Distinguish scientific statements from advocacy statements

-Release fewer statements

-Be mindful of unintended harms

-Prioritize and encourage open science practices