Conflict of interest disclosures don't alter the recommendations of peer reviewers

The study was conducted by researchers at Carnegie Mellon University, Harvard and the University of California, San Francisco (UCSF). It examined the evaluations of more than 3,300 reviewers of nearly 1,500 papers submitted between 2014 and 2018 to the Annals of Emergency Medicine (AEM). The authors believe it is the first randomly controlled experiment to examine the impact of COI disclosures on actual reviewer evaluations of research papers.

In the study, reviewers received a manuscript and were either informed within 24 hours if the authors had or had not revealed a COI during the submission process or were not informed of the authors' COIs one way or the other. Nearly one third of the papers had disclosures, primarily regarding funding sources. After returning their evaluation of the research, all reviewers were surveyed.

Although 78 percent of the reviewers in the first group indicated they had read and understood the COI disclosures, and most also agreed that COIs are an important issue, the disclosures had no effect on reviewers' recommendations regarding papers' suitability for publication.

The study was led by Harvard's Leslie John, who earned her Ph.D. in Behavioral Decision Research at Carnegie Mellon in 2011. She said the findings likely are related to the ambiguity of the review process as it relates to COI disclosures.

"Reviewers aren't given any explicit guidance on how to correct for possible COI," said John, the Marvin Bower Associate Professor at Harvard Business School. "If a disclosure reveals a significant COI, should the reviewer recommend rejection or apply special scrutiny to methods? There currently isn't a consensus about how reviewers should respond to COI disclosures, nor any direction provided about how to do so."

John added that most of the reviewers reported that they strongly believed they could correct for the biasing influence of conflicts of interest if and when they were disclosed.

"Even experienced peer reviewers usually cannot articulate a specific and effective approach to follow once COI is suspected or declared," said Michael Callaham, AEM's editor-in-chief, emeritus chair of emergency medicine at UCSF and a senior author of the paper. "As a result, most reviewers do not change their assessments due to this information. Clearly the guidelines for reporting COI are not achieving their goal and need to be further studied and improved."

George Loewenstein, Carnegie Mellon's Herbert A. Simon University Professor of Economics and Psychology and a co-author of the study, noted that the paper is the latest in a long line of research showing that disclosure is an inadequate response to COIs that are rampant in medicine.

"The medical and academic establishments think they've dealt with the issues of COIs by including disclosures," Loewenstein commented. "This experiment provides further evidence that they haven't."