Refuted Claims From Observational Studies Often Persist Despite Strong Evidence Against Them

"Some research findings that have received wide attention in the scientific community, as proven by the high citation counts of the respective articles, are eventually contradicted by subsequent evidence. A number of such high-profile contradictions pertain to differences between nonrandomized and randomized studies," the authors write. Despite these contradictions, findings from observational trials may still be supported in some scientific circles.

Athina Tatsioni, M.D., of the University of Ioannina School of Medicine, Ioannina, Greece, and colleagues evaluated the change in favorable vs. unfavorable citations for two highly cited epidemiological studies that proposed major cardiovascular benefits associated with vitamin E in 1993, which were strongly contradicted by evidence from large randomized clinical trials (RCTs). To assess the generalizability of these findings, the researchers also examined the extent to which two other major contradicted claims, the effectiveness of beta-carotene for preventing cancer and of estrogens for preventing Alzheimer dementia, continue to be supported in the current medical literature.

For the vitamin E claims, the researchers sampled articles published in 1997, 2001, and 2005 (before, early, and late after publication of refuting evidence) that referenced the highly cited epidemiological studies and separately sampled articles published in 2005 and referencing the major contradicting RCT (HOPE trial). They also sampled articles published in 2006 that referenced highly cited articles proposing benefits associated with beta-carotene for cancer (published in 1981 and contradicted by RCTs in 1994-1996) and estrogen for Alzheimer disease (published in 1996 and contradicted by RCTs in 2004).

The researchers found that for the two vitamin E epidemiological studies, even in 2005, 50 percent of citing articles remained favorable. A favorable stance was independently less likely in more recent articles, specifically in articles that also cited the HOPE trial and in general/internal medicine vs. specialty journals. In a sample of 29 articles published in 2005 that had cited the HOPE study, six (20.7 percent) were still favorable to vitamin E, 11 (37.9 percent) were equivocal (uncertain), and 12 (41.4 percent) were unfavorable. "Even among articles that cited the contradicting HOPE trial rather than the positive epidemiological studies, the majority in 2005 still could not conclude that vitamin E was ineffective," the authors write.

Regarding the claims for beta-carotene, 10 citing articles (62.5 percent) were favorable, three (18.8 percent) equivocal, and three (18.8 percent) unfavorable. For estrogen, 29 citing articles (61.7 percent) were favorable, 14 (29.8 percent) equivocal, and 4 (8.5 percent) unfavorable. All beta-carotene citations and all but two estrogen citations appeared in specialty journals.

"Counterarguments defending vitamin E or estrogen included diverse selection and information biases and genuine differences across studies in participants, interventions, co-interventions, and outcomes. Favorable citations to beta-carotene, long after evidence contradicted its effectiveness, did not consider the contradicting evidence."

"... it can be difficult to discern whether perpetuated beliefs are based on careful consideration of all evidence and differential interpretation, inappropriate entrenchment of old information, lack of dissemination of newer data, or purposeful silencing of their existence. Regardless of the reasons, better communication between research specialists and evidence-based clinical science may improve this situation and may lead to more rational and concerted translational efforts in basic, pre-clinical, and clinical research," the researchers conclude.

Journal reference: JAMA. 2007;298(21):2517-2526.