Scientific insurgents say 'Journal Impact Factors' distort science

The San Francisco Declaration on Research Assessment, or DORA, was framed by a group of journal editors, publishers, and others convened by the American Society for Cell Biology (ASCB) last December in San Francisco, during the Society's Annual Meeting. The San Francisco group agreed that the JIF, which ranks scholarly journals by the average number of citations their articles attract in a set period, has become an obsession in world science. Impact factors warp the way that research is conducted, reported, and funded. Over five months of discussion, the San Francisco declaration group moved from an "insurrection," in the words of one publisher, against the use of the prominent two-year JIF to a wider reconsideration of scientific assessment. The DORA statement posted today makes 18 recommendations for change in the scientific culture at all levels -- individual scientists, publishers, institutions, funding agencies, and the bibliometric services themselves -- to reduce the dominant role of the JIF in evaluating research and researchers and instead to focus on the content of primary research papers, regardless of publication venue.

Today's declaration is timed to coincide with editorials in scientific journals around the world including an endorsement of DORA by Bruce Alberts, Editor-in-Chief of Science Magazine in the journal's May 17th issue. Other editors signing DORA represent Journal of Cell Biology (JCB), Traffic, Genetics, eLife, Journal of Cell Science, Aging Cell, Molecular Biology of the Cell (MBoC), BioArchitecture, The EMBO Journal, Journal of Cell Science, Journal of Surfactants & Detergents, Cell Structure & Functions (Japan), Lipids, Genes, Journal of the Electrochemical Society, and Development. Along with ASCB, organizations signing DORA include the American Association for the Advancement of Science, the British Society for Cell Biology, the Wellcome Trust, the Howard Hughes Medical Institute, EMBO, the European Molecular Biology Laboratory, the Association for Psychological Science, the Electromechanical Society, the European Sociology Association, the European Mathematical Society, the Association of Australian Medical Research Institutes, the Company of Biologists, the American Physiological Society, the Linguistic Society of America, the Genetics Society of America, the Burnet Institute (Australia), the National Center for Genomic Analysis (CNAG, Barcelona, Spain), and the Bionics Institute (Australia). A complete list of signatories to date is at http://www.ascb.org/SFdeclaration.html.

Mark Patterson, Executive Director of eLife, points to the diverse list of DORA signers as evidence that concern about impact factors is not an Us vs. Them issue in scientific publishing. "You've got the very old like Science and the very new like PeerJ or eLife. So you've got the old and the new, the non-profit and the for-profit, the open access and the subscription model. It's really a mix," says Patterson. "The issue cuts right across and not just publishers of all shapes but all the different constituencies involved in research assessment, all the way from individual researchers to the institutions that they work in or are funded by to the journals that they publish in."

There are number of citation ranking systems today, but the oldest and most influential is the so-called "two-year JIF" devised by Eugene Garfield in the early 1950s and originally published by his Institute for Scientific Information (ISI) as a subscription buying tool for academic and medical librarians. The JIF, which appears once a year in Journal Citation Reports as part of the Thomson Reuters (ISI) Web of Knowledge, is the average number of citations received in a year per paper published in the journal during the two preceding years. The earliest that a new journal can have a JIF is the end of its third full year of publication.

Even though the JIF is only a measure of a journal's average citation frequency, it has become a powerful proxy for scientific value and is being widely misused to assess individual scientists and research institutions, say the DORA framers. The JIF has become even more powerful in China, India, and other nations emerging as global research powers. "It's maddening," says David Drubin, Editor-in-Chief of ASCB's journal MBoC. "This is a metric that really drives a lot of traffic. You really see it most clearly when you travel to foreign countries and I especially see it with my foreign postdocs. They only want to publish in journals with high impact factors."

The San Francisco declaration cites studies that outline known defects in the JIF, distortions that skew results within journals, that gloss over differences between fields, and that lump primary research articles in with much more easily cited review articles. Further, the JIF can be "gamed" by editors and authors, while the data used to compute the JIF "are neither transparent nor openly available to the public," according to DORA.

Since the JIF is based on the mean of the citations to papers in a given journal, rather than the median, a handful of highly cited papers can drive the overall JIF, says Bernd Pulverer, Chief Editor of the EMBO Journal. "My favorite example is the first paper on the sequencing of the human genome. This paper, which has been cited just under 10,000 times to date, single handedly increased Nature's JIF for a couple of years."

"The Journal Impact Factor (JIF) was developed to help librarians make subscription decisions, but it's become a proxy for the quality of research," says Stefano Bertuzzi, ASCB Executive Director, one of more than 70 institutional leaders to sign the declaration on behalf of their organizations. "Researchers are now judged by where they publish not by what they publish. This is no longer a question of selling subscriptions. The 'high-impact' obsession is warping our scientific judgment, damaging careers, and wasting time and valuable work."

The SF declaration urges all stakeholders to focus on the content of papers, rather than the JIF of the journal in which it was published, says Bertuzzi, "The connection is flawed and the importance of the finding as reflected by the light of a high JIF number is often completely misleading, because it is always only a very small number of papers published in a journal that receive most of citations, so it is flawed to measure the impact of a single article by this metric. Great papers appear in journals with low JIFs and vice versa."

One of the four editors of Traffic who signed DORA, Michael Marks acknowledges that the group realized that the scientific world has been using impact factors inappropriately. "Initially our gut reaction was to blame the JIF itself but it's not the JIF's fault," says Marks. "It's our use of the JIF that's the problem."

DORA's 18 recommendations call for sweeping changes in scientific assessment, says Drubin. They will hopefully lead to "a change in the culture where people will choose the journals that they publish in not on the prestige but on the fit. Is the format correct? Is the audience correct? Does the editorial board have the appropriate expertise?" A difficult change, Drubin concedes, but vital to the integrity of scientific self-assessment, which is the engine by which modern science advances. "For me, it was just a matter of when enough is enough," says Drubin of his role in the great JIF insurrection.