Analysis suggests China has passed U.S. on one research measure

Findings showed Chinese research ranked as high as or higher than U.S. work in the top 1% of scientific studies in 2019. This work is considered the most notable published science.

The new analysis, published recently in the journal Scientometrics, was done by three researchers: one from the United States, one from Europe and one from China. The measurement method used by the trio is different from what has traditionally been used -- and they say it is more appropriate to the task.

"We feel strongly that we have a better way to measure the impact of research when comparing nation-to-nation output," said study co-author Caroline Wagner, an expert on science policy and R&D investment and faculty member in the John Glenn College of Public Affairs at The Ohio State University.

"Using our measurement, China now has a slight lead over the U.S. in terms of scientific impact."

Wagner conducted the study with Lin Zhang of Wuhan University in China and Loet Leydesdorff of the University of Amsterdam in The Netherlands.

While there is no objective way of measuring the quality of research studies, scientists have traditionally used a proxy called citations. The more a study is mentioned (or "cited," as it is called) by other researchers in subsequent papers, the more impact it is seen as having on the field and the more it is assumed to be of high quality.

The gold standard is reaching the top 1% -- those scientific papers that are cited more often than 99% of others.

"These are the works that are seen as being in the class of Nobel Prize winners, the very leading edge of science," Wagner said. "The U.S. has tended to rank China's work as lower quality. This appears to have changed."

The issue is that papers in some scientific fields are generally cited much more frequently than papers in other fields. For example, the top papers in virology are cited more often than the top papers in sociology.

So what researchers have traditionally done is "field normalization," in which citation data are averaged in such a way as to allow the two fields to be compared side-by-side, taking into account statistically the differences in how citations are used in each of the fields.

When this method of measurement is used, the United States remains in the lead of producing top 1% papers.

But Wagner and her colleagues say weighting papers differently by scientific field makes no sense when you're comparing the research output of nations.

"When you're comparing one scientific field with another, then weighting by field makes sense. But it doesn't make sense when you're measuring the overall impact of one nation's science versus another, and it in fact produces erroneous results," Wagner said.

Instead of weighting citation results differently for papers in separate fields, Wagner and her colleagues simply combined papers in all fields and then calculated how nations compared using the raw citation data.

The researchers used the Web of Science database, which provides comprehensive citation data for studies in a wide variety of scientific disciplines. Using their measurement method, they found that China passed the United States for the top position in 2019, after passing the European Union in 2015.

In 2019, 1.67% of scientific articles with Chinese authors were in the top 1% of the most highly cited articles, compared to 1.62% of articles with U.S. authors. The U.S. was slightly ahead in 2018.

Even measurements using field normalizations have shown China to be improving rapidly in quality, even if they don't show China ahead of the United States. For example, in 2000, 1.77% of U.S articles were in the top 1%, compared to only 0.37% of Chinese articles, according to a report by the National Science Board.

But by 2016, that gap had closed considerably according to the field-normalized statistics: In that year, 1.88% of U.S. articles were in the top 1% and China was closing in with 1.12% in the highest ranks.

Overall, it has been widely recognized that China's total research output has been growing quickly in the past decade or more, Wagner said.

"But many experts had been saying that China was still lagging in quality, so there was no need for Western countries to worry. We don't think that gap exists anymore," she said.

"In an incredibly short time, China has advanced from far behind to producing research at the leading edge of science in some fields."

These new results showing China in the lead should help inform decisions on how the United States funds science, Wagner said.

"In the United States, we rely on science and technology for our national security, for our economy and for our health care," she said. "Being the world leader has been critically important to our economic success. A change in this position needs scrutiny."