Score, then rank: Researchers propose an integrated approach to grant review assessments

In many review programs, say the authors, proposals are funded primarily on the basis of reviewer ratings and summary statistics derived from them that represent reviewers' assessment of proposal quality. Relying solely on ratings, however, can pose problems for establishing funding priorities, "especially when distinguishing between similarly rated proposals, especially for projects of high quality, where only a few minor weaknesses can make the difference between a funded project and a rejection." Furthermore, say Gallo and colleagues, the use of ratings rubrics "often forces reviewers to penalize a project based on a series of identified but minor weaknesses, despite the fact that they may find the application overall to be potentially significant."

To address these concerns, the authors suggest adding rankings to the traditional rating approach and describe an innovative methodology called the Mallows-Binomial, which can then produce integrated proposal scores, as well as an induced preference ordering. In so doing, the approach allows funders to consider reviewers' scores alongside their opinions about how well the proposals stack up against one another. Pearce states that the inclusion of ranking will "more accurately distill reviewer opinion into a useful output to make an informed funding decision."

For research funders, the authors suggest clear benefits of the Mallows-Binomial protocol in comparison with ratings-only approaches. These include "providing a ranked priority list with confidence metrics, a higher degree of discrimination between similarly rated proposals, and a robustness to outliers and reviewer inconsistencies," features that will be important in distilling reviewer opinion "into a useful output to make the best, most informed decision" for funding crucial research.