Do Reviewers Read References? And If So, Does It Impact Their Scores?

The following is from the Open Mike publication of NIH Extramural News.  We have always recommended defending research efforts and community work with peer-reviewed literature from other investigators.  This article demonstrates the value of this inclusion.
_________________________

In March 2017, we wrote about federal funders’ policies on interim research products, including preprints. We encouraged applicants and awardees include citations to preprints in their grant applications and progress reports. Some of your feedback pointed to the potential impact of this new policy on the peer review process.

Some issues will take a while to explore as preprints become more prevalent. But some we can dig into immediately. For example, how do references cited in an application impact review?  To start to address this question, we considered another one as well: do peer reviewers look at references – either those cited by applicants or others – while evaluating an application?  We had heard anecdotes, ranging from “Yes, I always do,” to “No, I don’t need to,’ but we didn’t have data one way or the other. And if reviewers do check references, how does it impact their understanding and scoring of an application?

So, together with colleagues from the NIH Center for Scientific Review (CSR), we reached out to 1,000 randomly selected CSR reviewers who handled applications for the January 1, 2018 Council Round. There were an equal number of chartered (i.e. permanent) and temporary reviewers solicited to participate (n=500 each) over a three week period from November 16 to December 8, 2017.

Our survey focused on the last grant where they served as primary reviewer. Specifically, we asked if they looked up any references that were either included in the application (i.e. internal references), and if they also looked up any that were not included in the application (i.e. external references). Depending on their answers to each of these questions, we also proceeded to ask certain respondents follow-up questions to better understand their initial feedback. We felt it would be interesting to know, for example, how reading the paper or abstract impacted their understanding of the application and their score.

We received 615 responses (62% of total), including 306 chartered members and 309 temporary members.  Figure 1 shows the responses related to if they looked up references, either internal or external to the application.  Most reviewers answered yes – particularly for internal references.