Navigating the Complexities of Grant Peer Review: A Qualitative Exploration

The Untrained Reviewer: Learning by Doing in a Time-Constrained Environment

Grant peer review, a cornerstone of scientific funding, relies heavily on the expertise of volunteer reviewers. Yet, our study reveals a concerning trend: a "learn as you go" approach to training, where reviewers draw upon personal experience and philosophies rather than standardized instruction. This lack of formal training, coupled with the ever-present pressure of academic commitments, creates a challenging environment for reviewers striving to provide rigorous and unbiased evaluations.

"I learned from...some of my mentors and when I watched them as chairs...and then kind of learned from them."

Despite the availability of written guides, many reviewers reported not fully utilizing these resources, citing time constraints as a major barrier. This reliance on individual interpretation raises concerns about consistency and fairness in the review process.

"I did not receive training for any of those roles. Zero training. I mean I was given all the documents...the guides to review and so on."

The Meritorious Middle: Wrestling with Distinguishing Equally Strong Applications

The most time-consuming and arguably most critical aspect of grant review lies in evaluating the "meritorious middle"—the large pool of applications deemed fundable if resources were unlimited. Our study reveals the struggle reviewers face in differentiating these high-quality proposals. Without clear guidelines and facing immense pressure to distribute limited funds, reviewers often resort to subjective criteria like "interest" or “fashion,” risking inequitable outcomes.

"You can have a lot of grants where there’s nothing flawed...But there’s just another grant...that is scored marginally higher because it catches the eye and the interest of the review committee..."

This frustration is palpable, with some reviewers even suggesting a lottery system might be equally effective in this gray zone.

"That is really hard to grapple with in a peer review process…I honestly don’t think that the review Committee does a better job than a lottery."

The limitations of the current rating system, with its narrow scoring bands, further exacerbate this challenge, leading to a "mushy middle" where subtle distinctions carry significant weight. The need for improved calibration methods and potentially alternative evaluation approaches like ranking is evident.

"In the mushy middle [is the problem]. The exceptional ones, usually, you know, come through. But ones that are deeply, deeply flawed...we rarely give those really low rankings or really low scores, right?"

Reputation and Relationships: The Unintended Influence of Bias

Despite training on bias, unconscious biases, particularly regarding applicant reputation and established relationships, continue to permeate the review process. The "Matthew Effect," where past success begets future funding, puts early-career researchers and those from underrepresented groups at a disadvantage.

"You hope that it’s [grant review] based on merit, not who you are. But I have seen a degree of fascination with established career researchers who...get the benefit of the doubt..."

Furthermore, the social dynamics of in-person review meetings, while fostering collegiality, can inadvertently create an uneven playing field.

"It’s the side conversations sometimes away from the grant review that are enriching and rewarding as part of the process."

While some reviewers appreciate the networking opportunities, others question the potential for bias introduced by informal discussions outside the formal review setting, highlighting the need for clear guidelines and ethical considerations regarding social interactions during review periods.

The Essential Role of the Chair: Guiding the Process and Mitigating Bias

The Chair emerges as a crucial figure in navigating the complexities of grant peer review. From facilitating discussions and mediating conflicts to addressing bias and clarifying scoring procedures, the Chair's leadership is vital for ensuring a fair and rigorous process.

"it really sort of helps if you have a really good Chair."

Participants recognized the Chair's "responsibility" to manage potentially heated disagreements and ensure that all voices are heard.

"Sometimes discussions can get heated…, especially if you have a reviewer that really just doesn’t like something about the grant..."

However, our findings also suggest a need for more comprehensive training and support for Chairs, equipping them with the skills and resources to effectively manage these multifaceted responsibilities.