Peering at peer review revealed high degree of chance associated with funding of grant applications (original) (raw)
A new approach to grant review assessments: score, then rank
Research integrity and peer review, 2023
Background In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications. We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical "toy" examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges' evaluations. For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions. Conclusions A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.
Peer review of grant applications in biology and medicine. Reliability, fairness, and validity
Scientometrics, 2009
This paper examines the peer review procedure of a national science funding organization (Swiss National Science Foundation) by means of the three most frequently studied criteria reliability, fairness, and validity. The analyzed data consists of 496 applications for projectbased funding from biology and medicine from the year 1998. Overall reliability is found to be fair with an intraclass correlation coefficient of 0.41 with sizeable differences between biology (0.45) and medicine (0.20). Multiple logistic regression models reveal only scientific performance indicators as significant predictors of the funding decision while all potential sources of bias (gender, age, nationality, and academic status of the applicant, requested amount of funding, and institutional surrounding) are non-significant predictors. Bibliometric analysis provides evidence that the decisions of a public funding organization for basic project-based research are in line with the future publication success of applicants. The paper also argues for an expansion of approaches and methodologies in peer review research by increasingly focusing on process rather than outcome and by including a more diverse set of methods e.g. content analysis. Such an expansion will be necessary to advance peer review research beyond the abundantly treated questions of reliability, fairness, and validity.
Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices
PLoS ONE, 2012
Background: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.
A new approach to peer review assessments: Score, then rank
Research Square (Research Square), 2022
Background: In many peer review settings, proposals are selected for funding on the basis of some summary statistics-such as the mean, median, or percentileof review scores. There are numerous challenges to working with scores. These include low inter-rater reliability, epistemological differences, susceptibility to varying levels of leniency or harshness of reviewers, and the presence of ties. A different approach that is able to mitigate some of these issues would be to additionally collect rankings such as top-k preferences or paired comparisons and incorporate them in the analysis of review scores. Rankings and paired comparisons are scale-free and can enforce demarcation between proposals by design. However, analyzing scores and rankings simultaneously has not been done until recently due to the lack of tools for principled modeling. Methods: We first introduce an innovative protocol for collecting rankings among top quality proposals. This rankings collection is done as an add-on to the typical peer review procedures focused on scores and does not require reviewers to rank all proposals. We then present statistical methodology for obtaining an integrated score for each proposal, and from the integrated scores an induced preference ordering, that captures both types of peer review inputs: scores and rankings. Our statistical methodology allows for the collected rankings to differ from the score-implied rankings; this feature is essential when the two quality assessments disagree which, as we find empirically, often happens in peer review. We illustrate how our method quantifies the uncertainty in order to better understand reviewer preferences among similarly scored proposals. Results: Using artificial "toy" examples and real peer review data, we demonstrate that incorporating top-k rankings into scores allows us to better learn when reviewers can distinguish between proposals. We also examine the robustness of this system to partial rankings, inconsistencies between ratings and rankings, and outliers. Finally, we discuss how, using panel data, this method can provide information about funding priority that provides a level of accuracy in a format that is well suited for the types of decisions research funders make. Conclusions: The gathering of both rating and ranking data and the use of integrated scores and its induced preference ordering can have many advantages over methods relying on ratings alone, leveraging more information to most accurately distill reviewer opinion into a useful output to make the most informed funding decision.
Criteria for assessing grant applications: a systematic review
Palgrave Communications, 2020
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non- Western countries, and examine fields other than the medical and health sciences.
Research Evaluation
Evaluation for the allocation of project-funding schemes devoted to sustain academic research often undergoes changes of the rules for the ex-ante selection, which are supposed to improve the capability of peer review to select the best proposals. How modifications of the rules realize a more accountable evaluation result? Do the changes suggest an improved alignment with the program's intended objectives? The article addresses these questions investigating Research Project of National Interest, an Italian collaborative project-funding scheme for academic curiosity-driven research through a case study design that provides a description of how the changes of the ex-ante evaluation process were implemented in practice. The results show that when government tries to steer the peer-review process by imposing an increasing number of rules to structure the debate among peers and make it more accountable, the peer-review practices remain largely impervious to the change.
BMC Medicine, 2010
Background: The objectives of this research were (a) to describe the current status of grant review for biomedical projects and programmes from the perspectives of international funding organisations and grant reviewers, and (b) to explore funders' interest in developing uniform requirements for grant review aimed at making the processes and practices of grant review more consistent, transparent, and user friendly. Methods: A survey to a convenience sample of 57 international public and private organisations that give grants for biomedical research was conducted. Nine participating organisations then emailed a random sample of their external reviewers an invitation to participate in a second electronic survey. Results: A total of 28 of 57 (49%) organisations in 19 countries responded. Organisations reported these problems as frequent or very frequent: declined review requests (16), late reports , administrative burden , difficulty finding new reviewers (4), and reviewers not following guidelines (4). The administrative burden of the process was reported to have increased over the past 5 years. In all, 17 organisations supported the idea of uniform requirements for conducting grant review and for formatting grant proposals. A total of 258/418 (62%) reviewers responded from 22 countries. Of those, 48% (123/258) said their institutions encouraged grant review, yet only 7% (17/258) were given protected time and 74% (192/258) received no academic recognition for this. Reviewers rated these factors as extremely or very important in deciding to review proposals: 51% (131/258) desire to support external fairness, 47% (120/258) professional duty, 46% (118/258) relevance of the proposal's topic, 43% (110/258) wanting to keep up to date, 40% (104/258) desire to avoid suppression of innovation. Only 16% (42/258) reported that guidance from funders was very clear. In all, 85% (220/258) had not been trained in grant review and 64% (166/258) wanted this. Conclusions: Funders reported a growing workload of biomedical proposals that is getting harder to peer review. Just under half of grant reviewers take part for the good of science and professional development, but many report lack of academic and practical support and clear guidance. Around two-thirds of funders supported the development of uniform requirements for the format and peer review of proposals to help ease the current situation.