The troubles with peer review for allocating research funding (original) (raw)
Related papers
Peer Review at the NSF: A Dialectical Policy Analysis
Social Studies of Science, 1979
The controversy over peer review is viewed as a dialectic. The arguments espoused by advocates and critics of the system wherein research proposals are evaluated by advisors to funding agencies are reviewed, particularly the findings of two recent studies of peer review at the National Science Foundation These findings seem to establish merit as the primary factor m the recommendations of peer reviewers to fund proposals The findings also beg several questions as to 'acceptable' definitions of meritoriousness and innovativeness, the links among belief, perception, and evaluation, and the sanctioned operation of particularistic factors m the review process Future studies, it is suggested, must include psychological variables - especially measurement of applicants' and reviewers' 'cognitive styles' - if data are to narrow gaps in knowledge and inform the debate itself Finally, three models which undergird views of peer review are discussed and related to key so...
2015
One of the cornerstones of the scientific process is securing funding for one's research. A key mechanism by which funding outcomes are determined is the scientific peer review process. Our focus is on biomedical research funded by the U.S. National Institutes of Health (NIH). NIH spends $30.3 billion on medical research each year, and more than 80% of NIH funding is awarded through competitive grants that go through a peer review process (NIH, 2015). Advancing our understanding of this review process by investigating variability among review panels and the efficiency of different meeting formats has enormous potential to improve scientific research throughout the nation. NIH's grant review process is a model for federal research foundations, including the National Science Foundation and the U.S. Department of Education's Institute of Education Sciences. It involves panel meetings in which collaborative decision making is an outgrowth of socially mediated cognitive tasks. These tasks include summarization, argumentation, evaluation, and critical discussion of the perceived scientific merit of proposals with other panel members. Investigating how grant review panels function thus allows us not only to better understand processes of collaborative decision making within a group of distributed experts (Brown et al., 1993) that is within a community of practice (Lave & Wenger, 1991), but also to gain insight into the effect of peer review discussions on outcomes for funding scientific research. Theoretical Framework A variety of research has investigated how the peer review process influences reviewers' scores, including the degree of inter-rater reliability among reviewers and across panels, and the impact of discussion on changes in reviewers' scores. In addition, educational theories of distributed cognition, communities of practice, and the sociology of science frame the peer review process as a collaborative decision-making task involving multiple, distributed experts. The following sections review each of these bodies of literature.
Evaluation of research proposals by peer review panels: broader panels for broader assessments?
Science & public policy, 2023
Panel peer review is widely used to decide which research proposals receive funding. Through this exploratory observational study at two large biomedical and health research funders in the Netherlands, we gain insight into how scientific quality and societal relevance are discussed in panel meetings. We explore, in ten review panel meetings of biomedical and health funding programmes, how panel composition and formal assessment criteria affect the arguments used. We observe that more scientific arguments are used than arguments related to societal relevance and expected impact. Also, more diverse panels result in a wider range of arguments, largely for the benefit of arguments related to societal relevance and impact. We discuss how funders can contribute to the quality of peer review by creating a shared conceptual framework that better defines research quality and societal relevance. We also contribute to a further understanding of the role of diverse peer review panels.
Peer review practices: a content analysis of external reviews in science funding
Research Evaluation, 2010
The primary purpose of this study is to open up the black box of peer review and to increase its transparency, understanding, and credibility. To this end, two arguments will be presented: First, epistemic and social aspects of peer review procedures are inseparable and mutually constitutive. Second, a content analysis of written reviews indicates that certain elements of peer culture from the 17 th century are still active in the scientific community. These arguments are illustrated by a case study on the peer review practices of a national funding institution, the Swiss National Science Foundation. Based on the case study and the two arguments it will be concluded more generally that peer review procedures show a distinctive specificity to the reviewed objects (e.g. papers or proposals), the organisational format (e.g. panels or external reviewers), or the surrounding context (e.g. disciplinary or interdisciplinary). Scientists, administrators, and the public may conclude that appraising peer review procedures should not be done by way of general principals but should be based on concrete factual knowledge on the specific process under discussion.
Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices
PLoS ONE, 2012
Background: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.
Peer Reviewing: Weaknesses and Proposed Solutions
Weaknesses of grants' and journals' peer reviewing p. 116) also affirmed that regarding peer review there is "more evidence of harm than benefit… [and] Studies so far have shown that it is slow, expensive, ineffective, something of a lottery, prone to bias and abuse, and hopeless at spotting errors and fraud." Consequently, several potential improvements and alternatives have been proposed.
Reviewing the review process : new frontiers of peer review
2016
This news article introduces a new COST Action entitled PEERE (TD1306), which stands for New Frontiers of Peer Review (PEERE). PEERE is a trans-domain proposal which brings together researchers from various different disciplines and science stakeholders for the purpose of reviewing the process of peer review. PEERE officially began in May 2014 and will end in May 2018. Thirty-one countries, including Malta, are currently participating in the Action. In order to set the context in which this COST Action was initiated, we first look very briefly at the history of the process of peer review and various models of peer review currently in use. We then share what this COST Action hopes to achieve.
What works for peer review and decision-making in research funding: a realist synthesis
Research Integrity and Peer Review, 2022
Introduction Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom. Methods Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding. Results We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to ...