What works for peer review and decision-making in research funding: a realist synthesis (original) (raw)
Related papers
Evaluation of research proposals by peer review panels: broader panels for broader assessments?
Science & public policy, 2023
Panel peer review is widely used to decide which research proposals receive funding. Through this exploratory observational study at two large biomedical and health research funders in the Netherlands, we gain insight into how scientific quality and societal relevance are discussed in panel meetings. We explore, in ten review panel meetings of biomedical and health funding programmes, how panel composition and formal assessment criteria affect the arguments used. We observe that more scientific arguments are used than arguments related to societal relevance and expected impact. Also, more diverse panels result in a wider range of arguments, largely for the benefit of arguments related to societal relevance and impact. We discuss how funders can contribute to the quality of peer review by creating a shared conceptual framework that better defines research quality and societal relevance. We also contribute to a further understanding of the role of diverse peer review panels.
2018
In 2009, RAND Europe conducted a literature review in order to assess the effectiveness and efficiency of peer review for grant funding. This report presents an update to that review to reflect new literature on the topic, and adds case studies exploring peer review practice at six international funders. This report was produced with funding from the Canadian Institutes of Health Research. It will be of interest to government officials dealing with research funding policy, research funders including governmental and charitable funders, research institutions, researchers, and research users. Although the case studies focus on biomedical and health research, the literature review takes a broader scope and it is likely the findings may be of relevance to wider research fields.
Towards Bottom-Up, Stakeholder-Driven Research Funding–Open Science and Open Peer Review
Abstract: The current practices of research funding do not yet use means of communication and collaboration of the Internet age effectively. Combined with a number of information flow barriers associated with research funding this results in inefficiencies and intransparencies. We present a vision how an open science platform for research funding and cross-fertilization could be realized. It is based on stake-holder involvement and community self-organisation.
Peer review practices: a content analysis of external reviews in science funding
Research Evaluation, 2010
The primary purpose of this study is to open up the black box of peer review and to increase its transparency, understanding, and credibility. To this end, two arguments will be presented: First, epistemic and social aspects of peer review procedures are inseparable and mutually constitutive. Second, a content analysis of written reviews indicates that certain elements of peer culture from the 17 th century are still active in the scientific community. These arguments are illustrated by a case study on the peer review practices of a national funding institution, the Swiss National Science Foundation. Based on the case study and the two arguments it will be concluded more generally that peer review procedures show a distinctive specificity to the reviewed objects (e.g. papers or proposals), the organisational format (e.g. panels or external reviewers), or the surrounding context (e.g. disciplinary or interdisciplinary). Scientists, administrators, and the public may conclude that appraising peer review procedures should not be done by way of general principals but should be based on concrete factual knowledge on the specific process under discussion.
Technology &# 38; Innovation, 2010
Increasing demands on the part of the public for a demonstrable return on their investment in scientific and technical research have led to the widespread introduction of considerations of societal impacts into the peer review processes at public science and technology funding agencies. This answer to the accountability challenge also introduces a peculiar strain on peer review: expertise in particular areas of scientific and technical research is no guarantee of expertise in addressing the societal impacts of proposed research. Presenting preliminary results of a larger study, this article describes five current models of the peer review of grant proposals and shows that different agencies have very different ways of incorporating societal impacts considerations. The article also elucidates a notion of theoretical adequacy, which will be used to determine whether and how some peer review processes are better than others. The objectives of this article are to lay out the description of the agencies and to offer a preliminary assessment of each model's theoretical adequacy. The objective of our larger study is to determine the best ways to incorporate societal impacts considerations into the peer review of grant proposals, thus helping funding agencies respond to the demand for demonstrable results.
BMC Health Services Research, 2015
Background: Despite the widely recognised importance of sustainable health care systems, health services research remains generally underfunded in Australia. The Australian Centre for Health Services Innovation (AusHSI) is funding health services research in the state of Queensland. AusHSI has developed a streamlined protocol for applying and awarding funding using a short proposal and accelerated peer review.
F1000Research, 2017
Evaluating applications for multi-national, multi-disciplinary, dual-purpose research consortia is highly complex. There has been little research on the peer review process for evaluating grant applications and almost none on how applications for multi-national consortia are reviewed. Overseas development investments are increasingly being channelled into international science consortia to generate high-quality research while simultaneously strengthening multi-disciplinary research capacity. We need a better understanding of how such decisions are made and their effectiveness.An award-making institution planned to fund 10 UK-Africa research consortia. Over two annual rounds, 34 out of 78 eligible applications were shortlisted and reviewed by at least five external reviewers before final selections were made by a face-to-face panel. We used an innovative approach involving structured, overt observations of award-making panel meetings and semi-structured interviews with panel members ...
What do we know about grant peer review in the health sciences?
F1000Research, 2017
Background: Peer review decisions award >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 relevant papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the overall efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and th...