Evaluation for the allocation of university research project funding: Can rules improve the peer review? (original) (raw)
Related papers
The Success of Peer Review Evaluation in University Research Funding – the Case Study from Slovakia
2018
Public funding mechanism for excellence is widely used mostly due its aim to raise the performance of higher education institutions to an excellent level since the reallocation is based on competitiveness of institutions or researchers. The approaches which are currently used for research evaluation are either peer review or bibliometric techniques. Peer review is based on deep expertise of committees and experts. However, its application is questioned to some extent, especially due to its ineffectiveness and inefficiency. In Slovakia, peer review process is applied to selection of projects by the Scientific Grant Agency. This paper identifies whether there is a relationship between peer review score of project proposals and research productivity. Case study is applied to the Scientific Grant Agency and its grant selection in year 2009 when the results of peer review process are for the first time available to the public. Our results show that peer review in most fields failed to pr...
The thesis presents a reformative criticism of science funding by peer review. The criticism is based on epistemological scepticism, regarding the ability of scientific peers, or any other agent, to have access to sufficient information regarding the potential of proposed projects at the time of funding. The scepticism is based on the complexity of factors contributing to the merit of scientific projects, and the rate at which the parameters of this complex system change their values. By constructing models of different science funding mechanisms, a construction supported by historical evidence, computational simulations show that in a significant subset of cases it would be better to select research projects by a lottery mechanism than by selection based on peer review. This last result is used to create a template for an alternative funding mechanism that combines the merits of peer review with the benefits of random allocation, while noting that this alternative is not so far removed from current practice as may first appear.
Scientometrics, 2012
There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001-2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.
Peer review for the evaluation of academic research: lessons from the Italian experience
Research Evaluation, 2007
The aim of the paper is to control the reliability of peer review when evaluating academic research in the Three-Year Research Assessment Exercise developed in Italy. Our analysis covers four disciplinary sectors: chemistry, biology, humanities and economics. The results provide evidence that highlights strengths and weaknesses of peer review for judging the quality of the academic research in different fields of science, vis-à-vis bibliometric indicators. Moreover, some basic features of the evaluation process are discussed, to understand their usefulness for reinforcing the effectiveness of the peers' final outcome. P Emanuela Reale is at Ceris CNR,
Peer review practices: a content analysis of external reviews in science funding
Research Evaluation, 2010
The primary purpose of this study is to open up the black box of peer review and to increase its transparency, understanding, and credibility. To this end, two arguments will be presented: First, epistemic and social aspects of peer review procedures are inseparable and mutually constitutive. Second, a content analysis of written reviews indicates that certain elements of peer culture from the 17 th century are still active in the scientific community. These arguments are illustrated by a case study on the peer review practices of a national funding institution, the Swiss National Science Foundation. Based on the case study and the two arguments it will be concluded more generally that peer review procedures show a distinctive specificity to the reviewed objects (e.g. papers or proposals), the organisational format (e.g. panels or external reviewers), or the surrounding context (e.g. disciplinary or interdisciplinary). Scientists, administrators, and the public may conclude that appraising peer review procedures should not be done by way of general principals but should be based on concrete factual knowledge on the specific process under discussion.
What works for peer review and decision-making in research funding: a realist synthesis
Research Integrity and Peer Review, 2022
Introduction Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom. Methods Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding. Results We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to ...