Peer review for the evaluation of academic research: lessons from the Italian experience (original) (raw)
Related papers
Peer review for the evaluation of the academic research
The aim of the paper is to control the reliability of peer review when evaluating academic research in the Three-Year Research Assessment Exercise developed in Italy. Our analysis covers four disciplinary sectors: chemistry, biology, humanities and economics. The results provide evidence that highlights strengths and weaknesses of peer review for judging the quality of the academic research in different fields of science, vis-à-vis bibliometric indicators. Moreover, some basic features of the evaluation process are discussed, to understand their usefulness for reinforcing the effectiveness of the peers' final outcome. P Emanuela Reale is at Ceris CNR,
Bibliometric evaluation vs. informed peer review: Evidence from Italy
Research Policy, 2015
Italy* A relevant question for the organization of large scale research assessments is whether bibliometric evaluation and informed peer review where reviewers know where the work was published, yield similar results. It would suggest, for instance, that less costly bibliometric evaluation might -at least partlyreplace informed peer review, or that bibliometric evaluation could reliably monitor research in between assessment exercises. We draw on our experience of evaluating Italian research in Economics, Business and Statistics, where almost 12,000 publications dated 2004-2010 were assessed. A random sample from the available population of journal articles shows that informed peer review and bibliometric analysis produce similar evaluations of the same set of papers. Whether because of independent convergence in assessment, or the influence of bibliometric information on the community of reviewers, the implication for the organization of these exercises is that these two approaches are substitutes.
Scientometrics, 2012
There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001-2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.
Bibliometrically Disciplined Peer Review: on Using Indicators in Research Evaluation
Scholarly Assessment Reports, 2020
Evaluation of research uses peer review and bibliometrics, and the debate about their balance in research evaluation continues. Both approaches have supporters, and both approaches are criticized. In this paper, we describe an interesting case in which the use of bibliometrics in a panel-based evaluation of a midsized university was systematically tried out. The case suggests a useful way in which bibliometric indicators can be used to inform and improve peer review and panel-based evaluation. We call this 'disciplined peer review', and disciplined is used here in a constructive way: Bibliometrically disciplined peer review is more likely to avoid the subjectivity that often influences the outcomes of the peer and panel reviewbased evaluation.
ArXiv, 2021
In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates’ CVs. The NSQ divides academic disciplines into two categories, i.e. citation-based disciplines (CDs) and non-citation-based disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how...
Scientometrics
This article aims to introduce a special issue on ''Scientometrics of peer review'', which collects papers originally presented at workshops and conferences organised by the COST ACTION TD1306 ''New frontiers of peer review''. Peer review is the cornerstone of science and is one of the underlying processes that bring about publication traces that are at the heart of bibliometric studies. Unfortunately, despite its importance, quantitative studies on peer review are still poorly developed, often due to lack of data. The issue aims to promote the establishment of peer review as an interdisciplinary field of research and stimulate further quantitative research.
Peer review as a science evaluation tool: main tensions and some alternative proposals
e-Ciencias de la Información, 2023
Peer review plays a crucial role in scientific and academic research. However, the different ways that have been implemented have been criticized by the international scientific community. This essay aims to identify the main questionings raised about peer review as a science assessment tool and propose alternative solutions to these discussions. The field of study from which the research was approached was science and technology evaluation studies, a qualitative methodology of exploratory and descriptive scope was applied that included the search, compilation and analysis of various sources of scientific information in English, Spanish and Portuguese languages that addressed the proposed categories. A brief overview of peer review as a science assessment tool is presented, along with a summary of the main types of peer review, as well as their advantages and disadvantages. The text addresses the questionings and biases present in the peer review system that can perpetuate existing scientific paradigms, discourage novel ideas, and reinforce systemic inequalities within academia. Although measures to address these biases have been put in place, peer review remains a human-driven process and is not entirely free of bias or limitations. A series of alternatives are proposed to improve the peer review process with the purpose of strengthening the quality and reliability of peer review, through transparency, diversity and collaboration in scientific research.
A Scientometrics-Informed Peer-Review Exercise
2015
A scientometrics-informed peer review exercise is described, as applied at the “Babeș-Bolyai” University in Romania for an internal young researcher short-term grant competition. The applicants were graded from A (“internationally-leading”) to B (“regionally-leading”), C (“nationally-leading”) and D (“low-impact”). Judgment was delegated to panels of three evaluators per applicant, and was informed by such criteria as number of corresponding-author articles in higher-impact journals (as defined mainly by those in a range of highest impact factors per field, or variations thereof), citations, patents, books present in large numbers of libraries, and others. Importantly, the grade was dictated solely by the applicant’s own achievements within their field, rather than by comparison to other applicants. Furthermore, the criteria specifically rest it upon the evaluators to judge the quality of the scientific content, to the extent where a higher grade may be refused in spite of an appare...