Peer Review at the NSF: A Dialectical Policy Analysis (original) (raw)
Related papers
The dedisciplining of peer review
Minerva, 2012
The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with the question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes.
Peer review practices: a content analysis of external reviews in science funding
Research Evaluation, 2010
The primary purpose of this study is to open up the black box of peer review and to increase its transparency, understanding, and credibility. To this end, two arguments will be presented: First, epistemic and social aspects of peer review procedures are inseparable and mutually constitutive. Second, a content analysis of written reviews indicates that certain elements of peer culture from the 17 th century are still active in the scientific community. These arguments are illustrated by a case study on the peer review practices of a national funding institution, the Swiss National Science Foundation. Based on the case study and the two arguments it will be concluded more generally that peer review procedures show a distinctive specificity to the reviewed objects (e.g. papers or proposals), the organisational format (e.g. panels or external reviewers), or the surrounding context (e.g. disciplinary or interdisciplinary). Scientists, administrators, and the public may conclude that appraising peer review procedures should not be done by way of general principals but should be based on concrete factual knowledge on the specific process under discussion.
Evaluation of research proposals by peer review panels: broader panels for broader assessments?
Science & public policy, 2023
Panel peer review is widely used to decide which research proposals receive funding. Through this exploratory observational study at two large biomedical and health research funders in the Netherlands, we gain insight into how scientific quality and societal relevance are discussed in panel meetings. We explore, in ten review panel meetings of biomedical and health funding programmes, how panel composition and formal assessment criteria affect the arguments used. We observe that more scientific arguments are used than arguments related to societal relevance and expected impact. Also, more diverse panels result in a wider range of arguments, largely for the benefit of arguments related to societal relevance and impact. We discuss how funders can contribute to the quality of peer review by creating a shared conceptual framework that better defines research quality and societal relevance. We also contribute to a further understanding of the role of diverse peer review panels.
2018
Public funding mechanism for excellence is widely used mostly due its aim to raise the performance of higher education institutions to an excellent level since the reallocation is based on competitiveness of institutions or researchers. The approaches which are currently used for research evaluation are either peer review or bibliometric techniques. Peer review is based on deep expertise of committees and experts. However, its application is questioned to some extent, especially due to its ineffectiveness and inefficiency. In Slovakia, peer review process is applied to selection of projects by the Scientific Grant Agency. This paper identifies whether there is a relationship between peer review score of project proposals and research productivity. Case study is applied to the Scientific Grant Agency and its grant selection in year 2009 when the results of peer review process are for the first time available to the public. Our results show that peer review in most fields failed to pr...
In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following col-laborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process. As the primary means by which scientists secure funding for their research programs, grant peer review is a keystone of scientific research. The largest funding agency for biomedical, behavioral, and clinical research in the USA, the National Institutes of Health (NIH), spends more than 80% of its $30.3 billion annual budget on funding research grants evaluated via peer review (NIH 2016). As part of the mechanism by which this money is allocated to scientists, collaborative peer review panels of expert scientists (referred to as 'study sections' at NIH) convene to evaluate grant applications and assign scores that inform later funding decisions by NIH govern-ance. Thus, deepening our understanding of how peer review ostensibly identifies the most promising, innovative research is crucial for the scientific community writ large. The present study builds upon existing work evaluating the reliability of peer review by examining how the discourse practices of reviewers during study section meetings may contribute to low reliability in peer review outcomes. The NIH peer review process is structured around study sections that engender distributed expertise (Brown et al. 1993), as reviewers evaluate applications based on their particular domain(s) of expertise but then share their specialized knowledge with others who have related but distinct expertise. The very structure of study sections
Peer review: The experience and views of early career researchers
Learned Publishing
This paper presents selected findings from the first year of a 3-year longitudinal study of early career researchers (ECRs), which sought to ascertain current and changing habits in scholarly communication. Specifically, the aims of the paper are to show: (1) how much experience and knowledge ECRs had of peer review both as authors and as reviewers; (2) what they felt the benefits were and what suggestions they had for improvement; (3) what they thought of open peer review (OPR); and (4) who they felt should organize peer review. Data were obtained from 116 science and social science ECRs, most of whom had published and were subject to in-depth interviews conducted face-to-face, via Skype, or over the telephone. An extensive literature review was also conducted to provide a context and supplementary data for the findings. The main findings were that: (1) most ECRS are well informed about peer review and generally like the experience, largely because of the learning experiences obtained; (2) they like blind double-peer review, but would like some improvements, especially with regards to reviewer quality; (3) most are uncomfortable with the idea of OPR; and (4) most would like publishers to continue organizing peer review because of their perceived independence.
The Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process
Perspectives on Psychological Science, 2009
Anonymous peer review has served as the bedrock of research dissemination in scientific psychology for decades and has only sporadically been questioned. However, other disciplines, such as biomedicine and physics, have found the traditional peer-review system to be wanting and have begun to test and try alternative practices. In this article, we survey criticisms of the traditional peer-review system and describe several alternatives in the interests of facilitating discussion and debate. We also consider why the natural sciences tend to employ fewer reviewers and have lower rejection rates than do the social sciences. Our two recommendations are that a serious discussion of problems and alternatives to peer review should be started at all levels of psychology and that a science of research communication should be a priority, with psychologists as part of its advance guard because of their relevant substantive and methodological knowledge.
Cellular Oncology, 2009
Does past performance influence success in grant applications? In this study we test whether the grant allocation decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of the applicants in terms of publications and citations, and with the results of the peer review process organized by the Council. We show that the Council is successful in distinguishing grant applicants with above-average performance from those with below-average performance, but within the former group no correlation could be found between past performance and receiving a grant. When comparing the best performing researchers who were denied funding with the group of researchers who received it, the rejected researchers significantly outperformed the funded ones. Furthermore, the best rejected proposals score on average as high on the outcomes of the peer review process as the accepted proposals. Finally, we found that the Council under study successfully corrected for gender effects during the selection process. We explain why these findings may be more general than for this case only. However, if research councils are not able to select the 'best' researchers, perhaps they should reconsider their mission. In a final section with policy implications, we discuss the role of research councils at the level of the science system in terms of variation, innovation, and quality control.
Peer review as a science evaluation tool: main tensions and some alternative proposals
e-Ciencias de la Información, 2023
Peer review plays a crucial role in scientific and academic research. However, the different ways that have been implemented have been criticized by the international scientific community. This essay aims to identify the main questionings raised about peer review as a science assessment tool and propose alternative solutions to these discussions. The field of study from which the research was approached was science and technology evaluation studies, a qualitative methodology of exploratory and descriptive scope was applied that included the search, compilation and analysis of various sources of scientific information in English, Spanish and Portuguese languages that addressed the proposed categories. A brief overview of peer review as a science assessment tool is presented, along with a summary of the main types of peer review, as well as their advantages and disadvantages. The text addresses the questionings and biases present in the peer review system that can perpetuate existing scientific paradigms, discourage novel ideas, and reinforce systemic inequalities within academia. Although measures to address these biases have been put in place, peer review remains a human-driven process and is not entirely free of bias or limitations. A series of alternatives are proposed to improve the peer review process with the purpose of strengthening the quality and reliability of peer review, through transparency, diversity and collaboration in scientific research.