Studying grant decision-making: a linguistic analysis of review reports (original) (raw)
Related papers
This paper discusses a scientometric-statistical model for inferring attributes of 'frontier research' in peer-reviewed research proposals submitted to the European Research Council (ERC). The first step conceptualizes and defines indicators to capture attributes of frontier research, by using proposal texts as well as scientometric and bibliometric data of grant applicants. Based on the combination of indicators, the second step models the decision probability of a proposal to be accepted and compares outcomes between the model and peer-review decision, with the goal to determine the influence of frontier research on the peer-review process. In a first attempt, we demonstrate and discuss in a proof-of-concept approach a data sample of about 10% of all proposals submitted to the ERC call (StG2009) for starting grants in the year 2009, which shows the feasibility and usefulness of the scientometric-statistical model. Ultimately the overall concept is aiming at testing new methods for monitoring the effectiveness of peer-review processes by taking a scientometric perspective of research proposals beyond publication and citation statistics.
Cellular Oncology, 2009
Does past performance influence success in grant applications? In this study we test whether the grant allocation decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of the applicants in terms of publications and citations, and with the results of the peer review process organized by the Council. We show that the Council is successful in distinguishing grant applicants with above-average performance from those with below-average performance, but within the former group no correlation could be found between past performance and receiving a grant. When comparing the best performing researchers who were denied funding with the group of researchers who received it, the rejected researchers significantly outperformed the funded ones. Furthermore, the best rejected proposals score on average as high on the outcomes of the peer review process as the accepted proposals. Finally, we found that the Council under study successfully corrected for gender effects during the selection process. We explain why these findings may be more general than for this case only. However, if research councils are not able to select the 'best' researchers, perhaps they should reconsider their mission. In a final section with policy implications, we discuss the role of research councils at the level of the science system in terms of variation, innovation, and quality control.
Peering at peer review revealed high degree of chance associated with funding of grant applications
Journal of Clinical Epidemiology, 2006
Background and Objectives: There is a persistent degree of uncertainty and dissatisfaction with the peer review process underlining the need to validate the current grant awarding procedures. This study compared the CLassic Structured Scientific In-depth two reviewer critique (CLASSIC) with an all panel members' independent ranking method (RANKING). Eleven reviewers, reviewed 32 applications for a pilot project competition at a major university medical center.
A concept for inferring 'frontier research' in grant proposals
Scientometrics, 2013
This paper discusses a concept for inferring attributes of 'frontier research' in peer-reviewed research proposals under the popular scheme of the European Research Council (ERC). The concept serves two purposes: firstly to conceptualize, define and operationalize in scientometric terms attributes of frontier research; and secondly to build and compare outcomes of a statistical model with the review decision in order to obtain further insight and reflect upon the influence of frontier research in the peer-review process. To this end, indicators across scientific disciplines and in accord with the strategic definition of frontier research by the ERC are elaborated, exploiting textual proposal information and other scientometric data of grant applicants. Subsequently, a suitable model is formulated to measure ex-post the influence of attributes of frontier research on the decision probability of a proposal to be accepted. We present first empirical data as proof of concept for inferring frontier research in grant proposals. Ultimately the concept is aiming at advancing the methodology to deliver signals for monitoring the effectiveness of peer-review processes.
Le Centre pour la Communication Scientifique Directe - HAL - memSIC, 2013
HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
The journal of research administration, 2015
While Elizabeth Barrett Browning counted 25 ways in which she loves her husband in her poem, "How Do I Love Thee? Let me Count the Ways," we identified only eight ways to evaluate the potential for success of a federal research grant proposal. This may be surprising, as it seems upon initial glance of the review criteria used by various federal funding agencies that each has its own distinct set of "rules" regarding the review of grant proposals for research and scholarship. Much of the grantsmanship process is dependent upon the review criteria, which represent the funders' desired impact of the research. But since most funders that offer research grants share the overarching goals of supporting research that (1) fits within its mission and (2) will bring a strong return on its financial investment, the review criteria used to evaluate research grant proposals are based on a similar set of fundamental questions. In this article, we compare the review criteri...
Linguistic analysis of grant proposals: European Union research grants
English for specific purposes, 1999
Grant proposals are a genre that all academics will have to come to terms with at some point of their career, usually the sooner the better. Yet, there has been very little research on their characteristic features and they are not included in most courses of academic writing. We seek to remedy some of this problem in this paper, which is based on a sample of 34 proposals from European Union (EU) research grant applications written mainly by Finnish-led research teams. Our approach draws on Swalesian genre analysis as well as a social constructionist theory of genre . In our analysis, we identified ten recurrent moves in the proposals, reflecting the generic affinity of grant proposals to both academic research papers ) and promotional genres (Bhatia, 1993, in addition to moves specific to the grant proposals genre. The results should benefit both genre research and the teaching of academic writing. Published by Elsevier Science Ltd
Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices
PLoS ONE, 2012
Background: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.
Appraisal in the research genres: An analysis of grant proposals by Argentinean researchers
Revista signos, 2010
Writing an effective grant proposal poses many challenges for researchers, in particular for those whose native language is not English and who may have received little formal instruction in writing. While considerable research has been undertaken on the genre of the research article, grant proposals have received much less attention, despite their major significance in research settings. This study aims at investigating how grant proposal writers in the disciplines of chemistry and physics position themselves in relation to other researchers and theorists in key obligatory stages of this genre, namely Benefits and Importance Claim (Connor & Mauranen, 1999). Two successful and praiseworthy examples from disciplinary areas where access to international funding is regarded as important are analyzed. They were written by Argentinean non-native English speakers working at the universidad Nacional de San Luis. The proposals are analyzed by using the Appraisal framework proposed by Martin (2000) and Martin and White (2005), with a focus on the system of ENGAGEMENT. Results indicate that the proposals were highly heteroglossic, and that a variety of ENGAGEMENT resources were used. The instances were predominantly expansive, suggesting that authors tend to invite rather than challenge their colleagues' views. This may be interpreted as an attempt to address a potentially varied audience. These results can assist authors in becoming aware of the interpersonal resources they may use to position themselves and align their audiences when writing their grant proposals.
Criteria for assessing grant applications: a systematic review
Palgrave Communications, 2020
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use to assess grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as ‘criterion’ into an evaluated entity (i.e., the object of evaluation) and an evaluation criterion (i.e., the dimension along which an entity is evaluated). In total, the synthesis includes 12 studies on grant peer review criteria. Two-thirds of these studies examine criteria in the medical and health sciences, while studies in other fields are scarce. Few studies compare criteria across different fields, and none focus on criteria for interdisciplinary research. We conducted a qualitative content analysis of the 12 studies and thereby identified 15 evaluation criteria and 30 evaluated entities, as well as the relations between them. Based on a network analysis, we determined the following main relations between the identified evaluation criteria and evaluated entities. The aims and outcomes of a proposed project are assessed in terms of the evaluation criteria originality, academic relevance, and extra-academic relevance. The proposed research process is evaluated both on the content level (quality, appropriateness, rigor, coherence/justification), as well as on the level of description (clarity, completeness). The resources needed to implement the research process are evaluated in terms of the evaluation criterion feasibility. Lastly, the person and personality of the applicant are assessed from a ‘psychological’ (motivation, traits) and a ‘sociological’ (diversity) perspective. Furthermore, we find that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Grant peer review could therefore be considered unfair and biased. Our findings suggest that future studies on criteria in grant peer review should focus on the applicant, include data from non- Western countries, and examine fields other than the medical and health sciences.