Assessment, evaluations, and definitions of research impact: A review (original) (raw)
Related papers
The Journal of Technology Transfer
National governments spend significant amounts of money supporting public research. However, in an era where the international economic climate has led to budget cuts, policymakers increasingly are looking to justify the returns from public investments, including in science and innovation. The so-called 'impact agenda' which has emerged in many countries around the world is part of this response; an attempt to understand and articulate for the public what benefits arise from the research that is funded. The United Kingdom is the most progressed in implementing this agenda and in 2014 the national research assessment exercise, the Research Excellence Framework, for the first time included the assessment of research impact as a component. For the first time within a dual funding system, funding would be awarded not only on the basis of the academic quality of research, but also on the wider impacts of that research. In this paper we outline the context and approach taken by the UK government, along with some of the core challenges that exist in implementing such an exercise. We then synthesise, together for the first time, the results of the only two national evaluations of the exercise and offer reflections for future exercises both in the UK and internationally.
WORKING PAPER Research Impact Evaluation, a Wider Context: Findings from a Research Impact Pilot
2014
In the face of increasing pressure to demonstrate the socio-economic impact of funded research, whether it is funded directly by research councils or indirectly by governmental research block grants, institutions have to tackle the complexity of understanding, tracking, collecting, and analysing the impact of all their research activities. This paper attempts to encapsulate the wider context of research impact by delineating a broad definition of what might be classified as impact. It also suggests a number of different dimensions that can help in the development of a systematic research impact assessment framework. The paper then proceeds to indicate how boundaries and criteria around the definition of impact and these dimensions can be used to refine the impact assessment framework in order to focus on the objectives of the assessor. A pilot project, run at Brunel University, was used to test the validity of the approach and possible consequences. A tool specifically developed for...
Measuring the Impact of Research: Lessons from the UK’s Research Excellence Framework 2014
PLOS ONE, 2016
Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom's national research quality exercise, the Research Excellence Framework (REF) 2014, which piloted impact assessment as part of the overall evaluation offers a lens to view how impact of research in different disciplines can be measured. Overall research quality was assessed through quality of outputs, 'impact' and research environment. We performed two studies using the REF 2014 as a PLOS ONE |
Rethinking Research Impact Assessment: A Multidimensional Approach
Austrian Presidency of the EU Council Conference on the Impact of Social Sciences and Humanities for a European Research Agenda – Valuation of SSH in Mission-oriented Research, 2018
An interest in the evaluation of research impact – or the influence of scientific research beyond academia – has been observable worldwide. Several countries have introduced national research assessment systems which take into account this new element of evaluation. So far, research on this practice has focused mainly on the practicalities of the different existing policies: the definition of the term ‘research impact’, different approaches to measuring it, their relative challenges and the possible use of such evaluations. But the introduction of a new element of evaluation gives rise not only to challenges of a practical nature, but also to important ethical consequences in terms of academic identity, reflexivity, power structures, distribution of labour in terms of workloads etc. In order to address these questions and the relevant needs of researchers, in this paper we propose a multidimensional model that considers different attributes of research impact: Responsiveness, Accessibility, Reflexivity, Ecology and Adaptability. This holistic, multidimensional model of evaluation, designed particularly for self-assessment or internal assessment, recognises the qualities a project has on these different scales in a broader perspective, rather than offering a simple and single numerical evaluation. This model addresses many of the ethical dilemmas that accompany conducting impact-producing research. To exemplify the usefulness of the proposed model, the authors provide real-life research project assessment examples conducted with the use of the Multidimensional Approach for Research Impact Assessment (MARIA Model).
2018
Research impact is often talked about, but how clear is it what this term really means? Kristel Alla, Wayne Hall, Harvey Whiteford, Brian Head and Carla Meurk find that academic literature discusses research impact but often without properly defining it, with academic discourses mostly drawing on bureaucratic definitions originating from the UK. The authors highlight four core elements that comprise most research impact definitions and propose a new conceptualisation of research impact relevant to health policy.
How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework
There is growing interest in the wider impact of research outside academia. In particular, there is concern about the extent to which the topic and principal beneficiaries of such impact can be foreseen before research is even funded. Does research impact tend to emerge largely as planned, or are eventual impacts unrecognisable from initial plans?To explore the unexpectedness of impact, we draw on one of the largest research impact assessment exercises in the world - the UK’s Research Excellence Framework. We exploit REF impact traced back to research funding applications, as a dataset of 2,194 case-grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex-ante and ex-post impact.We found impact claims in these case-grant pairs are often congruent with each other, with 76% showing alignment. This indicates that, conditional on REF-inclusion, research delivers impact with largely the same orientation as envisaged at...
Evaluating impact from research: A methodological framework
Research Policy, 2021
Background: Interest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. Aims: This paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. Method: An adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. Results: Recognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research. Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches. Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor amongst many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. Conclusion: Using the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.
Political Studies Review
One of the most extensively discussed requirements introduced in the 2014 REF was impact. In this review piece we focus on the linear and temporal consequences of the REF impact system. We link such consequences to our own research agendas to to provide a sense of empirical richness to the broad concerns that arise from the impact agenda and to highlight the effects of the REF's linear focus, and, crucially, the types of alternative narratives it potentially silences. This "silencing" does not render alternative narratives impossible, but rather makes them difficult to articulate as 'safe' options within the existing framework. We highlight how a focus on direct impact could miss the collective nature of impact endeavours, as well as the broader social and cultural benefits of research, and potentially shape and limit the possible research questions posed within this national system. We conclude by opening up some broader questions for the future of impact raised through the consideration of linearity including the question of 'measurement'.
ISRIA statement: ten-point guidelines for an effective process of research impact assessment
Health research policy and systems, 2018
As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented toward...