Melanie Ehren | UCL Institute of Education (original) (raw)

Papers by Melanie Ehren

Abstract: This article uses a policy scientific approach to reconstruct assumptions underlying th... more Abstract: This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Educational Supervision Act. We show an example of how to reconstruct and evaluate a pro-gram theory that is based on legislation of inspection. The assumptions explain how inspection leads to school improvement. Evaluation of these assumptions is used to predict the (in)effective-ness of this legislation. The article concludes by discussing the advantages and drawbacks of this kind of approach as a starting point for impact and effect studies. As the program theory of inspec-tion includes elements common to other kinds of educational interventions and reforms, these ele-ments can also be considered for other types of program theory.

This book presented the methods and modalities of effective school inspections, summarizing the c... more This book presented the methods and modalities of effective school inspections, summarizing the current evidence base and discussing issues of validity and reliability of school inspections in relation to school effectiveness research. In our book we have focused on inspections of single schools which are currently dominating the landscape of both research and practice. As we discussed in Chaps. 1 and 2, school inspections have a duty in keeping national government informed about the quality and standards achieved in the areas inspected and their frameworks to assess school quality are common to all schools and nationally defined. The current landscape is however changing rapidly and these centralized top down inspection models are becoming increasingly outdated and unfit for their purpose of improving education quality in a more decentralized and fast changing system. This chapter will start with a brief outline of the changes we have seen over the past decades, outlining recent re...

Methods and Modalities of Effective School Inspections, 2016

Educational Assessment, Evaluation and Accountability, 2013

School inspection is used by most European education systems as a major instrument for controllin... more School inspection is used by most European education systems as a major instrument for controlling and promoting the quality of schools. Surprisingly, there is little research knowledge about how school inspections drive the improvement of schools and which types of approaches are most effective and cause the least unintended consequences. The study presented in this paper uses interviews with inspection officials and a document analysis to reconstruct the "program theories" (i.e. the assumptions on causal mechanisms, linking school inspections to their intended outcomes of improved teaching and learning) of Inspectorates of Education in six European countries. The results section of the paper starts with a summary of the commonalities and differences of these six national inspection models with respect to standards and thresholds used, to types of feedback and reporting, and to the sanctions, rewards and interventions applied to motivate schools to improve. Next, the intermediate processes through which these inspection models are expected to promote good education (e.g. through actions of stakeholders) are explained. In the concluding section, these assumptions are critically discussed in the light of research knowledge.

British Journal of Educational Studies, 2006

This article describes a theory about the ambition of most Inspectorates to realise 'school impro... more This article describes a theory about the ambition of most Inspectorates to realise 'school improvement through inspection'. Literature about a number of direct and indirect interventions, such as reciprocity, communication and feedback is used to build a theoretical model stating the relations between working methods of school inspectors, reactions of schools and resulting effects and side effects. Finally two types of inspections strategies are described that can be used in different types of schools. We expect schools with a low innovation capacity and few external impulses to be helped best by a directive approach in which an inspector clearly points to the strong and weak points of the school, the probable causes of their level of functioning, and potential ways for improvement. The inspector should pressure the school to change by making written agreements on how to change and by asking the school to work out these agreements in an improvement plan. A school with a high innovation capacity and strong external impulses is expected to do better with a more reserved inspection approach. Inspectors only need to provide this school with some insight into their strong and weak points.

American Journal of Evaluation, 2005

This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Ed... more This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Educational Supervision Act. We show an example of how to reconstruct and evaluate a program theory that is based on legislation of inspection. The assumptions explain how inspection leads to school improvement. Evaluation of these assumptions is used to predict the (in)effectiveness of this legislation. The article concludes by discussing the advantages and drawbacks of this kind of approach as a starting point for impact and effect studies. As the program theory of inspection includes elements common to other kinds of educational interventions and reforms, these elements can also be considered for other types of program theory.

Starting in January 2008, school inspections are targeted at schools with high risks of failing q... more Starting in January 2008, school inspections are targeted at schools with high risks of failing quality and school boards are now the primary contact in school inspections. In this paper a program theory approach was used to reconstruct the assumptions describing intended effects of these inspection methods. The assumptions were evaluated to predict the (in)effectiveness of the inspection methods.

The effects of school inspections on school improvement have been investigated only to a limited ... more The effects of school inspections on school improvement have been investigated only to a limited degree. The investigation reported on in this article is meant to expand our knowledge base regarding the impact of school inspections on school improvement. The theoretical framework for this research is partly based on the policy theory behind the Dutch Educational School Supervision Act (the latter includes assumptions about how school inspections lead to school improvement). Interviews and a survey with school inspectors gave insight into how school inspectors implement the Supervision Act and how they assess schools, and stimulate schools to improve. The results of ten case studies showed that all schools started to improve after a school visit. The innovation capacity of the school and the school environment do not seem to contribute to school improvement after school inspections. No effects were found on school-improvement processes of the number of insufficient scores that school...

This chapter describes how Inspectorates of Education operationalize different inspection goals (... more This chapter describes how Inspectorates of Education operationalize different inspection goals (control, improvement, and liaison) in their inspection indicator frameworks. The chapter provides an overview and examples of the indicators used across a number of countries and how these are incorporated in inspection frameworks to evaluate and assess schools with the purpose of control, improvement and liaison. We elaborate on potential inspection frameworks to inspect and assess the processes and results of schooling (which includes making expert judgements and –value addedmodels to evaluate school output), and discuss their value and adequateness in the light of recent school effectiveness research. Evaluating the value and adequateness of inspection frameworks and measures is an important condition of valid inspections; a topic we will talk about in more detail in the Chap. 3 .

Methods and Modalities of Effective School Inspections

This chapter identifies and summarizes the findings from international empirical research on the ... more This chapter identifies and summarizes the findings from international empirical research on the impact of school inspections. A systematic literature review was undertaken to describe the empirical evidence on (positive) effects from school inspections in four categories: school staff intentions to improve and reflections on school quality, behavioural change of teachers (and school leaders) to improve effective school and teaching conditions, the improvement and/or introduction of school self-evaluation, and improved student achievement results. Specific examples from different countries will be presented to understand which types of inspections have more or less impact. The final section of this chapter also includes a summary of findings from a review on school inspection impact in low and middle income countries, highlighting how a lack of resources, inefficient management and an organisational structure not adapted to current realities, as well as the limited capacity in schools and in the education system cause an overall lack of impact in these countries. The chapter suggests that it is not only the question if a school inspection may generally lead to rising student test scores or not but also how and why it may lead to different effects on school improvement and student performance in schools. A question we will turn to in Chap. 7.

Methods and Modalities of Effective School Inspections

This chapter introduces four mechanisms of change to explain how school inspections may lead to i... more This chapter introduces four mechanisms of change to explain how school inspections may lead to improvement of school quality and higher student outcomes. schools accepting and using inspection feedback, the setting of expectations around inspections standards and methods, voice, choice and exit of stakeholders to motivate school improvement, and capacity-building of educators and stakeholders to effectively implement improvements. Findings from a systematic literature review will be presented to demonstrate examples of such processes from across a number of studies. The final section will reflect on how these processes are interlinked in creating an overall logic of change across an education system.

Methods and Modalities of Effective School Inspections, 2016

This chapter introduces three categories of unintended consequences from school inspections: 1) i... more This chapter introduces three categories of unintended consequences from school inspections: 1) intended strategic behaviour where schools manipulate the inspection assessment through window dressing, misrepresentation or gaming, 2) unintended strategic behaviour when schools narrow their educational practices as a result of the behaviour of the assessor and/or by the method of working used for the assessment, and 3) other types of consequences, such as stress, anxiety and increased workload. As many inspection systems use standardized student achievement tests to evaluate school output, a fourth category on unintended responses to high stakes testing will also be introduced. The results from a recent systematic literature review will be used to provide evidence of responses in each of the four categories. The review shows that most studies present examples from England and previous case study work from Perryman (2006) will therefore offer more in-depth views of how an English school responds strategically to school inspections. The final section of the chapter provides explanations of the conditions under which such responses may occur.

Professional Development in Education

Japanese Lesson Study (JLS) is a professional development method, involving teachers collaborativ... more Japanese Lesson Study (JLS) is a professional development method, involving teachers collaboratively planning lessons, observing their enactment, then discussing observations of teaching and learning. This paper explores translation of JLS internationally, seeking to understand how and why it is adapted and how an understanding of national culture and implementation paradigms might support translation. We begin by examining evidence on adaptation and challenges of JLS implementation internationally, finding both deviation from the seven components of JLS, and qualitative evidence of perceived challenges to successful implementation. Further we explore two bodies of the literature explaining how and why such adaptations occur. First, implementation science reveals that full fidelity appears not to be amenable to the complexity of education innovations like LS, but that adaptation is fraught with challenges, with no linear pathway. Secondly, Hofstede's and colleagues' dimensions of culture enable us to hypothesise about how Japan's culture might have framed development of JLS, and to predict possible challenges when translated into a host nation. Finally, we hypothesize as to the relationship between adoption of either fidelity or adaptation implementation paradigms, and identified differences between Japan and the host nation's national culture, suggesting avenues for further research which may serve to test hypotheses empirically.

Trust, Accountability and Capacity in Education System Reform

School Leadership & Management

The public education system in Chile was reformed in 2017, introducing networking as a principle ... more The public education system in Chile was reformed in 2017, introducing networking as a principle for governing and improving schools. Between 2018 and 2025, 70 Local Public Education Services (SLE) will replace 345 municipalities as the intermediate level of the new system. SLEs are mandated to lead the formation of networks to support the improvement of schools. This paper presents a qualitative case study exploring how professionals from one SLE in lead the formation of mandated networks. Findings from 20 in-depth interviews were analysed using the network formation model proposed by Segato and Raab ([2019]. “Mandated Network Formation.” International Journal of Public Sector Management 32 (2): 191–206.). School networks were configured by SLE professionals, introducing hierarchical elements into their coordination. SLE professionals acknowledged a tension between guiding networks to produce horizontal and collaborative relationships and coordinating school networks vertically as functional units to support and supervise schools. In the discussion we argue that this tension emerged from SLE professionals’ mindset where prior experiences with vertical and centralised coordination, associated with hierarchical governance, predominated over new forms of horizontal and distributed leadership, associated with network governance.

Journal of Educational Change

Trust and accountability are often positioned as opposites, the argument being that accountabilit... more Trust and accountability are often positioned as opposites, the argument being that accountability is based on distrust and correction of identified deficiencies. Yet, trust is also important in order for accountability to lead to improvement; only when teachers and principals are open about the quality of their teaching and their school can there be a meaningful discussion about change. How can we overcome this dilemma? This paper will address the inextricable interaction between trust and accountability, presenting examples from a study in South Africa of how external control in a setting of distrust can undermine agency and improvement, and how high levels of trust can promote more effective accountability relationships. Our study provides relevant insights into why some education systems are unable to generate, evaluate and scale innovations in learning when a lack of trust and capacity leads to strong opposition to external accountability, and when strong bureaucratic accountability creates further inefficiencies in pressurizing educators across the education system to report and monitor on various aspects of education where these efforts do not actually improve the quality of teaching in the classroom or provide information on good practices.

Journal of Professional Capital and Community

Educational Assessment, Evaluation and Accountability

This paper explores the impact of external accountability on four mechanisms of network-internal ... more This paper explores the impact of external accountability on four mechanisms of network-internal quality control and the properties of (mandated) interorganizational networks. An explorative case study approach examines the external accountability of a newly established educational network (MAT) and how schools and the Trust are held accountable by the English Inspectorate of Education, Ofsted and the Department for Education (DfE), through Regional Schools Commissioners (RSCs). Analysis of documents and interviews were used to reconstruct a sequence of events over a period of five and a half years. Our findings indicate that the combination of accountability activities led to more centralized control over, and monitoring of curriculum, assessment and school improvement. This was accompanied by a pause in the growth of the Trust and the establishment of a regional structure of school collaboration. As accountability is particularly organized around individual school performance (through Ofsted inspections), this limited development of internal mechanisms of quality control around network-level outcomes. A short-term focus dominated the allocation of resources to schools in special measures or with an upcoming inspection visit. In the discussion, an alternative model is presented for network-level accountability and a set of indicators for effective accountability of collaborative arrangements.

Abstract: This article uses a policy scientific approach to reconstruct assumptions underlying th... more Abstract: This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Educational Supervision Act. We show an example of how to reconstruct and evaluate a pro-gram theory that is based on legislation of inspection. The assumptions explain how inspection leads to school improvement. Evaluation of these assumptions is used to predict the (in)effective-ness of this legislation. The article concludes by discussing the advantages and drawbacks of this kind of approach as a starting point for impact and effect studies. As the program theory of inspec-tion includes elements common to other kinds of educational interventions and reforms, these ele-ments can also be considered for other types of program theory.

This book presented the methods and modalities of effective school inspections, summarizing the c... more This book presented the methods and modalities of effective school inspections, summarizing the current evidence base and discussing issues of validity and reliability of school inspections in relation to school effectiveness research. In our book we have focused on inspections of single schools which are currently dominating the landscape of both research and practice. As we discussed in Chaps. 1 and 2, school inspections have a duty in keeping national government informed about the quality and standards achieved in the areas inspected and their frameworks to assess school quality are common to all schools and nationally defined. The current landscape is however changing rapidly and these centralized top down inspection models are becoming increasingly outdated and unfit for their purpose of improving education quality in a more decentralized and fast changing system. This chapter will start with a brief outline of the changes we have seen over the past decades, outlining recent re...

Methods and Modalities of Effective School Inspections, 2016

Educational Assessment, Evaluation and Accountability, 2013

School inspection is used by most European education systems as a major instrument for controllin... more School inspection is used by most European education systems as a major instrument for controlling and promoting the quality of schools. Surprisingly, there is little research knowledge about how school inspections drive the improvement of schools and which types of approaches are most effective and cause the least unintended consequences. The study presented in this paper uses interviews with inspection officials and a document analysis to reconstruct the "program theories" (i.e. the assumptions on causal mechanisms, linking school inspections to their intended outcomes of improved teaching and learning) of Inspectorates of Education in six European countries. The results section of the paper starts with a summary of the commonalities and differences of these six national inspection models with respect to standards and thresholds used, to types of feedback and reporting, and to the sanctions, rewards and interventions applied to motivate schools to improve. Next, the intermediate processes through which these inspection models are expected to promote good education (e.g. through actions of stakeholders) are explained. In the concluding section, these assumptions are critically discussed in the light of research knowledge.

British Journal of Educational Studies, 2006

This article describes a theory about the ambition of most Inspectorates to realise 'school impro... more This article describes a theory about the ambition of most Inspectorates to realise 'school improvement through inspection'. Literature about a number of direct and indirect interventions, such as reciprocity, communication and feedback is used to build a theoretical model stating the relations between working methods of school inspectors, reactions of schools and resulting effects and side effects. Finally two types of inspections strategies are described that can be used in different types of schools. We expect schools with a low innovation capacity and few external impulses to be helped best by a directive approach in which an inspector clearly points to the strong and weak points of the school, the probable causes of their level of functioning, and potential ways for improvement. The inspector should pressure the school to change by making written agreements on how to change and by asking the school to work out these agreements in an improvement plan. A school with a high innovation capacity and strong external impulses is expected to do better with a more reserved inspection approach. Inspectors only need to provide this school with some insight into their strong and weak points.

American Journal of Evaluation, 2005

This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Ed... more This article uses a policy scientific approach to reconstruct assumptions underlying the Dutch Educational Supervision Act. We show an example of how to reconstruct and evaluate a program theory that is based on legislation of inspection. The assumptions explain how inspection leads to school improvement. Evaluation of these assumptions is used to predict the (in)effectiveness of this legislation. The article concludes by discussing the advantages and drawbacks of this kind of approach as a starting point for impact and effect studies. As the program theory of inspection includes elements common to other kinds of educational interventions and reforms, these elements can also be considered for other types of program theory.

Starting in January 2008, school inspections are targeted at schools with high risks of failing q... more Starting in January 2008, school inspections are targeted at schools with high risks of failing quality and school boards are now the primary contact in school inspections. In this paper a program theory approach was used to reconstruct the assumptions describing intended effects of these inspection methods. The assumptions were evaluated to predict the (in)effectiveness of the inspection methods.

The effects of school inspections on school improvement have been investigated only to a limited ... more The effects of school inspections on school improvement have been investigated only to a limited degree. The investigation reported on in this article is meant to expand our knowledge base regarding the impact of school inspections on school improvement. The theoretical framework for this research is partly based on the policy theory behind the Dutch Educational School Supervision Act (the latter includes assumptions about how school inspections lead to school improvement). Interviews and a survey with school inspectors gave insight into how school inspectors implement the Supervision Act and how they assess schools, and stimulate schools to improve. The results of ten case studies showed that all schools started to improve after a school visit. The innovation capacity of the school and the school environment do not seem to contribute to school improvement after school inspections. No effects were found on school-improvement processes of the number of insufficient scores that school...

This chapter describes how Inspectorates of Education operationalize different inspection goals (... more This chapter describes how Inspectorates of Education operationalize different inspection goals (control, improvement, and liaison) in their inspection indicator frameworks. The chapter provides an overview and examples of the indicators used across a number of countries and how these are incorporated in inspection frameworks to evaluate and assess schools with the purpose of control, improvement and liaison. We elaborate on potential inspection frameworks to inspect and assess the processes and results of schooling (which includes making expert judgements and –value addedmodels to evaluate school output), and discuss their value and adequateness in the light of recent school effectiveness research. Evaluating the value and adequateness of inspection frameworks and measures is an important condition of valid inspections; a topic we will talk about in more detail in the Chap. 3 .

Methods and Modalities of Effective School Inspections

This chapter identifies and summarizes the findings from international empirical research on the ... more This chapter identifies and summarizes the findings from international empirical research on the impact of school inspections. A systematic literature review was undertaken to describe the empirical evidence on (positive) effects from school inspections in four categories: school staff intentions to improve and reflections on school quality, behavioural change of teachers (and school leaders) to improve effective school and teaching conditions, the improvement and/or introduction of school self-evaluation, and improved student achievement results. Specific examples from different countries will be presented to understand which types of inspections have more or less impact. The final section of this chapter also includes a summary of findings from a review on school inspection impact in low and middle income countries, highlighting how a lack of resources, inefficient management and an organisational structure not adapted to current realities, as well as the limited capacity in schools and in the education system cause an overall lack of impact in these countries. The chapter suggests that it is not only the question if a school inspection may generally lead to rising student test scores or not but also how and why it may lead to different effects on school improvement and student performance in schools. A question we will turn to in Chap. 7.

Methods and Modalities of Effective School Inspections

This chapter introduces four mechanisms of change to explain how school inspections may lead to i... more This chapter introduces four mechanisms of change to explain how school inspections may lead to improvement of school quality and higher student outcomes. schools accepting and using inspection feedback, the setting of expectations around inspections standards and methods, voice, choice and exit of stakeholders to motivate school improvement, and capacity-building of educators and stakeholders to effectively implement improvements. Findings from a systematic literature review will be presented to demonstrate examples of such processes from across a number of studies. The final section will reflect on how these processes are interlinked in creating an overall logic of change across an education system.

Methods and Modalities of Effective School Inspections, 2016

This chapter introduces three categories of unintended consequences from school inspections: 1) i... more This chapter introduces three categories of unintended consequences from school inspections: 1) intended strategic behaviour where schools manipulate the inspection assessment through window dressing, misrepresentation or gaming, 2) unintended strategic behaviour when schools narrow their educational practices as a result of the behaviour of the assessor and/or by the method of working used for the assessment, and 3) other types of consequences, such as stress, anxiety and increased workload. As many inspection systems use standardized student achievement tests to evaluate school output, a fourth category on unintended responses to high stakes testing will also be introduced. The results from a recent systematic literature review will be used to provide evidence of responses in each of the four categories. The review shows that most studies present examples from England and previous case study work from Perryman (2006) will therefore offer more in-depth views of how an English school responds strategically to school inspections. The final section of the chapter provides explanations of the conditions under which such responses may occur.

Professional Development in Education

Japanese Lesson Study (JLS) is a professional development method, involving teachers collaborativ... more Japanese Lesson Study (JLS) is a professional development method, involving teachers collaboratively planning lessons, observing their enactment, then discussing observations of teaching and learning. This paper explores translation of JLS internationally, seeking to understand how and why it is adapted and how an understanding of national culture and implementation paradigms might support translation. We begin by examining evidence on adaptation and challenges of JLS implementation internationally, finding both deviation from the seven components of JLS, and qualitative evidence of perceived challenges to successful implementation. Further we explore two bodies of the literature explaining how and why such adaptations occur. First, implementation science reveals that full fidelity appears not to be amenable to the complexity of education innovations like LS, but that adaptation is fraught with challenges, with no linear pathway. Secondly, Hofstede's and colleagues' dimensions of culture enable us to hypothesise about how Japan's culture might have framed development of JLS, and to predict possible challenges when translated into a host nation. Finally, we hypothesize as to the relationship between adoption of either fidelity or adaptation implementation paradigms, and identified differences between Japan and the host nation's national culture, suggesting avenues for further research which may serve to test hypotheses empirically.

Trust, Accountability and Capacity in Education System Reform

School Leadership & Management

The public education system in Chile was reformed in 2017, introducing networking as a principle ... more The public education system in Chile was reformed in 2017, introducing networking as a principle for governing and improving schools. Between 2018 and 2025, 70 Local Public Education Services (SLE) will replace 345 municipalities as the intermediate level of the new system. SLEs are mandated to lead the formation of networks to support the improvement of schools. This paper presents a qualitative case study exploring how professionals from one SLE in lead the formation of mandated networks. Findings from 20 in-depth interviews were analysed using the network formation model proposed by Segato and Raab ([2019]. “Mandated Network Formation.” International Journal of Public Sector Management 32 (2): 191–206.). School networks were configured by SLE professionals, introducing hierarchical elements into their coordination. SLE professionals acknowledged a tension between guiding networks to produce horizontal and collaborative relationships and coordinating school networks vertically as functional units to support and supervise schools. In the discussion we argue that this tension emerged from SLE professionals’ mindset where prior experiences with vertical and centralised coordination, associated with hierarchical governance, predominated over new forms of horizontal and distributed leadership, associated with network governance.

Journal of Educational Change

Trust and accountability are often positioned as opposites, the argument being that accountabilit... more Trust and accountability are often positioned as opposites, the argument being that accountability is based on distrust and correction of identified deficiencies. Yet, trust is also important in order for accountability to lead to improvement; only when teachers and principals are open about the quality of their teaching and their school can there be a meaningful discussion about change. How can we overcome this dilemma? This paper will address the inextricable interaction between trust and accountability, presenting examples from a study in South Africa of how external control in a setting of distrust can undermine agency and improvement, and how high levels of trust can promote more effective accountability relationships. Our study provides relevant insights into why some education systems are unable to generate, evaluate and scale innovations in learning when a lack of trust and capacity leads to strong opposition to external accountability, and when strong bureaucratic accountability creates further inefficiencies in pressurizing educators across the education system to report and monitor on various aspects of education where these efforts do not actually improve the quality of teaching in the classroom or provide information on good practices.

Journal of Professional Capital and Community

Educational Assessment, Evaluation and Accountability

This paper explores the impact of external accountability on four mechanisms of network-internal ... more This paper explores the impact of external accountability on four mechanisms of network-internal quality control and the properties of (mandated) interorganizational networks. An explorative case study approach examines the external accountability of a newly established educational network (MAT) and how schools and the Trust are held accountable by the English Inspectorate of Education, Ofsted and the Department for Education (DfE), through Regional Schools Commissioners (RSCs). Analysis of documents and interviews were used to reconstruct a sequence of events over a period of five and a half years. Our findings indicate that the combination of accountability activities led to more centralized control over, and monitoring of curriculum, assessment and school improvement. This was accompanied by a pause in the growth of the Trust and the establishment of a regional structure of school collaboration. As accountability is particularly organized around individual school performance (through Ofsted inspections), this limited development of internal mechanisms of quality control around network-level outcomes. A short-term focus dominated the allocation of resources to schools in special measures or with an upcoming inspection visit. In the discussion, an alternative model is presented for network-level accountability and a set of indicators for effective accountability of collaborative arrangements.