External Validity in the Context of RCTs: Lessons from the Causal Explanatory Tradition (original) (raw)
Related papers
In randomized control trials (RCTs) in the education field, the complier average causal effect (CACE) parameter is often of policy interest, because it pertains to intervention effects for students who receive a meaningful dose of treatment services. This report uses a causal inference and instrumental variables framework to examine the identification and estimation of the CACE parameter for two-level clustered RCTs. The report also provides simple asymptotic variance formulas for CACE impact estimators measured in nominal and standard deviation units. In the empirical work, data from ten large RCTs are used to compare significance findings using correct CACE variance estimators and commonly-used approximations that ignore the estimation error in service receipt rates and outcome standard deviations. Our key finding is that the variance corrections have very little effect on the standard errors of standardized CACE impact estimators. Across the examined outcomes, the correction term...
Critical Studies in Education, 2024
This paper seeks to raise awareness among educational researchers and practitioners of some significant weaknesses and internal contradictions of randomised control trials (RCTs). Although critiques throughout the years from education scholars have pointed to the detrimental effects of this experimental approach on education practice and values, RCTs are considered the gold standard for assessing the impact of education policies and interventions. By drawing on the approach of immanent critique, we elucidate substantial argumentative gaps between the assumptions and applications – that is, between the theory and reality – of RCTs in empirical research. This kind of analytic exercise complements existing critiques from outside the experimental discourse based on moral and epistemic principles. The present paper, in contrast, contributes to the literature by highlighting internal limitations and contradictions that can be seen by probing the logic espoused by those who are proponents of RCTs. In fleshing out our argument, we seek to encourage more informed and critical engagement by educators, policymakers, and researchers, among other stakeholders, when they are confronted with proposals for education programmes and reforms supported by findings from RCTs.
Avoiding Bias in Randomised Controlled Trials in Educational Research
British Journal of Educational Studies, 2003
Randomised controlled trials (RCTs) are often seen as the 'gold standard' of evaluative research. However, whilst randomisation will ensure comparable groups, trials are still vulnerable to a range of biases that can undermine their internal validity. In this paper we describe a number of common threats to the internal validity of RCTs and methods of countering them. We highlight a number of examples from randomised trials in education and health care where problems of execution and analysis of the RCT has undermined their internal validity. However, awareness of these potential biases can lead to careful planning to avoid or reduce their occurrence. If good quality randomised trials are to inform policy and practice in education then rigorous trials need to be designed that are the least susceptible to threats to their validity.
The Need for Randomised Controlled Trials in Educational Research
British Journal of Educational Studies, 2001
This paper argues for more randomised controlled trials in educational research. Educational researchers have largely abandoned the methodology they helped to pioneer. This gold-standard methodology should be more widely used as it is an appropriate and robust research technique. Without subjecting curriculum innovations to a RCT then potentially harmful educational initiatives could be visited upon the nation's children.
Randomised controlled trials in educational research: Ontological and epistemological limitations
Evidence-based practice (EBP) is positioned as an inherent good in the medical and clinical literature, and not without reason. It relies on the integration of research evidence, clinical expertise and patient preferences, [1] and has become a foundation on which health systems are built and improved. There is however, a growing body of literature that takes a more critical stance towards EBP, especially when practitioners make assumptions about what constitutes 'the evidence' and how the data informing that evidence are gathered. [2,3] The evidence upon which EBP is premised is usually derived from experimental research conducted in professional disciplines that are firmly rooted in the positivist paradigm; the research method most closely associated with this is the randomised controlled trial (RCT). RCTs are quantitative, controlled experiments in which the effect of an intervention can be determined more objectively than by observational studies. [4] There is no doubt that the method has utility in determining cause-effect relationships between medical treatments and patient outcomes, making it a powerful design for intervention studies with the objective of determining the influence of one variable on another. [5] In an educational context it may initially seem reasonable to expect that an experimental design could determine the effect of a teaching intervention that aims to improve student learning. The argument is that by using randomisation to average out the differences between students, one would be able to demonstrate which teaching and learning strategies lead to the largest effect sizes. These data, presumed to be free of subjective interpretation, could then inform policies that drive the implementation of effective teaching interventions. [6] However, if we assume that the evidence gathered via experimental research provides insight into an objective reality, we must take a position on teaching and learning that is at odds with our best explanations for how learning happens. Therefore, if we want to use RCTs in educational research, we must assume that there is a cause-effect relationship in the teaching and learning interaction that can be objectively measured. In this article we argue that RCTs are an inappropriate design choice for educational research because they force one to assume ontological and epistemological positions that are at odds with theoretically informed perspectives of learning. RCTs in educational research We begin by highlighting the biased way in which RCTs are positioned relative to other forms of research endeavours, explicit in the language employed by RCT proponents. Goldacre [7] suggests that 'Evidence-based interventions in teaching could … replace the current system where untested methods are passed to teachers through a variety of often dubious outlets' , and 'We need a slow revolution that puts evidence at the heart of teaching'. Torgerson [8] asserts that RCTs are the ultimate expression of evaluative research, referring to 'the importance and supremacy of the RCT' , and expresses concern that educational research tends to rely on 'manifestly inferior' qualitative methods. It seems clear that those who most strongly advocate the use of RCTs in education have an inherent bias against other methods of data collection, strongly positioning themselves within a positivist interpretation of reality. This does not mean that RCTs and other forms of experimental research are not valuable tools in the repertoire of the researcher; randomisation is rightly considered an appropriate design choice in clinical trials. By controlling for the influence of all other variables between groups – through the assumed equal distribution of those differences in a large enough randomised sample – any differences in outcome can be more confidently attributed to the intervention. [5] As the RCT is a powerful tool successfully used in medical and clinical research, some have suggested that it should therefore underpin all 'good' research regardless of context'. [9] However, one cannot assume that RCTs can provide more – and better – evidence, which inevitably leads to improvements in education. [6] RCT proponents hope that these trials can do for educational researchers what they have done for medical researchers, i.e. provide clear-cut answers around the relative benefits of one intervention over another. [7] RCTs are presented as a gold standard, able to determine 'the truth' by simplifying and Randomised controlled trials (RCTs) are a valued research method in evidence-based practice in medical and clinical settings because they are associated with a particular ontological and epistemological perspective that is situated within a positivist world view. It assumes that environments and variables can be controlled to establish cause-effect relationships. However, current theories of learning suggest that knowledge is socially constructed, and that learning occurs in open systems that cannot be controlled and manipulated as would be required in a RCT. They recognise the importance and influence of context on learning, which positivist research paradigms specifically aim to counter. We argue that RCTs are inappropriate in education research because they force one to take up ontological and epistemological positions in a technical rationalist framework, which is at odds with current learning theory.
Quantifying ‘promising trials bias’ in randomized controlled trials in education
2020
Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect, if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results. We conduct a retrospective design analysis to quantify this bias across 23 promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trials bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently...
The RCTs-Only Doctrine: Brakes on the Acquisition of Knowledge?
2007
The debate about how to rigorously gauge the outcomes of policies, programs, and other initiatives continues to rage around the globe. One camp pushes the notion that randomized controlled trials (RCTs) are always the preferred way to go, using scoring schemes such as the “Maryland Scale,” which yields a Scientific Methods Score that places RCTs at the top for rigor (Sherman et al., 1998). The other camp in the debate argues that methods should be matched to the situation.
Partially Nested Randomized Controlled Trials in Education Research: A Guide to Design and Analysis
2014
This paper, commissioned by the National Center for Education Research, provides readers with an introduction to PN-RCTs and ways to design and analyze the results from them. This paper was written primarily for applied education researchers with introductory knowledge of quantitative impact evaluation methods. However, those with more advanced knowledge will also benefit from some of the technical examples and appendices.