Explaining Variation in Findings From Efficacy and Effectiveness Studies for English Reading Interventions for English Learners (original) (raw)

2019, Journal of Research on Educational Effectiveness

Beginning in 2002, researchers developed, implemented, and evaluated the efficacy of an English reading intervention for first-grade English learners using multiple randomized control trials (RCTs). As a result of this efficacy work, researchers successfully competed for an IES Goal 4 effectiveness study using the same intervention. Unlike the efficacy studies, the findings from the effectiveness study begun in 2011 did not yield significant differences in favor of the treatment. We investigated the identification and pre-intervention performance levels of students in efficacy and effectiveness RCT studies and provide data and compare samples from these studies as one means of explaining variation in findings. Findings indicate that pre-intervention performance levels are higher for students in effectiveness RCT studies. This article serves as an exemplar for other effectiveness studies that may not replicate findings from previous efficacy trials. KEYWORDS program evaluation equating effectiveness study Today's classrooms reflect an increasingly diverse student population including students who may understand, speak, read, and write in English with varying levels of proficiency. Some may struggle in school while others thrive. English learners (ELs) may come to school with little formal schooling in their first language, experience in basic conversational English, or instruction in understanding and using academic English. In the classroom, this variability is demonstrated in students' receptive and expressive language skills in English, as well as differences in their reading, speaking, and writing in English. Recognizing the importance of early literacy supports for ELs, researchers developed, implemented, and evaluated the efficacy of both an English (Proactive) and Spanish (Proactiva) reading intervention for first grade ELs using multiple randomized control trials (RCTs; Vaughn et al, 2006a; 2006b, 2006c). As a result of this efficacy work, researchers successfully competed for an IES Goal 4 effectiveness study using the same intervention. Unlike the efficacy studies, the findings from the effectiveness study did not yield significant differences in favor of the treatment (Kulesz, Francis, Carlson, & Vaughn, 2017). 1 Failure of