Person:

Kim, James

The authors conducted a cluster‐randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence‐based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end‐of‐year comprehension lessons and materials sent to students’ homes in the summer months. In this study, 27 high‐poverty elementary schools (75–100% eligibility for free or reduced‐price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments, or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade‐level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed 0.12 standard deviation higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions and that teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence‐based summer literacy program.

Theory and empirical work suggest that teachers’ social capital influences school improvement efforts. Social ties are prerequisite for social capital, yet little causal evidence exists on how malleable factors, such as instructional management approaches, affect teachers’ ties. In this cluster-randomized trial, we apply a decision-making perspective to compare a literacy intervention managed under a “fidelity-focused” approach, in which teachers were expected to implement researcher-designed procedures faithfully, versus a “structured adaptive” approach, in which teachers collaboratively planned program adaptations. In the short term, the adaptive approach increased teachers’ accessing of intervention-related social capital, but decreased their accessing of social capital unrelated to the intervention. Short-term effects varied based on participants’ role in the intervention. No group differences were found on social capital measures one year later, suggesting that the structured adaptive approach did not make teachers more likely to form ties that would be useful outside of the intervention.

This study employs a sequential multiple assignment randomized trial (SMART) design to develop an adaptive intervention with personalized print and digital content for kindergarten to Grade 2 children (n = 273). In Stage 1, we ask whether it is better for children to receive an adaptive intervention based on (a) 10 conceptually coherent texts or (b) 10 leveled texts on a range of topics. In Stage 2, we ask how best to encourage nonresponding children. Findings indicate that children who received either conceptually coherent texts or leveled texts performed similarly on reading comprehension posttests, while augmenting and intensifying follow-up with gamification of the app and text messages to parents improved comprehension outcomes for nonresponders. Descriptively, we find that only 26% (n = 71) of parents accessed the app, highlighting the need for better implementation procedures to increase take up of app-based digital activities.

This study investigated the effectiveness of the Model of Reading Engagement (MORE), a content literacy intervention, on first graders’ science domain knowledge, reading engagement, and reading comprehension. The MORE intervention emphasizes the role of domain knowledge and reading engagement in supporting reading comprehension. MORE lessons included a 10-day thematic unit that provided a framework for students to connect new learning to a meaningful schema (i.e., Arctic animal survival) and to pursue mastery goals for acquiring domain knowledge. A total of 38 first-grade classrooms (N = 674 students) within 10 elementary schools were randomly assigned to (a) MORE at school (MS), (b) MORE at home, (MS-H), in which the MS condition included at-home reading, or (c) typical instruction. Since there were minimal differences in procedures between the MS and MS-H conditions, the main analyses combined the two treatment groups. Findings from hierarchical linear models revealed that the MORE intervention had a positive and significant effect on science domain knowledge, as measured by vocabulary knowledge depth (effect size [ES] = .30), listening comprehension (ES = .40), and argumentative writing (ES = .24). The MORE intervention effects on reading engagement as measured by situational interest, reading motivation, and task orientations were not statistically significant. However, the intervention had a significant, positive effect on a distal measure of reading comprehension (ES = .11), and there was no evidence of Treatment × Aptitude interaction effects. Content literacy can facilitate first graders’ acquisition of science domain knowledge and reading comprehension without contributing to Matthew effects.

Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist within outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term “item-level” HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions.

Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.

We developed a sustained content literacy intervention that emphasized building domain and topic knowledge from Grade 1 to Grade 2 and evaluated transfer effects on students’ reading comprehension outcomes. The Model of Reading Engagement (MORE) intervention emphasizes thematic lessons that provide an intellectual framework for helping students connect new learning to a general schema (i.e., how scientists study past events). A total of 30 elementary schools (N = 2,952 students; N = 144 teachers) were randomly assigned to a treatment or control group. Over 12 months, the treatment group students participated in (a) spring Grade 1 thematic content literacy lessons in science and social studies followed by wide reading of thematically related informational texts during summer, and (b) fall to spring Grade 2 thematic content literacy lessons in science. After implementation of Grade 1 thematic content literacy lessons and summer support for reading, treatment group students experienced smaller summer losses on a domain-general measure of reading than control group students. Following the sustained implementation of thematic content literacy lessons in science through Grade 2, treatment group students also outperformed their control group peers on a science content reading comprehension outcome (ES = .18). Furthermore, we found transfer effects on science content reading comprehension that varied by passage-item type (near-, mid-, and far-transfer passages determined by the inclusion and number of directly taught words in passages). A sustained content literacy intervention that aligns content and instruction across grades can help students transfer knowledge to novel reading comprehension tasks.

In a common approach for scaling up effective educational practice, schools adopt evidence based programs to be implemented with fidelity. An alternative approach assumes that programs should be adapted to local contexts. In this randomized trial of a reading intervention, we study a scaffolded sequence of implementation, in which schools first develop proficiency by implementing the program with fidelity before implementing structured adaptations. We find evidence supporting the scaffolded sequence: a fidelity-focused approach promoted learning and instructional change more so for teachers inexperienced with the intervention, while a structured adaptive approach was more effective for teachers experienced with the intervention. Students benefited more from the structured adaptive approach, but only when their teacher had prior experience with the fidelity-focused version of the program.

In a randomized trial that collects text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by human raters. An impact analysis can then be conducted to compare treatment and control groups, using the hand-coded scores as a measured outcome. This process is both time and labor-intensive, which creates a persistent barrier for large-scale assessments of text. Furthermore, enriching one’s understanding of a found impact on text outcomes via secondary analyses can be difficult without additional scoring efforts. The purpose of this article is to provide a pipeline for using machine-based text analytic and data mining tools to augment traditional text-based impact analysis by analyzing impacts across an array of automatically generated text features. In this way, we can explore what an overall impact signifies in terms of how the text has evolved due to treatment. Through a case study based on a recent field trial in education, we show that machine learning can indeed enrich experimental evaluations of text by providing a more comprehensive and fine-grained picture of the mechanisms that lead to stronger argumentative writing in a first- and second-grade content literacy intervention. Relying exclusively on human scoring, by contrast, is a lost opportunity. Overall, the workflow and analytical strategy we describe can serve as a template for researchers interested in performing their own experimental evaluations of text.