Building Evidence for Effective Education: Essays in Quantitative Research Methods

Abstract

This dissertation consists of three papers about quantitative research methods in education. Each paper is a response to potential limitations of the current approach to building evidence for effective education – an approach that is dominated by Randomized Controlled Trials of school programs, with mathematics and English achievement as primary outcomes. The first two papers quantify the risks of two important sources of statistical bias: selection bias from nonrandomized treatments, and attrition bias. Both papers use data from a large set of randomized trials in the UK, embedded in a census of English schools. The first paper finds no evidence of substantial selection bias in nonexperimental evaluations of primary school programs. In 42 comparisons between treatment effects in randomized and non-randomized evaluations, the distribution of underlying selection bias is centered around zero, with a mean absolute value of 0.03σ. The second paper examines the issue of attrition, arguably the biggest threat to the internal validity of field experiments. The data provide a unique glimpse into the post-intervention outcomes of children who dropped out of RCTs. We find that the typical magnitude of attrition bias is 0.015σ, with no estimate greater than 0.034σ. This suggests that, in practice, the risk of attrition bias is limited. The third paper turns to the challenge of broadening the set of outcomes that are available to researchers interested in education and skills. We conduct an experiment which uses a novel approach to measure individual contributions to group success. We use repeated random assignment, combined with conditioning on individual task-specific skill, to measure whether participants are ‘Team Players’: people who help a group succeed, regardless of their skill in the task at hand. The experiment demonstrates that Team Players score significantly higher on a well-established measure of social intelligence, but do not differ across other dimensions. Our approach may prove useful in developing measures of social skill and teamwork, thereby broadening the set of interventions and outcomes researchers can investigate in their efforts to improve education systems.