An Approach to Small-Scale Mixed-Methods Experimentation; Transparency for Development, Phase 2

Abstract

Faced with a promising but complex intervention, how can further refinement be evaluated? The typical approach is experimentation. Rigorously evaluated experimentation, for several centuries the province mostly of medicine and related research, is today a reality in a variety of fields of social science and practice. Its primary form, the randomized controlled trial (RCT), stems from its medical roots. By design, RCTs are a highly specialized instrument of inquiry: they seek reliability by focusing on a simple, singular causal relationship. Their relevance typically depends on the relevance of this causal relationship and whether it can be accurately represented and measured in one treatment or in a handful of modifications (or “arms”). But the growth of experimentation has brought randomized controlled trials into evaluations of complex interventions in policy areas like health care, education, water, or sanitation, which often occur at the group or society level, at a large scale, and the implementation of which can take myriad forms. For precisely evaluating the benefits of complex programs, RCTs remain the gold standard, frequently used when, for example, a health or education program is under consideration for scaling, is already being done at large scale but is of uncertain benefit, or is almost perfected save for a very specific design question. But often large-scale randomized controlled trials of complex interventions are inappropriate. For an intervention whose benefit is already widely accepted, they may be too expensive; for one whose benefit is uncertain, they may be too large-scale. When, for whatever reason, evaluation of a complex intervention is important but small-scale experimentation is more appropriate than a full randomized controlled trial, how might further refinement of that intervention be most rigorously and reliably evaluated?