Topics in randomized experiments: design, modeling, and power

Abstract

There are many statistical concerns in the design and analysis of randomized experiments. This dissertation considers design, modeling, and power in experiments with various types of complexity. Chapter 1 considers design: how to design a well-controlled study using experimental controls. Chapter 2 considers modeling: how to analyze data arising from an experiment that requires complex modeling approaches. Chapter 3 considers power: how to calculate power for multi-level experiments with multiple outcomes. Chapter 1 discusses experimental controls, which are measurements designed to detect systematic sources of unwanted variation in experiments. The chapter introduces clear, mathematically precise definitions of experimental controls using potential outcomes. These definitions provide a unifying statistical framework for fundamental concepts of experimental design from the biological and other basic sciences. Additionally, the chapter outlines the use of controls as tools for researchers to wield in designing experiments and detecting potential design flaws. The chapter concludes with a short example of applying experimental controls to examining the link between air pollution levels and autism. Chapter 2 models patient adherence to a treatment regime. Medication adherence is a widespread problem that can negatively impact a patient’s health. However, existing methods for determining patient adherence are either resource intensive or suffer from poor accuracy. The chapter develops an approach to infer medication adherence rates based on longitudinally recorded health measures that are likely impacted by time-varying adherence behaviors. The procedure employs a modular inferential approach and combines MCMC methods with a sequential Monte Carlo algorithm to infer patient adherence measures. Chapter 3 provides estimation approaches for power, sample size, and minimum detectable effect size. The chapter introduces the PUMP R package as a tool for analysts to estimate statistical power for multi-level randomized controlled trials with multiple outcomes. The PUMP package innovates by allowing for the use of multiple testing procedures, which is not supported by current power calculation software. A simulation-based approach is used to calculate power across a variety of experimental designs, models, definitions of power, and number of outcomes being considered. Analysts can also use PUMP to easily calculate power over a wide range of scenarios to explore sensitivity to underlying assumptions.