A Three-Study Examination of Test-Based Accountability Metrics

Abstract

Recent state and federal policy initiatives have led to the development of a multitude of statistics intended to measure school performance. Of these, statistics constructed from student test scores number among both the most widely-used and most controversial. In many cases, researchers and policymakers alike are not fully aware of the ways in which these statistics may lead to unjustified inferences regarding school effectiveness. A substantial amount of recent research has attempted to remedy this, although much remains unknown.

This thesis seeks to contribute to these research efforts via three papers, each examining how a commonly-employed accountability statistic may be influenced by factors unrelated to student proficiency or school effectiveness. The first paper demonstrates how the discrete nature of test scores leads to biased estimates of changes in the percentage of “proficient” students between any two given years and examines estimators that provide better recovery of this parameter. The second paper makes use of a state-wide natural experiment to show that a change in testing program, from paper-and-pencil to computer-adaptive, may cause apparent changes in achievement gaps even when relative student proficiencies have remained constant. The third paper examines “growth-based” accountability metrics based on vertically-scaled assessments, showing that certain types of metrics based on gain scores can be modeled via nonlinear transformations of the underlying vertical scale. It then makes use of this result to investigate the potential magnitude of impacts of such transformations on growth-based school accountability ratings.