Using Staff Ratings to Measure Variation in and Causal Relationships with Quality of Working Conditions in K-12 Schools

Abstract

A growing number of state and district education agencies collect staff ratings (SR) using multiple-choice item questionnaires to evaluate the quality of working conditions (QWC) in schools. Agencies often report school-level metrics derived from these data to the public, and some use these to inform decisions about practice and policy. Although these metrics may offer novel and useful information, there is limited empirical evidence for the interpretation of SR as indicators of QWC, and it is unclear whether the dimensions of QWC captured by these questionnaires are causally related to a range of important outcomes for students. My dissertation comprises two studies drawing on rich administrative data collected between 2014-15 and 2016-17 in California’s Fresno, Long Beach, and Santa Anna Unified School Districts. In my first study, I utilize Item Response Theory to evaluate the precision and transparency of scoring methods for SRQWC. I find that a polychotomously-scored information-weighted metric offers substantially greater precision than the dichotomously-scored equally-weighted metric typically favored by education agencies for its transparency. In my second study, I isolate within-school and within-student variation over time to estimate the potentially-causal relationship between SRQWC and student educational outcomes. Over time, I find that within-school changes in SRQWC are positively related to within-student changes in academic achievement and social emotional learning, and negatively related to within-school changes in the rate of disciplinary incidents. Collectively, these analyses contribute to a growing body of empirical literature on organizational context and teacher effectiveness by suggesting that improvements in educators’ quality of working conditions are associated with gains in student academic, social-emotional, and behavioral outcomes.