Objective, Your Honor: Exploring the Complex Interplay of Risk Assessment Instruments, Judicial Decisions, and Prediction

Abstract

A wide array of algorithmic risk assessment tools are used throughout the United States for the purposes of pretrial reform, promising a more objective, less disparate, and comprehensive approach to pretrial decision making. Using statistical methods on historical data, these tools aim to predict pretrial outcomes of interest such as failure to appear in court and new criminal arrest in order to determine whether to release a defendant into the community before their trial. Yet, using historical data to influence future predictions introduces its own set of biases, raising questions about the potentially disparate outcomes of these tools. This thesis looks closely at the Public Safety Assessment (PSA), one of the most commonly used pretrial risk assessment tools, as a case study to explore the validity, accuracy, and fairness of actuarial risk assessment instruments (RAIs). Previous validation studies have focused primarily on the predictive accuracy and fairness of the PSA. This research goes beyond validation to explore how the presence of the PSA recommendation influences judicial decision-making by using data from a randomized control trial (RCT) conducted in Polk County, Iowa in 2018. Using a variety of statistical analyses including Chi-Squared tests, logistic regression, ROC curves, and fairness evaluations, we find that little to no significant correlations exist between the features used to compute the recommendation scores and the outcomes observed. We also determine that the recommendations generated by this tool have little correlation with the judges’ decisions and that the tool is significant and correlated with outcomes for majority demographics groups, but less so for others. The results of this study highlight the consequences of using historical data to inform prediction, and how even demographic-blind evaluations can lead to disparate outcomes. These results raise questions about the usage of data-driven decision making tools in the criminal justice space, and highlight the limited benefits of supposed criminal justice reform measures, providing a critical analysis of RAIs in the pretrial reform space.