Watch Your Language: Detecting and Quantifying Political Bias In Large Language Models

Abstract

s large language model use begins to infiltrate academic and professional applica- tions, it’s important to understand their biases and idiosyncracies as they could have demonstrable effects on produced work that can reverberate throughout a field. This paper takes the unique approach in measuring aggregate biases through com- paring urtext to summarized text–a distinct technique that allows us to measure the true effects in a natural setting. Our corpus of data comes from oral arguments from Supreme Court cases, other miscellaneous texts associated with these cases and deci- sions, and speeches from the Congressional Record. This data is useful as it contains inherent proxies for political leaning. This data is also unique in how the political bias is less explicit and instead resides in interpretation, allowing the study to probe at the question at whether or not these languages models are conscious of their implicit po- litical biases. From an initial set of results from applying naive techniques and existing methodologies from political science literature, there were measurable differences be- tween urtext and summarized text in regard to their leanings and confirm those behav- iors through a second set of experimental techniques focused on more explicitly politi- cal speech. From these results, the paper also presents a preemptive policy solution for possible future harms presented by these models.