Speech and text psychometrics: Identifying suicide risk factors with large language models and acoustic networks

Abstract

Suicide is among the leading causes of death worldwide, with about one death every 11 minutes in the US in recent years. Suicide risk factors such as suicidal ideation, loneliness, and hopelessness have been mainly assessed through rating scales and clinical interviews, which have generally had low accuracy in predicting suicide. One challenge is that suicidal symptoms can be sudden and may not be present during a clinical interview. Another challenge is that many individuals are not receiving regular treatment to be assessed in the first place. Automated risk assessments of suicidal thoughts and behaviors (STBs) and their risk factors could be done more continuously and ecologically outside the clinic by using the speech and text data individuals produce naturally when they use social media, message a friend, journal, or use a crisis counseling service. Clinical assessments could also be enhanced by using speech and text data from therapy sessions and electronic health records. With recent advances in artificial intelligence, this seems promising, but how will we know whether these novel tools are valid? Psychological assessments using speech and natural language processing are generally not evaluated for the types of validity typically expected from screening questionnaires that use rating scales. This may be due to a gap between these fields and psychometrics. This dissertation seeks to bridge this gap. In chapter 1, I use generative AI to automatically build a Suicide Risk Lexicon of 49 risk factors for STBs. This results in an interpretable baseline model that can guarantee flagging certain high-risk language. I validate it with clinician judgments and by demonstrating it can predict imminent suicide risk during crisis counseling sessions. In chapter 2, I try to consolidate Text Psychometrics as the field that studies how to measure psychological constructs in text. I systematically compare a diverse set of models including lexicons, zero-shot deep learning methods, and generative AI on different desirable properties in their ability to predict 13 types of mental health crises. I evaluate clinical utility and introduce content validity test sets as a way to demonstrate whether models detect key expressions of the target variable. In chapter 3, to overcome that fact most approaches to suicide risk assessment use rating scales that simply capture the presence or frequency of STBs, we ask something that is relatively unknown: what are the actual thoughts in suicidal thinking? Participants with STBs answer this question over multiple days, and we compare the validity of categorizing their responses through theory-driven and data-driven methods. Our results reveal that measuring natural language through open-response questions captures suicidal ideation that rating scales miss. In Chapter 4, I analyze the speech patterns of individuals during and after their hospitalizations for suicide risk through daily recordings. I estimate networks of acoustic and psychological variables to capture the dynamic nature of how these variables affect each other over time. I provide a Suicide Speech Model to help explain potential mechanisms by which speech biomarkers could detect changes in STBs. In sum, I provide new methods to measure psychological constructs using speech and text data and lay out novel frameworks to validate these measurements more thoroughly.