Implementation Challenges and Approaches for Rule-Based and Machine Learning-Based Sepsis Risk Prediction Tools: A Qualitative Study

Abstract

Background: Mandated reporting of sepsis outcomes have led many institutions to implement surveillance software to improve sepsis outcomes. Commercial EMRs, external vendors, and home grown risk prediction tools offer a variety of approaches. Traditional rule-based models draw on the Systemic Inflammatory Response Syndrome (SIRS) criteria while newer predictive models utilize machine-learning (ML) based algorithms to predict sepsis risk. The purpose of this study is to identify challenges and approaches for successful implementation of sepsis surveillance tools. Methods: Semi-structured interviews were conducted with hospital leaders overseeing sepsis clinical decision support implementation at U.S. medical centers (n=14). Participants were recruited via purposive sampling. Interviews probed implementation process, challenges faced, and recommended approaches. Responses were independently coded by two coders with consensus approach and inductively analyzed for themes. Results: Challenges shared by institutions with both SIRS and ML models categorize to technical build, optimization of alerts, workflow integration, tool validation, implementation time, and working with external vendors. Institutions using ML models reported greater difficulty with clinician acceptance of these tools due to user expectation management, limited tool intuitiveness, distrust in the technology, and confusion. Successful institutions report multiple approaches to improving acceptance including user education, expert support, and practitioner-led efforts. Conclusion: In this small but diverse set of hospitals, we found that in addition to the known socio-technical challenges of implementing clinical decision support, less clinically intuitive ML models may require additional attention to user education, support, and expectation management.