Clinical Decision Support Rules Analysis: Methodology of Evaluating Success

Abstract

In medicine there are several recommendations for routine monitoring. These exist both for medications commonly used by clinicians and for age relevant screening lab tests. Clinical informatics system use decision support rules to aide physician in implementing these recommendations and supporting their execution. This is particularly helpful for complex or time-consuming rules, but when poorly done encumbers care and could potentially be harmful. Clinical decision support (CDS) rules implementation is measured by the number of rules implemented and not by any measure of value of those rules. Therefore, the most common complaint amongst clinicians is “I see too many alerts”. In this paper we discuss a methodology to move the conversation away from quantity to quality. Today’s CDS groups design is focused on clinical impact and modifications to the clinical process needed to accomplish the CDS goal. What is not generally discussed is impact to the clinician and the criteria that should be used to determine success. We propose a methodology based on work by Adam Wright and and Allison B. McCoy to add the metrics needed to assess success of the rule to the design process. We have focused on a five stage framework for CDS rule evaluation . The five stages are as follows: 1. Rule views, 2. Override rate, 3. Response to alert, 4. Intermediate outcome, and 5. Clinical outcome. We have demonstrated two key examples of this methodology. We generate the metrics and visualizations for this framework by analyzing the enormous amount of transactional data generated by modern CDS systems. The log data contains information necessary to evaluate rule view, override, and action taken. We extend this to intermediate markers and ultimately clinical outcomes based on the design intent of the rule using clinical data from the electronic health record (EHR).