Reinforcement Learning for Healthcare: From Model Development to Deployment

Abstract

Reinforcement Learning (RL) is a subfield of Machine Learning (ML) focuses on how agents learn to make optimal decisions over time to achieve particular goals. Given its emphasis on sequential decision-making, RL is amenable to the healthcare setting, in which clinicians often need to adaptively change treatment plans based on patients’ current physiological conditions. The deployment of trained RL models in real-world settings poses various difficulties. For example, the data collection phase may not be properly designed, so that the data cannot be used for downstream statistical analyses. As another example, the model is trained with mildly ill patients but is used on severely ill patients. As a last example, the clinicians do not trust the model due to its lack of transparency, and thus are not willing to use it. To help practitioners overcome those challenges, previous works propose a general framework that decomposes the complex ML workflow into individual stages. They also discuss the potential concerns and key factors that practitioners should consider at each stage of the pipeline. Applying this framework to RL models, which are a type of ML models, is not easy due to the distinct features of RL for healthcare (e.g., access to observational data only, poorly-defined objectives, infeasibility of running the model on target patients, etc). In this work, we take a further step towards facilitating the deployment by practitioners of RL in healthcare, using the general ML pipeline. We start by providing a systemic review of an ML pipeline that can help automate RL workflow, consisting of four steps: data preparation, model development, model assessment, and model deployment. We then discuss different challenges that arise when following each step of the pipeline to deploy RL into the healthcare system. We then focus on three specific challenges of model development and assessment and present solutions to address each of them. We conclude by discussing some open problems of current RL works in the healthcare domain with a view forward.