Off-Policy Evaluation of Reinforcement Learning in Healthcare

Abstract

Reinforcement learning is a method for learning optimal strategies for tasks which require making sequences of decisions. The ability to make decisions in a manner which balances short versus long term outcomes makes reinforcement learning a potentially powerful tool for planning of treatments in healthcare settings. Unfortunately, traditional reinforcement learning algorithms require random experimentation with the environment, which is usually not possible in healthcare. Nevertheless, reinforcement learning provides tools for evaluating decision making policies from observational data, a subfield known as off-policy evaluation.

In this work, we discuss the main challenges which make off-policy evaluation so difficult when applied to healthcare data, and develop algorithms to improve state of the art methods for performing off-policy evaluation. We describe several algorithms for improving the accuracy and statistical power of existing methods, and conclude by introducing a novel approach to increase the reliability of off-policy evaluation methods by developing an evaluation technique which integrates expert clinicians and their knowledge into the evaluation process.