The Point That Makes a Difference: Interpreting Influence in Linear Off-Policy Evaluation

Abstract

Evaluating how a decision-making policy will perform without actually running it is a core challenge in reinforcement learning, especially when working with pre-collected data. This task, known as offline policy evaluation (OPE), becomes particularly difficult when the data doesn’t fully cover the situations the new policy might encounter, making certain datapoints disproportionately important. This thesis explores how individual datapoints influence value estimates under two value-based OPE algorithms—Least-Squares Temporal Difference (LSTD) and Fitted Q-Evaluation (FQE)—using linear function approximation. We derive exact leave-one-out (LOO) estimates for both methods and, through experiments on structured environments, show how these influence measures can be used to understand convergence issues and offer insight to the reliability of OPE estimates. In particular, we show that FQE convergence provides a strong signal for when LSTD estimates can be trusted and offer an interpretation through influences for cases of FQE divergence. We further introduce a clipped version of FQE that bounds predicted values, reducing variance while maintaining stability, and provide exact LOO estimates for it as well.