Regularization in Reinforcement Learning: Equivalences and Novel Methods

Abstract

Reinforcement learning (RL) is a powerful framework for sequential decision-making, with applications ranging from robotics to healthcare. However, in real-world settings, such as mobile health (mHealth), RL faces challenges due to limited data and the need for generalization beyond observed experiences. Regularization -- a set of techniques that constrain model complexity to prevent overfitting and promote generalization-- plays a crucial role in overcoming these challenges. This dissertation critically examines existing RL regularization methods, uncovers novel connections between them, and introduces new approaches inspired by the challenges of mobile health studies. One focus of this work is establishing theoretical connections between existing regularization methods. We prove that discount regularization produces the same optimal policy as a Bayesian prior on the transition function and a penalized Q-function, and is also equivalent to a truncated lambda return. These relationships reveal underlying assumptions and limitations of discount regularization. This work also focuses on introducing novel regularization methods. First we introduce a state-action-specific regularization method that mitigates the limitations of discount regularization uncovered in our analysis. We also propose a novel Bayesian hypothesis testing-based regularization approach that leverages prior study data to improve learning while adapting to differences between the environments of the prior and current studies. This is particularly useful in mobile health applications where feedback is sparse and exploration is limited. Through theoretical analysis and empirical validation, this dissertation advances the understanding of RL regularization methods and introduces new techniques that enhance generalization in data-constrained environments. These contributions provide a principled foundation for improving RL applications in healthcare and beyond.