Theory-based reinforcement learning: A computational framework for modeling human inductive biases in complex decision making domains

Abstract

Throughout our day-to-day lives we are faced with a wide assortment of complex environments to master. Whether we have to work out the functions of a new device, the transportation options of a new city, or the norms of a new workplace, we are often able to extract the hidden principles of a new domain with surprisingly little instruction or exploration. This kind of learning would be im- possible without structured representations to compact the complexity of the world and inductive biases to help narrow down the set of plausible explanations. Yet while we have a solid understanding of many of the representations and biases that guide human learning in relatively simple domains, our knowledge of how humans make sense of the deeply interconnected systems of the real world is less precisely specified. In this work we study how humans learn to play video games in order to develop a clearer picture of the cognitive mechanisms that guide our behavior in more realistically complex domains. Complex task domains pose a key challenge for computational cognitive scientists. Models that precisely describe human decision making in simple tasks do not necessarily scale well to more com- plex domains. For tasks like video games, a single task instance (i.e., a game) can involve far more po- tential states and actions than can be reasonably stored in memory. Yet humans are often capable of generalizing knowledge of a few example states to choose rewarding actions from anywhere in these vast state spaces. In Chapter 1, we describe a novel task that highlights this kind of generalization in human decision making and demonstrate how the task representations in several existing modeling approaches fail to capture this behavior. We then integrate a broad literature on structured repre- sentation in human cognition to propose a new framework for modeling human decision making, which we refer to as “theory-based reinforcement learning,” that better accounts for human behav- ior in complex domains. Learning representations that are rich enough to capture complex environments introduces a second challenge. Any space of task representations large enough to capture the variety of envi- ronments that humans are capable of learning likely contains a plethora of representations capable of explaining the same observations. Identifying the most plausible of these theories thus requires inductive biases to better direct learning behavior. In the second portion of this dissertation, we ex- plore the kinds of inductive biases that guide learning by extending our theory-based framework of human decision making to incorporate structure learning. Chapter 2 presents a formal overview of theory-based RL, a novel computational framework for studying the inductive biases underlying human behavior in more realistically complex task domains. In particular, we demonstrate how this framework can be used to formally incorporate both syntactic biases about the structure of theory- based representations and semantic biases about the content of those representations. Finally, Chapter 3 provides an example of how theory-based RL can be used to empirically evalu- ate the inductive biases that humans use to learn in complex domains. We find evidence that human task learning is consistent with a range of syntactic and semantic biases, and that incorporating these biases into theory-based RL agents results in more human-like learning behavior. We then conclude with a brief discussion of how our framework can be further extended to help us understand the biases that guide human learning in day-to-day life.