Biological constraints and mechanisms for reinforcement learning

Abstract

This thesis concerns the application of the theory of reinforcement learning (RL) to neuroscience. Like many aspects of cognition, learning can be studied with a wide range of approaches from the computational, network and systems level, to the cellular and biophysical levels. This breadth of dis- ciplines makes the integration of findings challenging, causing critical insights to be missed. This the- sis addresses two challenges in integrating RL theories with the mechanistic study of learning in the brain. In Chapter 1 we examine how biological factors in the brain might facilitate existing theories related to risk sensitivity in RL. In Chapter 2, we use neural network models to generate hypotheses about how state representations could be implemented in the brain, and experimentally test them. We begin by highlighting the connection between dopamine’s role in RL and its underlying bi- ological mechanisms– the modulation of plasticity in the basal ganglia. We apply key insights from this experimental research to develop a biologically-informed RL model that aligns with dopamine- dependent plasticity rules. In this model we address the challenge that, while computational models estimate value predictions objectively, animals exhibit biased estimates and varied risk sensitivities. Our model highlights a previously overlooked factor—the modulation of dopamine receptor sensi- tivity by baseline dopamine levels—which naturally leads to risk sensitivities and biases in value learn- ing. This model not only explains experimental results that previous models failed to capture, it also provides a potential explanation to the persistent biases in value predictions seen in mental health disorders such as depression, Parkinson’s disease and addiction. We then tackle another challenge: while traditional models typically assume fully observable states, in natural environments states are often hidden and must be inferred. This leads to the need to esti- mate a probability distribution over possible states – called belief states. Computing belief states can become intractable in complex environments, raising questions about how the brain achieves this. Our previous modeling work showed that training recurrent neural networks to predict value in a task with hidden states leads to the emergence of network dynamics that resemble belief states, with- out being explicitly instructed to infer them. We therefore hypothesized that belief states could be instantiated through neuronal dynamics in the brain developed by optimizing value predictions. We test this hypothesis by recording population neural activity in frontal cortical regions in mice trained in the same task. Our findings show that neuronal dynamics consistent with belief states are indeed present in frontal regions in well-trained mice, but absent in primary motor cortical areas or during early stages of learning. Together, these results indicate that population neural dynamics facilitate the representation of belief states, and that reinforcement learning—both in the brain and artificial neural networks— refines these representations to enhance the accuracy of value predictions.