Dopamine and the Neurobiological Foundations of Learning and Performance

Abstract

The roles of phasic (fast-timescale) and tonic (slow-timescale) dopamine have been the subject of great speculation. While conventional models map phasic dopamine activity to 'reward prediction errors' (RPEs), or teaching signals that update an animal's estimates of rewards, more recent results suggest it reflects state values, on the basis that dopamine signals ramp up as a reward is approached in some tasks. One influential model posits that tonic dopamine signals the average reward in a given context, whereas Bayesian models assert that it controls the precision of stimulus encoding. Still other work has argued in favor of tonic dopamine controlling animals' tendencies to time precisely or to be exploitative, impulsive, or risk-seeking, although the directionality of these effects itself has been debated, with contradictory results obtained under seemingly similar experimental paradigms.

I show in this work that an unbiased but noisy agent will produce RPE ramps when sensory feedback is available throughout the task. This result captures the different fast-timescale dopamine behaviors and predicts a previously unobserved non-monotonic pattern, which is elicited experimentally and termed dopamine 'bumps.' I then generalize the RPE hypothesis to learning parameters beyond reward magnitudes and show that this generalized hypothesis reconciles a seemingly conflicting literature on dopamine's effects on interval timing. I then reconcile the two theories of tonic dopamine under the unifying framework of 'rational inattention,' which captures new findings that cannot be explained by either theory alone. By viewing animal behavior as a reward maximization problem subject to noise, I show that dopamine does not necessarily make animals more exploitative, impulsive, or risk-seeking at all, and rather that animals simply seek to maximize their rewards given the information they have access to. In sum, this work suggests that a common computational repertoire underlies the seemingly heterogeneous roles of dopamine.