Relationship of striatal dopamine to reward history and action during probabilistic decision making

Abstract

Adaptive behavior requires animals to flexibly switch between actions to obtain reward. It is unknown how the brain mediates transitions when the relative value between actions is unclear. To study these processes, we examined decision-making strategies employed by mice in a probabilistic two-armed bandit task (2ABT). We found that switching behavior in this task is modulated by previous action and outcome history, producing two distinct behavioral regimes. In one, given certain trial histories, mice nearly deterministically repeat their previous actions. In contrast, following other experiences, mice appear to choose actions randomly. We were able to capture the trial history dependence of switching behavior through several mathematically equivalent models, namely a recursively formulated logistic regression, an action-value reinforcement learning algorithm, and sticky Bayesian inference. Each model required a combined representation of action-outcome history, a perseverative bias, and a stochastic action policy.

Using externally observable variables alone, it was necessary that our behavior models capture the probabilistic nature of switching to recapitulate the structure of mouse trial-to-trial decision making. However, we examined whether accessing neural activity would reveal internally determined signatures of switching embedded in the action selection circuitry. We collected fiber photometry recordings of dopamine release into ventrolateral striatum (VLS) in a head-fixed 2ABT in which mice lick both to express decisions and evaluate outcomes. We have previously shown that lick direction and lick action outcome (i.e., the presence or absence of water) are dependent on and encoded in neural activity in VLS. We found evidence that dopamine activity in VLS is modulated by the same set of features necessary to model the behavioral strategy, producing responses to reward outcome consistent with reward prediction error (RPE) theory from reinforcement learning. Other components of VLS dopamine activity produce patterns that are distinct from RPE-encoding and are perhaps better aligned with features such as motivation or salience. Finally, we fail to connect modulation of VLS dopamine activity with future switching behavior through simple linear discrimination, despite the prior evidence that its encoding profile contains features behaviorally predictive of switching, suggesting an anatomical separation between evaluation and action selection.