Dopamine in the tail of striatum regulates post-assessment avoidance of stimulus novelty

Abstract

Novelty drives animals’ active sampling of their environment and influences learning. Novel objects evoke characteristic behavioral responses such as approach, retreat, and avoidance of these objects, but how the complex behavioral patterns are regulated remains poorly understood. In this project, I investigated the temporal development of behaviors induced by novelty in freely-moving animals and examined the role of dopamine in the posterior tail of the striatum (TS). I applied machine learning methods to analyze behavioral patterns using deep-learning-based supervised learning (DeepLabCut) and unsupervised learning (MoSeq). The former allowed me to track specific body parts in a video while the latter classifies behaviors into recurring motifs or “syllables” using a hidden Markov model. With these computational tools, I examined how mice in an open-field arena respond to two types of object-centered novelty: stimulus novelty (a novel object that the animal has never experienced) and contextual novelty (a familiar object in a place where the mouse has not experienced the object). My DeepLabCut-based analysis uncovered stereotypic risk assessment behavior characterized by cautious approach bouts where animals keep their tail farther away from the object relative to their nose. Risk assessment behavior was specific to the stimulus novelty condition. It also revealed variability in behavior after the initial risk-assessment period (either engagement with or avoidance of the object). By ablating TS-projecting dopamine neurons, I found that dopamine in TS biased post-risk assessment behavior towards avoidance. Using MoSeq, I found that 2 out of 3 syllables that were enriched in stimulus novelty compared to contextual novelty were risk assessment syllables: one corresponded to cautious approach and one corresponded to retreat. The two risk assessment syllables were both expressed less frequently in the mice with ablation of TS-projecting dopamine neurons, indicating that dopamine in TS plays a role in both approach and retreat behavior. In recording dopamine activity in TS during the novelty task, I found that natural individual variability in behavior corresponded to variability in dopamine activity. Specifically, the amplitude of dopamine response at retreat was negatively correlated with the timing of the transition from risk assessment to post-assessment engagement. Finally I propose a new model for novelty-induced behavior in which behavior is driven by three distinct components: one that facilitates risk assessment, one that facilitates post-risk assessment engagement, and one that suppresses engagement through threat prediction, learned with dopamine in TS. Incorporating dopamine recording data into the model, I show that the observed behavior can be explained by a simple reinforcement learning model with dopamine in TS signaling a threat prediction error. My dissertation work provides both a comprehensive explanation for novelty-related dopamine activity in the posterior tail of striatum as well as expands the toolset for investigating freely-moving behavior in mice.