Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target

Transcription

1 Reward and encoding in terminals of midrain dopamine neurons depends on striatal target Nathan F Parker, Courtney M Cameron, Joshua P Taliaferro, Junuk Lee, Jung Yoon Choi, Thomas J Davidson, Nathaniel D Daw & Ilana B Witten 6 Nature America, Inc. All rights reserved. Dopaminergic (DA) neurons in the midrain provide rich topographic innervation of the striatum and are central to learning and to generating actions. Despite the importance of this DA innervation, it remains unclear whether and how DA neurons are specialized on the asis of the location of their striatal target. Thus, we sought to compare the function of supopulations of DA neurons that target distinct striatal suregions in the context of an instrumental reversal learning task. We identified key differences in the encoding of reward and in dopamine terminals in dorsal versus ventral striatum: DA terminals in ventral striatum responded more strongly to reward consumption and reward-predicting cues, whereas DA terminals in dorsomedial striatum responded more strongly to contralateral s. In oth cases the terminals encoded a reward prediction error. Our results suggest that the DA modulation of the striatum is spatially organized to support the specialized function of the targeted suregion. Two essential aspects of ehavior are learning to perform actions that lead to reward, and generating those actions. The neuromodulator DA is implicated in oth processes 4, ut how DA supports these related ut distinct functions remains the suject of deate. Historically, dopamine was thought to e involved primarily in generating actions ecause of its role in movement disorders in humans (for example, Parkinson s disease) and the dramatic motor phenotypes following DA perturations in animals 3,5,6. However, seminal studies demonstrated that, rather than encoding actions, phasic activity in putative DA neurons in the ventral tegmental area (VTA) and sustantia nigra pars compacta (SNc) encodes a reward prediction error, or reinforcement signal 7,8, which in theory could e used to learn which stimuli or actions are likely to lead to reward. Consistent with this idea, phasic activation of DA neurons is sufficient to support learning 9,. Thus, the idea emerged that phasic DA activity may only have an indirect effect on movement, either y driving learning or y other mechanisms, such as altering motivational state, or affecting the alance etween activity in the output pathways of the striatum 3,4. However, most of our understanding of what DA neurons do and do not encode has emerged from recordings from putative DA neurons without knowledge of the striatal suregion targeted y the neurons. Notaly, different DA neurons in the VTA and SNc receive different inputs and project topographically to different striatal suregions 59. These striatal suregions themselves have specialized anatomical and functional organizations, with dorsal regions, such as the dorsomedial striatum (DMS), implicated in evaluating and generating actions and ventral regions, such as the nucleus accumens (NAc), implicated in processing reward 3,4. Indeed, these considerations lead to the intriguing hypothesis that DA projections to striatal suregions could support the specialized function of the target area. For example, supopulations of DA neurons might contriute to movement not merely as an indirect consequence of learning or motivation; instead, they could directly encode actions. To test the idea that supopulations of DA neurons that project to specific striatal regions encode specialized information, we sought to compare the representation of rewards and actions in DA axon terminals in the DMS and the NAc as mice performed an instrumental reversal learning task. Toward this end, we comined recent advances in measuring neural activity using calcium indicators in axons terminals of genetically specified neurons deep in the rain 5,6 with other complementary approaches, including a statistical model that allowed isolation of the calcium responses to individual ehavioral events, optogenetic perturations and fast scan cyclic voltammetry. RESULTS Task and ehavior Mice were trained to perform an instrumental reversal learning task (Fig. a). The start of each trial was signified y the illumination of a central nose port. After the mice entered the nose port ( nose poke ), two levers were presented ( lever presentation ). One lever corresponded to a high proaility of reward (7%) while the other lever corresponded to a low proaility (%); which lever (right or left) corresponded to the high versus low proaility of reward was reversed in a proailistic manner after at least ten rewarded trials. Lever presses that resulted in a reward were followed y one auditory stimulus (positive conditioned stimulus, or CS+) and lever Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, New Jersey, USA. Howard Hughes Medical Institute, University of California, San Francisco, San Francisco, California, USA. Correspondence should e addressed to I.B.W. (iwitten@princeton.edu). Received 8 Decemer 5; accepted 4 March 6; pulished online 5 April 6; doi:.38/nn.487 nature NEUROSCIENCE advance online pulication

2 6 Nature America, Inc. All rights reserved. Figure Mice continually learn which to make on the asis of recent experience. (a) A trial starts with the illumination of a central nose port ( Trial start ). As a consequence of entering the central nose port ( ), the mouse is presented with two levers. Pressing one lever results in reward with high proaility (7%; High pro ) and the other lever results in reward with low proaility (%; Low pro ). The identity of the high reward lever reverses on a pseudorandom schedule. The lue and red icons represent an auditory tone used as a positive and negative conditioned stimulus (CS+ and CS ), respectively. () The average proaility of pressing the high or low proaility lever relative to the trial in which the identity of the high proaility lever is switched. (c) Regression coefficients from a logistic regression model predicting each mouse s on a given trial on the asis of previous trial and outcome. Two sets of predictors were used: rewarded on previous trials (lue line), which identifies a as a rewarded right lever (+), rewarded left lever ( ), or unrewarded (), and unrewarded on previous trials (red line), which identifies a as unrewarded right lever (+), unrewarded left lever ( ), or rewarded (). Each predictor was included in the model, with a shift of 5 trials preceding the eing predicted (x axis represents trials etween the predicted and the presses that were not associated with reward were followed y a different auditory stimulus (CS ). A temporal jitter etween and s was introduced etween the nose poke and the lever presentation, as well as the lever press and the CS, to enale separation of the neural responses of temporally neighoring ehavioral events. Mice continually learned which lever to press on the asis of recent experience, as they were more likely to press the lever with the higher proaility of reward, and their s closely followed the reversal of the lever proailities (Fig. ). To quantify how mice were using previous trial outcomes to inform their, we used a logistic regression model to predict the animal s (right or left lever) on the asis of previous trial s and outcomes,7,8. In this model, a positive regression coefficient indicates that an animal was more likely to return to a previously chosen lever while a negative regression coefficient indicates that an animal was more likely to switch to the other lever. The model revealed that previously rewarded s increase the likelihood of returning to the same lever in comparison to unrewarded s (Fig. c). The effects of previous trials decayed roughly exponentially with increasing numer of trials ack (Fig. c). This pattern is consistent with error-driven learning models, in which reward (relative to lack of reward) drives learning aout an action s values, which ultimately determines the susequent s 8. The model provided a good fit to each mouse s ehavior, indicating that mice indeed learned from recent experience to guide their of action (Fig. d; R ranged from.9 to.49 with a median of.4, n = 6 mice). Timed inhiition of dopamine neurons disrupts learning To determine whether and how DA neuron activity affected the animal s on future trials, we transiently inhiited DA neurons a d Proaility Lever B press Proaility Lever A press Right Left.8.4 Time in trial Trial start.4 Lever B High pro Lever A High pro 5 5 Trials from lock switch Levers c Regression coefficient trials High pro Trials ack Rewarded Unrewarded High pro lock Choice Model previous /outcome that was used to predict it). Together, these predictors unamiguously identify the previous s and outcome cominations for the five trials preceding any. Positive regression coefficients for the two predictors correspond to greater likelihood of returning to a lever that resulted in reward, or no reward, respectively. (,c) Error ars, s.e.m. across animals (n = 8). (d) Example of one mouse s ehavior for trials. Black ars, identity of the high proaility lever; orange trace, the mouse s on each trial; gray trace, predicted y the ehavioral model. Low pro in the VTA SNc on a suset of trials as mice performed this task (Supplementary Fig. a). Mice expressing Cre recominase under the control of the tyrosine hydroxylase (Th) promoter (TH::IRES-Cre mice) received injections in the VTA of an adeno-associated virus (AAV) expressing either Cre-dependent NpHR-YFP (halorhodopsin from Natronomonas with yellow fluorescent protein) or YFP only (control virus), as well as ilateral optical fier implants aove each VTA (Fig. a c). To confirm the efficacy of NpHR-mediated inhiition in tyrosine hydroxylase expressing (TH + ) neurons, in a separate group of mice, we performed whole cell recordings in rain slices and oserved large photocurrents (689 ± 6 pa) and effective elimination of current-evoked spiking (Supplementary Fig. d). After mice were trained on the ehavioral task (Fig. a), we ilaterally inhiited DA neurons on a randomly selected % of trials throughout the duration of a trial (from the initial nose poke until either the end of CS or the end of reward consumption). We quantitatively descried the effect of DA inhiition on ehavior y incorporating the optical inhiition into the logistic regression model of introduced in Figure c. This model revealed a significant interaction etween the animals and the optical inhiition of DA neurons, as optical inhiition on previous trials decreased the proaility of returning to the previously chosen lever in comparison to previous trials without inhiition (Fig. d, n = 8 NpHR-YFP mice). The effect of optical inhiition was similar whether the previous trial was rewarded or unrewarded (Fig. d). This effect was greatest in the case of inhiition on the previous trial and declined with the numer of trials separating the inhiition and the, suggesting that the effect (like that of rewards) was mediated y error-driven learning. In contrast, light 7% 3% % 9%..5 Switch Return Proaility of right advance online pulication nature NEUROSCIENCE

3 6 Nature America, Inc. All rights reserved. Figure Inhiiting DA neurons in the VTA SNc alters an animal s on future trials. (a) Surgical schematic showing the optical fier implant (gray rectangle) and injection site (lack needle) in the VTA SNc (green). () Coronal section with anterior posterior (A P) optical fier tip locations for mice injected with AAV5 DIO-EFα-eNpHR3.-YFP (lack points) or DIO-EFα-YFP (gray points). (c) Confocal images of the VTA, showing expression of NpHR-YFP (left), anti tyrosine hydroxylase (TH) staining (middle panel) and an overlay of the two images (right) demonstrating colocalization. Scale ar, µm. Representative image from one animal; similar results were seen across 6 animals. (d) Coefficients from a logistic regression model demonstrating the influence of VTA SNc cell ody inhiition on lever in susequent trials in NpHR-YFP and YFP-control mice. A negative coefficient indicates a reduction in the proaility of returning to the lever chosen on the previous trial; a positive coefficient indicates an increase. Rewarded s with stimulation (stim) decreased the proaility of returning to the chosen lever in comparison to rewarded s without stimulation in NpHR-YFP mice (left; P =., t (7) = 3.39 for one trial ack, P =.4, t (7) =.45 for two trials ack; two-tailed t-test comparing coefficients across animals of rewarded in lue with rewarded + rewarded stim in purple). Likewise, unrewarded with stimulation significantly decreased the proaility of returning to the chosen lever compared to unrewarded alone (left; P =., t (7) = 3.35 for one trial ack, two-tailed t-test comparing unrewarded in red with unrewarded + unrewarded stim in orange). In contrast, no effect of stimulation on future was oserved in the YFP-control animals (right; P =.85, t (7) =., effect of stimulation on rewarded one trial ack; P =.4, t (7) =.876, effect of stimulation on unrewarded one trial ack; two-tailed t-tests). Error ars represent s.e.m. across animals (n = 8 for each panel, *P <.5). had no discernile effect on ehavior in YFP-control littermates (Fig. d, n = 8 YFP-only mice; light group were statistically significant in the case of oth rewarded and unrewarded s one trial ack; P =.43 for rewarded and P =.34 for unrewarded s; P-values from a mixed-effect logistic regression that predicted on the asis of previous rewarded and unrewarded, light and group, and all interactions thereof). Given recent reports of Cre expression in neurons that do not express TH near the VTA of THøIRES-Cre mice 9, we replicated the same experiment in mice expressing Cre recominase under control of the dopamine transporter (Dat) promoter (DAT::Cre mice) and oserved comparale effects of DA neuron inhiition on ehavior (Supplementary Fig. e,f). Together, these results indicate that activity in DA neurons help determine an animal s on future trials in this task and are consistent with dopamine neuron inhiition functioning as a negative reward prediction error. We also compared the latency to initiate the next trial after stimulation versus control trials and found no significant differences (average latency was 9.89 ±.7 s after stimulation versus 9.93 ±.3 s after control trials). This indicates that our manipulation is relatively specific for. To determine whether there is a time window within the trial that contriutes preferentially to the effect of inhiition on, we inhiited DA neurons either from trial initiation to lever press or else from lever press until s after reward consumption. Neither of these sutrial inhiitions had an oservale effect on (Supplementary Fig. g). Recordings and analysis of calcium signals from VTA SNøNAc and VTA SNøDMS terminals After confirming that DA neuron activity in the VTA SNc affected task performance, we employed a genetically encoded calcium indicator to measure and compare neural activity in the terminals of DA neurons originating in the VTA SNc and projecting to either the dorsomedial striatum ( VTA SNøDMS ) or the nucleus a TH::IRES-Cre c d Regression coefficient Optical fier Cre-dependent NpHR-YFP VTA SNc NpHR-YFP YFP-Control SNc VTA NpHR-YFP TH Overlay Rewarded Rewarded with stim NpHR-YFP (n = 8) * Trials ack * YFP-only (n = 8) A P 3.6 mm from regma accumens core ( VTA SNøNAc ) during the reversal learning task. We analyzed retrograde tracer injections in the two striatal regions and found that oth regions received inputs from the lateral portion of the VTA and medial portion of the SNc. Despite the fact that these two striatal regions received inputs from a partially overlapping part of the VTA SNc, quantification of the laeling profile of the retrograde tracers revealed that the supopulation of TH + neurons projecting to each region was largely non-overlapping (only 8 ± 4% of TH + neurons that projected to either area projected to oth areas, n = 487 neurons from n = 6 mice; Supplementary Fig. ). To record from the terminals of these DA neuron supopulations, a Cre-dependent AAV virus expressing the genetically encoded calcium indicator was injected in the VTA of THøIRES-Cre mice and optical fiers were implanted aove the DMS and NAc (Fig. 3; each mouse had one recording site in DMS and one in NAc). We first confirmed colocalization of and TH + in VTA SNøNAc and VTA SNøDMS neurons y quantifying overlap in VTA SNc etween a retrograde tracer injected into NAc or DMS, TH expression, and Cre-mediated expression (Fig. 3a). This revealed a high degree of specificity etween and TH expression in VTA SNøNAc and VTA SNøDMS neurons (99 ±.5% of VTA SNøNAc and 95 ± % of VTA SNøDMS neurons that expressed also expressed TH) as well as penetrance of expression in TH + neurons (9 ± % of NAc-projecting TH + cells and 88 ± 3% of DMSprojecting TH + cells expressed ; n = 9 VTA SNøNAc neurons and n = VTA SNøDMS neurons; n = mice). We measured fluctuations in fluorescence in the VTA SNøNAc and VTA SNøDMS terminals in the striatum through fier photometry 6,3 (Fig. 3c,d). In VTA SNøNAc and VTA SNøDMS terminals expressing, ut not terminals expressing GFP (control virus), we found large fluctuations in the fluorescence signal, reflecting underlying fluctuations in neural activity (Fig. 3d, change in fluorescence relative to aseline (df/f) of.6 ±.7 for VTA SNøNAc terminals, n = sites; peak df/f of.4 ±.5 * Trials ack Unrewarded Unrewarded with stim Switch Retum nature NEUROSCIENCE advance online pulication

4 a c d TH 4 µm optic fier TH::IRES-Cre Cre-dependent terminals terminals. df/f 5 s e Optical fier and microelectrode caron fier TH::IRES-Cre Bipolar stimulating electrode Cre-dependent CTB Overlay Striatum VTA SN Striatum VTA SN 6 Nature America, Inc. All rights reserved. f Applied voltage (V) na.4 V.3 V 7 na Current (na) g gcamp df/f [DA] 5 nm [DA] 5 5 h Latency of max (s).6.4. GFP control [DA] ** Figure 3 Calcium recordings in terminals of striatally projecting DA neurons. (a) Confocal image of expression in VTA SNc, along with tyrosine hydroxylase (TH) immunoreactivity and cholera toxin suunit B (CTB) laeling (retrograde tracer injected into the DMS). Scale ar, 5 µm. Representative image from one animal; similar results otained in mice. () Surgical schematic for terminal measurements; adeno-associated virus containing a Cre-dependent genetically encoded calcium indicator (AAV5 FLEX-CAG-) viral injection in the VTA SNc and optical fier placement in the striatum. (c) expression in DA axon terminals of the NAc (left panel) and DMS (right panel) suregions of the striatum (confocal images of sectioned tissue). Scale ar: 4 µm. Representative image from one animal; similar results were seen in animals. (d) Example df/f fluctuations in awake, ehaving mice in VTA SNøNAc terminals (top, orange trace) and j i Width at half max (s) Normalized max pulse *** 6 pulses 4 pulses pulses Normalized max [DA] VTA SNøDMS terminals (middle, lue trace). In contrast, control recordings taken from mice injected with adeno-associated virus containing Credependent GFP (AAV5 FLEX-CAG-GFP) (ottom, gray trace) show no modulation. Representative traces from three animals; similar traces seen in animals for recordings and 4 GFP control mice. (e) Schematic of simultaneous and voltammetry recordings. (f) Color representation of cyclic voltammetric data at one DMS site in response to electrical stimulation of the MFB (average of 5 trials; stimulation at s). The y axis is applied voltage, the x axis is time, and current measured at the caron fier electrode is plotted in false color. Inset: cyclic voltammogram.5 s after electrical stimulation is consistent with the chemical signature of DA. (g) Traces represent [DA] (lack) and (lue) over time for the data represented in f (average of 5 trials; stimulation at s; shaded region represents s.e.m.). (h) Latency etween stimulation onset and peak amplitude was longer for DA terminal responses than for [DA] transients in oth DMS and NAc (two-way repeated measures ANOVA revealed significant effect of recording method, F (,6) = 6.36, P =.6, and no effect of recording area nor area method interaction). (i) The width of the signal at half the maximum peak was significantly larger for [DA] than for responses in VTA SNøDMS (gray and lue) and VTA SNøNAc (gray and orange) (two-way repeated measures ANOVA revealed a significant effect of recording method, F (,6) = 4., ***P < 4 5, ut no effect of recording area nor area method interaction). (j) The relationship etween peak fluorescence and peak [DA] in NAc (orange) and DMS (lue) for 6 Hz stimulation with increasing numers of pulses (, 6,, 4). In h j data from 5 recording sites in the DMS and 4 sites from the NAc (across 3 mice). In f i, electrical stimulation of MFB was pulses of 6 Hz stimulation. Error ars represent s.e.m. in all panels. for VTA SNøNAc terminals, n = sites; and peak df/f of.6 ±. for GFP terminals, n = 4 striatal sites). To aid in quantitatively interpreting these recordings of calcium dynamics in DA terminals, we compared the dynamics of the terminal signal with the dopamine concentration ([DA]) in the same striatal location y affixing the optical fier for measuring DA terminal calcium dynamics ( fluorescence) to a voltammetry electrode for measuring extracellular [DA] (in anesthetized mice) and measuring oth signals simultaneously while stimulating DA fiers of passage in the medial forerain undle (MFB; Fig. 3e). Comparison of calcium dynamics and [DA] at an example recording site revealed faster dynamics of fluorescence relative to [DA] (Fig. 3g; stimulation of pulses at 6 Hz). Similarly, we found a significantly shorter latency to the peak response across the population of recording sites compared to latency to the peak [DA] for oth the DMS and NAc recording sites (Fig. 3h, n = 5 and n = 4 recording sites for DMS and NAc, respectively; two-way repeated measures ANOVA showed a significant effect of recording method, F (,6) = 6.36, P =.67 ut no effect of recording area nor of area method interaction). In addition, the width of the response at half the peak value was significantly shorter in the recording relative to the [DA] (Fig. 3i; two-way repeated measures ANOVA revealed a significant effect of recording method, F (,6) = 4., P = , ut no effect of recording area nor of area method interaction). Finally, to assess how the magnitude of the signal and the [DA] related to the strength of activity in DA neurons, we varied the numer of pulses of the MFB phasic electrical stimulation. A comparison of the peak response of the two measurements across the population of recording sites revealed a monotonically increasing relationship etween and [DA] with increasing pulse numer, a relationship that was remarkaly similar for DMS and NAc recording sites (Fig. 3j). At high pulse numers, the response egan to saturate relative to [DA], suggesting a saturation in either calcium concentration in the terminals or saturation in the sensor itself (Fig. 3j, n = 4 recording sites per condition in NAc and n = 5 recording sites per condition in DMS; peak was normalized relative to the largest response for each recording site). Given that a urst of phasic dopamine is typically 8 spikes 8,3, the data suggests a relatively linear relationship etween and [DA] in the relevant dynamic range (Fig. 3j). advance online pulication nature NEUROSCIENCE

5 Reward and prediction error encoding in striatal DA terminals To isolate the neural response associated with individual ehavioral events (nose poke, lever presentation, lever press, CS+, CS, reward consumption) while avoiding contamination from neighoring events, we fitted a linear regression model in which we modeled the measured trace as the sum of the convolution of each ehavioral event with a response kernel that corresponded to that event (Fig. 4a; details in Online Methods). We chose the coefficients of the kernels to minimize the difference etween the modeled and actual signal. The kernels for a particular ehavioral event can e interpreted as the isolated response to that event without contamination from other neary events, assuming linearity in the responses to each event. This statistical disamiguation of the responses to neighoring events was enaled y the temporal jitter that we had introduced etween events in the task, as well as y the variaility in each mouse s ehavioral latencies. We first examined the encoding of reward predicting stimuli and reward consumption in VTA SNøNAc and VTA SNøDMS terminals (Fig. 4). Aligning the d responses for all trials of all mice to the CS+ or the CS revealed a positive response to the CS+ and a negative response to the CS in oth the VTA SNøDMS and VTA SNøNAc terminals (Fig. 4c), showing that these responses are associated with the anticipatory component of reward prediction error, as previously reported in DA neuron cell ody recordings 8. The average CS+ response for VTA SNøNAc terminals was larger than for VTA SNøDMS terminals, which was evident in the singletrial heat maps and the average time-locked response (Fig. 4c,d and Supplementary Fig. 3a; time-locked VTA SNøNAc responses were 36% higher than VTA SNøDMS). This was further quantified y comparing the VTA SNøNAc and VTA SNøDMS CS+ kernels derived from the linear model of the signal (Fig. 4e and 6 Nature America, Inc. All rights reserved. a Trial time kernel Levers out kernel Lever press kernel CS kernel Model Levers out Bregma.54 A P Lever press CS Bregma.6 A P 5 s 5% df/f Bregma.74 A P d e c Regression coeffienct (n = ) Trial (n = ) Trial CS+ CS+ Lever press CS+ kernel gcamp6f CS Lever press CS CS kernel Reward consumption Reward consumption kernel gcamp6f Reward consumption CS+ gcamp6f.5.5 terminals terminals P <..5.5 Figure 4 Response to reward consumption and to reward-predictive cues dominates in VTA SNøNAc relative to VTA SNøDMS terminals. (a) Schematic of linear model used to predict signal from task events. Each ehavioral event is depicted as rectangular tick in the top row. Each kernel is time-shifted according to the time of the corresponding events (light gray traces) and then summed to generate the modeled trace (lack trace), which correlates with the recorded trace (dark gray trace). () Coronal sections of the mouse rain showing location of optic fiers used for terminal recordings. Orange and lue indicate VTA SNøNAc and VTA SNøDMS recordings, respectively. (c) Heat maps represent the of all single trial recordings time-locked to the CS+ (left), CS (middle), or onset of reward consumption (right), for recordings in VTA SNøNAc (top row) and VTA SNøDMS (ottom row) terminals. All recorded trials from all recording sites are represented, and the trials are sorted on the asis of the time of lever press (for left and middle heat maps) or the time of the CS+ (for the rightmost heat map). (d) Average across all recording sites time-locked to CS+ (left), CS (middle) and reward consumption (right, n = recording sites from animals for oth VTA SNøNAc and VTA SNøDMS). Orange traces are from VTA SNøNAc recording sites; lue, VTA SNøDMS. (e) Average kernel across all recording sites corresponding to CS+ (left), CS (middle), and reward consumption (right; n = recording sites for oth VTA SNøNAc and VTA SNøDMS). Kernels are coefficients from a linear regression model where each ehavioral event is convolved with a corresponding kernel and all convolved traces are summed to predict the measured signal, as schematized in a. Two-tailed t-test comparing VTA SNøNAc and VTA SNøDMS kernels in. s ins reveals statistically significant difference etween.7. s post-cs+ onset for CS+ kernel (left),.6. s post-cs onset for CS kernel, and.3.6 s post-reward consumption onset for reward consumption kernel (P <., significance represented y gray region). In d,e error ars represent s.e.m. across sites. nature NEUROSCIENCE advance online pulication

6 Figure 5 Evidence of reward prediction error encoding in oth VTA SNøNAc and VTA SNøDMS terminals. (a) The CS response in VTA SNøNAc terminal recordings (.. s post CS onset, time range selected on the asis of the CS+ and CS kernels in Fig. 4e) is predicted with a multiple linear regression with the reward outcome of the current trial (laeled trial ) and the previous trial(s) (laeled with negative numers) used as predictors. The average regression coefficient across mice is plotted for current and previous trials. The positive coefficient for the current trial and negative coefficients for previous trials indicate the encoding of a reward prediction error. () Same as a, ut for VTA SNøDMS terminal recordings. All error ars represent s.e.m. across mice; n = recording sites from animals in each panel; t-test, *P <.. a Regression coefficient * 4 3 Trials ack * Regression coefficient * 4 3 Trials ack * 6 Nature America, Inc. All rights reserved. Supplementary Fig. 3). The VTA SNøNAc and VTA SNøDMS kernels were statistically different from.7 to. s post-cs+ onset (Fig. 4e; peak kernel in VTA SNøNAc was 9% larger than in VTA SNøDMS; n = recording sites for oth VTA SNøDMS and VTA SNøNAc). Similarly, the negative response to the CS was more pronounced in the single-trial heat maps and time-locked average response for VTA SNøNAc relative to VTA SNøDMS terminals (Fig. 4c,d and Supplementary Fig. 3a; minimum time-locked VTA SNøNAc response was 78% more negative on average compared with VTA SNøDMS). A statistical comparison etween the two populations revealed a statistically significant difference in the kernel from.6. s post-cs onset across the two areas (Fig. 4e; minimum kernel was 85% more negative on average in VTA SNøNAc than in VTA SNøDMS; significance defined as P <. for two-tailed t-test comparing.-s time ins; n = recordings sites for oth VTA SNøDMS and VTA SNøNAc). In addition, the response to reward consumption was also stronger in the VTA SNøNAc terminals relative to the VTA SNøDMS Figure 6 Contralateral response preference in VTA SNøDMS terminals. (a) Heat maps represent the of recordings time-locked to the nose poke (left) or lever presentation (right), for VTA SNøNAc terminals. Heat maps shows all trials preceding either an ipsilateral lever press (top) or a contralateral lever press (ottom). Ipsilateral and contralateral lever presses are defined relative to the recording site in the rain. All recorded trials from all recording sites are represented (n = recording sites), and the trials are sorted on the asis of the time of illumination of the trial start light in the nose port (for left column), or the time of the nose poke (for right column). () Average d signal time-locked to the nose poke (left panel) and lever presentation (right panel) for ipsilateral (lack) or contralateral (orange) trials in VTA SNøNAc recording sites (averages taken across mice, n = recording sites). (c) and lever presentation kernels are derived from the same statistical model as descried in Figure 4a, ut in this case the events preceding an ipsilateral lever versus a contralateral lever are considered separately. No significant difference in the ipsilateral and the contralateral kernels (P >. for all time ins, two-tailed t-test comparing ipsilateral versus contralateral VTA SNøNAc kernels in ms ins). (d) Same as a ut for VTA SNøDMS terminal recordings (n = recording sites). (e) Same as ut for VTA c Regression coefficient terminals. This was evident in the single-trial heat maps as well as the time-locked average response (Fig. 4c,d; VTA SNøNAc response was 39% larger). In fact, the VTA SNøNAc terminals displayed a positive response efore and throughout the consumption of reward whereas the VTA SNøDMS terminals showed very small consumption responses (Fig. 4e; peak kernel in VTA SNøNAc was 4% larger than in VTA SNøDMS). Indeed, statistically comparing the magnitude of reward consumption kernels revealed a significant difference etween.3 and.6 s post-consumption (Fig. 4e; significance defined as P <., two-tailed t-test comparing. s time ins, n = recordings sites for oth VTA SNøDMS and VTA SNøNAc). Previous studies using electrophysiological recordings of putative DA neurons in rodents have also reported a residual response to fully predicted rewards A well-characterized feature of the electrophysiological recordings of activity in DA neurons in the VTA SNc is the encoding of a reward prediction error to outcomes, which may serve as a reinforcement signal,35,36. Specifically, the strength of the response to a reward a terminals d terminals Trial start Trial start Ipsilateral Contralateral Trial Trial kernel.5.5 kernal Contra Ipsi e f Regression coefficient kernal kernal Contra Ipsi P <. SNøDMS recording sites (n = ). (f) Same as c ut for VTA SNøDMS recording sites (n = recording sites). Two-tailed t-test comparing ipsilateral and contralateral kernels in. s ins revealed a significant difference (threshold of P <. for shading) etween.4.9 s for contralateral versus ipsilateral nose poke kernels (left) and.3.5 s for contralateral versus ipsilateral lever presentation kernel. Error ars in,c,e,f represent s.e.m. across sites advance online pulication nature NEUROSCIENCE

7 6 Nature America, Inc. All rights reserved. Figure 7 Contralateral response preference in VTA SNøDMS cell odies. (a) Surgical schematic. Cre-dependent was injected into terminals of the DMS (lack needle) and expression in VTA SNøDMS cell odies (green) was recorded through an optical fier (gray rectangle). () Coronal sections of the mouse rain showing location of optical fier recording locations (gray circles). (c) Confocal images of the VTA and SN. Expression of (top panel), anti tyrosine hydroxylase (TH) staining (middle panel) and an overlay of the two images (ottom panel) demonstrates retrograde travel following injection in the DMS. Scale ar, µm. Representative image from one animal; similar results were seen across 4 animals. (d) Heat maps represent the of recordings timelocked to the nose poke (left) or lever presentation (right), for VTA SNøDMS cell odies. Heat maps show all trials preceding either an ipsilateral lever press (top) or a contralateral lever press (ottom). All recorded trials from all recording sites are represented (n = 7 recording sites), and the trials are sorted on the asis of the time of illumination of the trial start light in the nose port (for left column), or the time of the nose poke (for right column). (e) Average time-locked to the nose poke (left panel) and lever presentation (right panel) for ipsilateral or contralateral trials in VTA SNøDMS neurons (average across n = 7 recording sites). (f) (left) and lever presentation (right) kernels for ipsilateral and contralateral trials derived from the statistical model descried in Figure 4a. Gray shading shows a significant difference etween ipsilateral and contralateral kernels from.4.9 s after time of nose poke and.4.6 s after lever presentation (two-tailed t-test, -ms ins, threshold of P <.). Error ars in e,f represent s.e.m. across sites. is modulated y reward history such that an unexpected reward leads to a stronger response than expected rewards. To determine whether reward prediction error was a feature of VTA SNøNAc and VTA SNøDMS terminals during this task, we adapted the analysis from Bayer and Glimcher 35 and created a multiple linear regression model in which we used the presence or asence of reward on the current and previous trials to predict the response to the conditioned stimulus (CS+ or CS ) on the current trial (response time window of.. s after CS onset, chosen on the asis of the time course of the CS+ and CS kernels; Fig. 4e). A reward prediction error would involve a positive modulation of the response y the current trial s reward, together with a negative modulation y the rewards on previous trials. (This is ecause the prediction error takes the form of otained minus predicted rewards, and reward predictions are expected to arise from a recency-weighted average over rewards otained on previous trials, as also suggested y ehavior; Fig. c). This is indeed what we saw, oth in VTA SNøNAc and VTA SNøDMS terminals: response to the CS+ was positively modulated y reward on that trial, ut negatively modulated y reward on previous trials (Fig. 5, P =.84 8, t () = 4. and P =.34, t () = 3.89 for current and one-trial-ack modulation, respectively, for VTA SNøNAc; P =.7, t () = 6.4 and P =.5, t () = 3.8 for current and one-trial-ack modulation, respectively, for VTA SNøDMS; two-tailed t-test). Choice encoding in striatal DA terminals We next examined how terminal activity in VTA SNøNAc or VTA SNøDMS neurons was modulated y the mouse s of action. In the VTA SNøNAc recordings, d recordings aligned to oth the nose poke and lever presentation revealed no consistent response preference across the population on trials with a Cre-dependent 4 µm optic fier TH::IRES-Cre c Striatum VTA SN A P 3.6 mm regma SNc VTA VTA TH Overlay SNc d Ipsilateral Contralateral e f Trial Trial Regression coefficient Trial start kernel an ipsilateral or contralateral lever, as expected for dopamine neurons (Fig. 6a,; ipsilateral and contralateral defined relative to the side of the recording). Similarly, a quantitative comparison of the VTA SNøNAc response kernels calculated for upcoming ipsilateral and contralateral s showed no significant difference etween responses of ipsilateral versus contralateral trials for either nose poke or lever presentation (Fig. 6c; paired t-test,. s time ins, P >., n = recording sites). In contrast, for the VTA SNøDMS recordings, the d response of trials time-locked with the nose poke and lever presentation showed a qualitatively larger response when the lever chosen was contralateral (rather than ipsilateral) to the recording location (Fig. 6d,e;,% larger peak response for the nose poke preceding contralateral versus ipsilateral s, 44% larger response for lever presentation preceding contralateral versus ipsilateral s). Indeed, a quantitative comparison of the response kernels associated with the nose poke and lever presentation preceding contralateral versus ipsilateral s showed a significantly larger response on contralateral trials etween.4 and.9 s for the nose poke kernel and etween.3 and.5 s for the lever presentation kernel (Fig. 6f; 5% larger response for contralateral versus ipsilateral nose poke kernel, 57% larger response for lever presentation; significance defined as P <. for two-tailed t-test comparing. s time ins; n = recordings sites for oth VTA SNøDMS and VTA SNøNAc; Supplementary Fig. 4). In addition, a direct comparison of the extent of contralateral response preference etween VTA SNøDMS versus VTA SNøNAc terminals revealed a significant difference (unpaired two-tailed t-test; maximum difference in ipsilateral versus contralateral responses etween VTA SNøDMS and VTA SNøNAc sites; P =., t () =.46 for nose poke response, P =.4, t () =.8 for lever presentation responses) Contra Ipsi kernel Contra Ipsi P <. nature NEUROSCIENCE advance online pulication

8 6 Nature America, Inc. All rights reserved. To control for the possiility that this stronger response for contralateral s could e related to a movement artifact rather than a true measure of neural activity, we examined the difference in ipsilateral and contralateral encoding in mice that expressed a calciuminsensitive GFP rather than in DA neurons. Unlike the nose poke and lever presentation kernels otained from striatal terminals of animals expressing, the nose poke and lever presentation kernels from striatal terminals of the GFP control mice did not show significant modulation (Supplementary Fig. 5, P >. for all timepoints, t-test comparing contralateral and ipsilateral kernels in.-s time ins; n = 4 recordings sites). Given the task structure, an animal s on one trial is correlated with its on the previous trial. To determine whether neural responses were etter explained y current trial versus previous trial, we created a new variant of the regression model that contained interactions etween the ehavioral event kernels and oth current and previous. The animal s current had a much larger influence on predicting VTA SNøDMS signal than its previous, indicating that the -selective activity was reflective of current, and not past, s (Supplementary Fig. 6). Choice encoding in VTA SNøDMS cell odies The preferential response to contralateral s in VTA SNøDMS terminals may e generated y axo-axonal interactions within the striatum or it may already e present in the VTA SNøDMS cell odies. To distinguish etween these two possiilities, we recorded from VTA SNøDMS cell odies in THøIRES-Cre mice, selectively targeted y injecting Cre-dependent virus into the DMS (Fig. 7a c). As in our findings in VTA SNøDMS terminals (Fig. 6d f), VTA SNøDMS cell odies showed qualitatively larger responses on contralateral trials to oth the nose poke and lever presentation (Fig. 7d,e; the average d response was 9% larger on contralateral trials for nose poke and 64% larger for lever presentation; averages across recording sites; n = 7; individual difference traces shown in Supplementary Fig. 7a). Similarly, the regression kernels for the nose poke and lever presentation were significantly larger for contralateral relative to ipsilateral trials (Fig. 7f; 49% larger for the nose poke kernel preceding contralateral versus ipsilateral s, 4% larger for lever presentation; significance defined as P <. for two-tailed t-test comparing. s time ins; n = 7, individual traces shown in in Supplementary Fig. 7). These results demonstrate that the encoding seen in dopaminergic terminals in the DMS is also present in the cell odies of these projecting neurons. DISCUSSION Here we recorded neural activity from striatal terminal regions of midrain DA neurons. We found striking differences in the encoding of reward and in VTA SNøNAc and VTA SNøDMS DA terminals, with VTA SNøNAc terminals preferentially responding to reward and VTA SNøDMS terminals preferentially responding to contralateral (versus ipsilateral). We also found an important commonality in task encoding in the two populations, as oth populations represented a reward prediction error, or reinforcement signal. Transient dopamine neuron inhiition affects future Our instrumental reversal learning task enaled mice to continually learn which lever to press on the asis of trial and error. The large numer of learning trials afforded the statistical power to quantitatively model the effect of transient dopamine neuron inhiition on, as well as to determine how neural activity related to reward,, and prediction error. Before examining neural correlates of the task in DA supopulations, we first confirmed that task performance depended on DA activity y optogenetically inhiiting activity throughout a trial on a randomly selected suset of trials (Fig. ). As expected, we found that future was affected in a manner consistent with dopamine neuron inhiition functioning as a negative reward prediction error. Our quantitative modeling of ehavior revealed a preserved differential effect of rewarded and unrewarded s on future after dopamine neuron inhiition relative to control trials, only with a shifted aseline, such that the mice had a tendency to avoid previous s associated with dopamine neuron inhiition. Note that this effect appeared to e specific to, as there was no change in latency to initiate the next trial after dopaine neuron inhiition. This specificity may e due to the fact that is easier to pertur than initiation latency in our task, given that mice are trained to adjust their s continually on the asis of reward feedack. We did not oserve an effect of inhiiting dopamine neurons for only a suset of time within a trial (inhiiting either efore or after the lever press; Supplementary Fig. g). This negative result may e caused y reound excitation after transient dopamine neuron inhiition cancelling the effect of inhiition. Alternatively, it may e that inhiition for only a suset of a trial is not a strong enough manipulation to pertur ehavior, especially considering that optogenetic inhiition throughout the VTA SNc was likely incomplete as a result of light scattering in neural tissue. Preferential responses to reward in VTA SNøNAc terminals A striking result in this study is the greater strength of responses to reward-predictive cues and reward consumption in VTA SNøNAc terminals relative to VTA SNøDMS terminals (Fig. 4c e). In oth areas, the stimulus that predicted reward (CS+) led to an increase in the calcium signal and the stimulus that predicted asence of reward (CS ) led to a decrease in the calcium signal. The magnitudes of responses to oth stimuli were much larger in the VTA SNøNAc terminals. In addition, a positive reward consumption response was evident in the VTA SNøNAc terminals, while if anything the VTA SNøDMS terminals exhiited a depression during reward consumption. These results support the notion that the VTA SNøNAc terminals are more involved in reward processing than are VTA SNøDMS terminals. Similarly, reward prediction errors were evident in oth populations; and the magnitude of such responses was also larger in VTA SNøNAc, consistent with our finding of more reward-related responses in that population (Fig. 5a,). Reward prediction error was quantified on the asis of a multiple linear regression model in which the response to the reward-predicting cue (CS+ or CS ) on the current trial was predicted on the asis of the reward outcomes of current and previous trials 35 (Fig. 5). In this analysis, the signature of a reward prediction error is a positive coefficient in the regression model for the current trial outcome along with negative coefficients for the previous trial outcomes, with more recent previous trials weighted more heavily. This represents a reward prediction error in that reward on previous trials decreases the strength of the response to reward on the current trial. In particular, if reward predictions are learned via the Rescorla-Wagner rule, then the regression coefficients for earlier trials outcomes should decline exponentially 7,35. The multiple linear-regression model revealed a profile consistent with these expectations in oth VTA SNøNAc and VTA SNøDMS terminal recordings (Fig. 5). Thus, these data support the view that oth dopaminergic projections encode a reward prediction error, or advance online pulication nature NEUROSCIENCE

9 6 Nature America, Inc. All rights reserved. reinforcement signal, and are consistent with recent evidence that DA terminals in oth dorsal and ventral striatum are sufficient to drive learning 37. Preferential contralateral responses in VTA SNøDMS terminals We found that contralateral s drive consistently stronger responses than ipsilateral s in VTA SNøDMS cell odies and terminals (Figs. 6 and 7). In other words, neural activity is greater preceding a contralateral versus an ipsilateral movement (lever press), supporting a role for this projection not only in learning and motivation, ut also in generating or expressing the movement that the mouse has chosen. There was a greater contralateral response preference in the VTA SNøDMS relative to the VTA SNøNAc terminals, although there appeared to e individual VTA SNøNAc recording sites that preferentially encode one or the other (Supplementary Fig. 4). To our knowledge, preferential responses for contralateral s have not een previously reported in DA neurons. This could e ecause previous recordings have not isolated responses in VTA SNøDMS neurons. In fact, electrophysiology experiments have often identified DA neurons in part y response to unexpected reward, which may have selected against recording from these neurons, given this supopulation s relatively weak responses to reward (Fig. 4). Our finding of contralateral response preference is consistent with the extensive literature demonstrating lateralized motor deficits as a result of unilateral manipulation of DA neurons 6. Given that the dorsal striatum is involved in action selection and generation, the contralateral preference we found in VTA SNøDMS terminal responses suggests that DA neurons projecting to that region are specialized to support its function. One possiility is that DA not only affects movement indirectly through an effect on learning or motivation, ut is also directly involved in signaling movement and even controlling movement execution (for example, via effects on medium spiny neurons in the contralateral striatum). Alternatively, it may e that contralateral selectivity arises as part of a specialized prediction-error signal appropriate to the dorsal striatum. For instance, while the standard prediction error is scalar and unitary, more sophisticated computational models have sometimes invoked vector-valued prediction errors with separate components specific to different actions, effectors or situations Prior evidence of spatial heterogeneity in striatal dopamine Previous electrophysiological recordings of putative midrain DA neurons in monkeys have related neural responses to the location of the recording electrode. Given the topography etween the VTA SNc and the dorsal and ventral striatum, the medial lateral axis of the VTA SNc should roughly distinguish etween DA neurons projecting to ventral versus dorsal striatum. Using this approach, one study reported more reward encoding in putative DA neurons in the VTA compared to SNc 4, while other studies have reported a more homogeneous distriution 8,4,43. This lack of consensus may arise from the challenge of determining the precise anatomical location of the recording electrode, coupled with the imperfect topography in VTA SNc projections to the striatum. In addition, neurochemical approaches have een employed to measure [DA] in specific suregions of the striatum during ehavior 36, Although most voltammetry studies have focused on the NAc 36,46 48, other studies have recorded in the dorsal striatum, and their results are consistent with our finding that reward responses are stronger in the ventral than in the dorsal striatum. For example, reward-related [DA] transients appear after more extensive ehavioral training in the dorsal striatum relative to the ventral striatum 49,5. Similarly, ramping patterns of [DA] as a rat approaches a reward location are more prominent in the ventromedial compared to the dorsolateral striatum 45. Our finding of differential encoding of reward and in the two terminal regions oth confirms and extends these results. In particular, the presence of features that are encoded more prominently in VTA SNøDMS terminals (contralateral preference) suggests that the oservation of weaker reward encoding in VTA SNøDMS terminals is not related to technical difficulties in otaining DA signals in the more dorsal striatal regions, ut is indeed a fundamental organizational feature of striatal DA. Conclusions In summary, given that the NAc is thought to e preferentially involved in reward processing while the DMS is thought to e preferentially involved in action selection and generation, the relatively strong reward-related responses in VTA SNøNAc terminals and the presence of contralateral selectivity in VTA SNøDMS terminals suggest that the DA innervation of striatal suregions is specialized to contriute to the specific function of the target region. This new concept, that the DA innervation of these striatal suregions contriutes to their specialized function, complements the major current view of striatal DA as encoding a reward prediction error signal to support learning. Methods Methods and any associated references are availale in the online version of the paper. Note: Any Supplementary Information and Source Data files are availale in the online version of the paper. Acknowledgments We thank C. Gregory, M. Applegate and J. Finkelstein for assistance in data collection; J. Pillow and A. Conway for advice with data analysis; M. Murugan and B. Engelhard for comments on the manuscript; and D. Tindall and P. Wallace for administrative support. I.B.W. was supported y the Pew, McKnight, NARSAD and Sloan Foundations, NIH DP New Innovator Award and an R MH6689-; N.F.P was supported y an NSF Graduate Research Fellowship; and J.P.T. and I.B.W. were supported y the Essig and Enright 8 Fund. AUTHOR CONTRIBUTIONS N.F.P., C.M.C., J.P.T. and J.L. performed the experiments; N.F.P., C.M.C., J.P.T., J.L. and J.Y.C. analyzed the data; T.J.D. provided advice on rig design; N.D.D. and I.B.W. provided advice on statistical analysis; N.F.P., N.D.D. and I.B.W. designed experiments and interpreted the results; and N.F.P. and I.B.W. wrote the manuscript. COMPETING FINANCIAL INTERESTS The authors declare no competing financial interests. Reprints and permissions information is availale online at reprints/index.html.. Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 5763 (6).. Schultz, W., Dayan, P. & Montague, P.R. A neural sustrate of prediction and reward. Science 75, (997). 3. Starr, B.S. & Starr, M.S. Differential effects of dopamine D and D agonists and antagonists on velocity of movement, rearing and grooming in the mouse. Implications for the roles of D and D receptors. Neuropharmacology 5, (986). 4. Wise, R.A. Dopamine, learning and motivation. Nat. Rev. Neurosci. 5, (4). 5. Marshall, J.F. & Berrios, N. Movement disorders of aged rats: reversal y dopamine receptor stimulation. Science 6, (979). 6. Aruthnott, G.W. & Crow, T.J. Relation of contraversive turning to unilateral release of dopamine from the nigrostriatal pathway in rats. Exp. Neurol. 3, (97). nature NEUROSCIENCE advance online pulication