Hierarchical Prediction Errors in Midbrain and Basal Forebrain during Sensory Learning

Transcription

1 Artcle Herarchcal Predcton Errors n Mdbran and Basal Forebran durng Sensory Learnng Sandra Iglesas, 1,2, * Chrstoph Mathys, 1,2 Kay H. Brodersen, 1,2 Lars Kasper, 1,2 Marco Pccrell, 2 Hanneke E.M. den Ouden, 3 and Klaas E. Stephan 1,2,4 1 Translatonal Neuromodelng Unt (TNU), Insttute for Bomedcal Engneerng, Unversty of Zurch and Swss Federal Insttute of Technology (ETH), 8032 Zurch, Swtzerland 2 Laboratory for Socal and Neural Systems Research (SNS), Unversty of Zurch, 8091 Zurch, Swtzerland 3 Donders Insttute for Bran, Cognton and Behavor, Radboud Unversty, Njmegen, 6500 HE, The Netherlands 4 Wellcome Trust Centre for Neuromagng, Unversty College London, London WC1N 3BG, UK *Correspondence: glesas@bomed.ee.ethz.ch SUMMARY In Bayesan bran theores, herarchcally related predcton errors (PEs) play a central role for predctng sensory nputs and nferrng ther underlyng causes, e.g., the probablstc structure of the envronment and ts volatlty. Notably, PEs at dfferent herarchcal levels may be encoded by dfferent neuromodulatory transmtters. Here, we tested ths possblty n computatonal fmri studes of audo-vsual learnng. Usng a herarchcal Bayesan model, we found that low-level PEs about vsual stmulus outcome were reflected by wdespread actvty n vsual and supramodal areas but also n the mdbran. In contrast, hgh-level PEs about stmulus probabltes were encoded by the basal forebran. These fndngs were replcated n two groups of healthy volunteers. Whle our fmri measures do not reveal the exact neuron types actvated n mdbran and basal forebran, they suggest a dchotomy between neuromodulatory systems, lnkng dopamne to low-level PEs about stmulus outcome and acetylcholne to more abstract PEs about stmulus probabltes. INTRODUCTION The noton that the bran has evolved to mplement a predctve machnery for antcpaton of future events has exsted snce early cybernetc theores (Ashby, 1952). The mechansms by whch the bran learns the probablstc structure of the world have been examned prmarly from the perspectve of renforcement learnng (RL), wth a focus on how reward learnng s drven by predcton errors (PEs) (Fletcher et al., 2001; McClure et al., 2003; O Doherty et al., 2003; Pessglone et al., 2006; Wunderlch et al., 2011). Another perspectve s provded by theores that vew the bran as approxmatng optmal Bayesan nference (Dayan et al., 1995; Doya et al., 2011; Frston, 2009; Knll and Pouget, 2004; Kördng and Wolpert, 2006). These theores go beyond reward learnng and have been appled to many aspects of percepton as, for example, n theores of predctve codng (Rao and Ballard, 1999) and the free energy prncple (Frston et al., 2006). A central postulate of these Bayesan perspectves s that the bran contnuously updates a herarchcal generatve model of ts sensory nputs to predct future events and nfer on the causal structure of the world. Ths belef updatng process rests on multple, herarchcally related PEs that are weghted by ther precson. Notably, these PEs are not restrcted to reward, but concern all types of sensory events as well as ther underlyng laws, e.g., probablstc assocatons and how these change n tme (volatlty; Behrens et al., 2007). Smply speakng, estmates of envronmental volatlty are updated n proporton to PEs about stmulus probabltes; n turn, estmates of stmulus probabltes are updated by PEs about stmulus occurrences. Whle several emprcal studes have examned human behavor and bran actvty from ths Bayesan perspectve, the herarchcal nature of PEs has receved lttle attenton so far. Ths s a sgnfcant gap, not only because herarchcally related PEs are at the heart of the Bayesan formalsm, but also because PEs at dfferent herarchcal levels may be lnked to dfferent neuromodulatory transmtter systems. Whle dopamne (DA) has long been related to the encodng of PEs about reward (Daw and Doya, 2006; Schultz et al., 1997), other modulatory neurotransmtters have been lnked to more abstract roles, such as encodng of expected uncertanty by acetylcholne (ACh) (Yu and Dayan, 2002, 2005). Notably, ths was (mplctly) operatonalzed as a hgher-level PE n that t represents the dfference between a condtonal probablty (degree of cue valdty) and certanty. Other computatonal concepts of ACh suggested that t may be representng the learnng rate (Doya, 2002). Agan, ths noton can be related to herarchcal Bayesan accounts where the learnng rate at any gven level s proportonal to the precson of predctons and evolves under the nfluence of the next hgher level n the herarchy (Mathys et al., 2011). Ths weghtng by precson (a form of adaptve scalng) s crucal and has been descrbed for DA responses to reward (Tobler et al., 2005) and novelty (Bunzeck et al., 2010). Such a functon may generalze across neuromodulators: t has been suggested that both DA and ACh may be nvolved n the precson-weghtng of PEs (Frston, 2009; Frston et al., 2012). Here, we present behavoral and fmri studes that examne possble lnks between neuromodulatory systems and herarchcal precson-weghted PEs durng assocatve learnng. The Neuron 80, , October 16, 2013 ª2013 Elsever Inc. 519

2 A C B Fgure 1. Task Desgn and Model (A) Task desgn. Subjects had to predct wthn 800 ms (behavoral study), 1,000 ms (frst fmri study), or 1,200 ms (second fmri study) whch vsual stmulus (face or house) followed an audtory cue (hgh or low tone). In the behavoral study and frst fmri study, a monetary reward (0.05 or 5.00 Swss Francs con) was randomly presented n one of the four corners. The type of con presented was uncorrelated to vsual stmulus outcome and was omtted n the second fmri study. (B) Black: tme-varyng cue-outcome contngency, ncludng strongly predctve cues (probabltes of 0.9 and 0.1), moderately predctve cues (0.7, 0.3) and nonpredctve cues (0.5); red: example of a subject-specfc trajectory of the posteror expectaton of vsual category. (C) HGF: generatve model. x 1 represents the stmulus dentty (category), x 2 the cue-outcome contngency (the condtonal probablty of the vsual stmulus gven the audtory cue) n logt space, and x 3 represents the log-volatlty of the envronment. See Equatons 2, 3, and 4 and Table S2. See also Fgures S1, S2, and S3 and Tables S1, S2, S4, S5, and S6. analyses rest on a recently developed herarchcal Bayesan model, the Herarchcal Gaussan Flter (HGF) (Mathys et al., 2011), whch does not assume fxed deal learnng across subjects but contans subject-specfc parameters that couple the herarchcal levels and allow for ndvdual expresson of (approxmate) Bayes-optmal learnng. Usng the subject-specfc learnng trajectores, we examned whether actvty n neuromodulatory nucle could be explaned by precson-weghted PEs, and f so, at whch herarchcal level. In partcular, we focused on dopamnergc and cholnergc nucle, usng anatomcal masks specfcally developed for these regons. Importantly, we examned 118 healthy volunteers from three separate samples, two of whch underwent fmri (n = 45 and n = 27, respectvely). Ths enabled us to verfy the robustness of our results and test whch of them would replcate across samples. RESULTS We report fndngs obtaned from three separate samples of healthy volunteers undergong purely behavoral assessment (n = 46) or combned fmri-behavor (n = 45 and n = 27). All three studes used a smple assocatve audo-vsual learnng task where partcpants had to learn the tme-varyng predctve strengths of audtory cues and predct upcomng vsual stmul (faces or houses) by button press (Fgure 1). Ths task requred herarchcal learnng about stmulus occurrences, stmulus probabltes, and volatlty that we modeled as a herarchcal Bayesan belef updatng process, usng a standard HGF wth three levels (Mathys et al., 2011); see Expermental Procedures for detals. Modelng of Behavoral Data In a frst step, we used random effects Bayesan model selecton (BMS) (Stephan et al., 2009) to examne the possblty that our subjects mght have engaged n a dfferent cogntve process than ntended, or may have used a dfferent model than hypotheszed. In the behavoral study and frst fmri study, we tred to ensure constant motvaton of our partcpants by assocatng each tral wth a monetary reward whose potental pay-out at the end of the experment depended on successful predcton of the vsual outcome (face or house). Even though subjects were explctly nstructed that these reward were random and orthogonal to the vsual outcomes, one may wonder whether subjects learnng mght nevertheless have been drven by (mplct) predcton of these tral-wse reward. To exclude ths possblty, we compared a three-level HGF assumng that audo-vsual assocatons were learned and guded subjects behavor (HGF 1 ; Fgure 1C) to a second HGF 520 Neuron 80, , October 16, 2013 ª2013 Elsever Inc.

3 that assumed that partcpants attempted to learn and predct tral-wse reward (HGF 2 ). A second queston was whether our partcpants were ndeed engagng n herarchcal learnng and updatng ther learnng rate dynamcally, as our Bayesan model assumed, or used a smpler learnng mechansm. To clarfy ths, we added two more models to our comparson set. The models were a Bayesan model wth reduced herarchcal depth (HGF 3 ) n whch the thrd level was elmnated from the herarchy, and a standard Rescorla-Wagner (RL) model wth a fxed learnng rate. Fnally, we mplemented a RL model wth dynamc learnng rate (Sutton, 1992) that was recommended by one of the revewers as a non-bayesan alternatve to HGF 1. See the Supplemental Expermental Procedures secton C (avalable onlne) for more nformaton on these models. Comparng these fve models, we found that, across studes, HGF 1 was the superor model n 86 out of our 118 partcpants. Examnng each study separately, random effects BMS yelded posteror model probabltes of 84% (behavoral study), 74% (frst fmri study), and 72% (second fmri study) for HGF 1, whch was fve to ten tmes hgher than for the next best model n each case (Table S1). As a consequence, n each study, the exceedance probablty n favor of HGF 1 (.e., the probablty that ts posteror probablty was hgher than that of any other model consdered) (Stephan et al., 2009) was ndstngushable from 100%. These results provde strong evdence that our partcpants dd learn the task-relevant condtonal probabltes of vsual stmul (nstead of predctng the ncdental reward) and were capable of updatng ther learnng rate dynamcally. We next examned the estmates of the free parameters (k, w, z) from the wnnng model (Table S2). These estmates were comparable across the three studes, as demonstrated by ANOVA: none of the model parameters showed sgnfcant dfferences across studes (k: F(2,115) = 1.04, p = 0.358; w: F(2,115) = 0.91, p = 0.405; z: F(2,115) = 2.98, p = 0.055). Addtonally, we used multple regresson to evaluate how well our model explaned subjects behavor (percentage of correct responses). Ths quantfed model performance n terms of varance explaned, complementary to the relatve model comparson by BMS above. Ths analyss showed that the lnear combnaton of the three model parameters predcted subjects task performance well (behavoral study: R 2 = 0.64, F(3,42) = 25.3, p < 0.001; frst fmri study: R 2 = 0.59, F(3,41) = 20.1, p < 0.001; second fmri study: R 2 = 0.63, F(3,23) = 13.2, p < 0.001). fmri Data Analyss As detaled n the Expermental Procedures secton, our fmri analyss focused on precson-weghted PEs and uncertanty estmates across the herarchcal levels of the HGF. For each of these varables, our analyss proceeded n three steps (see Expermental Procedures): frst, we performed whole-bran analyses; second, we focused on our anatomcally defned regons of nterest (ROIs), usng a combned mask of dopamnergc and cholnergc nucle n the bran stem and subcortex; fnally, we conducted these fmri analyses separately n two ndependent samples of n = 45 and n = 27 volunteers. Note that we only report those fndngs that survved strngent famly-wse error (FWE) peak-level correcton for multple tests (p < 0.05) and that could be replcated across studes. Replcaton was assessed usng a voxel-wse logcal AND operaton on the FWE-thresholded actvaton maps from both fmri studes, and only those actvatons are beng reported n whch ths procedure showed an overlap of sgnfcant actvatons n both fmri studes. Low-Level Precson-Weghted Predcton Errors Intally, we examned the precson-weghted PE about vsual stmulus outcome, ε 2 (for mathematcal detals, see Expermental Procedures and the Supplemental Expermental Procedures, secton A). In both fmri studes, our whole-bran analyses demonstrated sgnfcant actvatons n a wdely dstrbuted set of regons (Table 1; Fgure 2). In addton to the vsual cortex (around the calcarne sulcus), the actvty of numerous supramodal regons correlated postvely wth tral-wse estmates of ε 2, ncludng the mddle and nferor frontal gyr, anteror cngulate cortex (ACC), ntraparetal sulcus (IPS), and anteror nsula, all located blaterally. Perhaps the most notable fndng, however, was a sgnfcant actvaton of the mdbran (ventral tegmental area [VTA]/substanta ngra [SN]). In both fmri studes, ths VTA/SN actvaton not only survved FWE correcton wthn our anatomcally defned mask, but also across the whole bran (p < 0.05; Fgure 3). Ths fndng s remarkable because the precson-weghted PE ε 2 concerns a purely sensory event: the vsual stmulus category predcted by the audtory cue. Ths concluson s supported by the BMS analyss of the behavoral data descrbed above that demonstrated that n the frst fmri study subjects were not tryng to predct reward but vsual outcomes. Furthermore, n the second fmri, study rewards were omtted entrely whle keepng sensory stmulaton and task demands dentcal. Interestngly, as mpled by predctve codng theores (cf. Frston, 2005), regons whose actvty correlated postvely wth PEs about vsual nputs consderably overlapped wth regons that actvated on each tral, regardless of the computatonal state and stmulus category ( task executon per se ). Fgure 4 shows the results of a nested conjuncton analyss: ths combned the conjuncton analyses of contrasts testng for task executon per se (.e., a statstcal contrast on the base regressor encodng tral events, not the parametrc modulators) and for ε 2, respectvely, across both fmri studes. These results ndcated that n both studes, prmary vsual cortex (calcarne sulcus), blateral IPS, rght dorsolateral prefrontal cortex (DLPFC), and rght anteror nsula were actvated by the task per se and by precsonweghted PEs about stmulus category. Please note that ths s an extremely conservatve analyss: all conjuncton analyses tested the conjuncton null hypothess,.e., a logcal AND (Nchols et al., 2005), wth all contrasts thresholded at p < 0.05 (FWE whole-bran corrected), and the combnaton of these conjunctons across both studes corresponded to a double logcal AND. The results reported so far refer to the outcome predcton error ε 2 ; ths s the (precson-weghted) dfference between the actual vsual stmulus outcome and ts a pror probablty (.e., before tral outcome observaton). However, we can also use the predctons from our model to examne actvatons reflectng choce predcton error ε ch ; ths s the dfference between the correctness of the subject s choce and the a pror probablty of ths Neuron 80, , October 16, 2013 ª2013 Elsever Inc. 521

4 Table 1. Whole-Bran Actvatons by ε 2 fmri study 1 Hemsphere x y z t Score fmri Study 2 Hemsphere x y z t Score ε 2 : Postve Correlaton ε 2 : Postve Correlaton Mddle frontal gyrus/ Anteror/ R Mddle frontal gyrus R mddle cngulate cortex Insula R Anteror/mddle cngulate cortex R Inferor paretal cortex R Insula R Precuneus R Inferor paretal cortex R Intraparetal sulcus/ L Precuneus R nferor paretal cortex Inferor frontal gyrus L Intraparetal sulcus/ nferor L paretal cortex Insula L Inferor frontal gyrus L Mddle frontal gyrus L Insula L Mddle frontal gyrus L Mddle frontal gyrus L Lngual gyrus L Mddle frontal gyrus L Lngual gyrus R Lngual gyrus L Supramargnal gyrus R Lngual gyrus R Cerebellum L Cerebellum L Mddle temporal gyrus R Supramargnal gyrus R VTA / substanta ngra R Mddle temporal gyrus R Prefrontal cortex L VTA / substanta ngra R Prefrontal cortex L All results: p < 0.05 FWE whole-bran corrected. MNI coordnates and t values for regons actvated by ε 2, the precson-weghted PE about stmulus outcome, n the frst and second fmri study. Only those actvatons are lsted that were replcated across studes. The actvaton n the frst row consttuted a sngle cluster n the frst study, whereas t was splt nto two separate clusters n the second study. choce beng correct (see the Supplemental Expermental Procedures, secton B, for formal defntons of both PEs). In both fmri studes, choce PEs evoked promnent actvatons (p < 0.05 FWE whole-bran corrected; Fgure 5) n numerous regons, ncludng the blateral ventral stratum, ventromedal prefrontal cortex, OFC and ACC (for a complete lst, see Table S7). Actvatons of these regons are commonly found for reward PEs, and t s remarkable that we obtan a smlar actvaton pattern even though n our studes learnng was orthogonal to reward (fmri study 1) and reward were absent (fmri study 2). Fnally, t s notable that the actvaton of the ventral stratum also extended nto the basal forebran, as delneated by our anatomcal mask (p < 0.05 FWE corrected for the entre mask volume). Hgh-Level Precson-Weghted Predcton Errors Subsequently, we nvestgated precson-weghted PEs at the next hgher level of the herarchy n our Bayesan model. Ths PE, ε 3, concerns the cue-outcome contngency,.e., the probablty (n logt space) of the vsual stmulus category gven the audtory cue, and s used to update estmates of log-volatlty at the thrd level of the HGF. We found that the tral-wse expresson of ths PE correlated postvely wth actvty n the septal part of the cholnergc basal forebran (Table 2; Fgure 6). In both fmri studes, ths actvaton was sgnfcant (p < 0.05) when corrected for multple comparsons across the volume of our anatomcally defned mask (that ncluded all cholnergc and dopamnergc nucle n bran stem and subcortex). DISCUSSION In ths study, three ndependent groups of healthy volunteers (n = 118 n total) performed an audo-vsual assocatve learnng task that requred explct predctons about an upcomng vsual stmulus category (face or house) gven a precedng audtory cue. Because the cue-outcome contngences were varyng unpredctably n tme, optmal performance requred herarchcal learnng about condtonal stmulus probabltes and ther change n tme. Our analyses showed that partcpants were ndeed lkely to engage n such a herarchcal learnng process. Formal statstcal comparson of fve alternatve models ndcated that a herarchcal Bayesan model (a three-level HGF) best explaned the observed behavoral data. Applyng the computatonal trajectores from ths model to fmri data, we found that precson-weghted PEs about vsual outcome, ε 2, were not only encoded by numerous cortcal areas, ncludng dopamnoceptve regons lke DLPFC, ACC, and nsula, but also by the dopamnergc VTA/SN. Notably, we verfed both statstcally and expermentally that these PE responses concerned vsual stmulus categores and not reward. At the hgher level of the model s herarchy, precson-weghted PEs about cue-outcome contngences (condtonal probabltes of the vsual outcome gven the audtory cue), ε 3, were reflected by actvty n the cholnergc basal forebran. Our fndngs have two mportant mplcatons. Frst, our results are n accordance wth a central noton n Bayesan theores of 522 Neuron 80, , October 16, 2013 ª2013 Elsever Inc.

5 A B C frst fmri study x = 3, y = 25, z = 47 second fmri study x = 0, y = 25, z = 47 conjuncton across studes x = 0, y = 25, z = 47 Fgure 2. Whole-Bran Actvatons by ε 2 Actvatons by precson-weghted predcton error about vsual stmulus outcome, ε 2, n the frst fmri study (A) and the second fmri study (B). Both actvaton maps are shown at a threshold of p < 0.05, FWE corrected for multple comparsons across the whole bran. To hghlght replcaton across studes, (C) shows the results of a logcal AND conjuncton, llustratng voxels that were sgnfcantly actvated n both studes. See Table S3 for deactvatons. bran functon, such as predctve codng (Frston, 2005; Rao and Ballard, 1999): even seemngly smple processes of perceptual nference and learnng do not rest on a sngle PE but rely on herarchcally related PE computatons. As a corollary, one would expect a wdespread expresson of PEs wthn the neuronal system engaged by a partcular task. Indeed, we found a remarkable overlap of areas nvolved n the executon of the task and areas expressng PEs (Fgure 4). Second, our fndngs suggest a potental dchotomy wth regard to the computatonal roles of DA and ACh. Accordng to our results, the mdbran may be encodng outcome-related PEs, ndependent of extrnsc reward. In contrast, the basal forebran may be sgnalng more abstract PEs that do not concern sensory outcomes per se but ther probabltes. In the followng, we wll dscuss these two mplcatons n the context of the prevous lterature. Snce early accounts of general systems theory and cybernetcs (Ashby, 1952), the noton of PE as a teachng sgnal for adaptve behavor has taken an ncreasngly central place n theores of bran functon. In contemporary neuroscence, PEs play a pvotal role n two frameworks, renforcement learnng (RL) and Bayesan theores. Studes nspred by RL have largely focused on the role of reward PEs, suggestng that these are encoded by phasc dopamne release from neurons n VTA/SN (Montague et al., 2004; Schultz et al., 1997). In humans, ths has been supported by fmri studes that have demonstrated the presence of reward PE sgnals n the VTA/SN (e.g., D Ardenne et al., 2008; Duk et al., 2013; Klen-Flügge et al., 2011) or n regons targeted by ts projectons, such as the stratum (Gläscher et al., 2010; McClure et al., 2003; Murray et al., 2008; O Doherty et al., 2003; Pessglone et al., 2006; Schonberg et al., 2010). Whle RL models have also been used to study PE-dependent learnng n the sensory doman (den Ouden et al., 2009; Law and Gold, 2009), a more prevalent framework to study percepton has been the Bayesan bran hypothess that the bran constructs and updates a generatve model of ts sensory nputs (Doya et al., 2011). One partcular formulaton of ths hypothess s predctve codng (Frston, 2005; Rao and Ballard, 1999) that postulates that PEs are weghted by ther precson and are computed at any level of herarchcally organzed nformaton processng cascades, as n sensory systems. Ths has been examned by several fmri studes that contrasted predctable versus unpredctable vsual stmul, fndng PE responses n vsual areas specalzed for the respectve stmul used (Harrson et al., 2007; Summerfeld and Koechln, 2008) and precson-weghtng under attenton (Kok et al., 2012). Other studes have used an explct model of tral-wse PEs, usng vsual (Egner et al., 2010) or audo-vsual assocatve learnng (den Ouden et al., 2010; den Ouden et al., 2009) paradgms. Notably, these studes dd not have explct readouts of subjects predctons and used relatvely smple modelng approaches: they ether descrbed mplct learnng processes (n the absence of behavoral responses) usng a delta-rule RL model (den Ouden et al., 2009; Egner et al., 2010), or dealt wth ndrect measures of predcton (e.g., reacton tmes) usng an deal Bayesan observer wth a fxed learnng trajectory across subjects (den Ouden et al., 2010). Our present study goes beyond these prevous attempts by (1) requrng explct tral-by-tral predctons, and (2) characterzng learnng va a herarchcal Bayesan model that provdes subjectand tral-specfc estmates of precson-weghted PEs at dfferent herarchcal levels of computaton. Based on these advances, the present study shows much more wdespread sensory PE responses than prevously reported. Replcated n two separate groups, these responses were not only found n the vsual cortex, but also n many supramodal areas n prefrontal, cngulate, paretal, and nsular cortex (Fgure 2). Whereas a dstrbuton of reward (Vckery et al., 2011) and value sgnals (FtzGerald et al., 2012) across the whole bran have recently been demonstrated n humans, ths has not yet been shown, to our knowledge, for PEs; n ths case, precson-weghted PEs about the sensory outcome (vsual stmul). Perhaps the most nterestng aspect of our fndngs on sensory outcome PEs, ε 2, was the sgnfcant actvaton of the mdbran. In humans, strong emprcal evdence exsts for DA nvolvement Neuron 80, , October 16, 2013 ª2013 Elsever Inc. 523

6 A frst fmri study B second fmri study C conjuncton z= -18 Fgure 3. Mdbran Actvaton by ε 2 Actvaton of the dopamnergc VTA/SN assocated wth precson-weghted predcton error about stmulus category, ε 2. Ths actvaton s shown both at p < 0.05 FWE whole-bran corrected (red) and p < 0.05 FWE corrected for the volume of our anatomcal mask comprsng both dopamnergc and cholnergc nucle (yellow). (A) Results from the frst fmri study. (B) Second fmri study. (C) Conjuncton (logcal AND) across both studes. n processng reward PEs (Montague et al., 2004; Schultz et al., 1997) and novelty (Bunzeck and Düzel, 2006). In anmal studes, dopamnergc mdbran responses to vsual stmul have been reported n the absence of reward; however, ths requred that the stmul were novel, arousng or physcally smlar to reward-related stmul (Horvtz, 2000; Redgrave and Gurney, 2006; Schultz, 1998). In contrast, n our study the VTA/SN responses scaled wth tral-by-tral precson-weghted PE about the stmulus category; these were nether reward-related, arousng nor novel (we kept repeatng two to four face and house stmul n each study). One could thnk of VTA/SN actvty reflectng condtonal novelty (Bayesan surprse); however, ths s not a tght lnk because ε 2 s only related but not dentcal to Bayesan surprse (see Supplemental Expermental Procedures). An mportant caveat s that we cannot clam wth certanty that the mdbran actvaton we found specfcally reflects the actvty of DA neurons n VTA/SN because ths regon s not homogenous n ts cellular composton and also contans glutamatergc and GABAergc neurons (Nar-Roberts et al., 2008). In partcular, our anatomcal mask does not dstngush pars compacta and pars retculars of the SN; the latter contans GABAergc neurons whose contrbuton to the blood oxygen level-dependent (BOLD) sgnal s not well understood (Logothets, 2008). Whle multmodal nvestgatons have demonstrated good correspondence between stratal DA release and BOLD sgnal n VTA/SN n response to reward PEs or novel stmul (see Düzel et al., 2009 for revew), ths relaton stll remans to be establshed for sensory PEs. Smlar caveats apply to our fndngs on the basal forebran, whch also contans other neurons than only cholnergc ones (Zaborszky et al., 2008). Wth ths caveat n mnd, our study suggests that n humans the dopamnergc mdbran may not only encode PEs about reward, but also precson-weghted PEs about purely sensory outcomes. To our knowledge, smlar mdbran actvatons have not been reported n prevous studes on reward-unrelated learnng (e.g., d Acremont et al., 2013; Gläscher et al., 2010). Notably, our experments were desgned to detect branstem actvatons, ncludng an optmzed fmri sequence and careful correcton for physologcal (cardac and respratory) nose. Last but not least, our studes had consderably larger sample szes, and consequently hgher statstcal power, than prevous fmri studes on reward-unrelated learnng. It s worth mentonng that the recent study by Ide et al. (2013), whch reports actvty for unsgned PEs (Bayesan surprse) n ACC durng a Go/NoGo task, does show a mdbran actvaton (ther Fgure 3); however, ths s not a sensory PE but reflects a man effect of stop versus go trals. Another recent fmri study (Payzan-LeNestour et al., 2013) on neuromodulatory mechansms durng learnng focused on dfferent forms of uncertanty and on the noradrenergc system but dd not report any fndngs related to PEs, nor to DA or ACh, as n ths study. In anmal studes, dsentanglng responses to sensory and reward aspects of stmul s often dffcult because stmulusbound reward are requred to mantan motvaton (Maunsell, 2004). In our study, however, the fndng of a sensory PE response n the mdbran cannot easly be explaned by any (hdden) reward effect snce we controlled for the potental nfluence of reward n two ways. In the frst fmri study, we orthogonalzed reward delvery to the task-relevant predctons about vsual stmul; addtonally, we verfed by model comparson that our subjects decsons were unlkely to be drven by reward predctons. In our second fmri study, we entrely omtted any reward, yet found exactly the same VTA/SN response to PEs about vsual stmul as n the frst fmri study (Fgure 3). Beyond PEs about vsual stmulus category, our herarchcal model also enabled us to examne hgher-level PEs. Specfcally, n both fmri studes, we found a sgnfcant actvaton of the cholnergc basal forebran by the precson-weghted PE ε 3 about condtonal probabltes (of the vsual stmulus gven the audtory cue) or, equvalently, cue-outcome contngences. Ths fndng provdes a new perspectve on possble computatonal roles of ACh. In the prevous lterature, the release of acetylcholne has been assocated wth a dverse range of functons, ncludng workng memory (Hasselmo, 2006), attenton (Demeter and Sarter, 2013), or learnng (Dayan, 2012; Doya, 2002). A recent nfluental proposal was that ACh levels may encode the degree of expected uncertanty (EU) (Yu and Dayan, 2002, 2005). Operatonally, EU was defned (n slghtly dfferent ways across artcles) n reference to a hdden Markov model representng the relaton between contextual states, cue valdty, and sensory events. Notably, Yu and Dayan (2002, 2005) mplctly defned EU as a hgh-level PE, n the sense that t represents the dfference between a condtonal probablty (degree of cue valdty) and certanty. Despte clear dfferences n the underlyng models, ths defnton s conceptually related to ε 3 n our model (see Supplemental Expermental Procedures, secton A, for detals) that we found was encoded by actvty n the basal forebran. Our emprcal fndngs thus complement the prevous theoretcal arguments by Yu and Dayan (2002, 2005), offerng a 524 Neuron 80, , October 16, 2013 ª2013 Elsever Inc.

7 A B C y=37 y=-39 y=42 y=-39 y=42 y=-39 y=20 y=20 y=20 conjuncton frst fmri study conjuncton second fmri study conjuncton of conjunctons Fgure 4. Overlap of Actvatons by Task Executon Per Se and ε 2 Conjuncton analyss ( logcal AND, conjuncton null hypothess) of the contrasts testng for tral events and for the precson-weghted predcton error about stmulus vsual outcome, ε 2. (A) Frst fmri study. (B) Second fmri study. (C) Results of a double conjuncton,.e., the conjuncton of the results from (A) and (B) across both studes. related perspectve on ACh functon by conceptualzng t as a precson-weghted PE about condtonal probabltes (cueoutcome contngences). The precson-weghtng of ths PE also relates our results on basal forebran actvaton to the prevous suggeston of a lnk between ACh and learnng rate (Doya, 2002). Ths s because, n ts numerator, c 3 (the precson weght of ε 3 ) contans an equvalent to a dynamc learnng rate (Preuschoff and Bossaerts, 2007) for updatng cue-outcome contngences (see Equaton A.10 n the Supplemental Expermental Procedures, secton A and Equaton 27 n Mathys et al., 2011). In summary, our fndngs are mportant n two ways. Frst, they provde emprcal support for the mportance of precsonweghted PEs as postulated by the Bayesan bran hypothess. Furthermore, they contrbute to the ongong debate about the computatonal roles of neuromodulatory transmtters (Dayan, 2012), suggestng a more general role for DA than only encodng reward-related PEs and provdng emprcal evdence for ACh nvolvement n representng hgher-order PEs (about condtonal probabltes). Our results are compatble wth the noton that multple neuromodulators may be nvolved n the precsonweghtng of PEs (Frston, 2009), but suggest separable roles for DA and ACh at dfferent herarchcal levels of learnng. In future analyses, we wll focus on elucdatng how these PEs may be used as teachng sgnals for synaptc plastcty (expressed through changes n effectve connectvty; cf. den Ouden et al., 2010). We hope that, eventually, ths work wll contrbute to establshng neurocomputatonal assays that allow for nference on neuromodulatory functon n the brans of ndvdual patents. If successful, ths could have far-reachng mplcatons for dagnostc procedures n psychatry and neurology (Maa and Frank, 2011; Moran et al., 2011; Stephan et al., 2006). EXPERIMENTAL PROCEDURES Subjects Ths artcle reports fndngs obtaned from three separate samples of healthy volunteers. The three studes used nearly dentcal expermental paradgms, enablng us to test whch results would survve replcaton, both n the presence of monetary reward (behavoral study and frst fmri study) and n ther absence (second fmri study). The frst sample contanng 63 male volunteers (mean age ± SD: 21 ± 2.2 years) was examned behavorally only. The second sample (48 male volunteers; 23 ± 3.1 years) and thrd sample (27 male volunteers; 21 ± 2.2 years) underwent both behavoral assessment and fmri (the thrd sample corresponded to the placebo group from a pharmacologcal study whose results wll be reported elsewhere). We only employed male partcpants to exclude varatons of hormonal effects on the BOLD sgnal durng the menstrual cycle. The partcpants were all nonsmokers, wthout any psychatrc or neurologcal dsorders n ther past medcal hstory and were not takng any medcaton. All three studes employed a near-dentcal audo-vsual assocatve learnng task (see below). Pror to data analyss, each subject s data was examned for nvald trals. These were defned as mssed responses or as trals wth excessvely long reacton tmes (late responses; >1,100 ms n the behavoral study, >1,300 ms n the frst fmri study, and >1,500 ms n the second fmri study). Subjects wth more than 20% nvald trals or less than 65% correct responses were excluded from further analyses. These crtera led to the excluson of 17 partcpants n the behavoral study and three partcpants n the frst fmri study; no partcpants were excluded from the second fmri study. As a consequence, the fnal data analyss ncluded 46 subjects from the behavoral study (21 ± 2.3 years), 45 subjects from the frst fmri study (23 ± 3.0 years), and 27 subjects from the second fmri study (21 ± 2.2 years). All partcpants gave wrtten nformed consent before the study, whch had receved ethcs approval by the local responsble authortes (Kantonale Ethkkommsson, KEK /3 for the behavoral and frst fmri study, KEK /3 for the second fmri study). Expermental Desgn: Assocatve Learnng Task A cross-modal assocatve learnng task (audo-vsual stmulus-stmulus learnng [SSL]) was used n all three studes (Fgure 1) where partcpants had to learn the predctve strength of audtory cues and predct a subsequent vsual stmulus. Notably, ths predcton was explct and ndcated by button press before the vsual stmulus appeared. The task desgn was near-dentcal n all three studes; the only varatons concerned: (1) response nterval (800 ms n the behavoral study, 1,000 ms and 1,200 ms n the frst and second fmri studes), (2) duraton of the vsual outcome presentaton (150 ms n the behavoral and frst fmri study, 300 ms n the second fmri study), and (3) the presence or absence of tral-wse monetary reward (see below). Stmul were presented usng Cogent2000 ( Cogent/ndex.html). Trals were presented wth a randomzed ntertral nterval Neuron 80, , October 16, 2013 ª2013 Elsever Inc. 525

8 A B C frst fmri study x = 0, y = 9, z = -9 second fmri study x = 0, y = 9, z = -9 conjuncton across studes x = 0, y = 9, z = -9 Fgure 5. Choce Predcton Error Actvatons by choce predcton error, ε ch, n the frst (A) and the second fmri study (B). Both actvaton maps are shown at a threshold of p < 0.05, FWE corrected for multple comparsons across the whole bran. To hghlght replcaton across studes, (C) shows the results of a logcal AND conjuncton, llustratng voxels that were sgnfcantly actvated n both studes. See also Table S7. (ITI) of s. At the begnnng of each tral, partcpants heard one of two possble audtory cues for 300 ms, a hgh (576 Hz) or a low tone (352 Hz). To ensure that both tones were perceved equally loudly, subjects performed an ntal psychophyscal matchng task n whch they had to adapt the volumes untl they perceved both cues as equally loud (cf. den Ouden et al., 2010). Followng the cue, partcpants had to sgnal ther predcton by button press (rght ndex and mddle fnger), as quckly and as accurately as possble, whch of two possble vsual outcome categores (houses and faces) would follow. These comprsed a small subset of stmul (two to four) from our prevous work (den Ouden et al., 2010). Crtcally, n our task the cue-outcome assocaton strength changed over tme (.e., reversal learnng), ncludng strongly predctve (probabltes of 0.9 and 0.1), moderately predctve (0.7, 0.3), and nonpredctve cues (0.5). Each subject completed 320 trals, dvded nto ten blocks of dfferent assocaton strengths. Our stmulus sequence (Fgure 1B) had two key features: both block length (24 to 40 trals) and magntude of changes n cue-outcome contngency vared unpredctably across blocks. Over the experment, ths led to changes n two related varables of nterest: (1) volatlty, and (2) precson-weghted predcton error about cue-outcome contngency ε 3 (a proxy to expected uncertanty ; see Dscusson). Please note that n our modelng framework, there s a formal connecton between the concepts of volatlty and expected uncertanty: ε 3 depends on the prevous estmate of log-volatlty m 3 ; n turn, ε 3 determnes the updatng of m 3 (see Equatons A.10 and A.11 n the Supplemental Expermental Procedures). The probablty sequence was pseudorandom and fxed across subjects to ensure comparablty of the nduced learnng process and thus model parameter estmates. Subjects were nformed n whch range the probabltes could change but not about ther order or possble values. Also, as n prevous work (den Ouden et al., 2010), they were explctly nstructed that the condtonal probabltes were coupled as follows (f: face; h: house; = [ : hgh tone; = Y : low tone): pðfj = [Þ = 1 pðhj = [Þ = pðhj = YÞ = 1 pðfj = YÞ: (Equaton 1) We ensured that the margnal probabltes of face and house outcomes were dentcal across the experment and could thus not bas the partcpants predctons. Ths was acheved by requrng that (1) the probablty of one outcome gven a partcular cue was the same as the probablty of the other outcome gven the other cue (Equaton 1), and (2) n each block, both cue types appeared equally often and n random order. Wth these two manpulatons, we ensured that, on average, before the cue was presented, the a pror probablty of a face or a house occurrng was 50% each. Thus, on any gven tral, t was not possble to make an nformed predcton about the outcome before havng heard the cue. In the behavoral study and frst fmri study, each tral was assocated wth a potental monetary reward. Specfcally, at the end of each tral the vsual outcome was presented for 150 ms n the center of the mage, together wth a con (5 CHF or 0.05 CHF) randomly located n one of the corners (Fgure 1A). Crtcally, reward sze was uncorrelated to the vsual outcome to be predcted. In other words, hgh and low reward appeared randomly on 50% of the trals each, ensurng that any cue would predct any reward wth 50% probablty. At the end of the experment, we appled a smple pay-out rule: 100 lowrewardng trals and one hgh-rewardng tral were randomly chosen, and the summed reward from correct trals only was pad out (note that the maxmal possble net value for both low- and hgh-reward trals was dentcal,.e., 5 CHF). Ths procedure was used to motvate the partcpants to delver constantly hgh performance throughout the experment: by mnmzng the number of ncorrect predctons about the vsual outcome, partcpants would maxmze ther expected total reward. Although we nstructed our partcpants explctly that the reward sequence was random and could not be learned, one mght wonder whether some subjects mght nevertheless have tred to predct upcomng reward nstead of vsual outcomes. We therefore also modeled any putatve learnng of the orthogonal reward and performed model comparson to quantfy whether predctons of vsual outcomes or reward would better explan the subjects observed behavor (see below). Fnally, n the second fmri study, we omtted reward. Ths enabled us to examne expermentally whether behavor and fmri actvatons would reman dentcal when monetary reward were absent. Herarchcal Gaussan Flter For behavoral data analyss, we appled a Herarchcal Gaussan Flter (HGF) that descrbes learnng at multple levels and allows for nference on an agent s belef about the causes of ts sensory nputs (Mathys et al., 2011). The HGF rests on a varatonal approxmaton to deal herarchcal Bayes, whch conveys two major advantages. Frst, the HGF allows for ndvdualzed Bayesan learnng: t contans subject-specfc parameters that couple the dfferent levels of the herarchy and determne the ndvdual learnng process. Second, the update equatons are analytc and contan renforcement learnng as a specal case, wth precson-weghted predcton errors (PEs) drvng belef updatng at the dfferent levels of the herarchcal model (see below). Here, we mplemented a three-level HGF as descrbed by Mathys et al. (2011) and summarzed by Fgure 1C, usng the HGF Toolbox v2.1 that s avalable as open source code ( The frst level of ths model represents a sequence of envronmental states x 1 (here: whether a face or house was presented), the second level represents the 526 Neuron 80, , October 16, 2013 ª2013 Elsever Inc.

9 Table 2. Basal Forebran Actvatons by ε 3 fmri Study 1 X y z t Score fmri Study 2 x Y z t Score ε 3 : Postve Correlaton ε 3 : Postve Correlaton Basal forebran Basal forebran MNI coordnates and t values for regons actvated by ε 3, the precson-weghted PE about stmulus probablty n the frst and second fmri study. Only those actvatons are lsted that were replcated across studes. cue-outcome contngency x 2 (.e., the condtonal probablty, n logt space, of the vsual target gven the audtory cue), and the thrd level the log-volatlty of the envronment x 3. Each of these hdden states s assumed to evolve as a Gaussan random walk, such that ts varance depends on the state at the next hgher level (Fgure 1C): p pðx 1 jx 2 Þ = sðxþ x 1 ð1 sðx 2 ÞÞ 1 x 1 = Bernoullðx 1 ; sðx 2 ÞÞ; (Equaton 2) 2 ; x ðkþ 3 x ðkþ 2 x ðk 1Þ p x ðkþ 3 x ðk 1Þ = N x ðkþ 2 ; xðk 1Þ 2 ; exp kx ðkþ 3 + u ; (Equaton 3) 3 ; w = N x ðkþ 3 ; xðk 1Þ 3 ; w ; (Equaton 4) where s($) s a sgmod functon. In Equatons 2, 3, and 4, w determnes the speed of learnng about the logvolatlty of the envronment; k determnes how strongly the second and thrd levels are coupled and thus how much the estmated envronmental volatlty affects the learnng rate at the second level; and u s a constant component of the step sze at the second level. Fnally, the predcted probablty of a vsual target gven the audtory cue (.e., the posteror mean of x 2 ) s lnked to tral-wse predctons of vsual stmulus category by means of a softmax functon wth parameter z (encodng decson nose). Our three-level HGF for categorcal outcomes thus has four parameters. In our mplementaton, three of them were free (w, k, z), whereas u was fxed to 4 n our analyses n order to ensure model dentfablty. Importantly, the varatonal approxmaton underlyng the HGF provdes analytc update equatons that share a general form: At any level of the herarchy, the update of the belef on tral k (.e., posteror mean m ðkþ of the state x ) s pro-. Ths weghted PE from the level below and a precson rato jðkþ portonal to the precson-weghted predcton error (PE) ε ðkþ s the product of the PE d ðkþ 1 m ðk + 1Þ m ðkþ : fj ðkþ d ðkþ 1 = εðkþ ; (Equaton 5) j ðkþ = bpðkþ 1 p ðkþ ; (Equaton 6) where bp ðkþ 1 represents the precson of the predcton about nput from the level below and p ðkþ encodes the precson of the belef at the current level. The form of ths general update equaton s remnscent of RL models. Specfcally, the precson-weghtng can be understood as (component of) a dynamc learnng rate (cf. Preuschoff and Bossaerts, 2007); see Mathys et al. (2011) and secton A of the Supplemental Expermental Procedures for detals. In our three-level HGF, two precson-weghted PEs ε occur. At the second level, ε 2 s the precson-weghted PE about vsual stmulus outcome that serves to update the estmate of x 2 (the cue-outcome contngency n logt space). At the thrd level, ε 3 s the precson-weghted PE about cue-outcome contngency that s proportonal to the update of x 3 (envronmental log-volatlty). These are the two quanttes of nterest that the fmri analyses n ths artcle focus on. For the exact equatons, see the Supplemental Expermental Procedures, secton A. fmri Data Acquston and Analyss The experment was conducted on a 3T Phlps Acheva MR Scanner at the SNS Lab, usng an eght channel SENSE head-col. Structural mages were acqured usng a T 1 -weghted sequence. For functonal magng, 500 whole-bran mages were acqured n the frst fmri study and 550 mages n the second fmri study, usng a T 2 *-weghted echo-planar magng sequence that had been optmzed for bran stem magng (slce thckness: 3 mm; n-plane resoluton: mm; nterslce gap: 0.6 mm; ascendng contnuous n-plane acquston; TR = 2,500 ms; TE = 36 ms; flp angle = 90 ; feld of vew = mm; SENSE factor = 2; EPI factor = 51). In order to reduce feld nhomogenetes a second order pencl-beam volume shm (provded by Phlps) was appled durng the functonal acquston. Functonal data acquston lasted 21 mn. Durng fmri data acquston, respratory and cardac actvty was acqured usng a breathng belt and an electrocardogram, respectvely. fmri data were analyzed usng statstcal parametrc mappng (SPM8). Followng moton correcton of the functonal mages and coregstraton to the structural mage, we warped both functonal and structural mages to MNI space usng the New Segment toolbox n SPM; see Appendx A n Ashburner and Frston (2005). The functonal mages were smoothed applyng a 6 mm full-wdth at half maxmum Gaussan kernel and resampled to 1.5 mm sotropc resoluton. In order to optmze sgnal-to-nose rato for crtcal regons such as the bran stem, we corrected for physologcal nose usng RETROICOR (Glover et al., 2000) based on an n-house mplementaton (Kasper et al., 2009) (open source code avalable at translatonalneuromodelng.org/tapas). For fmri data analyss, we specfed a voxel-wse general lnear model (GLM) for each partcpant. In the frst fmri study, ths GLM reflected a factoral desgn wth vsual outcome category (face, house) and ncdental reward stmulus (hgh, low) as factors. In the second fmri study, reward stmul were absent; therefore, the GLM only contaned the two vsual outcome condtons. Addtonally, we modeled mssed and late responses, respectvely, by separate regressors. All regressors were convolved wth a canoncal hemodynamc functon and ts temporal dervatve. The subject-specfc belef trajectores, obtaned from the HGF, were used n the GLM as parametrc modulators. These varables ncluded (cf. Equatons 2, 3, 4, 5, and 6; Fgures S1 and S2): (1) ε 2, the precson-weghted PE about vsual stmulus outcome (that serves to update the estmate of vsual stmulus probabltes n logt space); (2) ε 3, the precson-weghted PE about cue-outcome contngency (that serves to update the estmate of log-volatlty); (3) c 2, precson weght at the second level; ths corresponds to the learnng rate by whch estmates of cue-outcome contngency are updated; (4) c 3, precson weght at the thrd level; ths s proportonal to the learnng rate by whch log-volatlty estmates are updated; (5) m 3, the predcted log-volatlty; and (6) ε ch, the choce predcton error. Importantly, choce PE ε ch and precson-weghted outcome PE ε 2 have dstnct defntons (see sectons A and B of the Supplemental Expermental Procedures for mathematcal detals). The choce PE ε ch s the dfference between the correctness of the subject s choce (1 f choce was correct, 0 otherwse) and the a pror probablty of ths choce beng correct. Ths PE s postve when the subject s choce was correct and negatve when t was wrong. In contrast, ε 2 multples two components (Equatons 5 and 6): (1) the precson weght j ðkþ (that s always postve), and (2) d 1, the dfference between the actual vsual stmulus outcome and ts a pror probablty (also always postve); the latter corresponds to Bayesan surprse and s bounded between 0 and 1. Neuron 80, , October 16, 2013 ª2013 Elsever Inc. 527

10 A B frst fmri study Contrasts of nterest testng for each of the parametrc modulators specfed above were defned at the frst level and entered nto second level ANOVAs to allow for nference at the group level. We tested for both postve and negatve effects of our parametrc modulators. Please note that we only report results that (1) survved strngent famly-wse error correcton (FWE) at the voxel level (p < 0.05), based on Gaussan random feld theory (Worsley et al., 1996), across the whole bran and wthn ROIs, respectvely, and (2) were replcated n both fmri studes. Replcablty was assessed by testng the conjuncton null hypothess,.e., a voxel-wse logcal AND analyss (Nchols et al., 2005). In the man text of ths artcle, we focus on actvatons related to predcton errors; for other fndngs related to the remanng regressors, see Supplemental Expermental Procedures (Fgure S3; Tables S3, S4, S5, and S6). C second fmri study conjuncton across studes Bayesan Model Selecton To dsambguate alternatve explanatons (models) for the partcpants behavor, we used Bayesan model selecton (BMS). BMS s a standard approach n machne learnng and neuromagng (MacKay, 1992; Penny et al., 2004) for comparng competng models that descrbe how neurophysologcal or behavoral responses were generated. BMS evaluates the relatve plausblty of competng models n terms of ther log-evdences. The log-evdence of a model corresponds to the negatve surprse about the data, gven the model, and quantfes the trade-off between accuracy (ft) and complexty of a model. Here, we used a recently developed random effects BMS method to account for potental nterndvdual varablty n our sample (Penny et al., 2010; Stephan et al., 2009), quantfyng the posteror probabltes of fve competng models (see Results and Supplemental Expermental Procedures for detals). Fgure 6. Basal Forebran Actvatons by ε 3 Actvaton of the cholnergc basal forebran assocated wth precsonweghted predcton error about stmulus probabltes ε 3 wthn the anatomcally defned mask. For vsualzaton of the actvaton area we overlay the results thresholded at p < 0.05 FWE corrected for the entre anatomcal mask (red) on the results thresholded at p < uncorrected (yellow) n the frst (A: x = 3, y = 9, z = 8) and the second fmri study (B: x = 0, y = 10, z = 8). (C) The conjuncton analyss ( logcal AND ) across both studes (x = 2, y = 11, z = 8). Importantly, the GLM used all computatonal trajectores n ther orgnal form, wthout any orthogonalzaton. Thus, we dd not mpose any judgment on the relatve mportance of regressors for explanng the fmri data. Also, the tmngs of our events were chosen such that PE estmates were tmelocked to the vsual outcome at the end of the tral; predcton and precson regressors spanned the entre tral and changed at outcome, accordng to the update nduced by the PE. Our subject-specfc (frst-level) GLM also ncluded regressors representng potental confounds. Ths ncluded the realgnment parameters (encodng head movements) and ther frst dervatve, a regressor markng scans wth >1 mm scan-to-scan head movement, and physologcal confound varables (cardac actvty and breathng), provded by RETROICOR. In addton to whole-bran analyses, we performed ROI analyses based on anatomcal masks of dopamnergc and cholnergc nucle. These ncluded (1) the dopamnergc mdbran (SN and VTA), (2) the cholnergc basal forebran, (3) cholnergc nucle n the tegmentum of the branstem,.e., the pedunculopontne tegmental (PPT) and laterodorsal tegmental (LDT) nucle. For the VTA/SN, we used an anatomcal atlas based on magnetzaton transferweghted structural MR mages (Bunzeck and Düzel, 2006). The basal forebran was defned usng the maxmum probablty map from a probablstc cytoarchtectonc atlas warped nto MNI space (Eckhoff et al., 2005; Zaborszky et al., 2008). Ths map ncluded the dfferent compartments of the basal forebran wth cholnergc neurons (septum, the dagonal band of Broca, and subpalldal regons ncludng the basal nucleus of Meynert). Gven the lack of a publshed atlas for PPT and LDT, we used MRICron to manually trace the regon of these nucle accordng to anatomcal landmarks from the lterature (Nadch et al., 2009; Zrnzo et al., 2011). Note that we dd not use these anatomcal masks separately to test for actvatons; nstead, all regons mentoned above were combned nto a sngle mask mage, and each ROI analyss used ths combned mask for multple comparson correcton. SUPPLEMENTAL INFORMATION Supplemental Informaton ncludes Supplemental Expermental Procedures, three fgures, and seven tables and can be found wth ths artcle onlne at ACKNOWLEDGMENTS We acknowledge support by the Zurch Neuroscence Centre (S.I., K.E.S.), the René and Susanne Bragnsky Foundaton (K.E.S.), KFSP Molecular Imagng, and SystemsX.ch (K.E.S.). We are very grateful to Smon Eckhoff and Emrah Düzel for provdng us wth the anatomcal masks for delneatng the basal forebran and VTA/SN, respectvely. Accepted: September 3, 2013 Publshed: October 16, 2013 REFERENCES Ashburner, J., and Frston, K.J. (2005). Unfed segmentaton. Neuromage 26, Ashby, W.R. (1952). Desgn for a Bran. (London: Chapman & Hall). Behrens, T.E., Woolrch, M.W., Walton, M.E., and Rushworth, M.F. (2007). Learnng the value of nformaton n an uncertan world. Nat. Neurosc. 10, Bunzeck, N., and Düzel, E. (2006). Absolute codng of stmulus novelty n the human substanta ngra/vta. Neuron 51, Bunzeck, N., Dayan, P., Dolan, R.J., and Duzel, E. (2010). A common mechansm for adaptve scalng of reward and novelty. Hum. Bran Mapp. 31, d Acremont, M., Fornar, E., and Bossaerts, P. (2013). Actvty n nferor paretal and medal prefrontal cortex sgnals the accumulaton of evdence n a probablty learnng task. PLoS Comput. Bol. 9, e D Ardenne, K., McClure, S.M., Nystrom, L.E., and Cohen, J.D. (2008). BOLD responses reflectng dopamnergc sgnals n the human ventral tegmental area. Scence 319, Neuron 80, , October 16, 2013 ª2013 Elsever Inc.