Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding

Transcription

1 Dsentanglng the Roles of Approach, Actvaton and Valence n Instrumental and Pavlovan Respondng Quentn J. M. Huys 1,2,3 *, Roshan Cools 4, Martn Gölzer 5, Eva Fredel 5, Andreas Henz 5, Raymond J. Dolan 1, Peter Dayan 2 1 Wellcome Trust Centre for Neuromagng, Unversty College London, London, Unted Kngdom, 2 Gatsby Computatonal Neuroscence Unt, Unversty College London, London, Unted Kngdom, 3 Medcal School, Unversty College London, London, Unted Kngdom, 4 Donders Insttute for Bran, Cognton and Behavour, Centre for Cogntve Neuromagng, Radboud Unversty Njmegen, Njmegen, Netherlands, 5 Charté Unverstätsmedzn Berln, Campus Charté Mtte, Berln, Germany Abstract Hard-wred, Pavlovan, responses elcted by predctons of rewards and punshments exert sgnfcant benevolent and malevolent nfluences over nstrumentally-approprate actons. These nfluences come n two man groups, defned along anatomcal, pharmacologcal, behavoural and functonal lnes. Investgatons of the nfluences have so far concentrated on the groups as a whole; here we take the crtcal step of lookng nsde each group, usng a detaled renforcement learnng model to dstngush effects to do wth value, specfc actons, and general actvaton or nhbton. We show a hgh degree of sophstcaton n Pavlovan nfluences, wth appettve Pavlovan stmul specfcally promotng approach and nhbtng wthdrawal, and aversve Pavlovan stmul promotng wthdrawal and nhbtng approach. These nfluences account for dfferences n the nstrumental performance of approach and wthdrawal behavours. Fnally, although losses are as nformatve as gans, we fnd that subjects neglect losses n ther nstrumental learnng. Our fndngs argue for a vew of the Pavlovan system as a constrant or pror, facltatng learnng by allevatng computatonal costs that come wth ncreased flexblty. Ctaton: Huys QJM, Cools R, Gölzer M, Fredel E, Henz A, et al. (2011) Dsentanglng the Roles of Approach, Actvaton and Valence n Instrumental and Pavlovan Respondng. PLoS Comput Bol 7(4): e do: /journal.pcb Edtor: Antono Rangel, Calforna Instute of Technology, Unted States of Amerca Receved November 12, 2010; Accepted February 22, 2011; Publshed Aprl 21, 2011 Copyrght: ß 2011 Huys et al. Ths s an open-access artcle dstrbuted under the terms of the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal author and source are credted. Fundng: Ths work was supported by a Wellcome Trust Grant and a Max Planck Award to RJD. PD was supported by the Gatsby Chartable Foundaton. Publcaton costs were shared between the Max Planck Award and the Gatsby Chartable Foundaton. The funders had no role n study desgn, data collecton and analyss, decson to publsh, or preparaton of the manuscrpt. Competng Interests: The authors have declared that no competng nterests exst. * E-mal: qhuys@cantab.net Introducton The functonal archtecture of respondng nvolves two fundamental components that are behavourally [1] and computatonally [2] separable: Pavlovan and nstrumental. The nstrumental component respects the stmulus-dependent contngency between responses and ther outcomes (stmulus-response and acton-outcome learnng) [3]. By contrast, preparatory Pavlovan responses, chefly nvolvng approach and wthdrawal, are elcted by the appettve or aversve valence assocated wth predctve stmul n a manner that s not dependent on the consequences of those responses [3 5]. The nteractons between the two systems are most evdent when automatcally-elcted Pavlovan responses nterfere wth contngent nstrumental respondng [1,6 9]. For nstance, pgeons wll strkngly contnue to peck at a lght predctve of food (a preparatory approach elcted by the appettve predcton), even f the food s wthheld every tme they peck the lght (the nstrumental contngency) [10,11]. Pavlovan nterference lkely contrbutes to many qurks of behavour such as mpulsvty [12], framng and [13], endowment effects [14] and many other anomales [15], ncludng neurologcal [16 19] and psychatrc dseases [20 26]. Further, puzzlng facets of seemngly purely nstrumental behavour such as the dffcultes n learnng go responses to avod punshments; or nogo to obtan rewards (unpublshed data) and even the restrctons n assocatons evdent n evolutonarly preparedness [27,28] mght be traced to Pavlovan prncples. However, nstrumental and Pavlovan systems share overlappng neural hardware. Ther bdrectonal nteracton s charactersed by two key trads: rewards are ted to approach and vgour; and punshments to wthdrawal and behavoural nhbton. The neuromodulator dopamne (DA) responds predomnantly to rewards [22,29 31], nduces behavoural actvaton and enhances approach [32 35]. Each aspect of ths trad confounds the role of the phasc DA bursts n the flexble acquston of nstrumental values [36 42]. Serotonn appears to le at the heart of the aversve trad, havng been lnked to punshments [43 45], behavoural nhbton and wthdrawal [25,32,46 52], although dopamne actng va D2 receptors lkely also plays a role n lnkng absence of rewards to nogo [17,53,54]. Sgnatures of both trads are also evdent n neural crcuts nvolved n response and choce. In the dorsal stratum, there are nterdgtated pathways for go and nogo, wth the go pathways agan lnked postvely to rewards va dopamne [16,18,55,56]. The ventral stratum s prmarly organzed along an appettve/aversve axs wth drect lnks to approach and wthdrawal behavours [57,58]. The aversve trad s also tghtly lnked to the dorsal raphé and the peraquaeductal gray [59,60]. The man routes to the scentfc nvestgaton of these nteractons conssts of tasks n whch Pavlovan stmul are presented durng ongong nstrumental tasks. However, these have PLoS Computatonal Bology 1 Aprl 2011 Volume 7 Issue 4 e

2 Author Summary Beautful background musc n a shop may well tempt us to buy somethng we nether need nor want. Valenced stmul have broad and profound nfluences on ongong choce behavour. After replcatng known fndngs whereby approach s enhanced by appettve Pavlovan stmul and nhbted by aversve ones, we extend ths to wthdrawal behavours, but crtcally controllng for the valence of the wthdrawal behavours themselves. We fnd that even when wthdrawal s appettvely motvated, t s stll nhbted by appettve Pavlovan stmul and enhanced by aversve ones. Ths shows, for the frst tme, that the effect of background Pavlovan stmul depends crtcally on the ntrnsc valence of behavours, and dffers between approach and wthdrawal. as yet not explored the full set of nteractons charactersng the overlap between the two systems. Two crtcal confounds reman: The frst confound concerns the precse nature of the effect of Pavlovan stmul on nstrumental behavours. The nstrumental behavours studed have largely been appettvely motvated approach behavours (n Pavlovan-Instrumental Transfer (PIT) and condtoned suppresson tasks, [1,6 8,61 63]), and one nstance of aversvely motvated wthdrawal behavour [64]. The relatve role of the appettve-aversve motvaton axs versus that of the approach-wthdrawal axs s unknown. Ths n turn obscures the nature of the nteracton: whether Pavlovan stmul nteract wth the value of the nstrumental behavour, or by promotng specfc responses [1], or even smply by modulatng behavoural actvaton [5]. Second, the extent to whch the separaton of reward and punshment processng nto opponent motvatonal structures apples to nstrumental as well as Pavlovan learnng s ncompletely explored [1,27,28,65]. All these ssues can smultaneously be addressed n a combned PIT and condtoned suppresson task wth both approach and wthdrawal actons n whch the overall motvatonal component of approach and wthdrawal are matched (Fgure 1 and Table 1). The task separates the contrbutons of approach and wthdrawal by usng two counterbalanced blocks, one nvolvng approach go versus nogo, and the other wthdrawal go versus nogo. The comparson between go and nogo controls for effects of behavoural actvaton or nhbton. In each block, subjects frst underwent bref nstrumental tranng (Fgure 1A), learnng from postve and negatve feedback (monetary gans and losses of J0.20) whether to produce a go or a nogo response assocated wth sortng mushrooms. In the approach block (Fgure 1A, top, all 46 subjects), go responses nvolved movng the cursor onto a mushroom (to collect t), whle nogo nvolved dong nothng, thus not collectng the mushroom. To test for the effect of low-level motor varables, subjects performed one of two types of wthdrawal actons. In throwaway (24 subjects, Fgure 1A, mddle), go nvolved movng the cursor physcally away from the mushroom and clckng nto an empty blue box; nogo nvolved dong nothng, and thus keepng the mushroom. Importantly, both approach to and wthdrawal from the nstrumental stmulus were orthogonal to any approach and wthdrawal that mght be drected at the Pavlovan background stmulus. In release (22 subjects, Fgure 1. Task descrpton. A: Instrumental tranng. To centre the cursor, subjects clcked n a central square. In approach trals (top), subjects chose whether to move the cursor towards the mushroom and clck nsde the blue frame onto the mushroom (go), or not do anythng (nogo). In throwaway wthdrawal trals (mddle), they nstead moved the cursor away from the mushroom and clcked n the empty blue frame (go) or dd nothng (nogo). In release wthdrawal trals (bottom), subjects were nstructed to keep the button pressed after the ntal clck n the central square. The mushroom was then presented centrally, under the cursor. To throw away the mushroom, subjects released the button. Outcomes were presented mmedately after go actons, or after 1.5 seconds. B: Pavlovan tranng. Subjects passvely vewed stmul and heard audtory tones, followed by wns and losses. C: On Pavlovan query trals, subjects chose between two Pavlovan stmul. No outcomes were presented, but they were counted and added to the total presented at the end of the experment. D: Pavlovan-nstrumental transfer. Subjects responded to nstrumental stmul wth Pavlovan stmul tlng the background. No outcomes were presented, but subjects were nstructed that ther choces counted towards the fnal total. No explct nstructons about the contrbuton of Pavlovan stmul towards the fnal total were gven. do: /journal.pcb g001 PLoS Computatonal Bology 2 Aprl 2011 Volume 7 Issue 4 e

3 Table 1. Expermental layout. Approach Block A1 Instrumental tranng (60 trals) Probablstc renforcements 1 : J s I 1,2,3?approach p(rewjgo,s I 1,2,3 )~0:7, p(punjgo,si 1,2,3 )~0:31 s I 4,5,6?nogo p(rewjnogo,s I 4,5,6 )~0:7, p(punjnogo,si 4,5,6 )~0:31 A2 Pavlovan tranng (60 trals) Determnstc renforcements s P zz?reward s P z?reward s P 0? 1 J 0.10 J 0 s P {?punshment s P {{?punshment J 21 J A3 PIT (100 trals) No Renforcements Wthdrawal Block s P s I 1{6?? W1 Instrumental tranng (60 trals) Probablstc renforcements 1 : +0:20 J s I 7,8,9?wthdraw p(rewjgo,s I 7,8,9 )~0:7, p(punjgo,si 7,8,9 )~0:31 s I 10,11,12?nogo p(rewjnogo,s I 10,11,12 )~0:7, p(punjnogo,si 10,11,12 )~0:31 W2 Pavlovan tranng (60 trals) Determnstc renforcements s P zz?reward s P z?reward s P 0? 1 J 0.10 J 0 s P {?punshment s P {{?punshment J 21 J W3 PIT (100 trals) No Renforcements s P s I 7{12?? Note the numercal subscrpts on the nstrumental stmul s I here refer to ther denttes, not to the tme of presentaton. 1 For subject wth determnstc nstrumental renforcements, the outcome probabltes were 1 and 0 nstead of 0.7 and 0.3, respectvely. do: /journal.pcb t001 Fgure 1A, bottom), the subjects had to start by pressng the mouse button. Go nvolved releasng the button to avod collectng the mushroom; nogo nvolved contnung to press the button and thereby recevng the mushroom. In order to orthogonalse the approach-wthdrawal and appettve-aversve axes, the learned nstrumental values n approach and wthdrawal blocks needed to be matched. To acheve ths, both go and nogo responses were, f correct, rewarded. Addtonally, to avod the confound of actvaton, n each block (.e. n both approach and wthdrawal blocks) the go acton was desgnated as the correct response to half the nstrumental stmul, and the nogo acton to the other half (see Table 1). Incorrect responses had opposte outcome contngences to correct responses, yeldng more punshments than rewards. Ths ensured that go, nogo, approach and wthdrawal overall had the same learned assocaton wth rewards and punshments. We tested both determnstc and probablstc outcomes but found no dfferences. In the second part of each block, subjects passvely vewed unrelated, fractal, stmul pared wth separate audtory tones (Fgure 1B). Each compound Pavlovan stmulus s P was determnstcally assocated wth a monetary gan or loss,.e. ts Pavlovan value V(s P ) was equal to that monetary outcome. Every ffth tral n the Pavlovan block was a query tral (Fgure 1C), n whch subjects chose the better of two fractal vsual stmul wthout beng nformed about the outcome. Fnally, n the PIT stage, the nstrumental stmul were presented on a background of fractal Pavlovan stmul together wth the audtory tones, and agan wthout outcome nformaton. Our task addressed the key confounds descrbed above. Wth respect to the trads, we found that the Pavlovan nfluence s acton specfc: appettve Pavlovan cues boosted go approach responses and suppressed wthdrawal go responses; aversve Pavlovan cues dd the opposte. Addtonally, subjects were substantally based aganst wthdrawal, but we found no evdence that the nstrumental learnng component tself dffered between the approach and wthdrawal condton. Results The key results n ths paper concern the nteracton of valued Pavlovan stmul on nstrumental choces. We frst present a drect analyss of the choce data and reacton tmes. We then provde a detaled modellng analyss of the data, employng a strngent form of group-level model selecton that assesses each model s parsmony by weghng ts ablty to ft the data aganst ts complexty. The models quantfy Pavlovan values V(s P ), whch are the expectatons of a gan or loss gven Pavlovan stmulus s P, and nstrumental choce values Q t (a,s I ), whch are the tmevaryng expectatons of a reward gven a response a to an nstrumental stmulus s I. The structure of the most parsmonous model mples the nfluences and nteractons that were sgnfcant (for nstance rulng n a bas aganst actve wthdrawal, but rulng out any dfference between the nstrumental learnng rates assocated wth approach and wthdrawal); the values of the parameters n ths model ndcate the nature of those nfluences and nteractons. PLoS Computatonal Bology 3 Aprl 2011 Volume 7 Issue 4 e

4 Model-free analyses There was no dfference between the results for probablstc and determnstc feedback, and we therefore present the combned data. Analyss of the components of the experment ndcate robust, yet moderate, nstrumental condtonng that was stable durng the PIT perod, combned wth hghly robust Pavlovan condtonng. Fgure 2A shows the nstrumental probablty of choosng the more rewarded ( correct ) stmulus over tme. Subjects rapdly came to prefer the more rewarded acton. Preference was weaker for go wthdrawal, aganst whch there was a consstent bas. We ntended the nstrumental preference to be weak to avod celng effects when assessng PIT. Subjects also exhbted predctable varablty on a shorter tmescale: Fgure 2B shows the mmedate consequences of rewards and punshments on subsequent behavour. It s notable that punshments dd not reduce the repeat probablty below chance level (mean p(swtch t jpun t{1 ) s not v0:5, one-taled t-test pw:2). The same was found when analysng go and nogo choces separately: n both cases, p(swtch t jpun t{1 ) was not sgnfcantly dfferent from 0.5 (both pw:3, two-taled t-test), and was sgnfcantly smaller than p(stay t jrew t{1 ) (both pv4 10 {6, pared t-test). Whether ths really does represent an nsenstvty to punshments depends, however, on the average stay probablty, and on how ths average stay probablty s related to past renforcements. Subjects were nstructed that the outcomes of responses n the PIT block would be counted as n the nstrumental block. Fgure 2C shows that ths led to stable mantenance of the nstrumental response tendences throughout the PIT block. Fgure 2D shows that all but one (excluded) subject showed extremely good performance on the Pavlovan query trals nterleaved wth the Pavlovan tranng (mean correct w95%). Gven the success of nstrumental and Pavlovan tranng, we next analysed the raw effect of Pavlovan stmul on approach and wthdrawal choces. Fgure 2E shows a hghly sgnfcant nteracton between block and Pavlovan stmulus valence. Relatve to neutral stmul, postve Pavlovan stmul enhanced approach and nhbted wthdrawal go over nogo. Conversely, negatve Pavlovan stmul enhanced wthdrawal and nhbted approach go over nogo. A smlar analyss lookng at the probablty of respondng ncorrectly (outsde the blue box) showed no effect of the Pavlovan stmul n ether approach or wthdrawal condton and no nteracton (p~0:26,0:22,0:88 respectvely, ANOVA), suggestng that these results were not due to response competton. Note that the wthdrawal go probabltes were lower than the approach ones, agan reflectng the overall bas aganst go wthdrawal. Average reacton tmes for go approach and go wthdrawal actons dd not dffer (p~0:097, 2-taled t-test). Aganst our expectatons, Pavlovan stmul of both postve and negatve valence shortened reacton tmes n a parametrc manner relatve to neutral Pavlovan stmul (Fgure 2F, p = , ANOVA), although ths effect was not present n ether block separately (p = and p = respectvely, ANOVA). Model-based analyses The sze of the PIT effect may have been affected by the extent of nstrumental learnng (and thus the actual learned acton values), by response bases, and by generalzaton from the nstrumental to the PIT stage. In addton, there may have been dfferences n the nstrumental learnng of approach and wthdrawal actons (Fgure 2A). We decomposed and analysed all such factors usng a detaled renforcement learnng model. Ths contaned explct parameters capturng all the nstrumental and Pavlovan effects n Fgure 2. Raw choce probabltes. A&C: Average probablty (+1 standard error) of choosng the more rewarded ( correct ) acton n the nstrumental (A) and PIT (C) parts. Average performance was above chance n all cases, but worse when wthdrawal go was the more rewarded acton (red). There was no extncton durng the PIT block. Each pont s the average across subjects and across four trals. B: The bars show mean overall probablty of repeatng an acton n the nstrumental part gven that t was last rewarded n the presence of the current stmulus, or the probablty of swtchng gven a prevous punshment. Punshments do not lead to relable swtchng. D: Choce probabltes n the Pavlovan forced choce query trals. Most subjects were close to perfect. The grey bars show the probabltes of left: choosng a very good stmulus (++) over a good (+) or neutral (0) stmulus; mddle: choosng a bad (2) or neutral (0) stmulus over a very bad (--) stmulus; rght: choosng a postve (++ or +) stmulus over a negatve one (-- or -). Subjects that performed submaxmally n the appettve Pavlovan doman dd not necessarly have lower reward senstvtes n the nstrumental task, and vce versa for aversve Pavlovan stmul and punshment senstvty. E: PIT effects. The left part shows the approach PIT block, the rght part the wthdrawal PIT block. Each bar shows the log rato of the choce probablty (go/nogo) n the presence of one of the fve Pavlovan stmul. There was a sgnfcant effect of Pavlovan stmulus valence n each block. In addton, there was a sgnfcant block Pavlovan stmulus valence nteracton. Grey bars are means +1 standard error (red) and +95% confdence ntervals (green). F: Reacton tmes, pooled data for both PIT blocks. The bgger the absolute valence of the Pavlovan stmulus, the shorter the reacton tme. do: /journal.pcb g002 PLoS Computatonal Bology 4 Aprl 2011 Volume 7 Issue 4 e

5 Fgure 3. Model comparson. Each bar shows the dfferental BIC nt score relatve to the model wth the lowest BIC nt score (log e scale). Note that these BIC nt scores are for the group as a whole. Top: Models 1 7 were ftted to the nstrumental data only. Model 1 was a standard Rescorla- Wagner type model whch forced rewards and punshments to be equally nformatve. It assumed equally fast learnng about rewards and punshments, and no bases. Incluson of ether separate reward and punshment senstvtes (2r, Model 2) or separate bases n the approach and wthdrawal blocks (Model 4) mproved the ft. Separate learnng rates for rewards and punshments (Model 3) dd not mprove the ft as much as separate reward and punshment senstvtes (Model 2). The best model (5) ncluded a separate go bas n the approach and wthdrawal blocks, and separate reward and punshment learnng rates. Models that addtonally allowed separate renforcement senstvtes (Model 6), or separate learnng rates (Model 7) n the approach and wthdrawal blocks faled to mprove the ft. Bottom: Comparson of models on both nstrumental and PIT choce data jontly. Models 8 10 used the nstrumental component of Model 5. Models 8 10 ncluded ten Pavlovan factors, capturng the effect of each of the fve Pavlovan stmul n each of the two blocks. Model 9 allowed for extncton by ncludng an exponental decay of the nstrumental values durng the PIT part of the task. Model 10 ncluded random generalsaton nose and provded the best ft. do: /journal.pcb g003 the task, and was ft to the choce data of all subjects. We used group-level Bayesan model comparson [66] to choose amongst a varety of model formulatons (reportng DBIC nt scores relatve to the fnal model), and ensured that nference yelded correct parameter estmates when run on surrogate data generated from the assumed underlyng decson process. Instrumental learnng The fnal model ncluded 5 parameters assocated drectly wth the nstrumental requrements of the task. These comprse one learnng rate ; two parameters bas app and bas wth representng the bas towards go n the approach and wthdrawal blocks; and two separate free parameters r rew and r pun, representng the effectve strengths of rewards and punshments. At a group level, subjects were based aganst actve wthdrawal, but showed no bas for or aganst approach (p~8 10 {8 and p~0:70 respectvely, two-taled t-test), the dfference beng sgnfcant (p~5 10 {5, ANOVA, Fgure 4A). Wthdrawal bases n the release and throw away expermental subgroups dd not dffer (p~0:62, ANOVA), controllng for motor effects. The wthdrawal bas accounts for the lower performance on go wthdrawal n Fgure 2A. One concern s that dfferences n the bases mght have masked dfferences n learnng (.e. the reward senstvtes) n the approach and wthdrawal condtons. We tested ths by allowng for separate reward and punshment senstvtes n the two condtons (Model 6) or separate learnng rates (Model 7). The use of these extra parameters was structurally rejected by the model selecton process (DBIC nt ~12:6; 19:7 respectvely for the purely nstrumental trals); and the freedom to choose dfferent parameter values n these condtons was duly not used (Fgure 5). The absence of any dfference n the learnng parameters for approach and wthdrawal suggests that the nstrumental system treated approach and wthdrawal entrely equally. We wll see below that ths was not true for the Pavlovan system. Although, by desgn, rewards and punshments were equally nformatve, subjects chose to rely more on rewards than punshments (Fgure 4B). Rewards had a stronger effect than punshments both at a group level and for all ndvdual subjects, the dfference beng sgnfcant (pv1 10 {15, ANOVA). Indeed, the average punshment senstvty was not dstngushable from zero (p~0:37, two-taled t-test). Ths remaned true when we separately tested subjects who were gven determnstc (p~0:34, two-taled t-test) and probablstc (p~0:0627, two-taled t-test) feedback. Supplementary analyses (Text S1) excluded two further explanatons for the punshment nsenstvty: frst, that t s due to choce perseverance (Fgure S1 Text S1); and second that t s due to an emergng maxmsaton behavour (Fgure S2 n Text S1). Thus, t appears that the pattern seen n Fgure 2B s ndeed due to a dfferental senstvty to rewards and punshments. Generalzaton: Extncton versus nose We next analysed the generalzaton of nstrumental Q(s,a) values from the nstrumental to the PIT blocks. Generalzaton could be mperfect n two ways - the startng Q(s,a) values n the PIT block could dffer from the endng Q(s,a) values n the precedng nstrumental block, and the Q(s,a) values could then decay over tme or trals durng the PIT block gven the lack of nformaton about the outcomes. We constructed models ncludng such effects, and tested whether ther excess complexty was outweghed by ther ft to the data. PLoS Computatonal Bology 5 Aprl 2011 Volume 7 Issue 4 e

6 Fgure 4. Instrumental model parameters. A: Go bases for the approach and wthdrawal condton n the full experment. Subjects were only based aganst go, compared to nogo, n the wthdrawal block. B: Reward and punshment senstvty. Subjects were sgnfcantly more senstve to rewards than punshments. C: Generalzaton nose. Effectve Q value dfferences between go and nogo actons for all stmul and subjects, at the end of nstrumental learnng and durng the PIT block. Generalzaton seemed noser when acton preferences were weaker. D: Mean Q values of correct (.e. more frequently rewarded) actons. There was no dfference, and all correct actons had postve expectatons on average. E: PIT parameter estmates, correctng for nstrumental learnng, response bases and generalzaton nose. Postve Pavlovan stmul enhanced approach go actons and nhbted wthdrawal go, whle negatve Pavlovan stmul nhbted approach go actons and enhanced wthdrawal go actons. The nteracton was hghly sgnfcant, as were the two lnear man effects. F: There was no dfference between the effect of Pavlovan stmul on throwaway versus release go actons (all p values n E and F are ANOVA). Throughout, grey bars are pror means wth estmates of standard error (red) and 95% confdence nterval (green). Black dots show ndvdual data ponts, and ndvdual subjects parameters are connected by a dashed grey lne n A and B. do: /journal.pcb g004 As expected from the stable raw probabltes of choosng the correct (.e., more rewarded) opton (Fgure 2C), a model n whch the nstrumental Q(s,a) values decayed exponentally over tme durng the PIT block (mmckng extncton) dd not provde a good account of the data (Model 9, compared to Model 10 DBIC nt ~865). Fgure 5. Reward senstvtes and learnng rates n nstrumental approach and wthdrawal blocks do not dffer. A: The dark bars show the reward (left) and punshment (rght) senstvtes n Model 5, whch collapses across approach and wthdrawal condtons. The grey and lght grey bars show the senstvtes when ft separately for approach and wthdrawal blocks (Model 6). There s no dfference between blocks; and the jont parameter dffers from nether (all parwse comparsons pw:19). B: Dark bar shows learnng rate collapsed across both condtons n Model 5. Grey and lght grey bars show learnng rates when ft separately for approach and wthdrawal condton. Agan, no parwse dfference s sgnfcant (all pw:2). Throughout, black dots show ndvdual data; bars show pror means and red and green error bars 1 estmated standard error and 95% confdence nterval, respectvely. do: /journal.pcb g005 PLoS Computatonal Bology 6 Aprl 2011 Volume 7 Issue 4 e

7 Rather, the fnal model allowed for the addton of random generalzaton nose to each Q(s,a). These factors were drawn ndependently from the same normal dstrbuton for all stmulusacton pars, and the mean and varance of ths dstrbuton were both nferred wthout constrants (see Methods). Fgure 4C vsualzes the resultng changes; each dot represents the preference for the go acton (Q(s,go){Q(s,nogo)) for all subjects and all stmul. The abscssa shows ths at the end of the nstrumental stage, the ordnate after addton of the nose for the PIT stage. Importantly, there was no systematc dfference n mean correct acton values ether n the nstrumental or PIT stage (Fgure 4D). Pavlovan-Instrumental transfer We were manly nterested n the effect of the Pavlovan values on nstrumental performance. We therefore ftted 10 unconstraned parameters to separately capture the nfluence of each of the fve Pavlovan stmul on nstrumental go actons n both the approach and wthdrawal condton. All models accounted for performance n the PIT part by addng up nstrumental and Pavlovan nfluences pror to takng a softmax [67,68]. Ths amounts to treatng nstrumental and the Pavlovan controllers as separate experts, each of whch voted for ts preferred acton. The model captured n detal, and thereby controlled for, varablty n nstrumental learnng and generalzaton. The fnal model predcted the choces of every ndvdual subject better than chance (bnomal probablty, pv:0001 for every subject, overall predctve probablty ). The maxmum a posteror (MAP) estmates of ths model s parameters panted a pcture very smlar to that seen n the raw data. Fgure 4E shows the parameters of the model related to the nfluence of each Pavlovan stmulus. The pattern mrrored that seen n the raw data: there are hghly sgnfcant, and opposte, effects n the approach and wthdrawal blocks, wth appettve stmul (++ and +) promotng approach but nhbtng wthdrawal; and aversve stmul (-- and -) promotng wthdrawal but nhbtng approach. At a sngle subject level, the effect n the approach block was seen n 45/46 subjects (98%), whle t was seen n 30 subjects (65%) n the wthdrawal block. Snce there was no dfference n the learned value of go or nogo actons n ether approach or wthdrawal blocks, and n ether the nstrumental learnng or the PIT stages (Fgure 4D), any PIT effects are unlkely to be due to a preferental assocaton of a Pavlovan stmulus wth the learned value of an acton. Rather, they reflect the approach or a wthdrawal nature of the acton. We ncluded two separate groups of subjects who ether performed a throwaway wthdrawal acton, or a release wthdrawal acton. Ths was both to test the contrbuton of an approach/wthdrawal component amed at the Pavlovan stmul tlng the background, and n recognton of the sophstcaton of defensve reactons [27]. Fgure 4F shows that Pavlovan stmulus value had a sgnfcant, lnear effect on both wthdrawal acton types, and that ths overall lnear effect dd not dffer between the two acton types. At an ndvdual level, lnear correlatons were postve for 16 (72%) and 14 (58%) subject n the release and throwaway condton, respectvely. Psychometrc measures No psychometrc measure of anxety or depresson correlated wth any of the parameters n the man model. Dscusson Our task was desgned to look nsde the trads of valence, behavoural actvaton and nhbton, and specfc actons assocated wth Pavlovan nfluences. Ths ssue has been ncompletely explored n the past. Ether these trads as a whole have been nvestgated: aversve actons allowed avodance of, or escape from, a negatve renforcer; appettve actons, the acquston of a reward [6,8,64], or, as n negatve automantenance [10], the relevant Pavlovan contngences have been tghtly embedded n the nstrumental task. Here, we found that Pavlovan nfluences dstngushed approach from wthdrawal when carefully controllng for actvaton, for appettve versus aversve nstrumental motvaton, and for detals of the motor executon. Thus, for nstance, a Pavlovan stmulus predctng reward had opposte effects on two dfferent nstrumental actons (approach and wthdrawal) even though both those actons were themselves equally motvated by the acquston of reward. Approach and avodance were defned n two parallel ways: by the cogntve label for the acton ( throw away, collect ) and by the relaton to the stmulus (movng the mouse/fnger towards or away from the stmulus). Our task dd not set out to dstngush these two contrbutons (cogntve and motor), and we also dd not attempt to quantfy subjects explct nsght nto ther strateges. However, both possbltes are mportant. At a cogntve level, subjects should neglect the Pavlovan stmul: by desgn, they are not nformatve about the nstrumental task. Upon enterng the PIT stage, subjects were also explctly nstructed to contnue dong the nstrumental task as before. If despte these facts subjects were cogntvely swayed to nclude the rrelevant backgrounds n ther goal-drected decson process, then our fndng show that Pavlovan contngences extend even nto cogntve choces. Ths s of course consonant wth a large number of behavoural rregulartes n human decson makng [12 15]. The motor aspects are equally nterestng snce they suggest a fne level of detal n the archtecture of Pavlovan nfluences. There s qute some evdence for ths; for nstance, Pavlovan CRs are known to be hghly adaptve to the detals of the CS (for nstance evokng a groomng condtoned response to a rat whch functons as a food CS, rather than a gnawng CR [69]) and to the nature of the US [70]. In humans, a plexglass postoned between subjects and an appettve US abolshes an ncreased wllngness to pay [71]. The performance on the purely nstrumental porton of the task was also revealng. We observed a dfference n the nstrumental performance of approach and wthdrawal acton; and ths came (unlke n prevous tasks) after controllng for the motvatonal dfference between approach and avodance. Our model-based analyss revealed that the dfference was not due to a dfference n learnng (.e. a dfference n the nstrumental parameters relatng renforcements to performance), but due to a statc bas aganst performng a wthdrawal go acton. Of course, lke all other tasks, our nstrumental task also had embedded Pavlovan contngences, and, ndeed, a Pavlovan suppresson of actve wthdrawal by the overall appettve framng of the task (subjects on average chose the correct, rewarded, acton more often) could mrror what we saw n the PIT stage of the task. Alternatvely, ths could be the result of subjects experences upon enterng an expermental stuaton n whch they are gven a computer mouse. We have nterpreted such as bas n terms of evolutonary preparedness or programmng [2,9,24,50,72]. That s, the flexblty of the arbtrary outcome-contngent mappngs of nstrumental control comes at the prce of the experence necessary for t to be specfed. Pavlovan prors substtute nflexble hard-wred choces that are mmedately avalable for ths flexble nstrumental adaptatvty wth ts potentally substantal sample complexty (.e. the potental need for extended experence). Related bases are wdely known: dogs wll happly learn to run, but not to yawn, for food; teachng a PLoS Computatonal Bology 7 Aprl 2011 Volume 7 Issue 4 e

8 rat to escape s easer than teachng t to avod the shock [3,27,28]; humans perform actve go responses slower f nstructons are n terms of aversve feedback [51] or f they are followed by aversve nformaton [73]. Fnally, n humans, an nstructed joystck approach response to a happy face s qucker than a wthdrawal response, dependng on the cogntve/affectve label n a manner smlar to our own fndngs here [74]. Alternatve nterpretatons of the response bas nclude endowment effects [14], whereby an over-valuaton of tems notonally n one s possesson makes one reluctant to gve them up. Ths s unlkely because such a bas should be present across all nstrumental stmul,.e. across both stmul for whch a go and a no-go s the more rewarded acton (Fgure 4). Another possblty s a frame dependence [13] snce we compared go wth nogo rather than two alternatve go actons aganst each other. The negatve frame assocated wth sortng to remove bad mushrooms could have nhbted go actons. Neurobology One of the central motvatons for our nvestgaton was the observaton that the neural substrate does not respect the logcal ndependence of reward/punshment and approach/wthdrawal. Rather, as we have dscussed, these are ted together, va the structure of the stratum and also specfc neuromodulators. Whle the neural bass for the promoton of approach responses by appettve stmul s known to nvolve both amygdala and stratum [62,63,75], the neural bases for the effects of aversve Pavlovan stmul are less clear. There are no data on wthdrawal responses per se,.e. wth postve expectatons. Nevertheless, anmal models, genetc studes and pharmacologcal manpulatons suggest that serotonn plays a crucal role n the nhbton of actve behavours by aversve expectatons [25,47,48,50,73,76 78]. In humans, there s evdence for the serotonergc medaton of the nhbton of actve approach by aversve predctons [51], and of approach responses to stmul that are predctve of negatve renforcement [73]. It should be noted, though, that, actng va the ndrect path and D2 receptors, dopamne tself has also been suggested to be mportant n medatng nogo behavour due to punshments [18,53,79]. Aversve Pavlovan stmul can also potentate behavour [1,64,80,81], wth both serotonn and dopamne nvolved. Dopamne may have a domnant nfluence n ths: t s both known to be released, and nfluental, n some aversve settngs [82 85] and has a more evdent relatonshp to vgour [33,34]. Ths observaton has led to a re-nterpretaton of prevous notons [43] of the opponency between dopamne and serotonn, puttng an axs spannng nvgoraton and nhbton together wth spannng reward and punshment [52]. Thus, the lterature suggests three predctons for genetc correlates of the Pavlovan nfluences we observe. When consderng these, the caveats concernng the nteracton of genetc varaton wth psychopathology (e.g. anxety or depresson), and wth development need to be kept n mnd. Nevertheless, the condtoned suppresson effect of aversve Pavlovan stmul on approach should be enhanced by D2 receptors, and hence be postvely related to D2 stratal receptor densty thought to be modulated by C975T (rs6277; [17]). Second, condtoned suppresson should be ncreased n subjects wth hgher serotonn levels,.e. as mght be the case wth the less effcent (s) allelc varaton of the serotonn reuptake transporter (5HTTLPR SLC6A4 [86]). Thrd, gven dopamne s establshed postve correlaton wth approach and PIT [87,88], we expect genetc polymorphsms that boost DA levels, such as the SLC6A3 polymorphsm of the dopamne transporter [89], to ncrease the mpact of appettve Pavlovan stmul on approach. A smlar effect may be expected from DARPP-32, although ts closer relatonshp to synaptc plastcty would also suggest effects on nstrumental learnng [90 92]. Instrumental punshment nsenstvty Although the learnng parameters assocated wth nstrumental approach and wthdrawal dd not dffer, the mpact of rewards and punshments on the acquston of respondng was hghly asymmetrc. In general, subjects neglected punshments, whlst mantanng a fxed senstvty to reward. Ths was gratutous as, n our settng, rewards and punshments were equally nformatve. It s, however, the case that the optmal strategy can be arrved at by concentratng on ether. Subjects were not globally nsenstve to punshments, as ther choce behavour n the Pavlovan learnng was hghly accurate both for rewards and punshments. Furthermore, t should be emphaszed that ascrbng punshments a value of zero outcome would stll effectvely behave as a punshment because a zero outcome s well below the average expectaton of correct actons (Fgure 4D) and as such would reduce the tendency to emt the acton that caused t. The asymmetry has been noted before. Others have ftted models wth separate learnng rates for rewards and punshments and reported sgnfcantly slower learnng rates for punshments than rewards [93,94]. In some restrcted regmes, learnng rates and nverse temperature parameters can trade off, and we explctly tested both types of models to address ths. One potental confound s the emergence of determnsm. Subject were nstructed to perform choces relatve to mushrooms. Real world mushrooms are ether edble or posonous, and ths dchotomy may have predsposed subjects towards a determnstc, rather than a matchng, strategy. (For nstance, subjects may have chosen responses based on a classfcaton of the mushrooms nto good and bad ones, rather than on the partcular value of a response for a mushroom.) Indeed, n RL settngs t s typcally optmal to start wth a low, exploratory, senstvty to outcomes, but to ncrease ths over tme to encourage explotaton, culmnatng n a determnstc strategy [2]. However, subjects dd not behave determnstcally at any pont (Fgure 2A) and supplementary analyses showed that the tme-varyng pattern of renforcement senstvtes ths would predct s not observed n the data (Text S1). A further potental confound s the average stay probablty. If ths were precsely half-way between the stay probabltes after rewards and punshments n Fgure 2B, then rewards and punshments would have the same effect relatve to the baselne, and hence arguably be equally nformatve. However, ths argument would neglect the fact that the mean stay probablty tself must be a functon of the renforcement hstory; and that ths must be ncluded n makng nferences about the renforcement senstvty. We have prevously made the argument on theoretcal grounds that part of the asymmetry observed n appettve and aversve systems mght be due to the nherent dfference n how nformatve rewards and punshments are processed, enshrned agan n the archtecture of the stratum and neuromodulaton [50]. Rewards tell us what to do; punshments tell us what not to do. The former s more nformatve n naturalstc settngs where many optons are avalable but only few are good. The fact that subjects gratutously rely on rewards rather than on punshments n the present settng may reflect an mplct apprecaton of ths fact, although our fndngs are certanly n no way conclusve evdence. Interestngly, t s known that stronger optmalty results can be shown for a stochastc learnng automata rule called lnear reward-nacton, whch does not change propenstes n the lght of punshments but PLoS Computatonal Bology 8 Aprl 2011 Volume 7 Issue 4 e

9 only rewards ([95,96]; also known as a benevolent automaton [97]), than for a rule that changes propenstes for both. Modellng The computatonal model served several central roles. Frst, t encapsulated the manfold aspects of behavour and learnng jontly, thereby controllng for them: the bas aganst wthdrawals s not a due to a dfference n learnng; and varatons n learnng or generalzaton do not account for the PIT effects we saw. Secondly, ts close ft to the behavour argues that the PIT effects can be accounted for by a smple superposton of an nstrumental and a Pavlovan controller: the acton propenstes due to both controllers were smply multpled (as addtve factors n an exponental), rather than beng allowed to nteract n more complex ways. The smplcty of ths nteracton eschews questons about perpheral versus central response competton, whether appettve and aversve systems compete centrally [7], and whether Pavlovan learnng s nvolved n nstrumental learnng [1]. It takes the vew of multple, separate controllers contrbutng n parallel [98], and weghtng the ultmate choce by the reward expected from that choce. One alternatve would be to wegh contrbutons by dfferent controllers accordng to ther certanty [99], although t s unclear how to compute the Pavlovan controller s certanty. Lmtatons There are varous pressng drectons for future studes. Frst, despte the role the archtecture of decson-makng has played n the argument, our work does not drectly address the neural mechansms concerned. These could be examned usng magng and pharmacologcal manpulatons. Second, our task was not desgned to dstngush between outcome-specfc and general mechansms [63,75] as we reled on one, monetary, outcome throughout. Studyng dfferent outcomes s mportant, gven evdence for partly parallel pathways through dfferent nucle of the amygdala and dfferent targets n the nucleus accumbens [100,101]. Thrd, we are mssng one crucal further orthogonalzaton to do wth the overall framng of the nstrumental task. It s mportant to consder the case n whch subjects can at best avod losng money by dong the correct acton [51]. We would expect punshment to mantan ts nstrumental force n ths case; but there could also be a systematc dfference n the nature of the Pavlovan nfluences. Concluson Pavlovan responses are beleved to be hard-wred to reflect evolutonarly approprate atttudes to predctons, beng hghly adaptve and senstve to envronmental structures [102]. Here, we showed that Pavlovan nfluences on nstrumental behavour depend on the ntrnsc affectve label of an acton, ndependent of ts learned reward expectaton. It has long been known that prepared or compatble [27,69] behavours are easer targets for nstrumental condtonng. These ntrnsc bases, or prors, may serve a crucal functon both by reducng the need for collectng data (.e. sample complexty) about the effects of actons, and by reducng the need for executng complex processng necessary to work out optmal actons (.e. computatonal complexty). Both of these can be expensve or dangerous, partcularly n an aversve context. Our fndngs sharpen the understandng of the relatve contrbuton of Pavlovan and nstrumental contngences n general tasks. We showed clearly that the nteracton of Pavlovan and nstrumental behavours s organzed along the lnes of appettve and aversve motvatonal systems, and that a crtcal contrbutor to ths s the affectve nature of actons. Methods Subjects and procedure 54 healthy subjects of central European orgn were recruted from the Berln area. Subjects were screened for a personal hstory of neurologcal, endocrne, cardac and psychatrc dsorders (SCID- I screenng questonnare), and for use of drugs and psychotropc medcaton n the past 6 months. Subjects receved performancedependent compensaton (5 32 Euro) for partcpaton. Three subjects dd not meet ncluson crtera and one subject dd not complete the task; the data for three further subjects were lost due to a programmng error. One further subject was excluded from the analyss because the nstrumental task was not satsfactorly performed. The 46 remanng subjects were 25:3+4:7 years old. 59% were female (n~27). The study was approved by the local Ethcs Commttee and was n accord wth the Declaraton of Helsnk Subjects were gven detaled nformaton and gave wrtten consent. They were seated comfortably at a table n front of a laptop wth headphones and used a mouse wth ther domnant hand to ndcate ther choces. The amount earned was ndcated by the computer, and the sum pad n cash at the end of the sesson. The computer task was followed by completon of self-ratng scales. Task descrpton The task was wrtten usng Matlab and Psychtoolbox ( psychtoolbox.org). It conssted of one approach and one wthdrawal block separated by a 2 mnute break. Each block was n turn dvded nto a nstrumental tranng, a Pavlovan tranng and a PIT part. Table 1 llustrates ths. Instrumental tranng. The nstrumental task was framed n terms of a mushroom collectng and sortng task. Instrumental stmul were generc, coloured mushroom shapes. Trals started when subjects clcked n a central square (Fgure 1A). In the approach block, nstrumental stmul s I 1,2,3 and si 4,5,6 (wth subscrpts ndcatng the dentty of stmul, not the tme of presentaton) were then presented to one sde, surrounded by a blue frame (Fgure 1A, mddle column, top). Subjects ndcated that they wanted to collect the mushroom by movng the cursor onto the mushroom and clckng on t (approach go). They could also decde not to collect the mushroom by dong nothng for 1.5 seconds (approach nogo). At the end of each tral (after a clck for go trals or after 1.5 s for nogo trals respectvely), the stmulus dsappeared and the outcome was shown n the mddle of the screen (Fgure 1A). In the wthdrawal blocks, nstrumental stmul s I 7,8,9 and si 10,11,12 were presented. Subjects chose whether to throw away mushrooms (wthdrawal go) or do nothng (wthdrawal nogo). Two dfferent wthdrawal go actons were tested. The throwaway group (n~24) had to clck n a blue frame located on the opposte sde of the stmulus (see Fgure 1A, mddle column, mddle). The release (n~22) group was nstructed to press and hold the mouse button after clckng n the central square to begn the tral. The mushroom was then presented underneath the cursor (Fgure 1A, mddle column, bottom), and they could throw away a mushroom by releasng the button (wthdrawal go) or not throw away the mushroom by not releasng (wthdrawal nogo) untl 1.5 seconds had elapsed. Each block contaned three good (s I 1,2,3 and si 7,8,9 ) and three bad (si 4,5,6 and s I 10,11,12 ) mushrooms, randomly selected from the pool of 12 stmul. Subjects were gven explct renforcng feedback after every choce ( Correct, +20 cents or Wrong. 220 cents ), ether determnstcally (n~19) or probablstcally (n~27), but were not told whch mushrooms were good or bad. Correct trals were those on whch subjects threw away a bad or kept a good mushroom, and those on whch they collected a good or refraned from collectng a bad mushroom. Importantly, ths means that correct go actons of both types (approach ( collect ) and wthdraw ( throw away )) were followed PLoS Computatonal Bology 9 Aprl 2011 Volume 7 Issue 4 e

10 by both rewards and punshments. Thus, the renforcement expectances of correct approach and wthdrawal actons were equal and postve on average. Smlarly, ncorrect actons of both types were also followed by rewards and punshments, but more by the latter than the former. To ensure replcablty across expermental desgns, four expermental confguratons were ncluded, crossng determnstc/ probablstc nstrumental feedback and the two wthdrawal acton types ( throw away or release ). These manpulatons are beyond the mathematcal model descrbed below, and thus should not affect our fndngs. We present both data for all subjects and, testng nternal consstency, across the four groups. 10 subjects were n the determnstc throwaway group, 9 n the determnstc release, 14 n the probablstc throwaway and 13 n the probablstc release group. One-way ANOVA comparsons of MAP parameter estmates from the most parsmonous model (Model 10; see below) for determnstc and probablstc feedback dd not reveal any sgnfcant dfferences. Pavlovan tranng. Fve compound Pavlovan stmul consstng of a fractal vsual stmulus (Fgure 1B) and a tone were classcally condtoned. Each stmulus was presented 20 tmes and determnstcally followed, 1 second later, by the assocated outcome. Outcome presentaton lasted 1.5 seconds. Outcomes for the best (s P zz ), good (sp z ), neutral (sp 0 ), bad (sp { ) and worst (sp {{ ) stmul were, respectvely, gans of 100 cents, 10 cents, zero, and losses of 10 and 100 cents. To ensure that subjects pad attenton, every ffth tral was a query tral n whch subjects had to choose between two Pavlovan stmul (Fgure 1C). No feedback was gven n these trals, but subjects were nstructed that the choces would contrbute to ther compensaton. Pavlovan-Instrumental transfer. In the fnal part of each block, the nstrumental task was presented n extncton and on the background of Pavlovan stmul (Fgure 1D). Subjects were nstructed to contnue dong the nstrumental task; that choces were stll earnng them the same outcomes and were beng counted, but that they would not be told about the outcomes. Note, mportantly, that the Pavlovan stmulus was presented over the entre background, and as such could not by tself modulate the drectonalty of actons. Psychometrc measurements. After completng the tasks, subjects completed self-ratng scales (Beck Depresson Inventory II (BDI), Beck Anxety Inventory (BAI), State-Trat Anxety Inventory STAI [ ]), followed by the admnstraton of clncan rated scales (Montgomery-Ashberg Depresson Ratng Scale (MADRS), Hamlton Depresson Scale (HamD), Structured Intervew for the Hamlton Anxety Scale (SIGHA) and Clncal Global Impresson (CGI) [ ]). Models We modfed a standard renforcement learnng model to capture the behavoural choces n the experment. We frst descrbe the man model, and then the alternatve control models. Consderng frst the nstrumental part, let s I t be the nstrumental stmulus (out of up to 12;.e. the subscrpt t now desgnates tme rather than dentty as n Table 1) presented at tral t, and a t the acton (choce) on that tral. An acton can be one of four types: go wthdrawal and nogo wthdrawal n the wthdrawal block, and go approach and nogo approach n the approach block. Let also r t [ f{1,1g be the renforcement obtaned, ether {1 for a punshment, or z1 for a reward. We wrte the probablty of acton a t n the presence of stmulus s I t as a standard probablstc functon of ) the renforcement expectatons Q t (s I t,a t) assocated wth that par on that tral, and ) a tme-nvarant, fxed, response bas b(a t ): W I (s I t,a t)~q t (s I t,a t)zb(a t ) ð1þ p(a t js I t )~ exp WI (s I t,a X t) exp a WI (s I t,a ) ð2þ where W I s the nstrumental weght of acton a t, and where the varable b(a t ) can take on value bas wth for wthdrawal go actons, or bas app for the approach go actons. It s always zero for the nogo acton. There was no delayed outcome n the nstrumental task, and the expectatons were thus constructed by a Rescorla-Wagner-lke rule wth a fxed learnng rate. The mmedate, ntrnsc, value of the renforcements delvered n the experment may have dfferent meanng for dfferent subjects. To measure ths effect, we added two further parameters: the reward senstvty r rew and the punshment senstvty r pun, yeldng an update equaton for the expectatons: Q tz1 (s I t,a t)~q t (s I t,a t)z R t {Q t (s I t,a t) ( R t ~ r rew f r t w0 r pun f r t v0 Ths s model 5 n Table 2, whch has the lowest BIC nt score (see below). Alternatve models tested on the nstrumental data only are as follows: Model 1 assumes that {r pun ~r rew ~b, and that bas wth ~bas app ~0. Model 2 allows only for separate reward and punshment senstvtes and model 4 for separate bases. Model 3 agan assumes {r pun ~r rew ~b,andthatbas wth ~bas app ~0,but allows for two separate learnng rates,.e. n Equaton 3 s replaced by rew on trals where r t ~1, andby pun on trals where r t ~{1. Model 6 and 7 are expansons of the fnal model, allowng for separate reward and punshment senstvtes (model 6) and for separate learnng rates (model 7) n the approach and wthdrawal condtons. Our man measure of nterest s the effect of Pavlovan stmul on the approach and wthdrawal actons. Let addtonally s P t be the Pavlovan stmulus on tral t. We can then wrte an equaton smlar to equaton 2 for the trals where both nstrumental and Pavlovan stmul were present, but ncludng a term f (a,s P t ) that quantfes the effect of the partcular Pavlovan stmulus s P t on the acton a. Ths means that the acton weghts due to the nstrumental and Pavlovan controllers are added nsde the exponent of equaton 2, and that thus the probabltes each controller attaches to a partcular acton are multpled and renormalzed. The two controllers are therefore treated as two dstnct enttes, each separately votng for a partcular acton to be emtted. The nfluence of each system on acton choce s relatve to the strength wth whch the other enhances one partcular acton. We wrte the PIT weght of acton a as: W PIT (a,s I,s P )~W I (s I,a)zf (a,s P ) Here we force f (nogo,s P )~0 at all tmes. The go values f (go,s P ) can take on 10 separate, nferred, values, meanng that there s one separate parameter for each of the fve Pavlovan stmul s P n each of the two blocks. Each of these parameters captures how much s P boosts the go over the nogo acton (f f (go,s P )w0) orthenverse(f f (go,s P )v0). Note that because these are separately nferred, ndependent, parameters, ths formulaton does not mpose any assumptons about the effect of the value of the stmulus s P, or about the relatve effect of dfferent stmul s P wth dfferent values. Hence, ths controls for varaton n learnng durng the Pavlovan tranng block (though the query trals ndcate that learnng was very robust). Equaton 3 (Model 8 n Table 2) assumes that the stmulus-acton values Q(s I,a) at the end of the nstrumental block are perfectly and exactly generalzed to the PIT block. We frst tested an alternatve model (Model 9 n Table 2) that ncluded an exponental extncton ð3þ PLoS Computatonal Bology 10 Aprl 2011 Volume 7 Issue 4 e