Opponent appetitive-aversive neural processes underlie predictive learning of pain relief

Transcription

1 2 Nture Pulishing Group Opponent ppetitive-versive neurl proesses underlie preditive lerning of pin relief Ben Seymour 1, John P O Doherty 1,2, Mrtin Koltzenurg 3, Ktj Wieh 1, ihrd Frkowik 1,4, Krl Friston 1 & ymond Doln 1 Termintion of pinful or unplesnt event n e rewrding. However, whether the rin trets relief in similr wy s it trets nturl rewrd is unler, nd the neurl proesses tht underlie its representtion s motivtionl gol remin poorly understood. We used fmi (funtionl mgneti resonne imging) to investigte how humns lern to generte expettions of pin relief. Using pvlovin onditioning proedure, we show tht sujets experiening prolonged experimentlly indued pin n e onditioned to predit pin relief. This proeeds in mnner onsistent with ontemporry rewrd-lerning theory (verge rewrd/loss reinforement lerning), refleted y neurl tivity in the mygdl nd midrin. Furthermore, these rewrd-like lerning signls re mirrored y opposite version-like signls in lterl oritofrontl ortex nd nterior ingulte ortex. This dul oding hs prllels to opponent proess theories in psyhology nd promotes forml ount of predition nd expettion during pin. Self-preservtion nd evolution ordin tht nimls t optimlly or ner-optimlly to minimize hrm. One of the prinipl mehnisms for deteting hrm is the pin system, nd erly predition is essentil to diret pproprite pre-emptive ehvior. However, ny simple orrespondene etween predited sensory input nd ehviorl output is hllenged y onsidering the nture of relief: for exmple, mild pin will e rewrding if it diretly follows severe pin. This illustrtes ritil issue in our understnding of pin relief s n ffetive nd motivtionl stte 1 3 nd poses roder question in emotion reserh: how do the neurl proesses tht underlie motivtion dpt to the ontext provided y the ongoing ffetive stte? Aording to psyhologil theories 4 7, toni versive sttes reruit rewrd proesses to help diret ehvior towrd homeostti equilirium (whih eomes the motivtionl gol). This my offer insight into why relief is often plesurle: for exmple, the experiene of ooling oneself in swimming pool on hot dy. Indeed, the euphori of relief hs een used to help explin numer of seemingly prdoxil ehviors, from sky diving to sun thing 8, in whih relief is thought to eome the dominnt motivtionl drive. Despite supportive psyhologil evidene 9 12, diret oservtions of neurl tivity onsistent with suh ppetitive proesses re lking. Coneptully relted issues rise in diverse res suh s engineering, eonomis nd omputer siene nd offer potentil insight into the underlying neurl proesses involved in relief in nimls. Notly, omputtionl reinforement lerning models hve proved prtiulrly useful in formlizing how the rin lerns to predit rewrds nd punishments These models lern to mke preditions y ssessing previous ontingenies etween environmentl ues nd motivtionlly slient outomes. In theory, these models n e extended to del with toni reinforement nd relief, y omputing preditions reltive to n verge rte of reinforement, rther thn ording to solute vlues 2,21. However, the extent to whih verge rewrd/loss reinforement lerning strtegies re implemented in the rin is still unler. With respet to pin, this my hve dded importne, s motivtionl preditions (of pin or relief) re thought to exert sustntil influene on the susequent pereption of pin 22,23. Understnding the neurl mehnisms y whih preditions re lerned is therefore key to our understnding of how the rin intrinsilly modultes pin in physiologil nd linil situtions. We used fmi to investigte the pttern of rin responses in nineteen helthy sujets s they lerned to predit the ourrene of phsi relief from or exertion of toni pin (see Methods). We employed first-order pvlovin onditioning proedure with prtil (%) reinforement shedule (Fig. 1). Toni pin ws indued using the psiin-het model. Cpsiin is the pin-induing omponent of hili pepper; it indues sensitiztion to het y tivtion of temperture-dependent TPV1 ion hnnels expressed on peripherl noieptive neurons. This temperture sensitivity llowed us to deliver onstnt ut esily modifile levels of pin for long durtions, dpted for eh individul sujet, t tempertures whih do not use skin dmge. This provides unique experimentl tool to study pin, s it speifilly permits investigtion of the neurl proesses underlying the offset of pin: tht is, relief. The model hs the further dvntge tht it 1 Wellome Deprtment of Imging Neurosiene, 12 Queen Squre, London WC1N 3BG, UK. 2 Division of the Humnities nd Soil Sienes , Cliforni Institute of Tehnology, Psden, Cliforni 9112, USA. 3 Institute of Child Helth, University College London, 3 Guildford Street, London WC1N 1EH, UK. 4 Neuroimging Lortory, Fondzione Snt Lui, ome 179, Itly. Correspondene should e ddressed to B.S. (seymour@fil.ion.ul..uk). Pulished online 21 August 2; doi:1.138/nn VOLUME 8 [ NUMBE 9 [ SEPTEMBE 2 NATUE NEUOSCIENCE

2 2 Nture Pulishing Group Tril type: (ndom order) Cue Pin High Medium Low (relief) Appetitive predition error: 1 1 Aversive predition error: 1 1 einfored Unreinfored einfored Unreinfored A A B B C 4 s 4 s elief Time Time indues the hrteristi moleulr nd ellulr hnges tht mimi physiologil injury, nd so presents iologilly relisti model of relief in nturl nd linil environments. We pplied psiin topilly to n re (12. m 2 )ofskinonthe left leg, whih used lolized re of urning pin (whih feels similr to sunurn), nd mnipulted the intensity of this pin with n overlying temperture thermode tht mthed the psiin-treted re. Temperture ws djusted for individul sujets to im for evoking n verge seline mgnitude of pin rted s 6 on 1 tegoril sle. Phsi dereses in the seline temperture to 2 1C used omplete relief of pin, nd temperture inreses used exertion. We used visul ues (whih were strt olored imges) s pvlovin onditioned preditors of these hnges. Thus, in the fmi snner, sujets lerned tht ertin imges tended to predit imminent relief or exertion of pin. We used omputtionl reinforement lerning (temporl differene) model to identify neurl tivity onsistent with rewrd-like proessing. The hrteristi tehing signl of these models is the Pin Control ue Figure 1 Experimentl design nd omputtionl model. () Experimentl design. There were five tril types: ue A ws followed y temperture/pin derese on % of osions (reinfored nd unreinfored relief ue), ue B ws followed y temperture/pin inrese on % of osions (reinfored nd unreinfored pin ue) nd ue C ws followed y no hnge in temperture/pin (ontrol ue). () Appetitive omputtionl model: predited neuronl response. Shemti showing the men representtion of the temporl differene predition error ording to the different ue types, where relief is represented s rewrd. () Aversive omputtionl model: predited neuronl response. Shemti showing the versive temporl differene predition error, whih trets pin exertion s punishment. nd represent the verge predited neuronl response; the orresponding predited BOLD response is shown in Figs. 3 nd 4, respetively, following onvolution with nonil hemodynmi response funtion. predition error, whih is used to diret quisition nd refinement of preditions relting to individul ues. The predition error reords ny hnge in expeted ffetive outome, nd it thus ours whenever preditions re generted, updted or refuted. By treting relief of pin s rewrd, nd exertion s negtive rewrd, we sought to identify tivity tht orrelted with this predition error signl. We lulted the vlue of the predition error for eh sujet ording to the sequene of stimuli they reeived in order to provide sttistil preditor of fmi dt (s hs een done previously 17,18,24 ). The use of prtil (proilisti) reinforement strtegy, in whih the ues re only % preditive of their outomes, ensures onstnt lerning nd updting of preditions nd genertes oth positive nd negtive predition errors throughout the ourse of the experiment (Fig. 1,). Thus, inferene is sed on identifition of this dynmi nd highly hrteristi signl. In support of the model, our dt show tht rin tivity (tht is, lood oxygen level dependent, or BOLD, tivity) in the mygdl nd midrin orreltes with the rewrd predition error signl predited y verge rewrd temporl differene lerning. In ddition, we show n opponent, versive representtion of the predition error in lterl oritofrontl nd genul nterior ingulte ortex. Furthermore, these two signls pper to e oexpressed in the ventrl stritum. ESULTS Behviorl nd utonomi results Sujets rted the seline therml stimultion s pinful nd the dereses nd inreses in temperture s plesnt or more pinful, respetively (Fig. 2). In ddition, plesntness nd pin rtings were signifintly greter thn equivlent temperture hnges on djent skin not treted with psiin (P o., ll pir-wise omprisons; see Methods). In ehviorl version of the tsk outside of the fmi snner, we demonstrted onditioning to the relief nd exertions of pin y engging the sujets in supplementry ue-preferene tsk, fter the lerning tsk. In this, sujets (n ¼ 14) mde fored hoie preferene judgement of pirs of ues, presented side y side. This demonstrted Figure 2 Behviorl mesures. () Pin rtings. Pin nd plesntness rtings for the seline level of therml stimultion, nd the phsi inreses nd dereses in temperture. Sores re on 1 mgnitude rting, with error rs representing the s.e.m. The grph shows results for the psiin-treted skin nd n djent re of unffeted skin. () Preferene sores. After the lerning experiment, sujets mde fored hoies etween rndomized pirs of ues. The sores re out of mximum of 2 pirings for eh ue (with higher sores inditing more preferred). Sujetive rtings Pinful Plesnt Exertion Bseline Cpsiin-treted skin Untreted skin elief Preferene sore Exertion Control elief Cue type NATUE NEUOSCIENCE VOLUME 8 [ NUMBE 9 [ SEPTEMBE 2 123

3 2 Nture Pulishing Group Signl hnge (%).... einfored 1 1 elief.. 1 signifint preferene ordering, with the relief ue preferred to the neutrl ue (P o., Wiloxon sign rnk test), whih ws, in turn, preferred to the exertion ue (P o.1, Wiloxon sign rnk test; Fig. 2). On post-experimentl deriefing (see Methods), only four out of the 14 sujets ould report ny ontingent reltionship etween the ues nd the outomes. During the fmi version of the tsk, we used physiologil mesures to ssess the quisition of ue expettions. Hert rte hnges indued y the ues orrelted with the mgnitude of expettions (tht is, ue-speifi temporl differene vlues) oth of pin relief (P o.1) nd pin exertion (P o.1), lulted from the model (see Methods). This supports the hypothesis tht ue expettions re quired in mnner onsistent with the (temporl differene) lerning model, leit in vlene-insensitive mnner. Tht is, we oserved inresed hert rte with higher vlued ues, whether positive or negtive, onsistent with lerned rousl-like response ssoited with the expettions. fmi results We used the model to identify representtion of the ppetitive predition error in the rin (Fig. 1, ppetitive model). Ativity in left mygdl nd left midrin (in region onsistent with the sustnti nigr) orrelted with this signl (Fig. 3,). Time-ourse nlysis illustrtes the verge pttern of response ssoited with the different tril types in the mygdl, illustrting strong orrespondene with the preditions of the model (Fig. 3). These dt support the hypothesis tht relief lerning involves rewrd-like lerning signl. eent evidene indites tht temporl differene models lso provide n urte desription of versive lerning, suggesting the existene of seprte reinforement lerning mehnism enoding versive events 18. We therefore sought to identify whether n versive Pin.. Unreinfored Figure 3 Appetitive temporl differene predition error. (,) Sttistil prmetri mps (P o.1) showing () left sustnti nigr (xil plne) nd () left mygdl (oronl plne). () Time ourse of inferred men neuronl tivity for the four prinipl tril types in left mygdl. Blk line represents the dt (error rs represent 1 s.e.m.), nd thin gry line is the model ppetitive temporl differene predition error (from Fig. 1) fter onvolution with nonil hemodynmi response funtion. 1 Signl hnge (%).... einfored 1 1 elief Pin... representtion of the predition error ws expressed, in whih exertion of pin ws treted s positive punishment, nd relief s negtive punishment (Fig. 1, versive model). Ativity in ilterl lterl oritofrontl ortex nd genul nterior ingulte ortex orrelted with this signl (Fig. 4,). The time-ourse of this tivity (Fig. 4) illustrtes the opposite pttern of response to the ppetitive predition error. These dt indite the existene of n versive reinforement signl, distint from the rewrd-like signl. Psyhologil studies of ppetitive-versive intertions predit tht opposing, lerning-relted tivities should onverge in some res 1. This might our in res suh s the ventrl stritum (nd insul ortex), where preditive tivity hs een oserved in oth rewrd nd pin lerning tsks, leit in seprte studies 17,18,2 28. This rises question out how oexpressed versive nd ppetitive predition errors would e represented y the BOLD signl, prtiulrly if they intert. We therefore reted new sttistil model tht inluded two regressors, modelling predition error for relief nd exertion seprtely. This model reveled oexpression in the ventrl putmen, nterior insul nd rostrl nterior ingulte ortex (Fig. ). The responses in these regions showed n ppetitive predition error for the relief-relted ue, nd n versive predition error for the exertionrelted ue (Fig. d). This pttern of tivity is notle, s it nnot result simply from the liner superposition of ppetitive nd versive signls, ut implies either n intertion etween predition error nd ue-vlene, or the expression of single vlene-independent predition error. DISCUSSION Drwing on theoretil onsidertions provided y omputtionl reinforement lerning 11, our dt provide evidene in support of n opponent motivtionl model of toni pin. We oserved two distint. Unreinfored 1 1 Figure 4 Aversive temporl differene predition error. Sttistil prmetri mps (P o.1) showing () lterl oritofrontl ortex (xil plne) nd () genul nterior ingulte ortex, highlighted (sgittl plne). () Time ourse of inferred men neuronl tivity for the four prinipl tril types in left oritofrontl ortex. Blk line shows dt (error rs represent 1 s.e.m.), nd thin lk line is the model versive temporl differene predition error (Fig. 1) fter onvolution with nonil hemodynmi response funtion VOLUME 8 [ NUMBE 9 [ SEPTEMBE 2 NATUE NEUOSCIENCE

4 2 Nture Pulishing Group d Signl hnge (%).... einfored 1 1 elief Pin.. ptterns of neurl tivity, distinguishle y their expression in seprte rin res, tht orrelted with the predition error signls of n opponent temporl differene model. This extends our understnding of humn preditive lerning eyond the ourrene of phsi events rising from neutrl seline. Thus, during toni pin, versive nd ppetitive systems seem to e simultneously involved to enode pproprite gol-direted preditions ross the spetrum of positive nd negtive outomes. Our oservtions suggest forml frmework for understnding the homeostti nd motivtionl proesses engged y pin nd my offer prdigmti ount of motivtion during toni ffetive sttes. The use of the temporl differene lgorithm to represent positive nd negtive devitions of pin intensity from toni kground level pproximtes the lss of reinforement lerning model termed verge-rewrd models 2,21,29. Aordingly, preditions re judged reltive to the verge level of pin, rther thn ording to n solute mesure. This omprtive tretment of motivtionlly slient preditions is onsistent with oth neuroiologil nd eonomi ounts of homeostti motivtion, whih rely ritilly on hnge in ffetive stte 2,3,31. Impliit in ny suh model is representtion of the verge rte of reinforement, lthough the short time window of fmi preludes investigtion of this diretly. From n implementtionl perspetive, one rgument for opponeny reltes to onsidertion of how longrun verge ffetive stte might e represented. Given our demonstrtion tht positive nd negtive predition errors re oth enoded y one system nd re fully mirrored y opposite signls in n opponent.. Unreinfored Figure Appetitive relief-relted plus versive exertion-relted predition error. Sttistil prmetri mps showing tivity tht orreltes with the ppetitive predition error for the relief ue (P o.1), msked with the versive predition error for the exertion ue (P o.1). () Bilterl ventrl putmen. () Bilterl ventrl putmen nd right nterior insul. () ostrl nterior ingulte ortex. (d) Time ourse of inferred men neuronl tivity for the four prinipl tril types in left ventrl putmen. Thik lk line shows the dt (error rs represent 1 s.e.m.), nd the thin gry nd lk lines re the model ppetitive nd versive temporl differene predition error, respetively (from Fig. 1,) fter onvolution with nonil hemodynmi response funtion. 1 1 system, the requirement for one system to fully represent oth the toni levels of reinforement (tht is, y sustined elevted tivity) with positive nd negtive phsi preditions simply superimposed, would seem to e ovited. If this is the se, the toni level of pin would e free to hve distint representtion, signl tht hs een suggested to e onveyed y toni dopmine relese 11. Mirror opponeny hs mny similrities to the ppetitive-versive reiproity hrteristi of erly psyhologil opponent proess theories 4 7. In their vrious forms, these theories grew out of requirement oth to explin the dptive hnges tht our during nd fter toni reinforement, nd to understnd the intertions etween ppetitive nd versive proesses tht rise in ertin speifi lerning proedures suh s onditioned inhiition nd trns-reinforer loking. Notly, reent eletrophysiologil reordings of neuronl tivity in mie diretly indite the involvement of opponent proesses in (ontext-relted) onditioned inhiition, speifilly impliting the ventrl stritum nd mygdl 32. Thus it seems possile (nd fully onsistent with omputtionl ount) tht, t lest in the ventrl stritum, sfety signl tht predits the sene of future pin might shre the sme neurl sustrte s the relief-predition error seen here. However, we show n ppetitive representtion in the mygdl, rther thn n opponent versive representtion (whih we oserve in lterl oritofrontl nd genul nterior ingulte ortex). This points to the expression of multiple lerning-relted neurl signls in the mygdl, onsistent with the omplex, integrtive role of this struture (nd the vrious nulei within) in ssoitive lerning nd pin 33,34. The finding tht lterl oritofrontl ortex demonstrtes n versive predition error signl is onsistent with previous reports of role for this region in versive lerning 3. In prtiulr, this re hs een shown to e involved in evlution of versive stimuli in the ontext of different motivtionl sttes 36 s well s in short-time-sle pin predition reltive to hnging (lerned) seline rte of phsi pin 37.Tken with the present results, this suggests tht lerning of versive vlue preditions in this region my e medited y n version speifi predition error signl, prtiulrly in irumstnes tht require dptive representtions following hnging motivtionl stte or ontext. However, it should lso e noted tht lterl oritofrontl ortex my not e exlusively involved in versive proessing, s rewrd-relted responses hve lso een reported in this region in some irumstnes. In reltion to pin, other ortil res, speifilly insul nd nterior ingulte ortex, hve ler motivtionl roles nd hve previously een implited in the proessing of relief-relted informtion 3. For exmple, reent neuroimging studies investigting the expettion nd reeipt of pleo nlgesi implite these res in endogenously medited nlgesi 38,39. Our findings provide further support tht these res hve key role in homeostti funtions relting to pin 2. The BOLD signl is thought to orrespond to hnges (inreses or dereses) in synpti tivity, nd thus the tivity we desrie my reflet speifi fferent neuromodultory influenes tht originte elsewhere 4,41. Sustntil evidene indites tht mesolimi dopmine neurons oth enode rewrd-relted predition error 16,19 nd hve key role in nlgesi 42, suggesting tht dopmine ould onvey n ppetitive relief-relted predition error. This drws ttention to tivity in the ventrl stritum, region tht reeives strong mesolimi dopminergi projetions. Comprison with previous dt in this re highlights the oservtion tht ues signling lower-thn-predited pin use detivtion in the ontext of neutrl seline, s opposed to tivtion in the ontext of toni pin seline 18,26. This implites dptive hnges ourring during toni pin, influening ventrl stritl tivity nd onsistent with the representtion of n ppetitive signl for relief-relted ues. However, tken lone, it is possile tht NATUE NEUOSCIENCE VOLUME 8 [ NUMBE 9 [ SEPTEMBE

5 2 Nture Pulishing Group this ventrl stritl tivity is modulted y single predition-error signl for oth relief nd exertion ues 43,44, lthough reent eletrophysiologil evidene demonstrting suppression of midrin dopminergi neurons to versive stimuli would seem to require distint versive opponent 4. Either wy, this signl must intert with vlene-speifi informtion y some dditionl mehnism, possily through the involvement of different intrinsi su-popultions of ppetitive nd versive neurons within the ventrl stritum 46. Tht pin relief nd rewrd might shre ommon neurl sustrte is lso suggested y the ft tht mny drugs tht hve rewrding effets hve nlgesi properties. Aside from dopmine, there re mny neurotrnsmitters with ler omined roles in ppetitive nd versive motivtion, for exmple opioid peptides, serotonin, sustne P nd glutmte 3,47,48. Of prtiulr interest re serotonin-relesing neurons projeting from the dorsl rphe nuleus to the ventrl stritum, whih hve emerged s plusile ndidte to medite n versive predition error 11. In ddition to role in pvlovin motivtion, it is lso ler tht pin nd relief-relted expettions exert strong influene on the tul susequent experiene of pin, in tht pereption (of intensity) is weighted y the prior expetnies quired through onditioning. How preditive motivtionl vlues influene pereptul inferenes suh s pin intensity is not yet ler, lthough proilisti pereptul models tht inorporte eonomi ost funtions, suh s deision theory, my offer insight t theoretil level 49. From n implementtionl perspetive, one puttive mehnism exploits n influene of higher rin res on sending pin pthwys vi desending modultory ontrol enters. A possile trget is the on- nd off- ells of the periquedutl grey nd rostrl ventromedil medull, whih show opponent ntiiptory pin-relted tivity under pprent higher ontrol 3. Whtever the mehnisms, these influenes re thought to e linilly importnt oth in endogenous pin modultion (inluding pleo nlgesi) nd in the pthogenesis of some hroni pin syndromes 3,23,38,39, nd we suggest tht integrted psyhologil, neurophysiologil nd omputtionl pprohes offer some promise in furthering their understnding. METHODS Sujets. Thirty-three helthy right hnded sujets (14 in ehviorl version of the tsk, nd 19 in the fmi version of the tsk), free of pin or medition, gve informed onsent nd prtiipted in the study, pproved y the Joint Ntionl Hospitl for Neurology nd Neurosurgery (University College London, Ntionl Helth Servie Trust) nd Institute of Neurology (University College London) Ethis Committee. Sujets were remunerted for their inonveniene (4 GBP). Stimuli: psiin model. We pplied topil 1% psiin (8-methyl-Nvnillyl-6-nonenmide, 98%, Sigm, diluted in % ethnol-ky jelly) to the lterl spet of the left leg over n re of 2. m, under n olusive dressing, nd left it for 4 min, fter whih ll sujets reported feeling persistent (though erle) pin, t whih time the psiin nd dressing ws removed nd the skin lened. A thermode mthing the size of the psiin pplition re ws pplied with loose tourniquet (esily removle in se of unerle pin) to the treted skin. Temperture ws then mnipulted using n fmi-omptile Peltier thermode (MSA thermotest, Somedi). Phsi vritions in temperture were mde t rte of 1C/s to the predetermined upper nd lower levels nd were ontrolled y in-house softwre. Stimuli nd pre-experimentl set-up. Before the experiment, required temperture levels for eh individul sujet were set y slowly inresing the utneous temperture overlying the psiin tretment site from 2 1C in steps of. 1C, with ontinul monitoring of pin rtings (on 1 rting sle) to hieve seline level of 6/1. Susequently, sujets reeived progressively higher phsi inreses to determine stisftory temperture for the pin exertions, to t lest 8/1 ( just tolerle ). Pin relief ws indued y phsi ooling to 2 1C, whih olished pin in ll sujets. We otined sujetive rtings of pin for the inrese, seline nd dereses in pin. We sked the sujets, Cn you give sore, on sle of to 1, s to how pinful the pin is, where is no pin t ll, nd 1 is the worst imginle pin? We lso took sujetive rtings of plesntness for the phsi relief. We first sked the sujets, Did you find the hnge in temperture unplesnt or plesnt? to hek tht no sujets found the ooling s unplesnt, nd then, Cn you give sore, on sle of to 1, s to how plesnt you found it, where is not t ll, nd 1 is highest imginle plesure? Phsi hnges were repeted with pin nd plesntness rtings on psiin-treted skin nd on distnt re of non psiin treted skin on the sme lim well eyond the re of seondry hyperlgesi, nd repeted t the end of the experiment. We hieved men rtings (s.e.m. in prentheses) for the seline toni pin of./1 (1.1) on psiin treted skin nd.9/1 (1.) on untreted skin. Phsi inreses were rted t 9.3/1 (.9) for psiintreted skin nd 3.3/1 (3.6) on untreted skin. Phsi dereses (relief; mesured on the plesntness sle) were rted t 7./1 (2.4) nd 4.6/1 (2.3) on untreted skin. All omprisons (treted versus untreted) were signifint t P o.1 with orresponding t-tests. After trnsfer into the snner or ehviorl testing room (with the thermode tthed) sujets were in pin for pproximtely 4 min to 1 h y the time the experiment strted. The visul ues were strt olored pitures. Tsk. The tsk ws lssil pvlovin dely-onditioning proedure of temperture inreses (exertions of pin) or dereses (relief of pin). Visul ues were presented for 4 s, t the end of whih the phsi pin perturtion ws pplied for s. The preise timing ws determined in psyhophysil pilot testing (to ommodte thermode nd C-fier ltenies). There were three different visul ues, eh presented 3 times. Cue A (reliefrelted ue) ws followed y deresed temperture on /3 osions (%), ue B (pin exertion relted ue) ws followed y inresed temperture on /3 osions (%), nd ue C ws followed y no hnge in temperture on 3/3 osions. The ontrol ondition provides dditionl ontrol in our prmetri design, lthough it ws initilly inluded to permit more onventionl nlysis (dt not shown). The five different tril types were presented in rndom order. Behviorl mesures. Sujets performed retion-time tsk whih onsisted of judging whether the visul ue ppered to the left or right of enter on the disply monitor, s quikly s possile. The resulting retion times were tken s ehviorl index of onditioning. Performne on this tsk ws not ontingent on the stimuli presented, nd sujets were told efore imging tht their suess or filure t quikly judging the position would not ffet the mount of pin or relief reeived. The tsk ws performed with two-utton key press using the right hnd. Hert rte ws reorded using pulse oximeter in onjuntion with Spike 2 softwre (CED). A ehviorl version of the tsk ws performed tht ws identil to tht performed in the fmi snner, exept tht it ws performed in testing room with the sujet seted in front of omputer monitor. After this tsk, we performed supplementry ue-preferene tsk designed to investigte whether the sujets hd quired ppetitive nd versive preferenes for the ues s result of the onditioning proedure. In this tsk, we presented two ues side-y-side nd sked the sujet to judge whih ue they preferred, indited y left or right key-press. Eh ue-piring ws repeted ten times nd ws rndomized s to whih side the ue ppered on. We lulted the preferene sores y summing the totl numer of preferene hoies mde for eh ue (s in n ll-ply-ll gmes tle, with mximum sore of 2). Men sores for eh ue were ompred ross sujets using Wiloxon sign rnk tests. We did not ttempt to formlly ddress the issue of onsious versus nononsious quisition of onditioned expetnies. However, to gin some insight into the level of expliit expetny lerning, we sked the question, Did you reognize ny reltionship etween the pitures nd susequent hnge in pin level? t the end of the experiment (for the ehviorl version of the tsk only). Sujets were not told the experiment ws lerning nd onditioning study eforehnd ut rther were simply told tht it ws study 1238 VOLUME 8 [ NUMBE 9 [ SEPTEMBE 2 NATUE NEUOSCIENCE

6 2 Nture Pulishing Group Tle 1 MNI oordintes nd sttistil z-sores for the ppetitive, versive nd joint oexpressed ppetitive-versive temporl differene predition error egion Lterlity x y z z-sore Appetitive predition error Midrin (sustnti nigr) L Amygdl L Aversive predition error Lterl oritofrontl ortex L Genul nterior ingulte ortex Motor ortex Comined ppetitive-versive predition error Ventrl putmen L Anterior insul L ostrl nterior ingulte ortex Signifint fter whole rin orretion. of pin nd temperture proessing. Ten of fourteen sujets were unle to report ny ssoition etween ues nd outomes. Computtionl model. We used temporl differene model to generte prmetri regressor orresponding to the ppetitive predition error, whih ws pplied to the imging dt, s previously desried 17,18. Here, we used two time point temporl differene model with lerning rte ( ¼.3) determined from ehviorl results (see elow). In this model, the vlue v of prtiulr ue (referred to s stte s) is updted ording to the lerning rule: v(s) v(s) + d, where d is the predition error. This is defined s d ¼ r +v(s) t+1 v(s) t,wherer is the return (tht is, the mount of pin) nd is the verge mount of reinforement (toni pin) tht ws ssumed to e onstnt. We ssigned relief nd exertions of pin s returns of 1 nd 1, respetively (tht is, liner sle of pin from relief to exertion). This is n ritrry speifition, given tht it is diffiult to preisely sle the reltive oppositely vlened utilities of relief nd exertions of pin. Thus, the model trets preditions relting to relief of pin on equl pr with unexpeted omission of exertion of pin, nd, similrly, it trets exertion-relted preditions equivlently to unexpeted omissions of relief. Dt quisition nd nlysis: ehviorl nd utonomi mesures. These were tken s mesures of ue reinforement nd orrelted with the temporl differene vlue (tht is, the ue expetny). etion time dt were individully (tht is, on sujet-y-sujet sis) fit to gmm umultive distriution funtion (using mximum likelihood funtion), to llow nlysis ross sujets, nd orrelted with the temporl differene vlue. This yielded est fit with lerning rte of.3, nd signifint orreltion for oth the relief-relted nd exertion-relted trils, independently, nd in the sme diretion. Tht is, retion times were shorter for oth high rewrd vlues nd high versive vlues. To remove ny possile onfounding effets of erly trils, during whih retion time dt hitute sustntilly, we repeted this proedure fter removing the first ten trils. This yielded orreltion whih just filed to reh signifine (P ¼.6), ross oth ue types. We lso looked t sensitivity to the initil temporl differene vlue y setting this to the verge vlue of., whih yielded non-signifint orreltion. The hert rte ws found to e pproximtely normlly distriuted nd ws normlized to permit nlysis ross sujets. We found signifint hert rte orreltions with oth relief nd pin ue types (independently, s for the retion time). For oth exertion nd relief tril types, this yielded est fit with lerning rte of.3. Aross oth ue types, this remined signifint (P o., r ¼.19) fter removl of the first ten trils nd with use of different initil temporl differene vlues. This is roust orreltion nd is reported in the min text. Consequently, we used lerning rte of.3 for the temporl differene model used in the fmi nlysis. fmi. Funtionl rin imges were quired on 3-T Allegr Siemens snner. Sujets ly in the snner with fom hed restrint pds to minimize ny movement ssoited with the pinful stimultion. Imges were religned with the first volume, normlized to stndrd EPI templte nd smoothed using 6-mm FWHM Gussin kernel. elignment prmeters were inspeted visully to identify ny potentil sujets with exessive hed movement; none ws found. Imges were nlyzed in n event-relted mnner using the generl liner model, with the onsets of eh stimulus represented s delt funtion to provide stimulus funtion. We used prmetri design, in whih the temporl differene predition errors modulted the stimulus funtions on stimulus-y-stimulus sis. The sttistil sis of this pproh hs een desried previously. egressors were then generted y onvolving the stimulus funtion with hemodynmi response funtion (HF). Effets of no interest inluded the onsets of visul ues, the pin relief nd exertions themselves nd relignment prmeters from the imge preproessing to provide dditionl orretion for residul sujet motion. Liner ontrsts of ppetitive predition errors were tken to group level (rndom effets) nlysis y wy of one-smple t-test, nd the versive predition error ws tken s the inverse. MNI oordintes nd sttistil z-sores re found in Tle 1. This nlysis determines res whih orrelte to univlent ppetitive or versive predition error nd does not identify res in whih these signls overlp. To explore the possile representtion of distint predition error signls for the pin relief nd exertion trils, we generted two independent regressors for the predition error ourring t eh. Then, we took the ppetitive relief nd versive exertion omponents of the predition error to seond level nlysis of vrine nd exlusively msked the two individul ontrsts (tht is, we looked for res of overlp of the independent ppetitiverelief nd versive-exertion predition errors, oth t P o.1; Fig. ). Group level tivtions were lolized ording to the group-verged struturl sn. Ativtions were heked on sujet-y-sujet sis using their individul normlized struturl sns to ensure orret loliztion, s some of the reported tivtions re in smll nulei (for exmple, sustnti nigr). We report tivity in res in whih we hd prior hypotheses on the sis of previous dt, though without speifition of lterlity. These regions hve estlished roles in oth versive nd ppetitive preditive lerning, nd inluded ventrl putmen, hed of udte, midrin (sustnti nigr), nterior insul ortex, ereellum, nterior ingulte ortex, mygdl, lterl oritofrontl ortex, medil oritofrontl ortex, dorsl rphe nd ventrl tegmentl re. We report tivtions t threshold of P o.1, with minimum size of five ontiguous voxels. We lso report rin tivtions outside our res of interest tht survive whole-rin orretion for multiple omprisons (Tle 1) using fmily-wise error orretion t P o.. We performed supplementry fixed-effets nlysis on tril sis to determine impulse responses, s previously desried 18. Note tht this nlysis refers to the verge impulse response ross eh tril throughout the experiment nd does not emody the time-dependent nture of lerning inorported within the min prmetri nlysis. ACKNOWLEDGMENTS We wish to thnk P. Dyn nd N. Dw for mny helpful disussions nd O. Josephs, B. Johnssen nd C. ikrd for tehnil ssistne. This reserh ws funded y The Wellome Trust. COMPETING INTEESTS STATEMENT The uthors delre tht they hve no ompeting finnil interests. eeived June; epted 2 August 2 Pulished online t 1. Cn, M. Physiologil role of plesure. Siene 173, (1971). 2. Crig,A.D.Anew view of pin s homeostti emotion.trends Neurosi. 26, (23). NATUE NEUOSCIENCE VOLUME 8 [ NUMBE 9 [ SEPTEMBE

7 2 Nture Pulishing Group 3. Fields, H. Stte-dependent opioid ontrol of pin. Nt. ev. Neurosi., 6 7 (24). 4. Solomon,.L. & Corit, J.D. An opponent-proess theory of motivtion. I. Temporl dynmis of ffet. Psyhol. ev. 81, (1974).. Konorski, J. Integrtive Ativity of the Brin: n Interdisiplinry Approh (Chigo, University of Chigo Press, 1967). 6. Shull, J. A onditioned opponent theory of Pvlovin onditioning nd hitution. in The Psyhology of Lerning nd Motivtion (ed. Bower, G.) 7 9 (Ademi, New York, 1979). 7. Grosserg, S. Some norml nd norml ehviorl syndromes due to trnsmitter gting of opponent proesses. Biol. Psyhitry 19, (1984). 8. Solomon,.L. The opponent-proess theory of quired motivtion: the osts of plesure nd the enefits of pin. Am. Psyhol. 3, (198). 9. Solomon,.L. eent experiments testing n opponent-proess theory of quired motivtion. At Neuroiol. Exp. (Wrs.) 4, (198). 1. Dikenson & Dering, MF. Appetitive-versive intertions nd inhiitory proesses. in Mehnisms of Lerning nd Motivtion. (eds. Dikinson, A. & Bokes,.A.) (Erlum, Hillsdle, New Jersey, 1979). 11. Dw, N.D., Kkde, S. & Dyn, P. Opponent intertions etween serotonin nd dopmine. Neurl Netw., (22). 12. Tnimoto, H., Heisenerg, M. & Gerer, B. Experimentl psyhology: event timing turns punishment to rewrd. Nture 43, 983 (24). 13. Brto, A.G. Adptive ritis nd the sl gngli. in Models of Informtion Proessing in the Bsl Gngli (eds. Houk, J.C., Dvis, J.L. & Beiser, D.G.) (MIT Press, Cmridge, Msshusetts, 199). 14. Sutton,.S. & Brto, A.G. einforement Lerning: n Introdution. (MIT Press, Cmridge, Msshusetts, 1998).. Montgue, P.., Dyn, P. & Sejnowski, T.J. A frmework for mesenephli dopmine systems sed on preditive Hein lerning. J. Neurosi. 16, (1996). 16. Shultz, W., Dyn, P. & Montgue, P.. A neurl sustrte of predition nd rewrd. Siene 27, (1997). 17. O Doherty, J.P., Dyn, P., Friston, K., Crithley, H. & Doln,.J. Temporl differene models nd rewrd-relted lerning in the humn rin. Neuron 38, (23). 18. Seymour, B. et l. Temporl differene models desrie higher-order lerning in humns. Nture 429, (24). 19. Dyn, P. & Blleine, B.W. ewrd, motivtion, nd reinforement lerning. Neuron 36, (22). 2. Shwrtz, A. A reinforement lerning method for mximizing undisounted rewrds. in Proeedings of the Tenth Interntionl Conferene on Mhine Lerning (Morgn Kufmnn, Sn Mteo, Cliforni, 1993). 21. Mhdevn, S. Averge rewrd reinforement lerning: Foundtions, lgorithms nd empiril results. Mh. Lern. 22, 1 38 (1996). 22. Fields, H.L. Pin modultion: expettion, opioid nlgesi nd virtul pin. Prog. Brin es. 122, (2). 23. Prie, D.D. Psyhologil Mehnisms of Pin nd Anlgesi (IASP, Settle, 1999). 24. Tnk, S.C. et l. Predition of immedite nd future rewrds differentilly reruits ortio-sl gngli loops. Nt. Neurosi. 7, (24). 2. Ploghus, A. et l. Dissoiting pin from its ntiiption in the humn rin. Siene 284, (1999). 26. Jensen, J. et l. Diret tivtion of the ventrl stritum in ntiiption of versive stimuli. Neuron 4, (23). 27. MClure, S.M., Berns, G.S. & Montgue, P.. Temporl predition errors in pssive lerning tsk tivte humn stritum. Neuron 38, (23). 28. Setlow, B., Shoenum, G. & Gllgher, M. Neurl enoding in ventrl stritum during olftory disrimintion lerning. Neuron 38, (23). 29. Dw, N.D. & Touretzky, D.S. Long-term rewrd predition in TD models of the dopmine system. Neurl Comput. 14, (22). 3. Mrkowitz, H. The utility of welth. J. Polit. Eon. 6, 1 8 (192). 31. Cmerer, C., Loewenstein, G. & Prele, D. Neuroeonomis: how neurosiene n inform eonomis. J. Eon. Lit. (in the press). 32. ogn, M.T., Leon, K.S., Perez, D.L. & Kndel, E.. Distint neurl signtures for sfety nd dnger in the mygdl nd stritum of the mouse. Neuron 46, (2). 33.Wtkins, L.. et l. Neuroiruitry of onditioned inhiition of nlgesi: effets of mygdl, dorsl rphe, ventrl medullry, nd spinl ord lesions on ntinlgesi in the rt. Behv. Neurosi. 112, (1998). 34. Hollnd, P.C. & Gllgher, M. Amygdl-frontl intertions nd rewrd expetny. Curr. Opin. Neuroiol. 14, 148 (24). 3. O Doherty, J., Kringelh, M.L., olls, E.T., Hornk, J. & Andrews, C. Astrt rewrd nd punishment representtions in the humn oritofrontl ortex. Nt. Neurosi. 4, 9 12 (21). 36. Smll, D.M., Ztorre,.J., Dgher, A., Evns, A.C. & Jones-Gotmn, M. Chnges in rin tivity relted to eting hoolte: from plesure to version. Brin 124, (21). 37. Glsher, J. & Buhel, C. Forml lerning theory dissoites rin regions with different temporl integrtion. Neuron 47, (2). 38. Petrovi, P., Klso, E., Petersson, K.M. & Ingvr, M. Pleo nd opioid nlgesi imging shred neuronl network. Siene 29, (22). 39. Wger, T.D. et l. Pleo-indued hnges in FMI in the ntiiption nd experiene of pin. Siene 33, (24). 4. Logothetis, N.K., Puls, J., Augth, M., Trinth, T. & Oeltermnn, A. Neurophysiologil investigtion of the sis of the fmi signl. Nture 412, 7 (21). 41. Stefnovi, B., Wrnking, J.M. & Pike, G.B. Hemodynmi nd metoli responses to neuronl inhiition. Neuroimge 22, (24). 42. Altier, N. & Stewrt, J. The role of dopmine in the nuleus umens in nlgesi. Life Si. 6, (1999). 43. Horvitz, J.C. Mesolimoortil nd nigrostritl dopmine responses to slient nonrewrd events. Neurosiene 96, (2). 44. Smith, A.J., Beker, S. & Kpur, S. A omputtionl model of the funtionl role of the ventrl-stritl D2 reeptor in the expression of previously quired ehviors. Neurl Comput. 17, (2). 4. Ungless, M.A., Mgill, P.J. & Bolm, J.P. Uniform inhiition of dopmine neurons in the ventrl tegmentl re y versive stimuli. Siene 33, (24). 46. oitmn, M.F., Wheeler,.A. & Crelli,.M. Nuleus umens neurons re inntely tuned for rewrding nd versive tste stimuli, enode their preditors, nd re linked to motor output. Neuron 4, (2). 47. Johnsen, J.P. & Fields, H.L. Glutmtergi tivtion of nterior ingulte ortex produes n versive tehing signl. Nt. Neurosi. 7, (24). 48. Gdd, C.A., Murtr, P., De Felipe, C. & Hunt, S.P. Neurokinin-1 reeptor-expressing neurons in the mygdl modulte morphine rewrd nd nxiety ehviors in the mouse. J. Neurosi. 23, (23). 49. Dyn, P. & Aott, L.F. Theoretil Neurosiene: Computtionl nd Mthemtil Modeling of Neurl Systems (MIT Press, Cmridge, Msshusetts, 21).. Buhel, C., Holmes, A.P., ees, G. & Friston, K.J. Chrterizing stimulus-response funtions using nonliner regressors in prmetri fmi experiments. Neuroimge 8, (1998). 124 VOLUME 8 [ NUMBE 9 [ SEPTEMBE 2 NATUE NEUOSCIENCE