Medial prefrontal cortical activity reflects dynamic re-evaluation during voluntary persistence

Transcription

1 Medil prefrontl corticl ctivity reflects dynmic re-evlution during voluntry persistence Joseph T McGuire & Joseph W Kle 215 Nture Americ, Inc. All rights reserved. Deciding how long to keep witing for future rewrds is nontrivil prolem, especilly when the timing of rewrds is uncertin. We crried out n experiment in which humn decision mkers wited for rewrds in two environments in which rewrd-timing sttistics fvored either greter or lesser degree of ehviorl persistence. We found tht decision mkers dptively clirted their level of persistence for ech environment. Functionl neuroimging reveled signls tht evolved differently during physiclly identicl delys in the two environments, consistent with dynmic nd context-sensitive repprisl of sujective vlue. This effect ws oserved in region of ventromedil prefrontl cortex tht is sensitive to sujective vlue in other contexts, demonstrting continuity etween vlution mechnisms involved in discrete choice nd in temporlly extended decisions nlogous to forging. Our findings support model in which voluntry persistence emerges from dynmic cost/enefit evlution rther thn from control process tht overrides vlution mechnisms. Pursuing long-run rewrds often requires persistence in the fce of dely nd short-run costs. The cpcity to dely grtifiction is centrl to the notion of self-control in humn decision-mking, nd filures of persistence cn pper to reflect impulsivity, inconsistency or self-control filure 1,2. We used functionl mgnetic resonnce imging (fmri) to exmine rin ctivity ssocited with sustining or curtiling persistence towrd delyed rewrds. Much is known out neurl systems involved in vlue-sed decision-mking 3 6, ut it is unknown wht role these mechnisms hve in temporlly extended persistence. Most intertemporl choice reserch hs focused on discrete choices mong outcomes tht differ in dely 7 9. Dely-of-grtifiction scenrios, in contrst, involve prolonged dely period with continuously ville opportunity to give up 1. These two types of future-oriented ehvior re widely thought to involve different mentl processes. A previous ccount 1 rgued tht the initil selection of delyed rewrd depends on rtionl cost/enefit ssessment, ut tht the susequent ility to wit for it depends on self-regultory dynmics (competition etween hot nd cool motivtionl systems 11 ). We previously hypothesized tht oth successes nd pprent filures of persistence emerge from dynmic vlue mximiztion 12,13. Becuse the exct timing of future events is usully uncertin, there is no gurntee tht decision mker who ws willing to egin witing for delyed rewrd should necessrily e willing to keep witing indefinitely. In some situtions, including mny tht seem to chllenge self-control, long dely so fr is predictive of longer thn expected dely yet to come One wy to nvigte such situtions would e to ressess the sujective vlue of the wited rewrd s time psses, sed on continuously updted estimte of the remining dely time 12. Such ressessment might e encoded in the sme neurl vlution system, comprised of ventromedil prefrontl cortex (VMPFC), ventrl stritum (VS) nd posterior cingulte cortex (PCC), tht encodes sujective vlue in highly generl mnner cross mny other kinds of decisions 3 6. The sujective vlue representtions encoded in VMPFC re known to e sensitive to oth immedite nd delyed outcomes 7,8, primry nd secondry forms of rewrd 3,16, gol-relted nd tempttionrelted fctors 9,17, nd high-level tsk contingencies 18,19. Other theoreticl perspectives mke different predictions. One lterntive possiility is tht successful persistence depends principlly on cognitive control mechnisms externl to the vlution system. Although some ccounts hold tht the VMPFC vlution system medites cognitive control 9,17,2, other ccounts posit form of control tht overrides or competes with vlution 2,11,2 23. If the ltter control mechnism is prmount, successful persistence might e etter understood s rule dherence thn s vlue mximiztion, nd curtiling persistence might reflect lpse in control-relted rin ctivity (for exmple, in lterl PFC). A second lterntive possiility is sed on the structurl prllel etween dely of grtifiction nd certin kinds of forging scenrios 13,15, It hs recently een hypothesized tht single-lterntive forging decisions for exmple, whether to exploit one s current food ptch or deprt to forge elsewhere might depend, not on the VMPFC vlution system, ut on representtion in dorsl nterior cingulte cortex (dacc) of the vlue of deprting 26,28. To exmine vlution signls during temporlly extended persistence, we conducted n fmri experiment in which prticipnts repetedly decided how long to keep witing for future monetry rewrds (Fig. 1). On ech tril, the prticipnt viewed token tht hd no initil vlue, ut mtured to vlue of $.3 (3 ) fter rndom dely. The prticipnt could sell the token nytime nd initite new tril, iming to mximize totl ernings in fixed time period. Unlike some previous studies 1,13, no smll rewrd ws delivered if the prticipnt quit erly; insted, the min incentive to quit ws the possiility tht the next tril might mture with shorter dely. Deprtment of Psychology, University of Pennsylvni, Phildelphi, Pennsylvni, USA. Correspondence should e ddressed to J.T.M. (mcguirej@psych.upenn.edu) or J.W.K. (kle@psych.upenn.edu). Received 12 Jnury; ccepted 1 Mrch; pulished online 6 April 215; doi:1.138/nn.3994 nture NEUROSCIENCE dvnce online puliction

2 Willingness-to-wit tsk Decision to quit SOLD Dely-length distriutions HP environment LP environment c Expected monetry return HP environment LP environment Token ppers Dely (5 9 s) 3 Token mtures RT (medin = 475 ms) SOLD 3 Key press Feedck/ITI (2 s) Proility mss Dely durtion (s) Proility mss Dely durtion (s) Return ( s 1 ) Witing policy (s) Return ( s 1 ) Witing policy (s) Figure 1 Experimentl tsk nd timing conditions. () Schemtic of the willingness-to-wit tsk. () Discrete proility distriutions governing the scheduled dely times in ech environment. (c) Expected monetry rtes of return under vrious witing policies, where ech policy is defined y giving-up time. The rewrd-mximizing policy ws to wit up to 4 s in the HP environment (tht is, never to quit), ut only up to 2 s in the LP environment. These rtes of return were contingent on the fixed 2-s inter-tril intervl (ITI). 215 Nture Americ, Inc. All rights reserved. The idel strtegy depended on the distriution of dely times, which differed etween two environments (Fig. 1,c). In highpersistence (HP) environment, the most productive strtegy ws to wit for every rewrd (up to 4 s). In limited-persistence (LP) environment, the est strtegy ws to wit 2 s nd then quit if the rewrd hd not rrived. Prticipnts lerned out the timing sttistics through direct experience during preliminry trining. The environments were presented in lternting 1-min runs, mrked y different-colored tokens. We predicted tht prticipnts would quit erlier in the LP environment thn in the HP environment 13. In ddition, our theoreticl model predicted tht prticipnts sujective vlution of the wited token would evolve differently in the two environments, incresing more rpidly with elpsed time in the HP environment thn the LP environment. Using neuroimging, we tested whether cnoniclly vlue-responsive rin regions would reflect this dynmic ressessment. We would lso hve een le to detect lterntive possiilities such s representtions of sujective vlue elsewhere in the rin, lpse in control-relted ctivity ssocited with quitting or representtion of the vlue of quitting in dacc. RESULTS Behviorl results Prticipnts (n = 2) quit efore receiving the rewrd more often in the LP environment (medin = 5.% of trils, interqurtile rnge (IQR) = %) thn in the HP environment (medin = 3.1%, IQR = 15.6%). In the LP environment, the time wited efore quitting (medin of medins) ws 29.3 s (IQR = ). Within-suject (cross-tril) vriility in quit timing ws comprtively smll: the medin size of the within-suject interqurtile rnge ws 9.1 s. Prticipnts were willing to wit longer in the HP environment thn in the LP environment. We used survivl nlysis to estimte ech prticipnt s proility of surviving vrious lengths of time without quitting 13. Figure 2 shows verged suject-wise empiricl survivl curves ginst idel performnce. The re under the curve (AUC) estimtes how much of the first 4 s prticipnt ws willing to wit on verge (Fig. 2). Medin AUC ws 38.9 s in the HP environment (IQR = ; idel = 4 s) nd 3.2 s in the LP environment (IQR = ; idel = 2 s). All 2 prticipnts persisted longer in the HP environment (medin difference = 7.6 s, IQR = , signedrnk P <.1). Persistence in the two environments ws modestly correlted (Spermn ρ n = 2 =.37, P =.11; Fig. 2) nd ehvior ws stle cross the fmri experiment (Supplementry Fig. 1). Rection time (RT) to sell rewrded tokens trcked time-vrying rewrd expectncy. When n event s ltency is uniformly distriuted, expectncy theoreticlly increses with elpsed time 28 (Fig. 2c). Accordingly, suject-wise Spermn correltions etween dely nd RT were relily negtive in the HP environment (medin single-suject ρ =.27, IQR =.36 to.16, signed-rnk P <.1), indicting fster responses to rewrds tht were preceded y longer delys (Fig. 2d) nd implying tht prticipnts successfully encoded the tsk s timing sttistics. Theoreticl modeling The pssge of time cn drive dynmic ressessment of wited rewrds y furnishing informtion out the remining dely 12,29. Intuitively, rewrds in the HP environment grew nerer nd more 1. 4 Figure 2 Behviorl results. () Survivl curves reflecting the proility tht prticipnt ws still witing t ech elpsed time, provided tht the rewrd hd not yet een delivered. Empiricl survivl curves were verged cross sujects t 1-s intervls (± s.e.m.). Idel performnce is plotted for reference (dshed lines). () Are under the curve (AUC) vlues clculted from individul prticipnts survivl curves. The mximum possile vlue ws 4 s. The red point mrks idel performnce. All 2 prticipnts persisted more in the HP environment. (c) Stem plots show the ground-truth hzrd rte for rewrd in ech environment: tht is, the proility of the rewrd rriving t ech time, conditionl on not hving rrived lredy. Fded lines illustrte hypotheticl continuous hzrd functions incorporting endogenous temporl uncertinty (Online Methods). (d) Rewrd RT t ech dely (medin nd IQR of sujectwise medins). RTs re expressed s devitions from ech suject s grnd-medin RT (medin = 475 ms, IQR = ms) to disply within-suject effects. RTs for 5 2-s delys did not differ etween the environments (HP medin = 472 ms, IQR = ; LP medin = 494 ms, IQR = ). Survivl rte c Hzrd rte HP LP Idel HP LP d HP AUC (s) Devition from grnd medin RT (ms) Empiricl Idel LP AUC (s) Dely durtion (s) dvnce online puliction nture NEUROSCIENCE

3 215 Nture Americ, Inc. All rights reserved. Figure 3 Theoreticl sujective vlue of the wited token s function of elpsed time in ech environment. () A token s sujective vlue incresed over time in the HP environment, ut not in the LP environment. These time courses re sed on the discrete ground-truth timing distriutions nd would e smoothed y sujective temporl uncertinty. () Simulted ehvior from model in which sujective vlue linerly influenced the log-odds of continuing to wit (men ± s.e.m. of suject-wise model fits). Dt from Figure 2 re overlid for reference. (c) Sujective vlue (SV) time courses convolved with cnonicl hemodynmic response function (HRF). (d) Predicted BOLD time courses otined y pplying our fmri nlysis to idelized synthetic dt (men ± s.e.m. of individul suject results). Visul differences from c reflect tht the HP nd LP environments hd independent selines, nd there ws smll degree of crryover cross trils. In spite of these differences, the theoreticl difference time courses (HP minus LP) were highly correlted etween c nd d (medin r 2 =.88, IQR =.84.89). sujectively vlule s time pssed, ut rewrds in the LP environment ecme progressively less likely to e delivered efore the prticipnt quit. We formlized this intuition in theoreticl model of sujective vlution. The model estimtes the wited token s sujective vlue t ech point in the dely intervl, ccounting for the chnging proility distriution over remining dely durtions. Our model extends formlism from the optiml forging literture known s the potentil function 25. The expected remining dely is multiplied y the opportunity cost of time nd sutrcted from the expected rewrd. Sujective vlue t given elpsed time equls the expected net return in the reminder of the current tril, mximized over ll possile giving-up times. Its minimum is zero, s the gent cn lwys quit immeditely. If the sujective vlue exceeds zero, this signifies tht the decision mker could do etter y witing thn y quitting immeditely. The level of sujective vlue t ech time reflects the mrgin of preference for witing over quitting (Online Methods). In the HP environment, the token s theoreticl sujective vlue incresed with elpsed time, reflecting the progressive shortening of the expected remining dely (Fig. 3). In the LP environment, the token s sujective vlue remined positive until 2 s, ut then fell to zero, reflecting tht the est strtegy ws to quit if the rewrd hd not rrived y then. Differences etween the sujective vlue trjectories in the two environments were primrily driven y the evolving proility tht the rewrd would rrive efore the optiml giving-up time (Supplementry Fig. 2). We modeled the empiricl ehviorl dt s stochstic function of theoreticl sujective vlue using logistic regression (Fig. 3 nd Online Methods). Greter sujective vlue ws ssocited with higher odds of witing (medin coefficient =.26, IQR =.5.78, signed-rnk P <.1). The sujective vlue model significntly outperformed n intercept-only model (suject-wise likelihood-rtio tests: medin z = 4.26, IQR , signed-rnk P <.1) nd n lterntive model tht directly fit different overll rtes of quitting in the HP nd LP conditions (suject-wise difference of model devinces: medin = 6.45, IQR = , signed-rnk P =.33). Figure 4 Model-sed contrst results. () Whole-rin nlysis. Displyed in red is the VMPFC cluster tht showed significnt reltionship with the theoreticl sujective vlue time courses in Figure 3d. In yellow, for reference, re regions identified in previous met-nlysis of vlution effects (the regions reported in Fig. 3d of ref. 3). Overlp ws oserved in VMPFC, ut not in PCC or stritum. () Model-sed contrst vlues for ech prticipnt, sptilly verged in met-nlytic ROIs. Sujective vlue effects were significntly positive in VMPFC nd significntly greter in VMPFC thn in stritum. Sujective vlue ( ) c HRF-convolved SV HP LP Neuroimging results Our fmri nlyses tested for rin signls tht evolved differently during physiclly identicl dely intervls in the two environments. Tril onset locked BOLD time courses were flexily estimted in ech environment using finite impulse response (FIR) model consisting of series of single time-point sis functions in generl liner model (GLM). Ech tril ws modeled from onset up to 1 s efore the outcome (rewrd cue or quit response). Becuse trils hd different durtions, erlier time points were oserved on more trils thn lter time points (Supplementry Fig. 3). Group nlyses focused on the intervl from s, for which 19 of 2 prticipnts contriuted complete dt. Becuse the HP nd LP conditions were presented in seprte scnning runs with independent selines, our nlyses focused on differentil chnge over time, not the overll offset etween the two conditions. Significnce ws ssessed using whole-rin permuttion tests to control for multiple comprisons (Online Methods). A model-sed fmri contrst tested directly for effects of theoreticl sujective vlue on BOLD ctivity. For ech suject nd voxel, the empiricl difference time course (HP minus LP) ws regressed on the predicted difference (Fig. 3c,d nd Online Methods) long with constnt intercept. The resulting contrst coefficient reflected the degree to which BOLD signl incresed more steeply with elpsed time in the HP environment thn the LP environment. s were sumitted to whole-rin, two-tiled, group-level test (n = 2). This identified single significnt cluster, locted in VMPFC (Fig. 4 nd Tle 1), in which BOLD signl ws positively relted to theoreticl sujective vlue. No negtive effects of sujective vlue on BOLD were identified, even in follow-up nlyses tilored to detect signls reflecting the difficulty of persistence (Supplementry Fig. 4). z = L x = 3 SV met-nlysis Empiricl effect Overlp Survivl rte d SV coefficient HP: model LP: model.2 HP: dt LP: dt Stritum VMPFC PCC SV met-nlysis region nture NEUROSCIENCE dvnce online puliction

4 215 Nture Americ, Inc. All rights reserved. Tle 1 Pek foci of BOLD effects Region x y z Cluster extent Pek vlue Cluster P vlue Tril onset locked time courses: model-sed contrst (t sttistic) VMPFC Tril onset locked time courses: condition time interction (F sttistic) L VMPFC R VMPFC L posterior prietl L superior temporl gyrus Anticiptory ctivity in quit-relted time courses: min effect of time (F sttistic), 12.5 s to 2.5 s R posterior prietl L posterior prietl R nterior insul DMFC R nterior PFC The oserved effect in VMPFC echoes the effects of sujective vlue tht re seen in rod rnge of other contexts 3 6. We formlly juxtposed our results with previous findings y quntifying the sptil overlp etween our empiricl results nd cnoniclly vlutionrelted rin regions derived from 26-study met-nlysis 3 (Fig. 4). The met-nlysis hd identified clusters showing preferentilly positive effects of vlue in VMPFC (9.67 cm 3 ), stritum (21.41 cm 3 ) nd PCC (2.62 cm 3 ). There ws 1-voxel (2.7 cm 3 ) region of overlp in VMPFC (27.9% of the cnonicl region nd 32.2% of the empiricl cluster). As n lterntive test of the sme question, the three cnonicl vlution res were tested s regions of interest (ROIs). Model-sed contrst coefficients were sptilly verged in ech ROI for ech prticipnt. The effect of sujective vlue ws significnt in VMPFC (signed-rnk P =.2), ut non-significnt, leit with positive trend, in stritum (P =.79) nd PCC (P =.62; Fig. 4). Piredsmples comprisons identified greter effect in VMPFC thn stritum (signed-rnk P =.12) nd no significnt differences etween the other two pirs of ROIs (P >.11). In summry, results suggest tht the region of VMPFC previously found to encode sujective vlue during discrete choices nd outcomes lso reflects dynmic ressessment of sujective vlue during voluntry persistences. This ws true to greter degree for VMPFC thn stritum. We lso conducted less-constrined nlysis tht could detect BOLD time course differences predicted y either our model or lterntive frmeworks. Tril onset locked time courses were nlyzed t the group level in whole-rin voxel-wise repeted-mesures ANOVA (n = 19), with fctors for condition (HP versus LP) nd time point. We focused on the condition time point interction, seeking to identify signls tht exhiited different ptterns of chnge over time in the two environments. This nlysis voids priori ssumptions out either the form of the difference or the loction of effects in the rin. A significnt interction ws oserved in left nd right VMPFC, left posterior prietl cortex, nd smll region of left superior temporl gyrus (Tle 1 nd Fig. 5). Time course plots (Fig. 5 e) suggested tht, in VMPFC nd prietl regions, the effect took the form of greter signl increse with elpsed time in the HP environment, consistent with theoreticl sujective vlue. Further nlyses tested for evidence of rewrd prediction error (RPE) signls 3. When rewrd occurs, RPE is the difference etween the otined nd expected outcome. Becuse rewrd expectncy theoreticlly rose over time in the HP environment (Fig. 2c; see lso RT dt ove nd hert rte dt elow), rewrds t short delys should hve een more surprising nd evoked lrger RPEs thn rewrds t longer delys. We tested whether the mplitude of the phsic BOLD response to rewrd ws modulted y the dely durtion tht preceded it. A negtive effect would reflect n RPE-like pttern (smller rewrd responses fter longer delys, pttern seen previously in the firing rtes of dopminergic midrin neurons 31 ). To focus on phsic rewrd responses while controlling for nonspecific effects of elpsed time, we compred the modultory effect in the post-rewrd epoch ginst the sme effect in pre-rewrd epoch. We oserved no significnt negtive modultory effect of elpsed time on the rewrd-relted BOLD response in ny loction. We did, however, identify n occipitoprietl cluster with n effect in the opposite direction: higher mplitude BOLD response to rewrds t longer delys, which theoreticlly were more strongly nticipted (Supplementry Fig. 5). Expectncy-driven mplifiction of rin responses hs een seen efore 32, including in visul cortex 33 ; these effects er fmily resemlnce to the fcilittory effects of sptil ttention 34. Numerous rin res responded differentilly to rewrd nd quit key presses, including some tht exhiited rmp-up in ctivity efore quit responses. We used GLM to estimte suject-wise peri-event time courses for the two event types seprtely (using ll key presses cross ll four runs), nd sumitted the difference etween rewrdrelted nd quit-relted time courses to group-level ANOVA. Effects occurred diffusely cross dorsomedil frontl cortex (DMFC), lterl PFC, nterior insul, precentrl sulcus, nd occipitl nd posterior prietl cortex (Fig. 6). In DMFC, nterior insul, posterior prietl cortex nd nterior PFC, the difference consisted of n erlier elevted response for quit responses thn rewrd responses. Other regions, including occipitl cortex nd left inferior frontl gyrus (IFG), responded more strongly to rewrds. Brodly, these effects reflect L L VMPFC c R VMPFC.4 HP LP.2 d.3 L prietl e.4 L STG z = 12 z = z = Figure 5 Model-free nlysis of tril onset locked BOLD time courses. () Clusters showing significnt time point environment interction (Tle 1). ( e) Sptilly verged signl time courses for significnt clusters (men ± s.e.m.), illustrting the form of the oserved interctions. Although voxel selection effects would distort follow-up inferentil tests of these time courses, we descriptively summrized their reltionship to our theoreticl predictions in terms of the correltion etween the verge theoreticl (Fig. 3d) nd oserved HP minus LP difference time courses. The resulting Person r vlues were.91,.89,.9 nd.68 for the results shown in e, respectively. dvnce online puliction nture NEUROSCIENCE

5 215 Nture Americ, Inc. All rights reserved. Figure 6 Regions in which BOLD signl differentited rewrd-relted nd quit-relted key presses, ssessed on the sis of the event type (rewrd versus quit) y time point interction. ( f) Wrm colors represent F sttistics for the nlysis of full time courses nd crosshirs mrk locl peks. Blue outlines mrk regions significnt in the nlysis of pre-quit time points only. Time courses (men ± s.e.m.) re plotted for 6-mm-rdius (33 voxel) sphere centered t ech focus point (green crosshirs). Blck dshed lines mrk the key press time; lue dshed lines mrk the medin rewrd cue time (for rewrd-relted key presses). tht rewrds involved visul cue, wheres e quitting ws freely timed nd volitionl. To test directly for signl chnges tht preceded decisions to quit, we performed group-level ANOVA on only the first five points in the quit-relted time course ( 12.5 to 2.5 s). A significnt effect of x = 6 time point in this nticiptory intervl ws oserved in posterior prietl cortex, DMFC, nterior insul nd nterior PFC (Fig. 6 nd Tle 1). VMPFC showed no effects in either of the ove nlyses; tht is, there ws no evidence tht sujective vlue effects in VMPFC could e lterntively explined in terms of role in response preprtion. Somtic rousl To test whether sujective vlue effects in BOLD ctivity were ccompnied y chnges in generl physiologicl rousl, we performed explortory nlyses of hert rte (inter-et intervl mesured vi pulse oximetry, n = 17) s function of tsk events. Hert rte trnsiently ccelerted fter key presses, ut did not differ etween the two conditions s function of dely time (Fig. 7). In the HP condition, there ws greter trnsient crdic ccelertion for rewrds preceded y longer delys (Fig. 7), olstering our conclusion lso supported y RTs nd occipitoprietl BOLD effects tht sujective rewrd expectncy incresed with elpsed time in the HP condition. Compring hert-rte time courses for rewrd nd quit events reveled crdic decelertion, well-known correlte of motor preprtion 35, efore quit responses (Fig. 7c). In summry, pre nd post key press IBI (% of seline) IBI (% of seline) c x = 42 x = Rewrd events (HP condition) Short dely Long dely Reltive time (s) Tril onset Reltive time (s) c HP LP Rewrd versus quit events HP rewrd LP quit Reltive time (s) R nterior insul Rewrd Quit R posterior prietl DMFC Time from key press (s) d f x = 33 x = 12 x = 42 rin responses (Fig. 6 f) co-occurred with chnges in generl somtic rousl, ut there ws no evidence tht rousl effects (s indexed y hert rte) ccompnied the tril onset locked BOLD effects of theoreticl sujective vlue (Figs. 4 nd 5). DISCUSSION Decision mkers fced with uncertin dely should repprise wited rewrds s time psses. Depending on the sttistics of the environment, the pssge of time my either decrese or increse one s estimte of how long dely remins. This type of dynmic ressessment offers rtionle for sustining or curtiling persistence. We elicited either greter or lesser willingness to persist in lortory environments y mnipulting the timing sttistics tht governed rewrd delivery. Decision mkers clirted their level of persistence dptively; this extends previous demonstrtions of environmentspecific clirtion of intertemporl choice ehvior 13,36. Convergent RT, BOLD nd hert-rte dt suggest tht prticipnts encode the relevnt timing sttistics, responding more vigorously to more strongly expected rewrds 28. Behvior still fell short of optimlity, nd n importnt gol for future work is to determine whether this ws result of inexct sttisticl lerning, strong prior expecttions, stochstic noise, unmodeled sources of vlue (for exmple, nticiption 37 ) or other cuses. Future work should lso test whether R nterior PFC Occipitl L inferior frontl gyrus Time from key press (s) Figure 7 Effects of tsk events on men crdic inter-et intervl (IBI; lower vlues correspond to fster hert rte). Error nds show s.e.m.; red nds mrk significnt differences. () Men tril onset locked IBI time course in ech condition. Verticl red dshed line mrks tril onset; gry dshed line mrks the preceding key press. Ech tril contriuted dt until 1 s efore the tril ended (lter time points therefore hve fewer oservtions thn erlier time points). No significnt differences were oserved. () Comprison etween rewrds rriving t shorter (5 2s) versus longer (25 4 s) delys in the HP condition. The mplitude of post key press hert-rte ccelertion ws greter for rewrds tht followed longer delys (lg, 1 s to 2.75 s; permuttion-sed P =.18). (c) Comprison etween rewrd events in the HP condition nd quit events in the LP condition, ech restricted to trils with durtion >1 s. Verticl red dshed line mrks the time of the rewrd cue or quit key press. Results suggest trnsient crdic decelertion efore quit responses (lg, 1 s to 3 s; permuttion-sed P =.45). nture NEUROSCIENCE dvnce online puliction

6 215 Nture Americ, Inc. All rights reserved. performnce would differ if immedite or viscerlly tempting rewrds were t stke (for exmple, ppetizing foods insted of money) 1. The success of the ehviorl mnipultion enled us to exmine time-dependent rin signls ssocited with either high or limited ehviorl persistence. We oserved signls in VMPFC consistent with dynmic nd context-sensitive ressessment of the wited outcome s sujective vlue. This effect ws identified using oth model-guided nd explortory fmri time course nlyses, oth t the whole-rin level nd in ROIs previously implicted in sujective evlution. VMPFC nd persistence Persistence towrd future rewrds hs een clssiclly understood to depend on self-regultory psychologicl processes tht compete with nd override more impulsive, rewrd-sensitive processes 1,2,11. Dulsystem psychologicl models hve given rise to the neuroscientific hypothesis tht competitive dynmics exist etween rin regions suserving cognitive control nd rewrd processing In contrst with this stndrd view, we hve proposed tht dely-ofgrtifiction decisions depend on dynmic repprisl of the wited future rewrd 12,13. This ccount ttriutes differences in witing ehvior cross individuls nd situtions to fctors such s temporl eliefs, perceived outcome vlues nd the perceived cost of time, not merely to differences in the cpcity to exert self-control 12. Here we elicited differences in witing y mnipulting temporl eliefs nd otined evidence for time-vrying representtion of sujective vlue. The hypothesized signl is context dependent, evolving over time in mnner tht depends on the timing sttistics of the current environment. A corresponding BOLD trjectory ws identified in VMPFC, corticl region regrded s prt of finl common pthwy in the prospective evlution of choice lterntives 6. These results re consistent with the view tht persistence depends on the sme neurl nd cognitive processes tht guide other forms of rewrd evlution nd economic choice. This view implies tht dptive persistence depends on ccurtely representing the vlue of witing, nd need not depend on the enggement of effortful inhiitory control processes 38. Our results dd to the lrge ody of evidence tht VMPFC vlution processes utilize detiled representtion of higher order tsk structure 18,19. Our findings lso extend current conceptions of VMPFC function; lthough VMPFC ctivity is known to encode phsic sujective vlue during discrete choices 3 8, we found tht it lso trcked sujective vlue in temporlly extended mnner (see ref. 39 for relted finding). Our neuroimging results suggest tht there is no need to posit ntgonistic dynmics etween neurl rewrd systems nd control systems to explin voluntry persistence (though we cnnot, of course, rule out such dynmics in other situtions). Our nlyses could hve detected ptterns suggestive of dul-system competition. For exmple, the nlyses in Figures 4 nd 5 nd Supplementry Figure 4 could hve detected ctivity scling with the difficulty of persistence, ut no such effects were found. The nlyses in Figure 6 could hve detected lpse in control-relted ctivity efore decisions to quit, ut insted the opposite occurred: n ensemle of regions previously implicted in cognitive control, lterl PFC, DMFC, insul nd prietl cortex, incresed ctivity efore quits, consistent with rin responses found to precede shifts of strtegy in other tsk prdigms 26,4. Our findings re more comptile with the hypothesis tht cognitive control opertes vi vlue modultion 9,17,2. The vlue modultion hypothesis stipultes tht control mechnisms in lterl PFC operte y modulting sujective vlue representtions in VMPFC. The hypothesis therefore posits VMPFC signl tht incorportes ll relevnt informtion nd suffices s finl common pthwy to guide decisions, consistent with our findings. It lso posits tht this signl depends on lterl PFC inputs. On this point our dt re mostly silent. We found no evidence for condition-dependent ctivtion trjectories in lterl PFC; nevertheless, we ssume tht vlue computtion involves interctions mong multiple rin regions, nd we cnnot exclude the possiility tht lterl PFC prticiptes. Vlue representtion during forging The prolem of clirting persistence in our willingness-to-wit tsk is closely nlogous to the ptch deprture prolem in forging It hs recently een hypothesized tht forging, which typiclly involves succession of ccept-or-reject decisions, imposes fundmentlly different informtion-processing demnds from stndrd multi-lterntive economic choice 41. Recent work hs implicted dacc in signling the vlue of exiting forging ptches 26 or of shifting wy from defult lterntives 41, lthough other findings hve questioned this ide 42. We did not find evidence for continuous, prospective encoding of the vlue of quitting (nlogous to ptch deprture) in dacc. Such signl would theoreticlly hve followed n inverted version of the vlue of witing (Fig. 3c,d), nd could hve een detected in either our model-sed nlysis (s negtive effect) or our explortory time course nlyses. We did, however, oserve response in dacc nd other frontl nd prietl regions in nticiption of quit decisions. This pttern is consistent with generl motor preprtion s well s with the possiility tht decision-relted signls in dacc mnifest predominntly during overt choice execution 26,43. Our results point to role for VMPFC vlution signls even in forging-like sitution in which decision mkers encounter one opportunity t time nd seek to mximize their overll rte of return. VMPFC ctivity correlted with the vlue of the current opportunity (witing for the current token). This finding grees with the ide tht VMPFC encodes est minus next est comprtive vlue signl 41 even when the next est is the constnt ckground option of moving on to new opportunity. This prllels previous demonstrtions tht VMPFC reflects the sujective vlue of individul options tht re evluted in turn ginst fixed reference lterntive 7. Our findings suggest continuity etween the vlution mechnisms involved in temporlly extended forging scenrios nd multi-option economic choice. Rewrd prediction error The willingness-to-wit tsk theoreticlly involves oth positive nd negtive RPE. Positive RPE should ccompny rewrd delivery, s, given temporl uncertinty, rewrds re not fully predicted t the specific time they re delivered 31. Conversely, the pre-rewrd intervl (when the rewrd could hve occurred, ut did not) presumly involves negtive RPE 44,45. Long delys in the HP environment highlight the dissociility of vlue nd RPE signls. Rewrd expectncy rmped up over time (Fig. 2c), so non-rewrd should e ssocited with progressively lrger negtive RPE, even s the wited rewrd s sujective vlue stedily increses (Fig. 3). Even though decision mkers my e incresingly surprised tht the rewrd did not come now, they re lso incresingly confident tht it will rrive soon. One potentil explntion for the lck of cler RPE signls in our neuroimging dt might e tht, t lest t the resolution of fmri, RPE nd sujective vlue signls were superimposed. Sujective vlue is cnoniclly ssocited with BOLD ctivity in VMPFC, PCC nd stritum 3 5, nd rod stnding question is how these regions might differ in their computtionl contriutions to decision ehvior. One possiility is tht stritum preferentilly encodes RPE 46,47, wheres VMPFC preferentilly encodes prospective dvnce online puliction nture NEUROSCIENCE

7 215 Nture Americ, Inc. All rights reserved. decision vlues 6,47. Our findings pper to e comptile with such distinction: dynmic signl of prospective sujective vlue ws oserved in VMPFC, ut ws significntly less evident in stritum. However, these results will need to e integrted with insights gined using other neuroscientific techniques; recent evidence from direct dopmine recordings suggests tht stritum my indeed exhiit rmping pre-rewrd signl 48, nd other work points to n importnt role for serotonergic neuromodultion in ehviorl persistence 49. It will lso e importnt for future reserch to ssess the fidelity with which VMPFC encodes the individul components of sujective vlue (Supplementry Fig. 2), to isolte vlution from relted fctors such s moment-y-moment rewrd proility 33,5, nd to test the generlity of these effects cross different mgnitudes nd types of rewrds. Reserch on these topics will yield n enriched picture of how the rin s vlution mechnisms contend with the complexity of rel-world decision environments. Methods Methods nd ny ssocited references re ville in the online version of the pper. Note: Any Supplementry Informtion nd Source Dt files re ville in the online version of the pper. Acknowledgments This reserch ws supported y NIH grnts DA387 to J.T.M. nd DA29149 to J.W.K. AUTHOR CONTRIBUTIONS J.T.M. nd J.W.K. designed the experiment, developed the nlysis procedures, nd wrote the pper. J.T.M. collected nd nlyzed the dt nd developed the theoreticl model. COMPETING FINANCIAL INTERESTS The uthors declre no competing finncil interests. Reprints nd permissions informtion is ville online t reprints/index.html. 1. Mischel, W. & Eesen, E.B. Attention in dely of grtifiction. J. Pers. Soc. Psychol. 16, (197). 2. Bumeister, R.F., Vohs, K.D. & Tice, D.M. The strength model of self-control. Curr. Dir. Psychol. Sci. 16, (27). 3. Brtr, O., McGuire, J.T. & Kle, J.W. The vlution system: A coordinte-sed met-nlysis of BOLD fmri experiments exmining neurl correltes of sujective vlue. Neuroimge 76, (213). 4. Clithero, J.A. & Rngel, A. Informtic prcelltion of the network involved in the computtion of sujective vlue. Soc. Cogn. Affect. Neurosci. 9, (213). 5. Liu, X., Hirston, J., Schrier, M. & Fn, J. Common nd distinct networks underlying rewrd vlence nd processing stges: met-nlysis of functionl neuroimging studies. Neurosci. Bioehv. Rev. 35, (211). 6. Levy, D.J. & Glimcher, P.W. The root of ll vlue: A neurl common currency for choice. Curr. Opin. Neuroiol. 22, (212). 7. Kle, J.W. & Glimcher, P.W. The neurl correltes of sujective vlue during intertemporl choice. Nt. Neurosci. 1, (27). 8. Kle, J.W. & Glimcher, P.W. An s soon s possile effect in humn intertemporl decision mking: Behviorl evidence nd neurl mechnisms. J. Neurophysiol. 13, (21). 9. Hre, T.A., Cmerer, C.F. & Rngel, A. Self-control in decision-mking involves modultion of the vmpfc vlution system. Science 324, (29). 1. Mischel, W., Ayduk, O. & Mendoz-Denton, R. Sustining dely of grtifiction over time: hot-cool systems perspective. in Time nd Decision: Economic nd Psychologicl Perspectives on Intertemporl Choice (eds. Loewenstein, G., Red, D. & Bumeister, R.F.) (Russell Sge Foundtion, New York, 23). 11. Metclfe, J. & Mischel, W. A hot/cool-system nlysis of dely of grtifiction: dynmics of willpower. Psychol. Rev. 16, 3 19 (1999). 12. McGuire, J.T. & Kle, J.W. Rtionl temporl predictions cn underlie pprent filures to dely grtifiction. Psychol. Rev. 12, (213). 13. McGuire, J.T. & Kle, J.W. Decision mkers clirte ehviorl persistence on the sis of time-intervl experience. Cognition 124, (212). 14. Rchlin, H. The Science of Self Control (Hrvrd University Press, 2). 15. Dsgupt, P. & Mskin, E. Uncertinty nd hyperolic discounting. Am. Econ. Rev. 95, (25). 16. Kim, H., Shimojo, S. & O Doherty, J.P. Overlpping responses for the expecttion of juice nd money rewrds in humn ventromedil prefrontl cortex. Cere. Cortex 21, (211). 17. Hre, T.A., Mlmud, J. & Rngel, A. Focusing ttention on the helth spects of foods chnges vlue signls in vmpfc nd improves dietry choice. J. Neurosci. 31, (211). 18. Hmpton, A.N., Bosserts, P. & O Doherty, J.P. The role of the ventromedil prefrontl cortex in strct stte-sed inference during decision mking in humns. J. Neurosci. 26, (26). 19. Dw, N.D., Gershmn, S.J., Seymour, B., Dyn, P. & Doln, R.J. Model-sed influences on humns choices nd stritl prediction errors. Neuron 69, (211). 2. Hutcherson, C.A., Plssmnn, H., Gross, J.J. & Rngel, A. Cognitive regultion during decision mking shifts ehviorl control etween ventromedil nd dorsolterl prefrontl vlue systems. J. Neurosci. 32, (212). 21. Csey, B.J. et l. Behviorl nd neurl correltes of dely of grtifiction 4 yers lter. Proc. Ntl. Acd. Sci. USA 18, (211). 22. Figner, B. et l. Lterl prefrontl cortex nd self-control in intertemporl choice. Nt. Neurosci. 13, (21). 23. Hetherton, T.F. & Wgner, D.D. Cognitive neuroscience of self-regultion filure. Trends Cogn. Sci. 15, (211). 24. Chrnov, E.L. Optiml forging, the mrginl vlue theorem. Theor. Popul. Biol. 9, (1976). 25. McNmr, J. Optiml ptch use in stochstic environment. Theor. Popul. Biol. 21, (1982). 26. Hyden, B.Y., Person, J.M. & Pltt, M.L. Neuronl sis of sequentil forging decisions in ptchy environment. Nt. Neurosci. 14, (211). 27. Fwcett, T.W., McNmr, J.M. & Houston, A.I. When is it dptive to e ptient? A generl frmework for evluting delyed rewrds. Behv. Processes 89, (212). 28. Nickerson, R.S. Response time to the second of two successive signls s function of solute nd reltive durtion of intersignl intervl. Percept. Mot. Skills 21, 3 1 (1965). 29. Griffiths, T.L. & Tenenum, J.B. Optiml predictions in everydy cognition. Psychol. Sci. 17, (26). 3. Montgue, P.R., Dyn, P. & Sejnowski, T.J. A frmework for mesencephlic dopmine systems sed on predictive Hein lerning. J. Neurosci. 16, (1996). 31. Fiorillo, C.D., Newsome, W.T. & Schultz, W. The temporl precision of rewrd prediction in dopmine neurons. Nt. Neurosci. 11, (28). 32. Cui, X., Stetson, C., Montgue, P.R. & Eglemn, D.M. Redy.go: mplitude of the FMRI signl encodes expecttion of cue rrivl time. PLoS Biol. 7, e1167 (29). 33. Bueti, D., Bhrmi, B., Wlsh, V. & Rees, G. Encoding of temporl proilities in the humn rin. J. Neurosci. 3, (21). 34. Tootell, R.B.H. et l. The retinotopy of visul sptil ttention. Neuron 21, (1998). 35. Lcey, J.I. & Lcey, B.C. Some utonomic-centrl nervous system interreltionships. in Physiologicl Correltes of Emotion (ed. Blck, P.) (Acdemic Press, New York, 197). 36. Schweighofer, N. et l. Humns cn dopt optiml discounting strtegy under reltime constrints. PLoS Comput. Biol. 2, e152 (26). 37. Loewenstein, G. Anticiption nd the vlution of delyed consumption. Econ. J. 97, (1987). 38. Duckworth, A.L., Gendler, T.S. & Gross, J.J. Self-control in school-ge children. Educ. Psychol. 49, (214). 39. Jimur, K., Chushk, M.S. & Brver, T.S. Impulsivity nd self-control during intertemporl decision mking linked to the neurl dynmics of rewrd vlue representtion. J. Neurosci. 33, (213). 4. Helfinstein, S.M. et l. Predicting risky choices from rin ctivity ptterns. Proc. Ntl. Acd. Sci. USA 111, (214). 41. Rushworth, M.F.S., Kolling, N., Sllet, J. & Mrs, R.B. Vlution nd decisionmking in frontl cortex: one or mny seril or prllel systems? Curr. Opin. Neuroiol. 22, (212). 42. Shenhv, A., Strcci, M.A., Cohen, J.D. & Botvinick, M.M. Anterior cingulte enggement in forging context reflects choice difficulty, not forging vlue. Nt. Neurosci. 17, (214). 43. Blnchrd, T.C. & Hyden, B.Y. Neurons in dorsl nterior cingulte cortex signl postdecisionl vriles in forging tsk. J. Neurosci. 34, (214). 44. McClure, S.M., Berns, G.S. & Montgue, P.R. Temporl prediction errors in pssive lerning tsk ctivte humn stritum. Neuron 38, (23). 45. Hollermn, J.R. & Schultz, W. Dopmine neurons report n error in the temporl prediction of rewrd during lerning. Nt. Neurosci. 1, (1998). 46. Berns, G.S., McClure, S.M., Pgnoni, G. & Montgue, P.R. Predictility modultes humn rin response to rewrd. J. Neurosci. 21, (21). 47. Hre, T.A., O Doherty, J., Cmerer, C.F., Schultz, W. & Rngel, A. Dissociting the role of the oritofrontl cortex nd the stritum in the computtion of gol vlues nd prediction errors. J. Neurosci. 28, (28). 48. Howe, M.W., Tierney, P.L., Snderg, S.G., Phillips, P.E.M. & Gryiel, A.M. Prolonged dopmine signling in stritum signls proximity nd vlue of distnt rewrds. Nture 5, (213). 49. Miyzki, K.W. et l. Optogenetic ctivtion of dorsl rphe serotonin neurons enhnces ptience for future rewrds. Curr. Biol. 24, (214). 5. Jnssen, P. & Shdlen, M.N. A representtion of the hzrd rte of elpsed time in mcque re LIP. Nt. Neurosci. 8, (25). nture NEUROSCIENCE dvnce online puliction

8 215 Nture Americ, Inc. All rights reserved. ONLINE METHODS Prticipnts. The prticipnts were 2 memers of the University of Pennsylvni community (ge 18 3, men = 22, 11 femle). Two dditionl prticipnts were excluded for hed movement (shifts of t lest.5 mm etween >5% of djcent time points). Prticipnts were pid show-up fee ($15 per h) plus rewrds erned in the tsk (medin = $19.8). All prticipnts provided informed consent. The procedures were pproved y the University of Pennsylvni Internl Review Bord. No sttisticl methods were used to predetermine smple size, ut our smple size ws similr to those reported in previous pulictions 16,18,19,32,42,47. Tsk. The tsk ws progrmmed using Mtl (The MthWorks) with Psychophysics Toolox extensions 51,52. A circulr token, colored green or purple, ppered in the center of the screen, leled. After rndom dely, the token turned lue nd its vlue chnged to 3. Prticipnts could sell the token nytime y pressing key with their right hnd. The word SOLD ppered in red over the token for 1 s. After 1-s lnk screen, new token ppered. The previous token s vlue ws dded to the prticipnt s totl ernings, which were displyed only t the end of ech scnning run. Setting the token s initil vlue to ment tht, unlike erlier work using this procedure 13, prticipnts received no immedite rewrd upon quitting. This served to simplify the tsk without sustntilly ltering either its incentive structure or the resulting pttern of ehvior. A white progress r mrked the mount of time the current token hd een on the screen. The r s full length corresponded to 1 s. It grew continuously from the left nd reset when new token ppered. The progress r ws included to reduce intervl-timing demnds nd discourge strtegy of covertly counting time. The scnning session ws divided into four 1-min runs. New tokens were presented until time ws up. Ech run presented one timing environment (corresponding to one token color). The two environments lternted in successive runs. The order of environments nd the mpping of token color to environment were counterlnced cross prticipnts. Ech prticipnt completed preliminry ehviorl trining session consisting of four 1-min runs lternting etween the HP nd LP environments. Prticipnts were explicitly instructed tht the green nd purple tokens might differ in their timing, ut tht they hd to lern the nture of the differences from direct experience nd were free to dopt ny ehviorl strtegy they preferred. During ehviorl trining (ut not during scnning) the screen displyed the time left in the 1-min run nd the mount erned so fr, to help ensure tht prticipnts understood the structure of the tsk. Ech token during ehviorl trining ws worth 1. Prticipnts explored the tsk environments during trining, witing through full 9-s delys in the LP condition on medin of 3.5 trils (IQR = 1 5.5, > for 18/2 sujects). Prticipnts completed two dditionl 5-min runs (one per condition) outside the scnner just efore the fmri session. Witing ehvior over time cross trining, prctice, nd fmri sessions is plotted in Supplementry Figure 1. Prticipnts would hve fced fundmentlly the sme tril-y-tril decision prolem if they hd received explicit informtion out the proilistic contingencies 53 in lieu of experience-sed trining. However, there is evidence tht proilistic informtion is encoded differently when lerned from description versus direct experience ; our trining procedure ws designed to involve the type of experience-sed, implicit sttisticl lerning tht is thought to guide eliefs nd expecttions in rel-world domins 29, including ecologicl forging environments. Future work might introduce explicit informtion to help ssess whether devitions from optiml ehvior were due to inexct encoding of the relevnt proilities. The dely durtion on ech tril ws rndomly drwn from discrete proility distriution (Fig. 1). In the HP environment delys were drwn uniformly from the vlues 5, 1, 15, 2, 25, 3, 35 nd 4 s. In the LP environment delys were set to 9 s with proility.5, nd otherwise drwn uniformly from the vlues 5, 1, 15 nd 2 s. By design, rewrd proilities were identicl etween the two environments for the first 2 s of the dely, the period of gretest interest in our neuroimging nlyses. We imposed longer delys here thn in our previous work 13 to otin fluctutions in sujective vlue cross time period on the order of 3 s, which is well suited for detecting BOLD effects (this corresponds to the time scle of locked design with ~15-s locks; see further discussion nd simultion results elow). Delys were smpled in pseudorndom mnner tht pproximtely lnced the first-order trnsition sttistics etween delys in successive trils. This helped ensure tht the scheduled delys were representtive of the ground-truth distriution, while voiding the negtive utocorreltion tht would result from strictly lnced frequencies. The HP environment ws richer y design, with ll prticipnts receiving more rewrds in the HP environment (medin = 44, IQR = ) thn in the LP environment (medin = 25, IQR = 21 26). The difference in overll richness ws not the fctor tht determined the idel ehviorl strtegy (one could design richer LP environments nd poorer HP environments 13 ), ut emerged here s side-effect of our decision to mtch the sizes of individul rewrds nd the rewrd proilities over the first 2 s. These design choices mximized the comprility of the two conditions for purposes of our neuroimging contrsts. We quntified ehviorl persistence using Kpln-Meier survivl curves 57, which estimted the proility of surviving vrious lengths of time without quitting. This technique ccommodted the fct tht rewrd delivery censored oserved witing times 13. Modeling idel performnce. The rte-mximizing strtegy ws to wit through ll delys in the HP environment (up to 4 s), ut to give up fter 2 s in the LP environment. We determined this y clculting the expected rte of return for vrious giving-up times (Fig. 1c). This clcultion follows previous work 13 nd hs precedent in stochstic forging models 25. The rewrd s rrivl time t rewrd is rndom vrile. For policy of quitting t time T, let p T equl the proility of receiving the rewrd, p T = Pr(t rewrd T), nd let τ T equl the expected dely if the rewrd is received, τ T = E(t rewrd t rewrd T). Ech tril s expected rte of return, in s 1, is 3p RT = T t TpT T( 1 pt) 2 The numertor is tril s expected gin in cents nd the denomintor is tril s expected cost in seconds, ssuming 3 rewrd nd 2-s inter-tril intervl. The gol is to find the vlue of T tht mximizes R T. We use R* to denote the est ville rte of return. Figure 1c plots R T s function of T. The est policy in the HP environment ws to wit 4 s (R* = 1.22 s 1 ), wheres the est policy in the LP environment ws to wit 2 s (R* =.82 s 1 ). Modeling sujective vlue s function of elpsed time. At ech point in tril, the token s sujective vlue depended on three fctors: (1) the expected ernings from tht token, (2) the expected dditionl time to e spent on tht token, nd (3) the monetry vlue of time, which corresponds to R* from ove. We denote the expected ernings s T (t) nd the expected time s T (t). Ech of these depends jointly on the current elpsed time t nd the intended future quitting time T. For given vlues of t nd T, the expected return is gt ( t) = T ( t) ( R *) T ( t) The current tril s sujective vlue (denoted potentil in the model s originl formultion 25 ) equls the mximum vlue of g T cross ll possile quitting times g( t) = mx gt( t) T (1) (2) (3) Put differently, g(t) is the expected net return in the reminder of the current tril, ccounting for the cost of time, under the est ville witing policy. Its minimum is zero ecuse there is lwys n option to quit immeditely (we tret the ITI s prt of the susequent tril). The decision mker should continue witing if g(t) >. Figure 3 shows g(t) s function of t in ech environment (see Supplementry Fig. 2 for decomposition of g(t) into its components). The function pproches 3 t the lst possile rewrd time, when 3 rewrd is expected with no further dely. The est strtegy is to wit up to 4 s in the HP environment ut quit t 2 s in the LP environment. If decision mker in the LP environment were to hve wited 53.5 s lredy it would e etter t tht point to continue witing nture NEUROSCIENCE doi:1.138/nn.3994