A reservoir of time constants for memory traces in cortical neurons

Transcription

1 A reservoir of time constnts for memory trces in corticl neurons Alerto Berncchi, Hyojung Seo, Deyeol Lee & Xio-Jing Wng 11 Nture Americ, Inc. All rights reserved. According to reinforcement lerning theory of decision mking, rewrd expecttion is computed y integrting pst rewrds with fixed timescle. In contrst, we found tht wide rnge of time constnts is ville cross corticl neurons recorded from monkeys performing competitive gme tsk. By recognizing tht rewrd modultes neurl ctivity multiplictively, we found tht one or two time constnts of rewrd memory cn e extrcted for ech neuron in prefrontl, cingulte nd prietl cortex. These timescles rnged from hundreds of milliseconds to tens of seconds, ccording to power lw distriution, which is consistent cross res nd reproduced y reservoir neurl network model. These neuronl memory timescles were wekly, ut significntly, correlted with those of monkey s decisions. Our findings suggest flexile memory system in which neurl supopultions with distinct sets of long or short memory timescles my e selectively deployed ccording to the tsk demnds. In economic ehvior, choices tht hve higher rewrd expecttion re fvored nd dptive decision mking depends on our ility to lern rewrd expecttion through pst rewrds ssocited with our ctions. The neurl mechnisms underlying this process hve een the suject of growing interest, s they could provide importnt insights on how lerning occurs in the rin nd how humns nd other nimls mke economic decisions. Neurl correltes of rewrd vlution hve een oserved in different studies 1 3 nd interpreted in the frmework of reinforcement lerning theory 4,. In the reinforcement lerning model, rewrd expecttion is computed y weighting the previous rewrds through temporl filter, which quntifies the memory trce of rewrds. The optiml durtion of the filter (memory) depends on the predictility of the environment. If the pyoffs for the sme option chnge often nd unpredictly, then rewrds should e filtered on short timescles to trck the fst chnges in voltile environment; in contrst, if pst rewrds relily predict future ones, then they should e filtered on long timescles to exploit stle environment 6,7. The neurl mechnism underlying switching etween long nd short time constnts for computing rewrd expecttion remins poorly understood. On which timescle does the rin filter rewrds? To dte, few studies hve estimted the time constnt of this filter from ehvior nd ssessed how pst rewrds ffect choice selection 8 1, ut the neurl mechnisms responsile for such timescles re still unknown. To ddress this issue, we nlyzed the ctivity of corticl neurons in monkeys performing competitive gme tsk. Using method sed on the ide tht rewrd memory modultes neurl ctivity multiplictively, we found tht memory time constnts cn e extrcted from the ctivity of single neurons. We found tht different timescle for rewrd memory cn e ssocited with ech recorded neuron nd tht there is wide rnge of timescles cross neurons, oeying power lw distriution. The sme distriution is found cross three different corticl res: nterior cingulte cortex (ACCd), dorsolterl prefrontl cortex (DLPFC) nd lterl intrprietl cortex (LIP). Hence, ech re is endowed with reservoir of time constnts for rewrd memory, which re distriuted heterogeneously cross neurons. We found tht the time constnts estimted from pirs of simultneously recorded neurons re uncorrelted, implying tht our results cnnot e explined y single time constnt for ll neurons tht chnges slowly over time. On the other hnd, our nlysis of n niml s ehvior suggests tht the timescle over which rewrd events ffect decisions chnges cross experimentl sessions, possily reflecting the niml s ttempt to increse its pyoff y exploring different strtegies. The time constnts for rewrd memory t the ehviorl nd neuronl levels were wekly correlted cross experimentl sessions. Finlly, we found tht rndomly connected circuit model, kin to reservoir network 13 1, cn reproduce the oserved distriution of timescles, provided tht the network opertes t the criticl point (or edge of chos) Tken together, these findings suggest distriuted, flexile neurl system for rewrd vlution nd memory. RESULTS Multiplictive memory trces in corticl neurons We nlyzed single-neuron ctivity recorded from three corticl res, ACCd 19 (14 neurons), DLPFC (3 neurons) nd LIP 1 ( neurons) of six monkeys performing mtching pennies tsk 11, (Fig. 1). In ech tril, the monkey chose one of two trgets y shifting its gze nd the computer mde its choice y simulting rtionl opponent; the niml received rewrd if its choice mtched tht of the computer. We computed firing rtes of ech neuron y counting the spikes in 1 time intervls of ms (Fig. 1), which re referred to s epochs. This includes six epochs (1. s) efore sccde initition (pre-fixtion, fore-period nd dely) nd six epochs (1. s) fter sccde completion (choice fixtion, feedck nd post-feedck). Consistent with previous studies 3, we found tht the ctivity of neurons vried sustntilly in different tril epochs (99% of neurons, Deprtment of Neuroiology nd Kvli Institute of Neuroscience, Yle University School of Medicine, New Hven, Connecticut, USA. Correspondence should e ddressed to X.-J.W. (xjwng@yle.edu). Received 18 Octoer 1; ccepted 11 Jnury 11; pulished online 13 Ferury 11; doi:1.138/nn VOLUME 14 NUMBER 3 MARCH 11 nture NEUROSCIENCE

2 11 Nture Americ, Inc. All rights reserved. Figure 1 Behviorl tsk nd schemtic illustrtion of memory trces. () In the mtching pennies tsk, the monkey ws required to fixte centrl spot during the fore-period ( ms) nd dely period ( ms) while the two choice trgets (green disks) were displyed. The centrl spot then disppered nd the monkey mde sccdic eye movement to one of the two choice trgets nd mintined its gze on the chosen trget for ms (choice fixtion). A red ring ppering round the correct trget reveled the computer s choice, nd if it mtched the niml s choice (s illustrted), rewrd ws delivered ms lter. ITI, inter-tril intervl. Colored rs t the ottom show the 1 -ms intervls (epochs) used to compute the firing rtes in the nlysis. (,c) Two hypotheticl neurons. The neuron in hs constnt verge firing rte (lck line), wheres the firing rte of neuron in c depends on the tril epoch, repeting in ech of the three consecutive trils. Red lines show the chnge in ctivity s result of the outcome in the first tril (continuous line indictes rewrd, dshed line indictes no rewrd). The inset shows the memory trce of the rewrd, given y the difference etween the red nd lck lines. The memory trce of the neuron in shows simple decy, wheres tht of the neuron in c is multiplictively modulted y the epoch-dependent ctivity. 67 of 681, ANOVA, P <.). The time course of the ctivity in successive epochs differs sustntilly in different neurons. We then exmined the effect of rewrd on the ctivity of neurons. Neurl ctivity in ll three corticl res crries the informtion of pst rewrd events 19. We chrcterized the memory trce of ech neuron using new pproch (Fig. 1,c). Consider the time course of neurl ctivity in different epochs, verged cross trils nd hence cross rewrd/no rewrd conditions. The difference in ctivity from the verge time course, triggered y rewrd/no rewrd event, ws defined s the memory trce of the rewrd, which is positive for one of the outcomes nd negtive for the other. Thus, if the verge ctivity of neuron is zero in given epoch, then the chnge y either outcome must e zero, nd the memory trce in tht epoch is therefore lso zero. Strting with this intuition, we hypothesized tht the memory trce in given epoch is proportionl to the verge firing rte in tht epoch. In tht cse the memory trce is modulted (multiplied) y the verge firing rte. We define the epoch code s the firing rte verged cross ll trils, s function of the different epochs, denoted y g(k) (k = 1,,1 epochs, in temporl order). In one neuron recorded in ACCd, firing rte decresed fter the sccde to chosen trget nd incresed fter the feedck period (Fig. ). To seprte the contriutions of epoch nd rewrd memory to neurl ctivity, we modeled the firing rte mesured in tril n nd epoch k, denoted y FR(n,k), s the sum of the epoch code g(k) nd filter f(n,k) convolved with the niml s rewrd history in previous trils (lst five trils; in ech tril, Rew = +1 indictes rewrd; Rew = 1 indictes no rewrd). c -ms window Firing rte Firing rte Fore-period. s Simple decy of the memory trce Rewrd No rewrd Tril 1 Modulted decy of the memory trce Rewrd No rewrd Dely. s ITI (1. s) Fix Sccde. s Tril Tril 3 Tril 1 Tril Tril 3 n = : FR( n, k) = g( k) + f ( n, k) Rew( n n ) Time Time (1) The filter f descries how the rewrd in given tril ffects neurl ctivity in the susequent trils, ssuming tht the effects of rewrds in successive trils re dditive. For exmple, f(3,4) descries the effect of rewrd fter 3 trils during epoch 4. The filter f corresponds to our definition of memory trce (Fig. 1,c); it reflects the devition from the epoch-dependent time course g(k) resulting from rewrd event. Becuse Rew(n) is nerly rndom sequence 11 of +1 nd 1, verging the firing rtes over ll trils recovers the epoch code g(k) (sum over n of FR(n,k)). We estimted the memory trce f(n,k) y pplying multiple liner regression to the dt ccording to eqution (1). In n exmple negtive neuron (tht is, rewrd decreses the ctivity of this neuron in susequent trils), the memory trce does not decy monotoniclly, ut its strength is modulted throughout the tril consistent with the epoch code (Fig. ). According to the multiplictive Memory trce Memory trce Feedck. s Figure An exmple neuron in ACCd showing multiplictive modultion of memory trces y the epoch code. The colors in ll pnels denote tril epochs, following the formt of Figure 1. () The epoch code for n exmple neuron; tht is, the firing rte computed in 1 -ms epochs in tril nd verged over ll trils (lck squres, interpolted y the lck line, roken during the sccde). Colored disks correspond to the slopes fitted in c (error rs represent ±s.e.); their correltion with the epoch codes quntifies the multiplictive modultion Disk = memory trce f (color = epoch) = Exponentil fit 3.. A =.4 = Fore Dely Fix Feedck nd is referred to s the fctoriztion index (.97 in this exmple). () The memory trce f of pst rewrds in the sme neuron, up to five trils in the pst. Colored dots nd error rs (±s.e.) show the results of the multiple liner regression model (eqution (1)) nd the lck line is the exponentil fit (eqution (), continuous line, exponentil ex(t); roken line, modulted envelope g ex(t)). The prmeters for the fit re shown (A, mplitude; τ, timescle). (c) The memory trce f (from ), plotted s function of the exponentil function ex. The lines re lest-squres fit, ech line encompssing prticulr epoch nd ll five tril lgs. According to the fctoriztion, the slopes should correspond to the epoch code, f = g ex. The vlues of the slopes re plotted in (colored squres) nd compred with the epoch code g(k). Memory trce f (Hz) Exponentil decy c Memory trce f (Hz) Exponentil decy nture NEUROSCIENCE VOLUME 14 NUMBER 3 MARCH

3 11 Nture Americ, Inc. All rights reserved. ACCd DLPFC 4 c d 3 1 Fore Dely Fix Feedck e f g h 1 1 Fore Dely Fix Feedck 1 i j k l model (Fig. 1,c), we ssumed tht the memory trce f is fctorized into the epoch code g(k) nd n exponentil function ex(t). FR( n, k) = g( k) + g( k) ex( t) Rew( n n ) n = : The filter f considered in eqution (1) is now replced y the product of two fctors g(k) ex(t), where ex( t) = Ae t is n exponentil decy function nd t is the time elpsed since the outcome (Online Methods). By pplying this model to the exmple neuron (Fig.,), we otined timescle of memory decy τ = 6.9 trils nd n mplitude A =.4. According to the fctoriztion ( f = g ex), the constnt of proportionlity etween the memory trce f nd the exponentil function ex, estimted in different epochs (Fig. c), should reproduce the epoch code g(k). The epoch codes for the neuron closely followed these predictions, indicting tht the fctoriztion is nerly exct (Fig. ). The fctoriztion index of neuron, defined s the correltion coefficient etween the epoch code nd the proportionlity constnts (slopes), ws.97 for this neuron. The modulted decy of the memory trce ws oserved in the mjority of the recorded neurons in ll three corticl res. In some cses, the sum of two exponentil functions, ex( t) = A e Ae fitted the dt etter thn single exponentil, in which cse the memory trce often exhiited iphsic chrcteristic (with A 1 nd A of the opposite sign; Fig. 3). Using the Byesin Informtion Criterion, we found tht the est fit ws single exponentil for 69 neurons nd doule exponentils for 68 neurons, wheres the LIP Fore Dely Fix Feedck Fore Dely Fix Feedck Figure 3 Firing rtes nd memory trces for six neurons, two for ech of the three recorded res. For ech of the six neurons, epoch codes (first nd third column) nd memory trces (second nd fourth column) re shown, presented s in Figure,. The second column shows monotonic decy of the memory trce nd the fourth column shows iphsic memory trces (doule exponentil). Different neurons hd different firing rtes, oth in mgnitude nd time course, nd different types of memory decy, ut they were ll consistent with n exponentil (single or doule) decy of the memory modulted y the epoch code. The fctoriztion indexes for those neurons re.98 (,),.91 (c,d),.98 (e,f),.84 (g,h),.97 (i,j) nd.61 (k,l). () t t, Fore Dely Fix Feedck Fore Dely Fix Feedck remining 144 neurons were fitted est y model with ex(t) =. The ltter is interpreted s no memory nd the corresponding neurons were excluded from further nlysis. We tested the vlidity of the fitting procedure y rndomly reshuffling the order of trils in ech session nd we consistently found tht 96% of neurons (66 of 681) showed no memory fter reshuffling. We exmined the verge firing rtes nd memory trces of ll recorded neurons (exmples shown in Fig. 3). Although the ctivity of most neurons is consistent with n exponentil decy of the memory trce (79%, 37 of 681, single nd doule exponentils), frction of them did not show modultion of the memory y the epoch code. This is quntified y the fctoriztion index, which is significntly positive for pproximtely hlf of the neurons showing memory effect (46%, 49 of 37, P <., t test). We found smll, ut significnt, difference in the frction of neurons with memory cross different res (87% in ACCd, 7% in DLPFC nd 78% in LIP, χ test, P =.1). We next investigted how the timescles of memory trces were distriuted cross neurons in different corticl res. We determined tht the distriution of timescles in ll corticl res could e fit with power lw with n exponent of (Fig. 4). The power lw implies tht timescles re distriuted in wide rnge of vlues. In fct, for power lw distriution ~τ, the vrince increses with the smple size nd, in principle, ritrrily lrge timescles would e oserved with proportionlly lrge increment in the numer of recorded neurons. Note tht the power lw til pplies for timescles equl to or lrger thn one tril, which re those timescles tht might e involved in memory (see elow). Aout % of ll recorded neurons (133 of 681) VOLUME 14 NUMBER 3 MARCH 11 nture neuroscience

4 11 Nture Americ, Inc. All rights reserved ACCd DLPFC LIP All res hd timescle lrger thn one tril (9% in ACCd, 19% in DLPFC nd 13% in LIP; χ test, P =.; see Supplementry Fig. 1c e). Becuse the timescles from one- (τ) nd two-exponentil functions (τ 1, τ ) were distriuted similrly (Supplementry Fig. 1,), we pooled ll timescles ( totl of 8 timescles from 69 single exponentil nd 68 doule exponentil; tht is, 69 τ, 68 τ 1 nd 68 τ ). ACCd contriuted 197 timescles from 71 single exponentil nd 63 doule exponentil functions (71 τ, 63 τ 1 nd 63 τ ), wheres neurons hd no memory. A totl of 36 timescles were otined from DLPFC with 14 single nd 119 doule exponentil functions (14 τ, 119 τ 1 nd 119 τ ) nd 79 DLPFC neurons hd no memory. LIP neurons contriuted 46 timescles from 74 single nd 86 doule exponentil functions (74 τ, 86 τ 1 nd 86 τ ) nd 4 LIP neurons showed no memory. Comprison with ehvior Are the neurl memory timescles relevnt for lerning nd decision mking? The mtching pennies tsk tht we used does not necessrily require the memory of pst rewrds nd the optiml strtegy for the monkey is to choose rndomly nd unpredictly. Although the overll performnce of monkeys ws nerly optiml, their tril-ytril decisions, loclly in time, were influenced y previous rewrds nd ctions 11,19. We nlyzed the ehvior of monkeys in different experimentl sessions y fitting their decisions with stndrd reinforcement lerning model (reinforcement lerning, Online Methods). The lerning rte prmeter (α) of the reinforcement lerning model quntifies the ehviorl timescle of the memory trce (α ~1/τ). The resulting likelihood ws significntly lrger thn the likelihood for reshuffled trils nd the model fit with ehviorl dt ws significnt in 78% of the sessions (196 of, P <.). We found tht the timescles of ehviorl memory vried cross sessions, possily τ Figure 4 Distriution of the timescles chrcterizing the rewrd memory trces cross neurons. Blck disks show the density for the neurons in ll three corticl res in the corresponding in; tht is, the count of timescles divided y the in length (error rs represent ±s.e.). The inset shows the count of the timescles in the sme ins, in liner scle ( totl of 8 timescles). Grey mrkers show the density seprtely for ech of the three different corticl res (squre, ACCd, 197 timescles; upwrd tringle, DLPFC, 36; downwrd tringle, LIP, 46). The red line (red curve in the inset) shows power lw fit (exponent = ). suggesting tht monkeys dopted different strtegies in successive sessions. For the 196 sessions fitted y the reinforcement lerning model, the distriution of ehviorl timescles followed power lw distriution (Fig. ) nd the exponent ws consistent with tht mesured in the neurl distriution. Hence, the distriutions of ehviorl nd neuronl timescles qulittively mtched with ech other. This result suggests tht there might e reltionship etween the memory trce oserved t the neurl level nd tht oserved t the ehviorl level. We tested this hypothesis y compring the neurl timescle for rewrd memory oserved during given recording session with the ehviorl timescle fit in tht session (when oth re ville) nd we found smll, ut significnt, correltion cross sessions (R =.1, P =.3; Fig. ), suggesting tht the ctivity of single neurons is relted, leit wekly, to the ehviorl strtegy of the nimls. Do the rewrd memory timescles lso chnge in single session? We determined whether the timescles re stle in single recording session y dividing ech session into two seprte locks (hlves) of trils nd we re-estimted oth the neurl nd ehviorl timescles seprtely in the two locks. Both the ehviorl nd neurl memory timescles were firly stle in single session (Fig. 6). The neurl nd ehviorl timescles might fluctute together cross sessions, ut their smll correltion indictes tht there is only wek coupling. Indeed, we found tht t ny moment, the timescles of rewrd memory vried cross corticl neurons. In ech recording session, only few neurons were simultneously recorded (out two on verge). When we estimted memory timescles for pirs of simultneously recorded neurons, the correltion etween their time constnts ws not significntly different from zero (31 pirs of timescles, R =.7, P =.). This result suggests tht the rod distriution of memory time constnts oserved in the dt reflects vriility of timescles cross different neurons, rther thn resulting from memory timescle fixed for ll neurons tht collectively chnges cross sessions. Tken together, our results support the conclusion tht diverse collection of neurl memory timescles, reservoir, is ville cross corticl neurons t ny given time. The niml s ehvior my e determined y redout system tht is le to smple, t different times, Figure Distriution of ehviorl timescles nd their reltionship with the neurl memory timescles. () Time constnt τ estimted from the lerning rte α (τ ~1/α) of reinforcement lerning model fit to the monkey s ehviorl dt. Blck disks show the density in the corresponding in; tht is, the count of timescles divided y the in length (error rs represent ±s.e.). The inset shows the count of the timescles in the sme ins, in liner scle ( totl of 196 timescles). The red line (red curve in the inset) shows power lw fit (exponent = 1.9). () The sctterplot of ehviorl versus neurl memory timescles otined from ll sessions where oth were ville. Neurl timescles from different types of fit (τ from single exponentil nd τ 1, τ from doule exponentil) re shown in different colors. Behviorl nd neurl timescles show smll, ut significnt, correltion (R =.1, P =.3). τ 1.9 τ (ehvior) τ τ 1 τ R = τ (neurons) nture NEUROSCIENCE VOLUME 14 NUMBER 3 MARCH

5 τ (second lock of trils) Behvior R = τ (first lock of trils) τ (second lock of trils) τ τ 1 τ Neurons R = τ (first lock of trils) Figure 6 Stility of ehviorl nd neurl memory timescles in n experimentl session. (,) In oth pnels, the sctterplot of the timescles fitted in the second hlf of the trils is plotted ginst the timescles fitted in the first hlf of the trils in the sme session. The correltion ws significntly different from zero in oth cses (R =.4 for ehviorl timescles, R =.77 for neurl timescles), suggesting tht oth types of timescles re firly stle in single session. Neurl memory timescles from different types of fit (τ from single exponentil nd τ 1, τ from doule exponentil) re shown in different colors. 11 Nture Americ, Inc. All rights reserved. from vriety of timescles present in the reservoir. The reservoir might not e sttic nd it my chnge its distriution of timescles from dy to dy. During competitive gmes, the sujects might lso tke into ccount their recent choices to determine their future ehvior. We therefore tested whether ny memory trce of choice exists in the recorded neurons y pplying the sme nlysis of equtions (1) nd () nd sustituting rewrd with choice. We found tht multiplictive modultion nd power lw distriution of memory timescles lso hold for memory trce of pst choices (Supplementry Fig. ). Neurl network model for memory trces Wht neurl mechnism(s) ccounts for the sttisticl properties of rewrd memory descried ove? To ddress this question, we constructed simple neurl network model tht reproduces the oserved neurl memory trces (Fig. 7 nd Supplementry Fig. 3). Model neurons integrte the rewrd signls y receiving current impulse whenever rewrd is otined. Becuse neurons re recurrently connected nd form loops, their ctivities revererte nd re le to mintin the memory of rewrd events. However, those memories decy nd re slowly forgotten ccording to time course tht depends on the pttern of synptic connections mong neuron pirs. Specificlly, the ctivity of neurons evolve ccording to d v = J v( t) + h Rew( t), where v is vector of M components, ech dt component is the ctivity of different neuron in the reservoir (M = 1, neurons in simultions), J is the synptic connectivity mtrix of their interctions nd h is vector representing the reltive strength of the rewrd input Rew(t) to ech neuron. For our purposes, the specific form of the input signls is not importnt; the results depend only on the synptic mtrix J. We ssumed tht the connection weights (the entries of the mtrix J) were rndomly distriuted nd we looked for cndidte proility distriutions such tht the network model reproduces the distriutions of timescles nd mplitudes oserved in the neurl dt from ehving monkeys (see Supplementry Text). Amplitudes determine the extent of the immedite response of neurons to rewrd, with respect to the verge ctivity. Time constnts hd power-lw distriution (Fig. 4) nd the distriution of Figure 7 Neurl responses (memory trces) in the model nd distriution of timescles of the memory trces in model neurons. () The memory trces of four model neurons. () The lck disks show the density of timescles in the corresponding in; tht is, the count of timescles divided y the in length (error rs represent ±s.e.). The inset shows the count of the timescles in the sme ins, in liner scles ( totl of 1, timescles). The red line (red curve in the inset) shows power lw fit (exponent = ). Neurl response (model) mplitudes ws exponentil (Fig. 8, where we used A for one exponentil nd A 1 + A for two exponentils). First, we found tht the connection weights must e rodly distriuted mong neuron pirs nd tht this endows the network with wide vriety of timescles. Intuitively, the stronger the connection, the longer the reverertion of the input nd hence the timescle of the memory trce. However, if connections re lso heterogeneous, then weker connections nd smller timescles will lso contriute to the memory trces. If the width of the distriution of connection weights reches certin threshold, power-lw distriution of timescles is oserved (Fig. 7), which is chrcterized y high proility for oth smll nd lrge timescles. This is distinct type of network stte t criticl point (or edge of chos in nonliner systems), which hve een proposed to e desirle for mny kinds of computtions In our model, the criticlity corresponds to the sitution where the system is on the verge of losing stility. When the width of the connection distriution exceeds the criticl level, the liner system is unstle nd the model would need to e extended to include nonlinerities such s sturtion of neurl ctivity. For the ske of simplicity, we limited ourselves to the liner model, which is sufficient for the purpose of reproducing the oserved power-lw distriution of timescles under specific conditions. A second desirle property of the network is tht its dynmics re roust with respect to smll chnges of the connection strengths. If the coding of the memory chnges mrkedly s result of smll chnges in the connection strengths (for exmple, synptic noise), it would e difficult for downstrem system to interpret tht code. A known property of the connection mtrix J tht ensures tht kind of roustness is normlity, which gurntees tht there is n orthogonl set of eigenvectors 6 (ut see refs. 7 9 for non-norml neurl network models). If J is norml, we found tht the mplitudes of the memory trces followed n exponentil distriution (Fig. 8), consistent with the experimentl oservtions (Fig. 8). To the est of our knowledge, our results provide the first complete sttisticl description of Time τ Model VOLUME 14 NUMBER 3 MARCH 11 nture neuroscience

6 11 Nture Americ, Inc. All rights reserved exp( A) ACCd DLPFC LIP All res 4 4 Amplitude Amplitude network connection mtrix sed on in vivo neuronl recordings of ehving nimls (see lso refs. 3 3). DISCUSSION The power lw of timescles suggests tht the durtion of rewrd memory trce is highly diverse cross corticl neurons. The sme diversity is oserved cross three corticl res, suggesting tht the computtion of rewrd memory is distriuted process. This finding is consistent with n incresing pprecition tht neurl encoding of cognitive vriles is highly heterogeneous nd distriuted 33,34. Prefrontl cortex is importnt for dynmic decision processes encoding nd updting vlues 1 4. Although nterior cingulte cortex hs een implicted in monitoring conflict etween incomptile response processes 3 or detecting performnce errors 36, recent studies hve plced more emphsis on its role in representing oth positive nd negtive vlues 19,37. Prietl cortex hs lso een implicted in decision mking on the sis of the vlue representtion nd the ccumultion of sensory evidence 38,39. Our work provides comprehensive description of memory trces in terms of specific distriution of timescles cross popultion of neurons nd introduces frmework tht could potentilly e pplicle to different rin res nd different types of memory. The concept of multiplictive modultion of memory trces cn e used to deduce the neurl memory timescles in vrious tsks nd to test the ide tht different set of time constnts is selected to dpt to specific environment 6,7. Although the glol optiml strtegy for the mtching pennies tsk is to choose rndomly nd therefore does not require memory, the nimls mde their decisions lrgely on the sis of their rewrd history 11,19. Perhps in the persistent serch for n pproprite strtegy, they smpled different timescles cross experimentl sessions. We found tht those ehviorl timescles followed similr distriution nd were wekly, ut significntly, correlted with the timescles oserved t the neurl level. This suggests the possiility tht the ehvior might e driven y mechnism tht ppropritely smples from rnge of timescles in neurl network, which hs yet to e elucidted. Alterntively, this wek correltion might e cused y fctors tht re currently not understood. Note tht the oserved rnge is different for the neuronl versus ehviorl time constnts. Also, we hve not ttempted to fit the ehviorl dt y reinforcement lerning model endowed with multiple time constnts. Future work is needed to further ssess the correltion etween neurl memory trces nd ehvior. Regrdless, our results suggest Amplitude Amplitude Figure 8 Distriution of mplitudes of the memory trces in the neurl dt nd model. (,) In oth pnels, lck disks show the density in the corresponding in; tht is, the count of timescles divided y the in length (error rs represent ±s.e.). The inset shows the count of the mplitudes in the sme ins, in liner scle (37 mplitudes in the dt, 1, in the model). Amplitudes re plotted s solute vlues, s the distriution ws pproximtely symmetric (symmetry is shown in the inset). Grey mrkers show the density seprtely for the three different recorded res (squres, ACCd, 134 mplitudes; upwrd tringles, DLPFC, 43; downwrd tringles, LIP, 16). The red line (red curve in the inset) shows n exponentil fit (e A ) exp( A) Model tht rewrd memory with multiple time constnts might e used to compute the vlue functions in reinforcement lerning theory in more thn one timescle. Similrly, the doule exponentil decy of memory my correspond to rewrd prediction error signl; if the short timescle (τ 1 ) is smll enough (out one tril or smller), then the corresponding exponentil filter will respond primrily to the rewrd in the present tril, wheres the long timescle (τ ) my provide vlue signl y weighting the rewrds in the pst few trils. When the two exponentils hve opposite signs, they roughly sutrct the vlue from the ctul rewrd signl, therefore providing rewrd prediction error. It hs een noted tht iphsic filtering in dopmine neurons might provide rewrd prediction error 4. Besides the memory for rewrd, the ctivity of primte corticl neurons reflects other types of short-term memory. The time course of memory-relted ctivity vries cross different neurons nd different tsk protocols, including persistent, rmping nd multi-phsic ctivity Memory trces in the neurl signls re mixed with other tsk-dependent fctors 44,4 nd it hs een deted s to whether other processes involved in gol-directed ehvior could e inter-mixed with memory trce, such s sptil ttention 46, motor plnning 47, nticiption of future events 48 or timing 49. The epoch code in the present tsk might include mny of those processes nd we found tht memory signls could e dissocited from those fctors y ssuming multiplictive computtion. The hypothesis of multiplictive effect of memory on neurl ctivity could e tested y looking more closely t the multi-phsic time course of memory-relted ctivity oserved in other experiments. The computtionl dvntge of the multiplictive effect of memory needs to e further investigted. For exmple, it my serve the pproprite recll of memories t different epochs (see Supplementry Text), s oserved in recent study. Reservoir-type networks hve een the suject of ctive reserch in computtionl neuroscience nd mchine lerning 13 1, ut experimentl support tht such networks re dopted y the rin hs een lcking. Those models predict tht the memory of input signls is stored in lrge, recurrent nd heterogeneous network (reservoir) in distriuted mnner nd tht desired output is otined y trinle comintion of the response signls in the reservoir. The heterogeneous encoding of the input llows the flexile lerning of different output functions. In our context, tht my correspond to flexile chnge in strtegy resulting from the vriety of timescles for rewrd memory present in the reservoir. We present direct experimentl evidence, t the level of single neurons, for high-dimensionl reservoir network of rewrd memory trces in prefrontl, cingulte nd prietl res of the primte cortex. This empiricl finding is reproduced y simple computtionl model, which suggests tht rewrd filtering in the cortex involves dynmic reservoir network operting t the criticl point, leding to power-lw distriution of time constnts. The output of the network, supposedly driving the niml s ehvior, is not explicitly modeled in our equtions. Further studies re necessry to elucidte how the motor res red out the memory of rewrd nd choices nd how the two re comined to suserve dptive choice ehvior. Power-lw distriutions re unusul, s they imply high proility for oth lrge nd smll time constnts. A diversity of time constnts lso mens rod rnge of lerning rtes, s the two re inversely nture NEUROSCIENCE VOLUME 14 NUMBER 3 MARCH

7 11 Nture Americ, Inc. All rights reserved. relted to ech other. This is noteworthy, s shift from n exploitive to n explortory strtegy s the environment ecomes uncertin is often ssessed y n increse in the lerning rte 1. Our work suggests tht rod rnge of lerning rtes re ville in the system, suset of which (fst or slow) might e selectively utilized ccording to which strtegy is ehviorlly desirle. Ultimtely, this frmework could led to new model for predicting how rewrd expecttion is computed nd how rewrd memory ffects decision mking. Methods Methods nd ny ssocited references re ville in the online version of the pper t Note: Supplementry informtion is ville on the Nture Neuroscience wesite. Acknowledgments We thnk J. Mzer nd M.W. Jung for comments on n erlier version of the mnuscript, nd R. Chudhuri, M. Hrre nd J. Murry for discussions. This work ws supported y the US Ntionl Institutes of Helth grnt R1 MH6349 nd the Swrtz Foundtion (A.B. nd X.-J.W.), nd y US Ntionl Institutes of Helth grnts R1 MH7346 (X.-J.W. nd D.L.) nd DA933 (D.L.). AUTHOR CONTRIBUTIONS All of the uthors prticipted in the reserch design nd the preprtion of the mnuscript. H.S. collected the dt, A.B. nd H.S. nlyzed dt, nd A.B. nd X.-J.W. performed modeling. COMPETING FINANCIAL INTERESTS The uthors declre no competing finncil interests. Pulished online t Reprints nd permissions informtion is ville online t reprintsndpermissions/. 1. Kle, J.W. & Glimcher, P.W. The neuroiology of decision: consensus nd controversy. Neuron 63, (9).. Rushworth, M.F. & Behrens, T.E. Choice, uncertinty nd vlue in prefrontl nd cingulte cortex. Nt. Neurosci. 11, (8). 3. Wng, X.-J. Decision mking in recurrent neurl circuits. Neuron 6, 1 34 (8). 4. Soltni, A., Lee, D. & Wng, X.-J. Neurl mechnism for stochstic ehvior during competitive gme. Neurl Netw. 19, (6).. Sutton, R.S. & Brto,, A.G. Reinforcement Lerning, n Introduction (MIT Press, Cmridge, Msschusetts, 1998). 6. Behrens, T.E., Woolrich, M.W., Wlton, M.E. & Rushworth, M.F. Lerning the vlue of informtion in n uncertin world. Nt. Neurosci. 1, (7). 7. Dw, N.D., O Doherty, J.P., Dyn, P., Seymour, B. & Doln, R.J. Corticl sustrtes for explortory decisions in humns. Nture 441, (6). 8. Lu, B. & Glimcher, P.W. Dynmic response-y-response models of mtching ehvior in rhesus monkeys. J. Exp. Anl. Behv. 84, 79 (). 9. Corrdo, G.S., Sugrue, L.P., Seung, H.S. & Newsome, W.T. Liner-nonliner-Poisson models of primte choice dynmics. J. Exp. Anl. Behv. 84, (). 1. Kennerley, S.W., Wlton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optiml decision mking nd the nterior cingulte cortex. Nt. Neurosci. 9, (6). 11. Lee, D., Conroy, M.L., McGreevy, B.P. & Brrclough, D.J. Reinforcement lerning nd decision mking in monkeys during competitive gme. Brin Res. Cogn. Brin Res., 4 8 (4). 1. Kim, S., Hwng, J., Seo, H. & Lee, D. Vlution of uncertin nd delyed rewrds in primte prefrontl cortex. Neurl Netw., (9). 13. Mss, W., Ntschläger, T. & Mrkrm, H. Rel-time computing without stle sttes: new frmework for neurl computtion sed on perturtions. Neurl Comput. 14, 31 6 (). 14. Jeger, H., Lukosevicius, M., Popovici, D. & Siewert, U. Optimiztion nd pplictions of echo stte networks with leky-integrtor neurons. Neurl Netw., 33 3 (7). 1. Verstreten, D., Schruwen, B., D Hene, M. & Stroondt, D. An experimentl unifiction of reservoir computing methods. Neurl Netw., (7). 16. Sussillo, D. & Aott, L.F. Generting coherent ptterns of ctivity from chotic neurl networks. Neuron 63, 44 7 (9). 17. Bertschinger, N. & Ntschlger, T. Rel-time computtion t the edge of chos in recurrent neurl networks. Neurl Comput. 16, (4). 18. Lngton, C.G. Computtion t the edge of chos: phse trnsitions nd emergent computtions. Physic D 4, 1 37 (199). 19. Seo, H. & Lee, D. Temporl filtering of rewrd signls in the dorsl nterior cingulte cortex during mixed-strtegy gme. J. Neurosci. 7, (7).. Seo, H., Brrclough, D.J. & Lee, D. Dynmic signls relted to choices nd outcomes in the dorsolterl prefrontl cortex. Cere. Cortex 17, i11 i117 (7). 1. Seo, H., Brrclough, D.J. & Lee, D. Lterl intrprietl cortex nd reinforcement lerning during mixed-strtegy gme. J. Neurosci. 9, (9).. Brrclough, D.J., Conroy, M.L. & Lee, D. Prefrontl cortex nd decision mking in mixed-strtegy gme. Nt. Neurosci. 7, (4). 3. Lpish, C.C., Durstewitz, D., Chndler, L.J. & Semns, J.K. Successful choice ehvior is ssocited with distinct nd coherent network sttes in nterior cingulte cortex. Proc. Ntl. Acd. Sci. USA 1, (8). 4. Sigl, N., Kusonoki, M., Nimmo-Smith, I., Gffn, D. & Duncn, J. Hierrchicl coding for sequentil tsk events in the monkey prefrontl cortex. Proc. Ntl. Acd. Sci. USA 1, (8).. Jin, D.Z., Fujii, N. & Gryiel, A.N. Neurl representtion of time in cortico-sl gngli circuits. Proc. Ntl. Acd. Sci. USA 16, (9). 6. Trefethen, L.N. & Emree, M. Spectr nd Pseudospectr: The Behvior of Nonnorml Mtrices nd Opertors (Princeton University Press, Princeton, New Jersey, ). 7. Murphy, B.K. & Miller, K.D. Blnced mplifiction: new mechnism of selective mplifiction of neurl ctivity ptterns. Neuron 61, (9). 8. Gnguli, S., Huh, D. & Sompolinsky, H. Memory trces in dynmicl systems. Proc. Ntl. Acd. Sci. USA 1, (8). 9. Goldmn, M.S. Memory without feedck in neurl network. Neuron 61, (9). 3. Schneidmn, E., Berry, M.J., Segev, R. & Bilek, W. Wek pirwise correltions imply strongly correlted network sttes in neurl popultion. Nture 44, (6). 31. Brunel, N., Hkim, V., Isope, P., Ndl, J.-P. & Brour, B. Optiml informtion storge nd the distriution of synptic weights: perceptron versus purkinje cell. Neuron 43, (4). 3. Gnguli, S. et l. One-dimensionl dynmics of ttention nd decision mking in LIP. Neuron 8, 1 (8). 33. Duncn, J. An dptive coding model of neurl function in prefrontl cortex. Nt. Rev. Neurosci., 8 89 (1). 34. Rigotti, M., Ruin, D.B.D., Wng, X.-J. & Fusi, S. Internl representtion of tsk rules y recurrent dynmics: the importnce of the diversity of neurl responses. Front. Comput. Neurosci. 4, 4 (1). 3. Botvinick, M.M., Brver, T.S., Brch, D.M., Crter, C.S. & Cohen, J.D. Conflict monitoring nd cognitive control. Psychol. Rev. 18, 64 6 (1). 36. Holroyd, C.B. & Coles, M.G.H. The neurl sis of humn error processing: reinforcement lerning, dopmine nd error-relted negtivity. Psychol. Rev. 19, (). 37. Wllis, J.D. & Kennerley, S.W. Heterogeneous rewrd signls in prefrontl cortex. Curr. Opin. Neuroiol., (1). 38. Pltt, M.L. & Glimcher, P.W. Neurl correltes of decision vriles in prietl cortex. Nture 4, (1999). 39. Roitmn, J.D. & Shdlen, M.N. Response of neurons in the lterl intrprietl re during comined visul discrimintion rection time tsk. J. Neurosci., (). 4. Byer, H.M. & Glimcher, P.W. Midrin dopmine neurons encode quntittive rewrd prediction error signl. Neuron 47, (). 41. Riner, G. & Miller, E.K. Time course of oject-relted neurl ctivity in the primte prefrontl cortex during short-term memory tsk. Eur. J. Neurosci. 1, (). 4. Mchens, C.K., Romo, R. & Brody, C.D. Functionl, ut not ntomicl, seprtion of wht nd when in prefrontl cortex. J. Neurosci. 3, 3 36 (1). 43. Shfi, M. et l. Vriility in neuronl ctivity in primte cortex during working memory tsks. Neuroscience 146, (7). 44. Curtis, C.E. & Lee, D. Beyond working memory: the role of persistent ctivity in decision mking. Trends Cogn. Sci. 14, 16 (1). 4. Pssinghm, D. & Ski, K. The prefrontl cortex nd working memory: physiology nd rin imging. Curr. Opin. Neuroiol. 14, (4). 46. Leedev, M.A., Messinger, A., Krlik, J.D. & Wise, S.P. Representtion of ttended versus rememered loctions in prefrontl cortex. PLoS Biol., e36 (4). 47. Funhshi, S., Chfee, M.V. & Goldmn-Rkic, P.S. Prefrontl neuronl ctivity in rhesus monkeys performing delyed nti-sccde tsk. Nture 36, (1993). 48. Riner, G., Ro, S.G. & Miller, E.K. Prospective coding for ojects in primte prefrontl cortex. J. Neurosci. 19, 493 (1999). 49. Brody, C.D., Hernndez, A., Zinos, A. & Romo, R. Timing nd neurl encoding of somtosensory prmetric working memory in mcque prefrontl cortex. Cere. Cortex 13, (3).. Bromerg-Mrtin, E.S., Mtsumoto, M., Nkhr, H. & Hikosk, O. Multiple timescles of memory in lterl henul nd dopmine neurons. Neuron 67, (1). 37 VOLUME 14 NUMBER 3 MARCH 11 nture neuroscience

8 11 Nture Americ, Inc. All rights reserved. ONLINE METHODS Animl preprtion nd electrophysiologicl recording. All of the dt were collected using the sme ehviorl tsk nd electrophysiologicl techniques. These techniques hve een descried previously We used six rhesus monkeys (five mle nd one femle). The niml s hed ws fixed during the experiment nd eye movements were monitored t smpling rte of Hz with high-speed eye trcker (Thoms Recording). Animls performed n oculomotor free-choice tsk (mtching pennies; Fig. 1). Trils egn with the niml fixting smll yellow squre (.9.9 ) displyed t the center of the computer screen for.-s fore-period. Two identicl green disks were presented t eccentricity in dimetriclly opposed loctions long the horizontl meridin for.-s dely period. The extinction of the centrl trget signled the niml to shift its gze towrd one of the trgets within 1 s. After the monkey mintined its fixtion on the chosen peripherl trget for. s, red ring ppered round the trget selected y the computer. The niml ws rewrded only if it chose the sme trget s the computer, which simulted rtionl decision mker in the mtching pennies gme trying to minimize the niml s expected pyoff. Before ech tril, the computer mde prediction for the niml s choice y computing the conditionl proilities for the niml to choose ech trget given its choices nd rewrds in the preceding four trils. The computer mde rndom choice if the proilities were consistent with unised ehviors, otherwise it would is its selection ginst the prediction. Single-unit ctivity ws recorded using five-chnnel multi-electrode recording system (Thoms Recording) from three corticl regions: the ACCd 19 (re 4c, two mle monkeys, 8 1 kg), DLPFC, (nterior to the frontl eye field; four mle nd one femle monkeys, 1 kg) nd LIP 1 (two mle nd one femle monkeys, 11 kg). All the neurons were recorded without pre-screening. The plcement of the recording chmer ws guided y mgnetic resonnce imges nd confirmed y metl pins inserted in known ntomicl loctions t the end of the experiment in some nimls. In three nimls, two recording chmers were used for simultneous recording of DLPFC nd LIP. All the experimentl procedures were pproved y the Institutionl Animl Cre nd Use Committee t Yle University nd conformed to the Pulic Helth Services Policy on Humne Cre nd Use of Lortory Animls nd the Guide for the Cre nd Use of Lortory Animls. Multiple regression nlysis of memory trces. To estimte the memory trces f(n,k) from the oserved neuronl firing rtes nd sequence of rewrds, we computed the firing rtes in ech tril in 1 time intervls of ms ech (Fig. 1). The following model ws used to fit the firing rtes. The firing rte of neuron depends on the tril epoch k, following the epoch code g(k); fter the outcome is reveled (feedck period) in ech tril, the firing rte is chnged y n mount of +f(n,k) for rewrd nd f(n,k) for no rewrd, where n is the numer of trils elpsed since tht outcome. The effects of outcomes in successive trils re dditive. The firing rte FR(n,k) is thus descried y FR( n, k) = g( k) + f ( n, k) Rew( n n ) + noise n = : where the index k lels the epoch (k = 1,,1) nd the indices n nd n lel trils. The effect of rewrd extends up to five trils (n =,,), while the index n runs over ll N trils ville in ech neuron recording (strting fter the first five trils, n = 6,, N). To determine f(n,k) nd g(k), we pplied multiple regression model y using the known FR(n,k) nd Rew(n) (+1/ 1 for rewrd/no rewrd). Note tht the epoch code g(k) depends on the twelve different epochs within tril, wheres the rewrd Rew(n) depends only on tril numer. As consequence, the regression cn e pplied seprtely for ech epoch. For fixed epoch k, the seven unknown vriles g(k), f(,k), f(1,k), f(,k), f(3,k), f(4,k) nd f(,k) cn e determined y using the known vlues of FR(n,k) nd Rew(n) in N trils (n = 6,, N). Using prsimonious mtrix nottion nd omitting the epoch lel k, eqution (3) cn e rewritten s FR = Rewi f + noise where the vector of the known firing rtes FR is equl to FR =[ FR( 6, k), FR( 7, k),..., FR( N, k)] T (3) (4) () The seven unknown vriles hve een rewritten y single vector f f =[ g( k), f (, k), f ( 1, k), f (, k), f ( 3, k), f ( 4, k), f (, k)] T The mtrix Rew is known, given y 1 Rew6 Rew Rew4 Rew3 Rew Rew1 1 Rew7 Rew6 Rew Rew4 Rew3 Rew Rew = RewN RewN 1 RewN RewN 3 RewN 4 Rew N (7) Becuse the sequence of rewrds is nerly rndom nd N is lrge, different columns of the mtrix Rew re nerly orthogonl. This implies tht the mtrix product (Rew T Rew) is well conditioned nd tht the solution f sol minimizing the vrince of the noise (or squred error) is roust nd given y T 1 T fsol = ( Rew irew) irew ifr This expression is used to otin the results. The confidence intervls for f sol re derived from the residul errors ccording to the Mtl (Mthworks) function regress. The mtrix product (Rew T Rew) is pproximtely proportionl to the identity mtrix. When Rew T Rew = I, the filter is equl to the firing rte verged over ll trils, where the verge is conditioned on the pst rewrds. This is equivlent to the cross-correltion etween the input (rewrds) nd output (firing rtes) nd its ppliction would correspond to reverse correltion method, commonly used in the nlysis of sensory neurl coding. Here, however, we only showed results from the multiple regression nlysis. For simplicity, we used n verge over ll trils s the definition of epoch code g(k) in the min text, mking use of the ove pproximtion. Exponentil memory trces nd model selection. The model considered here is similr to tht of eqution (3), ut we ssumed tht memory trces re exponentil function ex(t) rescled y the epoch code g(k). FR( n, k) = g( k) + g( k) ex( t) Rew( n n ) + noise n =. The filter f considered in eqution (3) is replced y g(k) ex(t). We considered two different exponentil functions, single exponentil nd the sum of two exponentils. ex 1 ( t) = Ae t t ex 1 t ( t) = A 1 e + Ae (6) (8) (9) (1) (11) where τ 1 < τ. The physicl time t depends on ll indices k, n nd n ecuse the time elpsed etween different epochs nd etween successive trils is vrile, due to the vriility in the time tken y the niml to strt tril nd to mke sccde to one of the two trgets. On the sis of the time stmps generted during the experiment, we computed the physicl time t = t(n,k,n ) s the difference etween the time corresponding to given tril nd epoch (n,k) nd the time corresponding to the feedck epoch of n trils in the pst (up to five trils). Note tht the memory trce f otined y the multiliner regression is not computed in physicl time. In tht cse, we ssumed tht the sccde rection time of the niml in ll trils is equl to 1 ms (verge) nd tht the time elpsed etween the initition of two successive trils is 3.4 s (medin). The epoch code g(k) ws fixed y the firing rtes verged cross trils, wheres the prmeters of the exponentil function (two prmeters (A,τ) when using eqution (1) nd four prmeters (A 1,τ 1, A,τ ) when using eqution (11)) were estimted using nonliner curve-fitting procedure, implemented y the Mtl function fminserch, minimizing the vrince of the noise (sum of squred errors) in eqution (9). Fitting ws repeted ten times for ech neuron nd ech model in the serch for glol minimum of the error. Any prmeters resulting in unrelistic vlues were discrded, such s negtive vlues of τ, τ 1 or τ, vlues of doi:1.138/nn.7 nture NEUROSCIENCE

9 11 Nture Americ, Inc. All rights reserved. τ lrger thn trils, nd the solute vlue of A or (A 1 + A ) lrger thn 4. We determined the prmeters for ll neurons in oth exponentil models, single nd doule exponentil nd denoted the corresponding squre errors y σ 1 nd σ, respectively. We lso computed the vrince of firing rte, σ, s the squre error for zero filter model, tht is, ex = or FR = g + noise. Among the three models, the selection of the pproprite one for ech neuron ws determined ccording to the Byesin informtion criterion (BIC) BIC i = mlog( s i ) + p i log( m) where p i denotes the numer of prmeters in the model, nd p = 1, p 1 = 3, p =, for, 1 nd exponentil fit, respectively (note tht the vrince σ i is lso prmeter), nd m is the numer of dt points (m = 1(N ); 1 epochs nd N trils for ech neuron). The model with the minimum BIC ws chosen for ech neuron. As control of the fitting procedure, we reshuffled the lel n in the firing rtes FR(n,k), ssigning to ech firing rte the vlue of rndom tril, nd we repeted the entire procedure. Reinforcement lerning fit of ehvior. We pplied stndrd reinforcement lerning model, seprtely for ech recording session, to nlyze how the niml s choice ws influenced y the outcomes of its previous choices. For exmple, when right trget R ws chosen in tril t, the vlue function for R, denoted y Q R (t), ws updted ccording to QR( t + 1) = QR( t) + [ Rew( t) QR( t)] (1) (13) where Rew(t) denotes the rewrd received y the niml in tril t, nd the term inside squre is commonly defined s the rewrd prediction error; tht is, the discrepncy etween the ctul rewrd nd the expected rewrd. A similr eqution holds for the left vlue function Q L (t). The proility tht the niml would choose the rightwrd trget in tril t, P R (t), ws determined y the SoftMx trnsformtion exp( Q t P t R( )) R( ) = exp( QL( t)) + exp( QR( t)) (14) where β, referred to s the inverse temperture, determines the rndomness of the niml s choices. Model prmeters (α,β) were estimted seprtely for ech recording session y using mximum likelihood procedure, where the likelihood is the product of proilities in ll trils (eqution (14)), in ech tril using R or L ccording to the ctul monkey s choice. The prmeter vlues mximizing the likelihood were found y using the Mtl function fminserch. The significnce of the estimtion ws ssessed, for ech session, y constructing 1 surrogte sessions, ech one otined y reshuffling of the order of trils. The distriution of 1 mximum likelihoods otined y the estimtion procedure ws then compred with the mximum likelihood of the non-reshuffled cse, which ws considered to e significnt if not smller thn the five lrgest reshuffled likelihoods. Vlue functions nd rewrd prediction error signls cn e relted to the exponentil filters estimted for individul neurons. If single vlue function (for given stimulus/ction) nd single rewrd (delivered t time zero) re considered, the solution of eqution (13) cn e pproximted y n exponentil response, t Q( t) = ( 1 1 ) ~ exp( ), provided tht τ is lrger thn one tril. When t t sequence of rewrds is delivered insted of single one, the vlue is superposition of the exponentil responses for ech rewrd. nture NEUROSCIENCE doi:1.138/nn.7