doi:.38/nture72 Neurl correltes, computtion nd ehviourl impct of decision confidence Kepecs A., Uchid N., Zriwl H. nd Minen Z.F. Confidence estimtes in integrtor models of decision-mking Computing decision confidence requires tht in ddition to inry choice, the decision process yields grded vlue mesuring the reliility nd consistency of the internl vriles contriuting to decision. This cn e chieved for other clsses of models including models sed on the integrtion of evidence tht re le to lso ccount for other fetures of ehviour, such s rection times -3. To demonstrte the generlity of the model predictions (Fig. 4c,d) we simulted version of the integrtor model, the rce model. In rce models, seprte decision vriles ccumulte evidence for different options nd the decision tken is determined y which decision vrile reches threshold first (Supplementry Fig. 6). To Nsimulte this, in ech tril the stimulus is normlly distriuted rndom vrile s(t) N(µstim, σstim), where the sign of µstim sets the direction of correct choice nd the signl-to-noise rtio µstim/σstim sets the difficulty of discrimintion. In the simplest version of the rce model there re two independent decision vriles tht ccumulte evidence for nd ginst the hypothesis tht µstim >. Ech decision vrile, e(t), ccumultes evidence for one direction: where nd When one of the decision vriles, e + (t) or e - (t), reches predetermined threshold, θ, the rce is terminted nd decision is generted in fvour of the decision vrile crossing threshold first. Therefore t decision time, tθ, e + (tθ) = θ or e - (tθ) = θ. We simulted this model with the following prmeters: µstim U(-.2,.2), σstim =, θ =, nd dt =. Supplementry Fig. 6c shows the frction of choices in fvour of the + hypothesis, µstim >, s function of stimulus, µstim. This psychometric curve is qulittively similr to tht of rts (compre to Fig. c). An estimte of decision confidence cn e computed in rce model y mesuring the distnce etween the two decision vriles t the time the rce is terminted. This ws originlly proposed y Vickers 4, who termed it the lnce of evidence. To see tht the distnce etween decision vriles cn provide resonle estimte of confidence, we plot choice ccurcy s function of this distnce, Δe = e + (tθ) - e - (tθ) (Supplementry Fig. 6d, dshed line). This distnce, Δe, cn e normlized, Δe/θ, to yield the lnce of evidence mesure 4. Here insted we sought to compute n estimte, δ, tht is closer to the veridicl confidence nd reflects the ctul outcome proility. For perfectly clirted or veridicl confidence estimte δ would correspond to the proility of correct outcome, from chnce level (δ = ) to perfect (δ = ) performnce. The considertion of the theoreticlly pproprite clirtion method is eyond our scope. For given signl to noise rtio the correct clirtion function my e derived y considering the error rte s function of the decision threshold -3. Here we used n pproximtion, δ = f(δe) = 2/(+e c(δe/θ) ), with c =/3, which provides excellent performnce cross multiple stimuli (see Supplementry Fig. 6). The role of clirtion is illustrted in www.nture.com/nture
doi:.38/nture72 Supplementry Fig 6d, showing tht ccurcy is nerly liner function of δ (solid line) in contrst to the normlized distnce, Δe/θ (dshed line). Finlly, when this confidence estimte, δ, is plotted s function of stimulus nd outcome it shows the sme pttern s the model in min text (cf. Fig. 4d nd Supplementry Fig. 6e). Note tht for given stimulus rnge (signl-to-noise rtio, µstim/σstim), there is only single free prmeter (threshold, θ) tht determines the slope of the psychometric function. This qulittive pttern the opposing V-shped curves for correct nd error choices ws roust to different choices of prmeters. This exmple demonstrtes tht confidence estimtes cn e redily computed in other clsses of decision models, nd yield qulittively similr predictions. The confidence estimte exmined is not unique, however, nd my e generlized to other mesures of decision uncertinty. For instnce, the vrince or entropy of the decision vriles could provide sis for confidence estimtion. For the rce model, the vrince of the decision vriles is simply the squre of the estimte introduced ove: δ V =(e + (tθ) - e - (tθ)) 2. The entropy of decision vriles pi, is δ S =- i pi log(pi). For the rce model this cn e expressed s δ S =- p + log(p + ) - p - log(p - ) with p +/- = e +/- (tθ)/(e + (tθ) + e - (tθ)). The limiting cses of mximum nd minimum uncertinty re instructive nd esy to clculte. Uncertinty is highest when the two decision vriles rech threshold t the sme time, which would mke δ S = it, while uncertinty is lowest when ll the evidence ccrues in fvour of one decision vrile, which would mke δ S = it. Such mesures re likely to e useful when considering confidence estimtes sed on popultion of neurons 5. Note, however, tht δ S needs to e clirted in order to provide veridicl outcome predictions. Therefore mesures like δ S could serve s sufficient sttistics for computing confidence tht need to e clirted to yield n instntneous estimte of outcome likelihood. The pproprite clirtion function in turn cn e found using reinforcement lerning. Intuition for the confidence model To provide intuition into the unexpected pttern of uncertinty s function of stimulus nd outcome (Fig. 4c) we need to exmine how different stimulus nd memory smple configurtions led to choice nd n estimte of confidence (Fig. 4). Note tht the model (or suject) hs ccess only to stimulus smple nd not the stimulus type (e.g. 56/44) s plotted. First, consider tht error choices occur when on given tril the stimulus, si, nd memory, i smples re reversed compred to the men of their respective distriutions. This cn only occur within region where the two distriutions overlp, wheres correct ordering my occur over the entire rnge of stimulus vlues. Therefore the size of the overlp region will plce limit on the mximl confidence vlues tht cn e ttined for error choices. Since the region of overlp is smller thn the entire rnge, the mximl distnce etween smples will e smller for errors thn correct choices, resulting in lower confidence estimtes (higher uncertinty) in the choice on verge. Moreover, the further wy the stimulus is from the oundry, the smller the possile region of overlp etween their distriutions, nd therefore the smller the mximum possile distnce etween the two smples in n error tril. Consequently, for esy stimuli errors re rre ecuse the overlp is smll, nd in those few incorrect trils the smples cnnot possily e fr from ech other resulting in low confidence estimtes (high uncertinty). www.nture.com/nture 2
doi:.38/nture72 Decision confidence or risk? Decision confidence is form of sujective uncertinty, ut it is importnt to distinguish it from other forms of uncertinty. The term uncertinty is often used synonymously with rewrd or outcome risk 6-. Decision uncertinty nd outcome uncertinty re similr in tht oth re sed on clculting the vrince or entropy cross set of vriles. The criticl distinction is the set of vriles over which the mesure is clculted. Decision uncertinty is mesured cross vriles oserved in single tril. Outcome uncertinty, in contrst, is mesured cross outcomes oserved over multiple trils. In free choice tsks it is possile to independently mnipulte outcome proility nd outcome vrince. In contrst, in two-lterntive psychophysicl decision tsks outcome proility rnges from.5 to nd therefore covries with outcome vrince. Hence our dt re consistent with either representtion of outcome proility or outcome uncertinty signls. Either wy, the computtion of such signl must incorporte n estimte of decision uncertinty. References. Bogcz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optiml decision mking: forml nlysis of models of performnce in two-lterntive forced-choice tsks. Psychol Rev 3, 7-65 (26)/ 2. Mzurek, M. E., Roitmn, J. D., Ditterich, J. & Shdlen, M. N. A role for neurl integrtors in perceptul decision mking. Cere Cortex 3, 257-69 (23). 3. Rtcliff, R. & Smith, P. L. A comprison of sequentil smpling models for two-choice rection time. Psychol Rev, 333-67 (24). 4. Vickers, D. Evidence for n ccumultor model of psychophysicl discrimintion. Ergonomics 3, 37-58 (97). 5. Zemel, R. S., Dyn, P. & Pouget, A. Proilistic interprettion of popultion codes. Neurl Comput, 43-3 (998). 6. Critchley, H. D., Mthis, C. J. & Doln, R. J. Neurl ctivity in the humn rin relting to uncertinty nd rousl during nticiption. Neuron 29, 537-45 (2). 7. Hsu, M., Bhtt, M., Adolphs, R., Trnel, D. & Cmerer, C. F. Neurl systems responding to degrees of uncertinty in humn decision-mking. Science 3, 68-3 (25). 8. Fiorillo, C. D., Toler, P. N. & Schultz, W. Discrete coding of rewrd proility nd uncertinty y dopmine neurons. Science 299, 898-92 (23). 9. McCoy, A. N. & Pltt, M. L. Risk-sensitive neurons in mcque posterior cingulte cortex. Nt Neurosci 8, 22-7 (25).. Toler, P. N., O'Doherty, J. P., Doln, R. J. & Schultz, W. Rewrd vlue coding distinct from risk ttitude-relted uncertinty coding in humn rewrd systems. Journl of neurophysiology 97, 62-32 (27). www.nture.com/nture 3
doi:.38/nture72 Supplementry Figure. Antomicl loction of recording sites Nissl-stined coronl section of rt frontl cortex showing the lesion sites from one tetrode. Left shows the estimted re of the recordings from 3 rts overlid on section 3.6 mm nterior to regm. Recording loctions rnged from +3. mm to +4.2 mm. www.nture.com/nture 4
doi:.38/nture72 c Accurcy (%) Accurcy (%) 75 5 75 N-479-6.2 Low rtes High rtes model 32 44 56 68 Odor mixture (%A) 5 N=33 32 44 56 68 Odor mixture (%A) Supplementry Figure 2. Neurl ctivity predicts ehviourl ccurcy eyond stimulus informtion, Behviourl ccurcy s function of stimulus nd rte for single neuron (sme s in Fig 4.e,f). Men ccurcy +/- s.e.m is plotted s function of stimulus nd neurl firing rte. Trils with t lest one spike were ssigned to low or high firing rtes ccording to whether the spike count ws ove or elow the medin., Averge ehviourl ccurcy s function of stimulus nd rte for the negtive outcome selective neuron popultion (Fig 4.g,h). Error rs represent s.e.m. cross neurons. c, Accurcy s function of stimulus nd decision uncertinty in the model. Trils re divided into low nd high uncertinty groups (elow nd ove medin uncertinty levels). www.nture.com/nture 5
doi:.38/nture72 Confidence model Stimulus + side selectivity model Rte (spk/s).8.6.4.2 Rte (spk/s).8.6.4.2 5 Odor mixture (%A) 5 Odor mixture (%A) c d O.P. for 68/32 mixtures.5 -.5 32 & 68 O.P. 32 O.P. 68 O.P. N.S. - - -.5.5 O.S. for 32-68 mixtures.8.6.4 44-56 & 32-68 O.S..2 44-56 O.S. 32-68 O.S. N.S..2.4.6.8 Outcome preference (O.P.) for 32/68 mixtures Outcome selectivity (O.S.) for 44-56 mixtures Supplementry Figure 3. Outcome selectivity follows predictions of the confidence model,, Schemtics illustrting the predictions of the confidence model (), nd the side selectivity model (), with colours denoting the outcome of choice (correct: green; error: red). Dshed rrows signify the distnce etween error nd correct choices of given difficulty. The direction of solid rrows signifies whether error or correct choices hve higher rtes for given stimulus. c, Outcome preference index (O.P.) for 32/68 mixtures s function of O.P. for 68/32 mixtures. All neurons included were significnt when stimuli were pooled nd colours show (see inset) whether these vlues were significntly outcome selective when considered seprtely (P <.5, permuttion test). d, Outcome selectivity index (O.S.) for 32-68 mixtures (32/68 nd 68/32 comined) s function of O.S. for 44-56 mixtures (44/56 nd 56/44 comined). 68/32 mixtures. All neurons included were significntly outcome selective when stimuli were pooled nd colours show (see inset) whether these vlues were significnt when considered seprtely (P <.5, permuttion test). www.nture.com/nture 6
doi:.38/nture72 Rte (spk/s) 2 8 4 Correct Odor 56-44 68-32 Error Normlized rte.8.7.6.5.4 N49-579-2.3.3 N = 5 2 Time choice port entry (s) Time choice port entry (s) c.7 d Normlized rte.6.5.4 Accurcy (%) 9 8.3 N = 5 32 44 56 68 Odour mixture rtio (%) 7 N = 5.5 Normlized rte Supplementry Figure 4. Positive outcome selective neurl popultion, Activity of n exmple neuron showing positive outcome selectivity. Firing rte of single cell ligned to the time of entry into the choice port. Trils re grouped y stimulus difficulty (56/44 nd 44/56, redish; 68/32 nd 32/68 lueish colours) nd tril outcome (correct/error, light/drk colours; see inset). Only ctivity occurring efore the onset of wter delivery or choice port exit is verged into the PSTH. After the nticiption period (.3 s in this session) the PSTH curves re dshed, signifying time period during which in some trils rts experienced rewrd delivery., Men normlized firing of the positive outcome selective popultion (those showing incresed firing rte in error trils during the nticiption period). Only ctivity occurring while the rt ws in the choice port nd witing for rewrd is verged. Dshed curves signify time t which rewrd ws experienced in frction of sessions. Legend is sme s for pnel (). c, Popultion tuning curves for firing rte during the initil.4 s outcome nticiption period s function of stimulus type nd outcome for the sme popultion s in (). Individul tuning curves were normlized nd error rs represent s.e.m. cross neurons. d, Men ccurcy s function of the firing rte for the sme neurl popultion s in. Firing rtes were inned for individul neurons nd the men ccurcy ws clculted for ech rnge of firing rtes. These curves were normlized to mximl firing rte of nd verged. Error rs represent s.e.m. cross neurons. www.nture.com/nture 7
doi:.38/nture72 8 Coefficient vlue 6 4 2 N-479-6.2 4 α α 2 L β R β β β β 2 β 2 β 3 β 3 2 Coefficient vlue 3 2 8 6 4 2 Numer of neurons α α 2 L β R β β β β 2 β 2 β 3 β 3 Regression coefficients Supplementry Figure 5. Impct of outcome history on firing rtes, Regression coefficients for the multiple regression nlysis (see Methods) for the neuron presented in Figure 2,. Error rs mrk the stndrd devitions of the coefficients estimted using leve-one-out-ootstrp vlidtion. Filled circles indicte coefficients tht re significntly different from (P <.5, permuttion test). β t L/R re the coefficients for the outcome t tril t (t = for the current tril) on either the left or right (L/R) choice sides. α nd α 2 mesure the influence of the stimulus difficulty nd the choice., Men regression coefficients (left scle) for the popultion of negtive outcome selective neurons (n = 33) nd their stndrd error. Br plots show the numer of neurons (right scle) with significnt regression coefficients (P <.5, permuttion test). Only neurons with significnt regression coefficients re included in the verges. www.nture.com/nture 8
doi:.38/nture72 Evidence Evidence for A Threshold (θ) Evidence ginst A e Frction of trils.2. Confidence ( ).8.6.4.2 2 3 Time.2.4.4.8 Blnce of evidence ( ).4 Frction of trils c d Blnce of evidence ( ).2.4.6.8 e.8 % choice + 8 6 75 4 2 5 -.2.2.5 Stimulus ( stim ) Confidence ( ) δµ Accurcy (%) δ e/θδconfidence ( ).7 correct.6.5.4 error.3 -.2.2 Stimulus ( stim ) µ Supplementry Figure 6. Computing confidence in rce model of decision mking, Schemtic of tril for the rce model of decision mking. Evidence for nd ginst n lterntive A ccumultes seprtely over time. When the ccumulted evidence crosses threshold level decision is generted its fvour. Confidence out the decision my e estimted sed on the distnce etween the decision vriles, e., Clirtion of e provides the estimte of confidence, δ. Left histogrm shows the distriution of e cross trils with the clirtion function overlid. The resulting distriution of δ is displyed on the right. c, Percentge of choices towrds the + hypothesis, µ stim >, s function of stimulus. d, Choice ccurcy s function of confidence, δ (solid line), nd lnce of evidence, e/θ (dshed line). e, Men confidence estimtes generted y the model conditioned on stimulus nd tril outcome. www.nture.com/nture 9
doi:.38/nture72 8 % choice A 6 4 2 Unscled confidence ( s- ) 4 3 2 Correct SD.5. Error 2 4 6 8 Odour mixture (%A) Supplementry Figure 7. Prmeter roustness of decision confidence ptterns, Psychometric functions generted y the ctegoriztion model presented in the min text. The totl noise, σ noise = (σ 2 ound + σ 2 stim) /2, ws vried to produce psychometric functions with different slopes., Men unscled confidence estimtes, d i = s i -- i, generted y the sme model nd plotted s function of stimulus nd tril outcome. www.nture.com/nture
doi:.38/nture72.8.4 O.P. & D.I. D.I. O.P. N.S. Difficulty index -.4 -.8 -.8 -.4.4.8 Outcome preference index Supplementry Figure 8. Stimulus nd outcome selectivity re correlted Difficulty index (D.I.) s function of outcome preference (O.P.) s descried in the Methods. Ech dot is one neuron. Colours show (see inset) whether these vlues were significnt (P <.5, permuttion test). www.nture.com/nture