This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and

Transcription

1 Ths artcle appeared n a journal publshed by Elsever. The attached copy s furnshed to the author for nternal non-commercal research and educaton use, ncludng for nstructon at the authors nsttuton and sharng wth colleagues. Other uses, ncludng reproducton and dstrbuton, or sellng or lcensng copes, or postng to personal, nsttutonal or thrd party webstes are prohbted. In most cases authors are permtted to post ther verson of the artcle (e.g. n Word or Tex form) to ther personal webste or nsttutonal repostory. Authors requrng further nformaton regardng Elsever s archvng and manuscrpt polces are encouraged to vst:

2 Avalable onlne at Cogntve Systems Research 9 (28) Neural affectve decson theory: Choces, brans, and emotons Acton edtor: Ron Sun Abnnder Ltt a, *, Chrs Elasmth b,c, Paul Thagard b,d,e a Graduate School of Busness, Stanford Unversty, Stanford, CA , Unted States b Department of Phlosophy, Unversty of Waterloo, Ontaro, Canada N2L 3G1 c Department of Systems Desgn Engneerng, Unversty of Waterloo, Ontaro, Canada N2L 3G1 d Department of Psychology, Unversty of Waterloo, Ontaro, Canada N2L 3G1 e Cherton School of Computer Scence, Unversty of Waterloo, Ontaro, Canada N2L 3G1 Receved 25 June 27; accepted 29 November 27 Avalable onlne 14 Aprl 28 Abstract We present a theory and neurocomputatonal model of how specfc bran operatons produce complex decson and preference phenomena, ncludng those explored n prospect theory and decson affect theory. We propose that valuaton and decson makng are emotonal processes, nvolvng nteractng bran areas that nclude two expectaton-dscrepancy subsystems: a dopamne-encoded system for postve events and a serotonn-encoded system for negatve ones. The model provdes a rgorous account of loss averson and the shape of the value functon from prospect theory. It also suggests multple dstnct neurologcal mechansms by whch nformaton framng may affect choces, ncludng ones nvolvng antcpated pleasure. It further offers a neural bass for the nteractons among affect, pror expectatons and counterfactual comparsons explored n decson affect theory. Along wth predctng the effects of partcular bran dsturbances and damage, the model suggests specfc neurologcal explanatons for ndvdual dfferences observed n choce and valuaton behavors. Ó 28 Elsever B.V. All rghts reserved. Keywords: Computatonal neuroscence; Decson makng; Emoton; Framng; Prospect theory 1. Introducton How do people decde what clothes to wear, what to eat for dnner, what car to buy, or what knd of career to pursue? In tradtonal economcs, the standard answer s that people decde by maxmzng expected utlty, but psychologsts have found many problems wth ths knd of decson theory as a descrpton of human behavor (e.g., Camerer, 2; Kahneman & Tversky, 2; Koehler, Brenner, & Tversky, 1997; Rottenstrech & Hsee, 21; Tversky & Kahneman, 1991). Economsts commonly take preferences as gven, but from a psychologcal pont of vew t should * Correspondng author. E-mal address: altt@stanford.edu (A. Ltt). be possble to explan how preferences arse from cogntve and affectve processes. Work n ths sprt has made tremendous progress n revealng key features and dynamcs mssed by theores dsconnected from the study of cogntve, emotonal and socally motvated phenomena, such as a common hypersenstvty to losses over equvalent gans (Kahneman & Tversky, 1979) and the affectve nfluence of pror expectatons and counterfactual comparsons on preference judgments (Mellers, 2). Moreover, wth the rse of cogntve and affectve neuroscence, t should be possble to dentfy precse neural mechansms underlyng these behavoral-level explanatons of why people make the choces that they do. We propose neural affectve decson theory as a psychologcally and neurologcally realstc account of specfc bran mechansms underlyng human preference and /$ - see front matter Ó 28 Elsever B.V. All rghts reserved. do:1.116/j.cogsys

3 A. Ltt et al. / Cogntve Systems Research 9 (28) decson. The theory conssts of four prncples, whch we shall lst here and descrbe n detal later: 1. Affect. Decson makng s a cogntve affectve process, crucally dependent on emotonal evaluaton of potental actons. 2. Bran. Decson makng s a neural process drven by coordnated dynamc nteractons among multple bran areas, ncludng parts of prefrontal cortex as well as major subcortcal systems. 3. Valuaton. The bran forms preferences va nteractng but dstnct mechansms for postve and negatve outcomes, encoded prmarly by dopamne and serotonn, respectvely. 4. Framng. Judgments and decsons vary dependng on how the context and manner of the presentaton of nformaton ntate dfferent neural actvaton patterns. There s substantal emprcal evdence for each of these prncples, and when ntegrated n the precse manner we outlne they can explan the fndngs of a wde range of psychologcal and neurologcal phenomena. In order to connect these prncples wth expermental results n a mathematcally and neurologcally rgorous fashon, we have developed a neurocomputatonal model called ANDREA (affectve neuroscence of decson through reward-based evaluaton of alternatves). It operates wthn the neural engneerng framework (NEF) developed by Elasmth and Anderson (23), usng bologcally realstc populatons of neurons to encode and transform complex representatons of relevant nformaton. ANDREA smulates computatons among several thousand neurons to model coordnated actvtes n seven major bran areas that contrbute to valuaton and decson makng: the amygdala, orbtofrontal cortex, anteror cngulate cortex, dorsolateral prefrontal cortex, the ventral stratum, mdbran dopamnergc neurons, and serotonergc neurons centered n the dorsal raphe nucleus of the branstem. ANDREA successfully produces detaled neural-level smulatons of behavoral fndngs explored n prospect theory (Kahneman & Tversky, 1979) and the decson affect theory of Mellers and colleagues (1997). It shows how specfc neural processes can produce behavors observed n both psychologcal experments and real-world scenaros that have provded compellng evdence for these preference and choce theores. In partcular, ANDREA provdes neurologcal explanatons for the major hypothess of prospect theory that losses have greater psychologcal force than gans, as well as for the fundamental clam of decson affect theory that the evaluaton (and subsequent potental choce) of an opton s strongly nfluenced by ts perceved relatve pleasure, an emotonal determnant that s dependant on expectatons and counterfactual comparsons. In our concludng dscusson, we compare ANDREA to other models n decson neuroscence, descrbe promsng avenues of expanson for ANDREA and neural affectve decson theory, and suggest addtonal psychologcal phenomena that are lkely to fall wthn the scope of our theory. 2. Neural affectve decson theory We now examne n detal the four gudng prncples of neural affectve decson theory, ncludng connectons to and supportng evdence provded by a dverse array of research n both psychology and neuroscence. The ANDREA mplementaton of the theory we descrbe later provdes the formal ntegraton of these deas necessary for our detaled smulaton experments Prncple 1. Affect Accordng to our frst prncple, decson makng s a cogntve affectve process, crucally dependent on emotonal evaluaton of potental actons. Ths clam rejects the assumpton of tradtonal mathematcal decson theory that choce s a cold process nvolvng the calculaton of expected values and utltes (Kreps, 199; Von Neumann & Morgenstern, 1947). The orgnal 19th-century concept of utlty was a psychologcally rch, affectve one based on pleasure and pan (Kahneman, Wakker, & Sarn, 1997). In contrast, 2th-century economcs adopted the behavorst vew that utltes are mathematcal constructons based on preferences revealed purely by behavor. There s no room n ths vew for fndngs observed n both psychologcal experments and everyday lfe that people s decsons are often hghly emotonal, wth preferences arsng from havng postve feelngs for some optons and negatve ones for others. Whle psychology has ntroduced a more complex characterzaton of the cogntve processes underlyng decson makng, the specfc nfluence of affect on behavor has frequently been gnored. Rottenstrech and Shu (24) argue that ths neglect of affect may stem from an orgnal desre of psychologcal decson researchers to mnmze dfferences wth the termnology and general themes of classcal normatve decson theores. But there s ncreasng apprecaton n cogntve scence that emotons are an ntegral part of decson makng (e.g. Bechara, Damaso, & Damaso, 2, 23; Churchland, 1996; Lerner & Keltner, 2; Loewensten, Weber, Hsee, & Welch, 21; Sanfey, Loewensten, McClure, & Cohen, 26; Slovc, Fnucane, Peters, & MacGregor, 22; Wagar & Thagard, 24). Kahneman (23, p. 71) argues that there s compellng evdence for the proposton that every stmulus evokes an affectve evaluaton. Common experence suggests that emotons are both nputs and outputs of decson makng. Preference for one opton over another depends strongly on ther relatve emotonal nterpretatons, and the process of decson makng can tself generate emotons such as anxety or relef. The relevance of emoton to decson makng s consstent wth physologcal theores that regard emotons as reactons to somatc

4 254 A. Ltt et al. / Cogntve Systems Research 9 (28) changes (Damaso, 1994; James, 1884). It also fts wth some cogntve theores of emotons, whch regard them as judgments about the extent to whch ones goals are beng satsfed (Oatley, 1992). From a neurologcal perspectve, t s easy to see how emotons can be both cogntve and physologcal, as there are numerous nterconnectons among the relevant bran areas Prncple 2. Brans Accordng to our second prncple, decson makng s a neural process drven by coordnated dynamc nteractons among multple bran areas, ncludng parts of prefrontal cortex as well as major subcortcal systems. In partcular, actvty n bran regons nvolved n assessng and actng upon the appettve or aversve nature of stmul (commonly conceptualzed as part of the bran s reward system) seems most crucal to understandng judgment and choce behavor (for a revew, see Sanfey et al., 26). Emprcal neuroscentfc nvestgaton of the nature of preference and decson has been developng rapdly, and much work today s dentfyng specfc bran areas nvolved n producng decson-related behavors (e.g., Bayer & Glmcher, 25; Breter, Aharon, Kahneman, Dale, & Shzgal, 21; Knutson, Taylor, Kaufman, Peterson, & Glover, 25; McClure, York, & Montague, 24; Montague & Berns, 22). Ths nascent feld of decson neuroscence represents an exctng fronter of deep exploraton nto how and why people act, thnk and feel as they do n choce and judgment scenaros (Shv et al., 25). But takng a neural approach to decson makng allows for much more than smply dentfyng bran areas actvated n the subjects of behavoral studes. The development of bologcally plausble theores of how bran areas nteract to produce preferences and choces can provde more refned mechanstc explanatons of decson behavors. Moreover, nvestgaton at the neural level can suggest novel experments, for example the recent dscovery that an odorless nasal spray preparaton of the neuropeptde oxytocn ncreases trust n rsky choce scenaros, ncludng those nvolvng monetary transactons (Kosfeld, Henrchs, Zak, Fschbacher, & Fehr, 25). Such a fndng s one that decson neuroscence can reveal, but that would be mssed by hgher-level psychologcal study alone. Neuroscence can thus nform the development of more detaled predctons and rcher understandngs of behavoral-level observatons Prncple 3. Valuaton Our thrd prncple states that the bran forms preferences va nteractng but dstnct mechansms for postve and negatve outcomes, encoded prmarly by dopamne and serotonn, respectvely. There s extensve evdence that mdbran dopamne neurons, such as those n the ventral tegmental area and nucleus accumbens, are nvolved n the computaton of a dscrepancy between the expected and actual rewardng nature of an outcome (e.g., Knutson et al., 25; Schultz, 2; Sur, 22), although recent evdence suggests that ths actvty s only nvolved n the encodng of postve devatons from expectatons, that s, gettng more than one expected. (Bayer & Glmcher, 25). Daw, Kakade, and Dayan (22) descrbe a plausble alternatve bran mechansm for stuatons n whch one receves less than expected, argung that serotonn nnervaton from the dorsal raphe nucleus of the branstem s crucal for producng characterstc reactons to negatvely valued stmul and matters beng consdered (e.g., optons n a choce scenaro). There are thus neurobologcal reasons for vewng gans and losses as beng encoded and subsequently assessed n a fundamentally dfferent manner by the bran, nvolvng dstnct neural crcuts and actvaton patterns. Ths provdes the bass for our explanaton of the central fndng of prospect theory that losses loom larger than gans. Our neurocomputatonal model ANDREA smulates how nteractons of the dopamne and serotonn systems wth the amygdala and other bran areas may enable ths asymmetrc assessment of postve and negatve outcomes Prncple 4. Framng The last prncple states that judgments and decsons vary dependng on how the context and manner of the presentaton of nformaton ntate dfferent neural actvaton patterns. The mportance of framng s evdent from the long hstory of nfluental work by Tversky and Kahneman (1981, 1986; see also Kahneman & Tversky, 2). They demonstrated that framng a decson n terms of ether losses or gans can substantally affect the choces that people make, and related phenomena have been observed n many real-lfe arenas such as the stock market and consumer choce (Camerer, 2). We contend that framng can be understood even more deeply from the neural affectve perspectve we propose n our frst two prncples. The smulaton results we descrbe later show how ths enrched concepton of framng allows for the ntegraton of dverse lnes of behavoral decson research, as well as the postulaton of mportant new predctons and hypotheses. We take the concept of framng to encompass any potental effects of the manner or context of presentaton on decsons and judgments. Followng ths characterzaton, such fndngs as preference reversals when outcomes are evaluated jontly versus separately (e.g., Hsee, Loewensten, Blount, & Bazerman, 1999; Hsee, Rottenstrech, & Xao, 25) mght also be consdered to be framng results. Another such framng effect s llustrated by the trolleyfootbrdge dlemma (e.g., Greene, Sommervlle, Nystrom, Darley, & Cohen, 21): most people consder flppng a swtch to kll one person nstead of fve morally justfed, but consder t mmoral to personally push a person nto the path of an oncomng trolley, kllng that person but preventng the trolley from kllng fve others. We wll also explan some of the fndngs of the decson affect theory of

5 A. Ltt et al. / Cogntve Systems Research 9 (28) Mellers and colleagues (1997) as framng effects that dfferentally actvate specfc neural systems. These four prncples make strong clams about the processes that consttute human decson makng, but alone they are not suffcently precse to explan partcular expermental results. We now descrbe a rgorously defned, bologcally realstc neurocomputatonal model that specfes how dfferent bran areas mght nteract n a manner consstent wth neural affectve decson theory to produce observed behavoral phenomena. AMYG OFC DLPFC 5-HT DA VS 3. The ANDREA model A neuropsychologcal theory conssts of a set of hypotheses about how specfc bran operatons produce observed behavors. Because of the complexty of the bran, computatonal models are ndspensable for theoretcal neuroscence, both for precsely specfyng relevant neural structures and actvtes and for examnng ther mplcatons through approprate smulaton experments. Ltt, Elasmth, and Thagard (26) proposed a bologcally detaled neural model of reward system substrates of valuaton and decson. We descrbe here the prmary functonal components of ths model, and ntroduce several explanatorly valuable addtons regardng neural response characterstcs and the complexty of emotonal arousal encodng. Ths verson of the model we call ANDREA, for affectve neuroscence of decson through reward-based evaluaton of alternatves. Our model apples the neural engneerng framework (NEF) developed by Elasmth and Anderson (23), and has been mplemented n MATLAB usng the NEF smulaton software NESm (see Appendx A). Neural populatons ( ensembles ) and ther frng actvtes are descrbed n the NEF n terms of mathematcally precse representatons and transformatons, wth the dynamc characterstcs of neural computatons characterzed usng the tools of modern control theory. Appendx B outlnes the exact mathematcal nature of representaton, transformaton and dynamcs as defned by the NEF. Ths rgorous, generalzed mappng of hgh-level mathematcal enttes and transformatons onto bophyscal phenomena such as spke patterns and currents allows for bologcally constraned computatons and dynamcs to be mplemented n physologcally realstc neural populatons, and has proven successful n modelng phenomena rangng from the swmmng of lamprey fsh (Elasmth & Anderson, 2) to the Wason card task from cogntve psychology (Elasmth, 25b). Fg. 1 shows the connectvty structure between the dfferent bran regons we have modeled. A comprehensve examnaton of afferent and efferent transmsson among these regons would feature many more connectons than we have ncluded. The nteractons shown represent partcular paths of coordnated actvty that contrbute to observed behavors, rather than a full characterzaton of all relevant neural actvty. Appendx A provdes detals ACC Fg. 1. Basc connectvty framework. Dotted arrows represent external nputs to the model. Abbrevatons: 5-HT, dorsal raphe serotonergc neurons; ACC, anteror cngulate cortex; AMYG, amygdala; DA, mdbran dopamnergc neurons; DLPFC, dorsolateral prefrontal cortex; OFC, orbtofrontal cortex; VS, ventral stratum. regardng the specfc numbers of neurons used to model each of these bran areas, as well as the physologcal parameters used to model ndvdual neurons n each of these populatons. Each nput output relaton symbolzed by a connecton lne n Fg. 1 maps onto one or more specfc mathematcal transformatons, as summarzed n Appendx C. We now descrbe these coordnated neural computatons as they are relevant to explanng decsons and valuatons Subjectve valuaton by emotonal modulaton Valuaton of alternatves and other nformaton s an essental part of decson makng. Central to the performance of ths task by ANDREA s an nteracton between the amygdala and orbtofrontal cortex (Fg. 1). Much research has mplcated orbtofrontal cortex n the valuaton of stmul (e.g., Rolls, 2; Thorpe, Rolls, & Maddson, 1983), partcularly n lght of ts extensve connectons wth sensory processng areas of the bran. Several recent studes have ndcated an mportant role for orbtofrontal neurons n provdng a sort of common neural currency (Montague & Berns, 22) whch allows for the evaluaton and comparson of fguratve (or even lteral) apples and oranges (Padoa-Schoppa & Assad, 26). Recent studes of the amygdala have challenged ts tradtonal assocaton wth manly aversve stmulus processng, showng nstead actvaton based on the degree to whch stmul are salent or arousng, rather than a specfc valence type (for a revew, see McClure et al., 24). Ths has nspred a renterpretaton of classc results as ndcatng that negatvely apprased events may be n general more emotonally arousng than postve outcomes, perhaps because of a need to alter current behavor n response to aversve feedback. In accord wth research on the role of the amygdala n emotonal attenton (Adolphs et al., 25) and multplcatve scalng observatons for vsual attenton (e.g., Treue,

6 256 A. Ltt et al. / Cogntve Systems Research 9 (28) ), ANDREA performs a multplcatve modulaton by amygdala-encoded emotonal arousal of the valuaton computaton performed n orbtofrontal cortex (Fg. 2). That s, orbtofrontal valuatons are modeled as beng multplcatvely dampened or ntensfed, dependng on whether the ndvdual s n a lowered or heghtened state of affectve arousal, respectvely. Let V represent baselne orbtofrontal stmulus valuaton, based on ntal sensory and cogntve processng and provded as an nput n our model. Takng A to represent amygdala-encoded emotonal arousal, we characterze the output subjectve valuaton S at tme t as SðtÞ ¼ AðtÞ V ðtþ: Thus, ncreased levels of emotonal arousal wll amplfy the subjectve valuaton of stmul by orbtofrontal cortex, whle lower arousal levels lead to valuaton attenuaton. ð1þ As we dscuss n our later account of prospect theory, AN- DREA also ntroduces realstc bologcal constrants mposed by neural frng saturaton that help to explan valuaton behavors observed n humans, an advancement over our earler reward system model (Ltt et al., 26) Surprse as devaton from expectatons Fg. 2 shows that our model generates amygdala actvty upsurges accompanyng changes n the valuaton nput to orbtofrontal cortex, and that these upsurges are valenceasymmetrc: negatve changes n valuaton (losses) produce greater affectve arousal ncreases than equvalent postve changes (gans). Ths neurologcal behavor s produced manly through a modulaton of amygdala-encoded emotonal arousal by a reward predcton error sgnal. Ths s smply the dscrepancy between expected and actual stmu- a 2 OFC nput b AMYG output c OFC output tme Fg. 2. Arousal modulaton of valuaton. (a) The emotonless nput sgnal to orbtofrontal cortex conssts here of postve and negatve valuaton changes of varyng magntude. The vertcal axs can be nterpreted as a sort of neural currency scale: upward steps n the graph thus represent gans, whle steppng down ndcates a loss. Postve/negatve sgn corresponds to appettve/aversve valence. (b) Emotonal arousal reflected n amygdala actvty. Decoded output from spkng neuron populatons (see Appendx B). Upsurges correspond to arousal ncreases concdng wth changes n the externally provded stmulus value sgnal n (a), demonstratng the role such changes play n nfluencng emotonal engagement. (c) Multplcatve modulaton of the actvty presented n (a) by that shown n (b). Emotonal arousal can nduce sgnfcant changes n valuaton from the baselne nput sgnal.

7 A. Ltt et al. / Cogntve Systems Research 9 (28) lus valuaton, and as such represents the effect of the surprsng nature of a stmulus on how emotonally arousng t s. For ths computaton we employ the temporal dfference (TD) model (Sutton & Barto, 1998), due to ts smple mathematcal structure and robust correspondence wth expermental neural actvty observatons (e.g., Schultz, 2). TD computes reward predcton error (E) based on the dfference between the latest reward valuaton and a weghted sum of all prevous rewards (P). Usng our arousal-modulated sgnal S as the nput regardng current stmulus valuaton, ths leads to the modeled recurrent equatons EðtÞ ¼ SðtÞ Pðt 1Þ PðtÞ ¼ Pðt 1Þ þ a EðtÞ; where a s a learnng rate constant between and 1. Ths actvty has typcally been modeled by ncreased mdbran dopamne frng wth postve predcton errors (that s, gettng more than expected) and frng rate depresson for negatve errors (gettng less than expected) (Schultz, 1998, 2; Sur, 22). Whle ths approach seems vald for partcular ranges, recent work has called nto queston the feasblty of mdbran dopamne actng alone to perform ths computaton (Daw et al., 22; Dayan & Ballene, 22). In partcular, such physologcal constrants as low baselne frng rates make t dffcult to envson how actvty depresson could be used to well encode hghly negatve predcton errors, a concern supported by recent expermental fndngs (Bayer & Glmcher, 25). Accordngly, we adopt an nteractng opponent encodng of postve predcton errors by mdbran dopamne and negatve errors by serotonergc neurons n the dorsal raphe nucleus of the branstem. Ths s supported by a varety of expermental studes n humans and other anmals (Deakn, 1983; Evenden & Ryan, 1996; Mobn, Chang, Ho, Bradshaw, & Szabad, 2; Soubré, 1986; Vertes, 1991; for a revew, see Daw et al., 22). By separatng the encodngs of losses and gans, we are able to dstnctly calbrate the modulatory effects of postve and negatve valuaton changes to provde a plausble neural mechansm for loss averson (Appendx C). That s, asymmetres n loss gan valuaton can be modeled va dfferng amygdala senstvty to nputs from dorsal raphe or mdbran areas, perhaps realzed n actual brans through dfferences n specfc neurotransmtter receptor concentratons n the amygdala, or smlar mechansms of connectvty strength varaton (e.g., receptor senstvty dfferences) Increased behavoral salency of negatve outcomes ð2þ ð3þ Valence-asymmetry n emotonal arousal s further strengthened n our model through the actvtes we have assgned to the anteror cngulate and dorsolateral prefrontal cortces, specfcally va a proposed dssmlarty n the nfluences of losses and gans on requred behavoral plannng. Much evdence supports the mportance of dorsolateral prefrontal cortex n the plannng, representaton and selecton of goal-drected behavors (e.g., Owen, 1997), and the anteror cngulate cortex n the detecton of conflcts between current behavor and desred or expected results, nterfacng approprately wth dorsolateral prefrontal n the process (e.g., Bush, Luu, & Posner, 2). We hypothesze an ncreased behavoral salency of negatve reward predcton errors, as such results may ndcate that current behavor needs to be modfed, rather than be smply mantaned or strengthened as a postve error would ndcate. Such a stuaton would ntroduce the attendant cogntve resource requrements of new acton plan formaton and executon n response to the dspleasng outcome, as well as potental envronmental rsks stemmng from alterng current behavor. We model ths ncreased behavoral salency through a correspondng ncrease n emotonal arousal (Appendx C). Thus, feedback from dorsolateral prefrontal cortex and the anteror cngulate to the amygdala n our model further ncreases the affectve nfluence of losses over smlarly szed gans. In combnaton wth the prevously dscussed roles of dopamne and serotonn n nfluencng the amygdala, we arrve at our fnal characterzaton of how emotonal arousal As nfluenced by the salency of unexpected gans and losses, as well as potental behavoral modfcaton costs assocated wth the latter: AðtÞ ¼ A 1 ðtþ þ b DAðtÞ þ c 5-HTðtÞ þ CðtÞ: A 1 represents a degree of emotonal arousal determned by external factors unrelated to reward predcton error or the descrbed dorsolateral cngulate contrbuton. In prevous work we have provded ths as a straghtforward nput sgnal to the model (Ltt et al., 26). We shall descrbe later how ANDREA expands A 1 arousal by ncorporatng pror expectatons regardng valuaton targets, whch allows for a neurobologcal explanaton of decson affect theory (Mellers, Schwartz, Ho, & Rtov, 1997). DA and 5-HT are the opponent encodngs of postve and negatve reward predcton error, respectvely, wth c chosen to be a connecton-strength constant greater than b to smulate an ncreased nfluence of serotonn-encoded losses over dopamne-encoded gans on emotonal state. Fnally, C represents the addtonal costs assocated wth losses that ncrease ther behavoral salency, and hence emotonal mport, as determned by actvty n the anteror cngulate and dorsolateral prefrontal cortces. 4. A neural account of prospect theory Prospect theory, a theoretcal framework for understandng rsky choce developed by Kahneman and Tversky (1979, 1982, 2), has been appled to many preference and choce behavors commonly exhbted by people. The most famous such phenomenon s loss averson, whereby people behave asymmetrcally n ther personal valuatons of objectvely equvalent losses and gans. Central to prospect theory s resoluton of ths and ð4þ

8 258 A. Ltt et al. / Cogntve Systems Research 9 (28) other nconsstences between classcal decson research and actual behavor s a redefned characterzaton of the nature of subjectve evaluatons of decson outcomes. The resultng value functon proposed by the theory has the followng essental characterstcs (Kahneman & Tversky, 1979; Tversky & Kahneman, 1984): () subjectve value s defned on gans and losses that s, devatons from a neutral reference pont rather than on total wealth, as s typcal of expected utlty theory; () the value functon s concave for gans and convex for losses; and () the curve s steeper for losses than gans. Taken together, an asymmetrc sgmod value functon s the well-known result (Fg. 3) Loss averson Neural affectve decson theory, va the ANDREA model, provdes a compellng explanaton of loss averson. The combnaton of arousal modulaton of subjectve valuaton and the ncreased affectve mport of losses produces emotonally nfluenced orbtofrontal valuatons that overweght losses. Ths can be seen n Fg. 2, and even more vvdly n the smplfed smulaton of Fg. 4. The resultng effects on thnkng and behavor would produce the asymmetres n peoples evaluatons of and responses to gans and losses that have been documented and explaned by prospect theory. We turn now to extendng ths neural account of loss averson nto a detaled bologcal explanaton for the specfc shape of prospect theory s sgmod value functon The value functon of prospect theory In developng a neural theory of preference that meshes wth the behavorally nspred prospect theory value functon, the frst step s to dentfy bran regons that should be expected to produce responses correspondng to the Fg. 3. A hypothetcal prospect theory value functon, llustratng subjectve valuaton commonaltes observed n tests of numerous subjects. sorts of behavors montored n psychologcal studes of preference and valuaton. As dscussed earler, t seems natural to look to orbtofrontal cortex as the ste of actvty mappng drectly onto people s subjectve valuatons of gans and losses. It has been mplcated strongly n tasks related to valuaton and comparson of outcomes, events and perceved stmul n general, and we have descrbed a fundamental affectve modulaton of ths encodng that may form the bass of the subjectve nature of ultmate outcome valuaton. The next step s to dentfy features of the ANDREA model that mght explan the specfc nature of the sgmod value functon, as descrbed prevously n terms of three prmary characterstcs (Fg. 3). Feature () of the curve, valuaton n terms of reference pont devaton, dentfes the sort of nput sgnal to be modulated n orbtofrontal cortex. Snce the degree to whch a stmulus s consdered a loss or a gan s a representaton of ts dvergence from a neutral reference, evaluatng such changes n value calls for a step-style nput, smlar to those shown n Fgs. 2 and 4, where the subjectve valuaton of a devaton of sze X wll be determned by the emotonal modulaton produced by an nput valuaton step from to X. For example, to produce orbtofrontal actvty correspondng to the subjectve valuaton of a loss of $2, we measure the emotonally modulated output from orbtofrontal cortex to a step nput that moves from (the reference pont) to our target value ( 2). Leavng asde feature () for a moment, the thrd aspect of the prospect theory value curve, a steeper slope for losses than gans, s smply loss averson (Kahneman & Tversky, 1979). The bologcal account of loss averson we prevously descrbed wll thus serve as a crtcal component of our neural explanaton of the S-curve. The second feature of the sgmod value functon, the levelng-off of loss and gan valuatons at the extremes, requres appeal to addtonal neurologcal mechansms. In partcular, we ntroduce the noton of neural saturaton. Any type of neuron has hard bologcal constrants on how fast t can fre. Each acton potental s followed by a refractory perod of repolarzaton durng whch the neuron cannot fre, and ssues such as cellular respraton requrements and local neurotransmtter depleton ntroduce unavodable lmtatons on spke rates. In the context of neurocomputaton n the NEF (and hence ANDREA), ths means that the range of values that can be encoded by any neural populaton s lmted. Ths nherent restrcton can actually serve an explanatory purpose n the case of the prospect theory value functon. To explan how ANDREA could produce levelng-off valuatons at the extremes, note that we encode values through ncreasng neural spke rates n the encodng populaton as these values become larger (n ether the postve or negatve-drecton). Thus, n attemptng to provde subjectve valuatons for very large gans or losses, neurons n orbtofrontal cortex wll begn to saturate, as they smply cannot fre fast enough to produce lnearly dstnctve affectve responses to ncreasngly large value devatons.

9 A. Ltt et al. / Cogntve Systems Research 9 (28) a OFC nput 2 b AMYG output c OFC output tme Fg. 4. Unbalanced evaluaton of gans and losses. (a) The nput sgnal to orbtofrontal cortex conssts of postve and negatve changes n value of equal magntude. (b) Arousal level modulated by predcton error and lkely behavoral salency of stmul. The loss nduces a much greater arousal ncrease than the equal gan. (c) The outcome of the unevenness dsplayed n (b). Reductons n stmulus valuaton (losses) are dsproportonately amplfed compared to gans. An alternatve encodng of value magntude through actvated populaton sze, rather than the frng levels of a fxed populaton, would seem to deny ths saturaton-based account of dmnshng margnal senstvty. Such a scheme mght be akn to accumulator models that have been appled to numerosty encodng n ntraparetal regons (Rotman, Brannon, & Platt, 27). However, applyng ths approach to valuaton encodng seems less plausble from a neurophysologcal resources perspectve; whle the salent bran areas do have mllons of neurons, all of these would requre drect connectvty to areas nterpretng magntude nformaton n order to mpart the same nformaton as naturally rate-tuned transmtter release by a populaton whch encodes magntude va frng rates. Indeed, accumulator models generally allow for cardnal value encodngs on restrcted nteger scales such as 2-to-32, rather than the potentally arbtrary and quas-analog scale on whch valuatons often le. Fg. 5 llustrates the results of runnng smulatons based on the precedng descrpton. Each data pont at X along the horzontal axes of Fg. 5b and c s the result of measurng a modulated orbtofrontal valuaton output n a smulaton provdng orbtofrontal cortex a step nput from to X, n accord wth the reference-devaton characterzaton of value n prospect theory. As expected, the effects of loss averson are clear, wth the slope dfferental ndcatng a greater affectve mpact of losses over equvalent gans. The 2:1 slope rato observed here for moderate losses and gans mrrors behavoral evdence n the decson lterature (e.g., Kahneman, Knetsch, & Thaler, 1991). Fnally, the concavty features of the value functon have been successfully replcated n these smulatons. Fg. 6 shows the specfc role of neural saturaton n ths regard. Each row n these spke rasters represents an ndvdual orbtofrontal cortex neuron, and each pont represents a sngle acton potental at a specfc pont n tme produced by the neuron n queston. Clearly, equal changes n the sze of a loss or gan do not necessarly produce smlar changes n neural spkng, partcularly as neurons begn to saturate. Ths causes a decreasng dstnctness n frng response at the extremes, whch we propose as the neurologcal bass for the levelng-off n loss and gan valuaton observed n behavoral studes. Overall, the mechansms we have outlned combne to produce a detaled, bologcally plausble neural explanaton of the nature of the value functon descrbed by prospect theory Framng through reference-value manpulaton Understandng how ndvduals respond dfferently dependng on the manner n whch nformaton regardng a stuaton s presented s consdered to be one of the prmary

10 26 A. Ltt et al. / Cogntve Systems Research 9 (28) a $5 loss n = 226 event b $1 loss c event $15 loss n = 344 ( 52%) n = 43 ( 17%) event tme Fg. 6. Orbtofrontal cortex spke rasters. Note the clear dfference n actvty between (a) and (b) (a 52% ncrease n mmedate post-event spkng). Ths s n contrast to the relatve smlarty n spkng n (b) and (c) forced by neural saturaton, as the $5 change moves farther away from the reference-value of. Ths response characterstc forms the bass of our neural explanaton for the concavty features of the prospect theory value functon. Fg. 5. Value functon smulaton results. (a) A typcal prospect theory value functon. (b) Subjectve valuaton outputs from orbtofrontal cortex. Each data pont represents the emotonally modulated valuaton of the loss or gan value chosen along the horzontal axs. (c) Close-up of the central porton of (b), showng the dfferng slopes for loss and gan valuatons. explanatory successes of prospect theory. A famous llustraton of the power of framng s the presentaton of two dfferent choce-sets to subjects regardng the consequences of dfferent plans to handle the outbreak of a dsease expected to kll 6 people (Tversky & Kahneman, 1981, 1986): Problem 1: Program A 2 people wll be saved. Program B 1/3 probablty 6 are saved, 2/3 probablty nobody s saved. Problem 2: Program C 4 people wll de. Program D 1/3 probablty nobody wll de, 2/3 probablty 6 wll de. Faced wth the choce n Problem 1, 72% of subjects chose Program A over B, whereas only 22% of subjects chose Program C over D when faced wth Problem 2. Clearly, though, Programs A and C are objectvely equvalent, as are ther respectve alternatves. The framng of stuatons n terms of losses or gans may thus cause dramatc reversals of preference n decson scenaros.

11 A. Ltt et al. / Cogntve Systems Research 9 (28) The mechansms mplemented n the ANDREA model provde a realstc neural bass for such framng effects. We descrbe means by whch objectvely equvalent outcomes can produce markedly dfferent subjectve valuatons n orbtofrontal cortex, dependng on the manner n whch each s framed. In partcular, note that feature () of the prospect theory value functon, the defnton of subjectve valuaton on devatons from some neutral reference-value, ponts towards an obvous mechansm for the framng of decsons: varaton of the reference value tself. In the dsease example, Problem 1 s framed n terms of lves saved rather than lves lost, whle the reverse s true for Problem 2. Thus, choosng zero reference ponts from whch subjectve valuatons shall devate n each case should lead to dfferent values for each problem. For Problem 1, zero lves saved would ndeed correspond to on a scale measurng the total number of people of our orgnal 6 who are expected to be left alve after the choce of a gven program for combatng the dsease. Crucally, however, the zero lves lost reference pont for Problem 2 would correspond to the value 6 when measured on ths same scale, snce 6 out of 6 people alve ndcates that no lves have been lost, as requred. Thus, n the case of the Program A opton n Problem 1, a subjectve valuaton devaton descrbed as 2 people out of 6 wll be saved represents a postve-drecton devaton from lves saved to 2 lves saved. In contrast, the objectvely equvalent Program C opton n Problem 2, descrbed as 4 people out of 6 wll de, represents a negatve-drecton devaton from lves lost to 4 lves lost, that s, from 6 left alve down to 2 left alve. Note that both devaton construals end at the value 2 on the scale of people stll alve, snce they are objectvely equvalent outcomes. Nevertheless, because of our multplcatve modulaton of valuaton devatons by emotonal arousal, opposte drectons of devaton wll produce subjectve valuatons that are emotonally amplfed n opposte drectons. Fg. 7 llustrates the results of characterzng ths type of framng as a manpulaton of the devaton reference pont. We obtan a subjectve valuaton of orbtofrontal step nput correspondng to Program A that s much more postve than that of a step nput correspondng to Program C, smply because of opposte drectons of emotonal modulaton. Ths would explan the preference reversal that occurs upon swtchng decson frames, as what was seen as a gan n comparson to one reference-value s suddenly evaluated as a loss n comparson to a dfferent referent. Later we wll dscuss framng effects that operate n ways other than varyng reference-values, and how these dfferent sorts of framng can n combnaton explan the observed nteractons between affect, pror expectatons and counterfactual comparsons explored n decson affect theory Predctons Our neurologcal explanaton of prospect theory suggests a range of testable neural-level predctons and a OFC n b OFC out /6 de 2/6 saved Fg. 7. Smulaton results for framng n terms of gans and losses. (a) Objectvely equvalent outcomes (endng up wth 2 people alve) evaluated as devatons from dfferent reference ponts. The thn-lned step nput represents Problem 1/Program A, and the heavy lne Problem 2/ Program C (Tversky & Kahneman, 1981). (b) Opposte drectons of devaton produce opposte drectons of emotonal amplfcaton n subjectve valuaton, leadng to more a postve outlook towards Program A than Program C. hypotheses. Ltt et al. (26) outlne several such predctons n relaton to loss averson and the behavoral nfluence of serotonn. For example, the extent of a partcular ndvdual s hypersenstvty to losses s hypotheszed to be correlated wth the concentraton n the amygdala of a specfc serotonn receptor subtype, whch would nfluence the degree to whch negatve reward predcton errors affect amygdala actvty. As well, degraded connectvty between mdbran dopamne neurons and the raphe serotonn system s predcted to ncrease emotonally nfluenced overvaluaton of both gans and losses, due to mutual attenuaton effects that we have modeled between these systems. Such correlaton between loss and gan senstvty has ndeed been shown n recent work by Tom and colleagues (27). The partcular neural actvty they descrbe suggests the mechansm underlyng ths relatonshp may nvolve mportant addtons to the computatons captured n the ANDREA model, such as the effects of noradrenergc crcuts. Further emprcal nvestgaton of the neural correlates of loss averson can provde more such tests and potental falsfcatons of the relatonshps proposed n ANDREA, although practcal barrers to magng bran stem serotonergc actvty lmt the capacty of the approach of the Tom et al. study n ths respect. Addtonally, ths work and other exploratons of prospect theory on the bran by ths team dd not show any sgnfcant amygdala actvty relevant to loss averson (Tom, Fox, Trepel, & Poldrack, 27; Trepel, Fox, & Poldrack, 25). Besdes the studes we have prevously cted that ndcate an mportant role for the amygdala n emotonal valuaton, sgnfcant amygdala actvty

12 262 A. Ltt et al. / Cogntve Systems Research 9 (28) drectly correspondng to loss averson has contrastngly been drectly observed n other fmri experments (e.g., Weber et al., 27). Such dscrepances and lmtatons mght be resolved by employng alternatve expermental technques; for nstance, t s possble to temporarly dmnsh cerebral serotonn levels by an ngeston of a tryptophan depletng drnk (Cools et al., 25), whch our model suggests would specfcally dmnsh loss averson. Depleton studes of decson-related phenomena hold much promse for testng the valdty of neurocomputatonal models lke ANDREA, as well as advancng our understandng of the bologcal bases of complex behavors and cogntons. Varous other anomalous choce-related behavors arsng from specfc dsconnectons and damage patterns are also descrbed by Ltt et al. (26). The enrched conceptons of decson phenomena provded by neural-level exploraton allow for postulaton of potental nfluences on behavor that would be mssed by studes restrcted to examnng hgher-level psychologcal processng. A straghtforward set of addtonal predctons dervng from our ANDREA smulatons would be expectatons of actvaton of the bran regons descrbed n our model (Fg. 1) to be observed n magng experments nvolvng people performng preference and choce tasks. But a more nterestng explanatory and predctve utlty of the model nvolves the study of ndvdual dfferences n value functons. Whle the general features of the curve have been reproduced by many dfferent experments and n large numbers of people, the specfc functons of dfferent ndvduals are known to vary wdely (e.g., Kahneman & Tversky, 1982). Because our smulatons essentally represent the value functon produced by a sngle bran (to whch we have complete access, as ts desgners and bulders) the model offers bologcal reasons for why and how ndvdual value functons vary, whle preservng some common general features. As dscussed prevously, connectvty strength between dorsal raphe serotonn and the amygdala can nfluence loss averson, and thus the nature of the subjectve valuatons performed n orbtofrontal cortex. In persons wth heghtened serotonn senstvty n the amygdala, we would expect to see a value functon wth an even steeper slope for losses than that of gans. The opposte also holds, n terms of lessened serotonergc nfluence or stronger mdbran dopamnergc connectvty, wth ether of these cases predctve of less dscrepancy n slope between losses and gans (.e., reduced loss averson). The damaged nnervaton between the raphe and mdbran mentoned earler would have the effect of steepenng both the loss and gan portons of the curve, although the slope rato mght be unaffected. These connectvty manpulatons can be understood psychologcally as alterng the extent to whch one s emotonally aroused by losses and gans, wth ths affectve nfluence central to producng subjectve valuatons of these devatons from a reference pont. We also predct that specfc neural saturaton range dfferences between brans may underle ndvdual dfferences n value functon shape, wth more readly saturatng orbtofrontal populatons prone to producng curves that level-off qucker and more markedly. In the NEF, ths saturaton can tself be modeled asymmetrcally around, so we can readly create a neural system of specfc archtecture and connectvty that yelds sgnfcant levelng-off for losses but much less so for gans, or vce versa. In sum, the ANDREA model allows for numerous expermentally testable neural-level predctons regardng prospect theoretc behavor. 5. A neural account of decson affect theory The hedonc nfluence of pror expectatons and counterfactual comparsons on the subjectve valuaton of outcomes s characterzed by the work of Mellers and colleagues on what they call decson affect theory (Mellers & McGraw, 21; Mellers et al., 1997; Mellers, Schwartz, & Rtov, 1999). Its fundamental clam s that evaluaton by an ndvdual of an outcome, event or decson opton s strongly nfluenced by the relatve pleasure t s consdered to provde (Mellers, 2). Ths relatvty derves n part from the effects of counterfactual comparsons, as llustrated by the fndng that Olympc slver medalsts are more lkely to feel dsapponted than bronze medalsts because of generally hgher personal expectatons (McGraw, Mellers, & Tetlock, 25). Another factor s the degree to whch an obtaned outcome s consdered surprsng, wth greater emotonal mpact for unexpected results (ether good or bad) than for expected outcomes. The mathematcal expresson of decson affect theory s R O ¼ J½u O þ dðu O u E Þ ð1 s O ÞŠ (cf. Eq. (1) n McGraw et al., 25). R O s the emotonal feelng assocated wth the obtaned outcome and J s a lnear functon relatng the felt pleasure to a specfc numercal response. u O and u E are the respectve utltes of the obtaned and expected outcomes, and d(u O u E ) s a dsappontment functon that models how the obtaned outcome s compared to the alternatve expected outcome. s O s the subjectvely judged probablty of the obtaned outcome actually occurrng, so the weghtng by the complementary (1 s O ) term models the degree to whch the obtaned outcome was not expected (.e., the subjectve probablty that somethng else would occur). The mportance of emotonal nfluence becomes clear wth the fndng that people wll choose what feels best that s, make decsons n such a manner as to maxmze average postve emotonal experence (Mellers et al., 1997). Thus feelngs about outcomes and choces, and hence the decsons people may be expected to make, are greatly nfluenced by the sze and valence of the dscrepancy between antcpated and actual results. The expected emotonal reacton to ganng $2 wll be vastly dfferent f the pror expectaton s ganng $1, versus the case where the expected yeld s only $1. Indeed, the degree of nfluence of ð5þ

13 A. Ltt et al. / Cogntve Systems Research 9 (28) ths dscrepancy from antcpated results s such that an objectvely worse outcome can sometmes be morepleasurable than one whch s better. Consder the easly magnable experence of feelng happer n stumblng across a 2-dollar bll whle walkng home from work than from recevng an underwhelmng 1% rase the same day, despte the monetary value of the rase beng sgnfcantly hgher than that of the found note. Decson affect theory thus descrbes n a revealng and systematc fashon the nature of certan stuatons n whch less can actually feel lke more. In descrbng a neural bass for the theory suggested by ANDREA, we shall draw upon an ntegraton of the sort of framng dscussed n our examnaton of prospect theory wth the effects of the emotonal-context of nformaton presentaton on valuaton and choce, whch we now explore n depth Framng through drect emotonal arousal nfluence Our earler dscusson of framng effects focused on the type most dscussed n prospect theory, concernng alternatve descrptons of losses and gans. However, framng can affect decsons n other ways, as n the trolley-footbrdge experments of Greene and colleagues (21). These experments are not reference-value manpulatons, and are therefore not explanable by the neural mechansms that we descrbed for prospect theory. The judgment of moralty reversal occurs between two outcomes that are both descrbed as kllng one person to save fve others. The manpulaton nvolved here s not one of reference pont, but rather the personal or mpersonal nature of the specfc acton performed that leads to the descrbed outcome. It s thus dfferng contexts of choce presentaton (.e., the nature of the stuaton story) that produce the change n stuatonal evaluaton, rather than any suggestve presentaton of choces n terms of ether losses or gans. We wll call ths emotonal-context framng, n contrast to the referencevalue framng we dscussed n relaton to prospect theory. We propose that emotonal-context framng n the trolley-footbrdge dlemma occurs through ncreased arousal assocated wth the drect, personalzed acton of pushng a person to ther death, compared to the more detached and mpersonal act of flppng a swtch that wll cause a trolley to dvert from httng fve people towards httng a sngle person. Such an ncrease n emotonal engagement nduces greater amplfcaton of the subjectve evaluaton of causng a death n the personal case, whch would provde a neurologcal bass for the reversal n typcal judgments of the moralty of the actons n queston. fmri experments by Greene and colleagues seem to support ths neural account of the trolley-footbrdge dlemma, showng ncreased amygdala and orbtofrontal actvty n cases of hghly personal characterzatons of morally debatable actons (Greene & Hadt, 22; Greene, Nystrom, Engell, Darley, & Cohen, 24). Fg. 8 llustrates the neural explanaton we have descrbed for the type of framng produced by changng a OFC n b AMYG n c AMYG out d OFC out Fg. 8. Smulaton results for framng that changes emotonal-context. (a) A step nput to orbtofrontal cortex ndcatng a negatve change n value, such as the consderaton of a stuaton n whch one s actons cause the death another person. (b) Two dfferent base arousal nputs to the amygdala, correspondng to dfferng levels of context-produced emotonal engagement. (c) Because of dfferng base arousal levels, arousal upsurges correspondng to the negatve valuaton devaton dffer as well. (d) Hgher context-motvated base arousal leads to greater amplfcaton of the subjectve valuaton change, and thus a belef that the more arousng scenaro s actually worse than the objectvely equvalent but less arousng scenaro. the emotonal-context of the presented nformaton. Just as for reference-value framng, we get dfferent subjectve valuatons for scenaros that have dentcal objectve values, whch would allow for preference reversals to occur when the decson frame s altered. The prmary dfference s that ths mechansm employs a drect manpulaton of emotonal arousal state, whereas our neural bass for the reference-value framng caused emotonal modulaton changes ndrectly though manpulaton of valuaton devaton reference ponts Decson affect theory as an ntegrated framng phenomenon Explanaton of the expermental results examned n decson affect theory requres both reference-value framng and emotonal-context framng. The hedonc mpact of counterfactual comparsons can be produced by settng the reference pont of the valuaton devaton for an obtaned outcome to the value of the unobtaned counterfactual result, whch s exactly reference-value framng.