Bonsai Trees in Your Head: How the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees

Transcription

1 Bonsa Trees n Your Head: How the Pavlovan System Sculpts Goal-Drected Choces by Prunng Decson Trees Quentn J. M. Huys 1,2,3. *, Ner Eshel 4., Elzabeth O Nons 4, Luke Sherdan 4, Peter Dayan 1, Jonathan P. Roser 4 1 Gatsby Computatonal Neuroscence Unt, Unversty College London, London, Unted Kngdom, 2 Wellcome Trust Centre for Neuromagng, Insttute of Neurology, Unversty College London, London, Unted Kngdom, 3 Guy s and St. Thomas NHS Foundaton Trust, London, Unted Kngdom, 4 UCL Insttute of Cogntve Neuroscence, London, Unted Kngdom Abstract When plannng a seres of actons, t s usually nfeasble to consder all potental future sequences; nstead, one must prune the decson tree. Provably optmal prunng s, however, stll computatonally runous and the specfc approxmatons humans employ reman unknown. We desgned a new sequental renforcement-based task and showed that human subjects adopted a smple prunng strategy: durng mental evaluaton of a sequence of choces, they curtaled any further evaluaton of a sequence as soon as they encountered a large loss. Ths prunng strategy was Pavlovan: t was reflexvely evoked by large losses and perssted even when overwhelmngly counterproductve. It was also evdent above and beyond loss averson. We found that the tendency towards Pavlovan prunng was selectvely predcted by the degree to whch subjects exhbted sub-clncal mood dsturbance, n accordance wth theores that ascrbe Pavlovan behavoural nhbton, va serotonn, a role n mood dsorders. We conclude that Pavlovan behavoural nhbton shapes hghly flexble, goaldrected choces n a manner that may be mportant for theores of decson-makng n mood dsorders. Ctaton: Huys QJM, Eshel N, O Nons E, Sherdan L, Dayan P, et al. (2012) Bonsa Trees n Your Head: How the Pavlovan System Sculpts Goal-Drected Choces by Prunng Decson Trees. PLoS Comput Bol 8(3): e do: /journal.pcb Edtor: Laurence T. Maloney, New York Unversty, Unted States of Amerca Receved September 14, 2011; Accepted January 18, 2012; Publshed March 8, 2012 Copyrght: ß 2012 Huys et al. Ths s an open-access artcle dstrbuted under the terms of the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal author and source are credted. Fundng: NE was funded by the Marshall Commsson and PD by the Gatsby Chartable Foundaton. The funders had no role n study desgn, data collecton and analyss, decson to publsh, or preparaton of the manuscrpt. Competng Interests: The authors have declared that no competng nterests exst. * E-mal: qhuys@cantab.net Current address: Harvard Medcal School, Harvard Unversty, Boston, Massachusetts, Unted States of Amerca. These authors contrbuted equally to ths work. Introducton Most plannng problems faced by humans cannot be solved by evaluatng all potental sequences of choces explctly, because the number of possble sequences from whch to choose grows exponentally wth the sequence length. Consder chess: for each of the thrty-odd moves avalable to you, your opponent chooses among an equal number. Lookng d moves ahead demands consderaton of 30 d sequences. Ostensbly trval everyday tasks, rangng from plannng a route to preparng a meal, present the same fundamental computatonal dlemma. Ther computatonal cost defeats brute force approaches. These problems have to be solved by prunng the underlyng decson tree,.e.by excsng poor decson sub-trees from consderaton and spendng lmted cogntve resources evaluatng whch of the good optons wll prove the best, not whch of the bad ones are the worst. There exst algorthmc solutons that gnore branches of a decson tree that are guaranteed to be worse than those already evaluated [1 3]. However, these approaches are stll computatonally costly and rely on nformaton rarely avalable. Everyday problems such as navgaton or cookng may therefore force precson to be traded for speed; and the algorthmc guarantees to be replaced wth powerful but approxmate and potentally suboptmal heurstcs. Consder the decson tree n Fgure 1A, nvolvng a sequence of three bnary choces. Optmal choce nvolves evaluatng 2 3 ~8 sequences. The smple heurstc of curtalng evaluaton of all sequences every tme a large loss ({140) s encountered excses the left-hand sub-tree, nearly halvng the computatonal load (Fgure 1B). We term ths heurstc prunng a Pavlovan response because t s nvoked, as an mmedate consequence of encounterng the large loss, when searchng the tree n one s mnd. It s a reflexve response evoked by a valence, here negatve, n a manner akn to that n whch stmul predctng aversve events can suppress unrelated ongong motor actvty [4,5]. A further characterstc feature of respondng under Pavlovan control s that such respondng perssts despte beng suboptmal [6]: pgeons, for nstance, contnue peckng a lght that predcts food, even when the food s omtted on every tral on whch they peck the lght [7,8]. Whle rewards tend to evoke approach, punshments appear partcularly effcent at evokng behavoral nhbton [9,10], possbly va a serotonergc mechansm [11 15]. Here, we wll ascertan whether prunng decson trees when encounterng losses may be one nstance of Pavlovan behavoural nhbton. We wll do so by leveragng the nsenstvty of Pavlovan responses to ther ultmate consequences. We developed a sequental, goal-drected decson-makng task n whch subjects were asked to plan ahead (c.f. [16]). On each tral, subjects started from a random state and generated a sequence of 2 8 choces to maxmze ther net ncome (Fgure 2A,B). In the frst of three expermental groups the PLoS Computatonal Bology 1 March 2012 Volume 8 Issue 3 e

2 Author Summary Plannng s trcky because choces we make now affect future choces, and future choces and outcomes should gude current choces. Because there are exponentally many combnatons of future choces and actons, bruteforce approaches that consder all possble combnatons work only for trvally small problems. Here, we descrbe how humans use a smple Pavlovan strategy to cut an expandng decson tree down to a computatonally manageable sze. We fnd that humans use ths strategy even when t s dsadvantageous, and that the tendency to use t s related to mld depressve symptoms. The fndngs, we suggest, can be nterpreted wthn a theoretcal framework whch relates Pavlovan behavoural nhbton to serotonn and mood dsorders. heurstc of prunng sub-trees when encounterng large punshments ncurred no extra cost (Fgure 2C). Subjects here pruned extensvely: they tended to gnore subtrees lyng beyond large losses. Ths allevated the computatonal load they faced, but dd not ncur any costs n terms of outcomes because there was always an equally good sequence whch avoded large losses (see Fgure 2C). In contrast, n the second and thrd expermental groups subjects ncurred ncreasngly large costs for ths prunng strategy (Fgure 2D,E); yet, they contnued to deploy t. That s, the tendency to excse subtrees lyng below punshments perssted even when counterproductve n terms of outcomes. Ths persstence suggests that prunng was reflexvely evoked n response to punshments and relatvely nsenstve to the ultmate outcomes. Computatonal models whch accounted for close to 90% of choces verfed that the nature of prunng corresponded to the Pavlovan reflexve account n detal. These results reveal a novel type of nteracton between computatonally separate decson makng systems, wth the Pavlovan behavoural nhbton system workng as a crutch for the powerful, yet computatonally challenged, goaldrected system. Furthermore, the extent to whch subjects pruned correlated wth sub-clncal depressve symptoms. We nterpret ths n the lght of a theoretcal model [17] on the nvolvement of serotonn n both behavoural nhbton [14,15] and depresson. Results Fgure 3 shows representatve decson paths. Fgure 3A shows the decson tree subjects faced when startng from state 3 and asked to make a 3-step decson. In the 2140 group, there are two equally good choce sequences n ths stuaton: ether through states (wth returns {20z20{20~{20 net) or through states (wth returns {140{20z140~{20 net). When gven the choce, subjects relably chose the path avodng the large loss (even though ths meant also avodng the equally large gan). However, Fgure 3B shows that subjects could overcome the reflexve avodance of the large loss. In ths stuaton, because the large loss s much smaller ({70), t s best to transton through t to reap the even larger reward (z140) behnd t. Ths same behavour was less frequently observed n larger trees when large losses happened deeper n the tree. Fgure 3C shows the tree of depth 5 startng from state 1. The leftmost three-move subtree, hghlghted by the box, s dentcal to the tree startng from state 3 wth depth 3. Although t s stll optmal to transton through the large loss, subjects tended to avoded ths transton and thereby mssed potental gans. Note that n 3C, subjects also avoded an alternatve optmal path where the large loss agan dd not occur mmedately. Fgure 3D F shows the number of tmes subjects chose the optmal sequence through the decson tree, separatng out stuatons when ths optmal choce nvolved a transton through a large loss and when t dd not. Subjects were worse at choosng optmal sequences when the depth was greater. Subjects were also less wllng to choose optmal sequences nvolvng transtons through large losses (shown n blue) than those that dd not (shown n green). Ths appeared to be the case more n the group 2140 than the two other groups. However, ths statstc s dffcult to nterpret because n ths group there was always an optmal sequence whch avoded the large loss. Nevertheless, we separated the blue traces nto those cases where large losses appeared early or deep n the tree. For sequences of length 4 or 5, subjects were more lkely to choose the optmal sequence f the loss appeared n the frst rather than n the second half of the sequence (t-tests, p~0:0005 and p~0:038 respectvely). At depth of 6 or more there was no dfference, but the number of these events was small, lmtng the power. Gven these patterns n the data, we consdered that subjects made goal-drected decsons [18] by evaluatng decson paths sequentally. We drectly tested the hypothess whether they would avod paths nvolvng losses by termnatng ths sequental evaluaton when encounterng large losses. That s, n Fgure 3C, do subjects neglect the large reward behnd the large loss because they dd not even consder lookng past the large loss? Important alternatve accounts (whch the analyses so far do not fully address) Fgure 1. Decson tree. A: A typcal decson tree. A sequence of choces between U (left, green) and I (rght, orange) s made to maxmze the total amount earned over the entre sequence of choces. Two sequences yeld the maxmal total outcome of 220 (three tmes U; or I then twce U). Fndng the optmal choce n a goal-drected manner requres evaluatng all 8 sequences of three moves each. B: Prunng a decson tree at the large negatve outcome. In ths smple case, prunng would stll favour one of the two optmal sequences (yeldng 220), yet cut the computatonal cost by nearly half. do: /journal.pcb g001 PLoS Computatonal Bology 2 March 2012 Volume 8 Issue 3 e

3 Fgure 2. Task descrpton. A: Task as seen by subjects. Subjects used two buttons on the keyboard ( U and I ) to navgate between sx envronmental states, depcted as boxes on a computer screen. From each state, subjects could move to exactly two other states. Each of these was assocated wth a partcular renforcement. The current state was hghlghted n whte, and the requred sequence length dsplayed centrally. Renforcements avalable from each state were dsplayed symbolcally below the state, e.g. zz for the large reward. B: Determnstc task transton matrx. Each button resulted n one of two determnstc transtons from each state. For example, f the partcpant began n state 6, pressng U would lead to state 3, whereas pressng I would lead to state 1. The transtons n red yelded large punshments. These (and only these) dffered between three groups of subjects (2140, 2100 or 270). Note that the decson trees n Fgure 1A,B correspond to a depth 3 search startng from state 3. C E: Effect of prunng on values of optmal choces. Each square n each panel analyses choces from one state when a certan number of choces remans to be taken. The color shows the dfference n earnngs between two choce sequences: the best choce sequence wth prunng and the best choce sequence wthout prunng. In terms of net earnngs, prunng s never advantageous (pruned values are never better than the optmal lookahead values); but prunng does not always result n losses (whte areas). It s most dsadvantageous n the 270 group, and t s never dsadvantageous n the 2140 group because there s always an equally good alternatve choce sequence whch avods transtons through large losses. do: /journal.pcb g002 are a smple nablty to look so far ahead n ths task ( dscountng ), an overweghtng of losses relatve to rewards ( loss averson ), and nterference by other, non goal-drected, decson makng strateges ( condtoned attracton & repulson ). We assessed whether subjects decson and nference strateges showed evdence of prunng by fttng a seres of ncreasngly complex models assessng all these factors explctly and jontly. Ths allowed a quanttatve comparson of the extent to whch the varous hypotheses emboded by the models were able to account for the data. Decson makng structure The frst model Look-ahead emboded full tree evaluaton, wthout prunng. It assumed that, at each stage, subjects evaluated the decson tree all the way to the end. That s, for an epsode of length d, subjects would consder all 2 d possble sequences, and choose among them wth probabltes assocated monotoncally wth ther values. Ths model ascrbed the hgher acton value to the subjects actual choces a total of 77% of the tme (fracton of choces predcted), whch s sgnfcantly better than chance (fxed effect bnomal pv10 {40 ). The gray lnes n Fgure 4A separate ths by group and sequence length. They show that subjects n all three groups chose the acton dentfed by the full look-ahead model more often than chance, even for some very deep searches. Fgure 4B shows the predctve probablty,.e. the probablty afforded to choces by the model. Ths s nfluenced by both the fracton of choces predcted correctly and the certanty wth whch they were predcted and took on the value 0.71, agan dfferent from chance (fxed effect bnomal pv10 {40 ). These results, partcularly when consdered wth the fact that on half the trals subjects were forced to choose the entre sequence before makng any move n the tree, ndcate that they both understood the task structure and used t n a goal-drected manner by searchng the decson tree. In order to drectly test hypotheses pertanng to prunng of decson trees, we ftted two addtonal models to the data. Model Dscount attempted to capture subjects lkely reluctance to look ahead fully and evaluate all sequences (up to 2 8 ~256). Rather, tree search was assumed to termnate wth probablty c at each depth, substtutng the value 0 for the remanng subtree. In essence, ths parameter models subjects general tendency not to plan ahead. Fgure 4B shows that ths model predcted choces better. However, snce an mproved ft s expected from a more complex model, we performed Bayesan model comparson, ntegratng out all ndvdual-level parameters, and penalzng more complex models at the group level (see Methods). Fgure 4C shows that fttng ths extra parameter resulted n a more parsmonous model. Note that ths goal-drected model also PLoS Computatonal Bology 3 March 2012 Volume 8 Issue 3 e

4 Fgure 3. Choce sequences. Example decson trees of varyng depth startng from states 1 or 3. The wdths of the sold lnes are proportonal to the frequences wth whch partcular paths were chosen (aggregated across all subjects). Yellow backgrounds denote optmal paths (note that there can be multple optmal paths). Colours red, black, green and blue denote transtons wth renforcements of {X,{20,z20 and z140 respectvely. Dashed lnes denote parts of the decson tree that were never vsted. Vsted states are shown n small gray numbers where space allows. A: Subjects avod transtons through large losses. In the {140 condton, ths s not assocated wth an overall loss. B: In the {70 condton, where large rewards lurk behnd the {70 losses, subjects can overcome ther reluctance to transton through large losses and can follow the optmal path through an early large loss. C: However, they do ths only f the tree s small and thus does not requre prunng. Subjects fal to follow the optmal path through the same subtree as n B (ndcated by a black box) f t occurs deeper n the tree,.e. n a stuaton where computatonal demands are hgh. D,E,F Fracton of tmes subjects n each group chose the optmal sequence, deduced by lookng all the way to the end of the tree. Green shows subjects choces when the optmal sequence dd not contan a large loss; blue shows subjects choces when the optmal sequence dd contan a large loss. Coloured areas show 95% confdence ntervals, and dashed lnes predctons from the model Prunng & Learned (see below). do: /journal.pcb g003 vastly outperformed a habtual model of choce (SARSA; [19]) n whch subjects are assumed to update acton propenstes n a model-free, teratve manner (BIC nt mprovement of 314). The thrd model, Prunng, s central to the hypothess we seek to test here. Ths model separated subjects global tendency to curtal the tree search (captured by the c parameter of model dscount ) nto two separate quanttes captured by ndependent parameters: a general prunng parameter c G, and a specfc prunng parameter c S. The latter appled to transtons mmedately after large punshments (red 2X n Fgure 2B), whle the former appled to all other transtons. If subjects were ndeed more lkely to termnate ther tree search after transtons resultng n large punshments, then a model that separates dscountng nto two separate prunng parameters should provde a better account of the data. Agan, we appled Bayesan model comparson and found strong evdence for such a separaton (Fgure 4C). The fourth model added an mmedate Pavlovan nfluence on choce. The need for ths can be seen by comparng the observed and predcted transton (acton) probabltes at a key stage n the task. Fgure 4D shows the probablty that subjects moved from state 6 to state 1 when they had two or more choces left. Through ths move, subjects would have the opportunty to reap the large reward of z140 (see Fgure 2B), by frst sufferng the small loss of 220. Subjects duly chose to move to state 1 on w90% of these occasons n all three groups. Ths was well matched by the model Prunng. However, when subjects only had a sngle choce left n state 6, t would no longer be optmal to move to state 1, snce there would be no opportunty to gan the large reward afterwards. Instead, the optmal choce would be to move to state 3, at a gan of 20. Despte ths, on about 40% of such trals, subjects were attracted to state 1 (Fgure 4E). Ths was not predcted by the prunng model: pared t-tests showed sgnfcant dfferences between emprcal and predcted choce probabltes for each of the three groups: p~0:026, t 11 ~{2:57; p~0:040, t 14 ~{2:27; and p~0:0005, t 14 ~{3:10, for groups 270, 2100 and 2140 respectvely. Three subjects n group 270 and one subject n group 2100 were never exposed to depth 1 sequences n state 6. To accommodate ths characterstc of the behavor, we added a further, Learned Pavlovan component to the model, accountng for the condtoned attracton (or repulson) to states that accrues PLoS Computatonal Bology 4 March 2012 Volume 8 Issue 3 e

5 Fgure 4. Model performance and comparson. A: Fracton of choces predcted by the model as a functon of the number of choces remanng. For bars 3 choces to go, for nstance, t shows the fracton of tmes the model assgned hgher Q value to the subject s choce n all stuatons where three choces remaned (.e. bar 3 n these plots encompasses all three panels n Fgure 3A C). These are predctons only n the sense that the model predcts choce t based on hstory up to t{1. The gray lne shows ths statstc for the full look-ahead model, and the blue bars for the most parsmonous model ( Prunng and Learned ). B: Mean predctve probabltes,.e. lkelhood afforded to choces on tral t gven learned values up to tral t{1. C: Model comparson based on ntegrated Bayesan Informaton Crteron (BIC nt ) scores. The lower the BIC nt score, the more parsmonous the model ft. For gudance, some lkelhood ratos are dsplayed explctly, both at the group level (fxed effect) and at the ndvdual level (random effect). Our man gude s the group-level (fxed effect). The red star ndcates the most parsmonous model. D,E: Transton probablty from state 6 to state 1 (whch ncurs a 220 loss) when a subsequent move to state 2 s possble (D; at least two moves reman) or not (E; when t s the only remanng move). Note that subjects dsadvantageous approach behavor n E (dark gray bar) s only well accommodated by a model that ncorporates the extra Learned Pavlovan parameter. F: Decson tree of depth 4 from startng state 3. See Fgure 3 for colour code. Subjects prefer (wdth of lne) the optmal (yellow) path wth an early transton through a large loss (red) to an equally optmal path wth a late transton through a large loss. G: Phase plane analyss of specfc and general prunng. Parameter values for whch the left optmal yellow path n panel F s assgned a greater expected value than the rght optmal path are below the blue lne. Combnatons that are also consstent wth the noton of prunng c S wc G are shown n green. The red dot shows parameters nferred for present data (c.f. Fgure 6). Throughout, errorbars ndcate one standard error of the mean (red) and the 95% confdence ntervals (green). do: /journal.pcb g004 wth experence. Ths captured an mmedate attracton towards future states that, on average (but gnorng the remanng sequence length on a partcular tral), were experenced as rewardng; and repulson from states that were, on average, assocated wth more punshment (see Methods for detals). Fgure 4B,C show that ths model (Prunng and Learned) provded the most parsmonous account of the data despte two addtonal parameters, and Fgures 4D E show that the addton of the Learned parameters allowed the model to capture more fathfully the transton probabltes out of state 6. The blue bars n Fgure 4A dsplay the probablty that ths model chose the same acton as subjects (correctly predctng 91% of choces). The model s predcted transton probabltes were hghly correlated wth the emprcal choce probabltes n every sngle state (all pv:0005). Further, we consdered the possblty that the Learned Pavlovan values mght play the addtonal role of substtutng for the utltes of parts of a search tree that had been truncated by general or specfc prunng. However, ths dd not mprove parsmony. We have so far neglected any possble dfferences between the groups wth dfferent large losses. Fgures 3D F mght suggest more prunng n group 2140 than n the other two groups (as the probablty of choosng optmal full lookahead sequences contanng a large loss s mnmal n group 2140). We therefore ftted separate models to the three groups. Fgure 4B shows that the PLoS Computatonal Bology 5 March 2012 Volume 8 Issue 3 e

6 ncrease n the model flexblty due to separate pror parameters for each group ( Prunng & Learned (separate) ) faled to mprove the predctve probablty, ncreased the BIC nt score (Fgure 4C), and hence represents a loss of parsmony. Returnng to Fgure 3D F, we plotted the predctons of model Prunng & Learned for each of the three groups, and found that ths model was able to capture the very extensve avodance of optmal full lookahead sequences ncludng large losses n group 2140, and yet show a gradual declne n the other two groups. The qualtatve dfference between group 2140 and the two other groups n Fgure 3D F s also mportant because t speaks to the goal-drected nature of prunng. Prunng s only counterproductve n groups 270 and The apparent reducton n prunng suggested by the reduced avodance of optmal sequences nvolvng large losses n groups 270 and 2100 (Fgure 3E,F) could suggest that the extent of prunng depends on how adaptve t s, whch would argue aganst a reflexve, Pavlovan mechansm. It s thus mportant that model Prunng & Learned could capture these qualtatve dfferences wthout recurrence to such a goaldrected, clever, prunng. It shows that these dfferences were nstead due to the dfferent reward structures (270 s not as aversve as 2140). Fnally, we return to the decson tree n Fgure 3B. Ths would prma face seem nconsstent wth the noton of prunng, as subjects happly transton through a large loss at the very begnnng of the decson sequence. Fgure 4F shows a dfferent facet of ths. Startng from the state 3 agan, subjects n group 270 choose the optmal path that goes through the large loss straght away even though there s an optmal alternatve n whch they do not have to transton through the large loss so early. In fact, n the model, the relatve mpact of general and specfc prunng factors nteracts wth the precse renforcement sequence, and hence wth the depth at whch each renforcement s obtaned. More specfcally, let us neglect the entre tree other than the two optmal (yellow) sequences the subjects actually took, and let C G ~(1{c G ); C S ~(1{c S );. The value of the left sequence then equals {70{20C S z140c S C G {20C S C 2 G. A smlar, thrd-order polynomal n combnatons of c G and c S descrbes the value of the rght path, and ndeed ther dfference. The blue lne n Fgure 4G shows, for each value of c G, what value of c S would result n the left and rght sequences havng the same value. The combnatons of c S and c G for whch the chosen left path (wth the early transton through the large loss) has a hgher total value turn out to le below ths blue lne. In addton, prunng wll only be more pronounced after large losses f c S s larger than c G. The overlap between these two requrements s shown n green, and the group means for c G and c S are shown by the red dot. Thus, because the effects of general and specfc prunng nteract wth depth, the reflexve, but probablstc, prunng n the model can lead to the pattern seen n Fgure 4G, whereby subjects transton through large losses close to the root of the decson tree, but avod dong so deeper n the tree. Put smply, fxed, reflexve Pavlovan prunng n these partcular sequences of renforcements has dfferental effects deep n the tree. In these cases, t matches the ntuton that t s the explodng computatonal demands whch mandate approxmatons. However, ths s not a necessary consequence of the model formulaton and would not hold for all sequences. Loss averson An alternatve to the prunng account s the noton of loss averson, whereby a loss of a gven amount s more aversve than the gan of an equal amount s appettve. Consder the followng sequence of returns: ½{20,{100,140Š wth an overall return of z20. The prunng account above would assgn t a low value because the large termnal gan s neglected. An alternatve manner by whch subjects may assgn ths sequence a low value s to ncrease how aversve a vew they take of large losses. In ths latter account, subjects would sum over the entre sequence, but overwegh large losses, resultng n an equally low value for the entre sequence. To dstngush loss averson from prunng, we ft several addtonal models. Model Loss s equal to model Look-ahead n that t assumes that subjects evaluate the entre tree. It dffers, n that t nfers, for every subject, what effectve weght they assgned each renforcement. In the above example, for the overall sequence to be as subjectvely bad as f the renforcement behnd t had been neglected, the 2100 renforcement could be ncreased to an effectve value of By tself, ths dd not provde a parsmonous account of the data, as model Loss performed poorly (Fgure 5A). We augmented model Loss n the same manner as the orgnal model by allowng for dscountng and for specfc prunng. There was evdence for prunng even when renforcement senstvtes were allowed to vary separately,.e. even after accountng for any loss averson (cf. models Dscount & Loss and Prunng & Loss, Fgure 5A). Furthermore, addng loss averson to the prevous best model dd not mprove parsmony (cf. models Prunng & Learned vs Loss & Prunng & Learned ). Fnally, the Pavlovan condtoned approach also provded a more parsmonous account than loss averson (cf Prunng & Learned vs Prunng & Loss ). Replacng the four separate parameters n the Loss model wth two slope parameters to reduce the dsadvantage ncurred due to the hgher number of parameters does not alter these conclusons (data not shown). Fnally, the screen subjects saw (Fgure 2A) only showed four symbols: ++, +, 2 and 22.Its thus concevable that subjects treated a ++ as twce as valuable as a +, and smlarly for losses. A model that forced renforcements to obey these relatonshps dd not mprove parsmony (data not shown). The nferred renforcement senstvtes from model Prunng & Loss are shown n Fgure 5B. Comparng the nferred senstvtes to the largest rewards and punshments showed that subjects dd overvalue punshments (treatng them approxmately 1.4 tmes as aversve as an equal-szed reward was appettve; Fgure 5C), consstent wth prevous studes [20]. In concluson, there s decsve evdence for specfc Pavlovan prunng of decson trees above and beyond any contrbuton of loss averson. Prunng estmates We next examned the parameter estmates from the most parsmonous model ( Prunng & Learned ). If subjects were ndeed more lkely to termnate the tree search after large punshments, and thus forfet any rewards lurkng behnd them, then the specfc prunng probablty should exceed the general prunng probablty. Fgure 6A shows the specfc and general prunng parameters c G and c S for every subject. To test for the dfference we modfed the parametrzaton of the model. Rather than nferrng specfc and general prunng separately, we nferred the general prunng parameter and an addtonal specfc prunng boost, whch s equvalent to nferrng the dfference between specfc and general prunng. Ths dfference s plotted n Fgure 6B for the groups separately, though the reader s remnded that the model comparsons above dd not reveal group dfferences (Fgure 4C). The posteror probablty of no dfference between c S and c G was 4:46 10 {7. The parsmony of separate prors was tested earler (see Fgure 4C), showng that specfc prunng c S dd not dffer between groups. Ths s n spte of the fact that prunng n the groups 270 and 2100 s costly, but not n the 2140 group PLoS Computatonal Bology 6 March 2012 Volume 8 Issue 3 e

7 Fgure 5. Prunng exsts above and beyond any loss averson. A: Loss averson model comparson BIC nt scores. Red star ndcates most parsmonous model. The numbers by the bars show model lkelhood ratos of nterest at the group level, and below them at the mean ndvdual level. Prunng adds parsmony to the model even after accountng for loss averson (cf. Dscount & Loss vs Prunng & Loss ), whle loss averson does not ncrease parsmony when added to the best prevous model ( Prunng & Learned vs Loss & Prune & Learned ). B: Separate nference of all renforcement senstvtes from best loss averson model. C: Absolute rato of nferred senstvty to maxmal punshment (270, 2100 or 2140) and nferred senstvty to maxmal reward (always +140). Subjects are 1.4 tmes more senstve to punshments than to rewards. do: /journal.pcb g005 (Fgure 2C). The fact that prunng contnues even when dsadvantageous s evdence for a smple and nflexble prunng strategy whch neglects events occurrng after large losses when computatonal demands are hgh. Fgure 6C shows the cost of prunng n terms of the loss of ncome durng epsodes when the optmal choce sequence would have nvolved a transton through a large punshment. These results suggest that prunng s a Pavlovan response n the sense that t s not goal-drected and not adaptve to the task demands, but s rather an nflexble strategy reflexvely appled upon encounterng punshments. Psychometrc correlates We next tested two a pror predctons that relate the model parameters to psychometrc measurements. Based on pror modellng work [17], we hypotheszed that the tendency to employ the smple prunng strategy should correlate wth psychometrc measures related to depresson and anxety,.e. wth the BDI score and NEO neurotcsm. We also expected to replcate pror fndngs whereby the reward senstvty parameter b should be negatvely correlated wth BDI and NEO neurotcsm [21 24]. Because parameters for dfferent subjects were estmated Fgure 6. Prunng parameters. A: Prunng parameter estmates specfc and general prunng parameters are shown separately for each group. Specfc prunng exceeded general prunng across subjects, but there was no man effect of group and no nteracton. The dfference between parameter types was sgnfcant n all three groups, wth specfc exceedng general prunng for 14/15, 12/16 and 14/15 subjects n the 270, 2100 and 2140 groups respectvely. Blue bars show specfc prunng parameters (c S ) and red bars general prunng parameters (c G ). Black dots show the estmates for each subject. Gray lnes show the uncertanty (square root of second moment around the parameter) for each estmate. B: Equvalent parametrzaton of the most parsmonous model to nfer dfferences between prunng and dscount factors drectly. For all three groups, the dfference s sgnfcantly postve. C: Income lost due to prunng. On trals on whch the optmal sequence led through large punshments, subjects lost more ncome the more counterproductve prunng was (loss n group 270wloss n group 2100wloss n group 2140). Each bar shows the total ncome subjects lost because they avoded transtons through large losses. Throughout, the bars show the group means, wth one standard error of the mean n red and the 95% confdence nterval n green. do: /journal.pcb g006 PLoS Computatonal Bology 7 March 2012 Volume 8 Issue 3 e

8 wth varyng degrees of accuracy (see ndvdual gray error bars n Fgure 6), our prmary analyss was a multple regresson model n whch the nfluence of each subject s data was weghted accordng to how accurately ther parameters were estmated (see Methods). We found that BDI was postvely correlated wth the specfc prunng parameter c S (t 31 ~2:58, p corrected ~0:03, R 2 weghted ~0:27). Furthermore, ths effect was specfc n that there was no such correlaton wth general prunng c G. There was also a negatve correlaton between BDI score and reward senstvty b, although ths dd not survve correcton for multple comparsons (t 31 ~{2:28, p corrected ~0:059, R 2 weghted ~0:12). The regresson coeffcents for the BDI score are shown n Fgure 7A. Notably, these correlatons arose after correctng for age, gender, verbal IQ, workng memory performance and all other NEO measures of personalty. Thus, as predcted, subjects wth more subclncal features of depresson were more lkely to curtal ther search specfcally after large punshment. However, aganst our hypothess, we dd not dentfy any sgnfcant correlatons wth NEO neurotcsm. Fnally, we examned correlatons between all parameters and all questonnare measures n the same framework. We found a postve correlaton between NEO agreeableness and the weght of the Learned Pavlovan nfluence v whch survved full correcton for 60 comparsons t 31 ~4:07, p corrected ~0:018. Dscusson We employed a Bayesan model-fttng approach to nvestgate how Pavlovan choces mght shape goal-drected decson makng. Our full model was able to account for a hgh percentage of subjects choces, allowng us to draw strong conclusons about the lkely forces governng ther behavor. Influences were deemed Pavlovan when they were evoked n a fxed and nflexble manner n response to an outcome or a stmulus value, and goal-drected when senstve to the ultmate, possbly dstant, result of the choce [25]. Partcpants exhbted two hghly sgnfcant Pavlovan nfluences. Frst, subjects pruned to a very substantal degree. Whle part of ths prunng was valence ndependent and hence not Pavlovan (parameter c G n the model), and can be seen as a natural, f suboptmal, response to the exponentally explodng complexty of complete search n the model (rangng from 2 to 256 sequences), subjects also showed a substantal ncrease n ther propensty to prune n the face of a large negatve outcome (parameter c S n the model). Importantly, they dd so even at the expense of a substantal net loss n reward. It was strkng that subjects were no less lkely to prune (Fgure 2C D) even when we rendered t ncreasngly dsadvantageous (movng from group 2140 to group 270),. The second, Learned, Pavlovan nfluence was assocated wth the learned attractveness of prevously rewarded states. In our task, states could have been assocated wth large rewards on past trals, but lack the potental to lead to reward (or ndeed punshment) on a gven tral, because nsuffcent choces remaned (Fgure 4E). Subjects were sgnfcantly seduced by the effect of these past rewards (or repulsed by punshments), agan n a way that was counterproductve to optmal control. Note that by ncludng ths second Pavlovan nfluence, we could be sure that the prunng descrbed above was a pure nfluence on goal-based evaluaton, and was not corrupted by an ntrnsc repulson to the punshment (whch would have been ascrbed to ths second, Pavlovan, nfluence). The Loss models do suggest that subjects were more senstve to punshments than rewards (Fgure 5C). However, ths dd not explan away prunng. Also, f the prunng we observed was just a sgnature of loss averson, one would have expected the extent of prunng not to be the same across groups. Loss averson s a specfc phenomenon n behavoural economcs, whereby subjects are more strongly opposed to a gven probablty of losng a certan amount than to wnnng that same amount [26]. To the extent to whch loss averson can be descrbed as an nflexble, reactve, response to an aversve stmulus, t may represent a thrd nstance of Pavlovan responses to losses nterferng wth goal-drected decsons n ths task [27]. Next, subjects could transton through losses early on n the tree, but were more reluctant to do so when they appeared deeper n the tree. Pavlovan prunng thus appeared to have a partcularly strong effect deeper n the tree. Although ths makes ntutve Fgure 7. Psychometrc correlates. A: Subclncal depresson scores (Beck Depresson Inventory, BDI, range 0 15) correlated postvely wth specfc prunng (c S ), and negatvely wth senstvty to the renforcers (b). Each bar shows a weghted lnear regresson coeffcent. Red error bars show one standard error of the mean estmate, and green errorbars the Bonferron corrected 95% confdence nterval. ~p uncorrected v:05, red dot ~p Bonferroncorr v:05. B,C: Weghted scatter plots of psychometrc scores aganst parameters after orthogonalzaton. do: /journal.pcb g007 PLoS Computatonal Bology 8 March 2012 Volume 8 Issue 3 e

9 sense, t s not a feature explctly bult nto the models. Fgure 4G shows that ths can arse from the nteracton of the partcular sequence of renforcements (and thus renforcement depth) and the prunng and dscount factors. Although ths s not necessarly always the case, the fact that our best-performng model accounted so well for subjects choces (Fgure 4A) suggests that t was a suffcent mechansm for the partcular set of renforcement sequences encountered here. Fnally, although our sample of healthy volunteers, whch was thoroughly screened for past pathology, reported only very mld depressve symptoms (wth mean BDI scores of 3:7, range 0{15), we found that subjects propensty to prune specfcally n the face of negatve valence was postvely correlated wth self-reported sub-clncal depressve symptoms. Prunng, serotonn and depresson Our work was nspred by a prevous modellng paper [17], whch used the concept of behavoural nhbton to unfy two dvergent and contradctory fndngs on the relatonshp between serotonn and depresson. On the one hand, drugs that putatvely ncrease serotonn by nhbtng the serotonn reuptake mechansm are effectve for both acutely treatng [28], and preventng relapse of [29], depresson. On the other hand, a genetc polymorphsm that downregulates the very same serotonn reuptake transporter, thus actng n the same drecton as the drugs, has the opposte effect on mood, predsposng towards depresson and other related mood dsorders ([30]; though see also [31] for a dscusson of replcaton falures). Dayan and Huys [17] explaned ths paradox by suggestng that people who experenced hgh levels of serotonn and thus exaggerated Pavlovan behavoural nhbton durng early development [32] would be most senstve to the effects of any nterference wth ths nhbton n adulthood secondary to a drop n serotonn levels [33,34]. Thus, the nhbtory consequences of serotonn could account for both ts predsposng qualtes on a developmental tme-scale, and more acute relef durng depressve epsodes. The hypothess n [17] relates to two facets of the current study. Frst, f serotonn ndeed medates behavoural nhbton n the face of punshments [10,12 14] then t s a strong predcton that the prunng parameter c S, whch medates the nhbton of teratve thought processes, should be related to, and modulated by, serotonergc actvty. We plan to test ths drectly n future studes. There s already some, though far from conclusve, evdence pontng towards such an nfluence of serotonn on hgher-level cognton. Frst, serotonergc neurons project strongly to areas nvolved n goal-drected, affectve choces ncludng the medal prefrontal cortex [35]. Genetc varaton n the serotonn transporter allele modulates functonal couplng between amygdala and rostral cngulate cortex [36]. Next, orbtofrontal serotonn depleton mpacts cogntve flexblty, or the adaptve ablty to swtch between contngences, by mparng nhbtory control [37] n monkeys. Thrd, learned helplessness, whch can be nterpreted n goal-drected terms [17], depends crtcally on preand nfralmbc cortex n rats [38], and s known to be medated by serotonn [39]. Contrary to ths, there s a recent report that mood manpulaton, but not acute tryptophan depleton, mpars processng on the one-touch Tower of London (OTT) task [40], whch should certanly engage goal-drected processng. One possble explanaton for ths apparent dscrepancy s that although the OTT requres sequences of moves to be evaluated, there s no obvous aversve pont at whch Pavlovan prunng mght be nvoked. Further, although OTT s explctly framed as a cold task,.e. one whch does not nvolve affectve choces, there s also supportng evdence (see below). The second facet of our theoretcal model [17] concerns depresson. The model suggested that subjects prone to depresson exhbt decson makng that s more relant on serotonergc functon, expressed as excess prunng, but that the depressed state tself s charactersed by a low serotonn state and thus a loss of prunng. The stronger dependence on serotonn n at-rsk subjects would explan why only they are senstve to the mood effects of tryptophan depleton [34], and why ndvduals wth a polymorphsm n the serotonn transporter gene that reduces serotonn uptake are more lable to develop mood dsturbance, especally followng serotonn depleton [41,42]. That s, ths theory predcts excessve prunng to occur n subjects at rsk for depresson, and reduced prunng to occur durng a depressve epsode. The data presented here (a postve correlaton between mldly rased BDI scores and the tendency to prune when encounterng a large loss; Fgure 7) would be consstent wth ths theoretcal account f mldly rased BDI scores n otherwse healthy subjects (we screened for crtera for a major depressve epsode; and 94% of our partcpants had BDI scores v13, renderng depresson unlkely [43]) could be nterpreted as a vulnerablty or proneness to depresson. The mldly rased BDI scores do reveal a latent level of dysphorc symptoms amongst healthy partcpants [55]. Ths mght be n lne wth fndngs that levels of dysphorc symptoms correlate wth levels of dysfunctonal thnkng, and that a cyclcal nteracton between the two could, n the presence of certan envronmental events, crescendo nto a depressve epsode proper [45,46]. However, we are not aware of any drect evdence that mldly rased BDI scores measure vulnerablty, and maybe more crtcally, we dd not observe correlatons wth NEO neurotcsm, whch s an establshed rsk factor for depresson [47]. The strong predcton that serotonergc functon and behavoural nhbton n the face of losses should be reduced durng a major depressve epsode remans to be tested. However, there s already some evdence n favour of ths concluson. People actvely sufferng from depresson are mpared on the OTT [48,49]. The mparment relatve to controls grows wth the dffculty of the problem; and depressed subjects also spend ncreasng amounts of tme thnkng about the harder problems, wthout showng mproved choces [50]. Ths suggests that people who are sufferng from depresson have more dffculty searchng a deep tree effectvely (possbly also captured by more general, superfcal autobographcal recollectons; [51]). However, gven the fndng by [40], we note that t s at present not possble to nterpret ths conclusvely n terms of prunng. Fnally, the same group has also reported catastrophc breakdown n OTT performance n depressed subjects after negatve feedback [52]. Concluson We used a novel sequental decson-makng task n conjuncton wth a sophstcated computatonal analyss that ftted a hgh proporton of healthy subjects choces. Ths allowed us to unpack a central facet of effectve computaton, prunng. Importantly, most subjects were unable to resst prunng even when t was dsadvantageous, supportng our hypothess that ths process occurs by smple, Pavlovan, behavoural nhbton of ongong thoughts n the face of punshments [17]. Provocatvely, consstent wth ths model, we found a relatonshp between the propensty to prune and sub-clncal mood dsturbance, and ths suggests t would be opportune to examne n detal the model s predctons that prunng should be mpared n clncally depressed ndvduals and followng serotonn depleton. PLoS Computatonal Bology 9 March 2012 Volume 8 Issue 3 e

10 Methods Partcpants Fourty-sx volunteers (23 female, mean age years) were recruted from the Unversty College London (UCL) Psychology subject pool. Each gave wrtten nformed consent and receved monetary, partally performance-dependent compensaton for partcpatng n a 1.5-hour sesson. The study was conducted n accord wth the Helsnk declaraton and approved by the UCL Graduate School Ethcs Commttee. Excluson crtera were: known psychatrc or neurologcal dsorder; medcal dsorder lkely to lead to cogntve mparment; ntellgence quotent (IQ) v70; recent llct substance use and not havng Englsh as frst language. The absence of axs-i psychopathology and alcohol- or substance abuse/dependence was confrmed wth the Mn Internatonal Neuropsychatrc Inventory [53]. Personalty, mood, and cogntve measures were assessed wth the State-Trat Anxety Inventory [54], the Beck Depresson Inventory (BDI; [55]), the NEO Personalty Inventory [56], the Wechsler Test of Adult Readng (WTAR; [57]), and Dgt Span [58]. Subjects who were assgned to the dfferent groups, were matched for age, IQ and sex (all pw:19, one-way ANOVA). Ffteen subjects were assgned to group 270, 16 to group 2100 and 15 to group Mean age (+1 st. dev.) was 24:1+4:3, 24:6+4:3 and 22:7+3:6 years respectvely; mean dgt span scores were 18:4+3:2, 17:4+3:6 and 19:4+3:2; mean IQ scores (computed from WTAR) were 109:9+7:5, 110:3+3:9 and 111:9+2:1. There were 5 (33%), 8 (50%) and 10 (66%) men n each of the three groups. One subjects age nformaton, and one subject s STAI nformaton were lost. These subjects were excluded from the psychometrc correlaton analyses. Task Partcpants frst underwent extensve tranng to learn the transton matrx (Fgure 2A,B; [16]). Durng the tranng, subjects were repeatedly placed n a random startng state and told to reach a random target state n a specfed number of moves (up to 4). After 40 practce trals, tranng contnued untl the partcpant reached the target n 9 out of 10 trals. Most subjects passed the tranng crteron n three attempts. Reachng tranng crteron was mandatory to move on to the man task. After tranng, each transton was assocated wth a determnstc reward (Fgure 2B). Subjects completed two blocks of of 24 choce epsodes; each epsode ncluded 2 to 8 trals. The frst block of 24 epsodes was dscarded as part of tranng the reward matrx, and the second block of 24 epsodes was analysed. At the begnnng of each epsode, subjects were placed randomly n one of the states (hghlghted n whte) and told how many moves they would have to make (.e., 2 to 8). Ther goal was to devse a sequence of that partcular length of moves to maxmze ther total reward over the entre sequence of moves. To help the subjects remember the reward or punshment possble from each state, the approprate + or - were always dsplayed beneath each box. Regardless of the state the subject fnshed n on a gven epsode, they would be placed n a random new state at the begnnng of the next epsode. Thus, each epsode was an ndependent test of the subject s ablty to sequentally thnk through the transton matrx and nfer the best acton sequence. After each transton, the new state was hghlghted n whte and the outcome dsplayed. On half of the trals, subjects were asked to plan ahead ther last 2 4 moves together and enter them n one step wthout any ntermttent feedback. The reward matrx was desgned to assess subjects prunng strategy; and whether ths strategy changed n an adaptve, goaldrected way. All subjects experenced the same transton matrx, but the red transtons n Fgure 2C led to dfferent losses n the three groups, of 270, 2100 or 2140 pence respectvely. Ths had the effect of makng prunng counterproductve n groups 270 and 2100, but not 2140 (Fgures 2C E). At the end of the task, subjects were awarded a monetary amount based on ther performance, wth a maxmum of 20. They were also compensated 10 for tme and travel expenses. Model-based analyss In the look-ahead model, the Q-value of each acton a n the present state s s derved by ) searchng through all possble future choces; ) always choosng the optmal opton avalable n the future after a partcular choce; and ) assgnng the two actons at the present state the values of the mmedate reward plus the best possble future earnngs over the entre epsode. More concsely, the look-ahead (lo) model s a standard tree search model, n whch the value of a partcular acton s gven by the sum of the mmedate reward R(a,s) and the value of the optmal acton from the next state s ~T (a,s) Q lo (a,s)~r(a,s)z max Q lo (a,t (a,s)), ð1þ a where T s the determnstc transton functon. Ths equaton s terated untl the end of the tree has been reached [59]. For notatonal clarty, we omt dependence of Q values on the depth of the tree. To make the gradents tractable, we mplement the max operator wth a steep softmax. An explct search all the way to the end of the tree s unlkely for any depths w3, gven the large computatonal demands. The model Dscount (d) thus allowed, at each depth, a based con to be flpped to determne whether the tree search should proceed further, or whether t should termnate at that depth, and assume zero further earnngs. Let the probablty of stoppng be c. The expected outcome from a choce n a partcular state, the Q values, s now an average over all possble prunngs of the tree, weghted by how lkely that partcular number of prunngs s to occur: Q(a,s)~ XI ~1 Q lo (a,s)p(dc) where Q lo (a,s) s the full lookahead value of acton a n state s for the cut tree. Importantly, the number I s mmense. If the number of branches of a bnary tree s n~ X D d~1 2d, then there are I~ X n n possble ways of choosng up to n branches k~1 k of the tree to cut. Although ths overestmates the problem because branches off branches that have already been cut off should no longer be consdered, the problem remans overly large. We therefore use a mean-feld approxmaton, resultng n Q d values: Q d (a,s)~r(a,s)z(1{c) max Q d (a,t (a,s)) ð3þ a where, at each step, the future s weghted by the probablty (1{c) that t be encountered. Ths means that outcomes k steps ahead are dscounted by a factor (1{c) k{1. We note, however, that Equaton 3 solves a dfferent Markov decson problem exactly. Next, the Prunng (p) model encompassed the possblty that subjects were more lkely to stop after a large punshment had been encountered. It dd ths by separatng the stoppng ð2þ PLoS Computatonal Bology 10 March 2012 Volume 8 Issue 3 e