Expectations and outcomes: decision-making in the primate brain

Transcription

1 J Comp Physiol A (2005) 191: DOI /s REVIEW Allison N. McCoy Æ Michael L. Platt Expectations and outcomes: decision-making in the primate brain Received: 2 March 2004 / Revised: 28 July 2004 / Accepted: 12 August 2004 / Published online: 12 October 2004 Ó Springer-Verlag 2004 Abstract Success in a constantly changing environment requires that decision-making strategies be updated as reward contingencies change. How this is accomplished by the nervous system has, until recently, remained a profound mystery. New studies coupling economic theory with neurophysiological techniques have revealed the explicit representation of behavioral value. Specifically, when fluid reinforcement is paired with visually-guided eye movements, neurons in parietal cortex, prefrontal cortex, the basal ganglia, and superior colliculus all nodes in a network linking visual stimulation with the generation of oculomotor behavior encode the expected value of targets lying within their response fields. Other brain areas have been implicated in the processing of reward-related information in the abstract: midbrain dopaminergic neurons, for instance, signal an error in reward prediction. Still other brain areas link information about reward to the selection and performance of specific actions in order for behavior to adapt to changing environmental exigencies. Neurons in posterior cingulate cortex have been shown to carry signals related to both reward outcomes and oculomotor behavior, suggesting that they participate in updating estimates of orienting value. A. N. McCoy Æ M. L. Platt (&) Department of Neurobiology, Duke University Medical Center, 325 Bryan Research Building, Box 3209, Durham, NC 27710, USA platt@neuro.duke.edu M. L. Platt Center for Cognitive Neuroscience, Duke University, Durham, NC 27710, USA M. L. Platt Department of Biological Anthropology and Anatomy, Duke University, Durham, NC 27710, USA Introduction For even the simplest of organisms, adaptive behavior requires, to paraphrase Darwin, the preservation of favorable variants and the rejection of injurious ones. In other words, nervous systems must weigh the potential rewards and punishments associated with available options and select the behavior likely to yield the best possible outcome. This process of adaptive decisionmaking is well illustrated by the example of a financial advisor choosing whether to buy, sell or hold portions of stock based on calculations of the most likely monetary gain. In the face of sudden change, such as a stock market crash, the value associated with available options is rapidly updated. In this way, the decision process allows behavior to adjust to environmental flux, resulting in a net benefit to the organism. While the analogy of the financial advisor captures the iterative, adaptive nature of the decision process, how does the brain actually select an action from the behavioral repertoire? This is a question that has long puzzled philosophers, economists, behavioral ecologists, and neuroscientists. In fact, significant advancement in our understanding of the neural basis of decision-making has been made through a combination of economic theory and neurophysiological techniques (see Glimcher 2003, for exposition). This review describes current understanding of the neural mechanisms underlying decision-making, focusing on evidence for the neural representation, computation and revision of economic decision variables used to guide visual orienting in primates. Behavioral economics: expected value and decision-making In neurobiology, as in psychology, the sensory-motor reflex has been wide viewed as a useful model for directly linking sensation to action (Pavlov 1927; Sherrington 1906). As instructive as it has been, however, the simple

2 202 sensory-motor reflex can only begin to approximate the rich ethology of real-world behavior (Glimcher 2003; Herrnstein 1997; Platt 2002). Only recently have neurobiologists begun to investigate the idea that internal decision variables dynamically link purely sensory and purely motor processes and are explicitly represented in the nervous system (Basso and Wurtz 1997; Dorris and Munoz 1998; Kim and Shadlen 1999; Platt and Glimcher 1999; Shadlen and Newsome 1996, 2001). In contrast, over the past 400 years, economists have developed simple normative models to describe what rational agents should do when confronted with a choice between two options [Arnauld and Nichole 1662; Bernouilli (1758) in Speiser 1982]. The simplest economic model of decision-making, known as Expected Value Theory, posits that rational agents compute the likelihood that a particular action will yield a gain or loss, as well as the amount of gain or loss that can be expected from that choice (Arnauld and Nichole 1662). These values are then multiplied to arrive at an estimate of expected value for each possible course of action, and the option with the highest expected value is chosen (Arnauld and Nichole 1662). Maximizing expected value is the optimal algorithm for a rational chooser with complete information about the costs and benefits of different options, as well as their likelihood of occurrence. Expected value models, and variations on them, i.e., Expected Utility Theory [Bernouilli (1758) in Speiser (1982)], have been shown to be very good descriptors of the choices both people and animals make in a variety of simple situations (Stephens and Krebs 1986; Herrnstein 1997; Camerer 2003; Glimcher 2003). Expected value models of decision-making assume that rational choosers have access to information about the probability and amount of gain that can be expected from each action. Behavioral observations have repeatedly demonstrated that humans and animals are exquisitely sensitive to the expected value of available options (Herrnstein 1961; Stephens and Krebs 1986; Glimcher 2003). This suggests that nervous systems somehow represent information about the estimated costs and benefits of potential behaviors and use that information to dynamically link sensation to action. New studies have begun to shed light on this process. These studies suggest that neurons representing different eye movements compete to reach a firing rate threshold for movement initiation (Hanes and Schall 1996; Roitman and Shadlen 2002). The activity of these neurons is systematically enhanced by both sensory information favoring a particular eye movement (Shadlen and Newsome 1996, 2001; Roitman and Shadlen 2002) and increases in the expected value of that eye movement (Kawagoe et al. 1998; Leon and Shadlen 1999; Platt and Glimcher 1999; Coe et al. 2002). These observations intimate that the eye movement decision process may be instantiated rather simply by scaling the activity of movement-related neurons by the expected value of the eye movements they encode. Some of the important evidence supporting this model is described further below. Expected value and primate parietal cortex In an explicit application of economic theory to experimental neurophysiology, Platt and Glimcher (1999) manipulated the expected value of saccadic eye movements while monitoring the activity of neurons in the lateral intraparietal area (LIP), a subregion of primate posterior parietal cortex thought to intervene between visual sensation and eye movements (Fig. 1; Gnadt and Andersen 1988; Goldberg et al. 1990, 1998). Prior studies had suggested that the responses of LIP neurons were sensitive to the behavioral relevance of visual targets for guiding attention or gaze shifts (Gnadt and Andersen 1988; Colby et al. 1996; Platt and Glimcher 1997) as well as the strength of perceptual evidence favoring the production of a particular eye movement (Shadlen and Newsome 1996, 2001). These observations suggested that LIP neurons play a role in the oculomotor decision-making process (Platt and Glimcher 1999; Shadlen and Newsome 1996). In the first set of experiments designed to test this hypothesis (Platt and Glimcher 1999), monkeys performed cued saccade trials, in which a change in the color of a central fixation light instructed subjects to Fig. 1 A network for oculomotor decision-making. Medial and lateral views of the macaque monkey brain illustrating major pathways carrying signals related to saccadic eye movements (green) and reward (blue). Note: many of these pathways are bidirectional but have been simplified for presentation. AMYG amygdala, CGp posterior cingulate cortex, CGa anterior cingulate cortex, LIP lateral intraparietal cortex, SEF supplementary eye fields, FEF frontal eye fields, PFC prefrontal cortex, OFC orbitofrontal cortex, SC superior colliculus, NAC nucleus accumbens, SNr substantia nigra pars reticulata, SNc substantia nigra pars compacta, VTA ventral tegmental area. After (Platt et al. 2004)

3 203 shift gaze to one of two possible target locations in order to receive a fruit juice reward. A change from yellow to red indicated that looking to the upper target would be rewarded while a change from yellow to green indicated that the lower target would be rewarded. In successive blocks of trials, either the volume of juice delivered for each eye movement response or the probability that each possible movement would be cued and therefore reinforced was varied. The investigators found that the activity of LIP neurons was systematically modulated by both the size of the reward a monkey could expect to receive and the probability that a particular gaze shift would be reinforced the two variables needed to compute the expect value of the movement. Intriguingly, the effects of expected value on neuronal firing rates were maximal early on each trial but diminished after the rewarded target had been cued. The information encoded by LIP neurons thus appeared to track the monkey s running estimate of the expected value of a particular movement, which was maximal once that movement was cued (Glimcher 2002). These experiments demonstrated that LIP neurons carry information about the economic decision variables thought to drive rational decision-making. In those studies, however, monkeys did not have a choice about which eye movement to make and thus the link between decision-making and LIP activity could not be directly observed. In a second experiment (Platt and Glimcher 1999), the investigators studied the activity of LIP neurons while monkeys were permitted to freely choose between two movements differing in expected value. This allowed the investigators to derive a behavioral estimate of the monkey s subjective valuation of each response, which could be compared directly with LIP neuronal activity. Under these conditions, monkeys behaved like rational consumers: their pattern of target choices was sensitive to the amount of fruit juice associated with each target (Fig. 2b). LIP neuronal activity was also sensitive to expected value in the free choice task (Fig. 2a, c). Activity prior to movement increased systematically with its expected value in concert with the monkey s valuation of the same movement. Based on these data, Platt and Glimcher (1999) concluded that LIP neurons encode the expected value of potential eye movements. More specifically, Gold and Shadlen (2002) have argued that these data are consistent with the hypothesis that LIP neurons signal the instantaneous likelihood that the monkey will choose and execute a particular eye movement, and that this likelihood is computed, at least in part, based on the expected value of that movement. Neurons in several other brain areas have been shown to carry signals related to the expected value of eye movements. For example, neurons in the caudate nucleus were found to respond in anticipation of rewardpredicting stimuli on a memory saccade task (Kawagoe et al. 1998), and the strength of this signal was correlated with the latency of behavioral response (Takikawa et al. Fig. 2a c Monkeys and posterior parietal neurons are sensitive to eye movement value. a Firing rate as a function of time for a single LIP neuron when the monkey subject chose to shift gaze to the target in the neuronal response field. Black curve high value trials, grey curve low value trials. Tick marks indicate action potentials recorded on successive trials for the first ten trials of each block. b Proportion of trials on which a single monkey subject chose target 1 as a function of its expected value [reward size target 1/ (reward size target 1 + reward size target 2)]. c Average normalized premovement firing rate (±SE) for a single posterior parietal neuron in the same monkey as a function of the expected value of the same target. After Platt and Glimcher (1999). Reprinted by permission from Nature 2002). On a reaction time eye movement task, anticipatory activity in the caudate was maximal when the contralateral target was reliably rewarded, and tracked changes in the monkeys response time as the rewarded location was reversed (Lauwereyns et al. 2002). Anticipatory activity was absent, however, during a task in which the rewarded target unpredictably changed from trial to trial (Lauwereyns et al. 2002). Signals correlated with the reward value associated with eye movements have also been found in dorsolateral prefrontal cortex (Leon and Shadlen 1999), supplementary eye fields (Amador et al. 2000; Stuphorn et al. 2000a), substantia nigra pars reticulata (Sato and Hikosaka 2002), and even the superior colliculus (Ikeda and Hikosaka 2003). Expected value signals thus appear to penetrate the oculomotor system from the cortex through the final common pathway in the superior colliculus, presumably serving to bias eye movement selection towards maximization of reward. Behavioral measures indicate that expected value systematically influences not only target selection by eye movements, but also modulates saccade metrics, including amplitude, velocity, and reaction time

4 204 b Fig. 3a, b Monkeys rapidly learn to choose the high value target. a Proportion of trials on which a single monkey chose target 1 as a function of its expected value. Individual graphs plot choice functions for sequential five trial blocks following a change in reward contingencies favoring target 1. Red line indicates a cumulative normal function fit to the choice data. b Reward sensitivity over time. Graph plots the steepness of the psychometric function (Weber fraction) computed by dividing the standard deviation by the mean of the cumulative normal function fit to the choice data in each five trial block for two monkey subjects. After McCoy et al. (2003). Reprinted with permission from Neuron throughout the cortical and subcortical oculomotor afferents to the superior colliculus. Such modulations in saccade-associated activity presumably result in the differential activation of pools of motor neurons in the oculomotor brainstem that ultimately generate the patterns of muscle contraction responsible for shifting gaze. This model, however, has yet to be tested functionally using either microstimulation or inactivation techniques. Learning from mistakes: dopamine and prediction error (Leon and Shadlen 1999; Takikawa et al. 2002; McCoy et al. 2003). These behavioral observations are consistent with a model for oculomotor decision-making in which expected value systematically biases neuronal activity Recent studies by Platt and colleagues (McCoy et al. 2003) have demonstrated that the expected value of eye movements is rapidly updated in the primate brain when reward contingencies change. Figure 3a shows the pattern of choice behavior in the target choice task for one well-trained monkey as a function of time following the introduction of a novel set of reward contingencies. Each graph plots the frequency of choosing target 1 as a function of the relative value of that target (reward for target 1 divided by the sum of the reward values offered for targets 1 and 2) for data gathered with multiple different target values over several weeks of experimental sessions. The choice functions were well fit by cumulative normal functions, which became steeper over time but nearly reached asymptote after only about ten trials. Figure 3b plots reward sensitivity (Weber fraction) against time for two monkeys. These data indicate that the monkeys rapidly learned to choose the target associated with the greatest amount of reward. These data suggest that the brain continuously evaluates the reward (or punishment) outcome associated with movements and uses that information to update the expected value of those movements. Learning theorists have long argued that a comparison of reward expectation and reward outcome directly drives learning (Pearce and Hall 1980; Rescorla and Wagner 1972) and should therefore be encoded in the nervous system. The comparison of expected and actual reward is known as a reward prediction error (Sutton and Barto 1981), and is defined formally as the instantaneous discrepancy between the maximal associative strength sustained by the reinforcer and the current strength of the predictive stimulus (Schultz 2004). Reward prediction error models of learning have been invoked to explain previously elusive behavioral phenomena such as blocking. In traditional associative

5 205 models of learning (e.g., Skinner 1981; Thorndike 1898) any cue paired with reinforcement should be learned, but in blocking paradigms redundant cues paired with rewards fail to be learned (the cues are blocked from acquiring associative value). The typical blocking experiment introduces four stimuli, A, B, X, Y, which are either paired with a reward or with nothing. In the first stage of the experiment, a subject learns that stimulus A is paired with a reward while stimulus B is not. Once this is learned, the same stimuli are subsequently paired with two novel stimuli (X and Y), and, in this second stage of the experiment, the joint stimuli AX and BY are both paired with rewards. If learning were merely associative, the subject would respond to both novel stimuli X and Y as if they predicted the reward. In fact, under these conditions subjects learn to associate Y, but not X, with reward. The explanation for this effect lies in the fact that because stimulus A fully predicted reward in stage 1, stimulus X was redundant and was therefore not highlighted for learning (Rescorla and Wagner 1972). Reward prediction error models, such as the Rescorla-Wagner model or temporal difference learning models (Sutton and Barto 1981), nicely account for these results because once the relationship between cue A and reward has been learned, the prediction error term goes to 0 and remains unchanged when the combined stimulus AX is followed by a reward in stage 2; hence, no new learning occurs for the redundant cue. Over the past decade, Wolfram Schultz and colleagues have demonstrated that the activation of dopamine neurons in the midbrain represents, at least in part, an instantiation of a reward prediction error in neural circuitry (Schultz and Dickinson 2000). These conclusions rest on the observation that the activity of dopamine neurons in the substantia nigra and ventral tegmental area is elevated by the delivery of unpredicted rewards, unchanged following the delivery of predicted rewards, and depressed when expected rewards are withheld (Schultz 1998). Moreover, in blocking paradigms these dopamine neurons fail to respond to redundant cues (Waelti et al. 2001). The activity of these neurons has therefore been proposed to encode reward prediction error and thus to determine the direction and rate of learning: learning occurs when the error is positive and the reward is not predicted by the stimulus; no learning occurs when the error is 0 and the reward is fully predicted; forgetting or extinction occurs when the error is negative and the reward is less than predicted by the stimulus; and the rate of learning is determined by the absolute size of the error, whether positive or negative (Schultz and Dickinson 2000). Recent evidence has called into question whether the signals carried by dopaminergic midbrain neurons are specific to rewards or can be generalized to other salient stimuli (Horvitz 2002). Horvitz and others have shown that putative dopamine neurons respond to a variety of novel or arousing events, including aversive as well as rewarding stimuli (Salamone 1994; Horvitz 2000; but see Mirenowicz and Schultz 1996). These observations suggest that the responses of dopamine neurons might therefore signal surprising events in the environment regardless of whether they are rewarding or aversive (Horvitz 2002). Rather than encoding a reward prediction error per se, dopamine neurons might instead gate cortical and limbic inputs to the striatum following unpredicted events and thereby facilitate learning. The role of dopamine neurons in learning thus appears to be more complex than previously thought, and a deeper understanding will likely lie in further investigations of the response of dopamine neurons to all salient stimuli good or bad. Updating expectations: the role of posterior cingulate cortex Up to this point, we have discussed the evidence that neurons in parietal cortex, prefrontal cortex, the basal ganglia, and the superior colliculus carry information about the expected value of the decision about where to look. Midbrain dopamine neurons, on the other hand, appear to play a role in processing reward-related information in the abstract highlighting when expected and actual reward outcomes are in conflict. Such a signal would be useful for initiating the assignment of new estimates of the expected value of potential behaviors, and evidence increasingly suggests this to be the case (Montague and Berns 2002). But how are abstract prediction error signals generated by midbrain dopamine neurons linked to the selection and performance of specific actions, such as eye movements, that maximize reward? With this question in mind, Platt and colleagues (McCoy et al. 2003) investigated the properties of neurons in posterior cingulate cortex (CGp), a poorly understood part of the limbic system thought to be related to eye movements (Olson et al. 1996), spatial localization (Sutherland et al. 1988; Harker and Whishaw 2002), and learning (Bussey et al. 1997; Gabriel et al. 1980; Gabriel and Sparenborg 1987). Many CGp neurons are activated following saccades, in contrast with the activation of saccade-related neurons in parietal and prefrontal cortex prior to movement (Olson et al. 1996). Such a pattern of activation is suggestive of an evaluative, rather than generative, role in oculomotor behavior, an idea first suggested by Vogt and colleagues (Vogt et al. 1992). Anatomically, CGp is interconnected with reward-related areas of the brain (see Fig. 1), including the anterior cingulate cortex (Morecraft et al. 1993), orbitofrontal cortex (Cavada et al. 2000), and the caudate nucleus (Baleydier and Mauguiere 1980), as well as oculomotor areas such as parietal cortex (Cavada and Goldman-Rakic 1989), prefrontal cortex (Vogt and Pandya 1987) and the supplementary eye fields. These connections provide potential sources for a functional linkage of motivational and oculomotor information within CGp. To test the idea that CGp links motivational outcomes with eye movements, Platt and colleagues

6 206 c Fig. 4a d Representation of saccade value in posterior cingulate cortex. a Left panel, average firing rate (±SE) for a single CGp neuron plotted as a function of time on high reward (black curve) and low reward (grey curve) blocks of trials. Rasters indicate time of action potentials on individual trials. All trials are aligned on movement onset. Right panel, average firing rate (±SE) following movement onset (grey shaded region on left panel) plotted as a function of reward size, for the same neuron. Reward size is measured as the open time (ms) of a computer driven solenoid and is linearly related to juice volume. b left panel, average firing rate (±SE) for a second CGp neuron plotted as a function of time on high reward (black curve) and low reward (grey curve) blocks of trials. Rasters indicate time of action potentials on individual trials. All trials are aligned on reward offset. Right panel, average firing rate (±SE) following reward offset (grey shaded region on left panel) plotted as a function of reward size, for the same neuron. c Population data. Proportion of CGp neurons with significant modulation by reward size, peak saccade velocity, saccade amplitude, and saccade latency plotted as a function of time. Significant modulations P< 0.05 for each individual factor in a multiple regression analysis of firing rate in each of ten sequential 200 ms epochs. The size of liquid reward delivered on correct trials was controlled linearly by the open time of a computer-driven solenoid (volume= open time in ms). d Reward value scales the gain of CGp responses. The correlation coefficient between firing rate and reward size for each neuron was plotted as a function of movement response index, a logarithmically scaled measure of the degree to which each neuron was excited or inhibited after movement relative to fixation level activity on mapping trials. Each dot represents data for one cell (n=67) and the best-fit line is shown in grey. Note: two outliers were removed from this analysis. After McCoy et al. (2003). Reprinted with permission from Neuron recorded the activity of CGp neurons while monkeys shifted gaze to a single visual target for fruit juice rewards. In the first experiment, the size of the reward associated with visually-guided saccades was held constant for a block of 50 trials, and then varied between blocks, while the oculomotor behavior of the monkeys and the activity of CGp neurons were examined. The authors found that saccade metrics were sensitive to the size of reward associated with gaze shifts. Specifically, monkeys made faster, higher velocity saccades when expecting smaller rewards, a strategy consistent with reducing the delay to reinforcement and thereby increasing fluid intake rate in low reward blocks of trials. This observation is consistent with the idea that, even in the absence of an overt decision task, monkeys are sensitive to the expected value of gaze shifts. In this study, the activity of CGp neurons was also systematically modulated by reward value, illustrated by data from two example neurons in Fig. 4. Single neurons were sensitive to reward size following movement (Fig. 4a) as well as following reward delivery (Fig. 4b). These represented largely separate modulations since the events were separated in time by at least 500 ms. Across the population of studied neurons, approximately onethird of the population was sensitive to reward size following movement and another third following the receipt of reward (Fig. 4c). These modulations by reward size were independent of any effects of saccade metrics, demonstrated by the inclusion of saccade amplitude, latency, and peak velocity as independent factors in multiple linear regression analysis of firing rate as a function of reward size. These results thus demonstrate that information about both the predicted and experienced reward value of a particular eye movement is carried by the activity of CGp neurons. Such modulation in neuronal activity by reward size cannot be accounted for solely by reafferent input from motor areas. In this experiment, the net effect of changes in reward value was to scale the gain of spatially-selective neuronal responses following movement. Sixty-two percent of studied neurons in CGp responded in a spatially selective manner following eye movement onset. Among these, neurons excited following a particular movement were further excited when the expected value of that movement was increased; similarly, neurons suppressed following a particular movement were further suppressed when the expected value of that movement was increased (Fig. 4d). Across the population, reward value thus tuned the spatial sensitivity of the CGp neuronal population to saccade direction. Improved spatial sensitivity under high reward conditions may be associated with the slower, more deliberate saccades made by monkeys under high reward conditions in this experiment. The timing of modulations in CGp activity by reward value suggests that this area encodes both the predicted and experienced value of a particular movement. If so, CGp might also be expected to carry signals related to an error in reward prediction, arising from a discrepancy between predicted and experienced value. This hypothesis was addressed in a second experiment in which reward size was held constant but delivered probabilistically while monkeys shifted gaze to a target fixed in the area of maximal response for the neuron under study. Correct trials were reinforced on a variable reward schedule of 0.8, meaning that, on average, 80% of correct trials were followed by an auditory noise burst and reward ( rewarded trials ), while the remaining 20% of correct trials were followed by an auditory noise burst only but no juice ( unrewarded trials ). The relatively low frequency of unrewarded trials allowed them to serve as catch trials for which a predicted reward was unexpectedly withheld. Once again, the behavior of monkeys and the activity of CGp neurons were both sensitive to this manipulation in reward contingencies. After predicted rewards were withheld, monkeys made higher velocity saccades, similar to the effects of low reward on saccade metrics found in the previous experiment. Similarly, the activity of single CGp neurons was also significantly different on rewarded and unrewarded trials (Fig. 5a, b). Across the population of cells studied in this experiment, firing rate following the usual time of reward delivery was significantly greater when expected rewards were omitted (Fig. 5c). These data demonstrate that CGp neurons faithfully report the omission of expected reward, suggestive of a prediction error-like signal for eye movements (Schultz and Dickinson 2000). Intriguingly, many CGp neurons responded equivalently to the delivery of larger than average rewards and

7 207 the omission of predicted rewards (McCoy et al. 2003), unlike dopamine neurons which respond with a burst of action potentials following unpredicted rewards but are suppressed following the omission of predicted rewards (Schultz et al. 1997). The reward modulation of neuronal activity in CGp is therefore consistent with attentional theories of learning, which posit that reward prediction errors highlight unpredicted stimuli as important for learning (Mackintosh 1975; Pearce and Hall 1980). According to this idea, the absolute value of

8 208 Fig. 5a, b Representation of saccade reward prediction error in posterior cingulate cortex. a Firing rate as a function of time for a single CGp neuron on rewarded (black curve) and unrewarded (grey curve) delayed saccade trials. On average, eight out of ten randomly-selected correct trials were rewarded. Conventions as in Fig. 4. b Population response to reward omission. c Average (+SE) normalized firing rate measured after the normal time of reward delivery for the population of CGp neurons on rewarded and unrewarded trials. After McCoy et al. (2003). Reprinted with permission from Neuron the neuronal response correlates with the extent to which the reward event differed from expectation, whether in a positive or negative direction. While such a signal would not contain information about what needs to be learned about the relationship between a stimulus and reward, such a signal would be useful for instructing when and how rapidly learning should occur. Some of the saccade-related reward signals uncovered in CGp may best be understood in terms of such an attentional learning theory. Taken together, the results of these experiments suggest that neurons in CGp link motivational outcomes with gaze shifts. Recent studies have reported that neurons in the supplementary eye fields, with which CGp is reciprocally connected, also respond to reward outcomes associated with eye movements (Amador et al. 2000; Stuphorn et al. 2000b). Neurons in other areas of cingulate cortex also have been implicated in guiding actions based on reward value. While studying timing behavior in primates, Niki and Watanabe (1979) first uncovered two classes of rewarderror units in anterior cingulate cortex (CGa): one class responding after juice delivery, and the other responding following incorrect trials as well as following the omission of reward on correct trials. Single cells in anterior cingulate cortex (CGa) have also been found to carry signals related to reward expectation on a cued multi-trial color discrimination task (Shidara and Richmond 2002). In the rostral cingulate motor area (CMAr), Shima and Tanji (1998) found neurons that responded when a change in reward contingencies but not a neutral cue prompted monkeys to change the strategy of their behavioral response in order to maximize their receipt of reward. Inactivation of the CMAr by muscimol injection resulted in insensitivity of the monkeys to a reward decrement and an apparent inability to modify their behavioral strategy in order to maximize reward. Taken together, these data are consistent with a direct role for cingulate cortex and supplementary eye fields in linking motivational outcomes to action. Neurons in the primate ventral striatum (Cromwell and Schultz 2003; Hassani et al. 2001; Schultz 2004) and orbitofrontal cortex (Rolls 2000; Tremblay and Schultz 1999) have also been shown to carry information about predicted and experienced rewards. These neurons do not appear to be directly related to the generation of action, but their responses otherwise bear striking resemblance to those in cingulate cortex. In particular, neurons in ventral striatum and orbitofrontal cortex respond during delay periods preceding reinforcement, as well as following reinforcement, and these responses reflect the subjective preferences of subjects for particular rewards (Tremblay and Schultz 1999). Thus, orbitofrontal cortex and ventral striatum appear to convert information about rewards, punishments, and their predictors into a common internal currency (Montague and Berns 2002). Neurons in the dorsal striatum and dorsolateral prefrontal cortex, on the other hand, show similar response properties, but are activated in association with particular movements, much like neurons in CGp and supplementary eye fields. In summary, the response properties of neurons in ventral striatum and orbitofrontal cortex suggest they provide a valuation scale for a broad range of stimuli, (Montague and Berns 2002), while those in the dorsal striatum, dorsolateral prefrontal cortex, supplementary eye fields, and cingulate cortex may serve to associate this information with specific actions (Schultz 2004).

9 209 Decision-making in the human brain Recent neuroimaging studies suggest that some of the same neurophysiological processes guiding oculomotor decision-making characterize more abstract representations of reward, punishment, and decision-making in humans. Specifically, the striatum, amygdala, orbitofrontal cortex, prefrontal cortex, anterior cingulate cortex, and parietal cortex are activated by rewards and reward-associated stimuli (Elliott et al. 2003). Moreover, hemodynamic responses are modulated by reward uncertainty in orbitofrontal cortex, ventral striatum (Berns et al. 2001; Critchley et al. 2001), and CGp (Smith et al. 2002) and are linearly correlated with expected value in orbitofrontal cortex (O Doherty et al. 2001), ventral and dorsal striatum (Delgado et al. 2000, 2003), amygdala (Breiter et al. 2001), and premotor cortex (Elliott et al. 2003). Further, errors in predicting rewards or punishments evoke hemodynamic responses in anterior cingulate cortex (Holroyd et al. 2004), ventral striatum (Pagnoni et al. 2002; Seymour et al. 2004), CGp (Smith et al. 2002) and insula (Seymour et al. 2004), and error-related electrophysiological responses have been recorded from the medial frontal and anterior cingulate cortices with scalp electrodes in humans as well (Holroyd et al. 2003). These observations indicate that brain regions carrying value-related information are activated in a similar fashion in monkeys and humans performing disparate types of learning and decision-making tasks. Human neuroimaging studies, however, have not yet tested the hypothesis that pools of neurons coding different movements are activated in proportion to movement value. Conclusions In summary, brain regions have been identified that appear to participate in several stages of oculomotor decision-making, from sensation, to reward expectation, to action, to outcome evaluation. Neurons in parietal cortex, prefrontal cortex, the basal ganglia, and the superior colliculus have been shown to encode in their firing rates the expected value of eye movements. Dopamine neurons in the midbrain, on the other hand, appear to signal an abstract reward prediction error, encoding in their firing rates a discrepancy between predicted and actual rewards. Because dopamine neurons terminate on glutamatergic inputs to the striatum from orbitofrontal cortex and amygdala (among other cortical and limbic structures), they are well-situated to gate the flow of motivational information through the striatum and other basal ganglia structures, whose purpose is ultimately to produce adaptive behaviors. Neurons in the ventral striatum, orbitofrontal cortex, and cingulate cortices, in turn, appear to signal the relative value or salience of features in the environment that may be important for learning and controlling behavior. In particular, posterior cingulate and supplementary eye field neurons have been recently shown to carry signals related to both reward outcomes and oculomotor behavior, and may contribute to updating orienting value signals in parietal and prefrontal cortex. Activation of this complex network thus appears to underlie visuallyguided behavior in particular, how animals choose where to look- and serves as a model for understanding behavioral decision-making more generally. References Amador N, Schlag-Rey M, Schlag J (2000) Reward-predicting and reward-detecting neuronal activity in the primate supplementary eye field. J Neurophysiol 84: Arnauld A, Nichole P (1662) The art of thinking: Port-Royal logic. Translated by Dickoff J, James P. Bobbs-Merrill, Indianapolis Baleydier C, Mauguiere F (1980) The duality of the cingulate gyrus in monkey. Neuroanatomical study and functional hypothesis. Brain 103(3): Basso MA, Wurtz RH (1997) Modulation of neuronal activity by target uncertainty. Nature 389:66 69 Berns GS, McClure SM, Pagnoni G, Montague PR (2001) Predictability modulates human brain response to reward. J Neurosci 21: Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P (2001) Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30: Bussey TJ, Everitt BJ, Robbins TW (1997) Dissociable effects of cingulate and medial frontal cortex lesions on stimulus-reward learning using a novel Pavlovian autoshaping procedure for the rat: implications for the neurobiology of emotion. Behav Neurosci 111: Camerer CF (2003) Behavioral game theory: experiments in strategic interaction (Roundtable Series in Behavioral Economics). Princeton University Press, Princeton Cavada C, Goldman-Rakic PS (1989) Posterior parietal cortex in rhesus monkey: I. Parcellation of areas based on distinctive limbic and sensory corticocortical connections. J Comp Neurol 287: Cavada C, Company T, Tejedor J, Cruz-Rizzolo RJ, Reinoso-Suarez F (2000) The anatomical connections of the macaque monkey orbitofrontal cortex. A review. Cereb Cortex 10: Coe B, Tomihara K, Matsuzawa M, Hikosaka O (2002) Visual and anticipatory bias in three cortical eye fields of the monkey during an adaptive decision-making task. J Neurosci 22(12): Colby CL, Duhamel JR, Goldberg ME (1996) Visual, presaccadic, and cognitive activation of single neurons in monkey lateral intraparietal area. J Neurophysiol 76: Critchley HD, Mathias CJ, Dolan RJ (2001) Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29: Cromwell HC, Schultz W (2003) Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. J Neurophysiol 29:29 Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA (2000) Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84: Delgado MR, Locke HM, Stenger VA, Fiez JA (2003) Dorsal striatum responses to reward and punishment: effects of valence and magnitude manipulations. Cogn Affect Behav Neurosci 3:27 38 Dorris MC, Munoz DP (1998) Saccadic probability influences motor preparation signals and time to saccadic initiation. J Neurosci 18:

10 210 Elliott R, Newman JL, Longe OA, Deakin JF (2003) Differential response patterns in the striatum and orbitofrontal cortex to financial reward in humans: a parametric functional magnetic resonance imaging study. J Neurosci 23(1): Gabriel M, Sparenborg S (1987) Posterior cingulate cortical lesions eliminate learning-related unit activity in the anterior cingulate cortex. Brain Res 409: Gabriel M, Orona E, Foster K, Lambert RW (1980) Cingulate cortical and anterior thalamic neuronal correlates of reversal learning in rabbits. J Comp Physiol Psychol 94: Glimcher P (2002) Decisions, decisions, decisions: choosing a biological science of choice. Neuron 36: Glimcher P (2003) Decisions, Uncertainty, and the brain: the science of neuroeconomics. MIT Press, Cambridge Gnadt JW, Andersen RA (1988) Memory related motor planning activity in posterior parietal cortex of macaque. Exp Brain Res 70: Gold JI, Shadlen MN (2002) Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36: Goldberg ME, Colby CL, Duhamel JR (1990) Representation of visuomotor space in the parietal lobe of the monkey. Cold Spring Harb Symp Quant Biol 55: Gottlieb JP, Kusunoki M, Goldberg ME (1998) The representation of visual salience in monkey parietal cortex. Nature 391: Hanes DP, Schall JD (1996) Neural control of voluntary movement initiation. Science 274: Harker KT, Whishaw IQ (2002) Impaired spatial performance in rats with retrosplenial lesions: importance of the spatial problem and the rat strain in identifying lesion effects in a swimming pool. J Neurosci 22: Hassani OK, Cromwell HC, Schultz W (2001) Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J Neurophysiol 85: Herrnstein RJ (1961) Relative and absolute strength of response as a function of frequency of reinforcement. J Exp Anal Behav 4: Herrnstein RJ (1997) The matching law: papers in psychology and economics. Harvard University Press, Cambridge Holroyd CB, Nieuwenhuis S, Yeung N, Cohen JD (2003) Errors in reward prediction are reflected in the event-related brain potential. Neuroreport 14: Holroyd CB, Nieuwenhuis S, Yeung N, Nystrom L, Mars RB, Coles MG, Cohen JD (2004) Dorsal anterior cingulate cortex shows fmri response to internal and external error signals. Nat Neurosci 7: Horvitz JC (2000) Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96: Horvitz JC (2002) Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behav Brain Res 137:65 74 Ikeda T, Hikosaka O (2003) Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39: Kawagoe R, Takikawa Y, Hikosaka O (1998) Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci 1: Kim JN, Shadlen MN (1999) Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nat Neurosci 2: Lauwereyns J, Watanabe K, Coe B, Hikosaka O (2002) A neural correlate of response bias in monkey caudate nucleus. Nature 418: Leon MI, Shadlen MN (1999) Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron 24: Mackintosh NJ (1975) Blocking of conditioned suppression: role of the first compound trial. J Exp Psychol Anim Behav Process 1: McCoy AN, Crowley JC, Haghighian G, Dean HL, Platt ML (2003) Saccade reward signals in posterior cingulate cortex. Neuron 40: Mirenowicz J, Schultz W (1996) Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379: Montague PR, Berns GS (2002) Neural economics and the biological substrates of valuation. Neuron 36: Morecraft RJ, Geula C, Mesulam MM (1993) Architecture of connectivity within a cingulo-fronto-parietal neurocognitive network for directed attention. Arch Neurol 50: Niki H, Watanabe M (1979) Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res 171: O Doherty J, Kringelbach ML, Rolls ET, Hornak J, Andrews C (2001) Abstract reward and punishment representations in the human orbitofrontal cortex. Nat Neurosci 4: Olson CR, Musil SY, Goldberg ME (1996) Single neurons in posterior cingulate cortex of behaving macaque: eye movement signals. J Neurophysiol 76: Pagnoni G, Zink CF, Montague PR, Berns GS (2002) Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci 5:97 98 Pavlov IP (1927) Conditioned reflexes. Oxford University Press, London Pearce JM, Hall G (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87: Platt ML (2002) Neural correlates of decisions. Curr Opin Neurobiol 12(2): Platt ML, Glimcher PW (1997) Responses of intraparietal neurons to saccadic targets and visual distractors. J Neurophysiol 78: Platt ML, Glimcher PW (1999) Neural correlates of decision variables in parietal cortex. Nature 400: Platt ML, Lau B, Glimcher PW (2004) Situating the Superior Colliculus within the Gaze Control Network. In: Hall WC, Moschovakis A (eds) The superior colliculus: new approaches for studying sensorimotor integration. CRC Press, New York Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning. Variations in the effectiveness of reinforcement and nonreinforcement. In: Black AHaP WF (eds) Classical conditioning. II. Current research and theory. Appleton-Century- Crofts, New York Roitman JD, Shadlen MN (2002) Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J Neurosci 22: Rolls ET (2000) The orbitofrontal cortex and reward. Cereb Cortex 10: Salamone JD (1994) The involvement of nucleus accumbens dopamine in appetitive and aversive motivation. Behav Brain Res 61: Sato M, Hikosaka O (2002) Role of primate substantia nigra pars reticulata in reward-oriented saccadic eye movement. J Neurosci 22: Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1 27 Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr Opin Neurobiol 14: Schultz W, Dickinson A (2000) Neuronal coding of prediction errors. Annu Rev Neurosci 23: Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: Seymour B, O Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS (2004) Temporal difference models describe higher-order learning in humans. Nature 429: Shadlen MN, Newsome WT (1996) Motion perception: seeing and deciding. Proc Natl Acad Sci U S A 93: Shadlen MN, Newsome WT (2001) Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J Neurophysiol 86: Sherrington CS (1906) The integrative action of the nervous system. Scribner, New York

11 211 Shidara M, Richmond BJ (2002) Anterior cingulate: single neuronal signals related to degree of reward expectancy. Science 296: Shima K, Tanji J (1998) Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282: Skinner BF (1981) Selection by consequences. Science 213: Smith K, Dickhaut J, McCabe K, Pardo JV (2002) Neuronal substrates for choice under ambiquity, risk, gains, and losses. Mangement Science 48: Speiser D (ed) (1982) The works of Daniel Bernouilli. Birkhauser, Boston Stephens DW, Krebs JR (1986) Foraging theory. Princeton University Press, Princeton Stuphorn V, Taylor TL, Schall JD (2000a) Performance monitoring by the supplementary eye field. Nature 408: Stuphorn V, Taylor TL, Schall JD (2000b) Performance monitoring by the supplementary eye field. Nature 408: Sutherland RJ, Whishaw IQ, Kolb B (1988) Contributions of cingulate cortex to two forms of spatial learning and memory. J Neurosci 8: Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88: Takikawa Y, Kawagoe R, Itoh H, Nakahara H, Hikosaka O (2002) Modulation of saccadic eye movements by predicted reward outcome. Exp Brain Res 142: Thorndike LL (1898) Animal intelligence: an experimental study of the asssociative processes in animals. Psychol Rev Monogr [Suppl 11] Tremblay L, Schultz W (1999) Relative reward preference in primate orbitofrontal cortex. Nature 398: Vogt BA, Pandya DN (1987) Cingulate cortex of the rhesus monkey. II. Cortical afferents. J Comp Neurol 262: Vogt BA, Finch DM, Olson CR (1992) Functional heterogeneity in cingulate cortex: the anterior executive and posterior evaluative regions. Cereb Cortex 2: