Volume 7, Number 4, 2001 REVIEW

Transcription

1 ling by Dopamine Neurons Volume 7, Number 4, 2001 REVIEW Reward Signaling by Dopamine Neurons WOLFRAM SCHULTZ Institute of Physiology and Program in Neuroscience University of Fribourg Switzerland Dopamine projections from the midbrain to the striatum and frontal cortex are involved in behavioral reactions controlled by rewards, as inferred from deficits in parkinsonism, schizophrenia, and drug addiction. Recent experiments have shown that dopamine neurons are not directly modulated in relation to movements. Rather, they appear to code the rewarding aspects of environmental stimuli. They show short, phasic increases of activity following primary food and liquid rewards ( unconditioned stimuli ) and conditioned, reward-predicting stimuli of visual, auditory, and somatosensory modalities. They also display smaller activation-depression sequences after stimuli resembling rewards and after novel or particularly intense stimuli. Rewards are only reported as far as they occur differently than predicted. According to learning theories, a prediction error message may constitute a powerful teaching signal for behavior and learning. The phasic reward message is different from the more tonic enabling function of dopamine that is deficient in Parkinson s disease, indicating that dopamine neurons subserve different functions at different time scales. Neurons in other brain structures, such as the striatum, orbitofrontal cortex, and amygdala, code the quality, quantity, and preference of rewards. The dopamine reward prediction error signal may cooperate with these reward perception signals during the learning and performance of behavioral reactions to motivating environmental stimuli. NEUROSCIENTIST 7(4): , 2001 KEY WORDS Motivation, Learning, Parkinsonism, Schizophrenia, Drugs The studies were supported by the Swiss National Science Foundation, European Union (Human Capital and Mobility and Biomed 2 programs via Swiss Federal Office of Education and Science), James McDonnell Foundation, Roche Research Foundation, and British Council. Address correspondence to: W. Schultz, Institute of Physiology and Program in Neuroscience, University of Fribourg, CH-1700 Fribourg, Switzerland; phone: ; fax: ( wolfram.schultz@unifr.ch). Neurobiological research in the past several decades has demonstrated the fundamental importance of dopaminergic neurotransmission in a number of behavioral processes. The focus on dopamine has started with the discovery that major degenerations of the nigrostriatal dopamine system are associated with Parkinson s disease, in which subjects suffer from severe deficits in movement, motivation, attention, and cognition. These findings have been experimentally scrutinized in a number of animal parkinsonian models using transmitterspecific neurotoxic lesions and pharmacological dopamine depletions (Burns and others 1983). In a separate line of investigation, the observation that the blockade of dopamine receptors is beneficial for a number of psychotic symptoms has demonstrated the wide involvement of dopamine neurotransmission in cognitive processes. In addition, hundreds of behavioral studies have demonstrated a prime motivational role of dopamine projections, in particular, those directed to the nucleus accumbens and frontal cortex. These systems appear to be crucially involved in the use of reward information for learning and maintaining approach and consummatory behavior (Fibiger and Phillips 1986; Robbins and Everitt 1996). This view is also supported by the recent emphasis on the role of dopamine neurotransmission in the action of major drugs of addiction, including cocaine, amphetamine, heroin, nicotine, and alcohol (Wise and Hoffman 1992; Robinson and Berridge 1993). This review describes how the motivational functions of dopamine systems can be investigated by studying the impulse activity of individual dopamine neurons. In particular, it will describe how dopamine neurons respond with short latencies to rewards and reward-related stimuli. These responses may constitute a biological correlate of a positive reinforcement signal postulated on theoretical grounds. The material reviewed will cover only briefly the cellular and membrane actions of dopamine on target neurons. Functions of Rewards Rewards are events or objects that make subjects come back for more. They contrast with punishers, which lead to passive or active avoidance behavior. Rewards have three basic functions. They elicit approach and consummatory behavior and may serve as goals of voluntary behavior. In doing so, they interrupt ongoing behavior and change the priorities of behavioral actions. Second, rewards have positive reinforcing effects. They increase the frequency and intensity of behavior leading to such objects (learning) and maintain learned behavior by pre- Volume 7, Number 4, 2001 Copyright 2001 Sage Publications ISSN THE NEUROSCIENTIST 293

2 venting extinction. This function constitutes the essence of coming back for more and relates to the notion of receiving rewards for having done something useful. In their third function, rewards induce subjective feelings of pleasure (hedonia) and positive emotional states. However, this function is difficult to investigate in animals and will not be treated here. Recent accounts indicate that associative learning does not sufficiently advance when a stimulus is simply associated with a reinforcer. Rather, the reinforcer needs to occur unpredictably, and learning slows as reinforcers become increasingly predicted. Thus, reward-driven learning depends on the discrepancy or error between the prediction of reward and its actual occurrence (Rescorla and Wagner 1972). During learning, the reward occurs unpredicted, and the prediction error is positive. Subsequently, the reward becomes increasingly predicted, the prediction error becomes smaller, and learning is slowed. When a reward is omitted, the prediction error becomes negative, and the learned behavior will undergo extinction. The notion of prediction error relates intuitively to the essence of learning, which can be viewed as the acquisition of predictions of outcomes, for example, reward. Outcomes that are different than predicted modify behavior such that the discrepancy between the outcome and its prediction tends to become smaller. Changes in predictions and behavior continue until the outcome occurs as predicted and the prediction error becomes nil. No learning, and hence no change in predictions, occurs when an outcome is perfectly predicted. Detecting Rewards and Reward-Predicting Stimuli Cell bodies of midbrain dopamine neurons are located in groups A8 (dorsal to lateral substantia nigra), A9 (pars compacta of substantia nigra), and A10 (ventral tegmental area medial to substantia nigra). These neurons release dopamine with nerve impulses from axonal varicosities in the striatum, nucleus accumbens, and frontal cortex, to name the most important sites. We recorded the impulse activity from cell bodies of single dopamine neurons during periods of 20 to 60 min with moveable microelectrodes from extracellular positions while monkeys learned and performed specifically designed behavioral tasks. The neurons can be easily distinguished from other midbrain neurons by their characteristic polyphasic, relatively long impulses discharged at low frequencies (Schultz 1986). Dopamine neurons consistently fail to show major covariations with movements (DeLong and others 1983; Schultz and others 1983; Schultz 1986). By contrast, they show phasic activations with latencies of 50 to 110 ms and durations of < 200 ms following primary rewards and conditioned, reward-predicting visual, auditory, and somatosensory stimuli. They also show an activation-depression response to novel or intense stimuli and to stimuli closely resembling rewarding events. The responses to these different stimuli are very similar and occur in 60% to 80% of neurons in groups A8, A9, and A10 in a range of behavioral situations, whereas the remaining dopamine neurons do not respond at all. Tested situations include classical and operant conditioning, various simple and choice reaction time tasks, direct and delayed go-nogo tasks, spatial delayed responding, spatial delayed alternation, visual discrimination, and self-initiated movements. Neurons respond slightly more frequently and stronger in medial midbrain regions, such as the ventral tegmental area (A10) and medial substantia nigra, as compared with more lateral regions, a moderate difference that only sometimes reaches statistical significance. Thus, the dopamine response constitutes a relatively homogeneous population signal that is graded by the response magnitude of individual neurons and by the fractions of neurons responding. Phasic activations of dopamine neurons occur when animals touch a morsel of hidden food (Fig. 1A) or when drops of liquid are delivered to their mouth outside of behavioral tasks (Schultz 1986; Romo and Schultz 1990) or during learning episodes (Fig. 2) (Ljungberg and others 1992; Schultz and others 1993; Hollerman and Schultz 1998). They respond in a similar fashion to different food objects and to liquid rewards, but not to nonfood objects (Fig. 1B). Dopamine neurons are also activated by conditioned stimuli that have become valid reward predictors following repeated and contingent pairing with rewards in classical or operant conditioning procedures. Concurrent with the development of the dopamine response to a reward-predicting stimulus during learning, the response to the predicted reward itself is lost, as if the response is transferred from the reward to the reward-predicting stimulus (Fig. 3). The transfer is observed when the reward occurs surprisingly outside of any behavioral task or during individual learning episodes and subsequently becomes predicted after a task is fully acquired. Dopamine neurons show depressions or activationdepression responses following novel or intense stimuli (Strecker and Jacobs 1985; Ljungberg and others 1992; Horvitz and others 1997). However, they rarely show the short-latency, phasic response to other attention-generating events, such as aversive stimuli (Mirenowicz and Schultz 1996; Guarraci and Kapp 1999), and they are depressed by the attention-generating omission of reward (Ljungberg and others 1992; Schultz and others 1993). These data suggest that dopamine neurons are not in general activated by attention-generating, salient stimuli, although they may code attentional components associated with rewards. Some dopamine neurons show depressions or activation-depression responses to non-reward-predicting neutral stimuli or even aversive stimuli when these are physically similar to rewarding stimuli and occur in similar contexts and close associations with them (Mirenowicz and Schultz 1996; Schultz and Romo 1990). Generalizing dopamine responses are not observed when nonrewarded stimuli are physically rather dissimilar from 294 THE NEUROSCIENTIST Reward Signaling by Dopamine Neurons

3 Fig. 1. Dopamine responses to primary food reward. A, Response to the touch of food in the absence of phasic stimuli predicting the reward. The food is invisible to the animal but can be touched by its hand underneath the cover. The perievent time histogram of neuronal impulses is shown above the raster display in which each dot denotes the time of a neuronal impulse in reference to movement onset (release of resting key). B, Differential response to touch of a piece of apple (top) but not to touch of a bare wire (bottom) or inedible objects known to the animal. A side view of the animal s hand entering the covered food box is shown to the left. Touch of objects produces an electric pulse that provides a temporal reference (vertical line). Data are from Romo and Schultz (1990). rewarding stimuli or even have different sensory modalities, or when stimuli occur in contexts of low reward incidence (Romo and Schultz 1990). Only a few dopamine neurons show the phasic activations following aversive stimuli. The tested events include primary aversive stimuli, such as non-noxious air puffs to the hand or hypertonic saline to the mouth; classically conditioned aversive stimuli; and learned visual or auditory stimuli in active avoidance tasks in which animals release a key to avoid an air puff or a drop of hypertonic saline (Mirenowicz and Schultz 1996; Guarraci and Kapp 1999). Although most of these stimuli are non-noxious, they are aversive in that they disrupt behavior and induce active avoidance behavior. However, the considerable insensitivity to aversive stimuli concerns only the short-latency responses, and dopamine neurons react to pain pinch stimuli, electric shock, and stressful events with comparatively slower decreases or increases of impulse activity (Chiodo and others 1979; Schultz and Romo 1987) or increases in striatal dopamine release (Louilot and others 1986; Abercrombie and others 1989; Doherty and Gratton 1992; Young and others 1992). Taken together, most dopamine neurons show phasic activations following food rewards, liquid rewards, and conditioned, reward-predicting stimuli. They show depressions or smaller activation-depression responses following stimuli that resemble reward-predicting stimuli or are novel or particularly intense. Only few neurons show phasic activations following aversive stimuli. Thus, dopamine neurons appear to label environmental stimuli with an appetitive tag, predict and detect rewards, and signal alerting, motivating events. A Reward Prediction Error Signal and Its Potential Use All responses to rewards and reward-predicting stimuli depend on event predictability. Rewards are only effective in activating dopamine neurons when they are not predicted by phasic stimuli. Dopamine neurons are depressed at the habitual time of reward when a predicted reward fails to occur following an error of the animal, withholding by the experimenter, or delayed delivery (Ljungberg and others 1992; Schultz and others 1993). The depression occurs in the absence of a stimulus immediately preceding the omitted reward. This reflects an expectation process based on an internal clock that concerns the precise time of the predicted reward. On the other hand, an activation follows a reward that is presented at a different time than predicted. These data suggest that the predictions influencing dopamine neurons concern both the occurrence and the time of reward. A sudden variation in either of the two parameters will lead to a dopamine response, even in a general, rewardpredicting environmental context. The dopamine reward response appears to indicate to what extent a reward occurs differently than predicted, which can be termed a discrepancy or error in the prediction of reward. The dopamine neurons report rewards Volume 7, Number 4, 2001 THE NEUROSCIENTIST 295

4 Fig. 2. Changes of dopamine population response to reward during learning. Animals performed in an operant two-choice discrimination task in which two familiar or two novel stimuli were presented in separate trial blocks. Top. Absence of reward response when familiar pictures were used. Bottom. Strong activations during initial learning trials before reaching the learning criterion (the second correct trial in the first series of four consecutive correct trials). The reward response decreased progressively after the criterion. Population activity was averaged from 54 neurons tested in familiar and learning conditions with 20 correct trials. Numbers indicate the first 5 trials (familiar pictures, top) or refer to trials after reaching criterion. Reprinted from Hollerman and Schultz (1998). Copyright (1998) Macmillan Magazines Ltd. relative to their prediction, rather than signaling rewards unconditionally. They appear to be feature detectors for the goodness of environmental events relative to prediction, being activated by rewarding events that are better than predicted, remaining uninfluenced by events that are as good as predicted, and being depressed by events that are worse than predicted (Fig. 3). Their response appears to follow the equation DopamineResponse = RewardOccurrence RewardPrediction. However, dopamine neurons fail to discriminate between different rewards and thus appear to emit an alerting message about the surprising presence or absence of rewards rather than indicating the particular nature of each reward. They process the time and prediction of rewards but not the nature of the particular reward. The prediction error is also coded during learning episodes (Hollerman and Schultz 1998). The moderately bursting, short-duration, nearly synchronous response of the majority of dopamine neurons leads to optimal, simultaneous dopamine release from the majority of closely spaced varicosities in the striatum 296 THE NEUROSCIENTIST Reward Signaling by Dopamine Neurons

5 Fig. 3. Dopamine neurons report an error in the prediction of reward. Top. A drop of liquid reward (R) occurs although no reward is predicted at this time. The occurrence of reward thus constitutes a positive error in the prediction of reward. The dopamine neuron is activated by the unpredicted occurrence of the liquid. Middle. A learned stimulus (conditioned stimulus, CS) predicts a reward, and the reward occurs according to the prediction; hence, no error in the prediction of reward. The dopamine neuron does not respond to the predicted reward (right) but is activated following the reward-predicting stimulus (left). Bottom. Omission of predicted reward. The activity of the dopamine neuron is depressed exactly at the time when the reward would have occurred. Each line of dots shows one trial, the original sequence in each panel being from top to bottom. Reprinted from Schultz and others (1997). Copyright (1997) American Association for the Advancement of Science. Volume 7, Number 4, 2001 THE NEUROSCIENTIST 297

6 and frontal cortex (Gonon 1988; Kawagoe and others 1992; Garris and others 1994; Garris and Wightman 1994; Gonon 1997). The short puff of dopamine quickly leads to regionally homogenous concentrations likely to influence the dendrites of striatal and cortical neurons. In this way, the dopamine reward prediction error message is broadcast as a divergent, rather global reinforcement signal to the striatum, nucleus accumbens, and frontal cortex, phasically influencing a maximum number of synapses involved in the processing of stimuli and actions leading to reward. The reduction of dopamine release induced by depressions with omitted rewards would reduce the tonic stimulation of dopamine receptors by ambient dopamine released through spontaneous impulses. In view of the crucial role of prediction errors during learning (Rescorla and Wagner 1972), the phasic dopamine response reporting a reward prediction error may constitute an explicit teaching signal for approach learning (Schultz and others 1997). A model may be established that formalizes the synaptic influences of dopamine neurons on neurotransmission in target structures, such as the striatum and frontal cortex. The weights of striatal synapses may undergo short- or long-term modifications according to an anatomically based three-factor Hebbian learning rule. According to the basic anatomic arrangement (Freund and others 1984; Williams and Goldman-Rakic 1993; Smith and others 1994), the dopamine influence may involve a functional triad, comprising excitatory cortical terminals at the tip of dendritic spines as inputs, dendritic spines on postsynaptic neurons leading to output at the soma, and the dopamine reinforcement signal arriving at the varicosities contacting the same dendritic spines (Fig. 4). Dopamine-dependent plasticity has been shown to exist in the striatum and frontal cortex (Calabresi and others 1992; Wickens and others 1996; Calabresi and others 1997; Otani and others 1999; Kerr and Wickens 2001; Tang and others 2001). The dopamine response resembles closely (Montague and others 1996) the teaching signal used by the very effective temporal difference reinforcement model developed by Sutton and Barto (1981) on purely behavioral and theoretical grounds. Indeed, neuronal networks using this type of teaching signal learn to play world-class backgammon (Tesauro 1994) and acquire serial movements and spatial delayed response tasks in a manner consistent with the behavior of monkeys in the laboratory (Suri and Schultz 1998, 1999). In a more general way, the neuronal computation and use of prediction errors may contribute to the self-organization of goal-directed behavior. Brain mechanisms establish predictions, compare current inputs with predictions from previous experience, and generate and emit a prediction error signal once a mismatch is detected (Schultz and Dickinson 2000). The error signal may act as an impulse for synaptic changes that lead to subsequent changes in predictions and behavioral reactions. The process is reiterated until behavioral outcomes Fig. 4. Basic design of hypothetical influence of dopamine prediction error signal on neurotransmission in the striatum. Synaptic inputs from a single dopamine axon X and two cortical axons A and B contact a typical medium-size spiny striatal neuron I. Corticostriatal transmission can be modified by dopamine input X contacting indiscriminately the stems of dendritic spines that are also contacted by specific cortical inputs A and B. In the present example, cortical input A, but not B, is active at the same time as dopamine neuron X (shaded area), for example, following a reward-related event. This could lead to a modification of the A I transmission but leave the B I transmission unaltered. Thus, the dopamine reward signal X would modify the conjointly active Hebbian synapse A I but leave the inactive synapse B I unchanged. Learning at the A I and B I synapses would comply with the three-factor Hebbian learning rule ω = ε rio, where ω is synaptic weight, ε is a learning constant, r is a dopamine reward signal, i is a cortical input activity, and o is a striatal neuron activity. Anatomical drawing is based on anatomical data (Freund and others 1984; Williams and Goldman-Rakic 1993; Smith and others 1994). match the predictions and the prediction error becomes nil. In the absence of a prediction error, there would be no signal for modifying synapses, and synaptic transmission remains unchanged and stable. Different Modes of Dopamine Function The phasic reward response with a time course of tens of milliseconds may be a good physiological correlate for the proposed function of dopamine systems in motivational processes (Fibiger and Phillips 1986; Robbins and Everitt 1996; Wise and Hoffman 1992; Robinson and Berridge 1993). However, a compromise in reward function would not explain the majority of parkinsonian movement deficits. In addition, parkinsonian deficits can 298 THE NEUROSCIENTIST Reward Signaling by Dopamine Neurons

7 Fig. 5. Different temporal operating modes for different dopamine functions. be ameliorated to some extent by increasing the tonic dopamine receptor stimulation, although this cannot in any simple manner restitute the phasic information transmitted by dopamine neuronal impulses. It appears that the dopamine systems may act at several different time scales and have a different function at each scale. At the fastest scale, dopamine neurons signal a nearly homogeneous reward signal. At the slowest scale, they seem to have a tonic enabling function on a large variety of motor, cognitive, and motivational processes that are deficient in parkinsonian patients and experimentally lesioned animals. At an intermediate scale, dopamine reactions occur in the range of seconds to minutes; are measured by electrophysiology, in vivo dialysis, and voltammetry; and are involved in a considerable range of behavioral processes, including feeding and drinking behavior, aversive events, stress, and social behavior (Louilot and others 1986; Schultz and Romo 1987; Abercrombie and others 1989; Doherty and Gratton 1992; Young and others 1992) (Fig. 5). The tonic enabling dopamine function is based on low, sustained, extracellular dopamine concentrations in the striatum (5 10 nm) and other dopamine-innervated areas. The ambient dopamine concentration is regulated locally within a narrow range by spontaneous impulses, synaptic overflow, reuptake transport, metabolism, autoreceptor-controlled release and synthesis, and presynaptic transmitter interaction (Chesselet 1984). The tonic stimulation of dopamine receptors should be neither too low nor too high to permit an optimal function of a given brain region (Brozoski and others 1979; Simon and others 1980; Murphy and others 1996; Elliott and others 1997). Other neurotransmitters exist in similarly low ambient concentrations, such as glutamate in the striatum, cerebral cortex, hippocampus, and cerebellum; aspartate and GABA in the striatum and frontal cortex; and adenosine in the hippocampus. Neurons in many brain structures are apparently bathed in a soup of neurotransmitters, which have powerful, specific, physiologic effects on neuronal excitability. Given the general importance of tonic extracellular concentrations of neurotransmitters, it appears that the wide range of parkinsonian symptoms may not be due to deficient transmission of reward information by dopamine neurons but reflect a malfunction of striatal and cortical neurons due to impaired enabling with reduced ambient dopamine. Dopamine neurons would not be actively involved in the wide range of processes deficient in parkinsonism Volume 7, Number 4, 2001 THE NEUROSCIENTIST 299

8 but provide the crucial background concentration of dopamine to maintain proper functioning of striatal and cortical neurons involved in these processes. Distributed Reward Processing Information concerning food and fluid rewards is also processed in brain structures other than dopamine neurons, such as dorsal and ventral striatum, subthalamic nucleus, amygdala, dorsolateral prefrontal cortex, orbitofrontal cortex, and anterior cingulate cortex. However, these structures do not appear to emit a global reward prediction error signal similar to that of dopamine neurons. Neurons in these structures show transient responses following the delivery of rewards, often irrespective of reward prediction, and transient responses to reward-predicting cues. Many of these neurons differentiate well between different food or fluid rewards but not between the spatial locations or visual aspects of the stimuli signaling the rewards. They may be involved in the detection and perception of the specific nature of rewarding events (Schultz 1998, 2000). Some reward responses depend on reward unpredictability, as they are reduced or absent when the reward is predicted by a conditioned stimulus (Hollerman and others 1998), although it is unclear whether they signal global prediction error signals similar to dopamine neurons. Other groups of neurons in the mentioned structures show sustained activations over several seconds preceding predictable rewards and may therefore be involved in the expectation of imminent rewards. The activity of a considerable number of neurons in the striatum and dorsolateral frontal cortex is related to various sensory and motor processes but is also influenced by expected rewards (Hollerman and others 1998; Kawagoe and others 1998). These neurons show changes in sustained activity during a preparatory delay period preceding arm or eye movements, or during the execution of such movements. Thus, both the reward and the movement to the reward are represented by these neurons, as if they integrate reward information into processes mediating the behavior leading to the reward. These reward-integrating activities may contribute to mechanisms in which the expected reward is represented at the neuronal level and can influence neuronal processes underlying the behavior toward that reward. These activities are in general compatible with a goal directional account, according to which, rewards may serve as goals for voluntary behavior if information about the reward is present while behavioral reactions toward the reward are being prepared and executed (Dickinson and Balleine 1994). It thus appears that different components of reward events are signaled by different neuronal systems in the brain. The phasic responses of dopamine neurons seem to signal primarily reward prediction errors and may be involved in short-term or long-term changes in behavior following rewards. Neurons in the other mentioned cortical and subcortical structures may be involved in the perception and expectation of rewards and in the use of reward information for structuring behavior directed at rewards. Although this scheme seems to contrast the reward functions between dopamine neurons and the other structures, it appears that the appropriate and effective behavioral reaction to positive motivating events would ideally involve many of these mechanisms in parallel. Drug Addiction With the advancing knowledge on neuronal reward mechanisms, we may gain an understanding of how artificial drug rewards may exert their profound influence on behavior. We may ask whether drugs modify existing neuronal responses to natural rewards or alter responses to nonrewarding environmental events, or constitute rewards in their own right and as such engage existing neuronal reward mechanisms. However, this somewhat schematic approach may constitute an oversimplification that may have to be modified with increasing neurobiological knowledge. Psychopharmacological studies have identified the dopamine system and the ventral striatum, including the nucleus accumbens, as some of the critical structures for the action of major drugs of abuse, such as opiates, nicotine, cocaine, and amphetamine (Wise and Hoffman 1992; Robinson and Berridge 1993). These drugs increase the activity and release of dopamine (opiates, nicotine) or block the reuptake of dopamine and thus increase the extracellular dopamine concentration in target structures (cocaine, amphetamine). Neurophysiological studies on animals self-administering drugs indicate that some neurons in the ventral striatum of rats and monkeys do indeed treat drugs as rewards in their own right. These neurons are activated by drug delivery and show anticipatory activity during the expectation of drugs, as other neurons in these structures do with natural rewards (Carelli and others 1993; Chang and others 1994; Bowman and others 1996; Peoples and others 1997; Peoples and others 1998; Carelli and others 2000). Behavior-related activity appears to be influenced by drugs in a somewhat similar manner as by natural rewards (West and others 1997), possibly reflecting a goal-directional mechanism by which the expected outcome influences the neuronal activity related to the behavior leading to that outcome. It is thus possible that some drug rewards, after entering the brain via blood vessels, exert a chemical influence on neurons in the reward structures that in some way mimics or corresponds to the influence of neurotransmitters released by natural rewards, in particular, dopamine. This would be consistent with the observation that major drugs of abuse activate the dopamine systems. Mimicking or boosting the phasic dopamine reward prediction error might generate a teaching signal and produce lasting behavioral changes through synaptic modifications. Alternatively, it is possible that the reward value of drugs is assessed in other, presently unidentified, structures and transmitted via existing reward pathways, including dopamine neurons, to 300 THE NEUROSCIENTIST Reward Signaling by Dopamine Neurons

9 executive structures. The final common path for natural reward- and drug-seeking behavior may involve the influence of the neuronal reward message on activity underlying the behavior directed at the rewards. References Abercrombie ED, Keefe KA, DiFrischia DS, Zigmond MJ Differential effect of stress on in vivo dopamine release in striatum, nucleus accumbens, and medial frontal cortex. J Neurochem 52: Bowman EM, Aigner TG, Richmond BJ Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. J Neurophysiol 75: Brozoski TJ, Brown RM, Rosvold HE, Goldman PS Cognitive deficit caused by regional depletion of dopamine in prefrontal cortex of rhesus monkey. Science 205: Burns RS, Chiueh CC, Markey SP, Ebert MH, Jacobowitz DM, Kopin IJ A primate model of parkinsonism: selective destruction of dopaminergic neurons in the pars compacta of the substantia nigra by N-methyl-4-phenyl-1,2,3,6-tetrahydropyridine. Proc Natl Acad Sci U S A 80: Calabresi P, Maj R, Pisani A, Mercuri NB, Bernardi G Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J Neurosci 12: Calabresi P, Saiardi A, Pisani A, Baik JH, Centonze D, Mercuri NB, and others Abnormal synaptic plasticity in the striatum of mice lacking dopamine D2 receptors. J Neurosci 17: Carelli RM, Ijames SG, Crumling AJ Evidence that separate neural circuits in the nucleus accumbens encode cocaine versus natural (water and food) reward. J Neurosci 20: Carelli RM, King VC, Hampson RE, Deadwyler SA Firing patterns of nucleus accumbens neurones during cocaine self-administration. Brain Res 626: Chang JY, Sawyer SF, Lee RS, Woodward DJ Electrophysiological and pharmacological evidence for the role of the nucleus accumbens in cocaine self-administration in freely moving rats. J Neurosci 14: Chesselet MF Presynaptic regulation of neurotransmitter release in the brain: facts and hypothesis. Neuroscience 12: Chiodo LA, Caggiula AR, Antelman SM, Lineberry CG Reciprocal influences of activating and immobilizing stimuli on the activity of nigrostriatal dopamine neurons. Brain Res 176: DeLong MR, Crutcher MD, Georgopoulos AP Relations between movement and single cell discharge in the substantia nigra of the behaving monkey. J Neurosci 3: Dickinson A, Balleine B Motivational control of goal-directed action. Anim Learn Behav 22:1 18. Doherty MD, Gratton A High-speed chronoamperometric measurements of mesolimbic and nigrostriatal dopamine release associated with repeated daily stress. Brain Res 586: Elliott R, Sahakian BJ, Matthews K, Bannerjea A, Rimmer J, Robbins TW Effects of methylphenidate on spatial working memory and planning in healthy young adults. Psychopharmacology 131: Fibiger HC, Phillips AG Reward, motivation, cognition: psychobiology of mesotelencephalic dopamine systems. In: Bloom FE, editor. Handbook of physiology the nervous system IV. Baltimore: Williams and Wilkins. p Freund TF, Powell JF, Smith AD Tyrosine hydroxylase-immunoreactive boutons in synaptic contact with identified striatonigral neurons, with particular reference to dendritic spines. Neuroscience 13: Garris PA, Ciolkowski EL, Pastore P, Wightman RM Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain. J Neurosci 14: Garris PA, Wightman RM Different kinetics govern dopaminergic transmission in the amygdala, prefrontal cortex, and striatum: an in vivo voltammetric study. J Neurosci 14: Gonon F Nonlinear relationship between impulse flow and dopamine released by rat midbrain dopaminergic neurons as studied by in vivo electrochemistry. Neuroscience 24: Gonon F Prolonged and extrasynaptic excitatory action of dopamine mediated by D1 receptors in the rat striatum in vivo. J Neurosci 17: Guarraci FA, Kapp BS An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential Pavlovian fear conditioning in the awake rabbit. Behav Brain Res 99: Hollerman JR, Schultz W Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci 1: Hollerman JR, Tremblay L, Schultz W Influence of reward expectation on behavior-related neuronal activity in primate striatum. J Neurophysiol 80: Horvitz JC, Stewart T, Jacobs BL Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res 759: Kawagoe KT, Garris PA, Wiedemann DJ, Wightman RM Regulation of transient dopamine concentration gradients in the microenvironment surrounding nerve terminals in the rat striatum. Neuroscience 51: Kawagoe R, Takikawa Y, Hikosaka O Expectation of reward modulates cognitive signals in the basal ganglia. Nature Neurosci 1: Kerr JN, Wickens JR Dopamine D-1/D-5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. J Neurophysiol 85: Ljungberg T, Apicella P, Schultz W Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67: Louilot A, Le Moal M, Simon H Differential reactivity of dopaminergic neurons in the nucleus accumbens in response to different behavioral situations. An in vivo voltammetric study in free moving rats. Brain Res 397: Mirenowicz J,Schultz W Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379: Montague PR, Dayan P, Sejnowski TJ A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16: Murphy BL, Arnsten AF, Goldman-Rakic PS, Roth RH Increased dopamine turnover in the prefrontal cortex impairs spatial working memory performance in rats and monkeys. Proc Natl Acad Sci U S A 93: Otani S, Auclair N, Desce JM, Roisin MP, Crepel F Dopamine receptors and groups I and II mglurs cooperate for long-term depression induction in rat prefrontal cortex through converging postsynaptic activation of MAP kinases. J Neurosci 19: Peoples LL, Gee F, Bibi R, West MO Phasic firing time locked to cocaine self-infusion and locomotion: dissociable firing patterns of single nucleus accumbens neurones in the rat. J Neurosci 18: Peoples LL, Uzwiak AJ, Gee F, West MO Operant behaviour during sessions of intravenous cocaine infusion is necessary and sufficient for phasic firing of single nucleus accumbens neurones. Brain Res 757: Rescorla RA, Wagner AR A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF, editors. Classical conditioning II: current research and theory. New York: Appleton Century Crofts. p Robbins TW, Everitt BJ Neurobehavioural mechanisms of reward and motivation. Curr Opin Neurobiol 6: Robinson TE, Berridge KC The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res Rev 18: Romo R, Schultz W Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63: Schultz W Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J Neurophysiol 56: Volume 7, Number 4, 2001 THE NEUROSCIENTIST 301

10 Schultz W Predictive reward signal of dopamine neurons. J Neurophysiol 80:1 27. Schultz W Multiple reward systems in the brain. Nat Rev Neurosci 1: Schultz W,Apicella P, Ljungberg T Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13: Schultz W,Dayan P, Montague RR A neural substrate of prediction and reward. Science 275: Schultz W,Dickinson A Neuronal coding of prediction errors. Ann Rev Neurosci 23: Schultz W,Romo R Responses of nigrostriatal dopamine neurons to high intensity somatosensory stimulation in the anesthetized monkey. J Neurophysiol 57: Schultz W, Romo R Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J Neurophysiol 63: Schultz W,Ruffieux A,Aebischer P The activity of pars compacta neurons of the monkey substantia nigra in relation to motor activation. Exp Brain Res 51: Simon H, Scatton B, LeMoal M Dopaminergic A10 neurons are involved in cognitive functions. Nature 286: Smith Y, Bennett BD, Bolam JP, Parent A, Sadikot AF Synaptic relationships between dopaminergic afferents and cortical or thalamic input in the sensorimotor territory of the striatum in monkey. JComp Neurol 344:1 19. Strecker RE, Jacobs BL Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res 361: Suri RE, Schultz W Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res 121: Suri RE, Schultz W A neural network with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91: Sutton RS, Barto AG Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88: Tang KC, Low MJ, Grandy DK, Lovinger DM Dopamine-dependent synaptic plasticity in striatum during in vivo development. Proc Natl Acad Sci U S A 98: Tesauro G TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comp 6: West MO, Peoples LL, Michael AJ, Chapin JK, Woodward DJ Low-dose amphetamine elevates movement-related firing of rat striatal neurones. Brain Res 745: Wickens JR, Begg AJ, Arbuthnott GW Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70:1 5. Williams SM, Goldman-Rakic PS Characterization of the dopaminergic innervation of the primate frontal cortex using a dopamine-specific antibody. Cereb Cortex 3: Wise RA, Hoffman DC Localization of drug reward mechanisms by intracranial injections. Synapse 10: Young AMJ, Joseph MH, Gray JA Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48: THE NEUROSCIENTIST Reward Signaling by Dopamine Neurons