Neurobehavioural mechanisms of reward and motivation

Transcription

1 228 Neurobehavioural mechanisms of reward and motivation Trevor W Robbinsl and Barry J Everitt* The analysis of the behavioural and neural mechanisms of reinforcement and motivation has benefited from the recent application of learning theory and better anatomical knowledge of the connectivity of certain key neural structures, such as the nucleus accumbens. This progress has enabled the dissection of motivational processes into components that can begin to be related to the functioning of specific limbic cortical structures that project to different compartments the ventral striatum. Address Department of Experimental Psychology, University of Cambridge, Downing Street, Cambridge, CB2 3EB, UK lwr2@hermes.cam.ac.uk 2 BJEl O@cus.cam.ac.uk Abbreviations AP-5 2-amino-5-phosphonopentanoic acid CNGX 6-cyano-7-nitroquinoxaline-2,3-dione Dl type 1 dopamine receptor D2 type 2 dopamine receptor DNGX 6,7-dinitroquinoxaline-2,3-dione DOPAC dihydroxyphenylacetic acid GABA y-aminobutyric acid 5-HT 5-hydroxytryptamine 5-HIAA 5-hydroxyindoleacetic acid NMDA N-methyl-o-aspartate Current Opinion in Neurobiology 19.96, 6: Current Biology Ltd ISSN Introduction One of the ultimate goals of behavioural neuroscience research is to understand the neural basis of such subjective phenomena as pleasure in humans. At present, however, most of the viable experimental approaches toward this end are restricted to observing the behaviour of experimental animals. Indeed, it is difficult to relate the mechanisms by which classically conditioned or unconditioned incentive stimuli elicit approach or consummatory behaviour in experimental animals to subjective affective states that may accompany such behaviour. Moreover, Skinner s operational definition of a reinforcer as an event that increases the probability of an instrumental or operant response upon which it is contingent, makes no reference to subjective states. The term reward unfortunately connotes several of these overlapping theoretical constructs, including pleasure or hedonism, as well as reinforcement and incentive. Consequently, we will confine ourselves mainly to a consideration of the neural substrates of mechanisms of incentive-motivation and reinforcement [1,2**,3,4], as these terms have fewer connotations concerning subjective states than reward. of Even leaving aside the complications of the subjective aspects of motivation and reward, it is probable that further advances in characterizing the neural mechanisms underlying these processes will depend on a better understanding of the psychological basis of goal-directed or instrumental behaviour [?I. The main theme of this review is co show how neurobiological advances are being combined with a growing behavioural sophistication to enhance our understanding of the field. Before discussing the progress of the past few years, we briefly provide some background on the relevant areas of learning theory and the neurobiology of motivation and reinforcement. Associative structures underlying motivation There is behavioural evidence for different mechanisms governing appetitive behaviour (e.g. locomotor approach responses) in anticipation of a reinforcer and the consummatory behaviour (e.g. ingestion or sexual mounting) performed in its presence [l], in keeping with early distinctions made by ethologists. It is important to realize that such appetitive behaviour can be subject to both instrumental and classical (Pavlovian) conditioning contingencies (see [Z**]). Three major forms of associative mechanisms potentially control this behaviour. In the first of these, Pavlovian associations between the stimuli in the test situation and the reinforcer may exert direct effects on performance according to the current motivational state. Thus, for example, stimuli conditioned to food presentation may elicit increases in appetitive behaviour, such as locomotor activity in the hungry rat. A second associative process involves knowledge of the contingency between voluntary actions and the reinforcing outcome. Behaviour governed by this process is responsive to fluctuations in the incentive value of the reinforcer, according to previous experience. Thus, shifts in food deprivation will only affect instrumental performance if the animal has previously experienced food in that deprivation state. This may account for a notable dissociation of appetitive and instrumental behaviour, the phenomenon of resistance to satiation in which instrumental responding persists in the sated state [Z ]. Evidence also exists for yet a third associative mechanism controlled by stimulus-response associations that. give operant behaviour a habit-like or automatic quality that is impervious to fluctuations in the value of the reinforcer [3]. How such theoretical distinctions are reflected in the organization of neural systems that control motivation and reward is an important issue, but one that is only beginning to be addressed [4]. Reinforcers also contribute to mnemonic processes; for example, they can effect the retroactive consolidation of

2 Neurobehavioural mechanisms of reward and motivation Robbins and Ever&t 229 Figure Current Opmon in Neurobiology Schematic diagram of limbic-striatal-pallidal circuitry implicated in reward processes. Limbic and cortical processing of affective information occurs from the left-hand side, to interact with the dopamine-dependent functioning of the nucleus accumbens. The different compartments of this structure ( shell and core ; see 19.1) act as interfaces to motor (behavioural), endocrine and autonomic outputs at the hypothalamic and brain stem levels (bottom, right-hand corner). Note that the pedunculopontine tegmental nuclei (PPTg) is a major nucleus within what Mogenson et a/. [7] called the mesencephalic locomotor region. Cortical projections of the dopaminergic neurons and the cholinergic neurons of the nucleus basalis magnocellularis (NBM) are not shown. A9 and AlO, mesencephalic dopamine cell groups. long-term memories [4] or, alternatively, can enter into short-term associations that affect response choice through processes of working memory. Identification of specific neural systems for all of these hypothetical processes would support their validity at a psychological level. Neurobiology of motivation and reinforcement Interest in the neural substrates of reinforcement was boosted by the discovery of intra-cranial self-stimulation, which had the added advantage of being especially amenable to a psychophysical analysis of reinforcing effects. A central map of positively and negatively reinforcing brain sites was constructed that implicated structures within the limbic system, such as the amygdala, septum and prefrontal cortex, as well as the catecholaminergic neuronal projections from the brain stem. Subsequently, there was an emphasis on the role of the mesolimbic dopamine system projecting from the ventral tegmental area to the ventral striatum, including the nucleus accumbens. This direction of research was further encouraged by discoveries that the reinforcing effects of psychomotor stimulant drugs such as amphetamine and cocaine were probably mediated by dopamine-dependent mechanisms of the ventral striatum [S]. This led to the notions that, first, the reinforcing effects of other drugs of abuse may be mediated via this same system [6], and second, that the reinforcing effects of natural reinforcers such as food and sex were similarly dependent upon it [6]. Thus, the effects of brain-stimulation reward and psychomotor stimulant drugs were hypothesized to mimic aspects of the reinforcing effects of these natural reinforcers. Both of these hypotheses are still controversial, not least because of the problems of distinguishing genuine motivational effects from alterations in motor function. We will concentrate on the second hypothesis because the companion review by Wise (see, in this issue, pp ) focuses on addiction. A further important development has been the realization that the ventral striatum forms part of a system that receives afferents from several limbic cortical structures, including the basolateral amygdala, the hippocampal formation and the prefrontal cortex, and that, in turn, projects to output structures such as the ventral pallidum (Figure 1). This limbic-ventral-striatopallidal system could in principle provide, in Mogenson et al.5 memorable words, a neural mechanism by which motivation gets translated into action (page 91 in [7]). Subsequent evidence demonstrated that one of the functions of certain limbic afferents (i.e. basolateral and lateral amygdala) is to convey associative information about environmental stimuli that predict the occurrence of reinforcers to systems determining response selection [g]. In some

3 230 Cognitive neuroscience circumstances, therefore, reinforcement can be conceived as a form of interaction between the excitatory amino acid (glutamatergic) inputs from these limbic afferents and the ascending dopamine system, which together determine the output of the ventral striatal GABAergic medium spiny neurons projecting to the globus pallidus (i.e. ventral pallidum). Interest has also focused on the distinct functions of other various output structures innervated by ventral striatal outflow, including the cholinergic cells of the nucleus basalis and the brain stem pedunculopontine nuclei. A recent major advance has been the anatomical discovery that the nucleus accumbens is a heterogeneous structure, with a medial shell and more lateral core that can be distinguished using anatomical, neurochemical and functional criteria [9*]. These discoveries may help us to decompose different aspects of the reinforcement process. Several foci have dominated work over the past two years. One has been the utilization of techniques such as in r& microdialysis and voltammetry to provide on-line measures of activity of neurotransmitter systems during reinforced behaviour in rats. After a period of preoccupation with the technical difficulties involved in accurately measuring dopamine release, the analysis is now being extended to more sophisticated behavioural settings, and to anatomical structures other than the nucleus accumbens, as well as to neurotransmitters other than dopamine. This neurochemical analysis, together with electrophysiological studies of single cells, has enabled researchers to ask under what conditions are the dopamine neurons normally activated. Not all the evidence suggests that these conditions are universally rewarding : some evidence exists to show that the activity of mesencephalic dopamine neurons or dopamine release from their terminals may also change in response to stressors. The possibilities therefore exist that, first, enhanced activity of the dopamine neurons does not always necessarily lead to positively reinforcing effects, and second, that exposure to stressors may affect the reinforcing efficacy of stimuli operating via the mesolimbic dopamine system. We will not have the opportunity to cover the latter area in any detail, although it forms an important area for future research [10,11*,12*,13,14]. Understanding the necessary and sufficient conditions that determine when a stimulus acts as a positive reinforcer or incentive, and its basis in terms of both neuronal interactions and the functioning of a discrete neural system will be our main focus. Functions of the ventral striatum in reward-related processes Comparisons between drugs or brain-stimulation reward as reinforcers and natural reinforcers such as food continue to show significant overlap, but also differences, in response to experimental manipulations of the ventral striatum: for example, in response to dopamine receptor antagonists or dopamine depletion from the nucleus accumbens [15, Nevertheless, recent articles have delineated important functions of the nucleus accumbens in the motivation for food, water, sex, and maternal behaviour [8,19,20,21]. The behavioural distinction between appetitive (or preparatory) and consummatory phases of behaviour (see e.g. [8,21]) is supported by the effects of manipulations of the nucleus accumbens, which often preferentially affect the appetitive phase (but see also below). Thus, using a paradigm that pits a choice between instrumental and consummatory behaviour (in this case, eating) in various ways, Salamone and colleagues [ZP] have shown that dopamine depletion from the nucleus accumbens biases an animal from instrumental responding for a preferred food reinforcer to consumption of freely available but normally less preferred food, confirming other dissociations between appetitive and consummatory behaviour following manipulations of the mesolimbic dopamine system. In a related experiment [23], the dopamine-depleted rats preferred to exert less behavioural effort to gain access to lower densities of food. These results indicate that global hypotheses about a role for dopamine or the nucleus accumbens in reward or motivation are too simple, and appear, rather, to indicate fundamental deficits in instrumental behaviour. As indicated above, however, the mechanisms governing instrumental responding are complex [PI. For example, Balleine and Killcross [24*] have varied the incentive value of food to investigate whether the reduced operant responding for food in rats with excitotoxic lesions of the nucleus accumbens arises specifically from an insensitivity to response-reinforcer associations. Their evidence is not in favour of this possibility, but rather suggests that these rats exhibit more general impairments in incentive-motivation, which they term affective arousal. The analysis of possible conflicting interpretations in the results of this experiment [24*] and those of Salamone and colleagues [22*,23] would be premature before each paradigm has been studied with both excitotoxic (i.e. cell body) and dopamine-depleting lesions of the nucleus accumbens. Conditioned reinforcement and the ventral striatum The nucleus accumbens has been known for some time to play an important role in mediating the effects on behaviour of conditioned reinforcers, that is, those stimuli that gain their reinforcing efficacy by means of their association with unconditioned reinforcers such as food, water and drugs. As well as acting as reinforcers for instrumental responding, such stimuli may also function as conditioned incentive stimuli, inducing approach

4 Neurobehavioural mechanisms of reward and motivation Robbins and Everiti 231 behaviour and generally facilitating behavioural output. Infusion of amphetamine into the nucleus accumbens increases responding that is reinforced by stimuli formerly paired with water or sucrose, an effect shown to have behavioural, neural and neurochemical specificity [8]. Such effects are known to depend on dopamine-dependent interactions between the ventral striatum and its limbic afferents, particularly those from the basolateral amygdala [8]; however, the involvement of specific dopamine receptors has only recently been investigated. Ranaldi and Beninger [ZS] have shown that the effects of amphetamine on responding with conditioned reinforcement are attenuated by both Dl and D2 dopamine receptor antagonists when administered systemically, similar to the effects shown by other authors [261 when the drugs are administered intra-accumbens. These data should be compared with the complex effects of dopamine receptor agonists and antagonists, or mesolimbic dopamine depletion, on behaviour maintained by other forms of reinforcement, including intravenous cocaine [ls*], food [15*,16,17], ethanol [17] and brain-stimulation reward [18] (see above). The results of some of these experiments [15,18,26] suggest that Dl dopamine receptors more effectively modulate the effects of both primary and conditioned reinforcers, but a consensus is not yet forthcoming. Consummatory behaviour and the ventral striatum It is significant that opiates such as morphine elicit feeding when infused into the nucleus accumbens [27*], a phenomenon that exhibits both conditioning and sensitization. It would thus appear that dopamine 1281 and opiates [27 ] have different effects on feeding motivation within the nucleus accumbens, perhaps relating to appetitive and consummatory aspects, respectively [29]. The anatomical differentiation of the nucleus accumbens into core and shell compartments [9 ] may also reveal more heterogeneous functions of the structure than hitherto have been considered, not only in view of the specificity and diversity of innervation from limbic afferents, but also because of the distributed nature of the ventral striatal efferents to a variety of structures [30**]. For example, the shell compartment projects to the lateral hypothalamus (Figure l), which has been shown to be involved in the feeding responses produced by infusions of non-nmda glutamate receptor antagonists into the shell region [31**]. The importance of this observation is that it is consistent with the hypothesis that appetitive and consummatory behaviour are controlled by separate neural systems, which involve distinct streams of ventral striatal outflow. Neurochemical correlates of reinforcement and incentive-motivation Much of the recent evidence-though not all (see below)- using in V~VO methods to monitor dopamine release is consistent with the general premise made above that nucleus accumbens dopamine activity is enhanced during appetitive (i.e. anticipatory, preparatory) phases of motivated behaviour. This conclusion is exemplified by the finding of enhanced extracellular concentrations of dopamine during exposure to tastes associated with the post-ingestive consequences of nutritive gastric loads [32 ]. In V~O microdialysis has also shown that elevated extracellular dopamine concentrations may be associated with high rates of instrumental behaviour rather than food consumption [33*]. Greater temporal resolution, sometimes at the cost of neurochemical specificity, can be achieved using in C&JO voltammetric methods, such as high-speed chronoamperometry. Using such methods, gradual increases in dopamine-related oxidation current were shown before the first lever press in each session of food-reinforced instrumental behaviour. Subsequently, there were biphasic changes in which the peak signal occurred at the time of lever pressing followed by a sharp decrease when retrieving earned food [34*]. It will prove increasingly important to monitor neurotransmitter systems other than dopamine in the nucleus accumbens, as well as in other brain regions, in order to define the extent of the neural systems that change their activity during rewarded behaviour. Recently, in vfvo microdialysis has also been used to monitor acetylcholine release in the hippocampus and frontal cortex during the anticipation of a meal, presumably arising from activity in cholinergic neurons of the rostra1 basal forebrain [3P]. Further studies should focus on the functional significance of the behaviour accompanying the neurochemical changes. For example, it may reflect altered cortical arousal and the induction of attentional mechanisms ([36]; H Himmelheber eta/., SocNeun_m-iAh- 1995, 21:763.6) as one of several components of the response to a reinforcer. At first sight, the phenomena associated with increased acetylcholine release are similar to those seen for mesolimbit dopamine, which, together with data showing that Dl agonists can enhance acetylcholine turnover [37], encourage hypotheses of a functional linkage between the mesoaccumbens system and part of its outflow to the basal forebrain [38]. However, the same research group that found elevations in acetylcholine release in anticipation of a meal have reported similar elevations in mesolimbic dopamine during the actual presentation of food [39,40*], thus questioning an interaction between these systems. As these data would seem to contradict other evidence of a role for mesolimbic dopamine in preparatory or appetitive processes, further experiments should be designed to resolve these apparent discrepancies. In addition to providing evidence of a role for nucleus accumbens dopamine in appetitive aspects of male sexual behaviour, a recent study in male rats [41*] has demonstrated parallel, sometimes short-lived changes in

5 232 Cognitive neuroscience dopamine in two anatomically related brain regions, the medial preoptic area and the medial basal hypothalamus. In contrast, changes in S-hydroxytryptamine (5HT) and its metabolite 5hydroxyindoleacetic acid (5HIAA) were more closely associated with consummatory and satiety aspects of sexual behaviour. Related investigations [42] have concerned interactions of dopamine and S-HT mechanisms with oestradiol and progesterone, studied in ovariectomized ewes. Electrophysiological correlates of reinforcement and incentive-motivation It is also important to integrate the results of neurochemical studies with those on single cells with temporal resolution in milliseconds rather than seconds or minutes. A recent example is the comparison of effects of either self-administered cocaine or water [43*]. Of the four main patterns of activity observed in nucleus accumbens neurons, three were common to drug and water reinforcement. Another study [44] has shown single-cell activity in the ventral tegmental area and prefrontal cortex associated with instrumental performance in rats responding under a fixed ratio schedule of sucrose delivery. This is relevant to the extensive set of single-cell recording data amassed by Schultz and colleagues [45 ] in monkeys that shows activity initially to a food object and later exclusively to a stimulus predicting its delivery. As all of these in viero methods provide essentially correlational data, it will ultimately be necessary to integrate such findings with the effects of lesions or other acute manipulations (e.g. the use of intracerebral infusions or promoters with transgenic animals) that seek to establish the functional significance of such correlated activity for behaviour. A key future issue in the field is whether neuronal activity can be linked to representations of the sensory qualities of different reinforcers, or to correlates of more general affective or motivational processes. Potentially, both types of activity could be present within the same neural system; an obvious possibility is that the dopaminergic cells provide motivational signals, whereas cells in the ventral striatum itself may also provide representations of the goal or reinforcer, based in part on information received from their limbic cortical afferents. Limbic-striatal interactions There is considerable evidence that effects of conditioned stimuli on motivated behaviour depend on interactions between the basolateral amygdala and dopamine-dependent functions of the ventral striatum [8]. One hypothesis is that associative information about conditioned stimuli and reinforcement arises from the basolateral amygdala [8]. Consistent with this notion is recent evidence implicating glutamate receptors of the NMDA type in the amygdala in learning to approach stimuli predictive of food [46*]. There is evidence that such associative information is relayed to the ventral striatum where processes determining the selection of specific behavioural responses are known to be influenced by activity of the ascending mesencephalic dopamine projections [S]. More complex forms of appetitive behaviour such as foraging are also controlled by the ventral striatum, although probably under the influence of limbic structures other than the amygdala. Impairments in foraging in a spatial context are produced by permanent or reversible lesions of the nucleus accumbens in rats [47 ] and monkeys [48*]. Optimal foraging by rats- for example in the spatial setting of a maze-may require working memory processes. Thus, the reinforcer must be associated transiently with certain of the maze stimuli in order to guide the future choice of responses. Infusions of the NMDA receptor antagonist AP-5 or the non-nmda receptor antagonist DNQX into the core region of the nucleus accumbens impair such behaviour, which itself is presumably dependent on the projection predominantly from the dorsal subiculum of the hippocampal formation to this region ([49*]; see Figure 1). This hippocampal influence may also explain the finding that performance in the water maze escape task used to assess spatial memory in rodents is also sensitive to intra-accumbens infusions of the dopamine receptor antagonist haloperidol [SO]. Neuronal activity has been found consistent with the hypothesis that parts of the ventral striatum integrate spatial and reward information in a radial maze [Sl ]. The ventral striatum therefore provides a convergence point for complex spatial information, as well as discrete conditioned stimuli, to interact with mechanisms determining response selection. Interactions between glutamate and dopamine in the ventral striatum The ability of the indirect dopamine agonist amphetamine to potentiate the effects of food-related conditioned reinforcers has been shown to be blocked by intraaccumbens infusions of the glutamate receptor antagonists AP-5 and CNQX [52*]. This result supports the hypothesis [S] that these effects of conditioned reinforcers are mediated by limbic-striatal, probably mainly amygdaloid, glutamatergic afferents, interacting with projections horn midbrain dopamine neurons. The same study indicated that amphetamine-induced locomotor activation appeared to depend on a different neuronal substrate. Thus, infusions of AP-5 into the core of the nucleus accumbens blocked amphetamine-induced locomotor hyperactivity while causing spontaneous hypoactivity, whereas infusions of AP-5 into the shell actually increased locomotor activity [53,54 ]. The functional significance of this form of locomotor activity is still unclear, although it could potentially contribute to the foraging and approach responses elicited by incentive stimuli such as food, or by signals that predict its delivery. However, the data reviewed below suggest that mechanisms governing locomotor responses

6 Neurobehavioural mechanisms of reward and motivation Robbins and Everitt 233 can be further anatomically dissociated from the effects of conditioned reinforcers and conditioned incentives, at the level of the different structures targeted by ventral striatal efferents. Functions of ventral striatal output targets There is no effect of lesions of the nucleus accumbens or ventral pallidurn on the positive reinforcing effects of lateral hypothalamic brain-stimulation reward [55*]. These results indicate that these structures are not necessary for ail forms of reward, and that reinforcing effects occur in distributed brain systems-in this case, probably efferent to the ventral striatopallidal system. Together with the evidence on pharmacologically elicited feeding described above [27*,31 ], these data reveal important interactions between the nucleus accumbens and lateral hypothalamus in reward mechanisms. Other output mechanisms from the ventral striatum probably have different functions; for example, they may mediate the effects of conditioned reinforcers on response selection and on conditioned place preference. Thus, lesions of the pedunculopontine tegmental nucleus block both morphine and amphetamine conditioned place preference without affecting amphetamine-induced locomotion [.56 ]. This blockade of place preference conditioning also extends to natural reinforcers such as food [57], although in this case, the effects were found only in rats that were not food deprived. However, similar lesions appear to impair response selection processes based on foodassociated conditioned reinforcers in the food-deprived state, again without impairing drug-induced hyperactivity [.%*I. Several years after the original observations of Mogenson and colleagues [7] on the mesencephalic locomotor region, the precise neural substrates of the ventral striatal output for locomotor activity-presumed to be at the level of the midbrain tegmentum-still remain to be established. Conclusions This review has shown that the ventral striatum plays a role in many motivational processes besides those related to drugs as reinforcers. The most basic distinction, which apparently holds across many natural motivational states, is that between appetitive and consummatory behaviour. These aspects of motivation may depend upon different neurochemical systems and subregions within the ventral striatum and their associated outputs. In particular, the effects of conditioned as well as unconditioned stimuli are represented in the ventral striatum. For example, the effects of conditioned reinforcers that promote instrumental behaviour appear to be controlled by an interaction between certain limbic afferents and dopamine-dependent mechanisms of the nucleus accumbens. Other interactions of defined limbic structures with the ventral striatum are implicated in forms of foraging behaviour. A role of the nucleus accumbens in stimulus-response, habit learning seems less likely than for the caudate-putamen (dorsal striatum) [4,59], but cannot be ruled out as yet. The anatomical compartmentalization of the nucleus accumbens into core and shell subregions is helping to clarify its role in different forms of motivational function, such as controlling consummatory and appetitive behaviour. Some progress is also being made in the integration of information from neurochemical, electrophysiological and lesion experiments. There remain, however, significant disagreements, probably attributable to differences, for example in temporal resolution, in the currently available methods. There is mounting evidence for separate neural systems underlying distinct effects of reinforcers at a behavioural level that arise from distinct outputs of the ventral striatum. Thus, in rats, different efferent targets of the nucleus accumbens may independently control consummatory forms of behaviour (e.g. ingestive and sexual), as well as locomotor activity implicated in foraging behaviour and approach to goals. Other structures help to control mechanisms of behavioural choice (e.g. in instrumental behaviour) and modulate arousal and attentional mechanisms, which may serve to promote the memory and learning of appropriate goal-directed sequences of behaviour at a cortical level. We anticipate that an important issue for the next period of research is whether these limbic-striatal-pallidal neural systems only mediate the effects of appetitive events on behaviour, or whether there are conditions under which the system also mediates the effects of aversive events. For example, exposure to stress may change the dynamics of the system (for example, by altering mesocortical dopaminergic activity [60]) so that the normal calibration of a positive reinforcer is disturbed. It remains to be seen to what extent the nucleus accumbens and related structures, like many of the limbic structures that innervate the ventral striatum, mediate the effects of aversive aspects of motivation on behaviour [61*]. This would considerably complicate the old notion of specialized reward centres or circuitry separate from punishment or aversion regions. Overall, we do not feel that the past two years have been a particularly vintage period for this field, although previous findings have been consolidated and novel directions for research have been identified following a fresh look at the underpinning psychological theory and neuroanatomical organization of the limbic cortical and striatal systems. We were disappointed not to have seen a greater input from human cognitive neuropsychology. Even though the functions of the neocortex have been extensively studied in the functional neuroimaging context, reward processes, including subjective responses, have not yet been much investigated. Further barriers to progress are provided by the difficulties of resolution of the relevant subcortical structures with neuroimaging techniques. Nevertheless,

7 234 Cognitive neuroscience the solid progress made from the growing sophistication of methods and theory suggests that the neural substrates of reward processes at a behavioural, and perhaps even a subjective, level will soon be identified. Acknowledgements Our research is supported by the Medical Research Council (UK) and the Wellcome Trust. With thanks to the staff of Current Biology Ltd for assistance. References and recommended reading Papers of particular interest, published within the annual period of review, have been highlighted as:. of special interest.. of outstanding interest 1. Holland PC, Rescorla RA: The effect of two ways of devaluing the unconditioned stimulus after first and second order appetlttve conditioning. J Exp Psycho/ [Anim Behavl 1975, Dickinson A, Balleine B: Motivational control of goal-directed action. Anim Learning Behav 1994, 22:1-l 6. y very useful review of some of the progress made in understanding the psychological processes undertying goal-directed behaviour over the past decade or so. 3. Dickinson A: instrumental conditioning. In Animal Learning and Cognition. Edited by Mackintosh NJ. San Diego: Academic Press; 1994~ White NM: Reward or reinforcement: what s the difference? Neurosci Biobehav Rev 1989, l 86. Koob GF: Dopamine, addictfon and reward. Semin Neurosci 1992, 4:139-l 40. Wise R: Neuroiepttcs and operant behavior: the anhedonla hypothesis. Behav Brain Sci 1992, 5:39-W. Mogenson GJ, Jones DL, Yim CY: From motivation to action: functional interface between the llmblc system and the motor system. Prog Neurobiol 1980, 14: Robbins TW, Cador M, Taylor JR, Everitt BJ: Limbic-striatal interactions in reward-related processes Neurosci Biobehav Rev 1989, 13: Heimer L, Zahm DS, Alheid GF: Basal ganglia. In The Rat. Nervous System, edn 2. Edited by Paxinos G. Sydney: Academic Press; 1995: _... _a. An excellent overvrew ot the advances made-many by tnese autnors tnemselves - in understanding the organization and connections of the dorsal and ventral striatum, including the demarcation of the core and shell territories of the nucleus accumbens. 10. Salamone J: The involvement of nucleus accumbens dopamlne in appebtfve and averslve motivation. Behav Brain Res 1994, 61: Goeders NE, Guerin GF: Non-contingent electric footshock. facilitates the acquisition of intravenous cocaine selfadministration in rats. Psychopharmacology 1994, 114: A sophisticated demonstration of the effects of stress to sensitize acquisition of drug-taking behaviour using the classic triadic design well known to those familiar with the learned helplessness literature. 12. Phillips GD, Hovms SR, Whitelaw RB, Wilkinson L, Robbins TW,, Ever&t BJ: Isolation-rearing enhances the locomotor response both to a novel environment and to cocaine, but impairs the Intravenous self-adminlstratlon of cocaine. Psychopharmacology 1994, 115: This paper illustrates how enhanced locomotor effects produced by psychomotor stimulants in socially deprived rats (previously shown to be associated with increased extracellular striatal doparnine) do not necessarily lead to enhanced reinforcing effects. The speculation is that adaptations in the limbic afferent structures serve to regulate doparnine release (and hence reinforcing efficacy of the self-administered drug) in the isolated rats. 13. Mayer LA, Parker LA: Rewarding and aversive properties of IP and SC cocaine: assessment by place and taste conditioning. Psychopharmacology 1993, 112: Papp M, Klimek V, Willner P: Parallel changes in dopamine D2 receptor binding In limbic forebrain associated with chronic mild stress-induced anhedonia and its reversal by imlpramlne. Psychopharmacology 1994,115: Caine SB, Koob GF: Effects of dopamlne D-l and D-2. antagonists on cocaine self-administration under different schedules of reinforcement in the rat J Pharmacol Drp Ther 1994, 270: An elegant parametric study of the effects of dopamine receptor antagonists on responding maintained by cocaine and food in different portions of a multiple fixed ratio schedule of responding. Low doses of selective Dl receptor antagonists more potently reduced responding for cocaine than food, whereas the reverse was the case for 02 receptor antagonists. This double dissociation presumably indicates that cocaine and food have distinct reinforcing properties, separately mediated by different dopamine receptors, and rules out obvious motor effects of the dopamine blockers. A related study [I61 implicates mesolimbic dopamine in the effects of cocaine, but not food, in maintaining operant behavior. 16. Caine SB, Koob GF: Effects of mesolimbic dopamine depletion on responding maintained by sucrose and food. J Exp Anal Behav 1994, 61: Hodge CW, Samson HH, Tolliver GA, Haraguchi M: Effects of intra-accumbens injections of dopamine agonists and antagonists on sucrose and sucrose-ethanol reinforced responding. Pharmacol Biochem Behav 1994,4&l 41-I Ranaldi R, Beninger RJ: The effects of systemic and fntracerebral injections of Dl and D2 agonists on brain stfmulatton reward. Brain Res 1994, 651: Mitchell JB, Gratton A: Involvement of mesolimbic dopamlne neurons In sexual behaviors: impllcetlons for the neurobiology of motivation. Rev Neurosci 1994, 5: Hansen S: Maternal behavior of female rats with 6-OHDA. lesions In the ventral sblatum: characterlzstlon of the pup retrieval deficit Physiol Behav 1994, 55: The author demonstrates that deficits in maternal behaviour, such as in pup retrieval, following dopamine depletion from the ventral striatum can be ameliorated by separation of the lesioned mothers from their pups-a manipulation that increases maternal motivation. 21. Everitt BJ: Sexual motivation: a neural and behavioral analysis of the mechanisms underlying appetttlve copulatory responses of male rats. Neurosci Biobehav Rev 1990, 14: Salamone JD, Cousins MS, Bucher S: Anhedonia or anergia?. Effects of haloperidoi and nucleus accumbens dopamine depletion on lnstrumentai response selectfon in a T-maze cost/benefit procedure. Behav Brain Res 1994, 65: One of a series of papers (see also [10,231) arguing with simple but ingenious tests that dopamine depletion exerts its effects on response output by adjusting the parameters relating effort of response to reinforcing efficacy of stimuli. This analysis thus takes the usual motor versus reward controversy at least one step forwards. Experiments with excitotoxic lesions of the same structure are urgently required for the purposes of comparison (see, for example, j24.1). 23. Cousins MS, Salamone JD: Nucleus accumbens dopamine depletions In rats affect relative response allocation In a novel cost/benefit procedure. Pharmacol Biochem Behav 1994, 49: Balleine B, Killcross S: Effects of lbotenic acid lesions of the. nucleus accumbens on Instrumental action. Behav Brain Res 1994, 65: An ambitious attempt to define the functions of the nucleus accumbens in operant behaviour by testing the sensitivity of lesioned rats to changes in the value of the reward controlling both the lever press response and the entry to the food magazine. The results do not support a rote for the nucleus accumbens in mediating instrumental action, although the general rate of lever pressing is reduced in lesioned rats and they exhibit signs of reduced motivational excitement via Pavlovian mechanisms. This approach represents an important interaction between animal learning theory and behavioural neuroscience that deserves further attention. 25. Ranaldi R, Beninger RJ: Dopamlne Dl and D2 antagonists attenuate amphetamine-induced enhancement of responding for conditloned reward in rats. Psychopharmacology 1993, 113: Wolterink G, Phillips G, Cador M, Donselaar-Wolterink I, Robbins TW, Everitt BJ: Relative roles of ventral striatai Di and D2 dopamine receptors in responding with conditioned reinforcement Psychopharmacology 1993, 110: Bakshi VP, Kelley AE: Sensttizatlon and conditioning of feeding. following multiple morphine microinjections Into the nucleus accumbens. Brain Res 1994, 640~

8 Neurobehavioural mechanisms of reward and motivation Robbins and Everitt 235 This paper is interesting for highlighting the possible role of the opiatedependent mechanisms in the nucleus accumbens in aspects of feeding that are probably controlled by anatomical projections from the shell region of the nucleus accumbens to the lateral hypothalamus (see [31**]). 28. Phillips GD, Howes SR, Whitelaw RB, Everitt BJ, Robbins TW: Analysis of the effects of intra-accumbens SKF39393 and LY upon the behavloural satiety sequence. Psychopharmacology 1995, 117: Di Chiara G, North RA: Neurobiology of opiate abuse. Trends Pharmacol Sci 1992, 13:185-l Pennartz CMA, Groenwegen HJ, Lopes da Silva F: The nucleus.. accumbens as a complex of functionally distinct neuronal ensembles: an integration of behavioural, electrophysiological and anatomical data. Prog Neurobiol 1994, 42:71 Q-761. An important synthesis of electrophysiological, anatomical and behavioural evidence that tries to make sense of the anatomical convergences and neurochemical compartmentalization of the ventral striatum. The concept of neuronal ensembles may allow us to explore the integration of information from different limbic sources in a learning context. The impression one has is that the behavioural data are only just beginning to address this concept. 31. Maldonado-lrizarry CS, Swanson CJ, Kelley AE: Glutamate.. receptors In the nucleus accumbens shell control feeding behavior in the lateral hypothalamus. J Neurosci 1995, An important breakthrough study showing that blocking non-nmda receptors in the shell compartment with DNGX elicits eating that is blocked by infusion of the GABAergic agonist muscimol into the lateral hypothalamus and therefore confirming a functional link between the shell and its anatomically defined outputs. An interesting question to ponder is the nature of neural and psychological controls over feeding exerted at the level of the nucleus accumbens. 32. Mark GP, Smith SE, Rada PV, Hoebel BG: An appetitively. conditioned taste eliclts a preferential increase in mesolimblc dopamine release. Pharmacol Biochem Behav 1994, 48: Describes experiments showing increases in extracellular dopamine in the nucleus accumbens, but not the dorsal striatum, in rats exposed to a mildly bitter conditioned taste stimulus paired with a nutritive gastric load, but not to a mildly sour taste paired with a non-nutritive gastric load. The authors suggest a role for conditioned dopamine release in ingestive responses acquired on the basis of the post-ingestive consequences of food. 33. Salamone JD, Cousins MS, McCullough LD, Carrier ODL,. Berkowitz RJ: Nucleus accumbens dopamine release increases during instrumental lever pressing for food but not free food consumption. Pharmacol Biochem Behav 1994,49: This study shows elevated dopamine release associated with operant behaviour but not free food presentation. These results thus complement other data showing the opposite effects of dopamine receptor blockers and dopamine depletion. The study also provides a valuable technical counterpoint to [34*] by employing in viva microdialysis rather than voltammetric. methods, and apparently arriving at roughly similar conclusions. 34. Kiyatkin EA, Gratton A: Electrochemical monitoring of. extracellular dopamine in nucleus accumbens of rats leverpressing for food. Brain Res 1994, 652: The controversies over the chronoamperometric technique should not detract from this complex and largely convincing analysis of the temporal dy namics of instrumental (operant) behaviour and neurotransmitter function within the ventral striatum. The key finding is that the presumed dopamine signal peaks at the moment the animals press the lever, and is followed by an abrupt decrease as the animals retrieve and consume the food pellet. 35. lnglis FM, Day JC, Fibiger HC: Enhanced acetylchollne.. release in hippocampus and cortex during the anticipation and consumption of a palatable meal. Neuroscience 1994, 62:1049-l 056. This paper is important because it links together two neurotransmitter systems-the basal forebrain cholinergic and the mesencephalic dopaminergic projections-that have hitherto been investigated rather independently. What makes the results so intriguing is that they seem to show for the basal forebrain cholinergic system what many other investigators would have expected to see for the mesolimbic dopsmine system, namely a clear correlative relationship with incentive-motivation. However, the authors of this paper do not themselves observe such incentive-motivational effects with mesolimbic dopamine (see [39,40*1). Similar work is also being performed by Sarter, Bruno and associates (see H Himmelheber et al., Sot Neurosci Abstr 1995, 25:763.6) and, while they endorse the concept of dopsrninergic modulation of cortical acetylcholine, there are likely to be some disagreements about the exact behavioural context in which the cholinergic changes are seen. 36. Muir JL. Everitt BJ, Robbins TW: AMPA-induced excitotoxic lesions of the forebrain cholinergic systems: a significant role for the cortical cholinerglc system In attentional function. J Neurosci 1994, 14: Day J, Fibiger HC: Dopaminergic ragulatlon of cortical acetytcholine release: effenzts of dopamlne receptor agonists. Neuroscience 1993, Sarter M: Neuronal mechanisms of the ettentional dysfunction in senile dementia and schizophrenia: two sides of the same coin? Psychopharmacology 1994,114: Fibiger HC: Mesolimbic dopamine: an analysis of its role in motivated behavior. Semin Neurosci 1993, 5: Wilson C, Nomikos GG, Collu M, Fibiger HC: Dopaminergic. correlates of motivated behavior: importance of drive. J Neurosci 1995, 15: A study emphasizing the correlation of ventral striatal dopamine release with consummatory rather than anticipatory (appetitive) behaviour using in viva microdialysis, and the amplifying effects of food deprivation ( drive ). The argument is made that this pattern of results best fits the psychopharmacological effects of dopamine receptor antagonists. The hypothesis has to be considered in the light of other psychopharmacological evidence with dopamine receptor indirect and direct agonists (e.g. [25,261) and other neurochemical (e.g. [32*-34-l) and electrophysiological evidence [45 ] emphasizing correlations with instrumental and anticipatory behaviour during presentation of conditioned stimuli. The possibility of fast transients in dopamine release that remain undetected by the microdialysis technique complicates the issue, but in our view, the controversy may be a product of interpretative emphasis and procedural factors rather than representing a serious theoretical discrepancy. 41. Fumero B, Femandez Vera JR, Gonzalez Mora JL, Mas M:. Changes In monoamine turnover in forebrain areas associated with masculine sexual behavior: a microdialysis study. Brain Res 1994, 862: These microdialysis experiments confin the increase in extracellular dopamine in the nucleus accumbens of male rats exposed to oestrous, but not ovariectomized, female rats. In addition, the results show increased levels of the metabolites of dopamine and 5-HT (DOPAC and 5-HIAA, respectively) not only in the nucleus accumbens, but also in the medial preoptic area (strongly linked to the control of copulation in males) and the mediobasal hypothalamus. The relationship between neurochemical changes and sexual behaviour leads the authors to suggest the changes in DOPAC are related to appetitive sexual responses and those in 5-HIAA to copulation and ejaculation. 42. Fabre-Nys C, Blache D, Hinton MR, Goode JA, Kendrick KM: Microdialysis measurement of neurochemical changes in the mediobasal hypothalamus of ovariectomized ewes during ctestrus. Brain Res 1994,649: Carelli RM, Deadwyler SA: A comparison of nucleus accumbens. neuronal firing patterns during cocaine self-administration and water reinforcement in rats. J Neurosci 1994, 14: Comparisons such as this are valuable because they are so rare. They may have relevance for constructing a dimensional or attribute analysis of different reinforcers and their neural substrates, although the difficulty of equating the value and state dependence of the different reinforcers provides major difficulties. 44. Kosubud AE, Harris GC, Chaplin JK: Behavioral associations of neuronal activity in the ventral tegmental area of the rat J Neurosci 1994, 14: Schultz W, Romo R, Ljungberg T, Mirenowicz J, Hollerman JR,.. Dickinson A: Reward-related signals carried by dopamine neurons. In Models of Information Processing in the Basal Ganglia. Edited by Houk JR, Davis JL, Beiser D. Cambridge, Massachusetts: MIT Press; 1995: This is an excellent review of several years work on the functions of midbrain dopamine neurons in transmitting reward -related signals. The most intriguing aspects are the transitional role of these neurons in coding initially the unconditioned stimulus and then the conditioned stimulus, and their application to a simple connectionist model of learning in the striatum. Several other chapters from this exceptionally interesting book are relevant to this topic. 48. Burns LH, Kelley AE, Everitt BJ, Robbins TW: Intra-amygdala. infusion of the N-methyl-o-aspartate receptor antagonist AP-5 impairs acquisition but not performance of discriminated approach to a CS. Behav Neural Biol 1994, 61: This article demonstrates behaviourally selective deficits produced by low doses of NMDA receptor antagonists on conditioned approach to food, rather than learning in aversive contexts, which has been emphasized to date. This implication of the amygdala in appetitive as well as aversive motivation has important implications for how this structure governs reflexive, endocrine, autonomic and behavioural aspects of response output (see Figure 1).