How the Basal Ganglia Use Parallel Excitatory and Inhibitory Learning Pathways to Selectively Respond to Unexpected Rewarding Cues

Transcription

1 The Journal of Neuroscience, December, 999, 9(23): How the Basal Ganglia Use Parallel Excitatory an Inhibitory Learning Pathways to Selectively Respon to Unexpecte Rewaring Cues Joshua Brown, Daniel Bullock, an Stephen Grossberg Department of Cognitive an Neural Systems an Center for Aaptive Systems, Boston University, Boston, Massachusetts 0225 After classically conitione learning, opaminergic cells in the substantia nigra pars compacta (SNc) respon immeiately to unexpecte conitione stimuli (CS) but omit formerly seen responses to expecte unconitione stimuli, notably rewars. These cells play an important role in reinforcement learning. A neural moel explains the key neurophysiological properties of these cells before, uring, an after conitioning, as well as relate anatomical an neurophysiological ata about the peunculopontine tegmental nucleus (PPTN), lateral hypothalamus, ventral striatum, an striosomes. The moel proposes how two parallel learning pathways from limbic cortex to the SNc, one evote to excitatory conitioning (through the ventral striatum, ventral pallium, an PPTN) an the other to aaptively time inhibitory conitioning (through the striosomes), control SNc responses. The excitatory pathway generates CS-inuce excitatory SNc opamine bursts. The inhibitory pathway prevents opamine bursts in response to preictable rewar-relate signals. When expecte rewars are not receive, striosomal inhibition of SNc that is unoppose by excitation results in a phasic rop in opamine cell activity. The aaptively time inhibitory learning uses an intracellular spectrum of time responses that is propose to be similar to aaptively time cellular mechanisms in the hippocampus an cerebellum. These mechanisms are propose to inclue metabotropic glutamate receptor-meiate Ca 2 spikes that occur with ifferent elays in striosomal cells. A opaminergic burst in concert with a Ca 2 spike is propose to potentiate inhibitory learning. The moel provies a biologically preictive alternative to temporal ifference conitioning moels an explains substantially more ata than alternative moels. Key wors: opamine; substantia nigra; rewar; basal ganglia; conitioning; peunculopontine tegmental nucleus; lateral hypothalamus; striosomes; aaptive timing Humans an animals can learn to preict both the amounts an times of expecte rewars. The opaminergic cells of the substantia nigra pars compacta (SNc) have unique firing patterns relate to the preicte an actual times of rewar (Ljungberg et al., 992; Schultz et al., 993; Mirenowicz an Schultz, 994; Schultz et al., 995; Hollerman an Schultz, 998; Schultz, 998). Figures an 2 summarize some of their main properties, notably how learning enables the SNc cells to respon immeiately to unexpecte cues [conitione stimulus (CS)] but to omit responses in an aaptively time fashion to expecte rewars [unconitione stimulus (US)]. Because these firing patterns also act as learning signals in the striatum an elsewhere (Wickens an Kotter, 995), they have been suggeste to play a key role in both aictive behavior (Garris et al., 999) an reinforcement learning. In particular, opaminergic rewar signals seem to strengthen the incentive salience or wanting of a certain Receive July 2, 999; revise Sept. 5, 999; accepte Sept. 6, 999. J.B. was supporte in part by the Defense Avance Research Projects Agency an the Office of Naval Research (ONR N , ONR N J- 309, an ONR N ). D.B. was supporte in part by the Defense Avance Research Projects Agency an the Office of Naval Research (ONR N an ONR N J-309). S.G. was supporte in part by the Defense Avance Research Projects Agency an the Office of Naval Research (ONR N , ONR N J-309, an ONR N ) an the National Science Founation (NSF IRI ). Corresponence shoul be aresse to Daniel Bullock or Stephen Grossberg, Department of Cognitive an Neural Systems an Center for Aaptive Systems, Boston University, 677 Beacon Street, Boston, MA anb@cns.bu.eu or steve@cns.bu.eu. Copyright 999 Society for Neuroscience /99/ $05.00/0 rewar, that is, the motivation to work for the rewar in a given behavioral context, as istinct from the affective enjoyment or liking of a rewar once consume (Berrige an Robinson, 998). The liking may be meiate by areas other than the basal ganglia (McDonal an White, 993). Recent moels (Houk et al., 995; Montague et al., 996; Contreras-Vial & Schultz, 997; Schultz et al., 997; Berns an Sejnowski, 998; Suri an Schultz, 998) of the nigral opamine cells have note similarities between opamine cell properties an well known learning algorithms, especially temporal ifference (TD) moels (Montague et al., 996; Schultz et al., 997; Suri an Schultz, 998). Although proviing a egree of insight into the information carrie by the opamine signal, the TD approach has not been able to answer the questions of what biological mechanisms actually compute the signal, an how. In particular, how oes learning in the circuit that inclues these cells enable them to prouce a fast excitatory response to conitione stimuli an a elaye, aaptively time inhibition of response to rewaring unconitione stimuli, in all of the experimental conitions summarize by Figures an 2? We show here that the known anatomy an cell types in pathways afferent to opamine cells lea to an explanation with significant avantages over previous moels. We introuce a moel in which the learne excitatory an inhibitory responses are subserve by ifferent anatomical pathways, an the aaptively time inhibitory learning is meiate by metabotropic glutamate receptor (mglur)-riven Ca 2 spikes in striosomal cells. These Ca 2 spikes occur with a spectrum of

2 Brown et al. Basal Ganglia Learning an Unexpecte Rewars J. Neurosci., December, 999, 9(23): Figure. Dopamine cell firing patterns. Left, Data. Right, Moel simulation, showing moel spikes an unerlying membrane potential. A, In naive monkeys, the opamine cells fire a phasic burst when unpreicte primary rewar R occurs (e.g., if the monkey receives a burst of apple juice unexpectely). B, As the animal learns to expect the apple juice that reliably follows a sensory cue [conitione stimulus (CS)] that precees it by a fixe time interval, then the phasic opamine burst isappears at the expecte time of rewar, an a new burst appears at the time of the rewar-preicting CS. C, After learning, if the animal fails to receive rewar at the expecte time, a phasic epression in opamine cell firing occurs. Thus, these cells reflect an aaptively time expectation of rewar that cancels the expecte rewar at the expecte time. [The ata in Figure (column ) are reprinte with permission from Schultz et al. (997).] temporal elays. When a Ca 2 spike an a opamine burst occur at the same time, inhibitory learning is enhance at the corresponing elays. To explicate these excitatory an inhibitory pathways, the moel functionally explains an simulates the firing patterns of opamine cells, striosomal cells of the striatum, peunculopontine tegmental nucleus (PPTN) cells, ventral striatal cells, an lateral hypothalamic cells (see Figs. 3). Its mglurbase spectral timing mechanism helps to explain more ata than the temporal erivative operation that efines the class of TD moels previously use to escribe opamine cell behavior. This moel is shown schematically in Figure 4. MATERIALS AND METHODS Dopamine cell responses can be conitione to phasic cues whose offsets occur long before the rewar signals that they preict (Ljungberg et al., 992). To brige the temporal gap, a CS is assume to activate a sustaine working memory input to the moel (Funahashi et al., 989). A subsequent primary rewar signal from a US is assume to trigger a opamine burst, which augments the weights between the working memory site an the ventral striatum (Wickens et al., 996). This allows future CS presentations to elicit an immeiate excitatory preiction of rewar. The CS also activates a population of lagge inhibitory signals from the striosomes to the SNc. When a opamine burst occurs at a sufficient lag after CS onset, it strengthens the subset of lagge inhibitory signals that are active at that time. These two types of learning enable a CS to generate an immeiate, rewar-preictive opamine signal but also to cancel subsequent SNc excitation that woul otherwise be cause by the preicte rewar-relate signals. When a response is mae an rewar is receive, the working memory input is assume to shut off (Funahashi et al., 989). We propose that the PPTN is responsible for the phasic bursts of activity in SNc opamine cells (Figs. an 2) an thus plays a key role in the learning an maintenance of instrumental tasks. Experiments showing monosynaptic glutamatergic an cholinergic PPTN-to-SNc projections (Scarnati et al., 988; Cone, 992; Futami et al., 995) support this hypothesis. Cone (992) has suggeste that the PPTN provies the main source of excitation to the SNc, an PPTN cells have been foun to fire phasically in response to primary rewar or rewar-preicting conitione stimuli, or both, leaving them well situate to provie this kin of SNc input (Dormont et al., 998) (Fig. 3A). The phasic nature of PPTN signaling is attributable to habituation, or accommoation, in SNc-projecting PPTN cells (Takakusaki et al., 997). Lesions of the PPTN prouce hemiparkinsonian symptoms, as if the SNc itself ha been lesione (Kojima et al., 997), an reversible PPTN inactivation mimics extinction in an instrumental task, even while rewars, if provie, are reaily consume (Cone et al., 998). PPTN afferents. From where oes the PPTN receive these responsemotivating rewar an rewar-preicting signals? We propose that the primary rewar signals come from the lateral hypothalamus, whereas the excitatory rewar-preiction signals (which generate a CS-inuce opamine burst) travel via the ventral striatum ventral pallium pathway, which receives input mainly from limbic cortex (Schultz et al., 992) (Fig. 4). Lateral hypothalamic neurons are known to play a role in feeing behavior an to fire phasically in response to primary rewar (Nakamura an Ono, 986), as in Figure 3D. A strong lateral hypothalamus PPTN projection has been foun an confirme by both anterograe an retrograe labeling (Semba an Fibiger, 992), an the primary rewar signal explains the similar phasic rewar response in the PPTN. Thus, the lateral hypothalamus seems to be a principal source of excitation to the PPTN. Likewise, more than one-fourth of the ventral pallium projects collaterals to the PPTN (Mogenson an Wu, 986). The ventral pallium receives projections from the matrisomes of the ventral striatum (Yang an Mogenson, 987), which respons to both preicte an primary rewar (Schultz et al., 992), as in Figure 3B. The ouble inhibition from ventral striatum to ventral pallium to PPTN results in net excitation from ventral striatum to PPTN. We preict that the sustaine, CSinuce striatal activation that is shown in Figure 3B is attributable to receipt of a working memory trace of the CS from limbic cortex, which is enhance by learning of CS-rewar contingencies (Dias et al., 996). The transient component in Figure 3B results from a phasic primary rewar signal from the lateral hypothalamus (Nakamura an Ono, 986; Brog et al., 993). We suggest that the ventral striatum is a main pathway of excitatory rewar preictions. Other PPTN afferents are possible caniates for generating phasic PPTN responses. Some other possible sources, foun by retrograe labeling from the PPTN, inclue the central nucleus of the amygala (CNA) an the subthalamic nucleus (STN) (Semba an Fibiger, 992). The amygala oes not appear to provie the main source of excitation, espite its processing of emotional valence information. In particular, it has been shown that rats with amygala lesions coul still learn operant tasks (McDonal an White, 993). After CNA amage, rats can learn

3 0504 J. Neurosci., December, 999, 9(23): Brown et al. Basal Ganglia Learning an Unexpecte Rewars Figure 2. Dopamine cell firing patterns. Left, Data. Right, Moel simulation, showing moel spikes an unerlying membrane potential. A, The opamine cells learn to fire in response to the earliest consistent preictor of rewar. When CS2 (Instruction) consistently precees the original CS (Trigger) by a fixe interval, the opamine cells learn to fire only in response to CS2. [Data reprinte with permission from Schultz et al. (993).] B, During training, the cell fires weakly in response to both the CS an rewar. [Data reprinte with permission from Ljungberg et al. (992).] C, Temporal variability in rewar occurrence. When rewar is receive later than preicte, a epression occurs at the time of preicte rewar, followe by a phasic burst at the time of actual rewar. D, Likewise, if rewar occurs earlier than preicte, a phasic burst occurs at the time of actual rewar. No epression follows because the CS is release from working memory. [Data in C an D reprinte with permission from Hollerman an Schultz (998).] E, When there is ranom variability in the timing of primary rewar across trials (e.g., when the rewar epens on an operant response to the CS), the striosomal cells prouce a Mexican hat epression on either sie of the opamine spike. [Data reprinte with permission from Schultz et al. (993).] secon-orer conitioning although they fail to learn a conitione orienting response (Gallagher an Chiba, 996). Similarly, some stuies suggest a moulatory rather than an excitatory role of the STN-to-SNc projection (Smith an Grace, 992), an cell recoring stuies have not yet shown rewar-preicting activity in the STN. Striosomes. What suppresses the opamine burst response to primary rewar after conitioning has occurre, an what causes the transient activity rop when expecte rewar is not receive (Fig. )? The striosomal cells provie a significant source of GABAergic inhibition to the SNc (Gerfen, 992), which coul account for both of these phenomena. In turn, striosomal cells receive opaminergic projections from the SNc (Gerfen, 992). We propose that an intracellular spectral timing mechanism (Grossberg an Schmajuk, 989; Grossberg an Merrill 992, 996; Fiala et al., 996) provies the function neee. Specifically, the striosomal cells briefly inhibit SNc opamine cells, after a learne elay perio, to provie an inhibitory expectation of rewar. The moel incorporates striosomal cells in both the orsal an ventral aspects of the striatum. Likewise, moel opamine cells correspon to both orsal an ventral SNc cells, which espite certain ifferences have similar inputs an response properties. Gerfen (992) has note the istinction between the orsal an ventral tiers of the SNc: orsal tier SNc cells project to the matrisomes of the striatum (incluing the moel ventral striatal cells), whereas ventral tier SNc cells project to the striosomes. The moel lumps together the ventral an orsal tiers of the SNc on the basis of their similarities. It has been suggeste that striosomal cells provie aaptively time inhibition to the opamine cells (Contreras-Vial an Schultz, 997), much as cerebellar Purkinje cells provie aaptively time inhibition of interpositus nucleus cells (Fiala et al., 996), but this general hypothesis must be couple to a biologically supporte local mechanism. Given evience that striatal learning is suppresse by mglur blockers (Calabresi et al., 992a) an Ca 2 -chelators (Calabresi et al., 994), we suggest the following striosomal cell moel: conitione stimuli excite a glutamatergic corticostriatal pathway that activates mglurs on striosomal neurons. These in turn cause a elaye transient rise in intracellular Ca 2, at least partly via NMDA channels (Calabresi et al., 992b), which are known to be potentiate by mglur receptor activation (Pisani et al., 996). This Ca 2 response is propose to be a basis for both learning an generating an aaptively time inhibitory striosomal SNc signal. The moel uses a population of striosomal cells with a range of elaye responses (Fig. 5), which, taken together, constitute the spectrum of possible learne elays. Fiala et al. (996) propose a moel of aaptively time conitioning in which cerebellar Purkinje cells generate a spectrum of ifferently elaye Ca 2 spikes after excitation of mglur receptors. A Ca 2 spike by itself activates a Ca 2 -epenent K conuctance, which is hyperpolarizing. In aition, when a climbing fiber signal is receive at the same time as a elaye Ca 2 spike, it causes a long-term increase in the Ca 2 -epenent K channel conuctance. Thus, in the cerebellar moel, the Ca 2 spike is a basis for both immeiate hyperpolarization an learne long-term epression (LTD). We propose that a relate but istinct mechanism operates in striosomal cells, which, unlike Purkinje cells (Crepel et al., 996), possess NMDA receptors. In this context, a mglur- meiate elaye Ca 2 spike can be amplifie an thus serve to transiently increase rather than ecrease striosomal cell activity. A class of recently iscovere Cainhibite K channels (Joiner et al., 998) may also contribute to a Ca-epenent epolarization. A Ca 2 spike combine with a phasic

4 Brown et al. Basal Ganglia Learning an Unexpecte Rewars J. Neurosci., December, 999, 9(23): Figure 3. Traine firing patterns in PPTN, ventral striatum, striosomes, an lateral hypothalamus. Left, Data. Right, Moel simulations, showing moel spikes an unerlying membrane potential. A, PPTN cell (cat), showing phasic responses to both CS an primary rewar. [Data reprinte with permission from Dormont et al. (998).] In the moel, phasic signaling is cause by accommoation or habituation (Takakusaki et al., 997), which causes the cell to fire in response to the earliest rewar-preicting CS an US rewar, but not to subsequent CSs before rewar. B, Ventral striatal cells show sustaine working memory-like response between trigger an a US rewar, an a phasic response to the US rewar. [Data reprinte with permission from Schultz et al. (992).] C, A ventral striatal cell, preicte here to be a striosomal cell, shows builup to phasic primary rewar response. For the moel cell, j 39. [Data reprinte with permission from Schultz et al. (992).] D, A lateral hypothalamic neuron with a strong, phasic response to glucose rewar. [Data reprinte with permission from Nakamura an Ono (986).] The majority of these neurons fire in response to primary rewar but not to a rewar-preicting CS. The moel lateral hypothalamic input is a rectangular pulse. burst of opamine acting on striosomal D receptors woul also allow long-term potentiation (LTP) in striosomal cells. It has been suggeste that increase Ca 2 combine with a opamine burst coul result in a potentiation of glutamate receptors (LTP) (Houk et al., 995), an opamine bursts have been shown to reverse corticostriatal LTD an instea cause LTP (Wickens et al., 996). Thus, a elaye Ca 2 spike in the striosomal cells coul serve as both a signaling gate an one component of a learning gate. Recent work on the cerebellum (Finch an Augustine, 998; Takechi et al., 998) has supporte the Fiala et al. (996) cerebellar moel an emonstrate the feasibility of irect calcium imaging in local regions of a enritic arbor using high-spee confocal microscopy. We suggest that the same technique coul be use in neostriatal cells to investigate the preictions regaring striosomal Ca ynamics. Pharmacological inactivation of mglur an IP 3 might also verify whether they are essential components of the Ca spike cascae, as in the cerebellum. Functionally, the striosomal cells of the moel nee to receive a sustaine input that is activate when a CS first occurs, as a reference point for the elaye inhibitory signal. Striosomal cells receive excitatory signals from eep layer V of limbic cortex (Gerfen, 992). The sustaine working memory signal initiates a steay rise of the intracellular calcium level, e.g., via an mglur-ip 3 -Ca cascae (as in the cerebellum) (Finch an Augustine, 998; Takechi et al., 998), which causes a calcium spike on reaching a threshol. The sustaine input hereby leas to a elaye, phasic response within the striosomal cell. A relate property of the moel is that if the sustaine input strength is proportional to the CS intensity, then a weaker CS causes an increase in the rise time to threshol, resulting in a slower perceive rate of time passage. This property agrees with behavioral ata (Wilkie, 987), although because of the complexity of cortical processing, the striosomal inputs may not be irectly proportional to external stimulus intensity. The moel simulations assume a simple two-state working memory input that is either on or off an coul be generate by passing a graually rising input through a sharp sigmoial signal function. The maximum elay that a single spectrum can aaptively time is still unknown an nees to be investigate biochemically (cf. Fiala et al., 996). Spectral timing of a single event also nees to be supplemente by inter-event timing mechanisms that involve network interactions, incluing prefrontal cortex an cerebellum (Buonomano an Mauk, 994; Grossberg an Merrill, 996). RESULTS Given the above backgroun, the moel mechanisms can now be summarize as follows (Fig. 4). First, a primary rewar signal is generate in the lateral hypothalamus (Nakamura an Ono, 986) (Fig. 3D). This irectly excites the PPTN (Semba an Fibiger, 992), which fires a brief burst an then accommoates or habituates (Takakusaki et al., 997; Dormont et al., 998). This brief burst irectly excites the SNc by cholinergic an/or glutamatergic projections (Cone, 992) an thereby causes a phasic opamine burst to the striatum (Gerfen, 992) at the time of primary rewar.

5 0506 J. Neurosci., December, 999, 9(23): Brown et al. Basal Ganglia Learning an Unexpecte Rewars Figure 4. Moel circuit. Cortical inputs (I i ) excite by conitione stimuli learn to excite the SNc (D) via the ventral striatal ( S)-to-ventral pallial-to-pptn (P)-to-SNc path. The inputs I i excite the ventral striatum via aaptive weights W is, an the ventral striatum excites the PPTN, via ouble inhibition through the ventral pallium, with strength W SP. When the PPTN activity excees a threshol P, it excites the opamine cell with strength W PD. The striosomes, which contain an aaptive spectral timing mechanism (x ij, G ij, Y ij, Z ij ), learn to generate lagge, aaptively time signals that inhibit rewar-relate activation of SNc. Primary rewar signals (I R ) from the lateral hypothalamus both excite the PPTN irectly (with strength W RP ) an act as training signals to the ventral striatum S (with strength W RS ). Arrowheas enote excitatory pathways, circles enote inhibitory pathways, an hemiisks enote synapses at which learning occurs. Thick pathways enote opaminergic signals. Suppose that a CS is receive an store in prefrontal working memory at some time before the actual rewar. This CS trace generates output signals along aaptive pathways to both the ventral striatum an the striosomes. When primary rewar occurs, a opamine burst facilitates LTP in the limbic cortical ventral striatal path (Brog et al., 993). Thus, the CS representation in limbic prefrontal cortex learns to excite the opamine cells via the limbic cortical ventral striatal ventral pallium- PPTN-SNc pathway (Yang an Mogenson, 987). In the moel, the ventral striatum an ventral pallium are lumpe for simplicity into a single ventral basal ganglia noe, which causes net excitation of the PPTN. The limbic cortical projection to the striosomes (Gerfen, 992; Eblen an Graybiel, 995) activates a spectrum of elaye Ca 2 spikes in the striosomal cells via metabotropic glutamate receptors. When a opamine burst arrives from the SNc, it strengthens the CS-activate limbic cortical connections to any currently spiking components of the striosomal timing spectrum. The striosomal cells hereby learn to inhibit the opamine burst at its expecte time via the inhibitory striosomal SNc path (Gerfen, 992). On a later trial in the traine moel, when the CS is receive at the expecte time before an actual rewar, its working memory trace tonically activates the ventral striatal moel cell, which in turn excites the PPTN, causing an immeiate opamine burst in the SNc. The aaptively time inhibition via the striosomal cells then inhibits the SNc so that the subsequent primary rewar signal oes not elicit a opamine burst in the SNc. If the primary rewar signal is absent on a trial, then the striosomal inhibition causes a phasic ip in the opamine signal. These three properties explain the opamine cell ata of Figure. Figure 5. Striosomal spectral timing moel an closeup (inset), showing iniviual timing pulses. Each curve represents the suprathreshol intracellular Ca 2 concentration [G ij Y ij s ] of one striosomal cell. The peaks are sprea out in time so that rewar can be preicte at various times after CS onset, by strengthening the inhibitory effect of the striosomal cell with the appropriate elay. The moel uses 40 peaks, spanning 2 sec an beginning 00 msec after the CSs (Grossberg an Schmajuk, 989). Moel properties are robust when ifferent numbers of peaks are use. It is important that the peaks be sufficiently narrow an tightly space to permit fine temporal resolution in the rewar-canceling signal. However, a trae-off ensues in that more time signals must be use as the time between peaks is reuce. The time signals must not begin too early after the CS, or they will erroneously cancel the CS-inuce opamine burst. The 00 msec post-cs onset elay prevents this from happening. The moel was also use to simulate various other task situations for which opamine cell responses are known. It successfully reprouce all the key SNc opamine cell ata (Figs., 2) as well as firing patterns of known cell types in the PPTN (Fig. 3A) an ventral striatum (Fig. 3B), which are afferent to the nigral opamine cells. In particular, opamine cell responses were simulate in eight task situations (Figs., 2). First, the moel receive primary rewar (R) only an showe a strong response to the rewar (Fig. A). We then traine the moel with a CS preceing R. During training, the moel fire weakly in response to both the CS an R (Fig. 2B). As training neare completion, the moel SNc respone strongly an only to the CS (Fig. B). In the traine moel, we examine the effect of omitting R an foun a transient epression at the preicte time of rewar (Fig. C). To test the effects of higher-orer conitioning, we first traine the moel with the CS R association. Then we introuce an aitional conitione stimulus (CS2), which consistently occurre sec before the CS. With training, the moel opaminergic cells learne to respon only to CS2 (Fig. 2A). Recent work has examine opamine cell responses uner conitions of variable rewar timing (Hollerman an Schultz, 998). The moel successfully simulate these ata as well. When the rewar R was elaye (Fig. 2C), moel opamine cells respone with the characteristic epression at the expecte time of R an then showe a burst later when R i occur. Similarly, if R occurre before the expecte time, moel opamine cells again showe a burst in response to R. They i not, however, show a ip at the expecte time of R (Fig. 2D), in agreement with

6 Brown et al. Basal Ganglia Learning an Unexpecte Rewars J. Neurosci., December, 999, 9(23): the ata, because the working memory trace shut off when R was receive. In some cases, the timing of primary rewar may vary from trial to trial because of its epenence on an operant response. The moel opamine response was simulate when the timing of R varie ranomly on an interval spanning 200 msec before an after the expecte (mean) time of R, with a uniform ranom istribution. This cause moel striosomal cells to learn to inhibit the opamine signal uring the entire interval in which the opamine bursts occurre. Because this interval of inhibition is wier than the opamine burst, moel striosomal cells prouce tails of epresse firing on either sie of the opamine burst (Fig. 2E), generating a kin of temporal Mexican hat function, as in the ata (Schultz et al., 993). The PPTN moel responses also agree with the cell recoring ata from conitioning tasks (Dormont et al., 998), which show transient bursts in response to both CS an R (Fig. 3A). In aition, when a CS2 precee the CS, the moel PPTN response to the later CS isappeare. This lack of response to subsequent CSs agrees with the ata of Dormont et al. (998), which show a similar isappearance of the CS-inuce PPTN response in that elay task. Moel ventral striatal cells also simulate known cell firing patterns (Fig. 3B). After the moel learne the CS R association, CS onset prouce tonic activity, followe by a phasic burst in response to the R signal from the hypothalamus (Fig. 3D). DISCUSSION The present moel explains an preicts significantly more ata than previous moels through its use of parallel learning pathways. Several moels have attempte to escribe the opamine cell behavior by a TD algorithm (Montague et al., 996; Schultz et al., 997; Suri an Schultz, 998). These moels suggest that the opaminergic SNc cells compute a temporal erivative of preicte rewar. In other wors, they fire in response to the sum of the time-erivative of rewar preiction an the actual rewar receive. These moels have not been linke with structures in the brain that might compute the require signals. The Suri an Schultz (998) moel has simulate much of the known opamine cell ata. However, their moel can only learn a single fixe interstimulus interval (ISI) that correspons to the longesturation time signal [x lm (t)] in their moel. If the ISI is shorter than this, opamine bursts will strengthen all of the active stimulus representations preicting rewar at the time of the opamine burst or later. Thus, their moel generates inhibitory rewar preictions beyon the primary rewar time an preicts a lasting epression of opamine firing subsequent to primary rewar, which is not foun in the ata. In contrast to TD moels that compute time erivatives immeiately before opamine cells, our spectral timing moel uses two istinct pathways: the ventral striatum an PPTN for initial excitatory rewar preiction an the striosomal cells for time, inhibitory rewar preiction. The fast excitation an elaye inhibition are hereby compute by separate structures within the brain, rather than by a single temporal ifferentiator. This separation avois the problem of the Suri an Schultz (998) moel by allowing transient rather than sustaine signals to cancel the primary rewar signal, thereby enabling precisely time rewarcanceling signals to be traine, an preventing spurious sustaine inhibitory signals to the opamine cells. This separation also allows the inhibitory system to follow an precisely cancel the real-time ynamics of the primary rewar signal, as in Figure B, where the striosomal signals cancel the opamine burst espite its asymmetry. Where temporal uncertainty exists in rewar preiction, the tails of inhibition (Fig. 2E) in the ata are explaine by the moel s ability to learn temporally istribute net inhibitory signals that track the temporal ispersion of rewar. Like our moel, the TD moel of Schultz et al. (997) uses transient rather than sustaine timing signals. However, because this moel oes not separate the computation of excitation an inhibition, each transient pulse is temporally ifferentiate to prouce an onset burst followe by an offset epression. Over the course of many trials, the onset burst strengthens its preceing time signal weight, thereby recursively chaining backwar until all time signal weights between the CS an R have been activate by learning. This preicts that the opamine burst graually travels backwar in time an that the rewar response extinguishes well before the CS response occurs. The ata show instea that opamine bursts o not occur systematically in the mile of the ISI uring training, an moreover, the opamine burst occurs concurrently at both CS an R uring iniviual training trials (Ljungberg et al., 992). The Contreras-Vial an Schultz (997) moel of the opamine cell system is base partly on the ART2 moel (Carpenter an Grossberg, 987). They first suggeste that striosomes may generate a spectrum of aaptively time rewar preictions, base on the earlier spectral timing moels of Grossberg an colleagues (Grossberg an Schmajuk, 989; Grossberg an Merrill, 992, 996; Fiala et al., 996). Their striosomal moel nonetheless faces problems because it relies on lateral inhibition among striosomal cells, rather than intracellular timing mechanisms. GABAergic lateral inhibition among striosomal cells is weak (Jaeger et al., 994; Wilson, 995) an may not be strong enough to meiate the competitive choices require by their moel. In aition, their moel assumes aaptively time inhibitory rewar preiction learning at the striosomal SNc synapses instea of at the corticostriosomal synapses. This fails to incorporate ata on corticostriatal LTP/ LTD (Wickens an Kotter, 995). In their moel, corticostriatal LTP/ LTD woul cause erroneous timing preictions because the cell with the strongest corticostriatal input becomes active first an generates its aaptively time signal, whereas it suppresses its competing neighbor cells via strong lateral inhibition. After this, the winning cell remains refractory, an the cell with the next strongest corticostriosomal weight becomes active, an so on. If learning occurs in the corticostriosomal path, as much evience suggests, then the rank orering of corticostriosomal weights may change as the synaptic weights change relative to each other. This woul cause erroneous rewar timing preictions, because the moel striosomal cells woul become active in the wrong sequential orer. Our moel avois these problems by escribing an intracellular mglur-meiate aaptive timing mechanism rather than an extracellular one. Another significant ifference between the present moel an that of Contreras-Vial an Schultz (997) is the source of excitation to the opamine cells. Their moel assumes that matrisomal cells provie the excitatory input to SNc cells inirectly, via ouble inhibition through the substantia nigra pars reticulata (SNr). This polysynaptic, matrisomal cell-snr-snc pathway cannot be rule out as a source of net excitation to the opamine cells, but as we have shown above, it is not the main pathway of SNc excitation. It shoul also be pointe out that although the present moel attempts to represent the principal circuitry responsible for opamine cell responses, aitional afferent circuitry exists that may also be capable of eliciting phasic opamine

7 0508 J. Neurosci., December, 999, 9(23): Brown et al. Basal Ganglia Learning an Unexpecte Rewars cell responses, e.g., the SNr SNc projection, an the STN PPTN an STN SNc projections. Houk et al. (995) moele opamine cell firing using the irect an inirect basal ganglia pathways. They assume that the polysynaptic, net excitatory inirect path through the basal ganglia is faster than the monosynaptic, irect path. The inirect path is propose to generate the initial excitatory opamine burst, whereas the irect path is propose to meiate the slower inhibition of the opamine cells. With regar to the fast excitation of the opamine cells, Houk et al. (995) cite ata showing that striatal stimulation results in a fast EPSP followe by a slower IPSP in the globus pallius (Kita an Kitai, 99). However, it is unlikely that the EPSPs are polysynaptic, because they coul be elicite with as little as 2 msec latency (Kita an Kitai, 99). Likewise, the fast EPSP that results from cortical excitation (Kita, 992) might be better explaine as from a cortical-stn-pallial route. Moreover, STN activity may moulate rather than excite the SNc (Smith an Grace, 992). These ata contraict Houk an colleagues (995) assumption of net striatal SNc excitation via the moel inirect pathway. The ata are probably cause by STN SNr excitation an subsequent SNr SNc inhibition (Hajos an Greenfiel, 994; Tepper et al., 995). With regar to the slow inhibition of the opamine cells, Houk et al. (995) propose that the irect path provies a prolonge inhibition of the opaminergic cells, which persists from the time of the rewar-preicting CS through the time at which the rewar occurs. This is inconsistent with the ata in two istinct but relate ways. First, when the rewar-preicting CS occurs, it prouces a opamine burst, but the opamine cell firing then immeiately returns to baseline. There is no persistent epression in opamine cell firing, although the Houk et al. (995) moel must preict such a persistent epression. Secon, when an expecte rewar is omitte, there is a brief epression in the opamine cell firing, after which it immeiately returns to baseline. The Houk et al. (995) moel instea preicts a prolonge (although below baseline) response rather than a transient response to the omission of expecte rewar. The Berns an Sejnowski (998) moel suggests that the primary source of net SNc excitation is the pallium, via a hypothetical inhibitory neuron. No suggestion is given regaring the location of this neuron or from which pallial segment (internal or external) the signal originates. As in our moel, the Berns an Sejnowski (998) moel assumes that the striosomal cells are the main source of inhibition to the SNc, but their moel oes not treat opamine cell temporal ynamics, which woul be necessary for it to explain the ata of Figures an 2. The new spectral timing moel of nigral opamine activity provies functional explanations of known SNc afferents. The moel suggests how the ventral basal ganglia stream learns an excitatory preiction of rewar via the PPTN, whereas the striosomal cells learn an aaptively time inhibitory preiction of rewar. This analysis clarifies how the nigral opamine cells are linke to four other cell types that are irectly or inirectly afferent to the SNc: ventral striatal cells, PPTN cells, striosomal cells of the basal ganglia, an cells in the lateral hypothalamus. The moel preicts that an aaptive timing mechanism occurs at the striosomal cells. Key explanatory limitations of previous moels, incluing TD an irect/inirect pathway moels of nigral opamine cell responses, are overcome by the present moel. Table. Moel variables S I R W is N N I i P U P x ij G ij Y ij Z ij D D r j M APPENDIX This section lists the mathematical equations an parameters of the moel. The circuit in Figure 4 was moele using neurons with a single-voltage compartment. The moel variables are summarize in Table, an the fixe parameters are summarize in Table 2. The variables in Figure 4 obey the following equations. Moel ventral striatal cell activity S respons at rate S an is excite by primary rewar inputs I R an by CS inputs I i that are gate by aaptive weights W is : S t S A ss S I i W is I R W i RS. () The CS-to-striatal weights W is change only when S is positive. They are potentiate by a positively reinforcing opamine burst N an epresse by a negatively reinforcing opamine epression N, escribe below. The weights W is range between a minimum of zero an a maximum of W S max I i, an they ecay at a rate WS with negative reinforcement: WS t W is S N I i W S max W is WS N W is. (2) The PPTN activity P is excite by striatal inputs S an primary rewar inputs I R : P t P U PW UP P P SW SP I R W RP. Accommoation, or habituation, of PPTN activity is moele as a lasting afterhyperpolarization, which reuces the excitability of the PPTN in an activity-epenent way: UP Ventral striatal cell Rewar input signal from lateral hypothalamus CS-to-striatum synaptic weights Above-baseline opamine burst signal Below-baseline opamine ip signal CS input signal PPTN cell activity PPTN cell afterhyperpolarization Striosomal metabotropic response Striosomal calcium concentration CS input-to-striosomal synaptic weights Dopamine cell activity Baseline average opamine signal Striosomal activity builup rate parameter Membrane potential riving integrate-an-fire (IAF) spiking moel Gaussian noise input to IAF moel (3) t U P U P U P P. (4) The opamine cell activity D is excite by the rectifie PPTN activity [P P ], where P is a signal threshol, an a tonic arousal signal I D. The notation [x] max(x,0) enotes rectification. The opamine cell activity D is inhibite in an aaptively time fashion by the summe spectrum of signals:

8 Brown et al. Basal Ganglia Learning an Unexpecte Rewars J. Neurosci., December, 999, 9(23): Table 2. Moel parameters Symbol Description Value r Striosomal spectrum spacing 50.0 r Striosomal spectrum offset.0 G Calcium spike threshol 0.37 G Calcium activation rate 5.0 G Calcium passive ecay rate 20.0 B G Calcium concentration maximum 5.0 y Calcium recovery rate.0 y Activity-epenent calcium inactivation rate 80.0 Y Calcium inactivation threshol 0.8 S Striosomal output threshol 0.2 s Striosomal learning gain 0000 z Striosomal learning rate 0. w RS Hypothalamus-to-ventral striatum synaptic weight.2 S Ventral striatal cell response time constant 30.0 WS CS-to-ventral striatal learning rate 20.0 max W S Maximum CS-to-ventral striatal synaptic weight 2.5 WS CS-to-ventral striatal weight ecay rate 0.2 A S Ventral striatal activity passive ecay rate 0.7 N Phasic opamine signal threshol 0.0 P PPTN cell response time constant t UP PPTN afterhyperpolarization time constant 4.0 D Dopamine cell response time constant 5.0 W PD PPTN-to-opamine cell synaptic weight 50.0 W SP Ventral striatal-to-pptn cell synaptic weight 2.0 W RP Hypothalamus-to-PPTN cell synaptic weight 0.8 W UP PPTN afterhyperpolarization gain 40.0 P PPTN output signal threshol 0.35 D Baseline average opamine time constant 4.0 I D Tonic input to opamine cell 0.5 h D Dopamine cell maximum hyperpolarization 0. V I Integrate-an-fire (IAF) moel output 0.5 R IAF moel membrane resistance 333 C IAF moel membrane capacitance noise IAF Gaussian noise input 0.4 R DA IAF opamine cell membrane resistance 80 R PPTN IAF PPTN cell membrane resistance 6667 C PPTN IAF PPTN cell membrane capacitance PPTN IAF PPTN cell Gaussian noise input 0. from the striosomal cells: D G ij Y ij S Z ij (5) i,j t D D D P P W PD I D D h D G ij Y ij S Z ij. (6) i,j A tonic opamine signal is compute as a time average of the momentary opamine cell potential: D t D D D. (7) Transient eviations from this tonic signal constitute reinforcement learning signals (Wickens et al., 996). The positive reinforcement learning signal N erives from excitatory phasic fluctuations of the opamine signal above the baseline: N D D N. (8) The complementary negative reinforcement learning signal is erive from inhibitory phasic fluctuations of the opamine signal below baseline: N D D N. (9) Spectral timing in the striosomal cells is meiate by a number of interacting factors, which are represente by the simplifie intracellular system of Equations 0 4. A moel of spectral timing in the cerebellum has elsewhere propose etaile biochemical correlates of this type of learning in terms of mglur, Ca 2, Ca-epenent K channels, an intracellular secon messengers. See Fiala et al. (996) for this biochemically etaile treatment. Here we simplify an aapt this moel to provie a phenomenological account of intracellular processes that oes not attempt to preict the exact concentrations of particular chemical species. Subscript i inexes which CS activates the cells, whereas subscript j inexes the response rate of the j th population of cell sites in the striosomal cell. It is important to note that the moel oes not require a ifferent cell for each CS at each response rate, or elay, which woul lea to a combinatoric explosion. Instea, multiple CSs synapse onto a single set of striosomal cells that span a spectrum of elays. In aition, not all CSs may be represente. Ventral prefrontal cortex (which provies much of the striosomal input signals) seems to preferentially represent CSs that have some motivational salience (Tremblay an Schultz, 999). The spectrum-sharing property of the moel is mae possible by the intracellular rather than extracellular elay timing mechanism, which allows a issociation between the cortical (CS)- to-striosomal connection strength an the striosomal cell fixe Ca spike elays. The possibility of interference among coactive CSs woul still necessitate more than a single striosomal spectrum, possibly at ifferent enritic sites (cf. Fiala et al., 996). Cell recorings in SNc, PPTN, ventral striatal, an limbic cortical cells uring multiple overlapping stimulus-elaye rewar tasks might eluciate the nature of cortical CS representations an the extent to which CS signals may converge or interfere with each other in the excitatory an inhibitory pathways. The moel preicts that multiple excitatory CS signals converging on the same opamine cell will elicit multiple opamine bursts in the traine animal, provie that the CSs are not preictably paire uring training. Likewise, the moel preicts that multiple CSs converging on the same striosomal cell may impair the ability of that particular cell to preict later rewars in a series uring overlapping tasks. These preictions have yet to be teste. The spectral timing ynamics of the moel are efine as follows. Striosomal cell activity x ij respons to the i th CS at rate r j : t x ij r j x ij x ij I i. (0)

9 050 J. Neurosci., December, 999, 9(23): Brown et al. Basal Ganglia Learning an Unexpecte Rewars To provie a range of aaptively time Ca 2 spikes, the striosomal builup rate parameter spans a range of values for a given set of cells: r r j, j, 2,..., n. () r j The activities x ij inuce intracellular calcium ynamics to cause transient calcium spikes at elays that are etermine by r j. These Ca 2 spikes etermine the times at which the corresponing cells can learn from a opamine burst. In particular, quantity [G ij Y ij ] represents an intracellular Ca 2 spike (Grossberg an Merrill, 992), where an t G ij G B G G ij f G x ij G G G ij (2) t Y ij Y Y ij y G ij Y ij Y. (3) In Equation 2, f G (x) is a step function: 0 for x 0, for x 0. Parameters G an Y in Equations 2 an 3 are signal threshols. When G ij is activate by suprathreshol striosomal cell firing at a rate that varies with r j, it rapily increases the intracellular Ca 2. As the calcium concentration rises to its maximal level, the available Ca 2 (Y ij ) rapily ecreases, causing a rapi falloff in the Ca 2 concentration. The Ca 2 concentration remains low as long as the mglur receptors receive tonic input. Subsequent Ca spikes occur only when the tonic input is remove long enough for reset, in which the mglur receptor an available Ca return to baseline. In the brief interval when the calcium concentration excees the activity threshol S in Equation 6, striosomal cell transmitter release is significantly enhance, an the CS striosomal weight Z ij is potentiate via LTP if a opamine burst is receive: t Z ij z G ij Y ij S Z ij S N N. (4) Simulate spike trains were generate with an integrate-an-fire (IAF) moel using the cell membrane potentials M as input (efine for cells in Eq., 3, 6, an 0 above, by variables S, P, D, an x ij, respectively, an shown in Figs. 3B (S), 3A (P), an 2 (D), an 3C (x ij )): M V t C V. (5) RC The noise term was Gaussian with variance 2 noise. When the voltage exceee a threshol V I value, a spike was generate, an the voltage was reset to 0. Moel outputs were compute from the moel spiking response for 20 trials, an the moel spikes were groupe into 20 msec-wie bins to compute histograms. The efault IAF parameters (Table 2) were V I 0.5, R 333, C 0.025, noise 0.4, except that for the opamine cell, R 80; for the PPTN cell, R 6667, C 0.005, an noise 0.. The ifferent R an C values were necessary to moel the ifferent firing properties of the cells. The moel performe a series of simulate learning trials. Each trial laste 0 sec. The CS was active for 2 sec, an the R was active for 750 msec uring the CS, beginning.2 sec after CS onset. Numerical integration was performe with an aaptive step size fourth-orer Runge-Kutta metho except for the IAF moel, which use a first-orer metho an a iscrete stepsize of 0.00 sec. The aaptive stepsize output was converte to a fixe stepsize by linear interpolation, so that it coul be use to rive the IAF moel. The CS was active from t 2 sec into the trial, an it shut off when the primary rewar signal shut off, or after t 3.95, whichever was earlier. The primary rewar signal typically began at t 3.2 an laste for 750 msec, with a magnitue of.0. The CS input (I CS ) ha an amplitue of 0.6. REFERENCES Berns G, Sejnowski T (998) A computational moel of how the basal ganglia prouce sequences. J Cognit Neurosci 0:08 2. Berrige K, Robinson T (998) What is the role of opamine in rewar: heonic impact, rewar learning, or incentive salience? Brain Res Rev 28: Brog J, Salyapongse A, Deutch A, Zahm D (993) The patterns of afferent innervation of the core an shell in the accumbens part of the rat ventral striatum: immunohistochemical etection of retrograely transporte fluoro-gol. J Comp Neurol 338: Buonomano DV, Mauk MD (994) Neural network moel of the cerebellum: temporal iscrimination an the timing of motor responses. Neural Comput 6: Calabresi P, Maj R, Pisani A, Mercuri N, Bernari G (992a) Long-term synaptic epression in the striatum: physiological an pharmacological characterization. J Neurosci 2: Calabresi P, Pisani A, Mercuri N, Bernari G (992b) Long-term potentiation in the striatum is unmaske by removing the voltageepenent magnesium block of NMDA receptor channels. Eur J Neurosci 4: Calabresi P, Pisani A, Mercuri N, Bernari G (994) Post-receptor mechanisms unerlying striatal long-term epression. J Neurosci 4: Carpenter G, Grossberg S (987) ART 2: Self-organization of stable category recognition coes for analog input patterns. Appl Optics 26: Cone H (992) Organization an physiology of the substantia nigra. Exp Brain Res 88: Cone H, Dormont J, Farin D (998) The role of the peunculopontine tegmental nucleus in relation to conitione motor performance in the cat. II. Effects of reversible inactivation by intracerebral microinjections. Exp Brain Res 2:4 48. Contreras-Vial J, Schultz W (997) A preictive reinforcement moel of opamine neurons for learning approach behavior. First International Conference on Vision, Recognition, an Action: Neural Moels of Min an Machine. Department of Cognitive an Neural Systems, Boston University, Boston, MA, May 997. Crepel F, Hemart N, Jaillar D, Daniel H (996) Cellular mechanisms of long-term epression in the cerebellum. Behav Brain Sci 9: Dias R, Robbins T, Roberts A (996) Dissociation in prefrontal cortex of affective an attentional shifts. Nature 380: Dormont J, Cone H, Farin D (998) The role of the peunculopontine tegmental nucleus in relation to conitione motor performance in the cat I. Context-epenent an reinforcement-relate single unit activity. Exp Brain Res 2: Eblen F, Graybiel A (995) Highly restricte origin of prefrontal cortical inputs to striosomes in the macaque monkey. J Neurosci 5: Fiala J, Grossberg S, Bullock D (996) Metabotropic glutamate receptor activation in cerebellar purkinje cells as substrate for aaptive timing of the classically conitione eye-blink response. J Neurosci 6: Finch EA, Augustine GJ (998) Local calcium signalling by inositol-,4,5-trisphosphate in Purkinje cell enrites. Nature 396: Funahashi S, Bruce CJ, Golman-Rakic PS (989) Mnemonic coing of visual space in the monkey s orsolateral prefrontal cortex. J Neurophysiol 6: Futami T, Takakusaki K, Kitai S (995) Glutamatergic an cholinergic inputs from the peunculopontine tegmental nucleus to opamine neurons in the substantia nigra pars compacta. Neurosci Res 2: Gallagher M, Chiba A (996) The amygala an emotion. Curr Opin Neurobiol 6: