A modular neural-network model of the basal ganglia s role in learning and selecting motor behaviours

Transcription

1 Cognitive Systems Research 3 (2002) locate/ cogsys A modular neural-network model of the basal ganglia s role in learning and selecting motor behaviours Action editors: Wayne Gray and Christian Schunn Gianluca Baldassarre* Via di Pietralata 430 G/8, Roma, Italy Received 1 March 2001; acceted 1 Setember 2001 Abstract This work resents a modular neural-network model (based on reinforcement-learning actor critic methods) that tries to cature some of the most relevant known asects of the role that basal ganglia lay in learning and selecting motor behavior related to different goals. The model uses a mixture of exerts network for the critic and a hierarchical network with two levels for the actor. Some simulations with the model show that basal ganglia select chunks of behavior whose details are secified by direct sensory-motor athways, and how emergent modularity can hel to deal with tasks with asynchronous multile goals. A to-down aroach is adoted that first analyses some adative non-trivial interaction of a whole (simulated) organism with the environment, and its caacity to learn, and then attemts to imlement these functions with neural architectures and mechanisms that have an emirical neuroanatomical and neurohysiological foundation Elsevier Science B.V. All rights reserved. Keywords: Basal ganglia; Double-inhibition mechanism; Doamine; Modularity; Neural networks; Mixture of exerts networks; Reinforcement learning; Actor critic methods; Multi-goal tasks 1. Introduction and methodology dealt with by the same sensory-motor athway (imlemented by a neural module), while in the latter What is the role that basal ganglia lay in mam- cases different sensory-motor athways are selected. mals sensory-motor behavior? When organisms In fact if the behavioral resonse to associate with a have different needs and goals, sometimes they have given ercetion were different with different needs to associate slightly different behaviours with the and goals, using the same neural synases and same ercetion atterns, some other times they have athways would only cause interference. In this to associate comletely different behaviours with context the basal ganglia could lay a role in them. This work resents some simulations that selecting different sensory-motor athways when suggest that in the former case the differences are necessary. This work follows a to-down aroach, where *Tel.: ; fax: the starting oint of analysis is organisms behavior address: gianlucabaldassarre@hotmail.com (G. Baldassarre). and learning rocesses (cf. Meyer & Guillot, 1990). With this urose it resents a simulation of an / 02/ $ see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S (01)

2 6 G. Baldassarre / Cognitive Systems Research 3 (2002) 5 13 organism that has different needs (signals coming 2. The basal ganglia and the emirical evidence from the body and indicating a hysiological unbal- addressed ance (cf. Rolls, 1999)) or, alternatively, different goals (desired states of the body and the world) This section first resents a brief review of the associated with different ositions in the environ- basic anatomy of basal ganglia, and then illustrates ment (for examle we can assume that these differ- the emirical evidence secifically addressed by this ent ositions are occuied by resources that satisfy research. Basal ganglia (Houk, 1995; Houk, Adams different needs). The organism learns through classi- & Barto, 1995) are a set of nuclei situated in the cal and instrumental learning ((Lieberman, 1993); in dee art of the two cerebral hemisheres, inside the (Baldassarre & Parisi, 2000), these two learning white substance under the cortex. Basal ganglia mechanisms are integrated in a comrehensive receive their inut from the whole cerebral cortex actor critic model. Cf. (Barto, 1995; Sutton & Barto, and send their outut to the frontal areas of cortex, to 1998), for this model) to navigate in the environment some non-cortical motor systems (suerior colliculus in order to reach those ositions. Given this be- and substantia nigra ars reticulata) and to doahavior, the work attemts to roduce it by building a minergic systems (such as the substantia nigra ars neural-network controller that satisfies (some of) the comacta) in the midbrain. Fig. 1 shows the main constraints coming from the known emirical evi- nuclei that make u the basal ganglia, and the other dence about basal ganglia. Since the starting oint of nuclei and areas to whom they are connected. The this aroach is to simulate sohisticated organisms striatum (mainly made u by the utamen and behaviors, sometimes there is no emirical data nucleus caudatus) constitutes the inut comonent of suggesting which mechanisms underlie them. In the basal ganglia, while the globus allidus (internal these cases some comutational solutions are and external) constitutes their outut comonent. adoted that do not have a known emirical corre- The substantia nigra ars reticulata in the midbrain sondent (they will be referred to as arbitrary in the lays an outut role for the basal ganglia functionally rest of the aer). These solutions should be consid- similar to the role of the internal globus allidus. The ered as a useful theoretical exercise, eventually connections between the striatum and these two suggesting interesting ideas for emirical investiga- nuclei form the direct athway. This athway is tion, and should not be judged too severely on the articularly imortant for the control of movement. basis of the current neural evidence. In fact, the substantia nigra ars reticulata contri- Fig. 1. The nuclei of the basal ganglia (bold boxes) and their connections with other areas and nuclei. Dashed boxes: nuclei of the midbrain. Solid arrowheads: inhibitory connections. Thin arrowheads: excitatory connections. Dashed arrows: doaminergic signal. S: striosomes. M: matrix. Some connections shown in the grah are not relevant for the model resented here (cf. Fig. 3).

3 G. Baldassarre / Cognitive Systems Research 3 (2002) butes to control the eyes movement via the suerior Houk, Adams & Barto (1995) suggest a ossible colliculus, while the internal globus allidus contri- corresondence between the actor critic models butes to control the skeletal movements via the architecture and functioning (Barto, 1995; Sutton & thalamus and the frontal areas (refrontal, remotor Barto, 1998) and the architecture of the basal and motor areas). The striatum also rojects to the ganglia. In articular they roose that the cirexternal globus allidus that, in turn, rojects, via the cumscribed regions called striosomes (differently subthalamus, to the internal globus allidus and from matrisomes, they are identifiable for their substantia nigra ars reticulata. This is called the chemical make-u and outut connectivity, cf. Fig. indirect athway. The striatum also rojects, direct- 1) may imlement the function of the critic: redictly and indirectly, to the substantia nigra ars com- ing future rewards and yielding a ste-by-ste reward acta. This is an imortant athway since this signal in cases of delayed rewards. The surrounding nucleus contains doaminergic neurons that roject matrix regions may imlement the function of the to the striatum itself and to the frontal areas and lay actor: selecting actions or, as in the model resented an imortant role in learning (see below). here, sensory-motor athways. As we shall see, the The anatomical and hysiological evidence se- actor critic model is at the base of the model cifically addressed in this work is now illustrated. resented here. Chevalier and Deniau (1990) roose that a double- Lots of other asects of these contributions have inhibition mechanism is the basic rocess of the been incororated into the model, and will be basal ganglia s functioning (Fig. 1). They reort that resented in detail in the next section. The numerous in some exeriments where monkeys have to carry brain-imaging studies of basal ganglia s role in out a delayed saccade to a remembered target, some sequence learning are not directly addressed in this striatal cells (usually mute) are induced to fire with aer (see (Graybiel, 1998), for some references). local injection of glutamate. The striatal discharge Similarly the imortant role that basal ganglia lay in inhibits (via GABAergic connections) a grou of working memory is not addressed in this aer (see cells in the substantia nigra ars reticulata (usually Goldman-Rakic, 1995; Frank, Loughry & O Reilly, tonically active) that release from (GABAergic) 2000). inhibition a subset of cells of the suerior colliculus resonsible for the saccade. In the case of skeletal movements the striatum globus allidus thalamus 3. Scenario and model of basal ganglia athway imlements the double inhibition releasing mechanism. The authors reort that while in rodents The environment used in the simulations is a this mechanism is sufficient to trigger movements, in square arena with sides measuring 1 unit (Fig. 2). the reorted exeriments the execution of a saccade The organism cannot see the boundaries of the arena requires temoral coincidence of basal ganglia dis- and cannot exit it. Inside the arena there are 5 inhibition with command signals from other sources. circular landmarks/ obstacles that the organism can The model resented here reroduces this asect: see with a one-dimension horizontal retina covering basal ganglia select a articular sensory-motor ath- 360 degrees with 50 contiguous sensor units. Each way that then yields the detailed behavioral outut. unit gets an activation of 1 if a landmark is in its Graybiel (1998), addressing the role that the basal scoe, 0 otherwise, and is affected by noise (0.01 ganglia s neural modules lay in human slow habit robability of fliing). The signals coming from the learning and animal stimulus resonse association, retina are aligned with the magnetic north through a draws an abstract arallel between the striatum s simulated comass whose reading is affected by anatomical organization in artly interconnected Gaussian noise (0 mean, 1 degree variance). Before zones, called matrisomes, and the modular architec- being sent to the controller, these signals are reture of the neural networks roosed by Jacobs, maed into 100 binary units reresenting the image Jordan, Nowlan & Hinton (1991). As we shall see, contrasts (these units imlement edge detection). the comutational model resented here rooses a Two contiguous retinal units activate one contrast ossible way to secify such a arallel. unit if they are resectively on and off, another

4 8 G. Baldassarre / Cognitive Systems Research 3 (2002) 5 13 contrast unit if they are resectively off and on, no contrast units if they are both on or both off (cf. Fig. 2). At each cycle of the simulation the organism selects one of eight actions, each consisting of a 0.05 ste in one of eight directions aligned with the north (north, northeast, east, etc.). The outcome of these actions is affected by a Gaussian noise (0 mean, 0.01 variance). As a consequence each action takes the organism to a osition within a small area centered on the oint where it intended to go. The organism s task is to reach one of the three goal ositions shown in Fig. 2. Fig. 3 illustrates the main features of the organism s controller and the ossible brain areas and nuclei corresonding to the model s comonents. Now a comutational descrition of the controller is given, and its ossible links to the neural structures of mammals brain are illustrated. Notice that the units used in the model sometimes reresent whole neural assemblies and at other times single units. The matcher is a (arbitrary) hand-designed net- work resonsible for generating an internal reward signal r by detecting the similarity between the goal Fig. 2. (To) The scenario of the simulations containing three goals (marked with 3), five landmarks (black circles), the scoe of the organism s 50 visual sensors (delimited by the rays), and the organism (white circle at origin of rays). (Bottom) The activation of the visual sensors, its re-maing into contrasts, and the bottom left goal (contrast attern). Fig. 3. The comonents of the organism s controller. Labels in italics indicate the ossible brain areas and nuclei corresonding to the model s comonents. Thin arcs indicate one-to-one connections with weight 11 when not differently indicated. Dashed thin arrows indicate unit-to-unit or unit-to-area inhibitory connections (strong enough to make the target units and areas silent). Bold arrows indicate connections udated on the basis of the doaminergic signal. Dashed bold arrows indicate the doaminergic signal.

5 G. Baldassarre / Cognitive Systems Research 3 (2002) and the current inut contrasts. A goal is the select one action, the activation m k (interretable as contrasts attern corresonding to the goal osition. action merit ) of the outut units is sent to the When these atterns have at least 94% of bits with frontal areas where a stochastic winner-takes-all same value, the matcher returns 1 otherwise it cometition takes lace (cf. (Hanes & Schall, 1996), returns 0. The threshold 94% has been chosen regarding this ossibility). The execution of one because it roduces a satisfactory small size of the action has to be thought of involving the activation area recognized as goal. It is assumed that some memory rocess, not simulated in the model, evokes the goal atterns. When a goal is reached, another goal is evoked that is randomly chosen between the three goals. In real brains, goal atterns may be P[ak5 a w] 5 mkyo fm f. generated within frontal areas, for examle by the frontal eye fields in the case of saccades, and of a articular muscle temlate. The robability P that a given action ak becomes the winning action aw to execute is given by The role of the basal ganglia is to select an exert recognition could take lace here or in the sensory which, in its turn, has to select the actions to be areas themselves (see Fig. 3; cf. (Kosslyn, 1999), on executed through the mechanism of double inhibition this issue). illustrated reviously involving the matrix of the There is an alternative way to view this art of the striatum and the globus allidus. This is done with model. Animals are endowed with innate neural another winner-takes-all cometition analogous to structures that take inut from the environment and the revious one, but this time involving the exerts ma it into a reward or unishment internal instead of the actions (could this mechanism corresignal. This usually haens when some states of the sond to the bistable behavior of the striatum siny environment are achieved that are relevant for cells?). Notice that the basal ganglia can only adatation, for examle some food is ingested or the revent the roer exert from being inhibited, but body is hurt (rimary reinforcements). Notice that cannot trigger an action directly. these signals are roduced only if a corresondent The critic is a mixture of exerts network aetitive need (e.g. hunger) is resent (Rolls, 1999). (Jacobs, Jordan, Nowlan & Hinton, 1991) based on 6 In the model the resence of a certain need could be exert networks. Each exert is a two-layer feedthought of as corresonding to an arbitrary attern forward neural network that gets the goal and the (the goal attern) coming from the body, while the visual contrasts as inut and has one linear outut signal relevant for adatation is the signal coming unit. The critic learns to roduce the estimation from the sensors, for examle from the sensors in the V 9 [s t] of the evaluation V [s t] of the current mouth that detect the ingestion of food. In this case contrast attern s t. V [s t] is defined as the exected the matcher would yield a rewarding signal when a discounted sum of all future reinforcements r, given need and the corresonding satisfying inut attern the current action-selection olicy exressed by are resent together, and it would corresond to the actor limbic structures (cf. Rolls, 1999). In both cases the matcher s signal arrives to the substantia nigra ars V [s t] 5 E[g rt11 1 g rt12 1 g rt13 1???], comacta, and this generates a doaminergic signal that controls learning. where g [ [0, 1] is the discount factor, set to 0.95 in The actor, with the 6 exert networks (6 differ the simulations, and E is the mean oerator. In order ent inut areas thalamus frontal areas athways), to comute V 9 [s t] the outut vk of the exerts is imlements the organism s action selection olicy. weighted and summed, Each exert is a two-layer feed-forward neural network that gets the goal and the visual contrasts as V 9 [s t] 5O k[vkg k]. inut, and has 8 sigmoidal outut units that locally encode the actions. The exerts may corresond to The weight gk is comuted as the softmax activa- neural assemblies of the thalamus or frontal areas: tion function of the outut units ok of the gating here the details of the model are quite arbitrary. To network,

6 10 G. Baldassarre / Cognitive Systems Research 3 (2002) 5 13 gk5 ex[o k]/o f[ex[o f]]. where ck is a measure of the correctness of the exert k defined as This art of the model is arbitrary, but the 2 ck5 ex[20.5e kt]. modularity of the striosomes confers some lausibility: the model is an imlementation of what is The gating network weights zki are udated to suggested by Houk, Adams & Barto (1995) accordincrease the contribution in yielding V 9 [s t] of the ing to whom different striosomes may be secialized exerts who had low errors, in dealing with different behavioral tasks. As we shall see, this is an emergent feature of the model Dzki 5 j(hk 2 g k)y i, resented here. The last comonent of the critic (subthalamic loo and substantia nigra ars comtions. This algorithm leads the exerts to secialize where j is a learning rate set to 0.1 in the simulaacta) is a neural imlementation of the comutation of the temoral-difference error e defined as in the different regions of the inut-goal sace. (Houk, Adams & Barto, 1995) Notice that j is higher than h. This has been found to be a necessary condition for the controller to e 5 (r 1 gv 9 [s ]) 2V 9 [s ]. work. With j 5h the exerts did not secialt t11 t11 t ize and interference between different goals revented learning. Each critic s exert has a secific error defined as e 5 (r 1 gv 9 [s ]) 2 v [s ]. kt t11 t11 k t The actor is trained according to the doaminergic signal e t. In this case this signal is interreted as the actor s caacity to select actions that lead the organism to new states with an evaluation higher These error signals corresond to the doathan the average evaluation exerienced reviously minergic signals and are at the base of the learning dearting from that same state. The udating of the rocesses of the actor and critic. action merits of the selected exert (and only this) is Each critic s exert is trained on the basis of the done by udating the weights of the neural unit exert s doaminergic error signal that assumes the corresonding to the selected action a w (and only role of error in the estimation of V 9 [s t] in a this) as follows: suervised learning algorithm. The weights of the exerts are udated so that their estimation v [s ] Dwwi 5 ze t(4m w(1 2 m w))y i, k t tends to be closer to the target value (rt11 1 where z is a learning rate (0.01) and (4m w(1 2 m w)) gv 9 [s t11]). This target is a more recise evaluation is the derivative of the sigmoid function multilied of st because it is exressed at time t 1 1 on the basis by 4 to homogenize the size of the learning rates of of the observed rt11 and the new estimation the actor and the linear critic. The model s doa- V 9 [s t11]. The formula (a modified Widrow Hoff minergic signal affecting the sensory-motor athrule, cf. (Widrow & Hoff, 1960)) to udate the ways may corresond to the brain doaminergic weights of each exert is signal targeting the frontal areas downstream the Dw 5he y h, thalamus. For simlicity in the model these doaki kt i k mine-sensitive areas have been designed ustream where w is a weight of the exert k, h is a learning the thalamus. The weights of the winning gating ki rate (set to 0.01 in the simulations) and y is the network s unit are udated in the same way used for i activation of the goal and contrast units. h (absent in the exerts merits (learning rate 0.01). k the Widrow Hoff rule) is the (udated) contribution The learning mechanism of the critic and the actor of the exert k to the global answer V 9 [s ], and is differ because in the case of the actor it is not t defined as ossible to have a teaching attern to imlement a suervised learning algorithm. The stochastic nature h 5 gcyo [g c], of the actor is necessary in order to roduce new k k k f f f behaviors that are then strengthened or weakened

7 G. Baldassarre / Cognitive Systems Research 3 (2002) according to their outcome in terms of rewards. At seeds have roduced results with analogous quality. the beginning of the simulations the weights of the Concerning the critic, we see that each goal is dealt critic and actor (only those affected by the doamine) with by a different exert (in each ossible osition are randomized in the interval [20.001, ]. of the arena the contribution of this exert in This imlies that the evaluations exressed by the determining the evaluation is over The first linear critic are around 0, and the merits (rob- column of Fig. 5 shows the resulting gradient field of abilities) exressed by the sigmoidal actor (stochas- the evaluations for the three goals). This robably tic selector) are around 0.5 (0.125). This imlies that means that the ositions in the arena need to receive initially, the organism s behavior is a random walk. a different evaluation for the three different goals, so Then the critic and the actor are trained simul- that using the same weights (same exert) would taneously (olicy iteration): the evaluator learns to only cause negative interference. This also means evaluate the states of the world on the basis of the that the connections from the (contrast) inut attern actor s action-selection olicy, and the actor im- to the critic s gating network are redundant. The fact roves the olicy by increasing the robabilities of that different arts of the striosomes secialize for those actions that yield an evaluation higher than the different goals, as in the model, is an interesting exected one (cf. Sutton & Barto, 1998). hyothesis that has not yet been verified emirically. Notice that the controller is caable of not using some of the resources available (exert 1, 3, 4). 4. Simulations, results, interretations These resources could be used for other goals. With regard to the actor, Fig. 5 shows that the As mentioned, the task of the organism is to reach secialization of the exerts is much less roone of the three goal ositions shown in Fig. 2. When nounced. In articular the grahs of the second and a goal is reached a new one (randomly chosen third column of the Fig. 5 show that, while ursuing between the three goals) is assigned to the organism a goal, the actor uses different exerts in different which then has to reach it from its current osition. osition in the arena. The histograms show the Fig. 4 shows the organism s learning curve in terms frequency of use of the different exerts for the of number of stes taken to reach a goal (mobile different goals. Clearly the controller tends to use average for 100 successes, average for 10 random different exerts when dealing with different goals, seeds). The erformance imroves from about 1000 but now (differently from what is observed in the to about 30 stes. critic) the visual inut lays an imortant role. An Fig. 5 resents some data about how the neural- interesting fact coming out from the second and third network controller of one of the 10 simulations has column of Fig. 5 is that the same exerts are being self-organized during learning. The other random used for different goals (e.g. exert 1 for goal 1 and 3). Further investigation should show if this different use of exerts in the critic and in the actor are due to the differences in the role they lay or if it is due to the difference between the algorithms emloyed (suervised learning and stochastic unsuervised learning; cf. (Calabretta, Nolfi, Parisi & Wagner, 1998), on the evolutionary emergence of modular networks function through genetic algorithms). Notice that in the actor, as in the case of the critic, there is a artial use of the resources available (marginal role of exert 3, 4 and 5). The exloration of some arameters and simulation conditions has shown some limits of the control- Fig. 4. The learning curve of the organism. Y-axis: cycles er goal reached (mobile average for 100 successes, average for 10 random ler. Too high learning rates (esecially for the critic) seeds). X-axis: cumulated cycles. roduce instability, while too low rates roduce slow

8 12 G. Baldassarre / Cognitive Systems Research 3 (2002) 5 13 Fig. 5. Data about the self-organization of the controller during learning (1 out of 10 random seeds). The three rows of grahs are relative to the three different goals. The first column of grahs shows the gradient field of evaluations V 9 [s t] roduced by the critic in 400 different ositions (corresonding to the cells of the grid). The area of the white (ositive evaluations) and black (negative evaluations) cells is roortional to the evaluation roduced. The atches where no evaluation is resent corresond to the obstacles. The number in each grah indicates the exert that is used by the critic for the articular goal (only 1 exert er goal). The second column of grahs shows the order number of the actor s exert with highest robability of being selected (for the same 400 ositions of the revious column). The last column of grahs shows the histograms that summarize the frequencies of the exerts illustrated in the revious column. learning. The system is also quite sensitive to the aliasing roblem (this is the roblem that occurs when there are states of the world that aear to be the same or very similar, cf. (Whitehead & Ballard, 1991)). In articular if there are ositions that are similar to the goal ositions, the organism tends to waste time searching around them. This haens because they will tend to have a high evaluation. With more goals some roblems also occur: with some random seeds the same critic s exert is used for more than one goal. This roduces a gradient field with more than one eak. This causes the organism to ursue the ositions corresonding to these eaks at the same time so that the behavior results to be dithering. 5. Conclusion This work has resented a comutational model that attemts to summarize in a coherent icture some of the most relevant roerties of basal ganglia

9 G. Baldassarre / Cognitive Systems Research 3 (2002) regarding motor behavior. An attemt has been made Honolulu: International Society for Adative Behaviour,. to design a model that on one side is caable of Barto, A. G. (1995). Adative critics and the basal ganglia. In controlling an organism in a non-trivial behavioral Houk, C. J., Davis, L. J., & Beiser, G. D. (Eds.), Models of task, and on the other side is based on architectures information rocessing in the basal ganglia. Cambridge, MA: and mechanisms ossibly grounded on emirical The MIT Press. evidence about the anatomy and hysiology of basal Calabretta, R., Nolfi, S., Parisi, D., & Wagner, G. P. (1998). ganglia. The model has shown that the role of the Emergence of functional modularity in robots. In Pfeiffer, R. (Ed.), From animals to animats 5: Proceedings of the 5th striosomes in the striatum might be that of roducing International Conference on the Simulation of Adative Bean evaluation of the exected future rewards, and to haviour. Cambridge, MA: The MIT Press, build a doaminergic signal corresonding to revi- Chevalier, G., & Deniau, M. (1990). Disinhibition as a basic ously neutral inut atterns on the basis of some rocess in the exression of striatal functions. Trends in rimary reinforcers. The doaminergic signal is used Neurosciences, 13, Frank, M. J., Loughry, B., & O Reilly, R. C. (2000). Interactions to learn to exress the evaluations themselves on the between frontal cortex and basal ganglia in working memory: a basis of a suervised learning algorithm. The simula- comutational model. ICS Technical Reort Deartment tions have shown that the modularity of the strio- of Psychology, University of Colorado at Boulder. somes is used to deal with different behavioral tasks Goldman-Rakic, P. S. (1995). Toward a circuit model of working the organism meets during its life. The model has memory and the guidance of voluntary motor action. In Houk, C. J., Davis, L. J., & Beiser, G. D. (Eds.), Models of inalso shown that the role of the matrix in the striatum formation rocessing in the basal ganglia. Cambridge, MA: might be that of learning to generate stochastic The MIT Press. variants of behavior, eventually consolidated on the Graybiel, A. M. (1998). The basal ganglia chunking of action basis of the doaminergic signal. Here the role of the reertoires. Neurobiology of Learning and Memory, 70, 119 basal ganglia s double-inhibition mechanism is not 136. Hanes, D. P., & Schall, J. D. (1996). Neural control of voluntary that of directly triggering articular atterns of movement initiation. Science, 274, behavior, but that of releasing from inhibition sen- Houk, C. J. (1995). Information rocessing in modular circuits sory-motor athways that then yield a articular linking basal ganglia and cerebral cortex. In Houk, C. J., Davis, behavior suitably related to the current goals and L. J., & Beiser, G. D. (Eds.), Models of information rocessing recets. in the basal ganglia. Cambridge, MA: The MIT Press. Houk, C. J., Adams, L. J., & Barto, G. A. (1995). A model of how the basal ganglia generate and use neural signals that redict reinforcement. In Houk, C. J., Davis, L. J., & Beiser, G. D. Acknowledgements (Eds.), Models of information rocessing in the basal ganglia. Cambridge, MA: The MIT Press. The Deartment of Comuter Science, University Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adative mixtures of local exerts. Neural Comutaof Essex, funded the author s research. Secial tion, 3, thanks are exressed to Prof. Jim Doran (University Kosslyn, S. M. (1999). Image and brain. Cambridge, MA: The of Essex) and Prof. Domenico Parisi (Italian Nation- MIT Press. al Research Council) for their valuable contribution Lieberman, A. D. (1993). Learning behavior and cognition. of ideas, and James Adam for his recious hel in Pacific Grove, CA: Brooks/ Cole Publishing. Meyer, J. -A., & Guillot, A. (1990). Simulation of adative the rearation of the article. behavior in animats: review and rosect. In Proceedings of the First International Conference on Simulation of Adative Behavior. Cambridge, MA: The MIT Press, References Rolls, E. (1999). Brain and emotion. Oxford: Oxford University Press. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: an Baldassarre, G., & Parisi, D. (2000). Classical and instrumental introduction. Cambridge, MA: The MIT Press. conditioning: From laboratory henomena to integrated mecha- Whitehead, S. D., & Ballard, D. H. (1991). Learning to erceive nisms for adatation. In Meyer, J. -A., Berthoz, A., Floreano, and act by trial and error. Machine Learning, 7, D., Roitblat, H., & Wilson, S. W. (Eds.), From animals to Widrow, B., & Hoff, M. E. (1960). Adative switching circuits. animats 6: Proceedings of the 6th International Conference on IRE WESCON Convention Record, Part IV, the Simulation of Adative Behaviour, Sulement volume.