Combining associative learning and nonassociative learning to achieve robust reactive navigation in mobile robots

Transcription

1 Combining associative learning and nonassociative learning to achieve robust reactive navigation in mobile robots Carolina Chang Grupo de Inteligencia Artificial, Departamento de Computación y Tecnología de la Información Universidad Simón Bolívar, Apartado Postal Caracas 1080, Venezuela cchang Abstract We present a neural network that learns to control approach and avoidance behaviors in a mobile robot based on three forms of animal learning: classical conditioning, operant conditioning, and habituation. Mechanisms of classical conditioning are used to learn to predict the proximity of obstacles and sources of light. At the same time the robot learns through operant conditioning to generate the desired avoidance and approach behaviors. Learning takes place as the robot moves around an environment cluttered with obstacles and sources of light. The neural network requires no knowledge of the geometry of the robot or the configuration of the sensors. After learning the robot can choose among different behaviors depending on the moment-by-moment combination of sensory information and internal needs. We discuss the problem of the oscillatory movements observed when the robot navigates through narrow hallways. Results show that habituation to the proximity of the walls can lead to smoother navigation. Habituation to sensory stimulation to the sides of the robot does not interfere with the robot s ability to turn at dead ends and to avoid obstacles outside the hallway. This work shows that several biological mechanisms of learning can be combined to produce adaptive behaviors in real mobile robots. Keywords: Mobile Robots, Classical Conditioning, Operant Conditioning, Habituation, Unsupervised Neural Networks. 1 Introduction According to Arkin (1998), intelligence endows a system (biological or otherwise) to improve its likelihood of surviving within the real world. Learning, sometimes regarded as the acquisition of knowledge about the world (Tolman, 1932), is therefore an important component of intelligence. To survive in a complex, dynamically changing environment, an animal must learn to recognize informative cues and to predict the consequences of its own actions. Classical conditioning and operant conditioning are two forms of associative learning that enable animals to acquire such knowledge about the world. In the classical or Pavlovian conditioning paradigm, learning occurs by repeated association of a neutral conditioned stimulus (CS) with an unconditioned stimulus (UCS), which has significance for an animal and always gives rise to an unconditioned response (UCR). For example, a rat that is repeatedly shocked (UCS) shortly after a red light is turned on (CS) will associate the red light with fear (UCR), meaning that eventually presentation of the red light alone elicits a conditioned response (CR) resembling the fear response elicited by the shock itself. Hence, classical conditioning enables animals to recognize informative stimuli in the environment. In the case of operant conditioning, an animal learns the consequences of its own actions. More specifically, the animal learns to exhibit more frequently a behavior that has led to reward in the past, and to exhibit less frequently a behavior that has led to punishment. For example, a pigeon can be trained to peck at an illuminated key in order to receive a small food reward, while a rat learns to avoid the ingestion of food that makes it ill. Over the past years we have applied biological learning theories to achieve adaptive behaviors in mobile robots (Chang & Gaudiano, 1998). The behaviors are learned by an unsupervised neural network, which imitates principles of classical conditioning and operant conditioning. We have used the neural network to train a Khepera miniature mobile robot (K-Team S.A., Préverenges, Switzerland) to avoid obstacles and approach lights. The Khepera is a 55cm diameter differential drive robot. It has eight infrared proximity sensors, six of which cover the frontal 180 o, and

2 the remaining two sensors cover the back of the robot. In our implementation we have ignored the two rear-facing infrareds, using only the six frontal sensors. The robot learns simultaneously to generate more often those movements that lead to increased light levels, while suppressing those movements that lead to collisions. The neural network requires no knowledge about the robot s geometry and the location of the sensors on the robot s body. Moreover, no prewired reflex behaviors are used. Instead, the robot learns to sort out which sensors are associated with obstacle or light sensing (Chang & Gaudiano, 1998; Gaudiano & Chang, 1997). Our model for reactive navigation has proven to be robust and fast. However, oscillations in the robot s movements are observed when the robot navigates through narrow corridors. This is primarily due to the short range of the Khepera s infrared sensors, which can detect obstacles up to a distance of only about 2cm. The neural network learns to generate abrupt turns (avoidance behaviors) when it detects an obstacle. The continuous detection of the walls in a narrow hallway elicits the obstacle avoidance behaviors very often, yielding oscillatory movements. To address the problem of oscillations in hallway navigation we have included in our neural network a mechanism for habituation to prolonged sensory input. In addition to learning through classical conditioning and operant conditioning, the neural network described here includes a mechanism of nonassociative learning that is similar to habituation. Habituation is a decrease in the strength of a response, after a repeated presentation of the stimulus that produces the response (Mazur, 1994). For example, a person may fail to show a startle reaction after repeated presentation of a loud noise. Arguably, habituation is the simplest form of learning. From the protozoan Stentor coeruleus to Homo sapiens, many species of the animal kingdom have in common this type of nonassociative learning. For this reason, it has been hypothesized that the same physiological mechanisms of habituation may be shared by different species. Some researchers have investigated the mechanisms of habituation of simple animals. For example, the study of the Aplysia s gill-withdrawal reflex has led to the discovery of the physiological changes that are responsible for its habituation (Castellucci, Pinsker, Kupfermann, & Kandel, 1970; Kandel & Schwartz, 1982). When the animal s siphon is touched, its gill contracts for a few seconds. If the siphon is stimulated repeatedly, the gill-withdrawal reflex habituates (smaller contractions and faster return to a normal position). The study of the Aplysia s neural circuit revealed that the presynaptic (sensory) neurons released less transmitter into the synapses with repeated stimulus presentation. On the other hand, there was no change in the sensitivity to the transmitter of the postsynaptic neuron. Therefore, habituation in Aplysia results from changes in the effectiveness of existing synapses rather from anatomical changes such as the growth of new connections between neurons. It has been more difficult to identify the physiological changes involved in habituation in mammals due to the complexity of their nervous system. However, as in Aplysia, there is evidence of changes in existing synapses of the nervous system. Moreover, these changes take place in the sensory areas of the neural circuit (Davis, Gendelman, Tischler, & Gendelman, 1982; Condon & Weinberger, 1991). This article describes how classical conditioning, operant conditioning, and habituation are integrated to achieve robust reactive navigation in a mobile robot. Section 2 describes the neural network model of classical conditioning which is the core of this work. The combination of mechanisms of classical conditioning and operant conditioning in section 3 allows the robot to learn to generate avoidance and approach behaviors. Section 4 describes how habituation was integrated to the model to reduce oscillatory robot movements observed in narrow hallways. Finally, section 5 presents the conclusions of this work. 2 Classical Conditioning Our neural network model (Gaudiano & Chang, 1997; Chang & Gaudiano, 1998) is based on a detailed theory of learning proposed by Grossberg, which was designed to account for a variety of behavioral data on learning in vertebrates (Grossberg, 1971; Grossberg & Levine, 1987; Grossberg & Schmajuk, 1987). Figure 1 is a schematic of the overall structure of the neural network for classical conditioning. In the figure, populations of neurons are represented by boxes, while the interconnections between populations are represented by lines. The sensory nodes (CSs) at the upper left of figure 1 receive activation from the robot s range sensors. There is no knowledge built into the network about the kind of sensor information, or the position of the sensor on the robot s body. The first design consideration of the model is that those stimuli that are initially not significant to the organism (i.e., CSs) are unable to generate emotional or behavioral responses, whereas a few stimuli that are innately significant to the organism (i.e., UCSs) always lead to an emotional and behavioral response (UCR). This is represented in figure 1 by the modifiable connections (indicated by semi-circles) between the CS population and the Reward/Punishment population, and by modifiable connections between the Gated CS population and the behavior generation population

3 Conditioned Stimuli (CS) Gated Conditioned Stimuli Reward/ Punishment Unconditioned Stimuli (UCS) Behavior Generation (UCR/CR) Figure 1: Schematic of the neural network model. CR/UCR. In contrast, the UCSs operate through fixed, strong connections to these populations, which are represented by thick arrows in the figure. The Gated CS nodes require joint activation of the sensory (i.e., CS) and emotional (i.e., Reward/Punishment) input in order to be activated. Prior to learning, as long as the connections from the CS to the Reward/Punishment population are weak, the Gated CS nodes cannot be activated by a CS alone, and behaviors cannot be generated by the CS population. Through repeated pairing with a UCS, a CS can acquire the ability to generate emotional and behavioral responses that resemble those of the UCS with which it is paired. The nodes of the Reward/Punishment population, which Grossberg refers to as drive nodes, modulate learning and carry the emotional valence of the UCS. In our implementation, a collision is a UCS that elicits fear in the robot, while an increase in the ambient light level is a UCS that elicits pleasure. Different UCSs can generate different behaviors in the robot: collisions generate avoidance behaviors, while the sight of light generates approach behaviors. The Reward/Punishment (or drive) nodes restrict learning to stimuli that are paired with emotionally significant events. This is an important departure from traditional connectionist approaches where every input-output pair presented to the network is learned. Drives compete in a sensory-drive heterarchy (Grossberg, 1971). That is to say, the combination of sensory activation and drive activation determines which motor response will take place. At each time only one motor response is released. Initially the CSs have no special meaning; it is only through learning that the network starts to discover the causal structure of the environment. When a collision occurs or light is sighted, all sensory nodes are allowed to learn. However, after repeated associations, only sensory nodes systematically active during learning will have their connections strengthened. In our example, activation of the fear drive tends to co-occur with activation of proximity detector nodes, so that ultimately the activation of proximity detectors generates an avoidance behavior. By the same token, activation of the pleasure drive tends to co-occur with activation of ambient light detectors, which generates an approach behavior. The two forms of learning take place simultaneously while the robot is moving through the environment. The dynamics of the network determine which nodes learn in which conditions. After learning, the neural network is capable of exhibiting multiple behaviors depending on the events that take place in the environment. 3 Operant Conditioning The neural network described in the previous section allows the robot to predict the arrival of a collision or the sight of light based on the activation of the robot s sensors. Once knowing that a collision will occur soon, it is desirable that the robot would change its trajectory immediately in order to avoid the collision. Similarly, the robot should change its trajectory in order to reach for a predicted source of light. Our neural network makes no use of prewired reflex behaviors. Moreover, there is no built-in knowledge about the robot s geometry and the location of the sensors on the robot s body. For this reason, the robot has to learn what movements it should perform when a collision or a source of light is predicted. The robot learns to produce the desired movements through operant conditioning. The robot is trained by allowing it to make random movements in an environment cluttered with obstacles and light sources. Whenever the robot collides with an obstacle during one of these movements the network learns not only to associate sensor activation with an impending collision (classical

4 (a) (b) Figure 2: Multiple behaviors exhibited by the Khepera robot. (a) The robot follows a flashlight using approach behaviors. (b) The robot avoids collisions using the obstacle avoidance behaviors. conditioning) but also to inhibit the movement that caused the collision (operant conditioning). Likewise, approach behaviors are learned whenever the robot s movements lead to a significant increase in detected light level. As long as lights and obstacles do not overlap, the avoidance and approach behaviors can be learned simultaneously. In general, the neural network nodes representing range sensors on the right side of the robot will be most active just prior to a collision when the robot is moving to the right, and likewise for other nodes. It is for this reason that there is no need to encode any information about the sensor geometry in the network: each node learns to suppress those movements that caused collisions with objects that the node saw coming. In practice there are occasions in which the robot might collide, for example, on its left side even though it is turning to the right. However, on average this is a rare event: the gradual nature of learning and the use of a small decay rate jointly ensure that each node learns a pattern reflecting the statistical distribution of collisions foreseen by that node, which in turn reflects the position of the sensor on the robot s body. Learning of the avoidance and approach behaviors in the neural network develop at the same time. After training the robot is able to exhibit the approach and avoidance behaviors depending on the moment-by-moment activation of its sensors. Several interesting experiments that tested the capabilities of the network in a variety of situations and complex environments were described in Chang and Gaudiano (1998). For example, figure 2 shows the Khepera approaching light while avoiding obstacles. The figure is a digital image captured from a camera mounted above the Khepera s environment, which is made out of LEGO bricks. A tracking algorithm localizes the robot s position and direction (square) and traces the trajectory described by the robot (black dots). In panel (a), navigation was guided by the approach behavior, as the robot followed a flashlight directed by the experimenter. The experimenter brought the robot close to a wall, and at that moment removed the flashlight. In panel (b), the robot avoided the nearby wall and kept wandering the environment, avoiding further obstacles that it found in its path. A more interesting result is obtained when pleasure and fear compete in the sensory-drive heterarchy. The effect of multiple drive activation is shown in figure 3. A light beam was directed through a 2.5cm wide window in a wall. The width of the window ensured that the light would be easily detected, while the window configuration allowed the robot to detect an obstacle in the same place where it detected the source of light. As the activation of the light sensors predicted the existence of a source of light, the robot turned towards the window and moved forward. When the robot approached the wall, its sensors started to signal the proximity of an obstacle. Some zig-zag in the robot s movement occurred due to the competition of the pleasure and fear drives, as the robot tried to approach and avoid the wall at the same time. However, when the robot got very close to the wall the proximity sensors became very active, which allowed fear to ultimately win the competition in the sensory-drive heterarchy. The robot then turned to avoid the wall and continued its navigation. In brief, competition between pleasure and fear allowed the robot to avoid obstacles even when approaching a source of light. Additional experiments and details on this neural network can be found elsewhere (Chang & Gaudiano, 1998).

5 Figure 3: Pleasure and fear competition. A flashlight is placed behind a wall, directed towards a 2.5cm wide window. The robot simultaneously tries to approach the light and avoid the wall. Fear finally wins the competition and the robot avoids the collision with the wall. 4 Habituation Our reactive method for obstacle avoidance does not make use of any information about the location of the sensors on the robot s body, as discussed above. As a consequence, the robot never navigates parallel to walls because any activation of the side sensors predicts a collision, causing the robot to turn away from the walls. While this is not a problem when the robot navigates open environments, oscillations are observed when navigating through narrow corridors, as shown in figure 4(a). Oscillations in hallway navigation are undesired behaviors produced by our neural network model. Notice however that other approaches to reactive obstacle avoidance have similar problems when negotiating narrow hallways. For example, the built-in Braitenberg s Vehicle of the Khepera robot produces erratic movements very often (Braitenberg, 1984; K-Team, 1995). Moreover, the navigation of hallways is not collision-free, and the robot can get stuck at deadends. Other solutions such as the Distributed Adaptive Control (DAC) (Pfeifer & Verschure, 1992; Verschure, Kröse, & Pfeifer, 1992) cannot even let the robot pass through corridors because the obstacle avoidance behaviors cause it to back-off in the presence of strong sensory stimulation. The oscillatory trajectories in narrow corridors are caused by the strong and prolonged sensory input experienced by the robot. To solve this problem we added to the network a mechanism of sensory habituation. We wanted the robot to get used to the proximity of the corridor walls, which in turns would reduce the generation of avoidance behaviors. To model habituation we implemented transmitter accumulation-depletion mechanisms that have been proposed by Grossberg as part of his psychophysiological theory of learning (Grossberg, 1972, 1982). The transmitter gates imitate the role of chemical transmitter depletion in the neurons synapses. The amount of transmitter g determines the efficacy of the synaptic transmission. Each gate multiplies the input signal x, to form a gated output signal T : T = xg (1) The output signal is a joint function of the input and the availability of transmitter. Hence, the output is large when a strong input is presented and the level of chemical transmitter is high. However, the transmitter level g changes over time depending on the strength of the sensory input x. Through a process of habituation, the transmitter level decreases as the input increases, according to: dg dt = (? g)? T xg (2) where, and T are positive constants that determine the habituation rate of the transmitter. The term (? g) indicates that the transmitter g recovers to its maximum value at the rate. The term T xg indicates that the transmitter level decreases at a rate proportional to the amount of transmitter available. Equation 2 ensures that an

6 (a) (b) Figure 4: (a): Hallway navigation using transmitter gates. Sensory adaptation eliminates the robot s oscillations. (b): The obstacle avoidance behavior causes the robot to oscillate in narrow hallways. Number of turns no habituation 15.0 = 0: = 0: = 0: = 0: Table 1: Average number of turns for different transmitter replenishment rates. As decreases the robot s oscillations are reduced. The decrease in the number of turns is related to the reduction of the turning angle produced by the robot. For example, 2 turns are observed in figure 4(b). However, the turning angles are so small that navigation is smooth and free of oscillations. increase in the input signal depletes more transmitter, while the transmitter returns to its maximum level in the absence of an input signal. In our model, the activity of the sensory nodes is propagated to the rest of the network through transmitter gates. As the robot s sensors are activated by the nearby walls, habituation decreases the gated output of the sensory neurons, thereby preventing oscillations, as shown in figure 4(b). The values = 0:005, = 1:0, and T = 1:0 were used in the experiment of figure 4(b). Different levels of habituation can be obtained depending on the parameter values of the transmitter gates. In particular, we can study the effect of, the replenishment rate of the transmitter. When the transmitter recovery is fast, the robot does not habituate to the proximity of the walls. On the other hand, if the replenishment is slow, the robot habituates, reducing the number of turns it makes to avoid the walls. Table 1 shows the average number of obstacle avoidance behaviors observed in the robot for = 1:0, and T = 1:0 and different values of. The robot navigates 10 times a 55x9.5 cm hallway. The number of turns produced in each trial are counted, with the exception of the required turns at the two ends of the hallway. For example, 15 turns are observed in figure 4(a). Oscillations are also affected by the width of the hallway. When the walls are too far apart, they are not detected constantly by the robot. Therefore, fewer turns are observed as the hallway gets wider. Habituation does not eliminate such turns because there is more time for the transmitter to recover in wide areas. As the hallway gets narrower, there is more improvement of navigation due to habituation. Table 2 shows the average number of turns in 10 trials with and without habituation for different hallway widths. The parameter values of the transmitter gates were = 0:005, = 1:0 for all the trials. = 1:0, and T Oscillations become a real problem for the robot in very narrow corridors. It is in such cases that habituation is most valuable. As shown in table 2, there is a reduction of 96.76% in the number of turns in a 7.5 cm wide corridor. Notice that this is a very narrow corridor since the robot s diameter is 5.5 cm. Not only can the robot pass through, but also the resulting navigation is smooth along the entire corridor.

7 hallway turns turns percentage width (cm) no habitation habituation reduction Table 2: Reduction of the number of obstacle avoidance behaviors produced by the robot in hallways of different widths. More oscillations are produced in narrow corridors. In such environments habituation improves navigation by reducing the number of turns of the robot. Oscillations are not observed in wider environments but habituation still reduces the number of turns. 5 Conclusions We have described a neural network that learns to generate avoidance and approach behaviors for a wheeled mobile robot by using mechanisms of classical conditioning and operant conditioning. Training for the avoidance and approach behaviors can be done simultaneously, even though these behaviors are quite opposite. One of the main properties of the model is that it requires no knowledge about the robot s geometry and the configuration of the sensors on the robots body. Moreover, learning generalizes to any environment since it is based on the robot s egocentric frame of reference. Though versatile, this purely reactive solution must be combined with higher-level navigation schemes in order to address more complex navigation tasks. Reactive hallway navigation based only on infrared or ultrasonic sensors is a problem for many controllers. For example, the DAC architecture in its original version does not even allow a robot to enter a hallway. The excessive activation of the range sensors causes the robot to move backwards. This problem has been addressed by Salomon (1998), who incorporated proprioceptive sensors to differentiate strong sensory input from actual collisions. Training with the improved architecture allowed the robot to pass through narrow places. It is not clear, however, whether oscillations in hallways are observed with this new architecture. Our approach consisted in letting the robot habituate to the strong and continuous sensory input obtained in narrow places. To this end, we adopted the transmitter gates model proposed by Grossberg (1972, 1982). The mechanism of habituation reduces the transmitter released into the synapses when a stimulus is presented repeatedly. We have included transmitter gates only in the sensory neurons of the neurocontroller, as suggested by the available studies of habituation in biological organisms. Experimental results confirm that short-term habituation can be achieved with no need for anatomical changes, such as the growth of new connections between neurons. Since the mechanism of habituation only introduces changes in the effectiveness of existing synapses we were able to incorporate habituation with our neural network a posteriori. No additional training was required for the network to perform correctly. This suggests that habituation can be easily integrated with other architectures, not only to reduce oscillations in hallways but also to produce other behavioral changes due to repeated stimulation. The neural network for reactive robot navigation described in this article is based on a computational neural theory of animal learning. The work shows that different mechanisms of biological learning can be combined successfully within a single artificial creature. Each modality of learning makes a unique contribution to the achievement of truly adaptive behaviors in the mobile robot. As in biological systems, learning allows the robot to survive in complex, nonstationary environments. Acknowledgments Initial parts of this work were supported by the Office of Naval Research and the Navy Research Laboratory, through grant ONR The realization of this work was possible thanks to the generosity and support of Dr. Paolo Gaudiano, Director of the Boston University Neurobotics Lab. References Arkin, R. C. (1998). Behavior-based robotics. Intelligent robotics and autonomous agents series. MIT Press. Braitenberg, V. (1984). Vehicles: Experiments in Synthetic Psychology. MIT Press.

8 Castellucci, V., Pinsker, H., Kupfermann, I., & Kandel, E. R. (1970). Neuronal mechanisms of habituation and dishabituation of the gill-withdrawal reflex in Aplysia. Science, 167, Chang, C., & Gaudiano, P. (1998). Application of biological learning theories to mobile robot avoidance and approach behaviors. Journal of Complex Systems, 1(1), Condon, C. D., & Weinberger, N. M. (1991). Habituation produces frequency-specific plasticity of receptive fields in the auditory cortex. Behavioral Neuroscience, 105, Davis, M., Gendelman, D. S., Tischler, M. D., & Gendelman, P. (1982). A primary acoustic startle circut: Lesions and stimulation studies. Journal of Neuroscience, 2, Gaudiano, P., & Chang, C. (1997). Adaptive obstacle avoidance with a neural network for operant conditioning: experiments with real robots. In Proceedings of the 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA), pp Monterey, California. Grossberg, S. (1971). On the dynamics of operant conditioning. Journal of Theoretical Biology, 33, Grossberg, S. (1972). A neural theory of punishment and avoidance, II: Quantitative theory. Mathematical Biosciences, 15, Grossberg, S. (1982). A psychophysiological theory of reinforcement, drive, motivation and attention. Journal of Theoretical Neurobiology, 1, Grossberg, S., & Levine, D. (1987). Neural dynamics of attentionally modulated Pavlovian conditioning: blocking, interstimulus interval, and secondary reinforcement. Applied Optics, 26, Grossberg, S., & Schmajuk, N. A. (1987). Neural dynamics of attentionally modulated Pavlovian conditioning: Conditioned reinforcement, inhibition and opponent processing. Psychobiology, 15(3), K-Team, S. (1995). Khepera: User Manual. Switzarland. Kandel, E. R., & Schwartz, J. H. (1982). Molecular biology of learning: Modulation of trasmitter release. Science, 218, Mazur, J. E. (1994). Learning and Behavior (Third edition). Prentice-Hall, Inc. Pfeifer, R., & Verschure, P. (1992). Distributed adaptive control: a paradigm for designing autonomous agents. In Varela, F. J., & Bourgine, P. (Eds.), Toward a practice of autonomous systems, pp MIT Press, Cambridge, Massachusetts. Salomon, R. (1998). Improving DAC. In From Animals to Animats 5. Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior, pp MIT Press. Tolman, E. (1932). Purposive behaviour in animals and men. Appleton-Century-Crofts, New York. Verschure, P. F. M. J., Kröse, B. J. A., & Pfeifer, R. (1992). Distributed adaptive control: The self-organization of structured behavior. Robotics and Autonomous Systems, 9,