A computational model for basic learning and memory formation in Aplysia. Zhong Hui

Transcription

1 Thesis for the degree of Master of Science in Complex Adaptive Systems A computational model for basic learning and memory formation in Aplysia Zhong Hui Department of Applied Physics Chalmers University of Technology Göteborg, Sweden, 2007

2 A computational model for basic learning and memory formation in Aplysia ZHONG HUI c ZHONG HUI Department of Applied Physics Chalmers University of Technology Göteborg, Sweden Telephone: +46 (0) Chalmers reproservice Göteborg, Sweden, 2007

3 Abstract In this thesis, a biologically inspired model, which incorporates the neural mechanisms of short- and long-term sensitization in the sea slug Aplysia, is implemented and analyzed. In the literature, intensive studies of fundamental forms of learning, like habituation and sensitization, have been made, on a cellular level, using the sea slug Aplysia as a neural model. The proposed model consists of a set of first-order differential equations, describing the synaptic plasticity for learning-related information storage based on a simplified Aplysia gill withdrawal neural circuit. The underlying mechanisms of basic learning and memory formation, responsible for short- and long-term synaptic plasticity and gene regulation, are successfully simulated by the proposed model. For short-term memory, the results demonstrate how the synaptic efficacy between the siphon signal and the motor neuron is enhanced by the tail input and decreases to its normal value after a period of time. On the other hand, experiments with Aplysia, reported in the literature, have shown that long-term memory requires communication between genes and synapses in order to increase the synaptic efficacy persistently. The proposed model introduces the concept of a junction that can be used to modify existing synaptic efficacy, and involves the framework of gene regulation, which leads to permanent synaptic change. The experimental results, displaying how the motor output responds to the various series of simulated siphon and tail sensory inputs, are presented and analyzed combining the effect of the different sets of parameters. The model quantitatively reproduced the neural processes of short- and long-term sensitization in a simple constructed neural network. In addition, a literature study on learning and memory, both of which allow animals to modify subsequent behavior in response to experiences and environments perceived, is presented, along with a discussion of applications of basic learning in robotics.

4 Acknowledgment I have had a challenging and enjoyable study time in the international master program of Complex Adaptive Systems at Chalmers University of Technology. I am grateful to the interactive environment and all kind help from teachers and students. I would like to give my special thanks to my thesis advisor Mattias Wahde for his excellent guidance, all means of assistance and support, and amazingly careful and accurate work attitude. Without his encouragement, it would not have been possible for me to surpass myself in this thesis. Deepest appreciations go to my parents for their supporting in this meaningful journey. Also I would like to express warmest gratitude to my boyfriend and my roommate. I must say that it is so different with your company and love.

5

6 Contents 1 Introduction and motivation 1 2 Learning and memory in biological systems Introduction Aplysia Memory formation in Aplysia Short-term memory Long-term memory Memory formation in other organisms Related work regarding models of basic learning Computational models of habituation Stanley s model and the Wang-Arbib model Novelty detection using habituation Hallway navigation in mobile robots using habituation A neural model of basic learning in Aplysia Model for memory formation in Aplysia Short-term learning in artifical organisms Long-term learning in artifical organisms Results Simulation results for STM Simulation results for LTM Simulation results for STM + LTM Discussion and conclusion 26 i

7 Chapter 1 Introduction and motivation Learning and memory are two of the most crucial components for animal behavior. The ability to modify subsequent behavior in response to experiences is readily demonstrated not only in vertebrates, but also in simple organisms such as Aplysia [10] and C. Elegans [24] (Chapter 27). Kandel and his co-workers [10] first studied the fundamental forms of learning and memory using the giant sea snail Aplysia as a cellular model. Kandel s research has led to important insights into the mechanisms of learning as well as the formation and storage of memories. One of the major results is the discovery that short-term memory requires alterations in the release and function of neurotransmitters at specific synapses, while long-term memory involves the formation of new synaptic connections, synthesis of new mrna and new protein [15]. In an artificial world, learning supports artificial organisms, like robots and autonomous agents, in their quest to make themselves more efficient and to adapt better to the environment. There are several methods relevant to learning, such as backpropagation and evolutionary algorithms, both of which can be used to train neural networks by modifying network parameters. However, supervised methods (such as backpropagation) that require continuous error signals, cannot normally be used in connection with realistic learning in autonomous agents, due to the problem of assigning credit to individual actions. Instead, in order to learn a desired behavior, trial-and-error is carried out in various environments, using e.g. reinforcement learning [28]. However, currently available methods are not very realistic from a biological point of view; they make no attempt to imitate the process of learning and memory such as it occurs in biological organisms. It would be interesting to simulate the different mechanisms of short-term and long-term memory, and to use those mechanisms in a general method for learning in artificial autonomous agents. Inspired by biological systems, there are several models that simulate the detailed electrophysiological process of synaptic plasticity, such as different types of neuron cells and synaptic connections [14] [13]. However, the focus of this thesis is on the dynamics of synaptic efficacy responsible for basic learning. As mentioned above, extensive studies of basic learning mechanisms such as habituation, sensitization, and classical conditioning have been carried out on Aplysia. The relevant properties of these kinds of biological mechanisms can be simulated using artifical neural networks, and applied to artificial organisms, with the aim of generating new and better adaptive behaviors. The objective of this project is to study the neural mechanism of short-term and long-term memory, and to simulate one of the simple learning processes, sensitization in Aplysia. The simulation part involves forming artificial neural net- 1

8 Chapter 1. Introduction and motivation works based on a set of equations formulated by Wahde [30], incorporating short-term and long-term memory. This thesis is structured as follows: Chapter 2 introduces the general neurophysiological background of learning and memory. Chapter 3 gives a brief description of some related work on modeling basic learning, followed by a discussion regarding some robotics applications. Chapter 4 describes the proposed method in detail and, in Chapter 5, some experimental results are presented. Finally, the model is evaluated and the conclusions are given in Chapter 6. 2

9 Chapter 2 Learning and memory in biological systems 2.1 Introduction In a very general sense, learning is an adaptive change in behavior as a result of experience. Memory is the process of storing and retrieving that experience (see e.g. [25], Chapter 29). It is the way that the information about the environment affects behavior. Learning and memory allow animals to respond more efficiently to different situations. It is generally considered that both learning and memory are orchestrated by the central nervous systems of the whole organism (see e.g. [9], Chapter 7, [23], Introductory Chapter). The question is how the environment affects the central nervous system. What are the physical and chemical changes resulting from learning and memory? The recent progress in biology, such as the understanding of genes, their expression and the proteins they encode, provides the possibility for answering this kind of questions at the molecular level. It is now clear that the environmental input is processed and transferred in neuronal circuitry by continuously changing connections between neurons. Such learning-related synaptic plasticity occurs throughout the entire life time of the organism. The most elementary forms of learning are habituation and sensitization, each of which can be found in invertebrates as well as vertebrates. Habituation is the decrease in behavioral response to repeated innocuous stimuli [15] [25]. Such a decrease is favorable to the organism, as it has no reason to interrupt its activities in order to respond to stimuli that are of no consequence. This is the evidence of an adaptive behavior. In contrast, sensitization is the increase in the response to an innocuous stimulus occurring after an aversive stimulus. Once the stimulus is regarded as aversive, the frightened state can remain for some time. It results in an enhancement of the defensive reflex. 2.2 Aplysia The processes of habituation and sensitization are effectively exhibited by the see slug Aplysia [10], shown in Fig This marine animal uses a gill on its back for breathing, which is covered by the mantle and the shell. On one end of the mantle, there is a siphon the purpose of which is to expel water from the animal. Aplysia has a very simple defensive behavior: When the siphon is stimulated, by, for example, a jet of water directed at it, the gill is withdrawn under 3

10 Chapter 2. Learning and memory in biological systems Figure 2.1: Two views of the sea slug Aplysia. Photos by Daniel L. Geiger (left panel) and Jason Dunsmore (right panel). Reproduced with permission. the mantle shelf for protection (Fig. 2.1). This reflex is quite similar to the quick withdrawal of a hand from a hot object. The basic processes of habituation and sensitization are quite similar throughout a wide range of animals. In Aplysia, repeated stimulations of the siphon and gill lead to a weaker withdrawal response of the latter. Similarly, in humans, a stimulus such as a ticking clock, which may initially seem quite loud, gradually appears to fade. During sensitization, Aplysia demonstrates strong response to a neutral stimulus after an electrical shock on its tail. Similarly, a person becomes more sensitive to an unexpected sound after hearing a nearby gunshot. Thus, the similarities of the essential reflex are remarkable. The information gained from Aplysia is greatly helpful when studying the processes underlying learning and memory in many animals. Aplysia has also been used to study the implicit or nondeclarative memory which is a fundamental form of memory. It describes the nonconscious recall and essential reflexive response to stimuli [1] [9]. For instance, nonassociative learning, such as habituation and sensitization, is closely tied to implicit memory. On the other hand, explicit or declarative memory which involves thinking has only been proven in vertebrates (see e.g. [25], Chapter 29). It requires declarative knowledge of the environment [22]. Based on the duration of the memory, both implicit and explicit memory in turn have two phases: Short-term memory, lasting minutes to hours, and long-term memory, lasting more than days, even for a life time. In Aplysia, the difference between short-term memory and long-term memory depends on the number of spaced training trials. Few training trials result in a fleeting memory, while long-term memory requires additional trials and a consolidation process after the training sessions (see e.g. [26], Chapter 50). In addition to habituation and sensitization, another basic learning is called classical conditioning which is the simplest form of associative learning. Pavlov s dog is one of the most well-known examples of researches on conditional reflexes. Before training, dogs do not respond to a ringing bell which is the conditioned stimulus. Training sessions involving ringing a bell preceding food, which is the unconditioned stimulus, were carried out to dogs. The training result shows that the dogs salivate once hearing the bell. The ability to associate two different stimuli and learn to respond to the conditioned stimulus is also seen in Aplysia s gill-withdrawal response. Different from the order of stimuli in the sensitization training session, a weak teach on siphon as the conditioned stimulus is followed by a tail shock as the unconditioned stimulus. After a number of such training sessions, the response to the siphon 4

11 Chapter 2. Learning and memory in biological systems Figure 2.2: Spaced repetitive shocks determine the mean duration of siphon withdrawal, which is compared with control animals without being shocked. Different series of trainings are tested after 1, 4, or 7 days, and the number of animals in each group is indicated by N. For the 4-days experiment involving four trains of four shocks per day, the result, which is distinctly larger than the others, exhibits long-term sensitization. Figure taken from Frost et al. [8]. c The National Academy of Sciences of the United States of America. Reproduced with permission. touch is enhanced without the tail shock. Aplysia learns to predict the harmful unconditioned stimulus only when the conditioned stimulus is applied. It is a particular evolutionary advantage of memory that it does not reserve everything; instead it retains only salient features [10]. Some of the information, which may not be important enough to be transferred into long-term storage, is simply forgotten. This transient memory serves to understand the meaning of events at present [25]. For instance, a telephone number, which is used briefly in the office, will be immediately forgotten. In contrast, many experiences are valuable enough to be retained for a long time, such as the understanding of the environment intimately linked with survival. Certain short-term memories are selected to become more stable. A critical question arises: which factors contribute to form long-term memory? In humans, it has been shown [22] that in order to consolidate long-term memory, it is necessary to use active review. Interestingly, some difficult skills, like skating and riding a bicycle, once acquired, are almost never forgotten even without subsequent practice. In addition, emotional association facilitates long-term memory formation [22]. If an unusual event leading to sadness, happiness, or surprise takes place, it is easier to remember not only the event itself, but also other factors occurring at the same time. 2.3 Memory formation in Aplysia Aplysia provides a good model to analyse the memory mechanism of a simple organism. Eric Kandel, who was awarded the Nobel Prize in Physiology and Medicine in 2000, pioneered 5

12 Chapter 2. Learning and memory in biological systems Facilitator neuron Tail sensory neuron Unconditioned stimulus (US) Conditioned stimulus (CS) Cell bodies Tail Interneuron Gill muscle Siphon skin Motor neuron Figure 2.3: The neural circuit mediating the simple behaviors in Aplysia. After Kandel and Schwartz [11]. the Aplysia studies with his associates as they considered it to be the most experimentally accessible organism [10]. Aplysia has around 20,000 nerve cells. Compared with humans, whose brains consist of around a thousand billion neurons, its nervous system is quite small. Moreover, many of these nerve cells are so large that they can be recognized as unique individuals and worked on separately. For these reasons, it is possible to define the neural circuit of specific behaviors [10]. Kandel s work on Aplysia has mainly revealed short-term and long-term implicit memory processes. The animal s gill withdrawal reflex involves different sensory neurons distributed over the skin transferring the signal caused by the environment to motor nerves, which innervate a tiny muscle used to retract the gill. As noted above, habituation refers to a reduction in the elicited gill-withdrawal by repeated weak touch of the siphon. First, exposing Aplysia to a set of repeated stimuli which are spaced a few seconds apart, results in a form of short-term memory as it fades after a few minutes. When several sets of this stimulation are used, the habituation state can last for several weeks i.e. a form of long-term memory [9]. Similarly, in experiments involving sensitization carried out by Kandel, the duration of the memory increases proportionally with the number of training trials. If the siphon is given a strong touch following a single electrical shock to its tail, Aplysia senses the potential threat and responds more strongly than if the tail shock had not been administered. This sensitization state only lasts for minutes, but if five spaced shocks are applied to the tail, the memory of these aversive stimuli can last for several days. Moreover, shocks given over days lead to enhanced and more enduring withdrawal of the gill, lasting for weeks (Fig. 2.2). Such a long-lasting change in the gill withdrawal reflex is an example of long-term sensitization. Various forms of learning activate different changes in the synaptic strength between the sensory and motor neurons. For instance, habituation involves decreasing the synaptic strength at the homosynaptic connections, while sensitization involves increasing the synaptic strength at the heterosynaptic connections [10], The weakening or strengthening of the synaptic connections is a result of alterations in the release of neurotransmitters from sensory neurons to their target cells during short-term and long-term memory acquisitions. Besides these functional changes, long-term memory involves 6

13 Chapter 2. Learning and memory in biological systems Sensory neuron + Inactive K channel Ca + Motor neuron + Active K channel P Glutamate receptors Active kinase Na + Adenylate cyclase Serotonin in vesicle ATP CAMP Glutamate in vesicle Na + Na + Facilitator neuron Figure 2.4: An illustration of the increased release of the neurotransmitter glutamate in the siphon-gill pathway, resulting from the modulatory function of the facilitator neuron. After Kandel and Schwartz [11]. anatomical changes, which are not required for short-term memory. The distinction is that, in the formation of long-term memory, new proteins are synthesized to establish new synaptic connections. In this process, a dialog between genes and synapses is indispensable [10]. One of the remarkable findings is that short-term and long-term memory take place at the very same synaptic connection [10] [9]. It is not easy to separate short-term memory from long-term memory unless localized and timed injections of drugs are applied [29]. Repeated learning stimuli lead to the transition from short-term to long-term memory. However, the information is handled in the same circuit. Using sensitization in Aplysia as an example, the molecular mechanics of short-term and long-term memory are discussed below Short-term memory The neural circuitry involved in mediating the simple behavioral repertoire of Aplysia has been worked out by Kandel and his colleagues [10] (Fig. 2.3). It consists of two sensory neurons that innervate the siphon and the tail skin, respectively, a motor neuron which innervates muscles under the gill, and a facilitator interneuron [15] [23]. The siphon sensory neuron connects directly to the motor neuron, while the tail sensory neuron makes a connection to the motor neuron through the facilitator interneuron [10]. The axon of this interneuron forms a synaptic connection with the presynaptic terminal of the siphon neuron [15]. Once the tail sensory neuron detects a noxious shock, the interneuron is in turn excited so that it releases the neurotransmitter serotonin at the terminal of the interneuron [10] [15] [23]. A cell-surface receptor on the siphon neuron picks up serotonin and gives rise to an enzymatic cascade. In this process, a second messenger, cyclic AMP, is produced and in turn PKA (camp-dependent protein kinase) is activated by camp, which ultimately leads to the 7

14 EPSP (mv) EPSP (mv) Chapter 2. Learning and memory in biological systems Sensory neuron Motor neuron min. Tail nerve stimulation 20 min. 50 min. 50 min. 20s Time (ms) Time (ms) Time (ms) Time (ms) Figure 2.5: A schematic illustration of short-term sensitization: Note the increased response of the motor neuron following tail stimulation. After Kandel and Schwartz [11]. depression of potassium channels [10][25][9] [15] [23]. Blocked potassium channels result in prolonged depolarization, so that the action potential is reached more slowly. As the result of this cascade, the release of the transmitter glutamate in the siphon-gill pathway is enhanced and prolonged [9] [15] (see Fig. 2.4). The synaptic excitation of the motor neuron is caused by the effects both of the siphon sensory neuron and the facilitator interneuron. Touching the siphon triggers firing action potentials of the motor neurons more easily during shortterm sensitization (Fig. 2.5). Thus, the strength of the synaptic connection in the siphon-gill pathway is enhanced by the excitatory facilitator interneuron for minutes. The short-term memory only involves a change in the efficacy of preexisting synaptic connections Long-term memory The mechanism of short-term memory noted above is the basis of long-term memory. However, if repeated shocks are applied to the tail, the increase in release of serotonin at the terminal of the interneuron leads to long-lasting sensitization. Distinct from short-term memory, changes in gene expression, synthesis of new protein, and growth of new synapses are required for longlasting synaptic plasticity: When the second-messenger (camp-pka) pathway is activated, some PKA moves into the nucleus, where a transcriptional cascade is triggered [10] [1][23]. The activated transcription factor CREB-1 (camp Response Element Binding protein-1), which is repressed by CREB-2 during short-term sensitization, initiates the long-term process [10] [1]. As its name implies, CREB-1 binds to a camp Response Element, a DNA sequence in promoters of immediate memory-enhancer genes [10]. The transcription and translation of these genes produce new protein contributing to the growth of new synaptic connections [10] [1] [23]. As a result, the number of such connections for a typical sensory neuron can be more than doubled [10]. Thus, due to this structural change, memory becomes more stable and more enduring. The transcription factor CREB works as a genetic switch which tightly controls the transcriptional cascade [10][23]. Once the repressive effect of CREB-2 is removed, CREB-1 is in turn activated and the long-term memory storage starts. 8

15 Chapter 2. Learning and memory in biological systems 2.4 Memory formation in other organisms Besides the biological model of withdrawal reflexes in Aplysia, there have been many other studies of learning and memory, ranging from invertebrates to mammals, such as neural plasticity in C. Elegans [4], odor learning in Drosophila mutants [7], Barnes maze spatial memory in mice [2], eyeblink response [12], and long-term potentiation (LTP) in the mammalian hippocampus [3]. The nematode C. Elegans has a small nervous system consisting of only 302 neurons. Similar to Aplysia, it also supplies a model organism to study learning-related neural plasticity. It can be easily cultivated in laboratory with E. coli as its food source. It has a very simple movement in the fashion of going forward or backward in a sine wave. The nervous system of C. Elegans has been thoroughly studied, and the functional roles of individual neurons are defined based on certain behavioral circuits. As in Aplysia, there are generally three types of neurons: sensory neurons, interneurons and motor neurons. However, it is found that these neural functions are not identical for different behaviors. Studies of underlying mechanisms of learning-related information storage on the molecular level are strongly coupled to electrophysiological recordings of neural circuits. Even though C. Elegans has a very simple nervous system and various kinds of studies have been carried out regarding its nervous system, the underlying mechanisms of learning-related synaptic plasticity in C. Elegans have not been revealed well. On the molecular level, the changes in the release of neurotransmitter between synaptic connections and the morphological alteration in neural circuits (both caused by learning) have yet to be explained as deeply as in Aplysia. However, studies of learning and memory responsible for regulation of behaviors have been carried out in combination with observations of changes in behavior. Behavioral plasticity in C. Elegans suggests that synaptic changes occur in the nervous system as a result of learning and memory. Considering three types of elementary learning, habituation, sensitization and classical conditioning, different training sessions have applied in C. Elegans and similar results as in Aplysia can be concluded. One example of habituation in C. Elegans is the response to taps on the side of dish. The worm in the dish would either stay still or move away as a result of such a stimulus. After several taps, the extent of the movement decreases. The duration of habituation varies from seconds to 24 hours depending on the number of stimuli as well as the length of the interval between stimuli [24] (Chapter 27). Memory of habituation is maintained longer after training sessions involving longer intervals. The studies indicate that the underlying processes of short- and long-term memory might require different neural mechanisms. The reactive movement can be divided into two patterns, since head touch leads to moving backward whereas tail touch activates moving forward. Two neural circuits responsible for each pattern of behavior act simultaneously to generate the response to those taps. It implies that the reaction is integrated with different functions of certain neurons. 9

16 Chapter 3 Related work regarding models of basic learning In this chapter, computational models from the literature are described. In Chapters 4 and 5, a new model will be introduced and analyzed. 3.1 Computational models of habituation As described in Chapter 2, habituation is one kind of basic learning, which allows animals to reduce their response to repetitive stimuli, thus avoiding exhaustion. Rather than focusing on such stimuli, the attention of the animal is drawn to new and possibly more important stimuli. This can make learning more efficient by ignoring repetitive inputs and focusing on interesting ones. On the other hand, if a stimulus has not been perceived for a period of time, the response recovers. This process, called dishabituation [19], shows that the effect of the stimulus (that caused habituation in the first place) is forgotten gradually. Habituation and dishabituation can be combined with other learning mechanisms, and constitute two basic steps for the discovery of learning. The mechanisms of habituation have been revealed physiologically to some extent. Habituation is caused by synaptic depression, i.e. a decrease in neurotransmitter release in the relevant synapses. Some biologically inspired models have been proposed to simulate the cellular processes of habituation. Moreover, the studies carried out on different animals, such as cat, toad and Aplysia, provide sets of real data to measure habituation. Thus, the results from the analytical models can be compared with experimental data in order to study the validity of the models and to analyze habituation quantitatively Stanley s model and the Wang-Arbib model Stanley [27] introduced a model to simulate short-term habituation based on the data obtained from the cat spinal cord [31]. The model exhibits a reduction of synaptic efficacy, y(t), according to the external stimulation, S(t). The algorithm is described by the following first-order differential equation: τ dy(t) dt = α(y 0 y(t)) S(t), (3.1) 10

17 Chapter 3. Related work regarding models of basic learning y(t) y(t) tau = 10 S(t) a=1.05 a= t a = S(t) tau=10 tau= t Figure 3.1: An illustration of Stanley s model. In both subplots, the stimulus S(t) is switched off in the time interval [150,200]. Other than that, S(t) is given by 1 continuously. The presence of S(t) causes the decrease in synaptic efficacy y(t), while the absence of S(t) brings the rise in y(t). Two groups of curves display the effects of different parameters, α and τ, with one of them fixed respectively. where y 0 is the original value of y; τ represents the time constant that regulates the rate of habituation, and α governs the rate of recovery. The equation describes that the synaptic efficacy decreases exponentially when the input signal S(t) is presented continuously and recovers spontaneously to its maximum y 0 when the input signal is absent. Fig. 3.1 displays the value of y, in a case where S(t) decreases abruptly from 1 to 0, and then increases instantaneously to 1. When S(t) is switched to 0 in the middle of the figure, there is an increase in synaptic efficacy, which represents the recovery process between two continuous habituation sessions. The four similar curves show the effect of the varying values for τ and α separately. As is evident from Eq. (3.1), with the same value of τ, the rate of recovery goes up when α increases. By contrast, with the value of α fixed, the rates of habituation and recovery go down when τ increases. While Eq. (3.1) can explain a number of properties of short-term habituation, Wang and Arbib adjusted it and added a new variable, z(t). This model can exhibit not only short-term but also long-term habituation [31]. Specifically, their model takes the following form: τ dy(t) dt = αz(t)(y 0 y(t)) βy(t)s(t), (3.2) dz(t) = γz(t)(z(t) l)s(t), (3.3) dt where y, y 0, τ, and α represent the same quantities as in Eq. (3.1). The factor βy(t), multiplying S(t), controls the habituation. z(t), which decreases slower than y(t) in the presence of an 11

18 Chapter 3. Related work regarding models of basic learning S(t),z(t) y(t) S(t) z(t) t t Figure 3.2: An illustration of the Wang-Arbib model. Note that the recovery of y(t) after the second habituation session (time interval [200, 350]), is slower than after the first one (time interval [0,150]). τ = 20, α = 2, β = 1.1, γ = 0.01 and l = 1.1 external stimulus S(t), regulates the rate of recovery. The more continuous external stimuli are perceived, the smaller z(t) becomes, resulting in slower recovery. In other words, more time is needed to forget being habituated to certain stimuli and to get out of profound habituation. The parameter l in the equation of variable z(t) should be larger than the initial value of z(t) to make sure that z(t) can only decrease. Making a biological interpretation, Eq. (3.2) would describe changes in neurotransmitter release, and Eq. (3.3) would imply morphological changes leading to long-term habituation [31]. The simulations by Wang and Arbib were also compared with experimental results, using stimulus-specific habituation data obtained from toads [31]. The simulation show quite similar results to the experimental data in different training sessions. Fig. 3.2 shows that z(t) only changes when S(t) is active, and the corresponding curve is in the form of an inverse S-shape. It allows y(t) to drop faster when consecutive stimuli are presented. This is shown in the figure, where the recovery of y(t) after the second of two consecutive habituation sessions (of the same length), is slower than the recovery after the first one Novelty detection using habituation Habituation can be used to filter stimuli that have been perceived repeatedly. This process is known as novelty detection [6]. Only novel stimuli can drive the attention and result in a certain reaction. As defined by Marsland, who has implemented Stanley s model as a novelty filter in different neural networks and real robots [19] [20] [21], habituation is a way of defocusing attention from features that are seen often [16]. The repetitive information is less 12

19 Chapter 3. Related work regarding models of basic learning interesting than the new one. For instance, the appearance of a waving hand can catch attention. On the contrary, if the hand keeps waving for a long time, less attention will be paid. As a result, a large number of stimuli from the environment can be incorporated. Lorenzo and Hernandez implemented the Wang-Arbib habituation model to detect repetitive patterns in visual and auditory sensory data [6]. In order to simplify the detection, a spectrogram of sensory data was used. It was extracted using the Fourier Transform to get what Lorenzo and Hernandez called the auxiliary signal. If the sensory data is repetitive, the auxiliary signal is 0. If the sensory data is changing, the auxiliary signal is 1. This signal is used as the input for the habituation model. The signals that are constant, or change with a fixed frequency or periodically, are considered to be lacking in variety. In the spectrogram, such signals show a specific pattern. Thus, such signals are the target to cause habituation, leading to lower synaptic efficacy. Different experiments were carried out, such as showing different colour images in an office environment and presenting signals recorded with a PC microphone containing different kinds of sounds in sections. The results showed appropriate levels of habituation to repetitive stimuli. Furthermore, the effect of different parameter settings were investigated Hallway navigation in mobile robots using habituation Chang added a habituation mechanism to a neural network that learns to predict a collision and turns away from obstacles detected by the side sensors on a mobile robot [5]. Before the habituation mechanism was applied, this reactive navigation allowed the robot to show obstacle avoidance behaviors very well in open environments. Nonetheless, the robot could not follow walls because of the continuous sensory stimulation. If the robot went into a narrow hallway, it produced an oscillation along the hallway and got stuck at the dead end. To solve this problem, Grossberg s computational model of habituation, which describes the changes of neurotransmitter release, was studied. This model is based on the same biological mechanism as the previous two (see above). However, instead of the synaptic efficacy y(t) in Stanley s model and the Wang-Arbib model, a variable g is introduced, representing the amount of transmitter, which can be used to determine the synaptic efficacy. The term of g multiplied by input signal x forms a gated input so that habituation only acts when the sensory input is perceived. The equation of variable g is very similar to Stanley s and Wang-Arbib s model with different names of parameters, as following: dg dt = ρ(µ g) δxg, (3.4) The differential equation is quite similar to Stanley s, but the second term on the right, g multiplied the input signal x, forms the gated input. Similarly, ρ regulates the recovery rate; µ is the maximum value of the transmitter level, and δ controls the habituation rate. g decreases with an increase in the sensory input x, while it dishabituates in the absence of input. In the neural network used by Chang [5], the input neurons are connected to the sensors distributed on the mobile robot. When the sensors detects an obstacle in its range, the connected sensory neuron in turn gets activated. Then it transmits the information to the rest of the network using Eq. (3.4), leading to a change in the corresponding angular velocity. If the robot is placed in a narrow hallway, the sensory neurons are kept activated, resulting in lower synaptic efficacy. In this case, the short-term habituation mechanism is a valuable help to eliminate oscillations by ignoring continuous sensory input. Results show that the robot 13

20 Chapter 3. Related work regarding models of basic learning CS sensory neuron US interneuron motor neuron Figure 3.3: The simplified neural circuit used by Magri et al. [18]. CS denotes the conditioned stimulus and US the unconditioned stimulus. can pass through narrow corridors. Moreover the performance of navigation is smoother. By varying the parameters in Eq. (3.4), especially ρ, the level of habituation can be controlled according to the width of the hallway, compared with the robot s diameter. This application of habituation in mobile robots to avoid oscillations was very successful, especially in hallways. Grossberg s short-term habituation model was proved to be easily applicable to different types of neural networks, since it only modifies the existing synaptic efficacy. Furthermore, it does not require any extra training of the neural network. The obstacle avoidance behaviors are improved due to the use of this simple habituation mechanism. Thus, it is useful and simple to combine habituation with other forms of learning. The effect of this kind of combination should, however, be explored further to achieve better adaptive behaviors. 3.2 A neural model of basic learning in Aplysia Magri et al. [18] proposed a computational model describing the variation of synaptic efficacy leading to habituation, sensitization and classical conditioning in a simplified neural circuit representing the gill-withdrawal reflex in Aplysia. The neural circuit consists of three neurons; a sensory neuron, an interneuron, and a motor neuron, shown in Fig The sensory neuron from the siphon is connected with the motor neuron innervating the gill, the withdrawal of which constitutes the defensive behavior. The interneuron mediating the sensory input from the tail modulates the synaptic efficacy between sensory neuron and motor neuron. Using terminology from the study of classical conditioning, Magri et al. refers to the siphon touch as the conditioned stimulus (CS), while the tail shock, i.e. the input to the interneuron, is termed unconditioned stimulus (US). Magri et al. used a biologically realistic neural model that simulates the dynamics of 14

21 Chapter 3. Related work regarding models of basic learning the membrane potential as well as potassium and calcium conductance in the presynaptic terminals of the neurons. The model consists of a set of differential equations that characterize the changes of the neural state. The brief paper by Magri et al. simply presents a model, without giving any detailed results, and there appears to have been no follow-up study. Nevertheless, an attempt was made to interpret and implement their equations, using a simplified neural model, since the detailed simulation of electrophysiological properties of neurons is less relevant here than the underlying logic of the model. However, the equations presented by Magri et al. are, in fact, somewhat incomplete: In order to obtain any meaningful results, several assumptions had to be made, thus perhaps rendering a detailed investigation of their model less relevant. The results will therefore not be given here. 15

22 Chapter 4 Model for memory formation in Aplysia The biological mechanisms of memory formation in Aplysia have been discussed in Sect The detailed processes occurring during short-term and long-term sensitization were explained for a simplified gill withdrawal circuit. In this chapter, a computational model for artificial organisms will be introduced. The model, which incorporates both short-term and long-term learning, consists of a set of equations [30] simulating the changes of synaptic efficacy during sensitization in the gill withdrawal circuit. 4.1 Short-term learning in artifical organisms Similar to Fig. 2.3, Fig. 4.1 displays a simplified representation of the gill withdrawal circuit. It consists of two input elements (squares), I 1 and I 2, two neurons (circles), the interneuron and the motor neuron, and the junction (diamond). In general, the purpose of a junction is to modify the strength of a synapse [30]. Compared with Fig. 2.3, I 1 and I 2 represent the touch signals from the siphon and the tail, respectively. Through the junction, the input from I 1 is transmitted to the motor neuron, to form the output signal x 1. The output signal determines the strength of the gill withdrawal reflex. I 2 passes through the interneuron whose output x 2 is the weight-modifying signal that is transmitted to the junction that modifies the weight connecting the siphon touch signal to the motor neuron. The input weight reaching the motor Siphon I 1 w S w J x 1 Motor neuron Tail I 2 w T x 2 w M Interneuron Figure 4.1: A simplified representation of the gill withdrawal circuit in Aplysia. 16

23 Chapter 4. Model for memory formation in Aplysia neuron is w J, which may differ from w S as a result of tail input; without any such input, w J is equal to w S. w T functions, similarly with w S, as the weight between the tail input signals and the interneuron. The dynamics of the neurons are described by the following differential equations and τ 1 dx 1 dt + x 1 = σ(w J I 1 ), (4.1) τ 2 dx 2 dt + x 2 = σ(w T I 2 ), (4.2) where τ 1 and τ 2 are time constants and the sigmoid function σ(z) is given by σ(z) = { 1 e cz if z 0, 0 otherwise (4.3) In sensitization, the weight connecting the siphon sensor to the motor neuron is enhanced as a result of a shock to the tail. Thus, after a single touch to the tail, the value of the weight w J is increased due to the output from the interneuron. However, the effect of the tail shock lasts only for some time, and w J gradually returns to its original value. The modulation of the weight variable w J is given by x 2 τ w1 dw J dt + (1 x 2 )τ w2 dw J dt + (w J w S ) = (w max w S )σ (w M x 2 ), (4.4) where w max is a pre-specified maximal value (of w J ), and τ w1 and τ w2 are time constants. To make sure that w J increases much faster than it goes back to the original value, τ w2 τ w1. The sudden rise of x 2, which leads to the increase of w J, right after a single tail shock, is ensured by a very small τ 2 and a large w T. As a result, x 2 becomes large (i.e. close to 1) following a tail shock, so that the second term multiplied by (1 x 2 ) can be ignored, and σ (w M x 2 ) 1. Thus, Eq. (4.4) is reduced (approximately) to dw J τ w1 + (w J w max ) 0, (4.5) dt so that w J w max. On the other hand, after the tail shock, when x 2 is very small (around 0), the first term multiplied by x 2 can be ignored, and σ (w M x 2 ) 0, leading to τ w2 dw J dt + (w J w S ) 0, (4.6) so that w J w S. The unknown constants to be determined are thus w S, w T, w M, c, τ 1, τ 2, τ w1, and τ w2 (w J is initially set equal to w S ). Possibly, different c-parameters could be used for the three sigmoid functions. 4.2 Long-term learning in artifical organisms The equations for short-term sensitization in Aplysia described in Sect. 4.1 regulate the increase of the synaptic efficacy between the siphon sensor to the motor neuron. Biologically, this process is due to the increase of neurotransmitter secretion in the presynaptic terminal, which is mediated by serotonin. Accordingly, in the proposed model, the weight, w J, connected to the motor neuron is strengthened. However, the amount of neurotransmitter only controls the 17

24 Chapter 4. Model for memory formation in Aplysia w S w J g w M C Figure 4.2: A junction (diamond) functions with the marker substance C and the weightmodifying gene g. existing synaptic efficacy. To form long-term memory, morphological changes, which require new protein synthesis, are essential. In long-term sensitization, a series of spaced tail shocks are applied. They trigger the activation of genes residing in the nucleus of the neural cell. The genes govern the synthesis of proteins, which are in turn used to strengthen existing synapse, or even to build new synapses. In a single neural cell, the number of synapses innervating the target cells is variable (see e.g. [25], Chapter 8 and [10]). By contrast, only one set of genes determines the weights of certain synapses, stimulated by serotonin, to be modified. Thus, the specific synapse is marked by the stimulus which has activated short-term learning, and long-lasting plasticity begins with gene regulation and new protein synthesis leading to the growth of such synapses. Such a biological process is simplified in Fig Applied to the gill withdrawal circuit, the weight modification involving gene regulation only functions on one synapse connected to the junction from the siphon sensor to form persistent structural change. In Fig. 4.2, the weights, w S, w J and w M, have the same meaning as in Fig Several stimuli from the tail are processed into x 2, which is multiplied by w M, the modifier weight, going to the junction. C is used as an artificial marker and has two different functions. First, it modifies the activation level of genes for long-lasting synaptic change. Second, it marks the synapses where long-lasting weight modification occurs. Since only one modifiable synapse is studied in this simplified model, it is not really necessary to use the marker C. However, the equations described below are intended to be applicable in neural networks with more than one modifiable synapse, in which case the marker C will be needed. The dashed arrows indicate how a gene g regulates the modification of w S, whose value is fixed during short-term learning. Once the activation level of gene, changing with C, is high enough, the weight w S is strengthened persistently. Accordingly, a set of equations explains such a long-term sensitization mechanism: where τ C dc dt + C = σ (k Cx 2 ), (4.7) τ C = τ ax 2 b, (4.8) τ g dg dt + g = σ (k gc), (4.9) 18

25 Chapter 4. Model for memory formation in Aplysia and τ ws dw S dt = { wmax ν (k s Cg) if w S w max, 0 otherwise, (4.10) where τ 0, a, b, τ g, k C, k g, τ ws, w max, and k s ( s for strengthening) are constant parameters to be set. The time constant τ C is determined by x 2. When there is no tail input signal, so that x 2 = 0, τ C returns to its original value τ 0. However, the fact that τ C varies in a manner opposite to that of x 2 ensures that the decrease rate of C is smaller than its increase rate, so that a series of tail input signals leads to a gradual rise in C. ν(z) is a modified sigmoid given by { 1 e c(z T) if z T, ν(z) = (4.11) 0 otherwise, where T is a threshold constant, used to determine when the values of C and g are large enough to trigger permanent modification of w S. If T = 0, ν(z) is the same as its special case σ(z). The gene g belongs to the neuron which the junction is connected to. Thus, all the synapses from such a neuron are regulated by this gene. However, only the weights of those synapses j whose marker levels C j are sufficiently high can be modified persistently. As a result, the long-term learning becomes synapse-specific. In this simple case, there is only one modifiable synapse and therefore only one C j (i.e. C 1 = C). Finally, this model is able to incorporate both the short-term and long-term sensitization by modifying w S and w J at the same time. 19

26 Chapter 5 Results The algorithm described in Chapter 4 was implemented to test it with siphon and tail input signals. The results of short-term and long-term sensitization, as well as the combination of them, are presented and explained in separate sections below. The effects of different parameters used in the simulations are also discussed. 5.1 Simulation results for STM According to the synaptic potentials in siphon and tail sensory neurons, shown in Fig. 2.5, the siphon and the tail input signals are presented in the form of spaced pulses in the simulation. In short-term sensitization, the expected output signal x 1 from the motor neuron should have the similar wave form changing with the siphon input signal. Moreover, once the tail pulse is applied, the corresponding output pulse to the siphon pulse should be enhanced. With several repetitions of siphon pulses, the amplitude of the output signal reduces to its normal value gradually. It indicates that without further tail input, the sensitization level fades. The duration of such recovery from sensitization shows how long the memory of this sensitization lasts. Fig. 5.1 shows a satisfactory result of short-term sensitization obtained in simulation. The siphon and tail inputs respectively represent I 1 and I 2 in Eq. (4.1) and Eq. (4.2). Before the tail pulse, the first siphon pulse causes a normal output pulse. The amplitude of the output pulse is indicated by the dashed line. With the effect of the tail pulse after the first siphon pulse, the response in x 1 is increased and the strength of the response decreases to the normal level. Fig. 5.2 exhibits how the weight, w J, between the junction and the motor neuron is enhanced by the tail input, and it drops back to its original value, which is equal to w S. This process is essential to get short-term sensitization, as described in Eq. (4.4). The tail input only acts to increase w J through the junction. However, w J eventually changes back to equal w S after the tail input (and therefore x 2 ) drops to zero. The strength and the duration of this short-term sensitization can be controlled by the parameters of the equations in Sect In Eq. (4.1) and Eq. (4.2), the sigmoid function σ(z) regulates the changes of x 1 and x 2 in the interval [0,1] and ensures that x 1 and x 2 will fall off to 0 in the absence of input signals. The parameter c governs the amplitudes of the pulses of x 1 and x 2 caused by I 1 and I 2. With an increase in c, the amplitudes become larger. τ 1 and τ 2 regulate the rate of change of x 1 and x 2. The smaller τ 1 and τ 2 are, the faster x 1 20