Aperiodic Dynamics and the Self-Organization of Cognitive Maps in Autonomous Agents

Transcription

1 Aperiodic Dynamics and the Self-Organization of Cognitive Maps in Autonomous Agents Derek Harter and Robert Kozma Department of Mathematical and Computer Sciences University of Memphis Memphis, TN 38152, USA Abstract Aperiodic dynamics are known to be essential in the formation of perceptual mechanisms and representations in biological organisms. Advances in neuroscience and computational neurodynamics are helping us understand the properties of nonlinear systems that are fundamental in the self-organization of stable, complex patterns in many types of systems, from biological ecosystems to human economies and in biological brains. In this paper we introduce a neurological population model that is capable of replicating the important aperiodic dynamics observed in biological brains. We use the mechanism to self-organize cognitive maps in an autonomous agent. Introduction The study of complex systems and nonlinear dynamics has blossomed in all areas of science in the past decades for many reasons. Nonlinear dynamics provide new conceptual and theoretical tools that allow us to understand and examine complex phenomena that we have never been able to tackle before. Nonlinear dynamics seem to show up everywhere, in physical systems like electrical circuits, lasers, optical and chemical systems. But such dynamics are especially ubiquitous in the biological world, from fractal growth patterns in biological development and city formation to the self-organizing characteristics of population models, and the importance in regulating healthy biological rhythms such as the beating of the heart. Nonlinear systems in critical states have many interesting properties. Phenomenon such as stochastic and chaotic resonance (Kozma & Freeman 2001) are known which enable such systems to actually detect the presence of signals much better in noisy environments than nonlinear systems are capable of doing. Their greatest interest lies however in their fundamental relationship to self-organization and emergence of complex patterns and behaviors in complex environments. Aperiodic or chaotic dynamics are both an indication of and a mechanism for the emergence of such self-organizing properties. Insights in nonlinear systems theory are beginning to be applied to understanding the dynamics of the brains, Copyright c 2004, American Association for Artificial Intelligence ( All rights reserved. and how such processes produce cognition (Freeman 1999; Tsuda 2001; Freeman 2003). Aperiodic dynamics are know to play a fundamental role in the mechanisms for the selforganization of meaning in mammalian perceptual systems (Skarda & Freeman 1987; Freeman 1991). Neurological evidence has shown that perceptual meanings (of recognized smells) are created through the formation and dissolution of chaotic attractors in the olfactory bulb. We will discuss this example of the self-organization of a perceptual pattern of meaning. We use this type of organization in aperiodic systems to model the formation of cognitive maps in the hippocampus of biological organisms. K-Sets: A Neurodynamical Population Model Aperiodic Dynamics in Olfactory Systems In their influential paper, Skarda & Freeman argued that chaos, as an emergent property of intrinsically unstable neural masses, is very important to brain dynamics. In experiments carried out on the olfactory system of trained rabbits, Freeman was able to demonstrate the presence of chaotic dynamics in EEG recordings and mathematical models. In these experiments, Freeman and his associates conditioned rabbits to recognize smells, and to respond with particular behaviors for particular smells (e.g. to lick or chew). They performed EEG recordings of the activity in the olfactory bulb, before and after training for the smells. The EEG recordings revealed that in fact, chaotic dynamics (as shown by the observed strange attractors) represented the normal state when the animal was attentive, in the absence of a stimulus. These patterns underwent a dramatic (nonlinear) transition when a familiar stimulus was presented and the animal displayed recognition of a previously stored memory (through a behavioral response). The pattern of activity changed, very rapidly, in response to the stimulus in both space and time. The new dynamical pattern was much more regular and ordered (very much like a limit cycle, though still chaotic of a low dimensional order). The spatial pattern of this activity represented a well defined structure that was unique for each type of odor that was perceptually significant to the animal (e.g. conditioned to recognize). Figure 1 shows an example of such a recorded pattern after recognition of a stimuli of the EEG signals and the associated contour map. In this figure after recognition,

2 Figure 1: EEG carrier wave patterns (left) and contour map (right) of olfactory cortex activity in response to a recognized smell stimulus (from Freeman, 1991, p. 80) Figure 2: Change in contour maps of olfactory bulb activity with the introduction of a new smell stimulus (from Freeman, 1991, p. 81) all of the EEG waves are firing in phase, with a common frequency (which Freeman called the carrier wave). The pattern of recognition is encoded in the heights (amplitude modulations) of the individual areas. The amplitude patterns, though regular, are not exact limit cycles and exhibit low dimensional chaos. In other words, different learned stimuli were stored as a spatio-temporal pattern of neural activity, and the strange attractor characteristic of the attention state (before recognition) was replace by a new, more ordered attractor related to the recognition process. Each (strange) attractor was thus shown to be linked to the behavior the system settles into when it is under the influence of a particular familiar input odorant. Figure 2 shows the effects on the spatial attractor pattern due to learning. Every time a new odor was learned by the animal, all of the existing attractor patterns changed. In this figure the contour pattern of activity for sawdust is shown (before learning the banana odor), for the banana odor, and then again for sawdust. Notice that the spatial pattern for sawdust no longer resembles its previous pattern. Whenever an odor becomes meaningful in some way, changes in the synaptic connections between neurons in different parts of the olfactory cortex take place. Just as in the Hopfield model and other associative type neural networks, these changes are able to create another attractor, and all other attractors are modified as a result of this learning. However, in real brains, the attractors of perceptual meaning are not simple point attractors, but are specific strange attractors. Freeman suggests that an act of perception consists of an explosive leap of the dynamic system from the basin of one (high dimensional, in the attentive state) chaotic attractor to another (low dimensional state of recognition) (Freeman 1991). These results suggest that the brain maintains many chaotic attractors, one for each odorant an animal or human being can discriminate. Freeman and Skarda speculate on many reasons why these chaotic dynamics may be advantageous for perceptual categorization. For one, chaotic activity continually produces novel activity patterns which can provide a source of flexibility in the individual. But since chaos is a ordered state, such flexibility is under control. As Kelso (1995) remarks, such fluctuations continuously probe the system, allowing it to feel its stability and providing opportunities to discover new patterns. Another advantage of chaos is that it allows for very rapid switching between attractors, which random activity is not able to do. K-Set Model of Aperiodic Dynamics The K-set hierarchy, developed by Freeman and associates (Freeman 1975; 1999; Skarda & Freeman 1987; Freeman 1991), is both a model of neural population dynamics and a description of the architectures used by biological brains for various functional purposes. The original purpose of the K-set was to model the dynamics observed in the olfactory perceptual system. The lowest level of the hierarchy, the K0 set, provides a basic unit that models the dynamics of a local population of tens of thousands of neurons. The dynamics of the K0 set are described by a second order ordinary differential equation feeding into an asymmetric sigmoid function: ab d2 x(t) d 2 + (a + b) dx(t) + x(t) = f(t) (1) t dt This equation was determined by measuring the electrical responses of isolated neural populations to stimulation. The a and b parameters are time constants that were determined through such physiological experiments. x(t) is the pulse density of the modeled neural population, in other words the average number of neurons that are pulsing in the population at any given point in time. f(t) is a nonlinear asymmetric sigmoid function describing the influence of incoming activation, and is given in equation 2. f(t) = k[1 exp( ev 1 )] (2) k A K0 unit models the dynamics of an isolated neural population. From the basic K0 unit can be built up architectures that capture the observed dynamics of increasingly larger functional brain areas. The KI models excitatoryinhibitory feedback populations. KII models interacting excitatory-inhibitory populations and correspond to organized brain regions such as the olfactory bulb (OB) or the prepyriform cortex (PC). KIII combine 3 or more KII populations to model functional brain areas such as perceptual cortex or hippocampus, and are capable of aperiodic dynamics of the type observed in these regions to, for example, derive meaning from perceptual senses. In the simulations presented in this paper, we use a discretized version of the K-model (described in (Harter & Kozma 2003; 2002; 2004)) developed for use in large-scale autonomous agent simulations. This model, called K-sets for autonomous agents (KA sets), uses a discrete time difference equation for the basic units, closer to the type used in standard artificial neural network models. However, the KA model units

3 have intrinsic dynamics tuned to behave like the original K- sets, and are capable of reproducing the types of aperiodic dynamics generated by the original K-sets. In the K models, the purpose of the KIII set is to model the chaotic dynamics observed in rat and rabbit olfactory systems (Freeman 1987; Shimoide, Greenspon, & Freeman 1993). KII are capable of oscillatory behavior, as described above. When three or more oscillating systems (KII) of different frequencies are connected through positive and negative feedback, the incommensurate frequencies can result in aperiodic dynamics. The dynamics of the KIII are produced in just this manner, by connecting three or more KII units of differing frequencies together. The KIII set is not only capable of producing time series similar to those observed in the olfactory systems, but also of replicating power spectrums characteristics of biological and natural systems in critical states (Solé & Goodwin 2000; Bak, Tang, & Wiesenfeld 1987). The power spectrum is a measure of the power of a particular signal (e.g. an EEG time series) at varying frequencies. The typical power spectrum of a rat EEG (see Figure 3, top) shows a central peak in the Hz range, and a 1/f α form of the slope. The measured slope of the power spectrum varies around α = /f α type power spectra are abundant in nature and are characteristic of critical states, between order and randomness, at which chaotic processes operate. Power spectra of biological brains have been observed to vary from α = 1.0 to α = 3.0. The EEG spectra of biological brains are characterized by a central peak in the Hz range is known as the γ frequency band. This peak is associated with cognitive processes in biological brains. The K and KA models are capable of replicating the power spectra of biological EEG signals, as shown in Figure 3, bottom. The KIII sets are capable of organizing perceptual categories in the fashion observed in biological perceptual systems. The KIII used as such a pattern classifier is very robust and compares well with more standard methods of pattern classification (Kozma & Freeman 2001). Hippocampal Simulation Experimental Architecture Perceptual meanings are formed through aperiodic attractors in the spatio-temporal activation of neuronal groups in the perceptual cortex. The same basic mechanisms of aperiodic dynamics observed in olfactory perception have also been detected in other perceptual and cortical areas (Barrie, Freeman, & Lenhart 1996; Kozma, Freeman, & Erdi 2003). We believe these same mechanisms are also used in other parts of biological brains, for cognitive mechanisms such as memory and goal-orientation. This theory of the importance of aperiodic dynamics in cognitive mechanisms is supported by many observations, including the similarity in structural elements observed in hippocampal and perceptual areas. In this paper, therefore, we use the basic KIII architecture originally developed for cortical-perceptual modeling to simulate formation of the so called place cells of cognitive maps in the hippocampus of an autonomous agent. Power KA III model (Group 3) Figure 3: The power spectrum of a rat Olfactory Bulb EEG (top) is simulated with the KA-III model (bottom). The calculated 1/f slope of the EEG and model is approximately Rat OB data from (Kay, Shimoide, & Freeman 1995), KA power spectrum from (Harter & Kozma 2003) In this experiment, we used a simulator of the Khepera robot in a virtual environment (Michel 1996). Figure 4 (bottom left) shows the morphology of the Khepera agent. The Khepera robot is a simple agent that contains 8 infra-red and 8 light sensors. It has two independently controlled wheels that allow it to move forward, backward, and turn left and right in place. The environment for this experiment is shown in figure 4. In the environment we place 8 light sources, which will be used as salient environmental locations (i.e. they can be thought of as good food sources for the agent in the environment). The light sources are detectable to the agent at a distance, and the range where the food source is detectable is indicated in Figure 4. In addition to the 8 salient environmental locations, there are 4 landmarks. The landmarks are always detectable to the agent, and it knows the distance and direction to each of the 4 landmarks as part of its sensory information. The architecture of the simulated hippocampus is shown in Figure 5. The portions of the architecture that form the cognitive map of the environment are simulated by a KA- III. These are the CA1, CA2 and CA3 areas, and are based on biological evidence of the structure of the biological hippocampus. Each of the CA areas contains an 8x8 array of KA-II units (for a total of 64 units in each CA region). Units in each CA region are connected to units in the other two. Each unit has many projecting connections to the other two regions, usually on the order of 8 to 16 projections. The interconnection of these three CA regions via inhibitory and excitatory feedback forms a KA-III unit. The connections between CA regions will be changed via Hebbian modification. Hz

4 is applied to the units. The DG units project randomly into the CA3 region, and the connections between these areas are also subject to Hebbian modification. Figure 4: Agent morphology (bottom left) and environmental setup for hippocampal simulations. The environment contains landmarks, used as allocentric reference points by the agent, and salient environmental locations, such as food sources. The agent is only able to detect the presence of a food source when it is within a particular distance of it. Figure 5: Architecture of KA-III hippocampal simulations Orientation beacons are fed into the hippocampal simulation through the DG region (Figure 5, left). The DG contains an 8x8 matrix single KA units. Orientation signals from the 4 landmarks are fed into the DG units. Each of the 4 landmarks has 8 units associated with the direction to the landmark, and 8 units associated with the distance. Directions are broken into 8 cardinal units, North, NorthEast, East, SouthEast, South, SouthWest, West and NorthWest. Units are sensitive to the direction of a particular landmark, though we use a graded response with a normal distribution. This means that more than one unit can be active with the most highly active unit being the one whose direction category best matches that of the landmarks actual direction. Similarly there are 8 cardinal distance values VeryClose, Close, MediumClose, Medium, MediumFar, Far, VeryFar, Distant. Again a graded response with normal distribution Method We use two types of learning in the simulation, Hebbian modification and habituation. Hebbian modification only occurs when the robot is within a certain range of a light source (the detection range). Therefore the light sources provide a certain valence signal that acts as a stimulus to learn environmentally salient locations. When the robot is not within proximity to a light source, no reinforcement signal is produced. During these times habituation of the stimulus occurs. This has the effect of lessening the response of the simulated hippocampus to unimportant regions in the environment (Kozma & Freeman 2001). The expected effect of this stimulation is to form 2 distinct types of dynamical patterns in the CA regions. When the agent is out of range of an environmentally salient location, the dynamics should be in the high-dimensional chaotic state, receptive to input but not indicative of recognizing a salient event. When in range of a light source, the system should transition to a low dimensional attractor, indicative of recognition of the important location. Further, the spatial amplitude modulation patterns in the CA regions upon such recognition should form 8 unique patterns, one for each of the recognized regions. The agent is allowed to roam in the environment, using a low level mechanism to produce efficient, but random wandering. By this we mean that the robot moves in random directions while exploring, but has mechanisms that cause it to avoid returning immediately to recently explored areas. The result is that all areas in the simulated environment tend to be well covered and visited multiple times during a typical simulation. The agent roams for some time, 10,000 time steps in our simulations. In our simulation 10 time steps approximates 1 second of real world running time, therefore the totaled simulated time of an experiment is 1000 seconds. Results We first give examples of the time series produced in the CA regions. Two broad classes of activity patterns organize themselves as a result of the Hebbian and habituation weight modifications. The spatial-temporal patterns stay in a relatively high-dimensional background state when the agent is in an uninteresting location. This pattern changes to a more regular (e.g. cyclic) pattern when the agent is close to a food containing area. The differences in these patterns come about as a direct result of Hebbian modifications being contingent on being within a meaningful area. Evidence of this shift, between high dimensional background state and low dimensional recognition state, can be seen in Figure 6. In this Figure we show a state space plot of one of the units from the CA3 area after learning, the unit in column 3 row 5 of the CA3 region to be precise. We show the activity of the unit when it is outside of a food source area (left) and when it is within range of a food source (right). Notice that the dynamics for the unit are much more cyclic and regular when the agent is in a recognized area,

5 Figure 6: State space plots of activity of unit in column 3, row 5 in the CA3 hippocampal region. The state space plot shows the activity of the unit plotted against itself with a time delay of 12ms. Left we show the activity of the unit when the agent is outside of an important region. Right is the activity when the agent is within range of a food source. Most units develop similar responses, which can be interpreted as a recognition of being in an environmentally salient area. they appear to have collapsed to a smaller volume of the attractor. The patterns of most of the units in the modeled hippocampus show similar transitions in their patterns from unrecognized to recognized areas. Next we look at the amplitude modulation (AM) patterns produced by the hippocampal simulation after learning of the environment has occurred. Figure 7 shows an example of the AM patterns formed in the CA3 hippocampal region. In this Figure, we show a contour plot representing the activity of all of the units in the CA3 matrix over a certain time period. The two patterns on the left are contour plots of the activity of the CA3 region while the agent was next to food source 4 in the environment (E4), while the two on the right represent AM patterns at location 7 in the environment (E7). The patterns, top and bottom, at the two environmental locations are from two randomly chosen locations within proximity of E4 and E7 respectively. To produce the contour plots, we placed the agent at the locations next to E4 and E7 for 1 second of time. During that 1 second, the orientation beacons and DG regions were stimulated with the appropriate orientation signals. We then captured the 1 second time series of the activity of all of the 8x8 units in the CA3 region. In order to represent the amplitude of the activity of a unit during the 1 second period, we simply measured the standard deviation of the time series. The standard deviation gives a rough measure of the amplitude of the activity of a unit over a time period. This gives us an 8x8 matrix of values representing the amplitude of each of the units while the different orientation signals were being perceived. The matrix of values can be used to produce the contour plots of the AM activity of the units over the time period. The AM pattern contour plots, therefore, give us an idea of which units are more highly stimulated (higher Figure 7: Example of AM patterns formed in the CA3 hippocampal region. In this figure we show contour maps of the activity of the CA3 region at two environmental food source locations, E4 (left) and E7 (right). At each food source, two spots were chosen randomly within range of the food source. Distinct AM patterns are formed and exhibited around the 8 environmental food source locations. See text for details. amplitudes in their activity) and which are less so when presented with the particular orientation signals. As Figure 7 shows, distinct AM patterns are formed by the aperiodic activity of the units in response to the orientation signals. In particular, we see a similar large peak of activity in the lower right units at location E4, and a large peak to the mid-left and a smaller peak mid-right in the E7 AM patterns. As a more complete test of the formation of unique AM patterns, we tested the robot with input from randomly selected locations, net to the environmental food areas. At each of the 8 food source locations, 4 points were randomly chosen within proximity to the location. We captured the activity of the CA3 units for 1 second of stimulation from the orientation signals at the randomly selected locations. AM patterns were collected for the randomly selected regions and compared to one another by calculating the distance between each pattern We can think of the 64 standard deviations of the 8x8 units as a point in 64 dimensional space and simply calculate the euclidean distance between any two points to measure their similarity. This testing showed that, in fact, the patterns produced next to a food source were consistently more similar to one another, than those produced in another environmental region. The KA-III hippocampal simulation described here, therefore, forms distinct AM patterns for the 8 salient environmental regions. These patterns are aperiodic spatiotemporal activity in the units of the simulated CA regions. The characteristic activity peaks in the AM patterns are examples of so called place cell formation. Here we see high activity among certain units (and low among others) correlated with being in a particular environmental location. It

6 is possible to interpret the AM patterns as being correlated with environmental locations, and therefore typical examples of the place cell phenomenon. Discussion The next step in this research is to begin to understand how such AM patterns might be used in the service of goaldirected navigation. It is known that if you measure the onset time of place cells in a biological brain, this time gradually shifts back in phase as the animal moves through the environment. This phase shift of the onset of the place cells may be evidence of the formation of navigation planning in the biological brain. One possible interpretation is that when the animal forms an intention to travel to a goal location, a sequence of AM patterns cycle through the hippocampus. This sequence can be interpreted as sequences of locations the animal intends to visit, from the current one to the next one, etc. in order to reach the goal. As the animal moves through the environment, its idea of the current location changes, and thus this whole sequence shifts back in phase in real-time to represent the next few intended steps the animal is planning to take. For this type of mechanism to be organized, the AM patterns must not simply form in an isolated way, but connections between adjacent locations must be incorporated into the mechanism. If the agent learns which AM patterns are co-located to which others, it may be possible to set up such a mechanism to produce a goal-directed planning for navigating in the environment. Conclusion The self-organization of spatio-temporal patterns in nonlinear systems are essential to cognitive mechanisms in biological brains. We need to better understand how such mechanisms operate in order to build better models of cognition and smarter autonomous agents. This paper has demonstrated one such self-organizing mechanism for the creation of AM patterns in a cognitive map of an agents environment based on the formation of aperiodic attractors in a highly recurrent model of the hippocampal system. Acknowledgment This work was supported by NASA Intelligent Systems Research Grant NCC References Bak, P.; Tang, C.; and Wiesenfeld, K Self-organized criticality: An explanation of 1/f noise. Physical Review Letters 59(4): Barrie, J. M.; Freeman, W. J.; and Lenhart, M. D Spatiotemporal analysis of prepyriform, visual, auditory, and somesthetic surface EEGs in trained rabbits. Neurophysiology 76(1): Freeman, W. J Mass Action in the Nervous System. New York, NY: Academic Press. Freeman, W. J Simulation of chaotic EEG patterns with a dynamic model of the olfactory system. Biological Cybernetics 56: Freeman, W. J The physiology of perception. Scientific American 264(2): Freeman, W. J How Brains Make Up Their Minds. London: Weidenfeld & Nicolson. Freeman, W. J The wave packet: An action potential for the 21st century. Journal of Integrative Neuroscience. in press. Harter, D., and Kozma, R Simulating the principles of chaotic neurodynamics. In Proceedings of the 6th World Multi-Conference on Systemics, Cybernetics and Informatics (SCI 2002), volume XIII, Harter, D., and Kozma, R Chaotic neurodynamics for autonomous agents. IEEE Transactions on Neural Networks. submitted. Harter, D., and Kozma, R Aperiodic neurodynamics using a simplified k-set neural population model. In Proceedings of 2004 International Joint Conference on Neural Networks (IJCNN 2004). submitted. Kay, L.; Shimoide, K.; and Freeman, W. J Comparison of eeg time series from rat olfactory system with model composed of nonlinear coupled oscillators. International Journal of Bifurcation and Chaos 5(3): Kelso, J. A. S Dynamic Patterns: The Selforganization of Brain and Behavior. Cambridge, MA: The MIT Press. Kozma, R., and Freeman, W. J Chaotic resonance - methods and applications for robust classification of noisy and variable patterns. International Journal of Bifurcation and Chaos 11(6): Kozma, R.; Freeman, W. J.; and Erdi, P The KIV model - nonlinear spatio-temporal dynamics of the primordial vertebrate forebrain. Neurocomputing 52-54: Michel, O Khepera simulator package version 2.0. Downloaded from WWW at om/khep-sim.html. Freeware mobile robot simulator written at the University of Nice Sophia-Antipolis. Shimoide, K.; Greenspon, M. C.; and Freeman, W. J Modeling of chaotic dynamics in the olfactory system and application to pattern recognition. In Eeckman, F. H., ed., Neural Systems Analysis and Modeling. Boston: Kluwer Skarda, C. A., and Freeman, W. J How brains make chaos in order to make sense of the world. Behavioral and Brain Sciences 10: Solé, R., and Goodwin, B Signs of Life: How Complexity Pervades Biology. New York, NY: Basic Books. Tsuda, I Towards an interpretation of dynamic neural activity in terms of chaotic dynamical systems. Behavioral and Brain Sciences 24(4).