Learning Topography in Neural Networks

Transcription

1 Learning Topography in Neural Networks Towards a better understanding of cortical topography DISSERTATION zur Erlangung des Grades Doktor der Naturwissenschaften an der Fakultät für Physik und Astronomie der Ruhr-Universität Bochum von If our small minds, for some convenience, divide... (the) universe into parts physics, biology, geology, astronomy, psychology, and so on remember that nature does not know it! (Feynman, 967) Jan C. Wiemer geboren in Bremerhaven Bochum, im Oktober 2000 Gutachter: Prof. Dr. Werner von Seelen Prof. Dr. Christoph von der Malsburg

2 To my parents

3 i ii Acknowledgments Abstract Our brain is functionally organized in topographic structure: typically, nerve cells (neurons) that are anatomically close to each other are also functionally close to each other. The functions of individual neurons are not genetically determined, but instead result from learning processes that enable the system to adapt to its environment. The learning of topography may be a significant reason for the amazing capabilities of biological information processing systems. Learning topographies in the cerebral cortex is the subject of this work. The following results are obtained:. From biological investigations well-known topographic structures can be learned stimulus-induced in natural environments, that is, with natural sensor information. I demonstrate this on the basis of functional structures in the primary visual cortex. Learning on the basis of natural sensor information has consequences for the resulting coding of signals, for example, for the coupling of signal parameters. 2. Temporal signal relations are of significance for topographic structures. For example, the functional proximity of two signals can be deduced from their temporal proximity. In connection with a biological learning experiment, which shows stimulus-induced reorganizations of the primary somatosensory cortex, I develop a model for the learning of dynamic signal representations from time-to-space transformations. The work offers a new perspective on cortical topography and signal coding. Moreover, it predicts further topographic structures in the cerebral cortex. This PhD-thesis was written at the Institut für Neuroinformatik, Ruhr-Universität Bochum, an interdisciplinary institute where physicists, biologists, engineers, mathematicians, and computer scientists work together in a pleasant atmosphere to discover and apply principles of biological information processing. I want to thank Professor Dr. Werner von Seelen, advisor and reviewer of my thesis, for his commitment to provide and financially protect this place of research. His scientific far-sightedness provided guidance in the wide interdisciplinary research area neuroinformatics. Furthermore, I want to thank Professor Dr. Christoph von der Malsburg, the other professor of the Institut für Neuroinformatik and also reviewer of my PhDthesis, for several discussions and helpful advices. I was always impressed by his overview of theoretical neurobiology and his ingenuity of quickly grasping complex scientific issues. I am much indebted to Dr. Friederike Spengler who was the driving force for the start of the second part of my thesis. She provided the biological motivation to think thoroughly about the role of time for the learning of cortical topography. Moreover, she was tireless in discussions between physicists, biologists, and engineers and supported my work even after she had left the ivory tower of science, still feeling responsible for our project. Also, her clear criticism of the structure of an earlier version of the manuscript led to considerable improvements. I wish to express my gratitude to Peter Stagge, Dr. Frank Joublin, Dr. Sylvie Wacquant, and Dr. Friederike Spengler. We met regularly, often late in the evening, discussing an experiment about cortical plasticity. These meetings of the Fünf Freunde were of substantial help for the progress of my work. Moreover, I am grateful to Dr. Frank Joublin for the intuitive picture of water drop dynamics, and for several drawings that illustrate Chapter 4 and the cover page. I thank Peter Stagge for the friendly atmosphere in our joint office and for the many healthy and stimulating apple breaks that we shared. The outcome of our work seems to contradict the proverb An apple a day keeps the Dr. away. I thank Dr. Thomas Burwick for many discussions about cortical processing and vector quantization, and for his corporation within the Sonderforschungsbereich Neurovision. It was a great challenge for two theoreticians to meet deadlines.

4 iii iv Stefan Schneider, Dr. Axel Steinhage, and Dr. Hubert Dinse deserve my thanks for giving helpful feedback on a draft version of my thesis. Stefan Schneider deserves special thanks for checking the manuscript thoroughly, asking numerous justified questions about badly written paragraphs. I thank Anja Busse for her help, finding the appropriate statistical test that I was looking for. I am grateful to David Kastrup for proofreading several parts of the manuscript concerning my usage of the English language. Gunnar Friege and Sven Moch deserve my thanks for reading the two most important chapters of the manuscript, Introduction and Conclusion, from the perspective of interested physicists. They gave valuable feedback for improvements. I thank Michael Neef and Arno Berg for their good job in computer system administration. I am especially grateful to Michael Neef for a great job in data security, even protecting the user from the consequences of his own Unix commands. I want to thank Rainer Menzner, Carsten Bruckhoff, and Christian Igel for their Latex and layout support. Our secretaries Heide Berz and Katharina Weissmann deserve a big thanks for the commitment with which they took care of organizational matters. Also, Carsten Winkel deserves thanks for running his kiosk and supplying us with chocolate bars. Funds for my research were provided by grant DFG SFB 509 and DFG SE 25/42-. me to study what I wanted to study, and supported me in a way I suppose only parents can. Thank you very much! Jan Wiemer Special thanks from my heart to my beloved wife Gesine Wiemer for her psychological support, for her proofreading of the German summary of this thesis, and for her patience when I came home once again late from work, especially during the last year of the thesis. Moreover, she bore our son David in January 2000, and David has enriched our life, opening our eyes to new dimensions of life, and also to forgotten ones. Finally, I want to thank my parents, Reinhart Wiemer and Dorothee Denck, by dedicating my PhD-thesis to them. They have always believed in me, encouraged

5 vi Contents Contents Introduction. Subject of this work Biological motivation Analysis of theoretical concepts Level of model formation Overview Cortical topography 9 2. Topographies on different spatial scales Analysis of topographic structures Neural responses Selection of stimuli Topology of stimuli Plasticity of topographic structures Biologically motivated mathematical terminology Mappings between stimuli and neurons Two concepts of cortical topography Definition of topography Remarks on the definition of topography Benefits of topography for cortical information processing Outlook: are there further topographies? Learning cortical topography from natural binocular stimuli Multi-feature representation of three-dimensional space High-dimensional coding Modular cortical architecture Stereo geometry Self-organization The model Self-organizing map (SOM) Definition of feature operators Orientation operator Ocular dominance operator Horizontal disparity operator Numerical experiment Natural binocular stimuli Numerical results Conclusion Temporal stimulus distances affect cortical topography 6 4. Neurobiological experiment Experimental paradigm Experimental results The model (TIS) Spatiotemporal stimuli Network architecture Neural activity and dynamics Shifts of activation Adaptation of weights Numerical experiment v

6 Contents vii viii Contents 4.3. A simple form of the proposed model Simulation of ontogenesis Simulation of post-ontogenetic plasticity Psychophysics The saltation phenomenon Application of TIS Discussion of the TIS approach Neurobiological boundary conditions Perceptual time scales of behavioral relevance Dynamic stimulus representation Topography from dynamics Representation of high-dimensional stimuli Time-based interpretation of cortical maps Maladaptive plasticity Biological plausibility of wave dynamics Mapping stimulus probability onto the cortex Neural field dynamics Beyond cortical maps Conclusion Learning topography from spatiotemporal stimuli Topography from spatial stimuli Spatial stimulus structure SOM learning The time-organized map (TOM) algorithm Description of the TOM algorithm Comparison of algorithms Topography from temporal stimulus distances Trivial spatial topology Temporal stimulus relations Time-organized maps Temporal course of learning Optimal mappings Definition of map error Analysis of map errors Facilitation of topography learning Competition of topologies Deviating topologies Illustrative example Generalization of results Conclusion Outlook Natural spatiotemporal stimuli Further model development From SOM to TIS to TOM Self-organized map (SOM) Temporal integration and segregation (TIS) Time-organized map (TOM) Conclusion Theoretical questions and results Biological relevance of results References 33

7 Abbreviations fmri foi ISI LFP PSTH RF SOM TIS TOM TRN functional magnetic resonance imaging functional optical imaging interstimulus interval local field potential post stimulus time histogram receptive field self-organizing map temporal integration and segregation time-organized map topology representing network

8 2 Introduction Introduction The brain is a fascinating and complex system, the scientific study of the brain a great challenge. We are still at the very beginning of understanding information processing in the brain. Central issues still await to be (fully) understood, for example, the encoding of information and the learning of this encoding. Concerning information encoding, the importance of the precise temporal structure of neural spikes, electric impulses exchanged by neurons, is intensively and passionately discussed. Moreover, the anatomical level of information encoding is unclear: is information encoded on the level of groups of neurons (populations), single neurons, substructures of dendritic and axonal arborizations, or perhaps on all of these levels together? Concerning learning, the brain has to allow flexibility and ensure stability at the same time: flexibility in the form of adapting neural structure, and stability of information processing running on the structure that is adapted. An improved understanding of information processing has a direct impact on how we see ourselves. Here, the question arises whether there are limits for our understanding: is our brain capable to grasp itself? Certainly, the brain can create new worlds and is able to cope with them. This quality goes to make up intelligence. Our brain has developed higher cognitive capabilities that are not directly enforced by its evolutionary development. For example, they are expressed in the form of music, art, and science. Accordingly, there is reason to believe that no fixed borders exist for our understanding of the brain. However, our brain results from an evolutionary development. Brains have evolved according to their capability to generate behaviors that are well tuned to the environment of the corresponding organisms and that ensure their survival. The spatial scale of these interactions of an organism with its surroundings approximates the scale of the organism itself, that is, a macroscopic scale, and the complexity of these interactions seems to be assessable. From this perspective, it is not surprising that human intuition may not be usable to grasp processes acting on much smaller spatial scales. The wave-particle dualism of quantum mechanics can serve as an example: although we can describe it precisely in mathematical terms, it seems as if we cannot understand it intuitively. Neuroscience may encounter a similar problem: the place of the small spatial scale is taken by the complexity of the brain. The complexity of the brain may be quantified by the number of its neurons and the number of their connections: 0 0 and 0 4, respectively (Braitenberg and Schütz, 99). The dimensionality of dynamical processes that can be intuitively understood is many orders of magnitude lower. Accordingly, a detailed description of neuronal dynamics on the level of single cells (or even below the level of single cells) is not sufficient to understand information processing in the brain. Which alternative levels of description can be applied? On account of the huge numbers of neurons and connections, a statistical approach may prove to be beneficial: in analogy to statistical mechanics one tries to deduce macroscopic system behavior from microscopic processes (Tsodyks et al., 996; Spiridon and Gerstner, 999). An alternative approach constitutes the search for and characterization of functionally defined structures within the brain. Here, one assumes implicitly that the brain is to some degree hierarchically structured into modules of information processing. This perspective reduces the number of degrees of freedom necessary for an understanding of the brain considerably. A typical approach combines cortical neurons into functional columns. This may prove to be fruitful because systematic orders between such columns are experimentally observed in many cortical areas.. Subject of this work The subject of this work is Learning topography in neural networks. In the following, the terms neural network, topography, and learning are clarified:. A neural network is an information processing system that consists of basic units called neurons and connections in between these units. This definition covers biological neural networks as well as artificial neural networks.

9 . Subject of this work 3 4 Introduction In the case of biological neural networks, for example, of nervous systems of man or animal, the term neuron refers to nerve cells. Roughly speaking, a nerve cell consists of an input unit (dendritic tree), a processing unit (axon hillock on the cell body), and an output unit (axon), see, for example, (Kandel, 996). Nerve cells interact (or communicate) with each other via synapses. Synapses are spatial loci along axons where electric signals (spikes) are transmitted to other neurons. The strengths of these synaptic transmissions are essential for the information processing within neural networks and are usually considered to follow from learning, see below. In the case of artificial neural networks, the term neuron refers to mathematically defined models of biological nerve cells. Such artificial neurons are typically structured in a simple way (Haykin, 998), for example, they evaluate a weighted sum of their inputs by the unit step function (McCulloch and Pitts, 943). Here, the crucial point is that elaborated input-output transformations can already be achieved by the choice of adequate connections between simple (nonlinear) elements. 2. Topography (topographic structure) denotes the continuous (gradual) variation of neural selectivities in anatomical (e.g., cortical) coordinates or in grid coordinates of an artificial neural network. The selectivity of a neuron is the neural activity pattern or stimulus that activates the neuron maximally. This rough definition of topography in neural networks should suffice for the moment. It is sketched in Figure.. In Chapter 2 an accurate definition is presented that defines continuity on discrete neurons in the framework of neighborhood preservation. Furthermore, Chapter 2 clarifies the term topography by experimental examples and theoretical comments. Neurobiological experiments have revealed (a) the existence of many topographies in the brain, for example, in the cerebral cortex (Section 2.), and (b) the plasticity of cortical topographies, i.e., their (re-)organization according to applied stimuli (Section 2.3). 3. Learning denotes processes that change neural parameters, for example, resulting in changes of topographic structures. This corresponds to a common In a more general framework, neural selectivity denotes the feature that neurons are specific for certain activity patterns presented as input ( stimuli ) or generated as output (activation of muscles). Cortex cortical coordinates neighborhood preserving Stimulus stimulus topology Figure.: Sketch of topography. Neural coordinates are mapped onto neural selectivities (i.e., maximally activating stimuli). Topography means that adjacent neural coordinates are mapped onto adjacent neural selectivities. usage of the term learning especially in the field of artificial neural networks (Kohonen, 997; Haykin, 998). In the context of system behavior, learning denotes the process of improving the performance of a system with regard to a specified task. The latter definition of learning is connected to the former definition by the following hypothesis: changes in cortical topography imply changes in cortical information processing, in perception and motor performance. This hypothesis has been confirmed by many experiments (Jenkins et al., 990; Recanzone et al., 992; Recanzone et al., 993; Elbert et al., 995; Wang et al., 995; Byl et al., 996)..2 Biological motivation Biological questions concerning the learning of cortical topography constitute the motivation of this PhD-thesis:. Is topography a general principle of cortical information processing? Topographic structures are found in early cortical areas. It remains unclear whether topographic structures are also implemented in many other cortical areas. Does the continuous variation of neural selectivities in cortical coordinates constitute a principal feature of cortical areas?

10 .3 Analysis of theoretical concepts 5 6 Introduction 2. To what extent does the formation and reorganization of neural selectivities and cortical topographies rely on learning processes that extract features of applied stimuli? Only to limited extent are selectivities of cortical neurons genetically determined. Many neurobiological experiments reveal the variability of neural selectivities and cortical topographies by the deliberate application of adequate stimuli. From a theoretical perspective one might argue that system stability could be ensured by genetic boundary conditions whereas system flexibility could be achieved by learning. 3. What principles are used to learn cortical topographies? What neighborhoods of neural selectivities does this imply? The assumption that topographic structures are typically implemented in cortical areas is especially plausible if the underlying processes for the generation of topographic structures are simple. Simple learning processes consist of interactions and learning rules that can easily be realized in biological neural systems. In the following chapters, new avenues are followed that may contribute to answering the above questions..3 Analysis of theoretical concepts In this work theoretical and mathematical concepts are developed and applied in order to improve our understanding of topographic structures in cortical areas. I proceed as follows:. Development of a theoretical concept: Theoretical ideas are developed in close connection with biological findings. 2. Converting the concept into model form: The theoretical ideas are converted into the form of an artificial neural network. 3. Analysis of system behavior: The artificial neural network is analyzed by numerical simulations. representation input 2 c k w ki 2 s i N c k i... N s Figure.2: Feedforward network. Sensory neurons of the input layer activate neurons of the representational (e.g., cortical) layer via connection strengths w ki (s =(s i ) : sensory activity, c =(c k )=w s : feedforward activation). 4. Comparison with biological data: Results from numerical experiments are compared with results from biological experiments. 5. Assessment of the concept and deduction of predictions: The latter comparison allows assessing the biological plausibility of the theoretical ideas converted into model form. Moreover, predictions can be made about correlations between experimental conditions (parameters) and measuring quantities. Checking such predictions in neurobiological experiments allows to check the underlying theoretical concept..4 Level of model formation The formation of models for biological neural networks will be conducted on the following level:. feedforward structure: The applied artificial neural networks consist of feedforward connections between neurons ordered in layers. The direction of information flow through the networks is fixed; neural activities propagate from one layer to the next layer, see Figure spike rates: Neural interactions are not modeled on a time scale corresponding to neural interactions on the basis of single spikes, but instead on the basis of spike

11 .5 Overview 7 8 Introduction rates of populations of neurons (e.g., neurons within one cortical column). The spike rate of a population of neurons is the frequency of spike emission by these neurons during time intervals of fixed width. 3. Hebbian learning: Connections between neurons result from Hebbian learning. In the framework of feedforward structure and spike rates, Hebbian learning denotes the strengthening of connections between neurons that are simultaneously active. Simultaneous activity of two neurons means that two neurons have increased (higher than average) spike rates during a time interval given by the time scale of modeling..5 Overview Chapter 2 gives an overview over cortical topography. It starts from the experimental perspective by reviewing several topographic structures. The experimental definition of topography is stated in detail and the difficulties encountered in the analysis of topography are presented and discussed. The plasticity of topographic structure is briefly reviewed. A theoretical description is given with an introduction of mathematical terminology used for the characterization of numerical neural network simulations. Cortical topography is functionally interpreted as beneficial for cortical information processing. This leads to the question whether a careful and close analysis of functional cortical structure could reveal many further topographic representations. In Chapter 3 representations of three-dimensional space are generated based on a simple model of self-organization, the self-organizing map (SOM). It is applied to binocular high-dimensionally coded stimuli. The representations are read out for different stimulus parameters. This results in maps that are compared to topographic representations of stimulus parameters found in neurobiological experiments. The correspondence between numerical and biological findings is discussed with regard to the parameters orientation and ocular dominance. Next, the computationally generated representations of three-dimensional space are analyzed according to the stimulus parameter disparity. The numerical simulations result in topographic representations of disparity. The results of Chapter 3 have been published previously (Wiemer et al., 2000a). Chapter 4 introduces the idea that the temporal proximity of incoming stimuli yields a topology that expresses the stimuli s relatedness and functional similar- ity. The idea is motivated by neurobiological experiments concerning plasticity in the somatosensory cortex of adult monkeys. They demonstrate that synchronous stimuli are integrated (represented at one cortical location), whereas asynchronous stimuli are cortically segregated (represented at distant cortical locations). The temporal integration and segregation (TIS) model is presented: it allows incoming stimuli to interact spatiotemporally in their cortical representations. An elementary propagating wave transforms temporal stimulus distances into representational distances. The TIS approach is applied to simulate neurobiological findings. Moreover, its basic mechanism, deformation of neural activity according to spatiotemporal correlations, is linked to psychophysics. Neurobiological and psychophysical predictions are deduced. Finally, the TIS model is discussed with regard to. neurobiological boundary conditions, 2. concepts of self-organized learning and dynamic stimulus representation, 3. neurobiological and psychophysical predictions, 4. biological plausibility, and 5. possible model extensions. The results of Chapter 4 have been published previously (Wiemer et al., 2000b). Chapter 5 presents the time-organized map (TOM) algorithm. Like the TIS algorithm, the TOM algorithm is founded on the idea of transferring temporal stimulus distances into topography. However, the TOM algorithm is much simpler so that its necessary key elements of neural interaction and learning are more easily analyzed. It is compared with the SOM algorithm that can be regarded as a special case of TOM learning with vanishing interactions between successive stimuli. The TOM algorithm is applied to different sets of spatiotemporal stimuli resulting in different functional maps. Moreover, two learning behaviors are demonstrated: topography learning from spatial stimulus structure is facilitated by additional temporal stimulus distances, if spatially and temporally defined topologies match. Deviating spatial and temporal stimulus topologies result in competition: depending on the interaction strength between successive stimuli either one of the topologies dominates or an intermingled representation is formed. Chapter 6 recapitulates the progression from the SOM model to the TIS model to the TOM model. The algorithms are characterized and compared, and the use of three different models to one biological phenomenon, cortical plasticity, is justified. Chapter 7 refers back to the central questions raised in Section.2 and summarizes the obtained results.

12 0 2 Cortical topography 2 Cortical topography primary visual cortex primary somatosensory cortex dorsal stream primary motor cortex premotor cortex frontal cortex ventral stream Cortical topography denotes the typical experimental finding that cortical neurons reveal similar selectivities if they are anatomically close to each other. Both, anatomical closeness and similarity of neural selectivities, can be considered on different levels. Section 2. and 2.2 are used to comment on these two important issues. Section 2.3 presents a brief overview about the plasticity of cortical topographic structures. In Section 2.4 the mathematical framework is presented that will be used for the description of topographic structure in the following chapters. Benefits of cortical topography for neural information processing are presented in Section 2.5. They lead to the question whether cortical topography constitutes a principle of cortical information processing, and whether many more, so far unknown cortical topographic structures exist (Section 2.6). 2. Topographies on different spatial scales In more global terms, cortical topography denotes the specialization of cortical regions and areas to different aspects of information processing. Many neurophysiological experiments reveal that different information processing tasks, for example, evoked by the presentation of different stimuli, activate different populations of neurons maximally. Accordingly, certain areas of the brain are more concerned with one function than with others: Hemispheric asymmetry: In a greatly oversimplified but didactically useful way, we may think of our brains as consisting of a left hemisphere that excels in intellectual, rational, verbal, and analytical thinking, and a right hemisphere that excels in perceiving and in emotional, nonverbal, and intuitive thinking. (Kandel et al., 996) primary auditory cortex inferotemporal cortex Figure 2.: Functional characterization of different regions of the human cortex. Processing of different sensory modalities (Figure 2.), for example, in the primary visual cortex on the occipital lobe, the primary somatosensory cortex on the parietal lobe, the primary auditory cortex on the temporal lobe. Streams of information processing, for example, the dorsal stream from primary visual cortex into the parietal lobe as the where path analyzing where objects are located, the ventral stream from primary visual cortex to inferotemporal cortex as the what path analyzing what we are looking at (Mishkin et al., 983; Tanaka, 997; Wallis and Bülthoff, 999). Why are neurons within cortical regions specialized on similar aspects of information processing? Neurons can be regarded as local operators: on average, the number of connections a neuron shares with other neurons decreases with the distance from its cell body. Accordingly, neurons that are spatially close to each other in the cortex interact more strongly than neurons that are far apart. This constitutes the reason why functional groups of neurons are found as spatial clusters within the cortex. 9

13 2. Topographies on different spatial scales 2 2 Cortical topography A retinotopy B orientation map Figure 2.2: The homunculus of the primary motor cortex. The relative allotment of the primary motor cortex to various muscle groups is shown. From Kandel and Jessell, 996. C hand representation D tonotopy mm Cortical functions, like the ones listed above, are not entirely separated. Instead, long-range interactions exist between most cortical areas (Abeles, 99; van Essen et al., 992). Nevertheless, this PhD-thesis focuses on the geometric distribution of neural functions within cortical areas. Long-range interactions between cortical areas are ignored. This approach is justified by the assumption that short range interactions within cortical areas are stronger than long range interactions between cortical areas. Going to a smaller spatial scale of about cm, experiments reveal an homunculus in the primary motor cortex of humans: neighboring neural regions activate muscles that are anatomically next to each other (Kandel and Jessell, 996), see Figure 2.2. The motor map has been obtained by direct electrical stimulation of the motor cortex of the human brain (Penfield and Rasmussen, 950). Topographic structures that express the continuous variation of neural selectivities are called maps. Typically, cortical topographic structures seem to break down below a characteristic spatial scale. For example, no topographic structure has been observed in the primary motor cortex on the spatial scale of mm. On this spatial scale, the anatomical position of activated muscles does not vary monotonically with the cortical position of electrical stimulation. Instead, drift of neural selectivities is superimposed by scatter of neural selectivities. Drift denotes the tendency how neural selectivities averaged over cortical columns vary, corresponding to topo- Figure 2.3: Cortical topographies in primary sensory cortices on a spatial scale of about mm and below. A: retinotopy in the primary visual cortex of the monkey (from Tootell et al., 982), B: orientation map in the primary visual cortex of the cat (from Hübener et al., 997), C: somatosensory hand representation in the primary somatosensory cortex of the monkey (from Jenkins et al., 990), D: tonotopy in the primary auditory cortex of the monkey (from Recanzone et al., 993). graphic structure on a larger spatial scale. Scatter denotes random fluctuations of selectivities around the drift. On a smaller scale, there are further cortical maps. The best analyzed cortical areas are those in primary sensory cortices. Topographic structures on an anatomical scale of down to one millimeter are, for example, retinotopy in the primary visual cortex (Hubel and Wiesel, 962, Figure 2.3A): the retina (or the visual field) is continuously mapped onto the cortex, see also (Schwartz, 980; Mallot, 985), somatotopy in the primary somatosensory cortex (hand representation shown in Figure 2.3C) (Marshall et al., 937; Merzenich et al., 978): the skin is continuously mapped onto the cortex,

14 2.2 Analysis of topographic structures Cortical topography tonotopy in the primary auditory cortex (Recanzone et al., 993, Figure 2.3D): the spatial order of sensory receptor cells of the ear expresses the functional parameter frequency and is continuously mapped onto the cortex. Note that cortical neurons are characterized above by sets of sensors that activate them. The receptive field (RF) of a cortical neuron is defined as the region of sensors from which the neuron can be activated by a simple stimulus (Hartline, 938). In the above examples, simple stimuli can be small (relative to RF size) spots of light, local tactile taps, or short tones of one frequency. In any case, the definition of neural RFs depends on the definition of simple stimuli used for their measurement. Well known topographies on an even smaller scale are orientation (Figure 2.3B) and ocular dominance maps of the primary visual cortex (Blasdel and Salama, 986; Swindale et al., 987; Bonhoeffer and Grinvald, 99). A pinwheel-like structure of orientation maps was already predicted from theoretical grounds in von Seelen, 970a, von Seelen, 970b. Topographic structures are not limited to early sensory cortices. Instead, there is experimental evidence for topographic representations of complex three-dimensional objects (faces) in the higher visual cortex of monkeys (inferotemporal cortex), see Figure 2.4 (Wang et al., 996). There may be further topographic representations of complex objects in higher visual cortices. The theoretical concepts and numerical experiments that will be conducted in the following chapters refer to the spatial scale of about mm. From now on, topographic structure on the spatial scale of about mm will be denoted shortly as topographic structure. The terms topographic structure, topographic representation, topography, and cortical map will be used synonymously. 2.2 Analysis of topographic structures Discovery and analysis of topographic structures are based on three key aspects:. an appropriate measure for neural responses (spikes, firing rates, etc.), 2. appropriate stimuli, and 3. an appropriate topology ( measure of relatedness ) for the set of stimuli. I use this section to comment on these aspects. In particular the latter two points are delicate and deserve special attention. Systematic shifts of activities in higher visual cortex () (2) (3) (4) (5) o 0 D P mm p < 0.05 p < 0.0 p < 0.05 p < 0.05 Figure 2.4: Topography in higher visual cortex. Systematic shifts of activations are induced by the rotation of complex three-dimensional objects (faces) in the inferotemporal cortex of monkeys (from Wang et al., 996). Top row: different views () (5) of a rotated doll face. Bottom row: cortical regions activated by the presentation of the different views () (5). Activity is monotonically shifted in cortical coordinates with viewing angle. Results from one monkey are shown on the left (using three different thresholds), and similar results in two other hemispheres are shown on the right (rightmost: shift of cortical activity induced by the rotation of a living human face, D: dorsal, P: posterior) Neural responses Neural activity consists of many temporally localized binary events denoted spikes. The analysis of neural responses can be based on the temporal structures of sequences of spikes emitted by (groups of) neurons. For example, two neurons can be recorded simultaneously and the temporal correlation of their spike emissions can be determined (Perkel et al., 967; Aertsen et al., 989; Riehle et al., 997). The measured number of synchronous spikes is then compared to the expected number of these events resulting by chance (Grün et al., 999). In the case of stimulus induced neural activity it is questionable whether the effort of such a spike-timing analysis would be justified. Moreover, there are considerable theoretical difficulties implied by this approach, for example, concerning the expectancy of synchronous spikes by chance (assumption of Poisson statistics) and the extension of this analysis to more than the pairwise analysis of two neurons.

15 2.2 Analysis of topographic structures Cortical topography With regard to stimulus representation, temporal structures in neural responses are usually neglected. Instead, neural activities are integrated over some fixed time window that is triggered to stimulus presentation. This approximates the strength of the neural processing that concerns the presented stimulus. Neural responses are averaged over several stimulus presentations in order to obtain reproducible measurement results, that is, sufficiently good signal-to-noise ratios. Among many other methods of recording neural stimulus induced activity, the following three techniques are the most important ones:. Applying extracellular microelectrode recording, neural responses are measured by the number of spikes emitted within a specific time window after stimulus presentation, or by the maximum amplitude of post stimulus time histograms (PSTHs) (Gerstein and Kiang, 960; Hubel and Wiesel, 962; Miyashita, 988). The smallest possible spatial scale is reached by this method, as single cells can be recorded and their position in the cortex can be marked. 2. Alternative measures of neural responses can be obtained by functional optical imaging (foi) based on intrinsic signals (Grinvald et al., 986; Ts o et al., 990; Bonhoeffer and Grinvald, 99) or dyes (Grinvald et al., 984; Grinvald, 985; Blasdel and Salama, 986). The former measure expresses variations in blood flow, the latter electric potentials analogous to local field potential (LFP) measurements. The highest spatial accuracy obtainable with optical imaging is about 0.2 mm. 3. Neural responses can be measured by functional magnetic resonance imaging (fmri) (Menon et al., 997; Tootell et al., 998). fmri is restricted by its worse spatial and temporal resolution in comparison to electrophysiological and optical measurements of about mm and to the order of seconds (Dale et al., 2000). However, recent experiments show that the spatial resolution of fmri may be improved by the use of higher magnetic fields and new data analysis techniques to about the spatial accuracy of optical imaging (Kim et al., 2000). In this work, the formation and adaptation of topographic structure will be simulated on spatial scales close to mm. Therefore, microelectrode recording and optical imaging are appropriate techniques to test the resulting model predictions. In the near future, fmri analysis may also be applicable for such experimental analyses Selection of stimuli Having decided to base the measurement of neural responses on one of the above techniques, the second aspect can be put into the following form: What are appropriate stimuli and appropriate conditions of stimulation for a good characterization of neural selectivities? I argue that topography is related to natural conditions of stimulation. Therefore, the analysis of cortical responses should be related to and based on natural stimuli. Natural stimuli are spatiotemporal activity patterns on sensory arrays (retina, skin, cochlea) that result from the animals behavior in its usual environment. The term natural stimulus is introduced in contrast to geometrically simple stimuli that are commonly applied in neurobiological experiments. With regard to vision, common simple stimuli are stationary Gabor patches or lines or moving gratings presented on a computer screen to one of the two eyes of an animal. Simple stimuli are mathematically simple: they are entirely described by only few parameters, for example, position, orientation or spatial frequency. The successive analysis of many geometrically simple parameters is problematic because it introduces a possibly wrong perspective concerning the analysis and interpretation of early cortical stimulus processing: the perspective that stimulus processing is decomposed into entirely independent processing pathways of single stimulus parameters. Moreover, the stimulus parameters are more or less arbitrarily chosen by the experimentalist. This ansatz may pose serious problems with regard to a complex re-binding of many parameters in further cortical processing leading to stimulus perception. From the perspective of many experimentalists, the use of geometrically simple stimuli seems to be straight forward for the analysis of complex systems like the brain. How can one hope to understand the brain s processing of an applied stimulus if the stimulus itself is already complex? Certainly, the analysis should be based on simple stimuli. But what is a simple stimulus from the perspective of the brain? A stimulus whose mathematical parameterization and technical presentation is simple may not be simple with regard to stimulus processing in the brain. I suggest that, from the perspective of the brain, a simple stimulus is a typical stimulus: one that is usually frequently encountered by the animal. Concerning stimulus processing in the visual cortex of animals with stereo vision, for example, of cats and monkeys, stimuli are typically. binocular, 2. local in spatial coordinates corresponding to local objects in three-dimensional space,

16 2.2 Analysis of topographic structures Cortical topography and 3. subject to continuous dynamics due to self- and object motion. The latter point is true in between two saccades, or on higher levels of cortical processing where discontinuous saccadic eye movements are compensated. The use of natural stimuli implies experimental and theoretical problems: In experiments, recording cortical activity from awake and freely behaving animals is technically difficult: each measurement technology restricts the animal s behavior, and vice versa. From a theoretical perspective, the use of natural stimuli is difficult for the analysis of experimental data. One has to deal with highdimensional stimulus spaces that are hard to visualize and handle. However, the latter problems may be small in comparison to the difficulties one encounters if one tries to analyze the cortical processing of inappropriate artificial stimuli. In contrast to geometrically simple stimuli, natural stimuli are behaviorally relevant, often bound to specific contexts, and meaningful to the animal. The processing of natural stimuli should (in some sense) be optimized in the brain. Moreover, natural stimuli constitute the basis of learning processes in brains. The characteristics of natural stimuli should be expressed in cortical topographic structures. The necessity of using natural stimuli leads to the following question: what is the simplest form of stimulation that can still be called natural? In other words, what is an elementary natural stimulus? Basically, this question has not been addressed by neurobiological experiments. The answer to this question may depend on the cortical area under consideration. Concerning stimulus processing in the early visual cortex, for example, of cats and monkeys, an elementary natural stimulus may be a localized three-dimensionally oriented edge under typical lighting conditions, seen by an awake (fixating) animal. Concerning stimulus processing in the higher visual cortex, an elementary natural stimulus may be the view of a familiar object or a face. The differentiation between stimulus and context is to some degree arbitrary: a stimulus can always be defined in a way that its context is included. The differentiation is usually based on different characteristics that are related to different parts of the entire stimulus, for example, differences in the time scale of their dynamics or differences in their temporal and spatial location Topology of stimuli Topography is defined by the gradual variation of neural selectivities in cortical coordinates. The stimulus that activates a given neuron maximally is denoted the selectivity of that neuron. Thereby, gradual variations of neural selectivities correspond to gradual variations of stimuli. The latter can be defined by the introduction of topology (or adjacency) in the space of stimuli: a gradual variation of stimuli is composed of transitions from stimulus to adjacent stimulus (see Section 2.4 for more precision). Accordingly, the selection of topology for the space of stimuli is a central issue in the definition of topographic structure. Stimulus relations in topographic structures are often not obvious, even on the level of early cortical processing. For example, the primary visual cortex processes many stimulus parameters whose topographic relations are extensively analyzed in neurobiological experiments (Hübener et al., 997; Kim et al., 2000). This raises the question how an appropriate topology can be defined for visual stimuli. The problem becomes even more obvious if stimulus processing in higher cortical areas is considered. A common topology follows from spatial Euclidian metric: the distance of two stimulus vectors s i, s j is given by d(s i,s j )= ( Ns ) /2 s i (x) s j (x) 2 x= (2.) (N s : dimension of vectors s i, s j ). The distance between stimulus vectors follows from differences between their components. The spatial Euclidian distance depends on the selection of stimulus encoding, that is, the selection of components that parameterize stimuli. Figure 2.5 illustrates two examples in which the spatial Euclidian distance is an adequate (A, B) and a non-adequate measure of stimulus relatedness (C, D). An alternative topology follows from the average temporal proximity of stimuli. It is especially useful for complex stimuli in high-dimensional stimulus spaces. The idea is illustrated in Figure 2.6. A three-dimensional L-shaped object moves continuously relative to an observer, it is shifted and rotated. At different instants of time it is projected onto an array of photo-receptors (e.g., the retina of one eye). Different high-dimensional activity vectors v, v 2, v 3, v 4 result. They constitute the basis of stimulus processing in the brain. The Euclidian distances of

17 2.2 Analysis of topographic structures 9 A discretized Gaussian stimuli v v 2 v 3 v 4 y i B Euclidian distance d(v i,v j ) x C handwritten digits v x v 2 x v 3 x v j y i 20 2 Cortical topography t t t D Euclidian distance d(v i,v j ) P v v 2 P P P v v x x x x j Figure 2.5: Spatial Euclidian metric and stimulus relations. The spatial Euclidian distance of stimuli does not always express their functional relatedness. The extend of shift between overlapping Gaussian functions (A) is expressed well by the spatial Euclidian distance: the latter increases monotonically with relative shift (white: zero distance, black: maximum distance; B). The relatedness of more complex stimuli, for example, handwritten digits (C), is not captured by the spatial Euclidian distance (D). The handwritten digits are taken from the MNIST data base provided by Yann LeCun ( yann/ocr/mnist/index.html). such activity patterns do often not reflect that they all describe the same object at successive instants of time. In the example, temporal proximity yields a better measure of stimulus relatedness: the four activity patterns belong together because they follow each other closely in time. Furthermore, time also expresses how the different vectors are related to each other: the time that passes (on the average over many trials) between two vectors measures the degree of their relatedness. Accordingly, the vectors v and v 2 are closer related than the vectors v and v 3, etc. Further stimulus topologies are introduced by specific processing tasks. Depending on specific processing tasks, stimuli can be grouped differently into classes, and the resulting stimulus classes can be related to each other in different ways. The obtained task dependent stimulus topologies represent task relevant infor- array of sensors array of sensors array of sensors array of sensors P: Projection Figure 2.6: Projection of a three-dimensional object onto the retina. At four successive instants of time, a moving L-shaped object is projected onto an array of sensors yielding activity vectors v, v 2, v 3, and v 4. mation. For example, if the task is to determine spatial loci of stimuli, the corresponding task-specific topology groups stimuli with regard to their loci independent of other features like stimulus color. On the other hand, if the task is to determine colors of stimuli, the corresponding task-specific topology groups stimuli with regard to color independent of spatial loci. In both cases, small errors of stimulus representation in task-specific topology remain small in their impact on task performance. Different stimulus topologies imply different interpolations between stimuli. Let us imagine an array of ten sensors. Two stimuli s A, s B with equal normalized intensities are given by s A = s(t A )=e 3, s B = s(t B )=e 5, as shown in Figure 2.7A. The stimuli are elements of the ten-dimensional space consisting of elements s = 0 i= λ ie i, λ i R. The stimuli s A, s B could result from a stimulus that moves along the sensory array, stimulating first the third sensor at time t A and later the fifth sensor at time t B > t A. The lack of input at the fourth sensor in the time interval (t A,t B ) may be due to the noise of the sensors or due to a process reading out the sensory array at discrete instants of time. When the stimuli s A, s B are

18 2.2 Analysis of topographic structures Cortical topography.5 A elementary signals (s A, s B ) 2.3 Plasticity of topographic structures signal amplitude signal amplitude signal amplitude S A S B B spatial Euclidian interpolation (s interp, ) C functional interpolation (s interp,2 ) sensory coordinate (j) interpolated in Euclidian coordinates we obtain Figure 2.7: Interpolation between elementary stimuli. s interp, = 2 (s A + s B ) = 2 (e 3 + e 5 ) (2.2) (Figure 2.7B). In the case of stimulus movement from left to right, a functional more meaningful interpolation would be s interp,2 = e 4 (2.3) (Figure 2.7C). An analogous result of interpolation is achieved with spatial Euclidian interpolation if the stimuli s A, s B are encoded in the one-dimensional space of their spatial position: s A = 3 e, s B = 5 e, and s interp,3 = 4 e. In the following chapters, different topologies will be transferred into topographic structures by stimulus induced learning processes. Different cortical areas may be topographically structured according to different stimulus topologies. Several cortical maps were presented in Section 2.. In their simplest form they express neighborhood relations of sensors (retinotopy, somatotopy, tonotopy). It has been argued that the generation of such maps results from genetically determined ordered growth processes of axonal connections from two-dimensional arrays of sensors to about two-dimensional cortical areas. However, the following aspects must be taken into account:. Experiments with invertebrates and also with mammals indicate that tectal or cortical maps are formed even if the growth process of axonal connections is disturbed (Sperry, 963; von der Malsburg, 988; Weber et al., 996). 2. The one-dimensional (two-dimensional) order of sensors is not maintained by axons within the acoustic (optic) nerve (Kohonen, 997; Shatz, 996; Ding and Marotte, 997). 3. There are further cortical maps that do not only express the order of sensors. Instead, more abstract stimulus features vary gradually within cortical coordinates, for example, orientation, ocular dominance, spatial frequency, direction, and probably also disparity in the primary visual cortex (Hübener et al., 997; Kim et al., 999; Burkitt et al., 998), views of faces in higher visual cortex (Wang et al., 996). 4. Experiments have shown that cortical maps are plastic: they adjust dynamically to incoming stimuli. This issue is sketched in the following paragraphs. The effect of sensory experience on the development of primary visual cortex was already observed by Hubel and Wiesel in 962 (Hubel and Wiesel, 962; Hubel and Wiesel, 965; Hubel et al., 977). They found that the visual cortex of kittens is plastic during a temporary critical period of ontogenesis, and that this plasticity is lost afterwards. Accordingly, they regarded the developing cortex as dynamic and the mature cortex as static (Garraghty and Kaas, 992). Since then, many experiments have demonstrated that the mature cortex retains a remarkable capacity to undergo reorganization. Many experiments concern the somatosensory cortex (Merzenich et al., 983; Merzenich et al., 984; Clark et al., 988; Jenkins et al., 990; Allard et al., 99; Recanzone et al., 992). However, there is also increasing evidence for adult plasticity concerning other modalities, that is, audition (Robertson and Irvine, 989; Recanzone et al., 993;

19 2.3 Plasticity of topographic structures Cortical topography Rajan et al., 993) and vision (Kaas et al., 990; Chino et al., 992; Gilbert and Wiesel, 992; Schmid et al., 996; Sugita, 996). Cortical maps serve as reference frames for the analysis of cortical plasticity. The gradual variation of neural selectivities in cortical coordinates simplifies the detection of cortical reorganizations: it allows to interpolate neural function in between cortical penetration sites of microelectrode mapping and justifies spatial averaging, for example, in optical imaging. Topographic structures before and after the presentation of stimuli that evoke cortical reorganization can then be aligned and compared. Moreover, changes in cortical maps can be directly interpreted with regard to underlying topographically represented parameters. The experiments demonstrating representational plasticity can be divided into two groups (Buonomano and Merzenich, 998):. Reorganization due to lesions, that is, peripheral or central alterations of inputs. The elimination of a limited defined set of normal inputs yields an expansion of the representation of still intact inputs formerly represented by neighboring cortical regions. The representation of still intact inputs expands into the region that represented the eliminated input. This is found in animals (Merzenich et al., 983; Merzenich et al., 984; Pons et al., 99) as well as in humans following limb amputation (Fuhr et al., 992; Kew et al., 994; Yang et al., 994). 2. Reorganization due to use, a) caused by experience during learning tasks: typically, additional cortical area is allocated for a larger representation of stimuli that are proportionally more used and/or behaviorally more relevant, see (Jenkins et al., 990; Recanzone et al., 992; Wang et al., 995) for animal experiments and (Pascual-Leone and Torres, 993; Elbert et al., 995; Pantev et al., 998) for evidence in humans, b) following surgical changes of sensory surfaces, for example, formation of syndactyly: representations are formed according to temporal input correlations. Synchronously stimulated sensors are represented within the same cortical column, see (Clark et al., 988; Allard et al., 99) for animal experiments and (Mogilner et al., 993) for the analysis of the reversal of congenital digital syndactyly in humans applying magnetoencephalography (MEG) recordings. Accordingly, cortical maps are not entirely genetically determined. Instead, they adjust according to incoming stimuli. At least to some degree, cortical maps have to be regarded as resulting from stimulus induced bottom-up processes shaping neural selectivities. Such processes are commonly denoted as self-organization ( self-organizing processes ) in order to distinguish them from top-down learning processes (von der Malsburg, 973; Willshaw and von der Malsburg, 976; Amari, 980; Kohonen, 982; Haken, 995). 2 Topographic structures in the adult cortex seem to express dynamic equilibria that are based on the statistics and dynamics of represented stimuli (Garraghty and Kaas, 992; Joublin et al., 996). 2.4 Biologically motivated mathematical terminology 2.4. Mappings between stimuli and neurons Cortical topographies can be described mathematically by mappings Φ from sets of stimuli S R d s onto sets of vertices K = {,2,...,N c } of a graph (also neurons or neural units of a network or neural grid, d s : dimension of stimulus encoding, N c : number of neurons, index c for cortex) Φ : S K. (2.4) The set of stimuli, S, is supplied with a topology that defines neighborhood relations among stimuli. The choice of this topology is a central issue and may be problematic, as pointed out in Section The set of vertices, K, is supplied with a topology by the neural graph that defines neighborhood relations among neural units. The latter topology corresponds, for example, to the anatomical 2 The terms bottom-up and top-down refer to functional hierarchy in stimulus processing, from detecting single object features to perceiving whole situations and generating appropriate behavior. To some extend, such a hierarchy can be observed in the brain. 3 In neurobiological experiments the choice is usually based on ad hoc assumptions. The latter become especially obvious when higher cortical areas are mapped (Tanaka et al., 99; Wang et al., 996).

20 2.4 Biologically motivated mathematical terminology Cortical topography spatial relations of neurons in the brain. Typical choices of graphs are twodimensional rectangular grids (Chapter 3 and 4) and one-dimensional chains of neurons (Chapter 5). The common approximation of the cortex as a two-dimensional grid of neurons is justified anatomically by the ratio of its surface to its width, 000 cm 2 to 2 mm for man (Hubel, 995). Functionally, the approximation is justified by the columnar structure of the cortex: neural features typically remain about constant in the direction from cortical surface to white matter but vary in directions parallel to the cortical surface (Mountcastle, 957; Hubel and Wiesel, 962; Tanaka et al., 99). One-dimensional networks simulate the variation of neural responses along fixed directions parallel to the cortical surface. In the following, the mapping Φ of (2.4) will be determined by ordered sets of weight vectors ( pointers, representatives ) W =(w,w 2,...,w Nc ), w k R d s attached to the vertices k. Φ maps stimuli s onto vertices k (s) that are maximally activated by s, for example, via feedforward connections Φ W (s)=k (s)=argmax(w k s) (2.5) k ( denotes the scalar product), see Section 3.2 for details and Section 4.2, 4.3, and 5.2 for more general mappings that take temporal stimulus relations into account. In other words, a neural coordinate k (s) Kis assigned to every stimulus s (von der Malsburg, 973; Kohonen, 982; Martinetz and Schulten, 993). In biological terms, the mapping Φ W describes the localization of maximal neural responses in the brain generated by the presentation of a stimulus. The locus of maximal neural activity within a brain region of special interest can, for example, be detected by foi or fmri (Godde et al., 995; Wang et al., 996; Tootell et al., 998). Another experimental approach is inverse to the just described procedures foi and fmri: first, a locus within the brain is chosen, for example, the cortex is penetrated with a microelectrode. Then a set of stimuli is presented (Hartline, 938; Tanaka et al., 99; Fujita et al., 992; Kobatake and Tanaka, 994; Tanaka, 997). The stimulus yielding maximum response is attached to the chosen locus within the brain: it approximates the selectivity of the recorded neuron. Out of the presented set of stimuli this stimulus is best represented (strongliest processed) at the chosen site of the network. The second experimental procedure corresponds to inverse mappings Φ W from sets of vertices, K, to sets of stimuli S, for example, Φ W (k)=s (k)=argmax s (w k s). (2.6) Here, a stimulus s (k) is assigned to every vertex k K. If the stimuli are densely distributed within R d s, Φ W (k) approximates well the weight vector w k of the k-th neuron: Φ W (k) w k. (2.7) If Φ W is bijective then Φ W is the inverse mapping of Φ W, Φ W = Φ W. (2.8) However, many stimuli s are usually mapped onto the same vertex k (s), that is, Φ W is not injective. Moreover, not all neural units may be reached by the mapping Φ W, Φ W (S) K, that is, Φ W is not necessarily surjective. Neurons k K\Φ W (S) are superfluous from the perspective of vector quantization where stimuli are represented by single neurons as in (2.5). 4 Such neurons may be called dead units Two concepts of cortical topography Cortical topographic structures can be characterized by two different concepts: dimension reduction and neighborhood preservation. The two concepts constitute different perspectives onto the central issue of how stimulus relations are transferred to neural coordinates:. The concept of dimension reduction describes topographies by mappings from high-dimensional stimulus spaces to low-dimensional anatomical spaces of neural coordinates (Durbin and Mitchison, 990). Due to the difference in dimensionality, a topographic mapping cannot preserve all stimulus dimensions. Instead, only a few dimensions are preserved, the number of these dimensions is restricted by the dimensionality of neural coordinates. The benefit of dimension-reducing mappings is to focus on essential stimulus characteristics: the most relevant stimulus dimensions are selected and 4 In vector quantization signals are approximated by a finite number of codebook vectors. A signal s is approximated by the codebook vector that is closest to s.

21 2.4 Biologically motivated mathematical terminology Cortical topography neighborhood of neuron k spatial raeumlich-euklidische Euclidian Metrik metric temporal zeitliche Metrik metric _ k k k + Figure 2.8: Spatial neighborhood in a chain of neurons. less relevant dimensions are neglected. An example is the selection of signal dimensions according to their variance (principal component analysis, Oja, 982). From a functional perspective, the key question concerning the concept of dimension reduction is the following: which dimensions of the space of stimuli are selected and preserved? 2. The concept of neighborhood preservation stresses the importance of choosing topologies or metrics for the description of topographies. The space of stimuli and the space of neural coordinates are topological spaces: neighborhood relations are defined in these spaces. A neural network is neighborhood preserving, if the corresponding mapping from the space of stimuli to neural coordinates or the inverse of this mapping is neighborhood preserving (Martinetz and Schulten, 993; Villmann, 996). The benefit of neighborhood preservation lies in the preservation of locality: processes run locally in stimulus space if they are processed locally in the neural network (or vice versa). Here, the term locally refers to the respective topology. The key question concerning the concept of neighborhood preservation is the following: which topology can be applied to the space of stimuli and the space of neural coordinates so that the neural network is neighborhood preserving? In the following, I focus on the concept of neighborhood preservation Definition of topography I denote a neural network described by Φ W (s) and Φ W (k) topographic ( of topographic structure, cortical map ) if the mapping Φ W (k) is sufficiently neighborhood preserving. What is meant by sufficiently neighborhood preserving is now put into mathematical terms. The adjacency of neurons in the graph of a network is defined in a straight forward manner. Two neurons k, k in a one-dimensional chain of neurons are neigh- s 3 s s s i s j d( s i, s j ) s 2 s s 2 0 s 3 s s s i s j representative trajectory s s 2 0 d( s i, s j ) Figure 2.9: Spatial and temporal distance in the space of stimuli. Especially for high-dimensional stimuli, the two measures can yield different neighborhood relations, see Chapter 5. bors (adjacent) if there is no neuron in between them: k k, see Figure 2.8; or two neurons k =(k,k 2 ), k =(k,k 2 ) in a two-dimensional rectangular grid are neighbors if the condition k k k k holds for all k K\{k}. 5 The adjacency of stimuli Φ W (k) =s (k) attached to a neural network can be deduced from different measures of stimulus distances, for example, from the spatial Euclidian distance ( spatial distance ) d(s i,s j )= s i s j R ds. (2.9) R ds denotes the Euclidian norm in R d s, see (2.) and Figure 2.9, left. Selforganization of topographic structure based on purely spatial stimuli and their spatial Euclidian distances will be analyzed in Chapter 3. An alternative distance measure is based on the time that typically elapses between the occurrence of two stimuli ( temporal distance ) ds d(s i,s j )= s i v(s), (2.0) see Figure 2.9, right. Here, the integration is carried out along an average (or representative) trajectory. In Chapter 4 and 5, I will assume typical interstimulus intervals (time intervals between two stimuli, ISI s) that characterize the flux s j 5 Different norms can be used in the above expression, for example, the Euclidian norm k E = i k i 2 or the maximum norm k max = max i k i, see Villmann, 996 for the relevance and consequences of such differences. s2

22 2.4 Biologically motivated mathematical terminology Cortical topography of incoming stimuli. To every pair of stimuli (s i,s j ) an ISI isi(s i,s j ) will be assigned, defining temporal distances between stimuli. Following Martinetz and Schulten, 993, two stimuli Φ W (k) =s (k), Φ W (k )= s (k ) attached to a neural network are defined to be neighbors (adjacent) if and only if a stimulus s Sexists that possesses s (k) and s (k ) as the next and next but one stimulus representatives of the neural network: A topographic network B not topographic network s 2 s 2 s 2 s (k),s (k ) neighbors s S exists with d ( s,s (k) ) d ( s,s ( k) ) d ( s,s (k ) ) d ( s,s ( k) ) for all k K\{k,k }. (2.) d(, ) denotes a metric in stimulus space, for example, according to (2.9) or (2.0). Having defined adjacency for stimuli attached to the network, the term topography can be specified as follows:. The mapping Φ W : K Sis neighborhood preserving in k Kif and only if the attached stimulus Φ W (k )=s (k ) of every neighboring neuron k of k is a neighbor of Φ W (k)=s (k): k neighbor of k s (k ) neighbor of s (k). (2.2) 2. The neural network given by Φ W and Φ W is topographic of degree D if and only if Φ W is neighborhood preserving in D N c out of N c neurons. 3. The neural network given by Φ W and Φ W is topographic if and only if it is topographic of degree D θ D (θ D : some fixed threshold, e.g. θ D = 0.7). The definitions of neighborhood preservation and of topography are illustrated in Figure 2.0. Stimuli s =(s,s 2 ) of a stimulus space S =[0,] [0,] with spatial metric according to (2.9) are attached to a chain of neurons k K= {,2,...,0}. The neurons are drawn at the positions of their selectivities in the space of stimuli and connected by lines according to their adjacencies in the neural chain. In addition, the space of stimuli is partitioned by Voronoi polyhedra, that is by areas in which a certain neural selectivity constitutes the best stimulus approximation. An in all neurons k Kneighborhood preserving mapping Φ W (k) from neural coordinates to the space of stimuli is shown in Figure 2.0 A. All adjacencies in k=3 k= s 0 0 s s s Figure 2.0: Illustration of neighborhood preservation. Two one-dimensional neural networks are shown for the special case of a square stimulus space with spatial metric: a topographic network that is neighborhood preserving in all neurons (A) and a not topographic network that is neighborhood preserving only in one neuron (k = 3, B). the neural chain are preserved in the form of adjacencies of the Voronoi polyhedra in the space of stimuli. Accordingly, the neural network of Figure 2.0 A is topographic. Contrarily, the mapping Φ W (k) shown in Figure 2.0 B is only neighborhood preserving in neuron k = 3, that is, it violates neural adjacencies in all other neurons. Accordingly, the neural network presented in Figure 2.0 B is not topographic Remarks on the definition of topography Let me comment on the above definition of topography by the following remarks: Comparison with topology representing networks: The definition of topography requires only that the inverse mapping Φ W (k) from the space of neural coordinates to the space of stimulus encoding is neighborhood preserving. This is less restrictive than the definition of topology representing networks (TRNs) where both mappings, Φ W (s) and Φ W (k), have to be neighborhood preserving (Martinetz and Schulten, 993). TRNs express mappings between stimulus spaces and neural spaces with equal

23 2.4 Biologically motivated mathematical terminology Cortical topography numbers of dimensions. This stands in contrast to the concept of dimension reduction. More elaborate measures for topography can be deduced from measures for topology preservation, like the topographic product (Bauer and Pawelzik, 992) or the topographic function (Villmann, 996; Villmann et al., 997). The latter measures can be restricted to neighborhood preservation by the inverse mapping Φ W (k). Trivial stimulus topology: Note that this definition of topography is meaningless in the case of trivial stimulus topology. Trivial topology for a set of elements {a i i N} is defined by the trivial metric d(a i,a j )=c( δ ij ), (2.3) with c R >0. The trivial topology is special in the sense that all elements are adjacent to one another: This result is obtained by the use of (2.): a i neighbor of a j for all i, j. (2.4) d(a,a i ) d(a,a ) d(a,a j ) d(a,a ) (2.5) is satisfied for all a a i,a j by a = a i because d(a i,a )=c = max i,i d(a i,a i ) (2.6) for a a i. Accordingly, a mapping Φ W (k) that maps neural vertices into a stimulus space with trivial topology cannot violate neighborhoods; it is trivially neighborhood preserving of degree D =. Alternative definition: In some cases of topography learning it is feasible to derive optimal mappings Φ W,opt(k) concerning preservation of spatial and temporal stimulus metrics. In these cases, the following error measure can be applied: E top = N c N c k= w k Φ W,opt(k) (2.7) see Section 5.4. A network is topographic, if and only if its error lies below some fixed threshold: E top θ E. This definition of topography characterizes learned topographic structures sufficiently in Chapter 5 and will be applied there. 2.5 Benefits of topography for cortical information processing From the perspective of neural information processing, cortical maps possess several advantages:. Robustness in spike transmission: The representation of related stimuli next to each other allows to shorten neural communication links that concern the processing of these stimuli. The error probability in spike transmission increases with axonal length. Action potentials may be lost during their transmission, or their transmission time (and impact) may vary (Kohonen, 997). 2. Energetic efficiency: The maintenance of neural connections is energetically expensive for a biological neural system. An electrical potential difference that results from the separation of charge across the cell membrane has to be maintained by active (i.e., energy consuming) pumping of Sodium and Potassium ions (resting membrane potential: about 70 mv). Longer neural connections yield larger membrane surfaces that require a larger number of Na + K + pumps. 3. Sophisticated interactions: The number of synapses that are potentially shared by two cortical neurons decreases with their anatomical distance. Accordingly, the interactions between two nearby neurons can be more sophisticated, more pronounced, and more precise than between two distant neurons. It seems to be favorable to group neural selectivities according to the degree of their relatedness in frequent and significant information processing tasks. 4. Robustness concerning death of synapses: A strong cooperation in information processing between neighboring neurons could also imply a sound degree of redundancy in neural connections.

24 2.6 Outlook: are there further topographies? Cortical topography This increases cortical robustness with regard to the death of single neural connections (axons, synapses). For example, the death of one synapse may be compensated by a parallel, former redundant synapse that increases its strength. 5. Simplification of interactions between cortical areas: Clustering of stimulus features in cortical maps is a first step towards abstraction and categorization (Ritter and Kohonen, 989). Cortically localized, clustered, and ordered features may simplify interactions between cortical areas, for example, between somatosensory and motor cortex (Kohonen, 997). 2.6 Outlook: are there further topographies? There are many topographic structures of early sensory cortices that show gradual variations of neural selectivities at the scale of about mm (Section 2.). Furthermore, recent experiments indicate that the concept of topographic stimulus representation extends to higher cortical areas as well (Tanaka et al., 99; Wang et al., 996). The latter finding is contradictory to the widely held opinion that cortical maps are limited to early cortical areas. Accordingly, the following questions arise:. Are there further (so far unknown) topographic representations in the cortex? (Kohonen and Hari, 999) 2. On which criteria should experiments be based that aim to discover further topographic representations? Difficulties are encountered in the analysis of cortical representations and the discovery of cortical maps: stimuli and stimulus topologies have to be chosen appropriately. The topologies can be based on different spatial stimulus encodings or on temporal stimulus relations. The difficulty of making appropriate choices may constitute the reason why not many topographies have been observed experimentally in higher cortical areas. Concerning higher cortical areas, serious problems are encountered in the analysis of stimulus representations and the discovery of topographic structures. The neural connectivity between different cortical areas is complex (van Essen et al., 992). So far, it is impossible to determine the anatomical origin and functional meaning of all afferents that converge onto single cortical neurons or columns. Accordingly, all experimental analyses of cortical representations are restricted to the stimulation of subsets of afferents of recorded neurons. Subsets of afferents of neurons correspond to subsets of stimulus parameters that neurons represent (process). Due to this uncertainty in neural selectivities, the preservation of stimulus topologies in topographic structures is difficult to detect in higher cortical areas (Ritter and Kohonen, 989). Experimental findings of non-gradual variations of neural selectivities, for example, in higher visual cortex (Tanaka et al., 99; Fujita et al., 992; Kobatake and Tanaka, 994), are not contradictory to the existence of topographic representations. Instead, they may result from the experimental restriction to subsets of stimulated afferents. The variation of neural selectivities may still be gradual in anatomical coordinates with regard to the spatial structure of neglected stimulus parameters. Or the variation of neural selectivities may be gradual with regard to typical temporal stimulus relations. Concerning the motor cortex no further topographic structures on the spatial scale of about mm have been found. Researchers have argued that there are no further topographies in the primary motor cortex (Georgopoulos et al., 986; Georgopoulos et al., 993). However, we are here confronted with the problems outlined above. Neurons that activate muscles can be grouped differently, for example, by the anatomical position of the corresponding muscles or according to temporal relations between the activations of different muscles during elementary movements. The latter metric may be realized topographically in the motor cortex. In the following chapters further topographies are predicted:. topographic representations of disparity in the primary visual cortex and 2. topographic stimulus representations in higher visual cortex and also in motor cortex that are determined by temporal stimulus relations. Topographic structures of the latter type I denote time-organized maps. Topographic stimulus representations can be beneficial for cortical information processing (Section 2.5). Furthermore, they can result from simple learning processes (following chapters). Accordingly, the brain may be structured in the form of cortical maps wherever this form of stimulus representation is advantageous. I regard the latter issue as a strong argument to keep on searching for cortical maps even in higher cortical areas.

25 36 3 Learning cortical topography from natural binocular stimuli 3. Multi-feature representation of three-dimensional space 3 Learning cortical topography from natural binocular stimuli Information processing in the cortex is distributed over several areas. Moreover, each area seems to specialize in different parameters of the modality under consideration. A well-known example is the visual cortex that is concerned with early vision as well as integration of higher-level features, see Figure 2., 2.3a,b and 2.4. The processing of stimuli in the primary visual cortex is the subject of this chapter. Here, extensive biological experiments reveal a topographic representation of multiple stimulus parameters, for example, position, orientation, ocular dominance, and spatial frequency (Mallot, 985; Blasdel and Salama, 986; Obermayer et al., 99; Chino et al., 997; Kim et al., 999). Cortical maps for position and orientation were shown in the previous chapter (Figure 2.3). In this chapter, a computational model is presented that. extends the application of topography learning to natural binocular stimuli, 2. is compatible with observed multi-parameter representations, and 3. predicts the topographic representation and stimulus induced learning of further stimulus parameters, in particular the geometric representation of disparity on the level of hypercolumns. Section 3. briefly discusses aspects of multi-feature representations of three-dimensional space. These aspects motivate the model of self-organization formulated in Section 3.2. It yields numerical results that are presented in Section 3.3. The chapter is concluded by a summary and an outlook in Section 3.4. Neurons of the early visual cortex do typically not specialize on only one visual feature. Instead, they respond clearly to multiple stimulus parameters. Accordingly, a multitude of functional maps is found in the early visual cortex (Hübener et al., 997; Kim et al., 999). I denote this phenomenon multi-feature representation. In this chapter, multi-feature representations of three-dimensional space are generated. The generating process is based on the following concepts:. high-dimensional coding, 2. modular cortical architecture, 3. stereo geometry, and 4. self-organization. 3.. High-dimensional coding Stimulus induced self-organization based on Hebbian learning (Hebb, 949, Brown et al., 990; Cruikshank and Weinberger, 996) may play an essential role in the development of neural selectivities in the visual cortex (von der Malsburg, 973; Grossberg, 976; von der Malsburg, 993). The analysis of that role requires. to choose a model of self-organization and 2. to determine the encoding of stimuli and neural selectivities within the selected model. Concerning the encoding of stimuli, a differentiation between feature vector representation and high-dimensional stimulus representation has to be made. In feature vector representation stimuli and neural selectivities are coded as lowdimensional vectors. Each vector component represents independently one stimulus feature. For example, neurons in the primary visual cortex are often described by their selectivities concerning the two stimulus features orientation and ocular dominance (Obermayer et al., 99; Erwin et al., 995). In high-dimensional stimulus representation stimuli are activity patterns in a high-dimensional space, for example, images on discretized retina (Andres et al., 992; Riesenhuber et al., 996; Mayer et al., 998b; Wiemer et al., 998a). The corresponding components of neural weight vectors can be interpreted as effective strengths of connections from sets of retinal photo receptors to cortical neurons. The simulations of this chapter will be carried out in order to analyze the emergence of neural selectivities within cortical maps. Feature vector representation does not suit this purpose as it requires to determine a priori the type of features 35

26 3. Multi-feature representation of three-dimensional space Learning cortical topography from natural binocular stimuli Left Retina Right Retina In recent years increasing evidence has been accumulated that this topographic organization of the primary visual cortex extends to other parameters as well, for example, spatial frequency and direction of motion. These findings challenge the concept of modular architecture of the primary visual cortex leading to the question of how all these stimulus features can be combined in each module. Accordingly, the geometric relationships between different feature maps are intensively discussed in the literature (Obermayer et al., 992; Erwin et al., 995; Hübener et al., 997). left retinal area Cortical Modul right retinal area Figure 3.: Modular architecture of the primary visual cortex. A -2 mm block of the primary visual cortex of monkeys reads out retinal activities from the left and right eye. The two retinal areas correspond to a volume element in threedimensional space. that are neurally represented. In contrast, high-dimensional stimulus representation enables the self-organizing process to extract dominant visual features from incoming stimuli. Moreover, correlations between different stimulus parameters emerge within neural selectivities. With regard to multiple stimulus features, neural selectivities can be read out by the application of operators acting on highdimensional weight vectors that are attached to neurons Modular cortical architecture A modular functional architecture of the primary visual cortex of monkeys was first proposed by Hubel and Wiesel (Hubel and Wiesel, 977), see Figure 3.. Their analysis of topographic representation revealed that drift and scatter of receptive fields are roughly constant when measured in multiples of receptive field size. In their so-called ice-cube model a -2 mm block of visual cortex contains the complete machinery for the analysis of the corresponding part of the visual field. Orientation preference and ocular dominance were regarded as neural key selectivities Stereo geometry Under natural conditions of stimulation, three-dimensional objects are projected onto the two retinae. Due to stereo geometry, the left- and right-eyed images that are projected onto the two retinal areas of a visual module deviate from one another. What are typical correlations between the two projections? A simplified situation is sketched in Figure 3.2. A line oriented in three-dimensional space is projected onto two retinal planes. The resulting retinal images possess disparities: different left- and right-eyed horizontal shifts and orientations, see (Wiemer et al., 2000a) for their computation. The resulting orientation differences for vanishing horizontal shifts are shown in Figure 3.3. If uniformly distributed orientations of the line in three dimensions are assumed as natural conditions of stimulation, it can be deduced from the figure that differences in left- and righteyed orientation are typically of only a few degrees. Therefore, the corresponding neural representations should constitute similar orientation maps for the two eyes. This is observed in neurobiological experiments (Crair et al., 998). Furthermore, there should be non-zero differences in left- and right-eyed orientation maps. Differences between left- and right-eyed orientation maps may express the representation of three-dimensional space. They may not solely result from measurement errors as assumed in (Gödecke and Bonhoeffer, 996; Crair et al., 998) Self-organization In the following, representations of three-dimensional space are generated by self-organization. Here, self-organization is understood as stimulus induced learning: the formation of functional structure is determined by the applied set of

27 3. Multi-feature representation of three-dimensional space Learning cortical topography from natural binocular stimuli A Sketch of stereo geometry 50 o C c F φ. α orientation difference θ L θ R - 25 o θ = 5 ο θ = 0 ο θ = 25 ο θ = 40 ο θ = 55 ο θ = 70 ο θ = 85 ο 0 o 0 o 45 o 90 o O L left retina B Projection, left retina f b O R right retina C Projection, right retina f angle α Figure 3.3: Differences in left- and right-eyed orientations. A line, C, that passes through the fixation point, F, is parameterized by two angles, α, θ (confer Figure 3.2 for their definition), and projected onto the two retinae as illustrated in Figure 3.2. For fixed eye geometry (b x =6 cm and F b d =60 cm within reaching distance) the graph shows differences in left- and right-eyed retinal orientations as a function of the line s three-dimensional position. h L Θ L Figure 3.2: Projection of a line oriented in three dimensions. In (A) the figure corresponds to the plane through left-eyed focal point O L, right-eyed focal point O R, and fixation point F. The straight line C is shown with its orthogonal projection into the figure plane. The three-dimensional position of C is obtained by rotating this projection around the point F + so that it has an angle θ with the figure plane (Φ: convergence angle, b: base vector, f : focal length, : point of line C in figure plane relative to fixation point F, c: direction of line C projected in figure plane, α: angle between the projection of line C onto the figure plane and the left retina). In (B,C) the corresponding projections of C onto the retinae are shown (retinae approximated as planes, h l,h r : left- and right-eyed horizontal shift, θ l, θ r : left- and right-eyed orientation). h R Θ R stimuli in bottom-up direction. An additional supervisory level is not needed that guides the process of structure formation via feed-back circuits. I simulate self-organization at the spatial scale of a functional module of the primary visual cortex. Here, the approximation holds that all neurons are connected to the same retinal areas. Accordingly, a grid of neural units is considered in which all units are connected to the same two retinal areas, one in each eye. The strengths of these connections are determined by stimulus induced learning that is based on high-dimensionally coded, binocular stimuli. 3.2 The model Following the above discussion a detailed model is now presented. The approach differs from earlier versions (Andres et al., 992; Andres, 995; Andres, 996) in the effort to establish a close connection to biological experiments. Moreover, biologically plausible boundary conditions regarding the generation of natural binocular stimuli are taken into account. In Section 3.2. the algorithm is

28 3.2 The model Learning cortical topography from natural binocular stimuli initialize weights for t= to t f choose learning rate & width of neighborhood: draw stimulus: compute winning neuron: adapt weights: σ(t) α(t), s(t) (m,n) kl w (t) Figure 3.4: Sketch of the SOM algorithm. described that is used for the self-organizing process. In Section the orientation, ocular dominance, and disparity operators are defined that are used to analyze the resulting neural selectivities Self-organizing map (SOM) A piece of cortex is modeled by a two-dimensional grid of N N neural units ( neurons, N = 32). These units should be interpreted as groups of neighboring cortical neurons, not as single neurons. Retinotopy is assumed to be already established at the beginning of self-organization; the units of one module are connected to the same P P pixels from left and right eye providing a total of (2P) P connections (P = 6). The strengths of these connections form the neural selectivities and are specified by weight vectors w. The self-organizing map (SOM) algorithm is applied for the stimulus induced self-organization of neural selectivities (Kohonen, 997). The algorithm combines processes of lateral interaction and Hebbian learning in a simple geometric computation that compresses the data, preserves topology, and accounts for the probability of stimuli. The SOM algorithm is sketched in Figure 3.4. The incoming natural binocular stimuli are of the form s =(s ij ) i=,..,(2p); j=,..,p R (2P) P 0, (3.) where all vectors consist of positive components and are normalized to the same Euclidian norm s. The generation and normalization of stimuli is described in Section Each neuron (k,l), k,l =,..,N discrete grid coordinates, possesses a weight vector w kl =(w ij ) kl i=,..,(2p), j=,..,p R(2P) P 0 that is of the same dimension as the stimuli. The scalar product S(w,s)=w s = ij w ijs ij is used as a measure for the similarity of weight vector w and stimulus s. This similarity measure can be interpreted biologically as neural feedforward activation. As normalized stimuli are used and as weight vectors are shifted in the direction of stimuli during the learning process, maximal similarity as defined above corresponds to minimal Euclidian distance: s =, w gives s w 2 2 2w s. Therefore, the following process of self-organization may also be regarded as a process of vector quantization that aims at minimizing the average expected square of the quantization error (Kohonen, 997). At each discrete time step t = 0,, 2,... a stimulus s(t) is randomly chosen from the pool of stimuli (Section 3.3) and presented to the network. The best matching neuron (m,n) is determined according to S(w mn (t),s(t)) S(w kl (t),s(t)) for all k,l. (3.2) A Gaussian neighborhood function around the winner (m,n) h mn (k,l,t)=exp ( (k ) m)2 +(l n) 2 (3.3) 2σ(t) 2 is assumed to localize learning. This function may reflect a principle of continuous mapping for the formation of the spatial structure of cortical feature maps (Obermayer et al., 990). It may also be interpreted as an average population activity resulting from feedforward activation and lateral inhibition (Wilson and Cowan, 973; Amari, 977; Mallot et al., 990; Mallot and Giannakopoulos, 996). Regarding the SOM algorithm, Lo and Bavarian have shown that such a neighborhood function results in faster convergence than a neighborhood interaction set, where all neurons selected for weight adaptation are updated at the same rate (Lo and Bavarian, 99). After each stimulus presentation, weights are adapted according to the following learning rule w kl (t)=α(t)h mn (k,l,t) ( s(t) w kl (t) ) (3.4) (Kohonen, 997). Here, weight vectors are shifted towards stimuli. Interpreting the neighborhood function h mn as averaged neural activity, the learning rule can be regarded as normalized Hebbian learning resulting from pre-synaptic and post-synaptic activity (s and h mn, respectively) (Brown et al., 990; Montague

29 3.2 The model Learning cortical topography from natural binocular stimuli and Sejnowski, 994). The term proportional to w kl (t) expresses normalization, that is, it ensures the boundedness of neural weights. The duration of the learning process is set to t f = 0 6 ( f : final) in accordance with Kohonen s rule of thumb (Kohonen, 990). To obtain global order from randomly initialized weights, learning rate α and width of neighborhood σ decrease monotonously during learning, see Figure 3.5. Initially, the learning rate is set to a value close to (α i = 0.9) and the width σ to about half grid size (σ 0 = 6). After an initial linear decay (Kohonen, 990), convergence of the learning process is achieved by an exponential decline of the learning rate { αi α i α o t α(t)= o t : t t o (3.5) α o e t to τ : t > t o with τ =(t f t o )/ln(α o /α f ) and parameters α f = 0 4, α o = 0.2, and t o = t f /0. The width of neighborhood function σ determines the smoothness of the change of neural selectivities within the grid of neurons. This model parameter is adjusted to the scale of biological hypercolumns: the width converges to a value σ f so that the resulting variations in orientation preference correspond approximately to those within one hypercolumn, σ(t)= σ 0 + σ f (3.6) +t/t σ with σ f = σ(t )=2, t σ = Definition of feature operators High-dimensional binocular neural weight vectors will be the result of the just described self-organizing process (Figure 3.9). The selectivities of some neurons (e.g., their orientation preference) can already be detected visually. In order to systematically read out the feature preferences of the neurons, operators have to be defined acting on the neural selectivities. A weight vector w =(w L,w R ) t R P2 R P2 is composed of a left-eyed and a right-eyed part. Its analysis is carried out by feature operators F(w)=F(w L,w R ) that correspond to the way in which neural features are measured (and thus defined) in biological experiments. In the following, biologically plausible operators are suggested regarding monocular orientation, ocular dominance, and horizontal disparity. α i Learning Rate α (t) α o σ o t o t f 0 t o t f σ i Width of Neighborhood σ (t) Figure 3.5: Temporal course of learning rate and width of neighborhood function. Orientation operator In biological experiments bi-directional moving gratings of different orientations are monocularly presented to determine neural orientation preference (presentation of one orientation at a time with movements in the directions perpendicular to the chosen orientation; Blasdel and Salama, 986; Ts o et al., 990). Accordingly, oriented bars b φ,n of two pixel width are employed as parts of receptive fields comprising A φ,n pixels (orientation φ S φ = {0, π 4, π 2, 3π }, shift n =,...P, 4 Figure 3.6). These determine the neurons preferred monocular orientations in the following way. For a bar b φ,n the weights w kl ij are summed up and normalized, with i, j restricted to the pixels of the bar, yielding the unit (k,l) s bar specific activation. The maximal size-normalized bar specific activation is taken as the neuron s orientation specific response: ( ) ( ) a kl (φ)=max w kl ij min w kl ij. (3.7) n A φ,n n A φ,n ij φ,n ij φ,n The second term establishes a kl (φ) =0 for constant w kl ij, that is, homogeneous and isotropic receptive fields. It supports the robustness of the resulting operator (3.8). As in the analysis of biological data, the responses a kl (φ), φ S φ of each neuron (k,l) are vectorially summed in a way that orthogonal orientations subtract from

30 3.2 The model Learning cortical topography from natural binocular stimuli W L W R W L d W R d b π _ 4, b π _ 4, 3 b π _ 4, 5 Figure 3.6: Examples of oriented bars applied to determine neural orientation selectivity. one another, leading to the unit (k,l) s preferred orientation Φ kl = ( ( ) ( ) ) cos2φ 2 a kl (φ),, (3.8) sin2φ 0 ( v, v 2 ) denotes the angle between v, v 2. Ocular dominance operator φ Ocular dominance of a neuron (k,l) with weight vector w =(w L,w R ) t is defined by od kl ij = wr ij ij wl ij ij w [,+]. (3.9) ij An ocular dominance of (+) is assigned to monocular neurons, that is, those solely receiving input from left (right) eye. Binocular neurons that are driven with equal strengths by both eyes possess zero ocular dominance. Horizontal disparity operator In order to define the horizontal disparity preference of a neuron (k,l) with weight vector w =(w L,w R ) t, the correlation coefficient is calculated between 0.. d P-d.. P Figure 3.7: Sketch of shared regions of a neuron s receptive field used to determine its horizontal disparity selectivity. the left and right part of that neuron s weight vector in dependence on the horizontal shift d between the two parts. The comparison must be restricted to the shared region of width P d pixels, see Figure 3.7. Horizontal disparity tuning is defined by the correlation coefficient C kl (d)= (wl d w L d )(w R d w R d ) Var(wL d ), (3.0) Var(w R d ) with Var(w): variance of w, w : expectation of w, and ( ) w L d : w L d ij wl i, j+d, ( ) wr d : w R d ij wr i, j, (3.) for i =,..,P, j =,..,P d, and d 0 (analogously for d < 0). Horizontal disparity preference could now be defined as the average horizontal shift using correlation coefficients as weighting factors. However, this would average out some of the neurons disparity selectivity (Figure 3.2). On the other hand, tuning curves could be searched for their maximum. Such a measure would, however, be very sensitive to noise. This situation can be improved by using a mixture of both approaches: the neuron s horizontal disparity preference is defined as the average of the shift d using correlation coefficients as weighting factors and taking only high correlation values C kl (d) > K max d C kl (d) into account (K = 0.8 in the simulations). Horizontal disparity and orientation analysis are geometrically interrelated. Vertically oriented structures yield pronounced correlation differences depending on the horizontal shift d, while horizontal structures produce only slight differences

31 3.3 Numerical experiment Learning cortical topography from natural binocular stimuli in correlation. On account of the stereo-geometrical dominance of horizontal to vertical disparity, the analysis is restricted to the horizontal component of disparity. 3.3 Numerical experiment In the SOM algorithm neural units extract properties from presented stimuli. Therefore, the pool of stimuli used for the learning process plays an essential role. Section 3.3. describes the generation of natural binocular stimuli in detail. Section presents and discusses the results, obtained by applying the self-organizing process of Section 3.2. to these stimuli Natural binocular stimuli In the visual cortex stimulus induced learning of neural selectivities is based on two-dimensional retinal activity patterns that are formed by projecting threedimensional objects onto left and right retina. Accordingly, natural stimuli are generated by first creating a three-dimensional scene and then taking stereo pictures of it. Thereby, natural correlations between different stimulus features are obtained, for example, between orientation and disparity. Different stereo views of the model world are produced by variations of the fixation point. Typical binocular activity patterns seen by a visual module are assumed to correspond to small pictures taken to one half from the projection of the left camera and to the other half from the projection of the right camera. The left and right parts of one such stimulus are taken from corresponding areas of the two projections: the pictures are cut in ocular stripes of P pixel height, the stripes are aligned in alternating order, and out of the resulting fused projection binocular stimuli are cut (2P) P pixel in size (P = 6), see Figure 3.8. The frame of each such stimulus is chosen so that its upper part covers the full height of a left ocular stripe. Apart from this the positions of stimuli are evenly distributed inside the binocular representation. Further stimuli are generated by the application of two symmetry transformations in order to increase the variability of three-dimensional situations. The first one is denoted by S : it transforms a given binocular stimulus to another stimulus A Left-eyed View B Right-eyed View C Binocular Representation L (R) : left (right) ocular stripe L R L R L R L R L R L R D Pool of Stimuli Figure 3.8: Generation of natural binocular stimuli. Using a stereo camera system, a three-dimensional scene is projected onto two retinae (A,B) and fused into one binocular representation consisting of alternating ocular stripes of P pixel width (C). As an example, six stimuli of size (2P) P pixel are shown within the binocular representation. The frame of each such stimulus is chosen so that its upper part covers the full height of a left ocular stripe. Apart from this, the positions of stimuli are evenly distributed inside the binocular representation forming a pool of stimuli, exemplarily represented by six examples (D). that would have been obtained when looking at the three-dimensional scene in a mirror: S (s L ij,sr ij )=(sr i,p+ j,sl i,p+ j ), (3.2) for s =(s L ij,sr ij ) and i, j =,..,P. The second symmetry transformation S 2 corresponds to looking at an upside down version of the scene: S 2 (s L ij,s R ij)=(s L P+ i, j,s R P+ i, j). (3.3) New stimuli are generated by applying S, S 2, and S S 2. Stimulus pre-processing is kept to a minimum in order to preserve natural correlations between different stimulus features. However, some boundary conditions regarding stimulus statistics have to be met to achieve simulation results that are comparable with cortical topographic structures known from biological experiments. The process of cutting binocular stimuli out of fused stereo images yields some unnatural stimuli whose left- and right-eyed parts correspond to different threedimensional objects (e.g., some stimuli cut out of the bottom two ocular stripes in

32 3.3 Numerical experiment Learning cortical topography from natural binocular stimuli Figure 3.8C). Accordingly, those stimuli are excluded whose left- and right-eyed components correlate less than a fixed value, 0.3, for all horizontal shifts up to ±P/2. Preferred orientations are in essence evenly distributed in optical imaging data (Erwin et al., 995). Recent works on the distribution of orientation in natural scenes (Coppola et al., 998a) and in cortical representations indicate slight prevalence of vertical and horizontal orientations compared with oblique angles (Coppola et al., 998b; Chapman and Bonhoeffer, 998). The limited number of stereo views and objects forming the three-dimensional scene leads to an uneven distribution of preferred orientations in the initial set of stimuli. An approximately even orientation distribution is generated in our pool of stimuli by randomly choosing stimuli from an initial pool and removing them if their left-eyed or right-eyed orientation belongs to an orientation class that is more frequent than another. In addition, stimuli are normalized with respect to their intensities because of two reasons: first, weight vectors are shifted towards stimuli in the SOM algorithm. Without normalization, differences in the brightness of the used stimuli would be mapped onto the neural grid leading to groups of neurons that specialize on different stimulus brightness. This is not observed in biological neural systems. Second, pre-cortical processes adjust neural light sensitivity, thereby achieving a form of brightness normalization (Boff et al., 986). The stimuli are normalized as follows: to each stimulus an amount of DC-light is added so that all stimuli lie on the same hypersphere. This process does not affect the stimuli s orientation and horizontal disparity analysis. The resulting pool used for self-organization consists of elements with a distribution of orientation, ocular dominance, and horizontal disparity shown by histograms in Figure 3.4 (dashed lines) and Numerical results The application of the self-organizing process of Section 3..4 to the pool of stimuli generated according to Section 3.3. generates high-dimensional binocular neural selectivities. These reveal different specific shapes and asymmetries, many selectivities corresponding to edges (Figure 3.9). The neural weight vectors vary gradually along the two-dimensional grid of neurons. This expresses the tendency of the SOM algorithm to establish a neighborhood preserving map from the grid of neurons to the space of spatial stimulus encoding. Here, gradual variations of high-dimensional neural selectivities are the basis of gradual variations of features like orientation, ocular dominance, and horizontal disparity that are presented as feature maps (Figure 3.0 and 3.3). The monocular analysis of weight vectors regarding orientation yields left-eyed and right-eyed orientation maps, see Figure 3.0. As known from optical imaging of the early visual cortex, such maps reveal: (a) linear zones: regions of gradual variations of orientation in mainly one direction of the cortex, (b) fractures: regions of abrupt (or particularly fast) orientation changes, and (c) singularities: singular points adjacent to iso-orientation domains of all orientations (Erwin et al., 995). These characteristic elements can be found in the two orientation maps. They are indicated exemplarily by arrows, lines, and squares, respectively. The orientation singularities correspond to singularities of neural selectivity, that is, to neurons with about homogeneous weight vector components. Their small differences in weight vector components are, however, blown up to maximal contrast by the visualization in Figure 3.9. Typically, singularities of orientation and selectivity reveal the same orientation preference on opposite sides with complementary weight vectors, resulting in a 360 change of orientation on a closed path surrounding them (see Figure 3.9). This is in contrast to optical imaging data usually showing singularities with 80 change in orientation around them. These singularities are called pinwheels (Bonhoeffer and Grinvald, 99). The difference may be due to shifts of RFs in visual space (Das and Gilbert, 997) or due to pre-cortical stimulus processing based on neural onand off-responses. More detailed models of neural units should be applied in order to better simulate singularities. However, this work does not focus on the exact structure of singularities but on the interrelations between orientation and horizontal disparity preference and between left-eyed and right-eyed orientation maps. The similarity of left-eyed and right-eyed orientation maps and its development was studied by Crair et al. in the cat (Crair et al., 998). The described model can simulate such a developmental process, starting with randomly initialized weights. During self-organization, neural units develop stimulus induced correlations between left-eyed and right-eyed orientation preference leading to an increase in the similarity of monocular maps. If identical left-eyed and righteyed activity patterns are presented to the neural units in each learning step, that

33 3.3 Numerical experiment Learning cortical topography from natural binocular stimuli A Left-eyed Receptive Fields B Right-eyed Receptive Fields Figure 3.9: Maps of lefteyed and right-eyed neural selectivities. Using the SOM algorithm for natural binocular stimuli, highdimensional neural selectivities develop. They correspond to the left (A) and right eye (B). Each of the squares represents one monocular weight vector. They are scaled to maximal contrast for visualization. Singularities of neural selectivity are marked by frames consisting of 5 5 neurons, they correspond to orientation singularities (Figure 3.0). For the comparison of the left and right part of a neuron s selectivity confer Figure 3.3. is, if the projections of the three-dimensional environment onto both retinae are assumed to be the same, the left-eyed and right-eyed orientation maps become identical. However, taking the stereoscopic projection of stimuli into account, different monocular activity patterns lead to small stimulus induced differences A Orientation Map for Left Eye B Orientation Map for Right Eye Figure 3.0: Maps of monocular orientation. Orientation analysis of the highdimensional neural selectivities shown in Figure 3.9 reveals monocular orientation maps for left (A) and right eye (B). Both maps show two singularities with opposite spin indicated by frames of 5 5 neurons. In addition, two corresponding linear zones are exemplarily marked by dashed arrows and fractures (faster change of orientation) by dashed-dotted lines. The monocular maps are similar but not identical. of monocular orientation preference. Figure 3. compares numerical results based on stereoscopic projections with biological data taken from Crair et al., 998. Both developmental processes reveal an increase in similarity saturating to similar but not identical maps. Accordingly, differences between left- and righteyed orientation preference may be a consequence of stimulus induced learning. Moreover, the differences may even be used by the cortex for the representation of three-dimensional space. The post-natal development of cat visual cortex is dominated by contralateral neural responses suggesting contralateral receptive fields to serve as templates for the ipsilateral side (Crair et al., 998). In the above simulations this contralateral dominance was neglected. However, it can be incorporated by. initializing contralateral neural selectivities not randomly but as well-defined structures and 2. choosing a lower learning rate for contralateral connections reflecting less plasticity. In this case, self-organization leads to an adaptation of ipsilateral neural responses to their contralateral templates converging to similar but not identi-

34 3.3 Numerical experiment Learning cortical topography from natural binocular stimuli.0 A Horizontal Disparity Tuning around Left Singularity B Horizontal Disparity Tuning around Right Singularity Similarity : numerical simulation : biological experiment Time in postnatal days / learning steps 00 Figure 3.: Temporal development of similarity between monocular orientation maps. During the learning process map similarity corresponding to left-eyed and right-eyed orientation preference increases. Map similarity is defined as the correlation coefficient between two maps transformed to the interval [0, ] (: equality of maps, 0.5: no correlation). Stimulus induced self-organization based on natural binocular stimuli is compared with biological results from (Crair et al., 998, numerical simulations averaged over six runs). cal monocular selectivities; the difference again being the result of stereoscopic projection. Neural units develop selectivities for the binocular features horizontal disparity and ocular dominance in the applied model of self-organization. Figure 3.2 presents exemplarily horizontal disparity tuning curves for the two arrays of 5 5 neurons centered at the singularities and marked as squares in Figure 3.9 and 3.0. The geometric representation of binocular features is shown in Figure 3.3. Horizontal disparity and ocular dominance both vary at the considered spatial scale of one hypercolumn. The orientation singularities lie in opposite ocular dominance regions, the left (right) singularity being dominated by the left (right) eye. Different values of ocular dominance in neural selectivities correspond to ocular differences in the presented stimuli; they are due to slight differences in illumination and camera response. The use of symmetry transformation S to generate additional stimuli exchanges left- and right- eyed part resulting in stimuli with opposite ocular dominance. Accordingly, two singularities form representing mirrored geometric conditions and opposite ocular dominance. Neural Response Neural Response Neural Response Neural Response Neural Response Disparity Disparity Disparity Disparity Disparity Neural Response Neural Response Neural Response Neural Response Neural Response Disparity Disparity Disparity Disparity Disparity Figure 3.2: Horizontal disparity tuning curves around left (A) and right singularity (B). The correlation coefficient between shifted left and right part of a neuron s selectivity is taken as its disparity tuning (Section 3.2.2). An array of 5 5 neighboring neurons (k,l) is shown revealing the variation of disparity tuning in the neighborhood of the two orientation map singularities. Horizontal disparity can only be resolved for non-horizontal orientations. Accordingly, different horizontal disparities can be found in regions of vertical and oblique orientations in Figure 3.3. The maps reveal substructures in regions of constant orientation corresponding to different horizontal disparities. In particular this holds for the region connecting the two singularities. The finding points to topographic relations between orientation, ocular dominance, and disparity maps. In Figure 3.4 single feature histograms for orientation, horizontal disparity, and ocular dominance are presented. For monocular orientation preference and ocular dominance the frequency of neural representation corresponds approximately to stimulus frequency. Deviations occur as stochastic fluctuations and as stimuli possess different feature distinctiveness. An over-representation of small values around zero occurs for horizontal disparity. During the learning process neural selectivities average over different stimuli thereby forming coarse structures compared to single stimuli. This results in less pronounced horizontal disparity

35 3.3 Numerical experiment Learning cortical topography from natural binocular stimuli A Map of Binocular Receptive Fields B Binocular Orientation Map A Left-eyed Orientation B Right-eyed Orientation Frequency / [%] 0 Frequency / [%] 0 Orientation, left C Ocular Dominance Orientation, right D Horizontal Disparity C Ocular Dominance Map 0.3 D Horizontal Disparity Map 7.5 Frequency / [%] Frequency / [%] Ocular Dominance Horizontal Disparity Figure 3.3: Binocular feature representation. Left (upper) and right (lower) part of each neuron s selectivity (Figure 3.9) are fused into one representation (A). Weight vectors are compressed in vertical direction in order to retain the square shape of the neural grid. Analogously, monocular orientation maps (Figure 3.0) are fused to form a binocular orientation map (B). The application of binocular operators reveals maps for ocular dominance (C) and horizontal disparity (D). Positions of orientation singularities are marked by square frames in each map. Positive (negative) horizontal disparity is additionally indicated by horizontal light (dark) lines in (B) and (D). -4. Figure 3.4: Histograms for single features. Learned neural selectivities and the pool of stimuli are analyzed according to monocular orientation, horizontal disparity, and ocular dominance. The results are visualized in histogram form (solid line: neural representation, dashed line: pool of stimuli). The equal orientation distribution of stimuli was generated as described in Section Different values of ocular dominance reflect differences in illumination and camera response. tuning. Moreover, a tendency to strongly represent zero disparity follows from the definition of disparity preference: units with weak disparity preferences are counted as cells with zero disparity. Finally, two-dimensional histograms show the degree of correlation between single features in the pool of stimuli and in the neural representation (Figure 3.5 and 3.6, respectively). Left- and right-eyed orientation are strongly correlated in the pool of stimuli and in its neural representation. This reflects the geometry of