Towards Cortex Sized Artificial Nervous Systems

Transcription

1 Towards Cortex Sized Artificial Nervous Systems Christopher Johansson and Anders Lansner Department of Numerical Analysis and Computer Science, Royal Institute of Technology, Stockholm, Sweden, Fax: {cjo, Abstract. We characterize the size and complexity of the mammalian cortices of human, macaque, cat, rat, and mouse. We map the cortical structure onto a Bayesian confidence propagating neural network (BCPNN). An architectural structure for the implementation of the BCPNN based on hypercolumnar modules is suggested. The bandwidth, memory, and computational demands for real-time operation of the system are calculated and simulated. It is concluded that the limiting factor is the computational and not the communication requirements. 1 Introduction Our vision is that we one day will be able to emulate an artificial neural network (ANN) with a size and complexity comparable to that of the human cortex. The naïve approach to this grand task is to build a biophysically detailed model of every neuron in the human cortex. But doing this is not feasible within the foreseeable future even in a very long-term perspective. We will assume here that the functional principles of cortex reside on a much higher level of abstraction than that of the single neuron i.e. closer to abstractions like ANN and connectionist models. The target computational structure in our vision is not a super-computer that fills an entire room but a compact and low-power device no larger than the human brain. Having such an artificial brain as the core of an artificial nervous system (ANS) would take us closer to the goal of building truly intelligent robots and electronic agents. Though we foresee a steady progress towards this goal, given the complexity of the human brain we expect its materialization to lie at least a couple of decades ahead. Here we focus on the large-scale properties of a system comparable to the mammalian cortex. We do not consider, for instance, the internal representation or how the data is processed (feature extraction, temporal aspects etc). We are primarily interested in the complexity in terms of memory, computations, and communication required to emulate such a system.

2 2 Christopher Johansson and Anders Lansner 2 A Top-down View of the Mammalian Cortex The mammalian cortex is a structure composed of a vast number of repeated units, neurons and synapses, with largely similar functional properties though with several differences between brain areas, individuals, and species with regard to e.g. number and types of neurons and their anatomical and functional arrangement as well as connectivity. A long-standing hypothesis is that the computations in cortex are performed based on associations between units, i.e. cortex implements some form of associative memory. This function is supported by local and long-range connectivity displaying different forms of synaptic plasticity and learning rules. A substantial part of the cortical connectivity is recurrent, within areas as well as between them. It is reasonable to assume that this network is symmetrically connected, at least in some average functional sense [1], which makes it plausible to assume that the cortex to a first approximation operates as a fix-point attractor memory [2-4]. Recently additional support for the relevance of attractor states has been obtained from cortical slices [5, 6]. It has been suggested that about 10 ms is an appropriate timescale when one simulates ANN models of real neuronal networks [3]. The probable timescale for attractor dynamics, i.e. convergence to a fix-point, is ranging from ms [2]. Hebbian modifications (synaptic plasticity) occur on timescales of ms and longer [7]. This idea was captured already in the early work of Donald Hebb and Fridrich von Hayek (see e.g. [8] for a review). A prototypical attractor memory like the recurrent Hopfield network can be seen as a mathematical instantiation of Hebb s conceptual theory of cell assemblies. In a general sense, in the following we thus regard the cortex as a huge biological attractor memory system. This top-down view has abundant support in experimental observations and helps to define the problem of modeling and implementing a system of such a high dimensionality and complexity as the mammalian cortex. 2.1 Neuron and Synapse Numbers in the Mammalian Cerebral Cortex The cortex is generally quite homogenous [9] and has seen a great increase in size during evolution. Pyramidal neurons with far stretching axons and locally highly connected interneurons constitute its two major cell types. Approximately 75-80% of the neurons are of the pyramidal type and the remaining 20-25% are mainly inhibitory interneurons [10]. In humans the cortex is about 3 mm thick [11] and in mouse about 0.8 mm [10]. An interesting property that applies to all mammals is a constant neuron density of about 10 5 mm -2 [9, 10] except for V1 in primates [9, 12]. The average number of synapses per neuron found in different areas and in different species varies about an order of magnitude, i.e. between and [10]. The average number of synapses per neuron in mouse [10], cat [13], and man [14] is about The neuron volume density is lower in humans than in mouse (constant area density and an increased cortical thickness). This means that there is more space for human neurons and thus they can also be larger and have more synapses than neurons in mouse. An other reason for the lower neuron density in humans is the need for a more extensive intracortical wiring [9].

3 Towards Cortex Sized Artificial Nervous Systems 3 Cortex of all mammals is organized in layers. There are six layers in neocortex, where layer I is the most superficial one. Layer I is very sparsely populated by neurons and a lot of the neuropil contains dendritic arborization of neurons in the layers below and incoming afferent fibers. The distribution of neurons in the different layers is approximately [15]: 35% in layer II/III; 20% in layer IV; 40% in layer V/VI. Input to cortex, from the thalamic region enters layer IV. Corticocortical afferents are received in all layers. The neurons in layer IV provide input to layer II/III that connects laterally (intracortical horizontal connections), and also project to layer V/VI as well as to inhibitory neurons in layer IV. The large pyramidals in layer III and in layer V/VI have lateral connections as well as long-range corticocortical connections via the white matter. Layer V/VI pyramidals further project to inhibitory neurons in layer II/III. The cortical output is provided by efferents originating in layer V/VI. [16] A minicolumn is a vertical volume in cortex with some 100 neurons [17, 18] that stretches through all layers of the cortex. The minicolumn is both a functional and anatomical unit. Each minicolumn has afferent input, efferent output, and intracortical circuitry. Within each minicolumn there are both excitatory pyramidal neurons and inhibitory interneurons. The horizontal diameter of a minicolumn varies slightly between different cortical areas and mammalian species. It typically has a diameter of about 50 µm and an inner core with a diameter of approximately 30 µm where the neuron density is high [19]. Although there exists some differences between minicolumns located in different parts of the cortex (such as the exact size and structure and active neurotransmitters) and different species, it seems as though the minicolumn represents a general building block of the cerebral cortex. Another modular structure seen in the cortex of mammals is the hypercolumn. Hypercolumns are sometimes referred to as macrocolumns and also ocular dominance columns when found in the visual cortex. Mountcastle [20] together with Hubel and Wiesel [21] pioneered the study of the hypercolumns of cat s and macaque monkey s cortex. This structure, in the visual cortex, was named a hypercolumn by Hubel and Wiesel and we will use this term here. A hypercolumn contains a number of minicolumns and its size ranges between µm [17, 18, 21, 22]. Hubel and Wiesel showed by electrophysiological experiments in primates that the hypercolumn can function as a competitive, winner-take-all (WTA), circuitry for line orientations [21]. In Table 1 the number of neurons, synapses, and cortical area is listed for five different mammals [23]. Table 1. The cortex data is summarized for a number of mammals [23]. All types of neurons are included in the counts. Human Macaque Cat Rat Mouse Cortex Area (mm 2 ) Neurons Synapses If we assume that the average minicolumn is composed of 100 neurons and as we know the total number of neurons, we can calculate the number of minicolumns (Table 2). The area density of neurons is roughly constant, 10 5 mm -2, and therefore the average minicolumn diameter is about 36 µm. This diameter fits the figures found

4 4 Christopher Johansson and Anders Lansner in the literature. Based on the literature we assume that the hypercolumn is typically a circular structure with a diameter of about 400 µm. In this area it is possible to fit about 100 minicolumns with a diameter of 36 µm. Table 2. The number of mini- and hypercolumns in cortex calculated for a number of mammals. Human Macaque Cat Rat Mouse Minicolumns Hypercolumns An Abstract Model of Hyper- and Minicolumns In Fig. 1 we show schematically how the minicolumns are grouped into hypercolumns. The minicolumns are arranged around a pool of inhibitory basket neurons that provides inhibition for all types of neurons in the hypercolumn [24]. The pyramidal neurons in a minicolumn have excitatory connections to the basket neurons and they receive inhibitory input from these. The reciprocal connection between all minicolumns and the basket neurons provides a normalizing soft WTA circuitry within the hypercolumn [24]. Inside each minicolumn there is a pool of small inhibitory interneurons. These have a highly localized axonal arborization and provide inhibitory input to the local pyramidal neurons. Corticocortical connections exist between minicolumns in different hypercolumns. If an incoming axon terminates on pyramidal neurons in the targeted minicolumn, the connection is excitatory, whereas if it terminates on inhibitory interneurons the connection is inhibitory. hypercolumn minicolumn basket neurons inhibitory interneurons inhibitory connection excitatory connection cortical excitatory connection cortical inhibitory connection Fig. 1. Two hypercolumns and their internal structure. An interesting question is how many of the synapses in a minicolumn that originate from corticocortical connections. We know that some 75% of all neurons are pyramidal neurons, which are the main source of corticocortical connections. All neurons have an approximately equal number of synapses, and if we assume that all of the synapses on the pyramidal neurons are devoted to corticocortical connections we have that the number of synapses devoted to corticocortical connections in each minicolumn is around If we assume that horizontal projections in layer II/III stretches out some 3.5 mm, then a minicolumn is able to contact about other minicolumns by intracortical connections. Here we assume that the neurons in layer II/III are mainly devoted to these horizontal connections, which mean that there are approximately synapses supporting these connections. The average number of synapses supporting a connection between two minicolumns is thus around 5. This figure is in the following

5 Towards Cortex Sized Artificial Nervous Systems 5 taken to generalize to all corticocortical connections originating from a minicolumn. This means that ~33% ( ) of all corticocortical connections are intracortical and the remaining ~67% are intercortical connections that go through the white matter. An interesting property of this connectivity structure is its resemblance to a scale-free network [25]. The total number of corticocortical connections for a number of mammals together with the average connectivity between minicolumns is listed in Table 3. Table 3. The number of connections and connectivity in cortex. The connections are between minicolumns and the level of connectivity is computed as the fraction of full connectivity between the minicolumns. Human Macaque Cat Rat Mouse Corticocortical Connections Connectivity The BCPNN Model As stated earlier we propose that the cortex can be modeled with a multi-network of ANNs. The basic functionality of this network is imagined to be similar to that of a large attractor memory. In order to simplify the model and its implementation further we will analyze a single cortex-sized recurrent attractor network of a specific type, a Bayesian Confidence Propagation Neural Network (BCPNN) [26]. We aim for realtime operation in the sense of one attractor convergence, i.e. some hundred network updates, in two hundred milliseconds, Each cortical minicolumn is mapped to a unit in the BCPNN model. As in cortex, the units in the BCPNN are organized into hypercolumns of 100 units. Within each hypercolumn the units compete to be active. In each unit, the product of weights and other units activity is summed. In each connection, the weight value is continuously updated. The computational requirements of a large-size network model are determined by the computational requirements of its connections. In our BCPNN model, the state of a connection is represented by two state variables and determined by two differential equations. In a fixed-point arithmetic implementation, each equation requires on average slightly more than one operation and one table look-up [27]. Each state variable in the fixed-point implementation requires 8 bits, provided that random numbers are available so that fractional bits can be used [27]. In each update of the BCPNN, a unit in each hypercolumn (the one that won the competition for activation) transmits its state to the rest of the network. To avoid the communication bottleneck and achieve a scalable neural system we can use AER (Address Event Representation) [28, 29] in the inter-unit communication. AER means that the activity in the ANN is represented as discrete spikes and the only information that needs to be transmitted over the computer network is the address or an identification of the spiking units. Two levels of organization are recognized in the BCPNN, hypercolumns and units. A third level could be a population corresponding to a cortical area. But for our purpose here we do not need to distinguish this. The total memory required by a unit is 0.24 Mbytes and a hypercolumn requires 24 Mbytes. The number of operations per second (OPS) required by a unit to operate in real-time is and for a

6 6 Christopher Johansson and Anders Lansner hypercolumn We argue that the hypercolumn is the appropriate module for parallelism in a hardware or cluster computer implementation. 3.2 Implementation and Simulation The bandwidth required for communication between the hypercolumns is calculated for a network with a flat topology, e.g. Ethernet. For applications that require a high communication performance over Ethernet, UDP is commonly used. UDP based communication does not guarantee that messages arrive and there is neither any control of when messages arrive or in what order they arrive. It has been shown [30] that these types of communication errors well tolerated by the BCPNN and do not affect its operation. The benefit of UDP is a very small communication overhead. The bandwidth (Table 4) is calculated with eq. (1), where M is the amount of data generated by a node during a single update, k is the size of a message-header, C is the total number of nodes, and f is the frequency of updates. BUDP = C( M / C+ k) f (1) The scaling of UDP based communication was investigated on a 64 nodes cluster computer [31] (Fig. 2). These nodes are connected with a 100 Mbits Ethernet network, and each node is equipped with an Athlon XP processor running at 1400 MHz and 768 Mbytes of memory. The UDP based communication was implemented in Java and it was compared to TCP based communication implemented in Java and communication based on an MPICH program implemented in C. The Java implementations had much longer communication latencies than the MPICH-C implementation. The parameters for eq. (1) are: k=12 bytes and M=100 8/C bytes, where C {2,..,64}. Updates / s MPICH TCP UDP Bandwidth usages efficiency (%) MPICH TCP UDP Nodes Nodes Fig. 2. (Left) The number of completed activity updates per second for three different communication implementations. (Right) Percentage of the peak bandwidth utilized by the three different implementations. The MPICH implementation utilizes the bandwidth five times more efficiently than the UDP implementation, but the UDP implementation achieves a seven times higher update frequency. No more than 1% of the UDP updates were incomplete. 100 hypercolumns were simulated; each hypercolumn generated 8 bytes of data on each update. A message s header contained 12 bytes. The scaling was studied from 2 to 64 nodes. As a concrete example, the peak performance of a 90 nodes Itanium II clustercomputer is 648 GFLOP. The memory available in this cluster is 540 Gbytes. The nodes are connected with a 2 Gbit s -1 network. Assuming we could use these

7 Towards Cortex Sized Artificial Nervous Systems 7 resources optimally, we have no bandwidth limitation, up to R-size systems fit into memory, but we are at least an order of magnitude from running the Mo-sized system in real-time. Table 4. The bandwidth, memory, and operations per second (OPS) required for real-time operation. Parameters used in eq. (1); C=hypercolumns, M=32, k=12, and f=100. H-sized Ma-sized C-sized R-sized Mo-sized Bandwidth (Mbit s -1 ) Memory (Gbytes) OPS (10 9 ) Discussion Taking the view of the cortical minicolumn as the basic functional unit in cortex of mammals has allowed us to map cortical networks onto a BCPNN in a biologically reasonable way. We suggest that parallelization of the BCPNN on the hypercolumn level is appropriate for an implementation on either hardware or on today s standard cluster architectures. The hardware implementation requires large amounts of random numbers, which could be a potential problem for efficient realization. A conclusion from our estimations is that the limiting factor is the computational demands. It is therefore necessary to make a hardware implementation of the hypercolumn processing element in order to fulfill the vision of simulating an ANS the size of the human cortex. Special purpose hardware with a performance available in twenty years time provided that Moore s law continues to hold, will be required to reach sufficient performance. The robustness against noise in the computation will allow implementation using very small dimension stochastic computing elements. References 1. Lansner, A., E. Fransén, and A. Sandberg, Cell assembly dynamics in detailed and abstract attractor models of cortical associative memory. Theory in Biosciences, : p Fransén, E. and A. Lansner, A model of cortical associative memory based on a horizontal network of connected columns. Network: Comp. in Neural Systems, (2): p Rolls, E.T. and A. Treves, Neural Networks and Brain Function. 1998, New York: Oxford University Press. 4. Palm, G., Neural Assemblies: An Alternative Approach to Artificial Intelligence. Studies of Brain Function, ed. V. Braitenberg. Vol , New York: Springer-Verlag. 5. Cossart, R., D. Aronov, and R. Yuste, Attractor dynamics of network UP states in the neocortex. Nature, (6937): p Shu, Y., A. Hasenstaub, and D.A. Mccormick, Turning on and off recurrent balanced cortical activity. Nature, (6937): p Koch, C., Biophysics of Computation: Information Processing in Single Neurons, ed. M. Stryker. 1999, New York: Oxford University Press.

8 8 Christopher Johansson and Anders Lansner 8. Fuster, J.M., Memory in the Cerebral Cortex Rockel, A.J., R.W. Hiorns, and T.P.S. Powell, The Basic Uniformity in Structure of the Neocortex. Brain, : p Braitenberg, V. and A. Schuz, Cortex: Statistics and Geometry of Neuronal Connectivity. 1998, New York: Springer Verlag. 11. Hofman, M.A., Size and Shape of the Cerebral Cortex in Mammals: I. The Cortical Surface. Brain Behav. Evol., : p Beaulieu, C., et al., Quantitative Distribution of GABA-immunopositive and - immunonegative Neurons and Synapses in the Monkey Striate Cortex (Area 17). Cerebral Cortex, (4): p Beaulieu, C. and M. Colonnier, Number and Size of Neurons and Synapses in the Motor Cortex of Cats Raised in Different Environmental Complexities. J. Comp. Neurol., : p Pakkenberg, B., et al., Aging and the human neocortex. Exper. Gerontol., : p Dombrowski, S., C. Hilgetag, and H. Barbas, Quantitative Architecture Distinguishes Prefrontal Cortical Systems in the Rhesus Monkey. Cer. Cortex, (10): p Thomson, A.M. and A.P. Bannister, Interlaminar Connections in the Neocortex. Cerebral Cortex, (1): p Mountcastle, V.B., The columnar organization of the neocortex. Brain, : p Buxhoeveden, D.P. and M.F. Casanova, The minicolumn hypothesis in neuroscience. Brain, (5): p Buxhoeveden, D., et al., Quantitative analysis of cell columns in the cerebral cortex. Journal of neuroscience methods, : p Mountcastle, V.B., Modality and Topographic Properties of Single Neurons of Cat's Somatic Sensory Cortex. Journal of neurophysiology, : p Hubel, D.H. and T.N. Wiesel, Functional architecture of macaque monkey visual cortex. Proc. R. Soc. Lond. B., : p Leise, E.M., Modular construction of nervous systems: a basic principle of design for invertebrates and vertebrates. Brain Research Reviews, : p Johansson, C. and A. Lansner. Towards Cortex Sized Attractor ANN. in Bio-ADIT Lausanne, Switzerland. 24. Cürüklü, B. and A. Lansner. An Abstract Model of a Cortical Hypercolumn. in In Proc. of the 9th International Conference on Neural Information Processing Singapore: IEEE. 25. Barabási, A.-L. and R. Albert, Emergence of Scaling in Random Networks. Science, (5439): p Sandberg, A., et al., A Bayesian attractor network with incremental learning. Network: Computation in Neural Systems, (2): p Johansson, C. and A. Lansner, BCPNN Implemented with Fixed-Point Arithmetic, TRITA- NA-P , Nada, KTH: Stockholm. 28. Bailey, J. and D. Hammerstrom. Why VLSI Implementations of Associative VLCNs Require Connection Multiplexing. in Proc. of Internat. Conf. on Neural Networks San Diego. 29. Deiss, S.R., R.J. Douglas, and A.M. Whatley, A Pulse-Coded Communication Infrastructure for Neuromorphic Systems, in Pulsed Neural Networks, W. Maass and C.M. Bishop, Editors. 1999, MIT Press. p Jonsson, J., Pilot Study of a Parallel Implementation of a Bayesian Confidence Propagating Neural Network, TRITA-NA-E , Nada: Stockholm. 31. Johansson, C. and A. Lansner, Mapping of the BCPNN onto Cluster Computers, TRITA-NA- P , Nada, KTH: Stockholm.