Perceptual Grouping in a Self-Organizing Map of Spiking Neurons Yoonsuck Choe Department of Computer Sciences The University of Texas at Austin August 13, 2001
Perceptual Grouping Group Two! Longest Contour? What is in here? Goal: To understand the Neural Mechanisms of Perceptual Grouping.
Perceptual Grouping ffl How are constituents grouped together? ffl The neural mechanisms are not well understood. ffl As a first step, focus on the Low-Level.
Low-Level Perceptual Grouping Retina Visual Cortex Why focus on low-level? ffl Constraints on feature dimensions. ffl Abundant neuroanatomical data. ffl Bridge to high-level cognitive processes. ffl Example: Contour Integration
Research Questions V1 Retina Visual Cortex 1. What are the neural mechanisms? 2. How does the circuitry emerge?
Possible Answers RETINA CORTEX t=1 t=2 t=3 t=4 Time Retina Cortex 1. Synchronized activity represents grouping. 2. Circuitry is self-organized during development. (Unsupervised learning)
Basics(1): Human Visual Pathway V1 Retina ffl Receptive Fields (RF). Visual Cortex ffl Hierarchy of maps. 1: Topological organization. 2: Laterally connected.
Basics(2): Primary Visual Cortex V1 Retina Visual Cortex Blasdel (1992) ffl Neurons are orientation-tuned. ffl Nearby neurons prefer similar orientation. ffl Forms orientation map.
Basics(3): Lateral Connections Bosking et al. (1997) ffl Black dots: Lateral Connections. Connect similarly orientation-tuned neurons. ffl (Gilbert and Wiesel, 1989)
Synchronization RF STRONG WEAK NO Synchronization Synchronization Synchronization Eckhorn et al. (1988); Gray et al. (1989)
Contour Association Field Association Field D. Field et al.(1993) ffl Smaller relative orientation is preferred. ffl Edges aligned on a common path are preferred. ffl Conjecture: lateral interaction is needed.
Local Grouping Function Geisler et al.(2001) ffl Edge co-occurrence in natural images! Local grouping function. ffl Transitive grouping rule. Predict human contour integration performance.
Self-Organization Altered visual environment drastically changes the organization of the visual cortex. Input deprivation: Hubel and Wiesel (1962,1974), ffl Issa (1999), White et al. (2000,2001) Biased input: Hirsch and Spinelli (1970), ffl Blakemore and van Sluyters (1975) Visual input to auditory cortex: Sur et al. (1988), ffl Sharma et al. (2000)
Computational Models 1. Synchronization 2. Self-Organization ffl Coupled Oscillators von der Malsburg&Buhmann(1992) Terman and Wang (1995) Chakravarthy,Ramamurti&Ghosh(1995) ffl Leaky integrator Eckhorn et al. (1990) Reitboeck et al. (1993) ffl Integrate and fire neurons Nischwitz and Glünder (1995) ffl Orientation map von der Malsburg (1973) Obermayer et al. (1990) ffl Ocular dominance Miller et al. (1989) ffl LISSOM: SOM with lateral conn. Miikkulainen,Bednar,Choe&Sirosh(1997) 3. Contour Integration Fixed, pre-calculated lateral interaction patterns. ffl Li (1998), Yen and Finkel (1997)
Limitations 1. Lack temporal representation. 2. Limited to simple grouping rules such as connectedness. 3. Fixed lateral interaction: ffl No self-organization. ffl Does not explain how such patterns emerge. Does not explain hemifield differences in ffl performance.
Approach RETINA CORTEX t=1 t=2 t=3 t=4 Time Retina Integration of two computational principles: Cortex 1. Temporal Coding ffl Spiking neurons. 2. Self-Organization ffl Oriented RFs and patterned lateral connections.
Architecture: PGLISSOM 1. Spiking Neurons. 2. Afferent connections form oriented RF. 3. Excitatory and inhibitory lateral connections. Retina Cortex 4. Two layers, constituting a cortical sheet. Self-Organization ffl Grouping ffl
s(t) = i + s(t 1)e Spiking Neuron θ base θ abs θ rel i Input Spike S(t) ω > ω Leaky Synapse ω Σ Weighted Sum θ (t) Dynamic Threshold i Output Spike ffl Leaky Synapse ffl Dynamic Threshold (Refractory Period Term) (t) = base + fi rel (t) + abs (t)
ffi = g 0 B B fl a @ X z } Afferent + flc X z } Intra Column + fle X z } Exc:Lat: fli X z } Inh:Lat: 1 C C A Activation ξ µ η E ν ζ I Retina Cortex Eim m Iim m μij οj ν ik k j m k m
ij(t) + ffvixj P w [wij(t) + ffvixj] j Weight Adaptation V i w ij X j Normalized Hebbian Learning: wij(t + 1) = ffl Increase weight proportional to the activities. ffl Normalize with sum of weights to limit growth.
Summary of Results 1. Self-organized orientation map. 2. Self-organized lateral connections. 3. Lateral connection statistics. 4. Contour integration. 5. Contour segmentation. 6. Contour completion. 7. Hemifield differences in contour integration.
Results(1): Orientation Map ffl Trained with elongated, oriented Gaussians as input. ffl Activate, settle, and adapt weights. ffl Smooth orientation map develops in both layers.
Results(2): Lateral Connections b c d (a) MAP2 (b) 85 o (c) 141 o (d) 174 o ffl MAP2 excitatory connections are shown. ffl Connects similarly orientation-tuned neurons. ffl Aligned along the axis of the source neuron s RF. ffl Synchronizes remote areas.
Results(3): Connection Statistics (I) RF Retina φ δ θ Measures Cortex Connections prefer ffl similar orientation. Number of Connections % of Total Boutons Matches data from ffl Bosking et al. (1997). 0 30 25 20 15 10 5 0 16 14 12 10 8 6 4 2 Median -80-60 -40-20 0 20 40 60 80 Diffrence in Orientation Tuning (deg) PGLISSOM Median -80-60 -40-20 0 20 40 60 80 Diffrence in Orientation Tuning (deg) Bosking (1997)
Results(3): Connection Statistics (II) δ φ θ φ =90 o Relative Probability 1 Reference 0.1 0.01 φ=0 o δ=27 PGLISSOM Geisler et al. (2001) ffl Connected RFs are aligned along a smooth path. ffl Matches edge co-occurrence statistics found in nature.
Connection Statistics(II): Zoomed φ =90 o Relative Probability 1 0.1 0.01 φ=0 o δ=27
Why Co-Circular? RFs aligned on a common straight input are ffl stimulated. ffl Non-optimally aligned input can also cause activity. ffl Forms the basis for contour integration.
o 0 30 o 60 70 Results(4): Contour Integration o o Cortex Model: Correlation 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Model vs. Human Data Model Data -20 0 20 40 60 80 100 Orientation Jitter (degrees) Performance measure: ffl Correl. coeff. between Multi Unit Activities (MUAs). ffl As orientation jitter increases: correlation decreases (PGLISSOM). ffi accuracy decreases in humans(geisler et al. 2001). ffi 100 90 80 70 60 Data: Accuracy(%)
Demo Contour Integration
Results(5): Contour Segmentation Correlation Coefficient 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Segmentation of 3 Contours Correl Within Across ffl Each contour is grouped by synchronized activity. Separate contours are segmented by ffl desynchronized activity.
Demo Contour Segmentation
Results(6): Contour Completion (I) ffl Illusory contours : Kanizsa square. ffl Edge-inducers around the border. ffl Contour completion may be a low-level mechanism.
Demo Contour Completion
Results(6): Contour Completion (II) 1 2 3 4 5 Afferent Input 2.5 2 1.5 1 0.5 1 2 Afferent Contribution 3 0 0 10 20 30 40 50 Location (Column) Correlation Coefficient 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 4 5 0-0.1 Contour Completion Both Exc Aff ffl Gap region receives small amount of afferent input. ffl However, completion is not due to afferent input alone. ffl Completion is not due to excitatory input alone either.
Hemifield Differences 00000000 11111111 00000000 11111111 000000000000 111111111111 000000000000 111111111111 000000000000 111111111111 000000000000 111111111111 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 000000000000000 111111111111111 000000000000000 111111111111111 000000000000000 111111111111111 000000000000000 111111111111111 000000000000000 111111111111111 000000000000000 111111111111111 00000000000000 11111111111111 000000000000000 111111111111111 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 00000000000000 11111111111111 000000000000 111111111111 000000000000 111111111111 0000000000 1111111111 0000000000 1111111111 00000000 11111111 00000000 11111111 Contour and illusory contour detection performance differ: Fovea > Periphery (Hess and Dakin 1997) ffl ffl Lower > Upper Visual Hemifield (Rubin et al. 1996)
Results(7): Hemifield Differences(I) φ =90 o Relative Probability 1 φ =90 o Relative Probability 1 0.1 0.01 φ=0 o 0.1 0.01 φ=0 o δ=27 Lower Hemifield δ=27 Upper Hemifield ffl Input presentation frequency differed in the hemifields. ffl Resulting excitatory lateral connections are: 1. more co-circular in the lower hemifield, and 2. more collinear in the upper hemifield.
Hemifield Differences(I): Zoomed Lower Upper
Results(7): Hemifield Differences(II) Correlation 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Contour integration: Lower vs. Upper Hemifield Lower Upper -10 0 10 20 30 40 50 Orientation Jitter (degrees) ffl Performance is higher in lower hemifield. Performance gap is larger for task requiring ffl co-circular lateral interactions.
Summary(1) What has been shown? 1. Synchrony in model accounts for human performance. 2. Synchrony is established through lateral connections. 3. The lateral connections are a result of self-organization. 4. Changes in input cause difference in structure and performance.
Summary(2): Predictions 1. Correlation of MUA sequences can represent perceptual grouping. 2. V1 mechanisms can account for edge-induced illusory contours. 3. Layered architecture in V1 may be due to different functional requirements: (1) self-organization and (2) grouping. 4. Input difference can cause structural changes, and result in altered performance. 5. Straight inputs can cause co-circular lateral interaction properties to emerge.
1. Neuroscience: Future Work Verify functional connection statistics. ffl Connections before, during, and after development. ffl Effect of disrupted neural synchrony on perception. ffl 2. Psychophysics: ffl Extend the model further for full stimulus dimensions. ffl Role of higher areas on task performance. 3. Intelligent Systems: Application to real-world images. ffl Higher module for activity interpretation. ffl Multi-modal integration. ffl
Conclusion ffl Model based on synchronization and self-organization. Accounts for: ffl 1. structural formation (development) and 2. functional mechanisms in contour integration tasks. Contributes to: ffl 1. understanding the neural mechanisms of P.G. 2. laying a foundation for artificial vision systems.
Extra Slides
T (t) = e t=r INF vs. Dynamic Threshold x(t): Voltage T(t): Threshold Fire! Fire! Fire! Fire! Integrate and Fire Dynamic Threshold dx = x + I R dt vs. x(t) = IR(1 e t=r )
Correlation in MUAs Synchronized Desynchronized MUA MUA MUA MUA time time Correlation coefficient between MUAs X and Y : = Y ) r = Cov(X; ff Y X ff pp (X i μ X )(Y i μ Y ) i ppi i μ X ) 2p Pi (Y i μ Y ) 2 (X
Human Contour Integration Geisler et al. (2001)
Overlapping RF Retina Cortex Neurons responding to mixed inputs are not ffl measured.
Network of Spiking Neurons Summary of spiking neuron behavior in a network: ffl Excitation with fast decay causes synchrony. ffl Inhibition with fast decay causes desynchrony. ffl Noise helps desynchrony. Refractory period helps overcome high levels of ffl noise.