Neurobiological Models of Visual Attention

Transcription

1 Neurobiological Models of Visual Attention John K. Tsotsos Dept. of Computer Science and Centre for Vision Research York University Theories/Models J.K.Tsotsos 2 Müller (1873) Exner (1894) Wundt (1902) Pillsbury (1908) Broadbent 1958 (Early Selection) Deutsch, Deutsch & Norman 1963/68 (Late Selection) The number of models that address Treisman 1964 the neurobiology of visual attention Milner 1974 * is small (*( in the list). The number Grossberg (Adaptive Resonance Theory) * that Treisman & Gelade 1980 (Feature Integration Theory) have real computational tests von der Malsburg (Correlation Theory) * on actual images is even smaller ( ( Crick 1984 * in the list). However, many relevant Koch and Ullman 1985 ideas have appeared in Anderson and Van Essen 1987 (Shifter Circuits) * psychological models. Sandon 1989 Wolfe et al (Guided Search 1.0, ) A selected historical perspective on Phaf, Van der Heijden, Hudson 1990 (SLAM) the ideas important to the modelling Tsotsos et al (Selective Tuning) * Mozer 1991 (MORSEL) task appears in the following slides. Ahmad 1991 (VISIT) * Olshausen, Anderson & Van Essen 1993 * Niebur, Koch et al * Desimone & Duncan 1995 (Biased Competition) * Postma 1995 (SCAN) * Schneider 1995 (VAM) * LaBerge 1995 * Itti & Koch 1998 Cave et al (FeatureGate) 1

2 Issues J.K.Tsotsos 3 Models of visual attention need to include solutions to or exhibit observed neurobiological/psychophysical performance for: computational complexity of visual processes information routing through the processing hierarchy attentional control time course of attentive modulation single cell attentive modulation attentive modulation in (apparently) all visual areas suppressive surround effects serial/ parallel visual search performance binding of features to objects Format of Overview J.K.Tsotsos 4 Not all models are included, only those that have historical importance or that claim neuro-psycho relevance Due to space and time limits, each model is described only with: 1. key references 2. key ideas 3. neurobiological relationship (where possible) ( has supporting evidence X does not have supporting evidence? open question) Note that this can only be regarded as a partial review! 2

3 Inhibition or Enhancement? The nature of the attentional influence has been debated for over a century: J.K.Tsotsos 5 Müller (1873) Exner (1894) Wundt (1902) Pillsbury (1908) re-inforcement inhibition and re-inforcement inhibition attention is in dis-array Broadbent 1958 Early Selection Model Broadbent, D. (1958). Perception and communication,, Pergamon Press, NY. - short term store acts to extend duration of stimulus - stimuli could be partitioned into channels (modalities) - selective filter selects among channels - limited capacity channel processes selected channel X (split-span( experiments of Deutsch & Deutsch ) J.K.Tsotsos 6 3

4 Deutsch/Norman Model 1968 Late Selection Model J.K.Tsotsos 7 Deutsch, J., Deutsch, D. (1963). Attention: Some theoretical considerations, Psych. Review 70,, Norman, D. (1968). Toward a theory of memory and attention, Psych. Review 75,, Late Selection Model - all information is recognized before it receives the attention of a limited capacity processor X (shadowing( with target word tapping, Treisman) - recognition can occur in parallel - stimulus relevance determines what is attended Summary of early ideas on attention Treisman 1964 Treisman, A. (1964). the effect of irrelevant material on the efficiency of selective listening, American J. Psychology J.K.Tsotsos 8 - filter attenuates (is not binary) unattended signals casing them to be incompletely analyzed - filter can operate at different levels - signal or meaning - so attention is hierarchical (Kastner et al. 1998) 4

5 Milner 1974 Milner, P. (1974). A model for visual shape recognition, Psych. Rev. 81,, J.K.Tsotsos 9 - unity of a figure at the neuronal level defined by synchronized firing activity? - attention acts in two ways: to select relevant figure from among others to activate the feedback pathways (Felleman & Van Essen 1991) from the cell assembly to the early visual cortex for precise localization - feedback pathways communicate attentional instructions? Grossberg Adaptive Resonance Theory J.K.Tsotsos 10 Grossberg, S., Carpenter, G., et al. (1998). The what-and-where filter: a spatial mapping neural network for object recognition and image understanding, Computer Vision and Image Understanding 69(1): Grossberg, S. (1998). How does the cerebral cortex work? Learning, attention and grouping by the laminar circuits of visual cortex,technical Report CAS/CNS ART algorithms are clustering algorithms that obey the following: bottom-up activation can drive a cell if strong enough top-down priming can modulate a cell cell becomes active if it receives large enough top-down and bottom-up activation top-down activation, even small, can negate bottom-up activation feedback leads to resonance and convergence - top-down attentional mechanisms should occur in every cortical area where learning can occur? - specific circuitry for interactions (slide 12)? 5

6 J.K.Tsotsos 11 ART rule may be realized by a top- down on-center off-surround network J.K.Tsotsos 12 Top-down, bottom-up and horizontal interactions in LGN, V1 and V2 based on ART Rule. Green - preattentive excitatory mechanisms Red - inhibitory mechanisms Blue - top-down attentional mechanisms 6

7 Treisman & Gelade 1980 Feature Integration Theory Treisman, A., Gelade, G. (1980). A feature integration theory of attention, Cognitive Psychology 12: J.K.Tsotsos 13 - master map of locations - attentional spotlight X Predictions: - popout in visual search without attention - conjunction search requires attention X X (Wolfe, Nakayama) von der Malsburg 1981 Correlation Brain Theory J.K.Tsotsos 14 von der Malsburg, C. (1981). The correlation theory of brain function, Internal Rpt. 81-2, Dept. of Neurobiology, Max-Planck-Institute for Biophysical Chemistry, Gottingen, Germany. - synaptic modulation - synapses switch between conducting and non-conducting states - modulation governed by correlations in temporal structure of signals - momentarily useless connections are deactivated and interference between different memory traces are reduced and memory capacity increased - dynamic modulation (Moran & Desimone 1985) - brain does not contain complex feature detector cells X (e.g., face cells) - timing correlations signal objects? 7

8 Crick 1984 J.K.Tsotsos 15 Crick, F. (1984). Function of the thalamic reticular complex: The searchlight hypothesis, Proc. Natl. Acad. Sci. USA 81,, Treisman s searchlight is controlled by the reticular complex of the thalamus X - searchlight is expressed by rapid bursts of firing from subsets of thalamic neurons - conjunctions are mediated by rapidly modifiable synapses (Malsburg synapses) by these bursts? - activation of Malsburg synapses produces transient cell assemblies connecting neurons at different levels? Koch and Ullman 1985 J.K.Tsotsos 16 Koch, C., Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry, Human Neurobiology 4, 4, saliency map (Treisman s map)? winner-take-all competition (Findlay 199, Lee et al. 1999) - WTA selects items to route to central representation X inhibition of return for shifts? time to move attention requires time logarithmic in distance between stimuli X (Krose & Julesz 1989) no single cell modulations X 8

9 Anderson and Van Essen 1987 Shifter Circuits J.K.Tsotsos 17 Anderson, C., Van Essen, D. (1987). Shifter Circuits: a computational strategy for dynamic aspects of visual processing, Proc. Natl. Academy Sci. USA 84: : information routing is accomplished by simple shifting circuits starting in the LGN and input layers of primate visual area V1. X realignment is based on the preservation of spatial relationships X stages linked by diverging excitatory inputs. direction of shift by inhibitory neurons that selectively suppress sets of ascending inputs. stages are grouped into small and large scale shifts. control comes from pulvinar? Wolfe Guided Search J.K.Tsotsos 18 Wolfe, J., Cave, K., Franzel, S. (1989). Guided search: An alternative to the feature integration model for visual search, J. Exp. Psychology: Human Perception and Performance 15,, Wolfe, J. (1994). Guided search 2.0: a revised model of visual search, Psychonomic Bulletin and Review, 1(2): Wolfe, J., Gancarz, G. (1996). Guided Search 3.0: A Model of Visual Search Catches Up With Jay Enoch 40 Years Later, in V. Lakshminarayanan (Ed.), Basic and Clinical Applications Vision Science,, Dordrecht, Netherlands: Kluwer Academic. p attentional deployment of limited resources is guided by output of earlier parallel processes - activation map? 9

10 J.K.Tsotsos 19 Guided Search 3.0 Sandon 1990 J.K.Tsotsos 20 Sandon, P. (1990). Simulating visual attention, J. Cognitive Neuroscience 2: first real implementation of Koch & Ullman model - first real implementation of any attention model - hierarchical, multiscale (pyramid) connectionist network - translation-invariant object recognition - bottom-up feature guidance - Koch and Ullman WTA scheme - no neurobiological predictions 10

11 Phaf, Van der Heijden, Hudson 1990 SLAM J.K.Tsotsos 21 Phaf, R., Van der Heijden, A., Hudson, P. (1990). SLAM: A connectionist model for attention in visual selection tasks, Cognitive Psychology 22,, based on McClelland & Rumelhart 1981 model for visual word recognition - adds response selection and evaluation - inhibitory competition to reduce distractor interference - attended items enhanced Tsotsos Selective Tuning Model J.K.Tsotsos 22 Tsotsos,, J.K., Analyzing Vision at the Complexity Level, Behavioral and Brain Sciences 13-3,, p , Tsotsos, J.K. (1993). An Inhibitory Beam for Attentional Selection, in Spatial Vision in Humans and Robots,, ed. by L. Harris and M. Jenkin, p , Cambridge University Press. Tsotsos, J.K., Culhane, S., Wai, W., Lai, Y., Davis, N., Nuflo, F. (1995). Modeling visual attention via selective tuning, Artificial Intelligence 78(1-2),p p Tsotsos, J.K. (1995). Towards a Computational Model of Visual Attention, in Early Vision and Beyond,, ed. by T. Papathomas, C, Chubb, A. Gorea, E. Kowler, MIT Press/Bradford Books, p Tsotsos, J.K., Culhane, S., Cutzu, F., From Theoretical Foundations to a Hierarchical Circuit for Selective Attention, Visual Attention and Cortical Circuits,, ed. by J. Braun, C. Koch & J. Davis, MIT Press (in press). 11

12 - attention modulates neurons to earliest levels; wherever there is a many-to-one mapping - signal interference controlled by surround inhibition throughout processing network - task knowledge biases computations throughout processing network - inhibition of connections not units Hernandez-Peon, Scherrer, Jouvet (1956) - attentional control is local, distributed and internal - competition is based on WTA (different form than previous models) - pyramid representation with reciprocal convergence and divergence Salin &Bullier(1995) attentional spotlight J.K.Tsotsos 23 neuron sees this receptive field subject attends to single item The basic idea (BBS 1990) effective receptive field of selected unit in unattended case layers of input abstraction hierarchy inhibitory attentional beam "inhibit" zone "pass" zone Kastner, De Weerd, Desimone, Ungerleider, 1998 unit of interest at top J.K.Tsotsos 24 processing pyramid input pass pathways Caputo & Guerra 1998 Bahcall & Kowler 1999 Vanduffel, Tootell, Orban 2000 Smith et al inhibited pathways 12

13 Ahmad 1991 VISIT J.K.Tsotsos 25 Ahmad, S. (1992). VISIT: a neural model of covert visual attention, in Advances in Neural Information Processing Systems,, edited by J.E. Moody, et al., 4: , San Mateo, CA: Morgan Kaufmann. - complexity is linear in number of pixels - bottom-up connectionist model - can compute spatial relations as well as model visual search - gated feature maps that inhibit unattended features - peaks in weighted feature saliency map locate focus of attention Predicting the attentional roles for several cortical areas Architecture for visual search Mozer 1991 MORSEL J.K.Tsotsos 26 Mozer, M.C. (1991). The perception of multiple objects,, MIT Press, Cambridge, MA - connectionist model of spatial attention and object recognition - BLIRNET builds location invariant representations of letters and words - BLIRNET includes a pull-out net and an attentional mechanism to limit processing - pull-out net uses semantic and lexical knowledge to select best interpretation - attention selects location - guided bottom-up by locations of stimuli and by top-down task bias (as in controlling temporal order in reading) - attention gates input to BLRNET - does not inhibit, just transmits activation with lower probability - attention uses neural net optimization search 13

14 J.K.Tsotsos 27 Olshausen, Anderson & Van Essen 1993 J.K.Tsotsos 28 Olshausen, B., et al. (1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, J. of Neuroscience, 13(1): implementation of shifter circuits - forms position and scale invariant representations at the output layer X - control neurons, originating in the pulvinar, dynamically modify synaptic weights of intracortical connections to achieve routing? - the topography of the selected portion of the visual field is preserved - uses Koch & Ullman mechanism (luminance saliency only) for selection X - associative recognition at output layer 14

15 J.K.Tsotsos 29 only attended item reaches output layer Olshausen seeks to achieve translation-rotation invariant recognition Niebur, Koch et al J.K.Tsotsos 30 Niebur, E., Koch, C., Rosin, C. (1993). An oscillation-based model for the neural basis of attention, Vision Research 33,, Niebur, E., Koch, C. (1994). A model for the neuronal implementation of selective visual attention based on temporal correlation among neurons, J. Comput. Neuroscience 1(1), Usher, M., Niebur, E. (1996). Modeling the temporal dynamic of IT neurons in visual search: A mechanism for top-down selective attention, J. Cognitive Neuroscience 8:4,, selection by the Koch & Ullman mechanism - attentional modulation is added at V1 and affects only the temporal structure of the spike trains of V1 neurons but not their mean firing rate. - the existence of frequency-selective inhibitory interneurons are assumed in V4 - selective attention activates competition within a microcolumn of neurons in V4. - outputs of V1 neurons are tagged, their postsynaptic targets in V4 will win in the V4 level competition. - no attentional effects on firing rates in V1, only in V4 or higher areas. - refinement in 2nd paper proposes firing coincidences among V2 neurons is sufficient. - no attentional effects on firing rates in V1, only in V4 or higher areas. X - temporal synchrony/coincidence? 15

16 Postma 1994 SCAN J.K.Tsotsos 31 Postma, E., et al. (1997). SCAN: a scalable model of attentional selection, Neural Networks 10(6): hierarchical network of gating lattices (pyramid) - bottom-up WTA - leads to an attentional pathway J.K.Tsotsos 32 threshold classifier (follows ART) control network data part gating network 16

17 Desimone & Duncan 1995 Biased Competition J.K.Tsotsos 33 Desimone, R., Duncan, J. (1995). Neural mechanisms of selective visual attention, Annual Reviews of Neuroscience 18,, all stimuli in visual field participate in a competition; interactions due to different objects activating the same neurons are mutually suppressive - strength of competitive interactions depends inversely on distance between stimuli - bias to favour one stimulus in a cluttered field can arise through many mechanisms - feedback bias is not only spatial, but can be for a feature - main source of top-down bias is working memory, prefrontal cortex - result is the suppression of the neuronal representations of behaviourally irrelevant stimuli in extrastriate cortex Schneider 1995 VAM J.K.Tsotsos 34 Schneider, W. X. (1995). VAM: neuro-cognitive model for visual attention control of segmentation, object recognition, and space-based motor action, Visual Cognition 2, 2, selection-for-object-recognition and selection-for-space-based-motor-action - follows von der Malsburg model - what-based attention identifies locations that share features with the target - where-based attention locates differences among local stimulus elements - inhibition of return (1) The color representation of the object is globally segmented by this top-down signal (2) feedforward flow of an attentional signal from V1 to the type-level attention selection in V1 17

18 J.K.Tsotsos 35 (1) The color representation of the object is globally segmented by this top-down signal (2) feedforward flow of an attentional signal from V1 to the type-level (3) where-based attentional control selects a region (4) attentional signal to the type-level modules what and where based endogenous attentional control LaBerge 1995 Triangular Circuit Model J.K.Tsotsos 36 LaBerge, D. (1995). Attentional processing: The brain's art of mindfulness.. Cambridge, MA: Harvard University Press. - attention requires 3 simultaneous activities: expression, enhancement, control - expression - clusters of neurons in posterior and anterior cortex - enhancement - thalamic nuclei excitatory neurons activate neurons in cortical columns - control clusters of neurons in frontal cortex 18

19 Itti 1998 J.K.Tsotsos 37 Itti, L., Koch, C., Niebur, E. (1998). A model for saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Analysis and Machine Intelligence 20,, a newer implementation of Koch and Ullman s scheme - fast and parallel pre-attentive extraction of visual features across 50 spatial maps (for orientation, intensity and color, at six spatial scales) - features are computed using linear filtering and center-surround structures - these features form a saliency map - Winner-Take-All neural network to select the most conspicuous image location - inhibition-of-return mechanism to generate attentional shifts - saliency map topographically encodes for the local conspicuity in the visual scene, and controls where the focus of attention is currently deployed J.K.Tsotsos 38 19

20 Cave 1999 Feature Gate J.K.Tsotsos 39 Cave. K. (1999). The FeatureGate model of visual selection, Psychological Res. 62: : hierarchy of spatial maps encode features - inhibition of distractor locations during an attentive task - flow of information is governed by a set of gates that control competition and prevent interference - inhibition is applied at several levels of hierarchy to inhibit distractor locations - selection based on local differences in a bottom-up WTA (with top-down biases) - inhibition of return Hierarchy of Maps J.K.Tsotsos 40 top-down : chooses locations that share features with the target 20

21 J.K.Tsotsos 41 levels of spatial map hierarchy width indicates signal strength attentional gates J.K.Tsotsos 42 21

22 Conclusions J.K.Tsotsos 43 Several ideas have endured: Winner-Take-All for selection (competition) Hierarchies Inhibition of return to force serial search Some kind of gating process Inhibitory surrounds However, modelling seems to be still in its early days Progress will depend on whether modellers and experimenters can work together 22