CHAPTER 82 ERP Studies of Selective Attention to Nonspatial Features Alice Mado Proverbio and Alberto Zani ABSTRACT This paper concentrates on electrophysiological data concerning selective attention to nonspatial attributes (spatial frequency, color, shape, orientation, etc.), and the way these attributes are combined into a unified percept, so that it becomes identified as an object. Feature-based and object-based mechanisms of the brain as investigated with ERPs are analyzed. An overview is provided of studies reporting the differential activation of two cortical subsystems of the visual brain, the so-called dorsal, or Where, and ventral, or What systems, in conditions in which stimulus attributes must be selectively attended to separately and/or conjointly. Efforts are made to demonstrate the task-related relative segregation and complex interactions of the aforementioned systems during the separate or conjoint processing of stimulus attributes. I. NONSPATIAL SELECTIVE ATTENTION For a long time it was believed that the primary projection areas of brain cortex acted as simple analyzers of input features and were not directly involved in the so-called top-down selection mechanisms; that is mechanisms based on higher level cognitive strategies. Only recently has this conception been challenged as a result of new findings provided by hemodynamic bioimaging and neurophysiological and electromagnetic techniques. These techniques are able, on the one hand, to determine the functional activation of the cortical and subcortical areas, and, on the other, to reveal the early timing of the attentional influences on the processing stages. Event-related potential (ERP) studies on the selection of single nonspatial stimulus attributes (such as color, orientation, texture, shape, or spatial frequency) have indicated that the timing of attention modulation for such processing starts as early as 80 100ms poststimulus, and this is manifested in the form of a modulation of P/N80 or P120 components (see Chapter 85), continuing with a prominent increase in negativity at N1 and N2, called selection negativity (SN), and a large P300 response to targets. For example, the selection of checkerboard patterns based on their check-size produces an increase in amplitude of the P1 and N115 early responses recorded at electrodes O1 and O2 corresponding to mesial occipital areas (Zani and Proverbio, 1995). Likewise, selecting gratings on the basis of their spatial frequency (Zani and Proverbio, 1997) and orientation (Karayanidis and Michie, 1997; Proverbio, Esposito, and Zani, 2002), or selecting alphanumeric characters on the basis of their shape produces an increase in the evoked response at the sensory level. Color selection is also reported to occur at a very early latency, often even earlier than the selection of other nonspatial features such as size, shape, or orientation. It has also been reported that color selection specifically activates the ventral stream, unlike other stimulus attributes such as motion (Anllo-Vento and Hillyard, 1996). II. DORSAL AND VENTRAL STREAMS Several neuroimaging and neurometabolic studies have shown the existence of separate pathways for object recognition and spatial localization. These Neurobiology of Attention 496 Copyright 2005, Elsevier, Inc. All rights reserved.
II. DORSAL AND VENTRAL STREAMS 497 pathways are named the ventral and the dorsal streams of the visual system (Arrington et al., 2000; Fink et al., 1997; Haxby et al., 1991; Olson, 2001; Ungerleider and Mishkin, 1982): one projects to the inferior temporal cortex (What system), and the other to the parietal cortex (Where system). A. Properties Evidence has accumulated to indicate that the dorsal stream handles information on spatial position and motion of environmental stimuli, as it possesses collicular afferents (including ipsilateral ones), whereas the ventral system analyses physical features such as orientation, color, spatial frequency, and texture (Ungerleider and Mishkin, 1982). Although the former mostly receives afferent fibers from large magnocellular gangliar cells, the latter receives afferents from small parvocellular cells. There is evidence that these two systems are related to scotopic and peripheral vision, as opposed to photopic and foveal vision, to the vision of low as opposed to high spatial frequencies and, more generally, to visual attention mechanisms based on space rather than on the object (Fink et al., 1997). Hemodynamic functional anatomical studies have clearly shown that visual attention modulates the activity of both systems (O Craven et al., 1999; Arrington et al., 2000; Wang et al., 1999). data has indicated secondary and primary visual cortices as possible generators of the surface electrical activity recorded at lateral occipital and mesial occipital scalp sites, respectively (Kenemans et al., 2000; see also Chapter 84). These data indicate the possible starting point of the dorsal and ventral streams of visual processing in the visual cortex, investigated by means of the electrophysiological techniques. B. ERP Studies This modulation of activity has also been observed by measuring changes in amplitude, latency, and scalp topography of event-related potentials (ERPs) of the brain to visual stimuli as a function of task relevance and attention condition (e.g., Annlo-Vento and Hillyard, 1996; Martin-Loeches et al., 1999; Zani and Proverbio, 1995; Wang et al., 1999). Attention mechanisms based on spatial location and mostly involving the dorsal stream are extensively reviewed in Chapter 84. 1. Visual Evoked Potentials Modern studies of visual evoked potentials have provided robust evidence of a topographical and functional dissociation between visual areas devoted to the analysis of low versus high spatial frequency content of luminance-modulated gratings. In detail, there is greater activity of lateral occipital areas during processing of low spatial frequencies and greater activity of mesial occipital areas during processing of high spatial frequencies (Proverbio et al., 1996), as shown in the voltage and scalp current density maps (SCD) of Fig. 82.1. Dipole-source analysis performed on ERP FIGURE 82.1 Grand-average voltage (top) and Scalp Current Density (SCD) maps (bottom) of brain activity recorded in response to four different luminance-modulated gratings going from a low (1.5 cycles per degree, cpd) up to a high (12 cpd) spatial frequency, presented in the central visual field during passive viewing. Note the topographic dissociation between the pattern of response to high vs. low frequency gratings, reflecting the equivalent distinction between the dorsal/ventral type of activation. Reprinted from Brain Topography 9; A. M. Proverbio, A. Zani, and C. Avella; Differential activation of multiple current sources of foveal VEPs as a function of spatial frequency, pp. 59 69. Copyright (1996) with permission from Kluwer Academic/Plenum Publishers and authors.
498 CHAPTER 82. ERP STUDIES OF SELECTIVE ATTENTION TO NONSPATIAL FEATURES 2. ERPs Studies on Selective Attention Space-based and Frequency-based Attentional Selection The two mechanisms of object- and spacebased selective attention normally work in close interaction. Yet, since they are partly functionally segregated, and probably based on the activation of nonoverlapping visual neural areas, to some degree it is possible to investigate these areas separately in order to unveil their neurofunctional activation. Recently we carried out a series of experiments (see Chapter 85) in which we were able to observe the two functional mechanisms by inducing different types of attentional set in the viewers. This was done by adopting the same set of visual stimuli in different tasks lateralized isoluminant gratings of two spatial frequencies, namely 1 and 7 cpd and modifying the experimental instructions. In different sessions the same individuals were instructed to pay attention and respond to different stimulus properties (frequency or location) while brain-evoked responses were recorded with a 32-channel montage. In the first paradigm, participants were requested to pay selective attention to a spatial frequency, and to respond to the target frequency whatever its spatial location, whereas in the second they had to attend and respond to all the gratings solely on the basis of their spatial location, thus ignoring the frequency. The results revealed a very early attentional selection effect in both cases, although with completely different morphology and topographical activation (see Fig. 82.2a,b). Indeed, location selection affected the P1 component of the ERP at lateral occipital scalp sites. This effect, commonly reported in the literature on spatial attention (e.g., see Mangun, 2002, for an exhaustive review of studies reporting this effect), was followed by a large enhancement in positivity called P300. Conversely, the selection of spatial frequency produced a very early P/N80 followed by a considerable negative deflection (N1/N2 complex, strongly modulated by selection negativity) and a somewhat delayed P300. At the same time, the reaction times in the spatial selection task were about 100 ms faster than those obtained in the frequency selection task. These data confirm the partial functional independence of the two visual feature selection systems, both based on a very early sensory filter (early selection) although dependent on two anatomically and functionally separate neural streams. Observation of the topographic distribution of the attentional effect of ERP differences obtained by subtracting the response to the nontargets from that to the same stimuli when they were targets for the selection of spatial frequency identified a filter that was strongly linked to visual processing, first of area 17 (at the level of P80) and then of areas 18 and 19 (at the level of selection negativity), whereas the selection of the spatial position was apparently based on the functionality of the dorsal visual pathway. The 3D maps presented in Fig. 82.3 show the scalp distribution of attention-related activity in the range of selection negativity, probably reflecting the selective modulation of the ventral stream. Indeed, this activity was recorded only in the frequency-relevant task, and was absent during the location-relevant task. Object-based Attentional Selection and Selection Negativity Much evidence has been provided that attention can be directed to a conjunction of features (object-based attention) and that object representation actually is encoded during the earliest stages of sensory processing (still improperly called preattentive stages) as shown, for example, by Valdes-Sosa et al. (1998), who found that the P1 component of ERPs (100 ms) was affected by object-based attention even if the stimulus fell at an unattended space location. Again, O Craven et al. (1999) used an fmri study to demonstrate that attending to one attribute of an object (such as the motion of a moving face) enhanced the neural representation not only of that attribute but also of another attribute of the same object (for example, the face), thus providing physiological evidence that whole objects are selected even when only one visual attribute is relevant. Many advances have been made in the comprehension of the neural systems implicated in object processing (Olson, 2001), and there is currently agreement on the view that object knowledge is represented in the inferior temporal cortex (ventral stream). In a further work on feature-conjunction selection (Proverbio and Zani, 2002) we investigated the mechanisms of the combined selection of frequency and spatial location using the same gratings adopted in the previous attend-location and attend-frequency paradigms and recording ERPs with a 32-channel montage. The task consisted in selectively attending to both frequency and location dimensions of the gratings. The results indicated that selection of the frequency occurred much earlier than had previously been believed, that is, within 60 80 ms post-stimulus (at this regard see also Chapter 85). Scalp current density maps (SCD) of attention effects showed a lateral occipital activation for location selection (independent of frequency relevance) and a mesial occipital activation for frequency selection (independent of location relevance), probably suggesting a dissociation between secondary and primary visual areas. These data support the hypothesis that the two attentional mechanisms may act by differentially modulating the
FIGURE 82.2 Grand-average event-related potentials ERPs recorded in response to luminance-modulated gratings of different spatial frequency presented at different space locations, displayed as a function of stimulus relevance. In A only the spatial frequency of the stimulus was task-relevant and, respectively, target (F+) or nontarget (F-). In B only stimulus location was relevant: target (L+) or nontarget (L-). In C both frequency and location had to be selectively attended to in a conjoined attention task. Note that ERP waveforms shown here as a function of location-relevance (target: L+, or nontarget: L-) are grand-averages across responses to stimuli independent of frequency relevance.
500 CHAPTER 82. ERP STUDIES OF SELECTIVE ATTENTION TO NONSPATIAL FEATURES FIGURE 82.3 Realistic head three-dimensional voltage maps of brain activity recorded during the attend-frequency (left) and attend-location (right) tasks in the latency range of selection negativity (i.e., 180 280 ms). Maps were computed on the difference waves obtained from the target nontarget subtraction to show the effect of selective attention on ERPs. Note how space-selection did not produce any attention-related selection negativity, and how the map for frequency selection pointed at a focus centered over the baso-occipito-temporal area consistent with a ventral activation. activity of the dorsal and ventral streams for spatial and nonspatial selective attention, respectively. Interestingly, frequency selection depended on location relevance, in that ERPs were larger to stimuli relevant in both features (L+F+) rather than only in location (L+F-), whereas frequency relevance per se was not sufficient to enhance the ERP response when the location was irrelevant (L-F+). More importantly, stimuli relevant only in location elicited a large attentional selection negativity, similar to that elicited in the attention to frequency paradigm described previously, which was instead absolutely absent in the attendlocation paradigm (see Fig. 82.2c). Three-dimensional maps of voltage and scalp current density (SCD) for selective negativity provided evidence of a strong similarity in the distribution of this response across the conjoined-selection paradigm (also when spatial frequency was not relevant (L+F-), although still attentionally evaluated by the viewer) and the independent spatial frequency-selection paradigm for frequencyrelevant gratings (F+). This result suggests that the selection mechanisms of the two features operate in parallel right from the earliest stages of analysis (see also Chapter 67), and that feature selection, rather than being temporally preceded by space selection, depends on it, in that is centered on precise coordinates of the attended receptive location. On the other hand, location relevance, in a paradigm in which frequency and location must be conjointly attended to, is affected by stimulus spatial frequency even if this is not task-relevant (L+F-), since it activates sensory attentional mechanisms (reflected by selection negativity) usually involved in spatial frequency but not location selection. So far we have seen how it is possible to use laboratory investigations to study the way in which the visual system processes and attentionally selects one or more visual features (spatial frequency, depth, stereopsis, color, orientation, texture, luminance) of the surrounding environment, separately. Of course, in actual fact, we perceive a unitary environment and not a separate series of objects or individual attributes. (We do not address binding issues here; see Chapters 24 and 47). This perception of the unitary nature derives from the interaction between the Where and What systems which, although partially anatomically and functionally distinct, operate in parallel and in very close coordination. Clear evidence of this interdependence comes from neuroimaging and neuropsychological literature. For example, the clinical neuropsychological study by Friedman-Hill et al. (1995) indicated that patients with focused bilateral lesions of the parietal cortex are unable to combine correctly color and shape of stimuli presented in the two visual hemifields. This suggests that the integrity of the Where system is essential for the correct recognition of objects. On the other hand, a large body of very recent neuropsychological, neurophysiological, and behavioral data indicates that the spatial and nonspatial intentional mechanisms are probably not separated at all. Overall, these data support the view that object-directed or feature-directed selection, rather than being preceded by a space selection, is centered on line on precise coordinates of the attended space (i.e., object-centered space receptive fields; see Olson, 2001). In our view, our electrophysiological data on conjoined selection of frequency and space reported previously are in line with this view derived from other research lines in cognitive neuroscience. Object Selection: Color and Shape Processing In a conjoined selective attention study we recently studied the mechanism subserving the processing of color and shape of familiar objects, in order to provide ERP indices of attention effects during ventral stream activation (Proverbio et al., 2004). On the whole, compared to color selection, shape selection produced slower response times, and larger and later N2, Frontal Selection Positivity (FSP), and P300 components, thus further supporting the viewpoint that color dimension selection is faster and easier than shape selection. Unprecedented in the literature, however, was the
III. CONCLUSION 501 result that color-related selection negativity (SN) was significantly affected by shape content, but not vice versa. When color was the attended feature (e.g., yellow) the SN response was significantly greater if the stimulus shape, which was the irrelevant dimension, was canonically color-related (e.g., a chick) rather than unrelated (e.g., an elephant). This greater SN response was particularly marked over the left hemisphere. The opposite pattern did not occur in the attend-shape task, in which the stimulus color content did not affect target shape selection. The data suggest a strong interaction between the two mechanisms subserving the selection of shape and color, and indeed provide evidence that multiple features of an object are processed automatically as a perceptual whole and not independently of each other. The fact that, in our study, shape content affected color processing even when shape was the irrelevant dimension and despite color, as an independent feature, being processed faster than shape, strongly suggests that object processing does not depend on the processing output of independent visual features (in this case shape and color), but that it is carried out in parallel right from the earliest stages of processing within the occipito-temporal pathway, as supported by the findings of recent neuroimaging literature. We propose that the left occipito-temporal scalp area, where the selection negativity was shown to be maximally affected by stimulus prototypicity (e.g., a green artichoke as compared with a pink one), reflects the underlying activity of structures belonging to the ventral stream, which are involved in the brain s representation of object-color knowledge. III. CONCLUSION The view that emerges from the literature indicates that the dorsal and ventral streams of the visual system, although partially anatomically segregated, may be activated in parallel and in an independent or conjoined mode depending on the attentional demands and task requirements. In addition, the perception of multidimensional objects is accomplished through an active binding process of spatial and nonspatial features. This mechanism would not be based on hierarchically organized independent processes, but rather on the horizontal processing of visual cells that takes place at very early stages of analysis as advanced by models (see Chapter 24) assuming that object attributes may be initially conjoined in a single representation, while being separately analyzed in parallel, dimension by dimension. References Anllo-Vento, L., and Hillyard, S. A. (1996). Selective attention to the color and direction of moving stimuli: electrophysiological correlates of hierarchical feature selection. Percept. Psychophys 58, 191 206. Arrington, C. M., Carr, T. H., Mayer, A. R., and Rao, S. M. (2000). Neural mechanisms of visual attention: object-based selection of a region in space. J. Cogn. Neurosci. 12, 106 117. Fink, G. R., Dolan, R. J., Halligan, P. W., Marshall, J. C., et al. (1997). Space-based and object-based visual attention: shared and specific neural domains. Brain 120, 2013 2028. Friedman-Hill, S. R., Robertson, L. C., and Treisman, A. (1995). Parietal contributions to visual feature binding: evidence from a patient with bilateral lesions. Science 269, 853 855. Haxby, J. V., Grady, C. L., Horwitz, B., Ungerleider, L. G., et al. (1991). Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proc. Natl. Acad. Sci. USA 88, 1621 1625. Karayanidis, F., and Michie, P. T. (1997). Evidence of visual processing negativity with attention to orientation and color in central space. Electroencephalogr. Clin. Neurophysiol. 103, 282 297. Kenemans, J. L., Baas, J. M., Mangun, G. R., Lijffijt, M., et al. (2000). On the processing of spatial frequencies as revealed by evokedpotential source modeling. Clin. Neurophysiol. 111, 1113 1123. Mangun, G. R. (2002). Neural mechanisms of attention. In The Cognitive Electrophysiology of Mind and Brain, (A. Zani and A. M. Proverbio, Eds.), pp. 247 258. Academic Press, San Diego. O Craven, K. M., Downing, P. E., and Kanwisher, N. (1999). fmri evidence for objects as the units of attentional selection. Nature 401, 584 587. Olson, C. R. (2001). Object-based vision and attention in primates. Curr. Opin. Neurobiol. 11, 171 179. Proverbio, A. M., and Zani, A. (2002). Visual selective attention to object features. In The cognitive electrophysiology of mind and brain, (A. Zani and A. M. Proverbio, Eds.), pp. 275 308. Academic Press/Elsevier, San Diego. Proverbio, A. M., Burco, F., Del Zotto, M., and Zani, A. (2004). Blue piglets? Electrophysiological evidence for the primacy of shape over color in object recognition. Cogn. Brain Res. 18, 288 300. Proverbio, A. M., Esposito, P., and Zani, A. (2002). Early involvement of temporal area in attentional selection of grating orientation: an ERP study. Cogn. Brain Res. 13, 139 151. Proverbio, A. M., Zani A., and Avella, C. (1996). Differential activation of multiple current sources of foveal VEPs as a function of spatial frequency. Brain Top 9, 59 69. Ungerleider, L. G., and Mishkin, M. (1982). Two cortical visual systems. In Analysis of visual behavior (D. J. Ingle, M. A. Goodale, and R. J. W. Mansfield, Eds.), pp. 549 586. The MIT Press, Cambridge, Mass. Valdes-Sosa, M., Bobes, M. A., Rodriguez, V., and Pinilla, T. (1998). Switching attention without shifting the spotlight: object-based attentional modulation of brain potentials. J. Cogn. Neurosci. 10, 137 151. Wang, J., Zhou, T., Qiu, M., Du, A., et al. (1999). Relationship between ventral stream for object vision and dorsal stream for spatial vision: an fmri + ERP study. Hum. Brain Mapp. 8, 170 181. Zani, A., and Proverbio, A. M. (1995). ERP signs of early selective attention effects to check size. Electroencephalogr. Clin. Neurophysiol. 95, 277 292. Zani, A., and Proverbio, A. M. (1997). Attention modulation of short latency ERPs by selective attention to conjunction of spatial frequency and location. J. Psychophysiol. 11, 21 32.