Blue piglets? Electrophysiological evidence for the primacy of shape over color in object recognition

Transcription

1

2 Cognitive Brain Research 18 (2004) Research report Blue piglets? Electrophysiological evidence for the primacy of shape over color in object recognition Alice Mado Proverbio a,b,c, *, Fabiana Burco b, Marzia del Zotto a,c, Alberto Zani b,c a Department of Psychology, University of Milano-Bicocca, Milan, Italy b Cognitive Electrophysiology Laboratory, University of Trieste, Italy c Institute of Molecular Bioimaging and Physiology, CNR, Segrate, Milan, Italy Accepted 29 October Abstract The goal of the study was to investigate how the color and shape of visual stimuli are processed when they are conjointly presented and represent real, and familiar, entities for which normal individuals presumably have a specific object color knowledge (e.g., piglets are pink, artichokes are green). There is evidence, from event related potential (ERP) literature on selective attention to color in conjunction with other, arbitrarily related, stimulus dimensions (e.g., geometrical shape), that color is processed faster than shape, and that the processing of shape depends on color relevance. In this study we recorded ERPs from 28 scalp sites in right-handed volunteers performing selective attention tasks to either color or shape of pictures representing familiar objects and animals. The results revealed that the selection of color was faster, and probably less demanding, than that of shape. However, it was also evidenced that the selection of color depended on object shape, but not vice versa. Indeed, in the attend-color condition, the N2 responses were significantly greater when stimulus shape was prototypically associated, rather than unassociated, with the color perceived. Topographical mapping of difference voltages identified the posterior occipito/temporal region of the left hemisphere as the possible locus of conjoined color and shape processing. Overall, the data support object-based attention models. D 2003 Elsevier B.V. All rights reserved. Theme: Neural basis of behavior Topic: Cognition Keywords: Object color knowledge; Ventral stream; VEP; Selective attention; Vision; Hemispheric asymmetry 1. Introduction Both neuroimaging studies (e.g., Refs. [6,15,39]) and intracranial ERP recordings [1] have provided knowledge of the anatomical location of brain areas devoted to the processing of color and shape in humans. Whereas the selective processing of shape activates the left dorsal occipital region and bilateral temporal, parietal and parahippocampal regions, the processing of color involves bilateral occipital regions (inferior occipito-temporal cortex, including medial * Corresponding author. Department of Psychology, University of Milano-Bicocca, Piazza dell Ateneo Nuovo 1, 20126, Milan, Italy. Tel.: ; fax: address: mado.proverbio@unimib.it (A.M. Proverbio). and lateral lingual gyrus, posterior fusiform gyrus and inferior temporal gyrus). Event-related potential (ERP) studies on the selection of color [3], or the combination of color and other features (e.g., orientation, color and size) [20], have indicated that the timing of attentional modulation for such selective processing starts as early as 100 ms post-stimulus in the form of a P130, continuing with prominent selection negativity (SN) and a broad P300 distribution. This early effect is consistent with the view, advanced by Zani and Proverbio [38], of early selection mechanisms for non-spatial stimulus features. Dipole modeling analysis applied to high-density electrode montage has localized early P130 and SN sources as being in, respectively, the dorsolateral occipital and ventral occipital (close to the collateral sulcus) regions [3], findings that are consistent with neuroimaging data /$ - see front matter D 2003 Elsevier B.V. All rights reserved. doi: /j.cogbrainres

3 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Several electrophysiological studies carried out to investigate the neural mechanisms of color processing have employed the recording of ERP during selective attention paradigms where color and other object attributes were selectively conjoined. In most of the studies, the relationship between the shape and color of the presented stimuli was totally arbitrary as the stimuli were non-pictorial drawings such as geometric shapes [2,25], gratings [21], checkerboards [3], or alphanumeric characters [33 34], for which no color knowledge is stored. In these ERP paradigms it is frequently reported that color selection occurs before the selection of other nonspatial features such as size, shape or orientation [e.g., Refs. [8,20]. In addition, the selection of those characteristics very often depends hierarchically on color selection. Indeed the selection negativity ERP wave, that reflects attentional selection at the N1 N2 level, was absent when color was irrelevant, even though the other attribute was task relevant [33,34]. Quite interestingly, Michie and colleagues [25], in a recent ERP study adopting original geometric colored patterns (yellow, red, blue, gray) as stimuli, found that the earliest effect of color processing, an early anterior positivity (FSP), also occurred when color was the unattended feature (pattern being attended), but not vice versa. Thus, it can be seen that the available literature on color processing suggests that, compared to other nonspatial features, color might have a special status in attentional selection, as when it is linked arbitrarily to another visual feature like shape. The primary goal of the present study was to further investigate how the brain mechanisms involved in processing color and shape might interact. In line with an ecological approach to visual perception, we chose to present familiar, everyday objects and animals for which the visual brain might possess a specific object color knowledge, due to the repeated association of specific colors and shapes throughout the person s life span. Indeed, the existence of such association is supported by the available evidence in the literature [5]. To compare the discriminability of the colors and shapes of the stimuli used in the ERP experiment described below, we ran a simple choice RT study (Control Experiment 1) in which the identical stimuli used in the ERP experiment were presented to a different group of subjects. The goal was to determine whether the ability to discriminate color and shape dimensions differed per se and, if so, to characterize the nature of the difference. Furthermore, in order to identify which color should most appropriately be matched with a given shape, we ran another behavioral experiment (Control Experiment 2). In this study, for each of the eight colors used in Control Experiment 1 and ERP experiment, we compared two different shades of the color (e.g., light and dark) for the various shapes in a living/non-living object categorization task. 2. Control experiment Methods Participants Ten right-handed individuals (4 males and 6 females) with a mean age of 29 years volunteered to participate in this experiment Stimuli The stimuli were identical to those used in the ERP experiment and described in its Methods (see also Fig. 2a). Stimulus parameters (duration, ISI, luminance, etc.) were also the same (see later in the text). The only difference was the behavioral task Task Stimuli were presented in pairs, above and below the fixation point at the center of the screen. The pairs could be identical or different in shape or in color. During the attend-shape condition 256 different pairs of randomly mixed stimuli were presented to participants. Half the pairs were of identical shape but different in color (SAME), and the other half were different in both shape and color (DIFFERENT). During the attend-color condition 256 different pairs of stimuli were randomly presented to participants. Half the pairs were of identical color but different in shape (SAME), while the other half were different in both color and shape (DIFFERENT). The task consisted in deciding whether stimuli were equal or different (SAME/DIFFERENT judgment) for the relevant dimension, either color or shape, by pressing a response key with the right or left index finger. Hand order and attention condition were randomized across subjects Results and discussion RTs exceeding the mean F 2 standard deviations were excluded from the analysis. The data underwent two-way repeated-measures ANOVA, the factors for the analysis being attention condition (attend-color, attend-shape) and stimulus type (same, different). Results (displayed in Fig. 1, left) showed a significant main effect of attention ( F 1,9 = 5.37; p < 0.05), with faster RTs when the relevant dimension was color (556 ms) rather than shape (577 ms). The stimulus type factor was also significant ( F 1,9 = 29.3; p < ), with much faster RTs to SAME (550 ms) than DIFFERENT (583 ms) stimuli, which is a pattern very commonly reported in the behavioral literature. Overall, the results of Control Experiment 1 indicate that the ability to discriminate color and shape dimensions does differ, and that this probably depends on the specific nature of these dimensions and not on the particular choice of color/shape pairing used in the present study. This difference in discriminability is probably why, in the ERP study,

4 290 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Fig. 1. Mean reaction times (RTs) with standard errors obtained in Control Experiment 1 (N = 10) and Control Experiment 2 (N = 12), designed to test the discriminability of the two dimensions per se, and the typicality of canonical colors adopted in the main ERP experiment. color selection produced faster RTs, and earlier ERP components than shape selection. A possible explanation for this obvious difference in discriminability might lie in the fact that shapes are defined by a complex distribution of luminance and relative spatial frequency spectra across space, whereas colors are defined by the wavelength of electromagnetic radiation. 3. Control experiment 2 A second pilot behavioral experiment was designed to ascertain whether the canonical colors adopted in the ERP experiments had not been arbitrarily defined, although we had chosen, for each shape, the shade and color saturation most typically associated with that shape in the common iconography (children s books, encyclopedias, etc.). However, we sought to prove the appropriateness of our choice by designing a behavioral experiment in which the same shapes as those adopted in the ERP experiment were depicted in 16 different colors, 8 of which were the same as the colors actually used in the ERP experiment (canonical color1), while the other 8 (canonical color 2) were variations of the same hues obtained by changing the values of the RGB color palette in such a way that the resulting color was still judged highly related (canonically) to the associated shapes by the four experimenters. For example, the green used in the ERP study (and in Control Experiment 1) was a Forest Green, while in this control study crocodile and artichoke were paired with two different canonical colors: Green 1 (Forest Green, the hue used in the ERP study) and Green 2 (Light Green, new hue tested in this pilot study). Table 1 lists the 16 colors adopted and the corresponding color-palette specifications. The goal of this control experiment was to investigate whether the colors chosen as canonical in the ERP experiment were indeed less or more typical than other possible, unexplored color hues Participants Twelve right-handed individuals (4 males and 8 females) with a mean age of 29 years participated in this pilot study. Ten of these individuals also participated in Control Experiment Stimuli The stimuli were the same shapes used in the ERP experiment and described in the Methods section of the paper. Stimulus parameters, such as duration, ISI, luminance, etc., were also the same. The only difference was that three new shapes ( monkey, camel and carrot ) were introduced to balance the number of living and non-living entities. The category blue was eliminated because it was composed only of non-living objects ( water bowl and European road Table 1 List of 16 colors adopted in Control Experiment 2 along with their color palette specifications Color name RGB Color palette Red Green Blue White White Yellow Yellow Pink Pink Red Red Orange Orange Green Green Grey Grey Black Black They were generated with Neopaint.

5 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) sign ), and the category orange was introduced instead, with two exemplars, camel and carrot Task In order to have the viewers pay attention to stimulus shape, and to investigate the effect of color-shape pairings on object processing, we developed a task in which subjects were instructed to make a living/non-living categorization as quickly and accurately as possible, by pressing one of two response keys exactly as in the previous pilot experiment Results and discussion RTs were treated as previously described. The data were subjected to a one-way ANOVA, with stimulus color (canonical color 1, canonical color 2, non-canonical colors) as the only experimental factor. Results (displayed in Fig. 1, right) showed that color had no effect on object categorization, which is quite consistent with the result of the ERP experiment. Indeed, electrophysiological data indicated that, although color was processed faster and apparently more easily than shape, it depended on shape processing particularly when the shape was canonically related to target color, while the opposite did not occur. Substantially, the data obtained in Control Experiment 2 confirmed the hypothesis that shape holds a special status in object processing, a view supported by the neuroimaging data reported in the object-processing literature (e.g., Ref. [12]). As for the specific effect of prototypicity, neither of the prototypical color hues had an effect on object categorization, but, if anything, the hues actually adopted in the ERP experiment (canonical color1) tended to produce faster RTs than the ones used in Control Experiment 2, thus confirming our initial assumptions about their typicality. 4. ERP experiment In the present study, healthy right-handed individuals were presented several thousands of pictures of common objects and animals depicted in their canonical color, as well as in various other colors not associated with them. The task consisted in paying attention to one dimension of stimulus (either color or shape) and responding to targets (e.g., Color: RED whatever the shape; Shape: PIGLET of whatever color) while ignoring the other dimension. ERPs time-locked to stimulus onset were recorded and averaged separately as a function of the attended dimension, attentional relevance and stimulus prototypicity Methods Subjects Ten university students (5 females and 5 males) aged from 20 to 25 (mean age = 23) participated in the study as volunteers, but two had to be excluded from further analyses (before averaging) because of excessive ocular artifacts. All were right-handed, according to a self-report laterality questionnaire, had right-eye dominance and normal or corrected-to-normal vision. The purpose of the investigation was unknown to them. Experiments were conducted with the understanding and the consent of each participant according to the declaration of Helsinki [Br. Med. J. 302 (1991) 1194], and with approval of the Ethical Committee of the University of Trieste, Italy. The authors also declare that they have complied with APA ethical standards (1992, American Psychological Association) in the treatment of human volunteer Stimuli and procedure Familiar objects and animals, in picture form and in different colors, were displayed in the center of the screen of a PC monitor. Sixteen different drawings of animals, vegetables and objects were depicted in 8 different colors. One color only was typically related to the shapes and was for this reason named canonical ; for example: yellow for the shape of a banana (see Fig. 1 for a complete list of all shapes in their canonical colors). Thus, each participant was shown a total of 128 different images. Each stimulus was presented 24 times, making a total of 3072 pictures presented to each of the 10 volunteers. The number of colors was limited to 8, namely green, white, pink, blue, red, dark gray, yellow, and black, as too wide a color range would have led to difficulties in discriminating color, compared to the attend-shape condition. Nevertheless, as stimulus probability is known to affect ERP amplitude, the probability of color and shape targets vs. non-targets was kept constant (16/128: p = 0.125). Similarly, the probability of prototypical vs. unrelated stimuli was perfectly matched across the two categories (for both target and non-targets = 1:7). Therefore, any eventual differences between the attend-color and attend-shape condition cannot be ascribed to a difference in stimulus probability. Only one color for each shape was prototypical, so that no one shape could have more than one canonical color. The stimuli, subtending about 4j 4j of visual angle, appeared briefly (for 100 ms) on a light gray background in the central visual field. The inter-stimulus interval (ISI) varied randomly, being between 1 and 2 s. An experimental run consisted in the presentation of 128 stimuli and lasted about 4 min. Between each run, participants were invited to rest for a few minutes. The participants, seated comfortably in a dimly lit and acoustically shielded room, faced a window behind which a high resolution VGA computer screen was positioned 114 cm from their eyes. A small yellow cross (4 mm in size) located at the center of the screen served as a fixation point to minimize eye movement. The subjects were instructed to fixate the center of the screen and to avoid any eye or body movement during the recording session. The task was to pay selective attention to either color (e.g., yellow) or shape

6 292 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) (e.g., lamb) i.e., to one dimension, ignoring the other, of the stimulus, the tested dimension being announced by the experimenter at the beginning of each run. In this way, the stimulus was selected on the basis of color (color selection) independent of shape, or on the basis of shape (object selection) independent of color. Response was the pressing of a button with the index finger of the left or right hand, allowing the recording and measurement of the speed and accuracy of response, viz., reaction time (RT) and percentage of correct responses, respectively. Hand order and attention condition (i.e., attend-color vs. attend-shape) were randomized within and across subjects EEG recording The electroencephalogram (EEG) was recorded continuously from 28 scalp sites using tin electrodes mounted on an elastic cap (Electro-cap, Inc.). The electrode positioning on the scalp was frontal (Fp1, Fp2, FZ, F3, F4), central (CZ, C3, C4), temporal (T3, T4), posterior temporal (T5, T6), parietal (PZ, P3, P4), and occipital (O1, O2) of the International System. Additional electrodes were placed at half the distance between homologous posterior temporal and occipital sites (OL, OR), half the distance between O1 and O2 sites (OZ), 10% of nasion inion distance below OZ (INZ), 10% of nasion inion distance above OZ (POZ), 10% to the left and the right of the POZ site (PO1, PO2), 10% and 20%, respectively, to the left and right of the INZ site (IN1, IN2, IN3, IN4). To ensure that fixation was maintained the horizontal and vertical oculograms (EOG) were also recorded. Vertical eye movement was recorded by two electrodes placed below and above the right eye, whereas horizontal movement was recorded using electrodes placed at the outer canthi of the eyes. Electrode impedance was kept below 5 kv. The reference lead was linked earlobes. The EEG and EOG were amplified with a half-amplitude band pass of Hz. Continuous EEG and EOG were digitized at a rate of 550 samples per second. Trials contaminated by eye or body movement were rejected. Computerized artifact rejection before averaging was used to discard epochs in which eye movement, blinking, excessive muscle potentials or amplifier blocking occurred. The criterion for artifact rejection was a peak-to-peak amplitude exceeding F 50 AV, and the rejection rate was about 5%. ERP epochs were averaged off-line from 100 ms before to 1000 after stimulus presentation. Separate ERP averages were calculated for the two attention conditions (attend-color, attend-shape) and the four types of stimulus (canonical color/shape target, unrelated target, canonical color/shape non-target, unrelated non-target). See Fig. 2b for some examples. The terminology is as follows. An unrelated target associates atypical shape with target color (e.g., Fig. 2b, yellow elephant), or atypical color with target shape (e.g., Fig. 2b, pink artichoke), while in a canonical non-target, the irrelevant dimension (color/shape) is typically associated with relevant color/shape (e.g., Fig. 2b, green hand). Instead unrelated non-target is the term used for no association whatsoever between irrelevant dimension and relevant shape/color, not because there is atypical pairing of color and shape (e.g., the blue piglet of Fig. 2b). When color was the attended dimension (e.g., yellow), the so-called unrelated targets were all stimuli of the yellow class and of any shape except that prototypically associated with that color (i.e., in the given example canonical targets would be a banana or a chick). In the same way, unrelated non-targets were all non-yellow stimuli with unrelated shapes (e.g., a swallow). When shape was the attended dimension (e.g., strawberry), unrelated targets were all strawberries of any color whatsoever except for red (that would make them a canonical target). In the same way, unrelated non-targets were all non-strawberry stimuli (e.g., a piglet) of any color whatsoever except red (that would make them canonical non-targets). A computer program was used in the identification and measurement of the major ERP components, the baseline being the average voltage obtained in the 100 ms prior to stimulus onset. Quantification of the ERP components was done by measuring both peak latency and peak amplitude, and assessing the mean areas within a specific latency range centered approximately on the peak latency of the deflection observed in the grand average waveforms. P1 deflection was identified in the ms time window and measured at posterior sites (O1, O2, OL, OR, T5, T6, PO1 and PO2). The N1 and N2 negative deflections were also identified, being, respectively, in the ms and ms time windows, and measured at OL, OR, T5, T6, IN3, IN4 sites. Fronto-central positivity (FSP) was measured at F3, F4, C3, C4 between 160 and 220 ms post-stimulus. The P300 component was identified in the ms time window and measured at C3, C4, P3, P4, Cz, Pz, Cz and POz Data analysis In the statistical analysis, any reaction times (RTs) exceeding the mean F 2 standard deviations were excluded. The RT data, latency and amplitudes of the major ERP components were subjected to multifactorial repeated-measure ANOVA. The mean area of P1, N1, N2 and FSP components, peak latency measures of P1, N1, N2, FSP and P300, as well as peak amplitude measures of N2 and P300 components were subjected to statistical analyses. P300 peak amplitude measures were used to directly investigate the relationship between amplitude and latency of ERPs and motor response. Also, the percentages of correct responses (Hits), false alarms (FAs) and misses, after data conversion to arcsine estimates, were subjected to ANOVA. FAs were also normalized for stimulus frequency. Behavioral responses were analyzed by means of a two-way repeated-measures ANOVA, the factors for this analysis being attention condition (attend-color, attend-shape) and stimulus prototypicity (prototypical, non-canonical).

7 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Fig. 2. (a) Drawings of familiar objects and animals depicted in their canonical color. All stimuli were also presented in other seven unrelated colors. (b) Example of stimuli represented in their canonical as well as in unrelated colors when targets or non-targets. In the upper row, examples of pictures for each stimulus type are given for the attend-shape condition. In the lower row, examples are taken from the attend-color condition.

8 294 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Table 2 Total percentages (%) of emitted responses as a function of attention condition, stimulus type and prototypicity Stimulus type Attended Dimension Color (e.g., red) Shape (e.g., artichoke) Target Canonical non-target 0.5 (blue CHERRIES) 0.2 (green PIG) Unrelated non-target 1 (white LAMB) 0.6 (pink CHICK) Electrophysiological responses were subjected to fiveway repeated-measure ANOVA. In this case the factors of variability were attention condition (attend-color, attendshape), type of stimulus (target, non-target), stimulus prototypicity (unrelated, canonical), electrodes (depending on ERP component), hemisphere (left, right). Multiple comparisons of means were done by post-hoc Tukey s tests Results Behavioral data RTs to targets were faster for conditions of attend-color (389 ms) than for attend-shape (407 ms), as confirmed by a significant main effect of attention condition ( F 1,9 = 5.27, p < 0.05). Apart from that, there was no other indication in the behavioral data of a significant difference in difficulty for these two tasks, the hit percentage being about 99% for both conditions. False alarm rates were significantly greater ( F 1,9 = 26.5; p < ; e = 1) to canonical than unrelated non-targets, thus suggesting a strict interaction between the processing mechanisms of the two features (see Table 2). Overall, data showed that 1:2 times in the attend-color and 1:3 times in the attend-shape condition FAs were emitted to canonical rather than unrelated non-targets. If one considers that the theoretical probability of erroneously responding to a canonical non-target, on the basis of stimulus frequency, was of 1:7, it appears that the unattended dimension, and shape in particular, affected decision making when color was being attended, even in an extremely simple task as the one used here Electrophysiological data P1 component. No effect whatsoever of attention condition, stimulus type and prototypicity was observed at the processing stage indexed by the P1 component ( ms) N1 component. The ANOVA performed on the amplitudes of N1 component, peaking at about 165 ms, showed the first effect of attention ( F 1,7 = 14.1; p < 0.008) in that N1 was greater in amplitude for target ( 2.6 AV) than for non-target stimuli ( 1.97 AV). Furthermore, the type of stimulus hemisphere interaction ( F 1.7 =9.14; p <0.002) indicated that the attention effects (target/non-target difference) over posterior sites were greater on the left than on the right (see means in Table 3). This hemispheric asymmetry in N1 response was confirmed by analyzing the N1 latency values. Indeed, the significant interaction of attention condition hemisphere ( F 1,7 = 13; p < 0.01) and relative post-hoc comparisons indicated earlier N1 latencies to targets (163 ms) than to non-targets (165 ms), but only at left sites ( p < 0.01). Note also that N1 was earlier for electrode sites on the right (164 ms) than on the left (166 ms) (hemisphere: F 1,14 =9.5; p < 0.05) N2 component. The ANOVA performed on the N2 component mean area values ( ) showed a significant effect of the type of stimulus ( F 1,7 = 29; p < 0.001) and stimulus prototypicity ( F 1,7 = 21; p < ). N2 was much larger for targets ( 1.67 AV) than for non-targets (1.43 AV), and for canonical ( 0.27 AV) than for unrelated stimuli (0.04 AV). The significance of the electrode factor ( F 2,7 =4.15; p < 0.04) can be seen in the greater N2 response at the inion site ( 1.13 AV) than at other sites (lateral occipital = 0.37 AV; posterior temporal = 0.41 AV), especially over the left hemisphere, as indicated by significant electrode hemisphere interaction ( F 2,7 = 4.7; p < 0.03). For this reason, additional ANOVA analyses were performed on the peak amplitude and latency values recorded at only the IN3 and IN4 inion sites. As in the case of the mean area, the N2 peak amplitude was markedly affected by both stimulus prototypicity (canonical = 1.91; unrelated = 1.38 AV) and the type of stimulus ( F 1,7 = 229; p < ), its amplitude being much larger for targets than non-targets ( 3.37 vs AV). The interaction of task attention condition prototypicity hemisphere ( F 1,7 =5.2; p < 0.05; Greenhouse Geisser epsilon = 1) shows a significant difference between the tasks (e.g., attend-color vs. attend-shape). At the left inion site, N2 was much greater for shape unrelated targets ( 4.1 AV) than for color ( 2.28 AV), as indicated by the Tukey s test post-hoc comparison ( p < 0.01). On comparing the means, it was also found that there was a stimulus prototypicity effect at the left recording sites but not at the right ones (see waveforms of Fig. 3). In fact, the N2 to targets was significantly enhanced by stimulus prototypicity, but only at the left inion site and in the attend-color task, not vice versa. Indeed, at the IN3 site during this task N2 had greater amplitude for canonical shapes than for unrelated ones ( p < 0.01). Table 3 Mean amplitude values of N1 component as a function of hemisphere and attention-related stimulus type Hemisphere Stimulus type Target Non-target Left Right Note that N1 values reported were obtained collapsing ERPs across OL, T5, IN3, and OR, T6, IN4 posterior sites, respectively.

9 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Fig. 3. Grand average ERPs to target (canonical = solid line; unrelated = dashed line) and non-target stimuli (canonical = dotted; unrelated = dot dashed line) as recorded at left (IN3) and right inion site (IN4) in the attend-color and attend-shape conditions. Worthy of note is that ERPs to pictures displayed in the shape prototypically associated with target color (such as a yellow chick in the attend-yellow condition) elicited a bigger N2 component, as compared to unrelated targets (such as a yellow swallow) over the left hemisphere. The ANOVA performed on N2 latency also provided evidence of hemispheric asymmetry. Apart from being earlier for targets ( F 1,7 = 18.6; p < 0.004) than for non-targets (273 vs. 280 ms), N2 latency was affected by the attention condition in interaction with hemisphere ( F 1,7 =29; p < 0.002). Post-hoc comparison indicated earlier N2 response to targets (IN3 = 271 ms; IN4 = 275 ms) than to non-targets (IN3 = 280 ms; IN4 = 279 ms), especially at left sites. Fig. 4 shows the scalp distribution of the N2 component for targets in the two attend-color and attend-shape conditions, obtained by plotting color-coded isocontour voltage N2 amplitudes in the ms time window. It can be noted that the negative occipital response was much greater for shape than for color targets, especially over the left hemisphere. When color was the attended dimension N2 displayed, at the peak of its amplitude, two bilateral foci centered at the ventral mesial and lateral occipital electrode sites (IN1, IN3 and IN2, IN4), whereas for shape selection this component was spread more broadly across the bilateral ventral (IN1, IN3, and IN2, IN4), dorsal mesial (O1, O2), lateral occipital (OL, OR), and posterior temporal (T5, T6) scalp sites. To further analyze the effect of prototypicity on ERPs to target colors, difference maps were obtained by plotting Fig. 4. Time series of isocolor voltage maps of brain activity elicited by targets in the attend-color (upper) and attend-shape (lower) selective attention conditions computed between 230 and 280 ms post-stimulus (N2 latency range). Each map represents a 10-ms brain activity period. Note the left-sided asymmetry of the early activation for both conditions.

10 296 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) the values of the difference wave (i.e., canonical unrelated) computed by subtracting the grand-average ERPs to unrelated color targets from those to canonical color targets. The focus of activity observed at the peak latency of N275 was mostly asymmetrically spread toward the left hemisphere at the posterior temporal and parietooccipital areas (see Fig. 5) Fronto-central positivity (FSP). The analyses performed on the mean amplitude of FSP showed the significance of attention condition type of stimulus ( F 1,7 =8; p < 0.01) and type of stimulus electrode ( F 1,7 =5; p < 0.03). Indeed, target/non-target difference (the selection positivity per se) was larger in the attend-shape (3.82 vs AV) than attend-color condition (2.95 vs AV). Furthermore, target/non-target difference was much larger at frontal (target = AV; non-target = 0.7 AV) than central (target = 3.45 AV; non-target = AV) electrode sites P300 component. ANOVA performed on the P3 peak amplitudes showed a strong type of stimulus effect ( F 1,7 =64; p < 0.001); furthermore, the P3 peak also tended to be affected by stimulus prototypicity ( F 1,7 =5.3, p < ). Indeed, regardless of the attended-dimension Fig. 5. Isocolor voltage map of brain activity recorded in the attend-color condition between 235 and 285 ms and obtained by subtracting ERP responses to unrelated targets (e.g., pink crocodiles ) from that elicited by canonical targets (e.g., pink piglets ). Worth noting is the specific activation of the left inferotemporal area probably related to the representation of, or access to, object/color knowledge. (i.e., the attention condition), the findings show P300 to be greater for targets than for non-targets (16.1 vs. 3.8 AV), and for canonical stimuli rather than unrelated (10.2 vs. 9.7 AV). The interaction of attention condition electrode ( F 6,42 =6.27; p < 0.001) revealed the greatest amplitude to be at centroparietal sites (see ERP traces in Fig. 6a and b). P300 was distributed more anteriorly in the attend-shape than in the attend-color condition, the amplitude being, overall, greater in the former than in the latter condition at all considered sites (e.g., at CZ: color = AV; shape = AV). The P300 latency analysis confirmed the earlier occurrence of P300 to color targets (378 ms) than to shape targets (402 ms), evident also from the significant task factor ( F 1,7 = 30.2; p < 0.001). Moreover, P300 was earlier for targets (372 ms) than for non-targets (410), as confirmed by the significance of attention-condition ( F 1,7 = 12; p < 0.01) Discussion In agreement with the behavioral data, the ERP results indicate that the selection of color and shape occurred in parallel, though not independently, and affected the amplitude of the occipital temporal N1 (N165) and N2 (N275) components, as well as the amplitude of the fronto-central positivity (FSP) and central parietal P300. On the whole, compared to color selection, shape selection produced slower RTs, and larger and later N2, FSP and P300 components; this further supports the viewpoint that color selection is faster and easier than shape selection [9,17]. Unprecedented in the literature, however, is the result that color-related selection negativity (SN) was significantly affected by shape content, but not vice versa. When color was the attended feature (e.g., target = yellow) the SN response was significantly greater if the stimulus shape, that was the irrelevant dimension, was canonically color related (e.g., a chick) rather than unrelated (e.g., an elephant) to the target color. This effect was especially evident over the left hemisphere. The opposite pattern did not occur in the attend-shape condition, in which the stimulus color content did not affect target shape selection. The present data also provided evidence of a greater involvement of the left hemisphere in visual attentional selection, as was indexed by the N1 and N2 components. Such findings (left-sided asymmetry) have already been reported in the ERP [8,23,28,29,31,32,38] and neuroimaging literature [6,11], thus supporting the hypothesis of a specific left hemisphere involvement in the discrimination of object features. Furthermore, they are consistent with the findings of a left hemisphere dominance in the attentional selection and processing of local aspects of visual information, such as local elements of hierarchical configurations or high spatial frequency stimuli [29 31,38]. In order to analyze the brain activity related to featurebased attentional selection, ERP waveforms to non-targets

11 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Fig. 6. Grand average ERPs (N = 8) recorded from all the various anterior and posterior scalp sites. ERP traces are relative to the attend-color (a) and attendshape (b) condition and as a function of stimulus type (target vs. non-target). Note the huge P300 response to targets.

12 298 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) were subtracted from ERP waveforms to targets, for both attend-color and attend-shape conditions (see Fig. 7). The difference in the two conditions is quite clear. As already reported [25,35], selective attention to either dimension elicits an early increased positive response at anterior sites, known as frontal selection positivity (FSP). Interestingly, on comparing the two attention conditions, we found this positivity to be earlier, though much smaller when color was attended than when shape was the attended feature. Furthermore, as described in the ERP literature, P300 to target colors was much faster than to target shapes, further supporting the view that overall color selection may be a simpler and less demanding task than shape selection, as also suggested by the RT findings. Consistently, P300 amplitude data showed that it was distributed more anteriorly and was greater in amplitude in the attend-shape than in the attend-color condition. This piece of data may suggest that the attend-shape condition was either more source demanding or needing greater attentional resources than the attend-color one. However, stimulus shape, probably due to its greater or lesser prototypicity, appears to have a strong bearing on color selection, while the reverse does not hold true. The present data demonstrate that when the two features are bound together in a familiar object or animal (e.g., a white lamb), rather than in an abstract configuration like a blue/ yellow checkerboard where they are not arbitrarily related, the spatial content (shape) of the object can affect the color-based selection of that object even when the spatial content (i.e., shape) of the stimulus is task irrelevant. These results suggest a strong interaction between the two mechanisms subserving the selection of shape and color and indeed provide evidence that, at least in some cases, multiple features of an object can be processed automatically as a perceptual whole and not independently from each other. Duncan [7] was one of the first researchers to demonstrate the precedence of object processing over that of individual features. He found that participants were slower to name two features of separate objects than to name two features of the same object. Since then, much evidence has been provided demonstrating that attention can be directed to a conjunction of features (object-based attention) and that object representation is actually encoded during the earliest stages of sensory processing (improperly called pre-attentive). In a similar vein, Valdes-Sosa et al. [37] found that the P1 component of ERPs (100 ms) was affected by objectbased attention even when the stimulus appeared at an unattended space location. Again, O Craven et al. [26] demonstrated with an fmri study that attending to one attribute of an object (such as the motion of a moving face) enhanced the neural representation not only of that attribute but also of another attribute of the same object (for example, the face), thus providing physiological evidence that whole objects are selected even when only one visual attribute is relevant. Much progress has been made in understanding the neural systems implicated in object processing [27], and at present the accepted notion is that object knowledge is represented in the inferotemporal cortex. Indeed, several neuroimaging and neurometabolic studies have shown the existence of separate pathways for object recognition and spatial localization, named respectively the ventral and dorsal streams of the visual system [4,10,12 14,19,36], the former projecting to the inferotemporal cortex (What system), and the latter to the parietal cortex (Where system). Evidence gathered in nu- Fig. 7. Grand average difference waves obtained by subtracting ERPs to non-targets from ERPs to targets (independent of stimulus prototypicity) as a function of the attended dimension: Color (solid line) and Shape (dashed line). This figure highlights the major attention-modulated components, viz., frontal selection positivity (FSP), P300 and occipital selection negativity (SN).

13 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) merous ERP studies corroborates the existence of this dissociation [16,18]. Gerlach et al. [12] provided evidence that object recognition takes place automatically even when it is not directly subjected to selective attention. In their study, PET activations were recorded during a simple shape categorization task (oval vs. round) performed on drawings of both recognizable and. unrecognizable objects. Although subjects were required only to attend to the global shape of the stimuli, data showed that the posterior parts of the fusiform and inferior temporal gyri were more activated when the stimulus was a recognizable rather than an unrecognizable object. Interestingly, additional findings indicated that the anterior part of the left fusiform gyrus (BA 20) and the right inferior temporal gyrus were differentially activated by natural (e.g., animals) vs. artifacts, which suggested that object-related knowledge had been automatically accessed by viewers. The fact that, in our study, shape content affected color processing even when shape was the irrelevant dimension, and notwithstanding the fact that color, as an independent feature, was processed faster than shape, strongly suggests that object processing does not depend on the processing output of independent visual features (in this case shape and color), but is carried out in parallel from the earliest stages of processing within the occipito/temporal pathway, as supported by recently reported neuroimaging findings. Similar to the findings of Gerlach et al. [12] for stimulus categories (animals and artifacts), the effect of shape prototypicity on color processing observed here suggests that automatic access to canonical object color knowledge took place even if not required by the task. The ERP isocontour voltage mapping of the brain activity indicated that the posterior temporal and parietooccipital areas of the left hemisphere were strongly activated by the selection of the color dimension for familiar pictures (i.e., of objects or animals depicted in their canonical color). In fact, these areas appeared to be very sensitive to the prototypicity effect, thus providing evidence of the possible locus in human visual cortex of joint color-shape processing of real objects. Note that in the present experiment color selection was associated with the strong activation of areas not linked specifically to color, but rather to shape processing. This observation is of primary importance, given the consideration that in selective attention paradigms there is the general assumption that the selection of a stimulus feature (e.g., color) modulates the activation of brain regions apparently active during perception of the feature (in this case color), as has been suggested by several ERP and PET studies (e.g., Ref. [15]). However, it would seem that things are not so straightforward and simple; a very interesting PET study [5] recently reported that the cortical regions involved in color perception are not the same as those providing the information linked to object color (object color knowledge). The study appears to have many points in common with the electrophysiological results we have described. Furthermore, some recent neuropsychological cases have shown that semantic information about object color could be grounded in systems quite distinct from those involved in form and function knowledge [24]. While there is little data and some ambiguity in the available neurological literature (e.g., Ref. [22]), Miceli and colleagues [24] have reported a straightforward case of three patients with lesions in the mesial temporal region of the left hemisphere: the patients, despite their essentially normal ability to name colors and define functional properties of common objects ( Is a pencil made of glass?), were unable to decide whether or not a color matched a given familiar object ( Can a lion be red? ). The authors concluded that the inferotemporal cortex of the left hemisphere was specifically involved in representing or accessing color knowledge. These findings are in agreement with the topographical indications derived from our ERP data, despite the obvious ERP limitations in neural source localization due to poor spatial resolution, and possible voltage propagation through volume conduction. Could it be that the structures of this brain region, where surface activity is detectable at the occipito-temporal electrode sites (Fig. 5), are involved in representing or accessing object color knowledge? 4.4. Conclusions The ERP and RT findings of the present study have shown that color selection is faster and probably less demanding than shape selection. Furthermore, it is also suggested that when there is no arbitrary linking of the color of objects and their shape (as with the geometrical shapes or alphanumeric characters mostly used in previous studies), shape selection does not depend on color selection. Instead, we found that the N2 component and its attention-related selection negativity showed the opposite pattern. ERP data for the attend-color condition showed that the brain s response was significantly greater when stimulus shape, the irrelevant dimension, was associated canonically with the attended color; there was no similar pattern in the attend-shape task. Thus, it appears that color does not affect the shape selection of familiar objects and animals, at least not at the N2 component level. Spatiotemporal mapping performed using scalp voltages, measured in the N2 temporal window, identified the posterior occipito-temporal region of the left hemisphere as the possible locus of conjoined color and shape processing for real objects in human visual areas. The agreement between the topographical data obtained here and recent neuropsychological findings describing the left inferotemporal cortex as being involved in object color knowledge is further evidence of such a locus.

14 300 A.M. Proverbio et al. / Cognitive Brain Research 18 (2004) Acknowledgements This research was supported by the Italian Ministry for University and Scientific Research (MIUR) and the National Council of Research (CNR). References [1] T. Allison, A. Begleiter, G. McCarthy, E. Roessler, A.C. Nobre, D.D. Spencer, Electrophysiological studies of color processing in human visual cortex, Electroencephalogr. Clin. Neurophysiol. 88 (1993) [2] L. Anllo-Vento, S.A. Hillyard, Selective attention to color and the direction of moving stimuli: electrophysiological correlates of hierarchical feature selection, Percept. Psychophys. 58 (1996) [3] L. Anllo-Vento, S. Luck, S.A. Hillyard, Spatio-temporal dynamics of attention to color: evidence from human electrophysiology, Hum. Brain Mapp. 6 (1998) [4] C.M. Arrington, T.H. Carr, A.R. Mayer, S.M. Rao, Neural mechanisms of visual attention: object-based selection of a region in space, J. Cogn. Neurosci. 12 (Suppl. 2) (2000) [5] L.L. Chao, A. Martin, Cortical regions associated with perceiving, naming, and knowing about colors, J. Cogn. Neurosci. 11 (1999) [6] M. Corbetta, F.M. Miezin, G.L. Shulman, S.E. Petersen, Attentional modulation of neural processing of shape, color, and velocity in humans, Science 248 (1990) [7] J. Duncan, Selective attention and the organization of visual information, J. Exp. Psychol. Gen. 123 (1984) [8] M. Eimer, The N2pc component as an indicator of attentional selectivity, Electroencephalogr. Clin. Neurophysiol. 99 (1996) [9] M. Eimer, An event-related potential (ERP) study of transient and sustained visual attention to color and form, Biol. Psychol. 44 (1997) [10] G.R. Fink, R.J. Dolan, P.W. Halligan, J.C. Marshall, et al, Spacebased and object-based visual attention: shared and specific neural domains, Brain 120 (1997) [11] A.P. Georgopoulos, K. Whang, M.A. Georgopoulos, G.A. Tagaris, et al, Functional magnetic resonance imaging of visual object construction and shape discrimination: relations among task, hemispheric lateralization, and gender, J. Cogn. Neurosci. 13 (2001) [12] C. Gerlach, C.T. Aaside, G.W. Humphreys, A. Gade, et al, Brain activity related to integrative processes in visual object recognition: bottom up integration and the modulatory influence of stored knowledge, Neuropsychologia 40 (2002) [13] C.L. Grady, J.V. Haxby, B. Horwitz, M.B. Shapiro, et al, Dissociation of object and spatial vision in human extrastriate cortex: age-related changes in activation of regional cerebral blood flow measured with 15O water and positron emission tomography, J. Cogn. Neurosci. 4 (1992) [14] J.V. Haxby, C.L. Grady, B. Horwitz, L.G. Ungerleider, et al, Dissociation of object and spatial visual processing pathways in human extrastriate cortex, Proc. Natl. Acad. Sci. U. S. A. 88 (1991) [15] J.V. Haxby, B. Horwitz, L.G. Ungerleider, J.M. Maisog, P. Pietrini, C.L. Grady, The functional organization of human extrastriate cortex: a PET-rCBF study of selective attention to faces and locations, J. Neurosci. 14 (1994) [16] S.A. Hillyard, L. Anllo-Vento, Event-related brain potentials in the study of visual selective attention, Proc. Natl. Acad. Sci. U. S. A. 95 (1998) [17] S.A. Hillyard, T.F. Münte, Selective attention to color and location: an analysis with event-related brain potentials, Percept. Psychophys. 36 (1984) [18] S.A. Hillyard, L. Anllo-Vento, V.P. Clark, H.J. Heinze, S.J. Luck, G.R. Mangun, Neuroimaging approaches to the study of visual attention: a tutorial, in: M. Coles, A. Kramer, G. Logan (Eds.), Converging Operations in the Study of Visual Selective Attention, APA, Washington, DC, 1996, pp [19] N.M. Horwitz, C.L. Grady, J.V. Haxby, M.B. Shapiro, et al, Functional associations among human posterior extrastriate brain regions during object and spatial vision, J. Cogn. Neurosci. 4 (1992) [20] F. Karayanidis, P.T. Michie, Evidence of visual processing negativity with attention to orientation and color in central space, Electroencephalogr. Clin. Neurophysiol. 103 (1997) [21] J.J. Lange, A.A. Wijers, L.J.M. Mulder, G. Mulder, Color selection and location selection in ERPs: differences, similarities and neural specificity, Biol. Psychol. 48 (1998) [22] C. Luzzatti, J. Davidoff, Impaired retrieval of object-color knowledge with preserved color naming, Neuropsychologia 32 (1994) [23] A. Martínez, F. Di Russo, L. Anllo-Vento, S.A. Hillyard, Electrophysiological analysis of cortical mechanisms of selective attention to high and low spatial frequencies, Clin. Neurophysiol. 112 (2001) [24] G. Miceli, E. Fouch, R. Capasso, J.R. Shelton, F. Tomaiuolo, A. Caramazza, The dissociation of color from form and function knowledge, Nat. Neurosci. 4 (2001) [25] P.T. Michie, F. Karayanidis, G.L. Smith, N.A. Barrett, M.M. Large, B.T. O Sullivan, D.J. Kavanagh, An exploration of varieties of visual attention: ERP findings, Cogn. Brain Res. 7 (1999) [26] K.M. O Craven, P.E. Downing, N. Kanwisher, fmri evidence for objects as the units of attentional selection, Nature 401 (1999) [27] C.R. Olson, Object-based vision and attention in primates, Curr. Opin. Neurobiol. 11 (2001) [28] A.M. Proverbio, A. Zani, Electrophysiological indexes of illusory contours perception in humans, Neuropsychologia 40 (2002) [29] A.M. Proverbio, A. Zani, Visual selective attention to object features, in: A. Zani, A.M. Proverbio (Eds.), The Cognitive Electrophysiology of Mind and Brain, Academic Press/Elsevier, San Diego, 2002, pp [30] A.M. Proverbio, A. Zani, C. Avella, Hemispheric asymmetries for spatial frequency discrimination in a selective attention task, Brain Cogn. 34 (1997) [31] A.M. Proverbio, A. Minniti, A. Zani, Electrophysiological evidence of a perceptual precedence of global vs. local visual information, Cogn. Brain Res. 6 (1998) [32] A.M. Proverbio, P. Esposito, A. Zani, Early involvement of temporal area in attentional selection of grating orientation: an ERP study, Cogn. Brain Res. 13 (2002) [33] M. Rotte, H.J. Heinze, H.G.O.M. Smid, Selective attention to conjunctions of color and shape of alphanumeric versus non-alphanumeric stimuli: a comparative electrophysiology study, Biol. Psychol. 46 (1996) [34] H.G.O.M. Smid, H.J. Heinze, An electrophysiological study of the selection of the color and shape of alphanumeric characters in response choice, Biol. Psychol. 44 (1997) [35] H.G.O.M. Smid, A. Jakob, H.J. Heinze, An event-related brain potential study of visual selective attention to conjunctions of color and shape, Psychophysiology 36 (1999) [36] L.G. Ungerleider, M. Mishkin, Two cortical visual systems, in: D.J. Ingle, M.A. Goodale, R.J.W. Mansfield (Eds.), Analysis of Visual Behavior, The MIT Press, Cambridge, MA, 1982, pp [37] M. Valdes-Sosa, M.A. Bobes, V. Rodriguez, T. Pinilla, Switching attention without shifting the spotlight: object-based attentional modulation of brain potentials, J. Cogn. Neurosci. 10 (1998) [38] A. Zani, A.M. Proverbio, ERP signs of early selective attention effects to check size, Electroencephalogr. Clin. Neurophysiol. 95 (1995) [39] S. Zeki, J.D. Watson, C.J. Lueck, K.J. Friston, C. Kennard, R.S. Frackowiack, A direct demonstration of functional specialization in human visual cortex, J. Neurosci. 11 (1991)