The Role of Color and Attention in Fast Natural Scene Recognition

Transcription

1 Color and Fast Scene Recognition 1 The Role of Color and Attention in Fast Natural Scene Recognition Angela Chapman Department of Cognitive and Neural Systems Boston University 677 Beacon St. Boston, MA Phone: achapman@bu.edu Short Title: Color and Fast Scene Recognition Word Count: 2698

2 Color and Fast Scene Recognition 2 Abstract Recent studies show conflicting results as to whether color plays a role in fast scene recognition. Li et al. (2005) found that natural scenes with color required no more attention to classify than black and white scenes when presented quickly. However, they did not control their stimulus set for color diagnosticity and luminance. This study uses a dual-task psychophysics paradigm to investigate whether naturally colored scenes require less attention to classify than non-colored scenes when the stimuli are color-diagnostic and controlled for luminance. Results of the study show that black and white scenes and abnormally colored scenes required no more attention to classify than naturally colored scenes. However, color conditions do make a significant contribution to the accuracy of each subject s responses. I conclude that color aids in natural scene recognition when it provides information about scene category, but that it is not necessary for successful fast scene categorization.

3 Color and Fast Scene Recognition 3 Introduction The evolutionary advantage of color vision is an open question among vision researchers. While several groups have investigated the role of color in fast natural scene categorization tasks, there is no concurrence among results. Many researchers, using a 'go/no-go' scene recognition task (in which, for instance, subjects are instructed to respond if they detect an animal or food item in a scene), have claimed that the presence or absence of color in a scene does not affect either reaction time or accuracy of recognition (Delorme, Richard, and Fabre-Thorpe, 2000; Li et al, 2005). These studies have been used as evidence that there is an achromatic, magnocellular pathway at work in fast scene recognition, with the color-sensitive, high-resolution parvocellular pathway only useful at longer time scales (ibid.). In contrast, many other studies use fast natural scene categorization, or naming, tasks (for instance, subjects may be forced to decide what kind of natural landscape the scene showed). Using these tasks, the coloration of a scene appears to be important: Goffaux et. al. (2005) find that abnormally colored scenes take longer to categorize than black and white scenes, which in turn take more time to categorize than diagnostically colored scenes. Similarly, Gegenfurtner and Reiger (2000) report that subjects categorize briefly presented colored scenes more accurately than black and white scenes.

4 Color and Fast Scene Recognition 4 Because of these conflicting results, some authors have concluded that there is a fundamental perceptual dichotomy between the recognition and naming tasks (Ostergaard & Davidoff, 1985; Tanaka & Williams, 2001). However, Oliva & Schyns (2000) suggest a different interpretation: Scene recognition or categorization is not the fundamental dichotomy; rather, the differing results are due to differing degrees of color diagnosticity among scenes. In their terminology, color diagnostic scenes are categories such as beaches and forests, the color histograms of which very separate regions of color space. In contrast, color non-diagnostic scenes, such as indoor scenes versus street scenes, cannot be discriminated by their color histograms (Oliva & Scyhns, 2000). Using images carefully controlled for luminance equality and color diagnosticity, they showed that color was important in both the naming and recognition tasks but only when the images were color-diagnostic. This result of Oliva & Schyns is particularly interesting because most of the studies which report no role for color in scene recognition do not test or control the color diagnosticity of their images. For instance, Li et al. (2005) claim that color has no effect on a scene classification task where most of the subject's attention is directed elsewhere. To support this claim, they use a dual-task setup in which the subject must perform an attention-demanding task determining whether one of five randomly rotated T's or L's is different in some way while simultaneously responding when they detect an animal

5 Color and Fast Scene Recognition 5 in a briefly presented peripheral scene (Li et. al., 2005). They find that when the task was repeated with black and white scenes as opposed to color, subjects' performance was unchanged, and they concluded from this that color does not affect the amount of attention required to classify a natural scene. However, they did not investigate the color diagnosticity of their scenes; nor did they present discolored scenes. Thus, according to the findings of Oliva & Schyns (2000), it is questionable whether their result is valid, or if it was merely an artifact of a color non-diagnostic image set. This study aims to clarify the issue by running a task similar to that of Li et al. (2005) using a color-diagnostic, luminance-controlled stimulus set. Method Participants Two intelligent, highly-motivated graduate students were used as subjects for this task. Both had normal vision, one with correction and one without correction, and both report normal color vision. Apparatus Software and Platform The psychophysics experiment was programmed using the Psychophysics Toolbox (Brainard, 1997) in conjunction with Matlab 6.0. Experiments were run using a

6 Color and Fast Scene Recognition 6 laptop with a Windows XP processor; screen size was 8.5 x inches. The monitor was set to the highest color and pixel resolution possible, and screen refresh rate was 60 Hz. Subjects sat with their faces about twelve inches from the screen, which allowed the peripheral task to be six degrees away from the center of the visual field. Stimuli The stimuli for the peripheral task fell into three categories: Forest scenes, coastal scenes, and canyon scenes. The forest and coastal scenes were taken from Aude Oliva and Antonio Torralba s online database, and were the same as those used in Oliva & Torralba (2002). Canyon scenes were culled from miscellaneous, non-copyrighted online sources. All scenes were scaled to 256 x 256 pixels and selected for color-diagnosticity, and each category contained 79 different scenes. Color-diagnosticity of the stimulus categories was determined using the L*a*b* color space, which puts luminance on a separate axis from chroma. The L* axis indicates luminance value, a* ranges from red on the negative axis to green on the positive axis, and b* ranges from blue on the negative axis to yellow on the positive axis. In order to convert the RGB images to L*a*b*, I used the algorithm described in the appendix of Oliva & Schyns (2001); for further details on use of the L*a*b* color space, refer to that paper s discussion. After converting each image to L*a*b*, I plotted a histogram of each

7 Color and Fast Scene Recognition 7 image s projection onto the a*b* plane. Figure 1 shows the normalized sum of the image histograms for each category. Note that the canyon category shows a distinct bias toward the orange quadrant, the coastal category stretches primarily along the blue axis, and the forest category stretches into the yellow-green quadrant. These histograms indicate the general coloration of each category; from these, it is apparent that each has a qualitatively different composition. After verifying color-diagnosticity, I constructed black and white stimuli and abnormally-colored stimuli in a systematic manner, also taken from Oliva & Schyns (2001). Black and white stimuli were simply constructed by setting a* and b* values to zero, retaining only luminance values. This guaranteed that luminance was equal among all color conditions. In order to construct abnormally colored scenes, I used an invertand-swap operator, in which I first set a* = -a* and b* = -b*, then interchanged the a* and b* axes. Thus, the net peripheral stimuli set consisted of three categories, each with 79 normally-colored images, 79 black-and-white images, and 79 abnormally-colored images.

8 Color and Fast Scene Recognition 8 Task Central Task This task used a dual-task paradigm similar to that of Li et al. (2005). The central task has been previously shown to be a good measure of attentional resource allocation (Braun 1994). The central stimuli subtended the central two degrees of the subjects visual field and consisted of randomly rotated T s and/ or L s appearing in five of nine possible locations. The subject s task was to determine whether all the letters were the same, or whether one differed from the others. Each trial is preceded by a fixation cross for one second. The letter stimulus is then flashed for 250 ms before being covered by a mask of F s which are rotated in the same way as the letters they replace. The subject responds by typing either S for same or D for different. Peripheral Task The peripheral task was based on a combination of Li et al. (2005) and Oliva & Schyns (2001). The stimulus for this task consisted of a natural scene flashed six degrees to the periphery of the subject s visual field. The location of the scene was randomly varied between trials; it could appear at the top or bottom, and left or right of center. In the single-task condition, the scene was preceded by a central fixation cross for 1090 ms; it was then flashed for 160 ms before being covered by a noise mask. In the dual-task

9 Color and Fast Scene Recognition 9 condition, the central stimulus appeared after 1 second of fixation, and the peripheral stimulus appeared 90 ms later. For a more detailed description of the dual task paradigm, refer to Li et al. (2005). The subject s task was to categorize the natural scene as a forest, beach or canyon. They did so by typing F for forest, B for beach, or C for canyon. In the dual-task condition, they were instructed to respond to the central stimulus before responding to the peripheral stimulus. Training and Testing Subjects were trained on three conditions: Central-only, peripheral-only, and dual-task. The training stimuli were naturally colored scenes which were colordiagnostic but separate from the testing stimulus set. The subjects ran through training sets of 30 trials each until they could consistently achieve greater than 80% accuracy for all tasks under all conditions. Testing was conducted in three phases. The first phase consisted of two sets of central-only stimuli, with 80 trials in each. This provided a baseline for central task performance. The second phase consisted of peripheral-only stimuli. It contained three sets of normally-colored images, three sets of black and white images, and three sets of abnormally colored images. Each set contained 79 trials. This provided a baseline for

10 Color and Fast Scene Recognition 10 peripheral performance under the normal, black-and-white, and abnormally-colored conditions. The third phase was dual-task. It contained the same number and type of trials as the peripheral-only task, with the difference that the subject was forced to attend to the central task in addition to the scene classification task. Data Analysis Data analysis was performed between trial types for each subject s data. In order to verify significance or non-significance, two statistical tests were used. The first was a two-tailed t-test. The t-test assumes an underlying Gaussian distribution of the data, which I was unable to prove or disprove given the limited number of trials available, so I also used the Mann-Whitney Inverted U test, which makes no assumptions about the underlying data distribution. For both tests I used a significance threshold of Results Results for both subjects are shown in Figures 2 and 3. For all cases, the results of the Mann-Whitney Inverted U test and the t-test were in strong concordance, so there was no need to worry about the underlying data distribution any further. Figure 2 shows the results for the central task. No significant within-subject differences were found for the central task either between single- and dual-task conditions or between different color

11 Color and Fast Scene Recognition 11 conditions. This indicates that subjects were allocating the same amount of attention to the central task regardless of whether they were also performing a peripheral task. The overall central task performance of Subject 2 is slightly lower than that of Subject 1, most likely because Subject 2 spent less time in training sets than did Subject 1. This betweensubjects difference was not significant. Figure 3 shows the results of the peripheral tasks for each subject. These results were substantially more interesting than the central task results. Neither subject showed any significant difference between single- and dual-task conditions. However, both subjects showed marked differences between color conditions, and these differences reached high significance levels for both statistical tests. Both subjects performed much better for normally-colored scenes than they did for black and white scenes, and black and white performance was much better than abnormally-colored scene performance. The fact that all significance tests yielded precisely the same results for both subjects suggests that the effects seen in this experiment are highly robust, even in spite of the small subject population. Discussion The results discussed above reflect an interesting dichotomy, and partially support the hypotheses of both Li et al. (2005) and Oliva & Schyns (2001). The fact that there is

12 Color and Fast Scene Recognition 12 no difference in central task performance under any conditions suggests that the same amount of attention is being allocated to the central task in every condition. Inversely, this suggests that there is no difference in attentional allocation under any peripheral task condition: therefore, the same amount of attention is being used regardless of color condition. This is further supported by the fact that, for the peripheral results, there are no differences between single- and dual-task performances. This result supports the hypothesis of Li et al. (2005), that scene categorization without color requires no more attention than scene categorization with color. However, there is a significant difference between color conditions in the peripheral task. Subjects were better at classifying the normally-colored scenes than they were at classifying the black and white or abnormally colored scenes. This contradicts the results of Li et al. (2005), who found no differences in performance between their colored and black-and-white scenes. This suggest that color does in fact play some role in natural scene categorization when the categories are color-diagnostic, as Oliva and Schyns (2001) postulated. One major difference between the present experiment, the Oliva & Schyns experiment, and the Li et al. experiment is the training paradigm. For Li et al. (2005), subjects were trained for over fifteen hours on stimulus types before testing. Furthermore, they were trained on black and white images before being tested on them.

13 Color and Fast Scene Recognition 13 Thus it follows that they would find no performance differences between the colored and black and white conditions. Oliva & Schyns (2001), on the other hand, used a vocalresponse task paradigm which required no training at all. My study was about halfway between the paradigms, with subjects being trained for about half an hour, and only on the colored stimuli. So, some differences in experimental outcome might be attributable to the difference in training paradigm and the stimuli to which the subjects were exposed or habituated. In sum, it is probably safe to conclude from the present study that, when color information provides additional information about scene category, subjects do rely on it to make category distinctions. This has the character of a learned strategy rather than a perceptual or innate effect. However, it is also safe to conclude from these results, along with the results of Li et al. (2005), that classifying a natural scene with color does not require more attention than classifying one without color, even with luminancecontrolled, color diagnostic stimuli. This could be the case if, as Oliva & Torralba (2002) postulate, the most important visual component of natural scene classification is not the coloration, but the spatial envelope of the scene. Taken together, these results imply a complex interplay between color and attention in natural scene categorization.

14 Color and Fast Scene Recognition 14 References Biedermann, I., and Ju, G. (1988). Surface versus edge-based determinants of visual recognition. Cognitive Psychology, 20(1): Brainard, D.H. (1997). The Psychophysics Toolbox. Spatial Vision, 10: Braun, J. (1994). Visual search among items of different salience: Removal of visual attention mimics a lesion in extrastriate area V4. Journal of Neuroscience, 14: Delorme, A., Richard, G., and FabreThorpe, M. (2000). Ultrarapid categorisation of natural scenes does not rely on colour cues: A study in monkeys and humans. Vision Research, 40(16): Dunai, J., Castiello, U., and Rossetti, Y. (2001). Attentional processing of colour and location cues. Experimental Brain Research 138: Li, F.F., VanRullen, R., Koch, C., and Perona, P. (2005). Why does natural scene categorization require little attention? Exploring attentional requirements for natural and synthetic stimuli. Visual Cognition, 12(6): Gegenfurtner, K.R., and Rieger, J. (2000). Sensory and cognitive contributions of color to the recognition of natural scenes. Current Biology, 10:

15 Color and Fast Scene Recognition 15 Goffaux, V., Jacques, C., Mouraux, A., Oliva, A., Schyns, P.G., and Rossion, B. (2005). Diagnostic colours contribute to the early stages of scene categorization: Behavioral and neurophysiological evidence. Visual Cognition, 12(6): Oliva, A., and Schyns, P.G. (2000). Diagnostic colors mediate scene recognition. Cognitive Psychology, 41: Oliva, A., and Torralba,A. (2001). Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope. International Journal of Computer Vision 42(3): Ostergaard, A.L., and Davidoff, J.B. (1985). Some effects of color on naming and recognition of objects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11(3): Pelli, D.G. (1997). The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 10: Tanaka, J., Weiskopf, D., and Williams, P. (2001). The role of color in high-level vision. Trends in Cognitive Science, 5(5): Yip, A.W. and Sinha, P. (2002). Contribution of color to face recognition. Perception 31:

16 Color and Fast Scene Recognition 16 Figure Captions Figure 1. Image category histograms in L*a*b* color space, projected onto the a*b* plane. The horizontal a* axis ranges from green (positive) to red (negative), while the vertical b* axis ranges from yellow (positive) to blue (negative). On each histogram, lightness corresponds to the number of pixels falling into a given bin. Figure 1a shows the binned pixels of the 80 canyon images, Figure 1b shows the binned pixels of the 80 beach images, and Figure 1c shows the binned pixels of 80 forest images. Note that each histogram covers a different area of the a*b* color space; this indicates that the categories are color-diagnostic. Figure 2. Central task performance for both subjects. No significant differences were found between color conditions or between single- and dual-task conditions. BW Scenes : Black and white scenes; DC Scenes : Discolored scenes. Figure 3. Peripheral task performance for both subjects. Significant differences were found between color conditions, but no significant differences were found between single- and dual-task conditions. Figure 3a shows results for Subject 1; Figure 3b shows results for Subject 2. BW Scenes : Black and white scenes; DC Scenes : Discolored scenes.

17 Color and Fast Scene Recognition 17 Figure 1 (a) (b) (c)

18 Figure 2 Color and Fast Scene Recognition 18

19 Color and Fast Scene Recognition 19 Figure 3 (a) (b)