University of Nevada, Reno. Cortical Representation of Illusory and Surface Color

Transcription

1 University of Nevada, Reno Cortical Representation of Illusory and Surface Color A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology By Andrew John Coia Dr. Michael A. Crognale/ Dissertation Advisor August, 2016

2 THE GRADUATE SCHOOL We recommend that the dissertation prepared under our supervision by ANDREW COIA Entitled Cortical Representation Of Illusory And Surface Color be accepted in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Michael Crognale, Advisor Michael Webster, Committee Member Gideon Caplovitz, Committee Member Grant Mastick, Committee Member Thomas Nickles, Graduate School Representative David W. Zeh, Ph. D., Dean, Graduate School August, 2016

3 i Abstract In order to see the world as we do our visual system constantly performs many amazing feats that are largely unknown to us, the viewers. In order to create and update a seamless and coherent representation of the world, our eyes and brain have to deal with many obstacles that could interfere with object recognition such as occluding blood vessels in our eyes, layers of occluding retinal neurons, the blind spot that is devoid of photoreceptors, objects and shadows occluding other objects, and retinal inhomogeneities. Ultimately, we obtain information that encodes relative brightness and colors that allow us to recognize different objects and surfaces. How is it that we obtain the information to assign color to object surfaces? It has been hypothesized that we predominantly extract information about the color of an object from the spectral contrast at the edges and fill in the remaining areas (if the edge information is consistent with that from a uniform surface). These ideas have been put forth to explain striking visual illusions such as the Cornsweet Illusion, neon color spreading, and the watercolor effect. All of these illusions exist in both color and brightness domains. Thus, even though it may not be intuitively obvious, edges play a key role in our final percept of what color we see when viewing an object. It is said that we perceptually fill in color from edges similarly to how our visual system fills in the blind spot in our eye. Of course it should be noted that spectral information may also be available from within the regions away from the edges and this information may also play an important role in the perception of surface color. The theory that edge information alone determines color and that internal surface information is discarded implies that there is no difference in neural computation of physical surface color and edge

4 ii induced color spreading. Initial studies of this question suggest that illusory colors and actual surface colors may in fact show important differences. One goal of the present study is to employ both psychophysics and electrophysiology to determine under what conditions illusory colors and surface colors are differentiated to provide insights into the fundamental processes involved in surface color perception. If filling-in is indeed important for surface color perception, then how might color fillingin occur? One possibility is that information from the edges might be propagated in a feedforward manner that passively spreads until another edge or contradicting information is encountered. Another possibility is that edge information could be relayed to higher order form centers and then surface colors reconstructed from feedback from those higher regions. The question of feed-forward vs. feedback mechanisms of perception has been debated by scientists since at least the time of Hering and Helmholtz, Hering favoring a bottom up interpretation while Helmholtz argued that our visual system performs unconscious inferences. It is currently appreciated that a relatively sparse sampling of visual information from the world results in a rich visual percept, supporting the idea that much of our vision is actually reconstructed from past experience. Another goal of the present study is to apply electrophysiological methods (hdeeg) to determine the relative importance of feed-forward and feedback mechanisms in surface color perception. This work expands on previous research which developed a method of measuring the watercolor illusion with single channel VEP. This dissertation provides a literature review of the background and significance of the problems, presents preliminary data, outlines the series of proposed experiments, and lastly the results of the proposed experiments.

5 iii Acknowledgements I would like to thank my parents Anthony and Patricia-Radosevich Coia for supporting me through my graduate studies as well as my friends, other family members, and colleagues in the Cognitive and Brain Science program at the University of Nevada Reno. Thank you to Michael Crognale, John Erik Vanston, Gideon Caplovitz, and Annica Aguzzi for helping with the design, implementation, execution, and analysis of experiments. This work was supported by the Bilinski Fellowship and use of equipment maintained by the Centre of Biomedical Research Excellence (COBRE).

6 iv Table of Contents Abstract i Acknowledgments iii Table of Contents List of Tables List of Figures vi vi iv Chapter 1: Introduction General Goals 1 Background and Significance 3 Taxonomies of Filling In 4 How and Why Does Filling in Happen? 5 Chromatic Induction 7 Lightness, Brightness, and Color Constancy 8 Temporal Properties of Filling In 9 Spatial Frequency and Natural Scene Statistics 11 Afterimages, Adaptation, and Edges 12 Neurons and Visual Coding 13 Neurons and Filling In 14 Cortical Modularity of Color and Form 16 Color Centers in the Cortex 17 fmri, Retinotopic Maps, and Filling In 19 The Visual Evoked Potential (VEP) 23 Chromatic Visual Evoked Potentials 26

7 v Attentional Feedback in the VEP 27 Physiological Correlates of Watercolor Effect 28 Chapter 2: Preliminary Data 32 Materials and Methods 32 Subjects 33 Results (onset, electrodes) 33 Results (SSVEPs, electrodes) 35 Estimation of Cortical Activity 37 Results (onset, sources) 39 Results: (SSVEPs, sources) 47 Case Study 50 Conclusions 52 Chapter 3: Proposal 54 Proposed Studies 54 Behavioral Psychophysics 54 VEPs 55 Chapter 4: High Density Investigation of the Chromatic Visual Evoked Potential 57 Psychophysical Procedures 59 Participants 61 Results: Psychophysics 62 HdEEG Recordings 62 Stimulus Presentation 63 Data Pre-processing 64 Results: hdeeg 64 Source Localization 66 SSVEPs 71

8 vi Chapter 5: High Density Investigation of the Watercolor Illusion 74 Psychophysics 76 Results: Psychophysics 77 Results: EEG 78 Watercolor Source Localization 80 Watercolor SSVEPs 81 Discussion 83 Case Study: Deuteranomoly 83 General Discussion 84 References 87 List of Tables Table 1: Chromaticities for Gabors 60 Table 2: Chromaticities for Watercolor Stimuli 76 List of Figures Figure 1: Examples of illusions 3 Figure 2: How filling in may be achieved by the brain 6 Figure 3: SSVEP stimuli 28 Figure 4: Fundamental and 2 nd Harmonic 30 Figure 5: Average onset topography 34 Figure 6: Electrode Diagram 35 Figure 7: SSVEP Topography 36 Figure 8: Average SSVEP amplitudes. 37 Figure 9: Onset Cortical Activity 40 Figure 10: Regions of Interest 41 Figure 11: Latencies and Amplitudes for Regions of Interest. 42 Figure 12: T tests of illusion vs. control onset waveforms for ROIs 45

9 vii Figure 13: T tests for filled control vs. control onset waveforms for ROIs 46 Figure 14: Cortical FFT amplitudes 47 Figure 15: SSVEP 2 Hz amplitudes 49 Figure 16: SSVEP 4 Hz amplitudes 50 Figure 17: Polar Plots: illusion to filled control 51 Figure 18: Polar plots: illusion and align/misalign conditions 52 Figure 19: Multi Gabor Fields 59 Figure 20: Average heterochromatic flicker photometry matches 62 Figure 21: Average waveforms for multi Gabor crveps 64 Figure 22: Timeline of topographic activity for crveps 65 Figure 23: Regions of Interest in Source Localization Study 66 Figure 24: Average source estimates for crveps 68 Figure 25: Comparison of summed activity in early and ventral visual ROIs 69 Figure 26: Average SSVEP waveforms 70 Figure 27: Significance level of T-Circ test results 71 Figure 28: Estimates of 2nd harmonic SSVEP ROI activity 72 Figure 29: Watercolor stimuli 74 Figure30: Average psychophysical matches across observers 77 Figure 31: Average onset responses across observers to watercolor stimuli 78 Figure 32: analysis of latency and amplitude components of watercolor onset VEPs 78 Figure 33: Average source localization for watercolor stimuli 79 Figure 34: Average responses found in ventral visual area 3 (V3v) 80 Figure 35: Average SSVEP watercolor responses 81 Figure 36: Watercolor SSVEP source estimates 82 Figure 37: Comparison of Deutan subject to group average 83

10 1 Chapter 1: Introduction GENERAL GOALS The aim of these studies is to test theories about perceptual filling in by observing both behavioral and electrophysiological responses in humans to physical surface color as well as stimuli that induce perceptual filling in and comparing them to control stimuli which have edges but no induced surface color. AIM 1: Comparison of the strength of illusory hue and brightness The watercolor illusion exists in both chromatic and achromatic (brightness) forms, and previous studies show that the illusory color can be matched to physical color. The chromatic matches generally produce a desaturated hue of the inducing contour while the achromatic matches have reduced luminance contrast compared to the inducer. Are there differences in perceptual magnitude between chromatic and achromatic illusions? Since the watercolor illusion can be induced in achromatic as well as chromatic form, this study aims to contrast match achromatic and chromatic inducing contours and apply those matches to create chromatic and achromatic watercolor illusions. To measure the strength of the induction, the illusions will be matched to physical colors. AIM 2: Comparison of cortical activity during chromatic and achromatic visual evoked potentials It has been known for some time that robust evoked potentials can be elicited from isoluminant chromatic stimuli. It is currently unknown whether the neural generators of the chromatic visual evoked potential (VEP) are the same as those of achromatic VEPs. This aim sets out to contrast match isoluminant chromatic stimuli (Gabor patches) to achromatic stimuli.

11 2 HdEEG will be used to record responses to these stimuli and dipole modeling techniques will be used to estimate the cortical sources of chromatic and achromatic visual responses. AIM 3: Cortical activity during real and filling in of color and brightness The aim here is to determine whether there are characteristic differences in cortical activity recorded from the visual response to physical and illusory surface color and brightness. Are surface colors (real and illusory) generated in early visual areas in the brain, immediately after retinal signals are delivered? Or does the early visual system only respond to edges, with surfaces being filled in later on in image construction? In addition, are there differences in processing of real surface colors versus illusory ones as seen in the watercolor illusion? The matched chromatic and achromatic stimuli measured for each individual in the behavioral experiments will be used to evoke responses for high density EEG (hdeeg) recordings. HdEEG allows for an analysis of activity measured at the level of the electrodes which can also be used to estimate the activity of different areas in the brain using dipole modeling. The responses to illusory color spreading and their matched physical counterparts will be compared to a control stimulus which has contours of the same color and luminance but not surface color. In addition to looking at neural activity of edge induced vs. actual physical surface color, we hope to compare color vs. brightness evoked responses for different regions of interest in the brain. Comparisons will be made in both distribution of the response as well as the time course of the response in different regions. These data are particularly interesting since the source of the different components of the chromatic VEP have not been determined with certainty.

12 3 Background and Significance In order to construct a coherent visual scene, our visual system integrates many different cues such as borders and shading determining the brightness and color of objects in our surroundings. These cues help us recognize objects in different contexts. In certain instances different cues can send conflicting messages resulting in ambiguous percepts, as illustrated in the infamous dress picture that was popularized on the internet recently. Some cues can be exploited in color and brightness illusions, in order to make the appearance of color experienced where really there should be none. Many brightness and color illusions (see figure 1) exploit the role of edges in our determining the color of an area, and it is said that we perceptually fill in the surface color of objects from edges, causing shifts in appearance of brightness and color. Figure 1: Examples of illusions where physical brightness/color differs from perceived brightness/color. A: The Kanizsa triangle: an illusory white triangle is seen occluding three black circles and an outline of another triangle and appears brighter than the background. The perceived edges of the white triangle are created by the contextual cues. B: Simultaneous Contrast: The surroundings of the two smaller identical grey squares causes the one in the black

13 4 square to look slightly brighter than the one in the lighter square. C: The Craik Cornsweet O Brien Effect (CCOE): A sharp luminance edge going down the middle of the square causes the right side of the square to appear brighter than the left side. By covering the border region in the middle the illusion goes away. D: White s Effect: The small grey rectangles can change brightness whether they are placed on the white or black tracks. E: The watercolor effect (WCE): A thin black line traces a thin orange line which causes a bleeding of orange into the interior of each of the columns. F: Neon color spreading: The yellow lines in the middle are continuous with the surrounding black lines, making it appear as if there is a yellow filter covering that area, and we perceive a yellowish tint over the whole region. Taxonomies for Filling In Other examples of perceptual filling in involve filling of the blind spot (see Murakami, 2008, and Komatsu, 2006), filling in of real (Gerrits & Vendrik, 1970) and artificially created (Ramachandran and Gregory, 1991) scotomas, peripheral fading (Troxler, 1804), and retinal stabilization (Krauskopf, 1963). Characteristics such as color, brightness, texture, motion, and depth can be filled in. In a study of filling in of both texture and color, it was shown that first color was filled in and then texture (Ramachandran and Gregory, 1991). This implies active processing of separate mechanisms. Another type of filling in exists where we perceive illusory contours, delineating an object that is being partially occluded (Kanizsa (1976), figure 1A). Illusory contours can also induce color spreading (neon color spreading figure 1F) (see Bressan et al., 1997). This implies that illusory boundaries may be created that later give rise to surface properties. Although all these visual phenomena can be considered filling in, their individual underlying neural substrates may be different.

14 5 Several authors have created taxonomies categorizing these different types of filling in (Pessoa et al., 1998, Weil & Rees, 2011). Pessoa et al. designate different types of filling in into modal vs. amodal completion and boundary vs. featural completion. Amodal and modal completion can be seen in the Kanizsa triangle (figure 1A). Modal completion would be the illusory white triangle seen in the foreground, while amodal completion involves inferring that the black pacmen surrounding the triangle are partially occluded circles. Boundary vs. surface completion is also present in the Kanizsa triangle: the percept of a continuous boundary around the white triangle (boundary) causes the surface of the triangle to appear brighter. Weil and Rees (2011) argue that filling in phenomena should be separated into groups for delayed vs. instantaneous and stimulus dependent vs. stimulus independent. While filling in at the blind spot is stimulus independent (it always happens no matter what is being looked at), the illusions discussed in this proposal are stimulus dependent (in order for color to spread in the watercolor illusion, two lines must be bordering each other), and both are instantaneous in the sense that filling in happens quickly. Examples of delayed filling in are retinal stabilization and Troxler fading (when eye movements are suppressed, objects fade from view), which can take seconds to fade. How and why does filling in happen? A debate exists about whether the filling in processes is an active neural process or if the visual system is just ignoring the absence of sensory input. Some argue that filling in shouldn t be thought of as the brain painting over a surface, but it may just be ignoring the absence of contradictory information. Dennet (2002) brings up the example of neon color spreading illusion and points out the question of whether this mislabeling of color in an area is due to a spreading activity or if there is a single label of color applied to the area. If the brain

15 6 were applying a single color label over an area of space which the retina (or retinotopic maps such as V1) have numerous receptors (or receptive fields), this could be thought of as some form of neural data compression. How does the brain accomplish filling in? Figure 2: Possible explanations of how filling in may be achieved by the brain. A: Feed forward propagation of visual information from lower to higher levels. B: Lateral/horizontal connections combine information to infer context intermediate location. C: Feedback from higher level back to lower level causes filling in of intermediate location. (adapted from Pessoa et al., 1998) Figure 2 illustrates 3 hypothetical models to account for the neural basis of perceptual filling in (see Pessoa et al., 1998, and Spillmann & Werner, 1996). The circles represent neuron receptive fields, and the arrows going from level 1 to level 2 in A represent feed forward activity, where filling in could be completed in a bottom up manner. Figure 2 B represents an alternate mechanism of horizontal connections within a single cortical area, and Figure 2 C illustrates a feedback mechanism where after information is brought from level 1 to level 2, level 2 feeds back information to level 1. The current proposal aims to test these theories using electrophysiological responses to the watercolor illusion. In addition to these different neural models accounting for filling in, there is also a question of whether the induced area is isomorphically filled in (neurons with receptive fields

16 7 within the area are stimulated during filling in) or whether filling in happens on a more symbolic, or representational level, without the need to actually stimulate neurons within the retinotopic maps of the early visual cortex. These are referred to as the isomorphic and symbolic theories, respectively. Chromatic Induction Simultaneous contrast (figure 1 B) occurs with stimuli of low spatial frequencies, where the inducing and induced areas both cover a rather broad range but have a sharp high frequency border. This causes the test patch to contrast with its surrounding area. In most of the other illusions shown in figure 1, the opposite sensation of assimilation, or color spreading, occurs. The test areas appear more similar to the inducing surrounding (the area adjacent to the luminance increment will appear brighter). Researchers have shown that one can cause a shift from contrast to assimilation by increasing the spatial frequency of a square wave grating (Helson, 1963). Both contrast and assimilation are forms of chromatic induction. Although assimilation and contrast are opposing percepts, many of the assimilation illusions are caused by borders with opposing contrast polarity, such as the Craik-Cornsweet O Brien Effect (CCOE) and watercolor illusions. Both of these illusions are caused by a sharp change in contrast polarity at high spatial frequency, usually causing a spreading of the neighboring luminance or color. One principle of the watercolor illusion is that the color spreading is stronger for the line that contrasts less with the background (Pinna 2004). If the inducing contours are equally salient, both can spread within their allotted boundaries. It has been shown that the color spreading of the line that contrasts more with the background can be in the opposed direction of the outer/farther contour (Kimura and Kuroki, 2014).

17 8 Another point to emphasize is the link between color and form. This can be seen in the watercolor effect. In addition to the color spreading more on the side of where the line contrasts less with the background, this side also adopts a figural percept, the other side acting as the background (Pinna, Werner, & Spillmann, 2003). In contrast to the ambiguous figure ground percept that can switch in the face vase illusion, the dominant coloration of one side of the watercolor effect strongly supports a figural percept. Lightness, Brightness, and Color Constancy Factors involved in the percept of surface color include physical properties of the surface as well as properties of the illuminant. To account for this, there are two proposed perceptual dimensions: lightness and brightness (Gilchrist, 2007). While lightness refers to perceived reflectance, brightness refers to perceived luminance. Observing a blank grey background, there are no cues as to the reflectance of the surface: it could be a white object in shade or a black object in light. While these two examples would have the same brightness, they would have different lightness. Our ability to recognize colors of objects under different illuminants is known as color constancy. Cues such as edges, specular reflections, and shading give our visual system clues as to the lighting context, and can change the perceived lightness of a uniform grey area. Inferring the color of a surface from its edges may be beneficial for color constancy and being able to recognize uniformly reflective surfaces under a vast range of illuminations. One explanation of the watercolor effect invokes visual mechanisms involved in processing depth, reflectance, and illumination. Although the watercolor effect appears vividly when the two lines differ in brightness and color, the illusion still exists by juxtaposing purely

18 9 chromatic lines equal in luminance to each other (Devinck, Delahunt, Hardy, Spillmann & Werner, 2005). Conversely, an achromatic watercolor illusion can be created by juxtaposing black and white lines (Cao, Yazdanbakhsh, & Mingolla, 2011). For achromatic illusions spreading is along the dimension of brightness (Cao et al., 2011). Chromatic illusions resemble shifts in saturation of the inducing contour, but shifts in hue have also been observed (Devinck et al., 2005). Temporal properties of Filling In Although the watercolor effect can be matched to physical color, there may be differences in the underlying processes for these different percepts of colors. By modulating the surrounding color or brightness of the simultaneous contrast illusion (figure 1B), one can modulate the color or brightness of the center region, making the illusion dynamically change in time. By comparing this modulation to actually modulating the center color itself, De Valois et al. (1986) found differences in temporal tuning matches for physical vs. induced modulation in both chromatic and brightness domains (De Valois, Webster, De Valois and Lingelbach, 1986). When the test area itself was modulated, the modulation could still be seen at higher frequencies, while the illusory modulation was reported as being slower. At high temporal frequencies of surround modulation, no center modulation was seen. These temporal differences imply an induction process that is sluggish compared to physical contrast modulation. Other researchers have hypothesized that it should take time for filling in to propagate from the edges to the center of the stimulus. By briefly presenting a white disk followed by a mask, Paradiso and Nakayama (1991) demonstrated that while the border of the white disk was

19 10 visible to observers, the interior was black. In other words, a ring was seen instead of a disk. This implies a filling in process that takes time to travel from the edge to the surround for a normal uniform visual stimulus, which enforces the idea that filling in from edges is a general aspect of vision and not just specific to illusions. Increasing the spatial frequency of a modulating square wave grating has been shown to increase the critical flicker frequency of which brightness induction can be seen (i.e. the larger the induced area, the longer it takes to fill in) (Rossi and Paradiso, 1996). Further studies tested the frequency at which reversing Cornsweet illusions (figure 1C) at different spatial frequencies could be viewed and found similar results of increasing critical flicker frequency with increasing spatial frequency (Davey, Maddess, & Srinivasan, 1998). A following study testing a larger range of spatial frequencies showed an inverted u function for achromatic CCOE patterns (Devinck, Hansen & Gegenfurtner, 2007). In other words, at very high spatial frequencies, it actually takes longer to fill in compared to intermediate spatial frequencies. For chromatic isoluminant CCOE gratings, the critical frequency decreases with increasing spatial frequency. This goes against the filling in hypothesis which would predict decreasing critical frequencies for lower spatial frequencies. These differences in results between chromatic and achromatic stimuli seem to be consistent with the differences seen in the achromatic vs. chromatic contrast sensitivity functions (Mullen, 1985). This suggests that if filling in is happening, it is happening at different spatial scales and by different neural mechanisms. Looking at these studies, it appears that although both real surface and illusory brightness/color fill in from the edges, the current evidence shows contrast induced brightness/color (square wave gratings) to be spatially dependent (Rossi and Paradiso, 1996),

20 11 while illusory color (the CCOE effect) depends more on the contrast sensitivity function (Devinck, Hansen & Gegenfurtner, 2007). However Devinck et al. (2007) point out that for one observer, Rossi and Paradiso (1996) found a lower cutoff frequency for the highest spatial frequency. Further studies are needed to establish whether different forms of brightness and color induction have different critical flicker frequencies. Spatial Frequency and Natural Scene Statistics Images of natural scenes like rocks, forests, and landscapes have been shown to obey orderly statistics (Field, 1987). Natural scene statistics show that low spatial frequencies have higher intensities than high spatial frequencies by a function of 1/f. Some have posited that for stimuli like the Cornsweet edge, the visual system may be amplifying low spatial frequencies (Dakin & Bex, 2003). Since the spatial contrasts in the CCOE and WCE consist mainly of only high spatial frequencies, our visual system may be compensating for the lack of low spatial frequencies by filling in the closest color. Dakin & Bex (2003) also show that the CCOE can be nulled by scrambling low spatial frequency components of the image, but not high spatial frequency components. This challenges the filling in theory by proposing a filtering process rather than a spreading. Others have made multiscale filtering models of different spatial frequencies to account for illusions such as simultaneous contrast, grating induction, and White s effect (Blakeslee and McCourt, 1999). Although edges have traditionally been associated with sharp changes in luminance, an analysis of natural color images has shown that there also exist isoluminant chromatic edges in natural scenes independent of luminance edges (Hansen and Gegenfurtner, 2009). Hansen and Gegenfurtner analyzed a database of 700 chromatic images and found that isoluminant edges

21 12 were no less rare than luminance edges, and color information is independent of luminance in that it can be used to analyze images in a separate way. This agrees with other studies linking orientation with color (Clifford et al., 2003, Webster, Switkes, and DeValois, 1990), and also with the fact that isoluminant versions of the WCE and COCE exist (Devinck et al., 2005, Devinck, Hansen & Gegenfurtner, 2007). Afterimages, Adaptation, and Edges The role of edges has also been shown to have major effects on the appearance of afterimages. The location of an afterimage can be modulated by placing the contours used to define the afterimage in different regions (Anstis, Vergeer, & Van Lier, 2012). Additionally, the contrast of a surface can be temporarily erased after adapting to a flickering outline of the surface, an effect known as contour adaptation (Anstis, 2013). These observations show that boundaries are made before filling in can happen (Cox & Maier, 2015). It has also been shown the prolonged viewing of the neon color spreading illusion produces an afterimage that is filled in (Shimojo, Kamitani, & Nishida, 2002). Other studies created filled in afterimages of regions by adapting to reversing watercolor illusion stimuli (Hazenberg & van Lier, 2013).These studies exemplify the importance of edges in surface color perception and edges and illusory surfaces can affect afterimages. Expanding on the initial discovery of contour adaptation (Anstis, 2013), we have done some experiments on contour adaptation and the watercolor illusion (Coia and Crognale, in preparation). We replicated the general finding that adapting to flickering black and white outlines dulls out inner achromatic surfaces, and that the flickering black and white lines do not reduce the contrast (desaturate) of colored surfaces as they do achromatic surfaces. However,

22 13 adapting to black and white flickering lines does desaturate color matches for a chromatic watercolor illusion, while adapting to isoluminant chromatic lines increases the amount of saturation needed for a match. These findings indicate that adapting to black and white flickering lines can be effective for nulling illusory color but not physical surface color, while it is effective for both real and illusory achromatic surface brightness. Neurons and Visual Coding As mentioned earlier, many of the illusions presented in Figure 1 are a result of a sharp change in brightness or color. A starting point for the mechanism behind simultaneous contrast is thought to be lateral inhibition, a property of contrast enhancement stemming from the center surround receptive fields of neurons early on in the visual stream. A firing neuron with a receptive field on an edge will inhibit the activity of surrounding neurons with receptive fields in the surface. Recordings made from neurons in the optic nerve of a cat while viewing border contrast correlate nicely with the perception of border contrast (Baumgartner and Hakas, 1962, as discussed in Spillmann, 2009). Most of the physiological studies on assimilative filling in and filling in of the blind spot have examined early visual cortical areas, such as V1and V2. The studies of Hubel and Wiesel (1962, 1965) recording from cortical cells of the cat created a model of hierarchical information processing of vision, where the functions of neurons in cortical areas exhibit increasing complexity. This is evident in the simple, complex, and hypercomplex cells they found in Brodmann areas 17, 18, and 19. These regions have since been linked to areas being functionally active during visual processes and are now known as V1, V2, and visual association areas, respectively. These different neurons were stated to respond

23 14 primarily to edges but not uniform surfaces, and their initial discovery was in fact an accident when they noticed the cell firing to the edge of the projector slide while they were removing it. The hierarchical model of vision motivated by these studies portrayed neural processes as occurring in a largely bottom up, or feed forward manner (Van Essen, 1983). The primary visual cortex (V1) is where the lateral geniculate nucleus first distributes the retinal signal to the cortex, corresponding to Brodmann area 17. V1 then projects the information to extrastriate visual areas (V2,V3/Brodmann s areas 18 and 19) upstream. An influential idea on the way that the brain processes visual information is that there are two separate cortical processing streams known as dorsal and ventral streams. The dorsal stream runs from the occipital area V1 to upper temporal and parietal areas which have been functionally associated with spatial vision and where things are in space, as well as how we interact with objects (reaching and grasping). The ventral stream goes from V1 into the temporal lobes and is associated with object vision, recognizing shape, form, and color (Ungerleider, Mishkin, and Macko, 1983). Neurons and Filling In While cells in the LGN can respond to uniform surfaces, most of the cortical cells in areas V1 and V2 respond to edges (e.g. von der Heydt, Friedman, & Zhou, 2003). These neurons are also sensitive to orientation and contrast polarity. Von der Heydt and colleagues have searched for color filling in using stimuli that caused delayed filling in of a disk, i.e. Troxler fading. They trained monkeys to do color discrimination and were able to establish that they perceive filling in similarly to humans (Friedman, Zhou, & von der Heydt, 1999). They recorded from neurons in V1 and V2 whose receptive fields were on the edge and surface of the stimulus

24 15 during filling in of the surface. While both isomorphic and symbolic theories would predict a decrease in edge cell activity during filling in (the edge disappears), the isomorphic theory would also predict a change in activity for surface cells. While they found the expected decrease in edge cells during filling in, they failed to find activity in surface cells, which goes against isomorphic theory. They conclude that edge responses and border ownership contribute to surface color. Although surface cell firing does not seem to correlate with perceptual filling in during Troxler fading (as discussed in von der Heydt et al., 2003), other single cell recording studies of brightness filling in have measured uniform surface responses in the absence of sharp edges (Huang & Paradiso, 2008). By selectively studying cells in V1 that gave robust responses to surface brightness changes in time, Huang and Paradiso (2008) found that these neurons tend to have longer latencies when their receptive fields are farther from an edge in a visual stimulus. This implies that the filling in process extracts information from borders and moves inwards. This time course of activity relates to behavioral studies showing that filling in of edge induced illusory colors is a phenomenon that acts on a relatively slow timescale (DeValois et al., 1986, Davey et al., 1998, Rossi and Paradiso, 1996). A recent study has found a difference in the early processing of chromatic vs. achromatic surfaces. A study using a voltage sensitive dye technique measured responses to surface and edge color and brightness in monkey V1 (Zweig et al., 2015). They found an overall edge enhancement in responses for both chromatic and achromatic patterns, but only a later surface response for achromatic patterns. They refer to the isomorphic vs. symbolic debate and state that while their achromatic data could be argued as being isomorphic, the chromatic data cannot, agreeing with the previous findings of von der Heydt et al., 2003.

25 16 There has been at least one study recording from single cells in monkey while viewing physical and illusory brightness variation (square wave grating vs. illusory brightness (COCE)) (Roe, Lu, & Hung, 2005). Single cell recordings in anesthetized monkey show responses in V2 for illusory brightness but not V1, while V1 did respond to the real luminance grating. They conclude that V1 is not a suitable source of the COCE, but V2 may be. A similar finding was also found using illusory contours as stimuli, where responses in V2 were evoked by illusory contours, while responses from V1 were not (von der Heydt & Peterhans, 1989). Anatomically there are feedback projections from extrastriate cortical areas projecting back to earlier visual areas, and the functional purposes of these feedback connections are being explored. While the parietal areas (dorsal stream) may be receiving input from subcortical areas such as the superior colliculus and pulvinar, temporal areas of the ventral stream are more dependent on signals from the V1 retinogeniculatestriate path (Lamme, Super, Spekreijse, 1998). Another study showed that response latencies to visual stimuli took longer to reach ventral areas V2 and V4 than V1 (Schmolesky et al., 1998). One of the motivations of the current proposal is to investigate whether perceptual filling in of surface colors are instigated by feedback activity from extrastriate areas back to early visual areas V1/V2. Cortical Modularity of Color and Form A separate debate exists concerning the cortical mechanisms of vision and whether there are modules in the brain that perform discrete functions and whether certain visual attributes such as color, orientation, motion, and size are processed in segregated manner. Tissue staining studies using cytochrome oxidase have shown certain clusters of neurons stained better, which were called blobs in V1 and bands in V2. Neurons within these blobs and bands

26 17 seemed to have sensitivity to color stimuli, while neurons outside these areas tended to be more selective for orientation (Livingstone and Hubel, 1984). On average neurons in these regions may prefer one visual attribute or another, however the selectivity of neurons to orientation and color is best described as lying on a continuum where each neuron can have a certain amount of color and orientation selectivity (see Gegenfurtner, 2003). Physiologists have categorized color cells in visual cortex as single and double opponent (e.g. Shapely and Hawken, 2011). While single opponent neurons respond to uniform surfaces and are color selective, double opponent cells may be orientation selective, responding to color boundaries and textures. Both of these cell types are thought to exist throughout the cortical visual pathway in order to create the percepts of color and form, and may be analogous to edge and surface selective cells reported from other labs. Color Centers in the Cortex While early visual areas undoubtedly contribute to both chromatic and achromatic vision, ventral-occipital (VO) cortical regions have been shown to be necessary for the perception of color vision in humans. This has been known from neuropsychological studies of cerebral achromatopsia with brain damaged patients. Zeki (2002) refers to observations made by Verrey, describing a case where damage to the fusiform and lingual gyri in the right hemisphere of the brain caused achromatopsia for the entire hemifield. Since these brain lesions are located below the calcarine sulcus, one would expect that they may only affect the upper quadrants of the visual field, given what is known about the retinotopic map of V1. This implies that there is an area in the lower part of the cortex, independent of V1,that controls color vision for the whole hemisphere, implying a color center in the cortex.

27 18 Others have contested the role of area V4 as being a color center in the cortex. As discussed in Shapely and Hawken (2011). Lesion studies in macaque have shown the monkeys only have minor color discrimination deficits after lesions to V4 and found distinct areas in the inferotemporal cortex lesions which caused a monkey analogue to cerebral achromatopsia. These authors speculate that these temporal areas may serve as a memory bank of colors that feeds back to V1. Grossberg has created a model (called FAÇADE: Form and color and depth) theorizing about how the brain accomplishes filling in (see Pinna and Grossberg, 2005). In this model a surface processing stream of neurons in visual cortex can be influenced by a separate boundary stream. These are processed in parallel by the visual system, while the surface stream is thought to travel through the blob regions of the cortex, the boundary stream goes through the interblob regions. These cortical pathways are distinguished from the previously proposed color and orientation pathways. The boundary stream pools over opposite polarities and in doing this lose their information about color. While the boundary stream is invisible, it delineates the areas to which the surface stream will apply color, and sends feedback which the surface stream uses. One aspect of the watercolor illusion is that the edge of the colored boundary (inducing inner line) is weakened by the presence of the surrounding dark boundary, causing the color to spread inwards. This can be explained by spatial competition between the contours, in which the line contrasting more with the background suppresses the boundary signal of the less contrasting inducing contour, causing weakening and spreading of the inner color. This boundary process may be completing itself in V2, which will signal back to V1 to suppress the colored boundary signals (Pinna and Grossberg, 2005).

28 19 As previously mentioned, along with the illusory color and brightness seen in figure 1 examples, these illusions can be thought of as mechanisms for enhancing border ownership and figure-ground segregation. A neuron that shows border ownership is one which will respond differentially when an object (say a square) is placed on one side or another of its receptive field, even when contrast polarity is controlled for. Single neuron recordings in monkeys have found cells that code for figure ground organization and border ownership by somehow accessing information outside of their classic receptive field (Zhou, Friedman, and Von Der Heydt, 2000). These neurons were prevalent in V4 and V2, but also found in V1. Neurons in V1 are more responsive when on a figure, or the edge of a figure, compared to the receptive field being on the background of an image with figure ground properties (Lamme, 1995). These types of neurons may be getting feedback from higher visual areas or have long range connections with other neurons far away on the retinotopic map. When extrastriate regions are lesioned, V1 retains its increased output when on the edge, but loses the increased response when on the surface. This implies some sort of feedback mechanism occurring, which may be responsible for the filling in of the figure. fmri, Retinotopic Maps, and Filling In The spatial scene we see is imaged on our retina to some degree of precision, and the photoreceptors relay that information in the retina to cells with receptive fields, which can act as edge detectors. Retinotopic organization is present in the cortex, and can now be visualized using fmri by seeing how different parts of the cortex prefer different eccentricities and areas of visual space. It has been shown that there are many different retinotopic maps laid out over the cortex (see Wandell & Winawer 2011). All of these visual areas have their own

29 20 representation of visual space. The map of the primary visual cortex (V1) is organized contralaterally (left cortex~right side of visual field), as well as upside down (below calcarine sulcus~upper visual field). While the foveal representation of areas V1, V2 and V3 all converge near one area of the cortex, there are other foveal representations, one in the ventral occipital cortex of humans shown to prefer color to achromatic stimuli (Wade, Brewer, Rieger, and Wandell, 2002). Retinotopic maps are made up of neurons whose receptive fields occupy a portion of the visual field. With increases in the visual hierarchy, the receptive fields of cells gets larger, and there are some brain areas wherein neurons seem to respond to objects regardless of spatial location This implies that the neurons receptive fields are the whole visual field of the organism (Smith et al., 2001). In addition to the retinotopic map, the primary visual cortex is organized into columns representing different spatial frequencies and orientations. Monocular information from both eyes converges in V1 forming ocular dominance columns ranging in preference of eye input ratios. There have been several studies using fmri investigating the activity of different brain regions upon viewing the Cornsweet illusion (CCOE). Perna et al. (2005) performed an fmri imaging study on the CCOE and failed to differentiate illusory brightness edge from their control line in early visual areas. As a control stimulus, they turned the Cornsweet edges into lines which had a luminance gradient but did not reverse in polarity. This control stimulus created an illusion of depth without the illusion of brightness seen in the COCE. Perna et al. (2005) did find dorsal areas (caudal intraparietal and lateral occipital sulcus) responsive for brightness perception differences in theses stimuli, and brings up the possibility of some top down cognitive process from the dorsal stream involved in brightness computations.

30 21 Boyaci et al. (2007), using fmri, did find activity in early cortical visual areas with the presence of the COCE. They compared a COCE stimulus to a square wave control with matched physical brightness. They found significantly greater activation in V1 for physically matched brightness than the illusion. For areas V2 and V3, both real and illusory responses were greater than those for the control stimuli with less perceptual surface brightness. High resolution fmri has shown that the lateral geniculate nucleus responses have been correlated with the CCOE, indicating neural correlates for low level visual processing (Anderson, Dakin, & Rees, 2009). This study used the fact shown by Dakin and Bex (2003) that scrambling low spatial frequencies nulls the CCOE. They showed that a CCOE stimulus that had its low spatial frequencies scrambled produced a smaller BOLD activation in the LGN, compared to high spatial frequency scrambled (which doesn t affect the illusion). These different studies on the BOLD signal from CCOE stimuli all report different regions of enhanced activity, ranging from the LGN (Anderson et al., 2009), V2/V3 (Boyaci et al., 2007), to association areas in parietal/lateral occipital areas (Perna et al., 2005). While Anderson et al. (2009) and Boyaci et al. (2007) argue for a response in early visual areas that corresponds to the percept of illusory brightness, feedback from higher to lower visual areas cannot be ruled out. Although the illusions discussed above exist in both brightness and color domains, the neural correlates have been more thoroughly studied in the brightness domain. One study using fmri has been done on the neon color spreading illusion (figure 1F) (Sasaki & Watanabe, 2004). They found increased activity in the retinotopic area corresponding to the color spreading predominantly in area V1 when attention was controlled by having observers respond to a letter task. When attention was not controlled, increased activity for spreading was found in V1

31 22 through V4. The retinotopic areas corresponding to the inducing contours also activated areas V1-V4 in both illusion and control, indicating a separate contour processing stream. This was seen for both chromatic and achromatic forms of the illusion, indicating a similar mechanism for chromatic and achromatic neon color spreading. It was inferred from this study that when attention is allocated to another task, V1 is still activated by the illusion. In other words, the activity in V1 was not due to attentional feedback, although preconscious feedback could be possible. An fmri study looking at peripheral (Troxler) fading of a disk showed a decrease in activation of early visual areas (V1 and V2), but an increase in more anterior regions (V3A and V4v) (Mendola et al., 2006). This indicates that higher visual association areas are active while early visual areas may be suppressed during this type of perceptual filling in task. It should be noted that in this peripheral fading, luminance contrast is disappearing, while in other types of filling in (the watercolor illusion) color is spreading over a region. These authors also report increased activation of the parietal cortex during Troxler fading. They correlate this finding with fading being more effective behaviorally in the lower visual field, and the parietal cortex being more responsive to the lower visual field. In Troxler fading the filled in area appears to take on the color of its surroundings. Some studies show that actually what is happening is the filled in area is actually a mixture of the surrounding and center colors (Hsieh and Tse, 2010). In other words, when a blue disc is filled in on a red background, the area will actually become purple. Using fmri and pattern classification algorithms, Hsieh and Tse (2010) were able to make above chance predictions about whether an observer was viewing a blue, red, or purple disc, and applied this technique to obtain evidence of the perceptual feature mixing. This feature mixing could also explain why in the watercolor

32 23 effect the induced area is a mixture of the inducing contour and the physical surface. The results of Hsieh and Tse (2010) give evidence for filling in of early visual areas, their effects being strongest in V1. One study failed to find fmri activation correlated with illusory brightness/color perception in early visual cortex. Corniellesen et al. (2006) performed an fmri study with temporally modulating contrast effect similar to DeValois et al. (1986) study and looked at activity of retinotopic areas while modulating center vs. surround brightness. Even though a brightness modulation was perceived in the center when the surround luminance was modulated, no fmri activity was seen during surround modulation in the retinotopic area of V1 representing the center. This is evidence against isomorphic filling in, and they argue that other studies finding evidence for color filling in from edges in early visual areas may be confounded with figure ground and border ownership responses. They do however report extended edge responses and postulate about a mechanism such as filtering from edges that does not require a filling in mechanism. The Visual Evoked Potential (VEP) While fmri studies provide some spatial resolution of specific brain regions, the blood oxygen level (BOLD) signal is slow in time. In other words, it has rather poor temporal resolution when considering the speed at which neurons in the visual stream communicate with each other. While the speed of nerve processes happens on the order of milliseconds, the BOLD signal is on the order of seconds. In order to achieve higher temporal resolution other methods which record electrical activity directly can be used. EEG and MEG, which are recorded noninvasively at the scalp, can give gross potentials of brain activity. The trade off with EEG and

33 24 fmri is with timing and spatial resolution: EEG has rather poor spatial resolution: the exact location of the electrical source of neural activity is unknown. The temporal resolution for EEG is very high resulting in millisecond temporal precision. A study using dense array EEG was done to look at activity generated in response to a brightness illusion known as White s Effect (figure 1 D) (McCourt & Foxe, 2004). This study showed that the earliest measurable response (C1 component) was increased in White s Effect compared to control stimuli, indicating that the illusion may be detected almost immediately in the cortical responses, suggesting feed forward activity. While White s effect is an example of assimilation, where the induced area looks more similar to the inducer, the induced area is of short range compared to the long range spreading of the Cornsweet illusion and the watercolor effect. Although the spatial resolution of EEG is poor, estimates can be made of the conductance of electrical activity on the cortex using dense arrays of EEG/MEG (many recording electrodes). Source modeling techniques use dipoles as model neurogenerators and create head models using structural MRI and estimating the conductivity of the skull and scalp. Then an inverse model can be built which best explains the electrical activity measured from the electrodes, given the head model. Estimates of activity in functional regions of interest which are mapped out using fmri localizers can then be calculated (Cottereau, Ales, & Norcia, 2014). Di Russo and colleagues have estimated the sources for both transient and steady state visual evoked potentials (Di Russo et al., 2001, Di Russo et al., 2006). Transient visual evoked potentials briefly show an image followed by a blank screen. This allows for a signal to return to baseline. The transient VEP can be split up into 3 components: C1, P1, and N1. While the

34 25 source of C1 was found to be V1, P1 and N1 components were found to be in extrastriate areas (Di Russo et al., 2002). The steady state VEP (SSVEP) is an exchange or pattern reversal of two images (most commonly reversing black and white checkerboards). The resulting waveform elicited by this form of visual stimulation is sinusoidally shaped, never returning to baseline due to the repeated stimulation. Di Russo et al. (2006) state this as a disadvantage for SSVEP (compared to transient/onset presentation) because many brain regions may be contributing to the sinusoidal waveform. SSVEP data can be objectively quantified in the frequency domain using the Fourier transform. SSVEPs induced by achromatic Gabor patterns were found to have two sources: one located in V1 and another located near MT/V5, which is an area thought to be involved in motion processing (Di Russo et al., 2006). Using hdeeg in conjunction with dipole modeling, a study of cortical processing of figure ground segmentation showed that figure and background representations have different cortical activations (Appelbaum, Wade, Vildavski, Pettet, and Norcia, 2006). By modulating the figure and background of a stimulus at different frequencies, Appelbaum et al. (2006) were able to frequency tag separate responses for figure and background. Their source modeling showed that while the background frequency was strong over areas V1, V2, and V3, figure responses were dominant over lateral occipital areas (LOC). They cite other studies showing LOC activity in response to line drawings of objects and illusory contours, and state that their results are consistent with cortical feedback from high visual areas back to the early visual areas to enhance figure ground segmentation.

35 26 Chromatic Visual Evoked Potentials Most natural scenes contain both chromatic and luminance contrast, but on a computer monitor under controlled conditions it is possible to create chromatic stimuli that are isoluminant, containing no brightness contrast with the surrounding. While strong visual evoked potentials (VEPs) can be measured using black and white checkerboards, isoluminant chromatic stimuli also elicit VEP signals, referred to as chromatic visual evoked responses (CRVEPs) (see Rabin, Switkes, Crognale, Schneck, & Adams, (1994), Porciatti, & Sartucci, (1999). These studies used sine wave grating stimuli which isolated S (tritan axis) and L-M cone opponent channels along cardinal directions in color space (Derrington, Krauskopf, and Lennie, 1984, Macleod and Boynton, 1979). Furthermore the CRVEP can be measured at low contrasts (Porciatti, & Sartucci, 1999), which is important for trying to measure colors matched to the faint color spreading seen in the watercolor illusion. Important differences have been noted in chromatic and achromatic VEPs. One is that CRVEP responses seem to be lacking an early component that is present in achromatic stimuli (Gerth, Delahunt, Crognale, & Werner, 2003, Rabin et al., 1994) and in the adult, the waveform recorded over the occipital lobe consists of a large negative wave usually followed by a positive component. To generate responses to achromatic stimuli, pattern reversal (exchanging two images repeatedly) stimulation is known to be effective. For isoluminant chromatic stimuli pattern onset (showing color followed by a neutral grey background) generates robust responses compared to achromatic stimuli. At low spatial frequencies, S cone stimuli have been shown to elicit stronger VEP responses than achromatic stimuli (Rabin et al., 1994). Similarly, blurring high spatial frequency stimuli causes a more drastic reduction in achromatic than S cone patterns (Gerth et al., 2003).

36 27 Studies of cortical dyschromatopsia, a condition of incomplete color loss, also imply that the percept of color may be disrupted by damage to ventral visual areas. One piece of evidence comes from a case study of a stroke patient with lesions in ventral occipital regions, but a relatively spared V1 (Crognale et al., 2013). The damaged brain regions were bilateral and included posterior parahippocampal, fusiform, and lingual gyri. This brain damage resulted in a loss of color sensation where colors once known to be vivid to the patient appeared lacking, dulled out sepia tones. An interesting finding was that the early components of the patients visual evoked potentials for chromatic stimuli were normal compared to control subjects. While chromatic VEPs have previously been shown to be reduced in the case of retinal color deficiencies (Crognale et al., 1993) the dyschromatopic patient in Crognale et al. (2013) gave normal chromatic VEPs yet had a dysfunctional color percept due to higher level cerebral damage. It was also noted that the later components of this patient s CRVEP waveforms appeared to be lacking, implying activity occurring in later visual areas or a possible lack of neural feedback from the damaged regions. Importantly although there have been many studies using the CrVEP, the cortical origin of the major components in the response recorded over the occipital lobe is still uncertain. Attentional Feedback in the VEP One cognitive function that has been implicated with neural feedback is attention. By attending to a colored stimulus, its saliency and saturation are increased (Fuller and Carrasco, 2006). Attention has been shown to modulate the BOLD signal in ventral occipital areas more so than early visual areas (Kastner et al., 1998). Studies using the visual evoked potential found attentional effects for steady state achromatic stimuli, but failed to find attentional effects when

37 28 using transient isoluminant chromatic stimuli (Highsmith and Crognale, 2010, Gerth et al., 2003). This implies that the initial wave of signal in the chromatic VEP is independent of attention. Attentional effects on achromatic ERPs have also been reported in later components of the waveform occurring ~200ms after stimulus onset (Flevaris, Martinez, and Hillyard, 2014). Figure 3: SSVEP stimuli. A: Control pattern reversal where two mirror image control stimuli with no surface color are exchanged. B: Illusion condition where illusion (containing illusory surface color) is exchanged with control stimulus. C: Filled control condition where control with physical surface color is exchanged with control with no surface color. D: Same as A without orange lines. E: same as B without orange lines (and hence without illusion as well). Physiological Correlates of Watercolor Effect By using a steady state VEP (SSVEP) presentation, Coia et al. (2014) developed a technique to measure the watercolor illusion. This study involved comparing the frequencies of the VEP response elicited by exchanging pairs of images (illusion vs. control, control 1 vs. control 2, control vs. filled control, see figure 3 A, B, and C). The illusion stimulus was a series of

38 29 columns made of two thin outlines, each having colored lines in the interior bordered by luminance lines on the exterior, while in the control the colored and luminance lines were braided. A consistently stronger fundamental frequency component was found when exchanging illusion with control than when exchanging control 1 with control 2 (a mirror image of itself). In addition, when physically filled color was placed into one of the controls (Figure 3 C) in the control condition, this also produced a measurable fundamental frequency. Each image was presented for 250 ms (4 Hz exchange rate), and the color in the illusion and filled control conditions would appear once per cycle of the two images (2 Hz fundamental). While all 3 conditions produced a strong 4 Hz component due to the image updating frequency, a 2 Hz component was also present in the illusion and filled control conditions, but not in the control (see bottom of figure 4). Figure 4 (top) shows example SSVEP responses for one subject for control and illusion conditions. For this subject, the control condition (figure 4, left) appear symmetric and to be dominated by the 4Hz component (indicated by the 4 uniform peaks and large 4Hz component highlighted in red). The illusion condition (figure 4, right) has a large 2 Hz component compared to the control (highlighted in red).

39 30 Figure 4: Fundamental and 2 nd Harmonic. Top: SSVEP waveforms from one subject for illusion and control. Middle: Frequency tagging the waveform using Fast Fourier Transform (FFT) to objectively quantify stimulus response. Bottom: Results of Coia et al., (2014). Experimental (illusion) and filled control conditions produced larger 2Hz components than the control condition (left). There was no significant difference for the 4 Hz (right). Coia et al. (2014) showed the fundamental 2Hz frequency elicited by the illusion SSVEP to be dependent on the luminance and color of the lines relative to each other and the background. When comparing the steady state VEP conditions, one discrepancy that could be a potential confound is that there is an alignment of the same colored lines in the illusion condition that is not present in the control condition. The control condition is a pattern reversal in the sense that the two images being exchanged are mirror images of each other. In contrast, in the illusion condition, the two images are not mirror images, but rather the lines exchanged go from an aligned to misaligned state. Previous research has used similar stimuli to measure Vernier acuity in infants using SSVEP (Skoczenski & Norcia, 1999). They found that exchanging aligned and misaligned lines produced a strong fundamental harmonic, while exchanging

40 31 misaligned to misaligned did not. Therefore stimuli excluding the inducing lines but going from an aligned to misaligned state (Figure 3 E) should also produce a 2Hz signal. Including this condition may help to disambiguate additional activity hypothesized to be responsible for the illusory color from the alignment/misalignment (Vernier Acuity) response. The watercolor effect can propagate over large distances, and some have proposed that the watercolor illusion may be mediated by long range horizontal connections within the cortex (Werner, Pinna, Spillmann, 2007). Where in the cortex this takes place is unknown. While Coia et al. (2014) demonstrated that the illusion (along with filled surface color) are measureable using SSVEP, single electrode EEG could not give much insight about the possible difference in loci of these signals.

41 32 Chapter 2: Preliminary Data We have collected preliminary data measuring the watercolor illusion using high density EEG with two methods of stimulation: pattern onset (transient) VEPs and steady state (SSVEP). In the pattern onset, stimuli were shown for 250 ms followed by a blank screen for 750ms. There were 3 conditions: illusion, control and filled color. In these pilot studies the filled control and illusion were not contrast matched: a fixed contrast was used for every subject. Therefore any differences between illusion and filled control in these data may be due to differences in perceived contrast (the filled control was made to be more on the saturated side). The onset presentations are analogous to the stimuli in the left side of each column in figure 3(A-C), but instead of being exchanged with the control, were followed by a blank screen (same luminance as background) allowing the evoked potential to return to baseline. For the SSVEPs, two images per condition were exchanged at a rate of 2 cycles per second (see Figure 3). Three 60 second sessions for a total of 180 seconds were recorded for each condition in both onset and SSVEPs. Materials and Methods A 256 channel EGI (Electrical Geodesics, Eugene, Oregon) hydrocel system was used. The data were acquired using NetStation software and stored offline. A photogrammetry system was used to take pictures of the electrode placement on each participant, allowing for warping of a general head model based on subject s head size. Initially, data were run through Net Station software which filtered (band-pass 1-40Hz), segmented, performed artifact detection, bad channel replacement, baseline correction, and averaging. A separate analysis was done using the Brainstorm software in order to perform source localization (Tadel et al., 2011). In order to eliminate blinks and eye movements, a signal

42 33 space projection (SSP) algorithm was used. This consisted of using electrodes under the eyes to detect blinks, which cause massive electric potentials contaminating the data. An algorithm then smoothed the data around the blink, removing these artifacts. Subjects While 8 people were run in pattern onset VEPs, the data from one subject was not included because of insufficient response amplitudes. 8 subjects participated in the color SSVEP conditions (Figure 3 A-C), while 4 participated in the alignment experiment (Figure 3D and E). Results (onset, electrodes) The transient onset presentation allowed us to analyze components of the event related potential (ERP) and compare waveforms of the three conditions. A grand average of selected time points in the topography of the evoked potentials for the 3 different conditions (control, illusion and filled control) are shown in figure 5. These points were chosen corresponding to the prominent components of the waveforms from the oz electrode, which is the active electrode in single channel VEP studies. There is an initial peak around 100 ms (P1) after onset and reaching a low point at around 150 (N1), then peaking at 300 (P2) and 400 (P3) ms.

43 34 Figure 5: Average onset topography of activity recorded for 3 different conditions (left). Average waveforms for Oz electrode over back of head where VEPs are typically recorded. A subsequent analysis on the pattern onset data was done by taking an average of posterior electrodes on left and right hemispheres performing a paired sample t-test for illusion vs. control and filled control vs. control at each millisecond. These data are shown in figure 6. The areas between the curves highlighted in yellow are the areas in which p<.05. Difference between illusion and control can be seen early on (around ms after stimulus onset) in the waveforms for the left and right hemispheres, and in the right hemisphere for the filled control vs. control. The filled control vs. control also shows early difference in the right hemisphere and later differences (around 400 ms after onset).

44 35 Figure 6: Electrode Diagram. Diagram showing grouping of electrodes (top) and resulting waveforms averaged across participants (bottom). Areas highlighted in yellow are areas where paired sample t test p<.05. Results (SSVEPs, electrodes) Steady state VEPs were presented a rate of 2 cycles per second. The data were acquired in the same session and processed in the same way as the onset presentation. An additional frequency analysis was done on the SSVEPs performing a Fast Fourier Transform (FFT). Figure 7 shows averaged 2 and 4 Hz topographies for the 5 SSVEP conditions shown in figure 3. These were derived by calculating average 2 and 4 Hz amplitudes for each individual s electrodes and then averaging across subjects. The alignment and misalignment conditions were run in the same session but always after the first 3 conditions (priority was given to the color experiments). This later recording may have caused the data to be noisier because the electrodes may have dried out. As expected from Coia et al., (2014), the illusion and filled control produced greater 2 Hz activity than the control condition, which is reflected in the red

45 36 seen in the 2 Hz signals but not so much in the control 2 Hz. In line with previous studies on Vernier acuity EEG responses (Skoczenski & Norcia, 1999), the align/misalign condition appears to have more 2 Hz activity than the misalign/misalign condition. Figure 7: SSVEP Topography. Average topographies for 2 and 4 Hz amplitudes in different experimental conditions. In order to quantify these data, the amplitudes of 2 and 4 Hz frequencies were averaged across the same subsets of posterior electrodes as in the onset data shown in figure 6. These data are shown in Figure 8. These data show greater 2 Hz activity for illusion, filled control, and align/misalign conditions compared to control and misalign/misalign conditions, while 4 Hz are approximately all similar in magnitude.

46 37 Figure 8: Average SSVEP amplitudes. Bar graphs show 2 and 4 Hz amplitudes calculating from averaging across electrodes and then across subjects. Error bars represent +- 1 standard error of the mean. The results shown in figure 8 indicate that the 2 Hz component seen in the illusion condition may be due to alignment of the lines and not the percept of illusory color. The way that we propose to control for this confound of alignment is by changing the SSVEP stimuli: instead of exchanging from illusion to control, we will in the proposed study exchange two complementary illusions in a pattern reversal manner. Estimation of Cortical Activity While the activity recorded from the electrodes represents activity recorded from the scalp, estimates of cortical activity was made to see if different brain regions at different times and intensities for the different experimental conditions. Neural models of the watercolor effect have speculated that the illusion may be caused by neural feedback from higher visual areas to lower ones, which attributes color to regions that aren t receiving bottom up color

47 38 signals (Pinna and Grossberg, 2005). This hypothesis could be tested by estimating the activity of regions of interest and looking at the time course of activity for the illusion and comparing it to the control and filled control conditions. In order to estimate activity of cortical areas, a computer model of each individual s head was created using Brainstorm as well as the OpenMEEG BEM software (Gramfort et al., 2010 / Kybic et al., 2005). Since the experimenters did not have individual MRI data for 7 out of the 8 participants in this experiment, source estimation was done on a generalized head model (Colin27: MNI brain with a 1mm resolution). Panoramic pictures of each subject were taken after each recording session while wearing the electrode cap and were used to generate a 3-D head model of the individual. This 3-D model was later used to morph the general head model to give a more precise individualized model. The envelope of the cortex surface was represented by vertices, each having and X Y and Z orientation. Building this head model is referred to as solving the forward problem of how the electric signals in the brain propagate through inner skull, and outer skull and scalp to the electrodes. Each of these three layers was modeled with 1922 vertices. This head model assumes that the signals being measured by the electrodes come from pyramidal cells aligned parallel to the surface of the cortex. The surface of the cortex is not always perpendicular to the scalp, as can be seen by the presence of sulci and gyri, so activity of a certain brain region isn t always going to be under the closest electrode, per se. In order to estimate the noise coming from each sensor, chunks of baseline EEG recording were used to calculate a noise covariance matrix. This is used to solve the inverse problem, that is, estimate the most likely source of activity from the electrode recordings. This

48 39 experiment used a minimum norm estimate (Baillet et al., 2001) to approximate the activity of certain regions of the surface of the cortex. The measured electrode responses were transformed into a source space to give an estimated average evoked response for a given brain region in a 3/d model. Results Onset (sources) The same data as presented earlier were used in the following analyses. In addition to the head models created for each individual subject, a grand average across subjects was made and fit to a generalized head model. Figure 9 shows average cortical activity on a generalized 3D head model. The red and green outlines represent Brodmann Areas 17 and 18 which correspond to V1 and V2. In figure 8, activity appears to begin in all 3 conditions in the right hemisphere, lateral to V1 and V2. It then propagates to these areas in the right then left hemispheres, respectively. Although the standard model of cortical activity presumes visual activity to begin in V1, there are other subcortical visual pathways (one through the superior colliculus to the medial temporal lobe) that may be contributing to the early activity seen in the lateral occipital/parietal areas. This has been proposed as an explanation for blind sight in which patients have damage to V1 yet can still perform unconscious visual functions such as motion detection (see Cowey, 2010). For the N1 and P3 components the activity seems to be localized quite precisely to the V1/V2 areas.

49 40 Figure 9: Onset Cortical Activity. View of posterior cortical activity at temporal points of interest for different onset VEP presentations In addition to these early visual areas V1 and V2, some other regions of interest include areas in the dorsal and ventral streams thought to be associated with vision. Regions of interest (ROIs) were selected using the Desikan-Killiany brain atlas (Desikan et al., 2006). In addition to Brodmann areas V1 and V2, four other general regions of interest were picked: inferior parietal cortex, fusiform gyrus, lingual gyrus, parahippocampal gyrus (see figure 10). The inferior parietal cortex is in the dorsal stream and appeared to be where the activity in the grand

50 41 average source estimates started. The other three regions are in the ventral stream and are of interest because they are areas reported to be damaged in achromatopsia/dyschromatopsia patients (Zeki, 2002, Crognale et al., 2013) and are located in regions which are associated with color vision in fmri studies (Wade et al., 2002, Brewer et al., 2005). The fusiform gyrus has also been shown to be activated more in people with color grapheme synesthesia (see Hubbard and Ramachandran, 2005). Figure 10: Regions of Interest. Posterior view of extrastriate regions of interest where analyses were performed. Activity within an ROI can be averaged across dipoles to create a waveform of activity for that region. The amplitude and latency of the first three components (see figure 5) were recorded for each condition of each participant in each ROI. The averages of these values for ROIs in both hemispheres are shown in figure 11. The top 3 rows of graphs show the latencies of the components for each ROI averaged across subjects. The lower two rows are normalized ratios of the differences in amplitude between P1 and N1 and N1 and P2 which were calculated using the following formulas:

51 42 If the amplitudes are above zero, it means that a given condition has a greater amplitude than the control condition on average. Figure 11: Latencies and Amplitudes of onset waveforms for different regions of interest (ROIs). Error bars represent +-1 standard error of the mean.

52 43 Looking at the P1 latency for left hemisphere ROIs (top left), a gradual increase in latency can be seen going from left to right, indicating feed forward processing (Figure 11). The filled control left hemisphere P1 latencies appear slightly quicker than the control and illusion latencies, but overall the latencies of the three conditions appear similar within a given ROI. The P1 latencies for the right hemisphere appear similar to the left, except for the latency of the fusiform region are faster for all three conditions, being closer to V1 and V2 than the parahippocampal regions. There doesn t appear to be any significant differences in amplitudes of these early components for illusion or filled control in V1, V2, inferior parietal, or lingual gyrus. The illusion and filled control amplitudes do not on average differ from the control in these areas. Differences in P1 amplitude can be seen for the filled control condition in the fusiform and parahippocampal gyrus, indicating that these regions are responding to filled in color in the early component of the VEP. Both illusion and filled control show larger N1 to P2 amplitudes in the right fusiform gyrus. These data indicate that the filled color response elicits an increased early response (P1-N1) in the fusiform gyri, while the illusion is eliciting a larger later response (N2-P2) in the right hemisphere. In addition to looking at the components of the waveforms for the different ROIs, t tests were also computed for each time point within the ROI waveforms, as was done for the electrode data in Figure 6. Because some of the onset data for different ROIs had flipped polarity, all waveforms were flipped on their axes so the 2 nd component (n1) went negative, and then waveforms were averaged across subjects for each ROI. Figure 12 shows a comparison of illusion to control waveforms for the 5 different ROIs presented earlier. For V1 and V2, the illusion and control waveforms are nearly identical in the early stages (0-200 ms), which are consistent with the analysis of the early components for these areas in figure 11. There are

53 44 some differences in the later stages of the waveform ( ms), indicative of later processing, possibly feedback. Significant differences can be seen early on in the right inferior parietal cortex. These differences start happening right at the stimulus onset, which may indicate either they are anticipatory or happening very late (~1000ms after the previous trial). Figure 13 shows the comparison of filled control to control. In this comparison differences can be seen early on in right hemisphere V1, V2, and parietal cortex, as well as in the left parahippocampal cortex. Significant differences can also be seen later on in the waveforms, similar to the illusion vs. control analysis. These data indicate that (at least in the right hemisphere) the filled control is activating early visual areas V1 and V2 early on in the visual process, contrary to the illusion, which is only activating these areas preferentially over the control later on in time. The differences seen in the filled control in the left parahippocampal region are not present in the illusion condition either. It must be mentioned that these differences in effects could be due to increased perceived contrast of filled control than illusion. One possibility is that the differences seen in the onset data at the electrode level for illusion vs. control could be coming from the parietal areas and not V1 and V2, while differences in the filled control vs. control could be coming from V1, V2, and/or the parietal regions.

54 45

55 46 Figure 12: Results of paired sample t tests of illusion vs. control onset waveforms for ROIs. Areas highlighted yellow represent p<.05. Figure 13: Results of paired sample t tests for filled control vs. control onset waveforms for ROIs. Areas highlighted yellow represent p<.05.

56 47 Results: SSVEPs (sources) Source estimation of the same regions of interest as the transient onset VEPs were investigated for the SSVEP data. The data were analyzed by binning each condition into one second intervals and taking the Fast Fourier Transform of each one second interval and averaging the amplitudes and phases of the fundamental and second harmonics of each ROI. While amplitude corresponds to the peak to trough difference in voltage, phase is a measure of the relative latencies of the peaks and troughs. Figure 14 shows grand average two and four Hz amplitudes representing the topography shown in figure 7. These cortical estimates reflect the relative activity seen in the topographical representation in figure 7 in that the control shows relatively no 2 Hz activity compared to the illusion and filled control conditions. As with the electrode data, the align/misalign condition does appear to have more 2Hz activity than the misalign/misalign condition. All conditions show 4 Hz cortical activity. This was to be expected since it represents the rate at which the image is updating. Figure 14: Cortical activity estimates generated by inputting average FFT amplitudes for each electrode across subjects and performing source estimation on these average values on a general head model. Original electrode topography data are shown in figure 7.

57 48 While the 2 Hz activity for the filled control condition appears to be localized with V1 and V2 areas, the illusion and align/misalign 2Hz frequency appears to have more activity in the parietal/lateral occipital areas. In order to compare activity across conditions within the aforementioned ROIs, Fast Fourier transforms (FFTs) were computed on the raw SSVEP data after being transformed into source space. The 2 and 4 Hz amplitudes were extracted from the frequency analysis and averaged across participants, the amplitudes shown in figure 14. The alignment experiment data are plotted separately due to the different baseline levels of noise observed. The results for the SSVEP frequency analysis (figure 14 an15) show increased 2 Hz amplitudes for the illusion and filled control compared to the control bilaterally in V1 and V2. At the lingual gyrus the 2 Hz appears to be greater in the illusion than the control as well. The 2 Hz signal for the alignment/misalignment condition appears to be consistently larger than the misalign/misalign condition across most ROIs. While the heat maps in figure 14 were generated by importing 2 and 4 Hz amplitudes (averaged across participants) from each electrode into one general head model, the graphs in figure 11 were calculated by running individual subjects raw data through the model and doing an FFT on the source data. This may be why increased 2 Hz parietal activity is seen in the heat map (figure 14) of the illusion condition but not in the graph (figure 15). Another possibility is that the inferior parietal ROI is much bigger than the small spots of activity within them and these isolated regions of activity are not sufficient to drive up the signal of the whole region.

58 49 Figure 15: SSVEP 2 Hz amplitudes for regions of interest. Error bars represent +-1 standard error of the mean. In figure 16, the average 4 Hz amplitude across participants is plotted for the SSVEP illusion experiment. While the 2 Hz amplitudes showed an increase in the illusion and filled control for V1 and V2, no significant differences are seen in the 4 Hz amplitudes for these ROIs. For the lingual and parahippocampal areas, the 4 Hz component seem slightly smaller for the illusion compared to filled control, but this result is not clearly interpretable.

59 50 Figure 16: Average amplitude of 4 Hz frequencies within ROIs. Error bars represent +-1 standard error of the mean. To summarize these results, the illusion, filled control, and align/misalign conditions show larger 2 Hz activity in early visual areas V1, V2, and lingual gyrus, while the 4 Hz activity does not show any strong differences across conditions. The distribution of activity across ROIs for the three conditions showing greater 2 Hz activity are not clearly distinguishable from each other (i.e. the illusion, filled control, and align/misalign do not differ obviously from each other. Case Study One of the observers in the preliminary data set was part of the UNR database of people who had been retinotopically mapped using fmri. This observer s data was therefore used in an in depth analysis of activity in some functionally defined areas of visual cortex V1, V2, V3, hv4,

60 51 and ventral occipital areas 1 and 2 (as defined by Brewer et al., 2005). Figures 17 and 18 show polar plots comparing illusion to filled control (figure 17) and illusion to align/misalign (figure 18). These data are from the right hemisphere V1 as defined through retinotopic mapping, and are exemplary of all the early visual areas examined by this subject. T-circ tests were conducted to determine if the frequency components significantly differed between these conditions (Victor and Mast, 1991). For the illusion and filled control, both 2 and 4 Hz components were significantly different in this ROI, while for illusion and alignment/misalignment, neither component significantly differed across conditions. The main point here is that significant differences are seen between illusory color and real color, but not illusion and alignment. Figure 17: Polar Plots for one subject s SSVEP 2 and 4 Hz components comparing illusion to filled control conditions. Note that the conditions show significant differences for both frequencies.

61 52 Figure 18: Polar plots for one subject SSVEP 2 and 4 Hz components for illusion and align/misalign conditions. Note: neither component significantly differ from one another. Conclusions These data show a few interesting findings. For the onset VEPs there is no evidence for differences between illusion and control stimuli in early visual areas V1 and V2, although both conditions show strong response in these areas. Differences in the filled control vs. control can be seen in early components of the waveforms in the right hemisphere V1 and V2. The parietal area seems to be selectively activated early on for both illusion and real color, and this is a potential source for the illusion (see Perna et al. 2005). In addition, all 3 onset conditions seem to show a feed forward activity moving from early occipital areas to more ventral areas like the parahippocampal regions, which may be due to the contours and not necessarily the surface color. These ventral regions have been previously implicated in color processing (Zeki, 2004, Crognale et al., 2013, Brewer et al., 2005). The SSVEP data when analyzed in the frequency domain show larger 2 Hz amplitudes for illusion and filled control conditions compared to control, with the largest difference being in

62 53 early visual areas V1, V2 and lingual gyrus. This contrasts with effects seen in the onset data, but may be explained by the alignment response that is present in the illusion, which is present even when there is no illusory color spreading. The alignment response is just as strong in these early visual areas and with our current analyses we are unable to distinguish responses of illusion from those of alignment.

63 54 Chapter 3: Proposal Proposed Studies The proposed studies aimed to build off the methods used in the preliminary experiments and collect hdeeg data on participants who have been scanned and retinotopically mapped using MRI. This will allow us to constrain our regions of interest to functional visual areas as well as have a more accurate head model. There were 10 participants in the subject pool of scanned subjects and eight of these subjects were run in the following month. An additional question we wanted to ask in the proposed studies is whether differences in processing can be seen for chromatic and achromatic illusions, as well as their physically matched counterparts. Differences in VEP responses to chromatic and achromatic stimuli have been shown in previous studies (Rabin et al., 1994), and the source of these differences is unknown. Behavioral Psychophysics While the preliminary stimuli used the same filled control color for all subjects, the proposed study aimed to measure individual color matches to the illusion and use those individual matches in the VEP stimuli, as was done in Coia et al. (2014). A more accurate comparison of filled color to illusory color can be made if they are of equal perceptual magnitude. Instead of using black and orange inducing lines, the proposed experiments aimed to test 3 different opponent pairs of contours: those differing only in luminance (achromatic), and those along the 2 cardinal directions of color space (S-(L+M) and +-(L-M) defined by Macleod and Boynton (1979) and Derrington, Krauskopf, and Lennie (1984). These patterns were

64 55 contrast matched for equal salience (Crognale & Switkes, 1999). This involved matching contrast of the opponent pairs to each other. Next once the contours had been contrast matched the illusions were matched to a physically filled in control pattern in a color matching experiment. Inducing patterns were +S (violet) bordered by S (chartreuse), + (L-M) (pink) bordered by (L-M) (green/teal), and - (L+M+S) (black) bordered by + (L+M+S) (white). These three colored patterns have been previously shown to elicit illusions that can be behaviorally matched to physical color (Devinck et al., 2005, Cao et al., 2011, Coia et al., 2014). In addition to contrast matching watercolor illusion stimuli, Gabor Patches were contrast matched along the same axes in color space in order to create stimuli for traditional chromatic VEPs (Rabin et al., 1994). Flicker photometry was employed to measure isoluminance settings for individual observers. VEPs The contrast matched contours and surface colors were used to create custom VEP stimuli for each individual. SSVEP stimuli were pattern reversals, which will avoid the Vernier acuity issue raised in the preliminary data. The control condition will be the same, but the illusion and filled control will be different in the sense that they will be exchanging complementary surface colors, instead of being exchanged with uncolored control stimuli. We will therefore only be examining the 2 nd harmonic, as the surface color will be exchanged with the image update. Stimuli will be presented in 60 s blocks with 3 illusion, 3 filled control, and 3 unfilled control, totaling 9 minutes. Pattern onset stimuli had 3 illusory color conditions, 3 filled

65 56 controls, and 3 non filled controls for a total of approximately 9 minutes recording time. The additional one line control experiment will have 6 conditions, taking 6 minutes. Traditional crveps of Gabor patches onset and reversals (3 each) will take 6 minutes. The total VEP time should be approximately 30 minutes. The same analysis was performed as was done on the preliminary data set, except the ROIs will be customized according to individuals retinotopic mapping. These studies will contribute to our understanding of the difference in neural processing involved in chromatic and achromatic stimuli as well as physical vs. illusory brightness and color.

66 57 Chapter 4: High Density Investigation of the Chromatic Visual Evoked Potential The study of color vision combines the disciplines of physics, biology, psychology, and philosophy. Curiosity about mixing and matching colored light showed that most humans can combine amounts of three different lights to match any given colored light (Young, 1802). This then led us to find systematic differences between people in the sensation of different wavelengths of light, which later led to the discovery that different genetic sequences of photopigments in our photoreceptors can cause abnormal or color deficient vision at the level of the eye (Nathans et al. 1986). Other forms of color vision loss are caused by damage to regions of the cerebral cortex, distinct from the primary visual center (V1) (Meadows, 1974). These are known as achromatopsia and dyschromatopsia, which are the complete or partial loss of color vision with a sparing of achromatic (black/white) vision. Studies of color vision have characterized different levels of chromatic processing, one notably opponent processing, which is apparent in the experience of visual afterimages. Cone opponent cells exist in the eye and lateral geniculate nucleus going to the cerebral cortex. Neuroscientists have employed various methodologies in the quest to understand the neural mechanisms underlying color vision and the brain regions involved in these computations. Studying color vision at the level of spiking of individual neurons have revealed the existence of neurons with different properties towards chromatic stimuli (DeValois, Abramov and Jacobs, 1966, Livingstone and Hubel, 1988, Johnson, Hawken and Shapley, 2001, Wachtler, Sejnowski and Albright 2003). Others have used fmri as a tool to investigate brain responses to color

67 58 (Engel, Zhang, and Wandell, 1997, Zeki and Bartels, 1999, Wade et al., 2002, Brewer et al., 2005,). Electroencephalography (EEG) is another useful tool for probing the neural mechanisms of color vision by recording the gross potential of brain activity from the scalp (Regan, 1973, Kulikowski, Murray and Parry, 1989, Rabin et al., 1994). The EEG response to visual stimuli is called a visual evoked potential (VEP), crvep specifically for color. While the spatial resolution of EEG is limited, advancements have been made in estimating where the sources of EEG activity are by modeling the surface of the cortex as a series of dipoles and focusing on functionally defined areas(di Russo et al., 2002, Cottereau, Ales and Norcia, 2015). These emerging source localizing methodologies incorporate anatomical and functional magnetic resonance images into the analysis and source estimation of EEG activity. This series of studies probed the inner workings of electrocortical responses to opponent colors using high density electroencephalography (hdeeg). 3 color combinations were used defined along a luminance axis (L+M, achromatic), a red green (L-M, cyan and magenta), and blue yellow (S-(L+M), violet chartreuse). The color space used to define these colors varied along cone opponent axes (see Derrington, Krauskopf and Lennie, 1984, MacLeod and Boynton, 1979). Although it does not appear that strict adherence to these cardinal mechanisms is maintained at the cortical level (see Shapely and Hawken, 2011), the L-M pathway is a good representation of the color response in general, and the S-(L+M) pathway may have interesting properties since it is known that the S cones do not feed into all pathways equally and may contribute little to motion and luminance computations.

68 59 Psychophysical Procedures Before the electrical recordings were made, 2 psychophysical measures were taken in order to produce the stimuli. The first was heterochromatic flicker photometry (HFP). This was done on the two chromatic patterns in order to ensure that they were isoluminant. Participants were instructed to use computer keys to adjust the relative luminance of the opponent colors until the flickering was minimized. This procedure was done at 5 eccentricities and repeated 3 times for each color pair, then averaged at each eccentricity. The stimuli were similar to those used in Skiba, Duncan, and Crognale (2014). They were full field (20 deg) multi-gabor fields, consisting of 59 horizontally oriented Gabor fields. When viewed at 57 cm the Gabors subtended 2 cycles per degree in the center and 0.5 cycles per degree in the periphery (see figure 19). The colors were presented in the rings of five different eccentricity levels which covered the entire computer monitor. The average background chromaticity in CIE 1931 space was [x=.313, y=.322]. The luminance of the entire pattern was held at 18 cd. The chromaticity of the S-(L+M) Gabors was held constant and ranged from a max S [ ] to min S [.343,.39] with individual applied isoluminant settings (see table 1 for chromaticities). Figure 19: Multi Gabor fields along three cardinal directions in color space

69 60 The second experiment was suprathreshold contrast matching (Switkes and Crognale, 1999). This involved having participants view 2 opponent color pairs in temporal sequence and making judgments about which pair, the 1 st or 2 nd, was more visually salient. While the S-(L+M) pattern was held as a constant reference for all participants, the L+M and L-M stimuli were modulated along their axes along fixed steps using the method of constant stimuli. There were 8 steps ranging along the L+M and L-M axes (see table 1). These steps were then counterbalanced with the S-(L+M) reference as being presented either first or second. The outputs of the counterbalanced responses were than averaged, then fit to a curve from which the point of subjective equality was derived from the psychometric function. Most participants produced a relatively smooth psychometric curve after one trial, but if necessary additional trials were run. Table 1: Chromaticities for Chapter 1 Gabor Experiment

70 61 Participants Eight people (1 female) participated in the study. One subject had abnormal color vision (deuteranomaly) and was not included in the averages. All the others had normal color vision, as tested with the Ishihara plates. While some people produced much more pronounced waveforms than others, no data were excluded. All subjects were part of a database which had undergone anatomical as well as functional magnetic resonance imaging (MRI). The anatomical MRI data was used to make a realistic head model of each participant. The fmri data included retinotopy and was used to define areas of visual cortex known to exhibit retinotopic organization (this retinotopic analysis had been done previously in the lab of Professor Caplovitz: These included the primary visual cortex (V1), V2, and V3. In addition to the early visual areas a dozen or so other areas along the dorsal and ventral visual streams were also mapped out for each participant. Results: Results from psychophysics include average HFP settings at the five different eccentricities and average contrast matches of L+M and L-M to S-(L+M) reference. Figure 20 shows average HFP settings across subjects for the two cardinal chromatic axes across different eccentricity levels.

71 luminanc e amgle 62 5 Flicker Photometry l-m s-(l+m) eccentricity Figure 20: Average heterochromatic flicker photometry matches across observers. Error bars represent one standard error of the mean. While L-M settings remain relatively flat across eccentricities, there is a large shift in the S-(L+M) settings across eccentricities, people requiring more +S luminance in the foveal region than in the periphery. This is in line with previous studies (Skiba, Duncan and Crognale, 2014, Parry and Robson, 2012) and is believed to be due to the varying levels of macular pigment throughout the retina. The reader is referred to table 1 for average contrast match values for the stimuli. HdEEG Recordings A 256 channel system was used to obtain the hdeeg recordings (EGI: Eugene, Oregon). The circumference of the participant s head was measured in order to obtain the proper cap size, and the midpoint from the inion to nasion and the left and right periauricular points were marked to locate the vertex of the scalp. The net was then soaked in a mixture of water, potassium chloride, and baby shampoo, before being put on the participant s head. The electrodes were then set, impedances checked and corrected. After the recordings, panoramic

72 63 pictures were taken of the participant wearing the net (photogrammetry) and later coregistered with the head model. Stimulus Presentation Stimuli were presented in two forms: onset and steady-state (SSVEP). The onset presentation showed the full-field Gabors for 200ms, followed by 800ms of blank screen of equal average luminance to the Gabors. One minute of onset VEPs were recorded for each condition. The steady-state presentation showed a full-field Gabor for 200 ms followed by a phase-reversed Gabor for 200 ms (5 Hz 2 nd harmonic), which repeated for 48 seconds (120 cycles). While the onset stimuli were randomly interleaved across color combinations, the SSVEPs were presented in blocks. Data Preprocessing While the psychophysical data were processed on the fly to produce the custom EEG stimuli, the raw EEG data were stored off line for further analysis. This analysis was done with the Brainstorm software package (Tadel et al., 2011), and comprised bandpass filtering (1-40Hz), detection of blinks and other artifacts, removal of artifacts using signal space projection (SSP) projectors, and segmenting the data into 1 second blocks starting at stimulus onset. Chunks of data containing massive artifacts were removed altogether before averaging. Bad channel recordings that deviated drastically from the majority were deleted.

73 64 Results: hdeeg After averaging the processed data it was noticed that for most subjects a cluster of electrodes over the occipital area showed a negative going component at around 180 ms after stimulus onset, so these electrodes were averaged together for each subject to form a representative waveform. This negative going peak for both chromatic and achromatic stimuli has been previously reported, along with the earlier positive peak for achromatic stimuli (Rabin et al., 1994). Figure 21: Left: Average waveforms for multi Gabor crveps. Right: topographic representation of data showing strong negativity (blue) at the time of the first major negative component in waveform. Figure 21 shows averaged waveforms for the 3 conditions (left), and a topographic map of averaged electrodes highlighted in red. These average electrodes are immersed in a blue which represents the negative going activity seen at approximately 180 ms after stimulus onset.

74 65 Figure 22: Timeline of topographic activity for crveps. Figure 22 shows topography of three conditions across four points in time. While L+M shows increased activity in right posterior scalp (~133ms), all three conditions show strong negative component around 200 ms. The L+M negativity appears more laterally distributed around 200ms after onset compared to the more medial chromatic negativity. Source Localization Sources of the early component of the achromatic visual evoked potential have been largely attributed to the primary visual cortex (V1) (Di Russo et al., 2002). Di Russo and colleagues (2002) showed that the later n1 component of the VEP was localized to more ventral extrastriate regions in the fusiform gyrus. Our achromatic stimuli were low contrast due to contrast matching to isoluminant chromatic stimuli, and we did not see a very pronounced early component and choose here to focus on this later n1 component. We were particularly interested in comparing chromatic and achromatic responses in the dorsal and ventral streams

75 66 because fmri studies have shown responses to color in other areas in addition to V1, notably areas in ventral visual cortex ((Engel, Zhang, and Wandell, 1997, Hadjikhani et al., 1998, Zeki and Bartels, 1999, Wade et al., 2002, Brewer et al., 2005). Figure 23 shows the different retinotopic maps analyzed in these studies. The ventral stream includes V1, V2v, V3v, hv4, VO1 (ventral-occipital), and VO2 (Sereno et al., 1995, DeYoe et al., 1996, Engel, Zhang, and Wandell, 1997, Wade et al. 2002, Brewer et al. 2005). The dorsal regions include V2D, V3D, V3A, V3B, IPSO (intraparietal sulcus, also known as V7), and IPS1 (Press et al. 2001, Konen and Kastner, 2008). Figure 23: Regions of Interest in Source Localization Study: Regions of Interest were derived from retinotopic maps measured using fmri. Diagram shows posterior view of inflated cerebral cortex with ventral (left) and dorsal (right) retinotopic maps for one subject. Brainstorm software was used for source localization. This required importing Free Surfer folders with subjects MRI data and building a head model using the symmetric boundary element method and OpenMEEG software (Gramfort et al., 2005, Kybic et al., 2010). The

76 67 settings used were the default set by Brainstorm, estimating the layers of the skull, with outer and inner layers each having 1922 vertices. Once this forward model was generated, a minimum-norm estimate constrained normal to the cortex (15000 dipoles) was applied to solve the inverse problem of source localization (Hämäläinen and Ilmoniemi, 1994). This required the input of a noise covariance matrix which was generated using the last 300ms of raw data from each of the 1s onset VEP segments. The output converts scalp voltage to current density (amperes) and gives an estimate of activity at every dipole used to model the cortical surface. The dipoles within each region of interest (ROI) were summed together to get an average waveform for each ROI. The amplitudes of the source estimates were converted to z scores within subjects with respect to baseline (baseline: ms after stimulus onset) before being averaged across participants. This required taking the average of and standard deviation of the baseline, then at each time point taking the difference from the mean and dividing by the standard deviation. The z scored data were then rectified in order to avoid the issue of polarity flips: because the surface of the cortex undergoes many folds in the sulci and gyri, the polarity of the sources gets flipped depending on the dipole orientations. Without rectification inverted dipoles would cancel one another and produce little response. Figure 24 shows average source estimates for crveps in the dorsal and ventral ROIs.

77 68 Figure 24: Average source estimates for crveps. Left column shows responses in ventral stream, right column dorsal stream. Y axis is average z score of source estimates across participants Figure 24 shows differences in amplitude within conditions through the dorsal and ventral streams. Achromatic amplitudes have an attenuated n1 response in early visual areas compared to ventral areas in the early portion of the signal (~200 ms after onset). These early areas in the achromatic response have a large later component (~370ms). The chromatic waveforms have early components in both early and ventral visual areas, and the L-M seems to have more of a pronounced spike than S-(L+M). This suggests that the electrode activity for the early component in the achromatic response may be coming more from ventral visual areas, while chromatic is also distributed in early and ventral visual areas

78 69 Figure 25: Comparison of summed activity in early and ventral visual ROIs. Top: Same data as figure 6 but collapsed across early visual areas (V1, V2, V3) and ventral (hv4, VO1, VO2). Bottom: Average latency (left) and amplitude (right) of n1 component across subjects for above waves. Figure 25 shows average waveforms for early vs. ventral visual areas separately for each condition (top). Average latencies and amplitudes were averaged across observers (bottom), error bars represent standard error of the mean. A repeated measures ANOVA on the latencies revealed no main effect for condition or region, but a marginally significant interaction between region and condition F(2, 5) = 4.7, p =.089. While there doesn t appear to be a significant difference in chromatic latencies between early and ventral areas (if anything chromatic latencies are slightly faster in ventral than early areas), achromatic information seems to be coming later to the ventral areas. Although the achromatic condition seems largely attenuated in the average waveform, a repeated measure ANOVA on the amplitudes produced no significant main effects or interactions between condition and region.

79 70 SSVEPs Steady-state VEPs were also recorded in a continuous presentation where the same Gabor fields as in the onset were exchanged with an antiphase version of itself in a reversal paradigm. Each image was displayed for 200ms, resulting in an exchange rate of 5 Hz. Each of the three conditions was presented for 48s. These data were primarily analyzed in the frequency domain using the fast Fourier transform (FFT). This allows for objective quantification of amplitude of a particular frequency. In this case we were interested in the 2 nd harmonic, which is the rate at which the image is presented. Figure 26 shows average SSVEP waveforms across observers (left) and the average amplitude of the 2 nd harmonic (right). Figure 26: Average SSVEP waveforms (left) and the corresponding 2 nd harmonic amplitudes calculated using the FFT (right). There is a difference between the waveforms produced by onset and SSVEP presentations: while L-M produced strong onset response, its SSVEP response was smaller relative to the other two conditions. This is apparent in the red line in the left of figure 8 being

80 1-p value 71 of smaller amplitude than the blue and black lines, and the smaller L-M bar in the right graph of figure 26. Significance Level SSVEP 2nd Harmonic L-M L+M S-(L+M) Figure 27: Significance level of T-Circ test results. Each subject s value was an average of the 30 posterior electrodes p values. The y axis is 1-pvalue in order to display the data as increasing for more significant. A T circ test was done on each of the electrodes as in the onset, and the average significance averaged across observers for each condition is plotted in figure 27 above (Victor and Mast, 1991). These p values are inversely correlated to the amplitudes (i.e. higher amplitudes usually result in lower p values). These t-circ tests were also done on the dipole source data and the average p values are plotted below in figure 28. This was done by inputting the raw data into the minimum norm estimate, while for the onset data only the averages were input. The SSVEP source data was not rectified to positive in order to maintain the frequency component.

81 72 Figure 28: Estimates of 2nd harmonic SSVEP ROI activity. Looking at figure 28, there doesn t appear to be any significant differences between the significance levels of the different colors in these different regions, although it is notable that L- M seems to have less significance, reflecting the electrode results.

82 73 Chapter 5: High Density Investigation of the Watercolor Illusion While the color we experience is correlated with the wavelength of light absorbed by photoreceptors, our visual system performs other computations which lead to color sensation. This is apparent in many visual illusions such as simultaneous contrast, the Cornsweet illusion, White s effect, and neon color spreading. There are other long term aftereffects such as the McCollough effect where achromatic patterns appear chromatic much longer than the fleeting afterimages of non-patterned stimuli. The next series of experiments uses the same logic and methodology as the previous chapter but applies it to a color spreading illusion known as the watercolor effect (Pinna, Brelstaff and Spillmann, 2001, Broerse, Vladusich and O Shea, 1998). This illusion occurs when thin lines (of higher spatial frequency than the multi-gabor fields) form a contrasting boundary. The watercolor illusion has a similar contrast profile to the Cornsweet Illusion and has been shown psychophysically to be measurable for chromatic (Devinck et al. 2005, 2006, Devinck and Knoblauch, 2012) and achromatic (Cao Yazdanbakhsh and Mingolla, 2011) contours. One previous study demonstrated the watercolor illusion is measureable using SSVEP (Coia et al., 2014). Coia et al. exchanged the illusion with a control pattern (see figure 29) and produced a reliably larger fundamental harmonic, indicating that the illusion may be of cortical origin. The current study set out to determine the cortical locus of this illusion using the source localization techniques outlined in the previous chapter.

83 74 Figure 29: Watercolor stimuli: Rows designate color axes and columns designate condition. Left to right: Filled Control is control stimulus with physically filled in color. Middle is illusion (no physical inner surface color, just illusory spreading from contours. Right is control with braided contours and no surface color. These stimuli are similar in spatial structure to those used in previous studies (Coia et al a, Coia and Crognale, 2014 b). At a viewing distance of 114 cm, contours subtended 4.5 arcmin in width and the resulting columns were 0.66 s wide by 6.7 s long. Each group of patterns comprised seven columns, separated by 0.37 degrees. In the current study these contours are modulated exclusively along the three cardinal directions in color space.

84 75 Psychophysics Participants completed HFP on the control contours for individual isoluminance settings and subsequently contrast matching of L+M and L-M axes to a set S-(L+M) control patterns. The chromaticities of the stimuli are given in table 2. Method of constant stimuli in temporal sequence was used to compare the visibility of L+M and L-M contours of varying contrasts to a fixed S-(L+M) pattern. This involved showing one control before another control and asking the observer which was more visible. Once the contrasts of the contours were matched to each other, the illusion was measured by comparing it to a range of filled color put into the control. The same temporal method of constant stimuli was used. See table 2 for details about chromaticities of the patterns.

85 76 Table 2: Chromaticities for Chapter 2 Watercolor Experiment Results: Psychophysics Previous studies have measured the magnitude of the watercolor effect by the percentage of the inducing contour required to make a perceptual match to the illusion (Devinck et al., 2005, 2006, Coia et al., 2014 a, 2014 b). Coia et al. (2014 a) found a correlation between the perceptual matches to the illusion with the SSVEP amplitudes induced by different contour

86 Percent of Inducing Contour 77 pairs. Figure 30 shows the percentage of inducing colors required to make a match for the six different colors tested in this study. The percentages in figure 12 were calculated in MBDKL space. The average percentages in the bottom right column in table 2 were converted into cone contrast space. They are overall smaller percentages but match the overall trend of figure Psychophysical Magnitude of Illusion increment decrement 0 L+M S-(L+M) L-M Figure30: Average psychophysical matches across observers. Error bars represent +-1 standard error of the mean. For L-M and L+M, decrements have slightly larger percentage, while for S-(L+M) increments have a slightly larger percentage. L+M has an overall larger percentage than both L- M and S-(L+M) when poles are summed together. Results: EEG The data were processed in the same manner as chapter 1 concerning data acquisition and pre-processing. Data were segmented according to their specified color and condition. Graphs in figure 31 below show the average electrode responses averaged across the same set of occipital electrodes chosen for analysis in chapter 1.

87 78 Figure 31: Average onset responses across observers to watercolor stimuli. Different graphs represent different chromatic axes and different lines within graphs represent different conditions. While differences can be seen in the waves of the different conditions for achromatic (L+M) and S-(L+M), the L-M conditions appear more similar to each other. In the achromatic graph, the negative component for the filled control is slightly faster than the illusion, while in the S-(L+M) graph the illusion latency is faster than the filled control. This indicates that physical achromatic contrast is being processed more quickly by the visual system than physical +S contrast, while the illusion latencies seem to be on a similar timescale Figure 32: analysis of latency and amplitude components of watercolor onset VEPs. Above figure 32 shows average latencies and amplitudes for electrode data. In general, chromatic latencies are a bit longer than achromatic, even though perceptually they have been contrast matched. This is especially apparent in the filled control condition. While small effects in amplitude can be seen for the L+M and S-(L+M) illusion and fill conditions compared to their