EVENT-RELATED POTENTIAL CORRELATES OF VISUAL CONSCIOUSNESS: A REVIEW OF THEORIES AND EMPIRICAL STUDIES

Transcription

1 EVENT-RELATED POTENTIAL CORRELATES OF VISUAL CONSCIOUSNESS: A REVIEW OF THEORIES AND EMPIRICAL STUDIES Bachelor Degree Project in Cognitive Neuroscience 15 ECTS Spring term 2012 Granit Kastrati Supervisors: Antti Revonsuo and Judith Annett Examiner: Judith Annett

2 ERP Correlates of Visual Consciousness: A Review 2 Event-related Potential Correlates of Visual Consciousness: A Review of Theories and Empirical Studies Submitted by Granit Kastrati to the University of Skövde as a final year project towards the degree of B.Sc. in the School of Humanities and Informatics. The project has been supervised by Antti Revonsuo and Judith Annett I hereby certify that all material in this final year project which is not my own work has been identified and that no work is included for which a degree has already been conferred on me. Signature:

3 ERP Correlates of Visual Consciousness: A Review 3 Abstract Two influential theories of consciousness disagree about if consciousness initially arises along the occipitotemporal cortex to later engage frontoparietal regions and attentional mechanisms, or if it necessarily requires the latter. Consequently, different predictions are made about the temporal emergence of consciousness. The event-related potential (ERP) technique can be used to resolve the issue. It can temporally track neural activity of consciously perceived stimuli relative to stimuli bypassing consciousness. This essay describes the two theories and reviews ERP studies on visual consciousness and its relationship to attention. Three ERP correlates of consciousness have been proposed. The question is if they should be interpreted as supporting the one or the other theory. Most plausibly, visual consciousness arises along occipitotemporal regions and later incorporates frontal areas engaging higher cognitive functions. Importantly it seems that consciousness cannot arise without spatial attention/parietal regions. Keywords: Visual Consciousness, ERP, Attention, Phenomenal Consciousness, Reflective Consciousness, The Global Neuronal Workspace Theory, The Recurrent Processing Theory.

4 ERP Correlates of Visual Consciousness: A Review 4 Table of Contents Abstract 3 List of abbreviations 6 Introduction 7 Theories of consciousness 11 The Recurrent Processing Theory 11 The role of attention in the RPT 12 The Global Neuronal Workspace Theory 14 Three forms of processing 15 Summary 16 ERP correlates of visual consciousness 16 The First Positive 16 The P1 amplitude as the attentional modulation of stimuli 18 Visual consciousness without the P1 19 The Visual Awareness Negativity 20 The VAN and attention 24 The VAN and selective feature-based attention 24 The VAN and spatial attention 26 Visual consciousness without the VAN 26 The Late Positivity 27 The LP as confidence level 30 The LP and attention 31 The LP and spatial attention 32 The LP and the scope of attention 32 The LP and selective feature-based attention 33

5 ERP Correlates of Visual Consciousness: A Review 5 ECVC and the theories of consciousness 33 Discussion 34 References 37

6 ERP Correlates of Visual Consciousness: A Review 6 List of Abbreviations CB = Change blindness. ECVC = ERP correlates of visual consciousness. ERP = Event-related potentials. FFS = Feedforward sweep. GNWT = Global neuronal workspace theory. IB = Inattentional blindness. LP = Late positivity. RP = Recurrent processing. SOA = Stimulus onset asynchrony. VAN = Visual awareness negativity.

7 ERP Correlates of Visual Consciousness: A Review 7 Introduction In the last two decades, empirical studies on consciousness have grown exponentially. In approaching consciousness, many researchers have followed a strategy suggested in an influential paper by Crick and Koch (1998). The suggestion was that researchers could begin by studying visual consciousness because vision is the dominant sensory system in humans and can be manipulated easily in various ways. Researchers following this suggestion have aimed at elucidating the neural correlates of (visual) consciousness (NCC). NCC can be defined as the minimally sufficient neural correlate(s) of specific kinds of phenomenal content (Chalmers, 2000). In this essay discussion will be restricted to specific contents of visual consciousness (e.g. the experience of the color green) as distinguished from background states of consciousness (e.g. the state of dreaming or being awake) (For a discussion on states of consciousness see, Chalmers, 2000). Hence, the words conscious and unconscious in this essay refers to the content of consciousness and not to global states of consciousness. A further distinction is the one between phenomenal and reflective consciousness. The former is the subjective experience of for example seeing the color green (Block, 2002; Revonsuo, 2006). The latter refers to the cognitive operations on the contents of phenomenal consciousness, for example naming or evaluating objects (Revonsuo, 2006). A concept similar to reflective consciousness is the concept of access consciousness: the broadcasting of ones representations for wider cognitive access (Block, 2002). For the sake of simplicity, in this essay, only the concept of reflective consciousness is used to refer to the higher-order form of consciousness. In order to study the neural activity of the contents of visual consciousness specifically, neural activity related to the conscious perception of some stimuli has to be differentiated from neural activity of stimuli bypassing consciousness. This can be accomplished by manipulating visual perception so that stimuli sometimes enter

8 ERP Correlates of Visual Consciousness: A Review 8 consciousness and at other times not. Different methods used in manipulating perception are for example visual masking, binocular rivalry, change blindness and attentional blink (Kim & Blake, 2005). Using the various techniques of cognitive neuroscience e.g. functional magnetic resonance imaging (fmri) and event-related potentials (ERP), it is possible to record the differences in brain activity when stimuli enter visual consciousness and when they do not. Each technique has its own advantages and disadvantages. Some researchers have utilized the high spatial resolution of fmri to study the difference in brain activity between conscious and unconscious processing of stimuli. The fmri has a spatial advantage over other techniques (e.g. ERP) with a resolution of around 3mm 3. Some studies have associated the occipitotemporal cortex with visual consciousness (Bar et al., 2001; Moutoussis & Zeki, 2002; Tong, Nakayama, Vaughan & Kanwisher, 1998). Other studies have suggested that the activity along the visual ventral stream may be necessary but not sufficient for conscious perception. Instead, frontoparietal regions need to be active as well (Beck, Rees, Frith & Lavie, 2001; Dehaene et al., 2001; Lumer & Rees, 1999; Vuilleumier et al., 2001). The latter findings suggest a role for attention for conscious perception as frontal and parietal areas have previously been related to attention (Corbetta, Kincade, Ollinger, McAvoy, & Shulman, 2000; Corbetta & Shulman, 1998). The complex relationship between consciousness and attention is an important issue in the science of consciousness, one that remains to be solved (van Boxtel, Tsuchiya, & Koch, 2010). Although the fmri has a high spatial resolution, its temporal resolution is lower and extends over seconds. For this reason, the high temporal resolution of the ERP technique (a few milliseconds) can be used to track the temporal evolution of neural activity leading to consciousness of visual stimuli. Electroencephalography (EEG) measures the electrical activity of the brain as it is reflected at the scalp (Luck, 2005). The signals received are seen

9 ERP Correlates of Visual Consciousness: A Review 9 as temporal changes in voltage or electrical potentials in response to stimuli. When signals elicited from specific events are averaged, ERPs can be extracted from the EEG. The ERP waveforms consist of characteristic peaks, components or amplitudes that are either negative or positive. They are numbered to indicate in what temporal position they appeared. For example P1 is the first positive peak and N1 is the first negative peak. Although giving some information about temporal sequencing, the spatial resolution of the ERP is low and it is not possible to locate the exact neural source of the electrical potentials. Researchers have come to different conclusions about the timing of consciousness. Fig. 1 shows the three candidate ERP correlates of visual consciousness (ECVC): A first positive peak around 100 ms (P1) (e.g., Pins and ffytche, 2003; Roeber et al., 2008), a negative peak around 200 ms called the visual awareness negativity (VAN) (e.g., Ojanen, Revonsuo, & Sams, 2003; Railo & Koivisto, 2009a), and a late positive peak around 300 ms called the late positivity (LP) (e.g. Del Cul, Baillet & Dehaene, 2007; Lamy, Salti, & Bar-Haim, 2009). These ERP correlates suggest either that visual consciousness arises early in the visual cortex or later when also non-visual areas are incorporated (Railo, Koivisto & Revonsuo, 2011). The ERP technique has also been used to study attention (e.g. Hillyard & Anllo- Vento, 1998). Psychological phenomena such as inattentional blindness (IB) (Mack & Rock, 1998) and change blindness (CB) (Simons & Levin, 1997) indicate that attention is necessary for consciousness. Koivisto, Kainulainen and Revonsuo (2009) suggest that attention and consciousness should be tested with the ERP technique to see if any form of consciousness (phenomenal or reflective) is separable from any type of attention. The results of such research could lead to the inclusion or exclusion of attention and the relevant brain regions in the generation of consciousness. There are two influential but conflicting theories about the nature of consciousness. Specifically, they disagree about if it corresponds to phenomenal consciousness or to

10 ERP Correlates of Visual Consciousness: A Review 10 reflective consciousness. According to the Recurrent Processing Theory (RPT) (Lamme, 2010) consciousness can be dissociated from attention and other higher cognitive functions and thus corresponds to phenomenal consciousness. Phenomenal consciousness is generated relatively early (around 200 ms) in the occipitotemporal cortex by local recurrent processing (RP). When the RP reaches a global scale, incorporating also frontoparietal regions, information becomes available for cognitive manipulation, corresponding to reflective consciousness. However, the Global Neuronal Workspace Theory (GNWT) (Dehaene & Naccache, 2001) takes consciousness to be a later process, necessarily connected to higher cognitive functions e.g. attention and output systems. Consciousness, according to the GNWT, is a global process in the brain involving both the occipitotemporal and the frontoparietal cortex and thus corresponding to reflective consciousness. The aim of this essay is to review ERP studies on visual consciousness in order to find out if they can help resolve the debate between the two theories of consciousness. What can the ERP studies tell us about the nature of consciousness? If it arises early, it should correspond to phenomenal consciousness. If it emerges later as the LP, then it can be considered to be a higher-order form involving also non-visual areas. First, the two theories of consciousness are described and the role they give to attention for conscious perception highlighted. Then the studies suggesting different ECVCs are contrasted. Here, also ERP studies investigating the relationship between forms of attention and types of consciousness are examined in order to further elucidate the nature of consciousness.

11 ERP Correlates of Visual Consciousness: A Review 11 Fig. 1. The suggested ERP correlates of visual consciousness; P1, VAN and LP are shown. The ERP correlates are seen as the difference between ERP responses to conscious (aware) and unconscious (unaware) conditions. The VAN here overlaps with the N1, P2 and N2 components and the LP overlaps with the P3 component. Negative is plotted upwards according to convention. From Tracking the processes behind conscious perception: A review of event-related potential correlates of visual consciousness, by H. Railo, M. Koivisto and A. Revonsuo, 2011, Consciousness and Cognition, 20, p.973. Copyright 2011 by Elsevier Inc. Reprinted with permission of the authors. Theories of Consciousness The Recurrent Processing Theory The RPT distinguishes two kinds of neural processing: the feed forward sweep (FFS) and the RP (Lamme, 2000, 2004, 2010). When a visual stimulus is presented to the eye, the information is transferred from the retina to the visual cortex where it sweeps forward along the occipitotemporal and the occipitoparietal cortex, towards motor and prefrontal areas (Lamme, 2010). During this sweep, starting around 30 ms and continuing to 100 ms after stimulus onset, neurons respond to information they are selective to, extracting basic features of the stimuli. For example area V5 and the inferotemporal cortex tune in to motion and to faces respectively. According to the theory, individuals remain unconscious of the stimulus information during the FFS (Lamme, 2000). For consciousness to arise, those neurons that have gone through the FFS have to interact both among themselves and with earlier areas, thus reaching the RP. During RP, starting around 200 ms, neurons relate features, bind, segregate and organize (Lamme, 2000, 2004, 2010). The RP is necessary for visual

12 ERP Correlates of Visual Consciousness: A Review 12 consciousness although it must reach some critical mass for it to also to be sufficient for it. Lamme (2010) distinguishes between local and global RP. During the local RP, stimulus information already in the visual ventral stream may reach phenomenal consciousness. When the RP reaches higher brain areas (around 300 ms) such as frontal and parietal areas, the information becomes available for other higher cognitive functions, corresponding to reflective consciousness (Lamme, 2000, 2004, 2010). Furthermore, visual attention and visual consciousness are defined differently from each other and are clearly distinguished (Lamme, 2003). They are not only distinguished from each other in function but can occur independently of each other. The role of attention for the RPT The model for attention and consciousness suggested by Lamme (2003, 2004) takes into account both conscious and an unconscious information processing. According to this model, attention can be directed to either one of these forms of processing. There are thus four types of processing according to the model: Conscious information that is or is not attended to and unconscious information that is or is not attended to. When information is processed at the level of consciousness but is not attended to it reaches pure phenomenal consciousness (Lamme, 2003, 2010). When attention is directed to the contents of phenomenal consciousness the information becomes available for among other things, conscious report (Lamme, 2003). An example from the CB paradigm is used by Lamme (2010) to illustrate his point. In this experiment, participants are shown several figures in two successively presented images separated by a brief blank screen. During the blank screen a change may occur that is usually not consciously detected when asked to identify the change in the second image (Simons & Levin, 1997). If the figure that may have changed is cued during the second presentation

13 ERP Correlates of Visual Consciousness: A Review 13 most people do not notice the change. If the figure-to-change is cued in the first presentation then it is hard not to notice the change. If the location of the figure-to-change is cued during the blank screen most people notice the change although slightly less so than if cued during the first presentation. This example illustrates that we are phenomenally conscious of the figures first presented and that we may fail to report the change because of the failure to attend towards the location of the figure-to-change. Lamme (2003, 2010) suggest that CB should be considered to be a failure of memory and not of consciousness. This is because attention towards an item enables it to be stored in working memory, which further enables the first image to be compared with the second image. In a similar way the phenomenon of IB, the failure to notice stimuli when it appears suddenly and when attention is deployed somewhere else (Mack & Rock, 1998), may also be a failure of memory. This is because the task involves participants being asked afterwards if something unusual was seen, or asked about something not related to the task. Participants have to search in their memory and will probably not find anything because attention was deployed elsewhere and so the consciously experienced unexpected stimuli may not have been stored in memory (Lamme, 2004). The role of attention is to resolve competition among stimuli that are competing for access to executive systems (Lamme, 2003, 2004). Competition among stimuli in the visual system may be resolved as a result of sensory processing and memory that is shaped by genetics and visual experience (Lamme, 2003, 2004). In this way, neurons may be predisposed to "choose" some stimuli over others. We may process some features of the world over others, for example moving stimuli over stationary, which are shaped by genetics and visual experience (Lamme, 2004). Alternatively the brain chooses one stimulus over another because an earlier stimulus has left the system biased towards this one stimulus by leaving traces (memory), paving the way for it to be chosen (Lamme, 2003, 2004). We may be phenomenally conscious of more than one stimulus when they are represented at early

14 ERP Correlates of Visual Consciousness: A Review 14 visual areas (local RP). By the attentional mechanisms described above (sensory processing and memory), the local processing in the visual system may spread to frontoparietal areas, thus being globally distributed (global RP). In other words, information selected by attentional mechanisms becomes available for executive systems and for motor output, reflecting reflective consciousness (Lamme, 2003). The Global Neuronal Workspace Theory According to the global workspace theory, a cognitive theory of consciousness proposed by Baars (2002, 2005), visual consciousness arises when visual information becomes globally distributed, allowing cognitive functions to access this information. The function mediating this global access is called a workspace. The workspace enables different functions to connect so that different specialized areas may have access to information processed at other sites. A neural version of the workspace theory called the GNWT has been put forward by Dehaene & Naccache (2001). This theory suggests "a distributed neural system or workspace with long-distance connectivity that can potentially interconnect multiple specialized brain areas in a coordinated, though variable manner" (Dehaene & Naccache, 2001, p.13). Examples of specialized areas is the fusiform area selective for faces, the color area V4 and the language systems. Top-down attentional amplification is the mechanism that brings together the different modules into the global workspace and enables information to be maintained over a certain time period (Dehaene & Naccache, 2001). Attention is therefore necessary for any conscious experience to occur. A broad neural network is associated with consciousness, involving the coactivity of frontoparietal areas with occipitotemporal areas (Dehaene Changeux, Naccache, Sackur, & Sergent., 2006). The necessary frontoparietal regions activate at around 300 ms, indicating the time at which

15 ERP Correlates of Visual Consciousness: A Review 15 consciousness arises (Del Cul et al. 2007). Because information in occipitotemporal regions must be distributed to frontoparietal areas, and thus to higher cognitive functions, consciousness equals reflective consciousness and pure phenomenal consciousness is an illusion (Dehaene et al., 2006). The GNWT allows for a different interpretation of CB than the interpretation made by Lamme (2010). It could be interpreted as illustrating that there is no conscious experience of the scene until attention is deployed there (Dehaene et al. 2006). People may be confident of consciously seeing the whole scene but it is shown that they fail to notice a change when the change occurs during e.g. a brief blank screen separating the first and the second presentation of the scene. Dehaene et al. notes further that if the change draws attention it can more easily be detected. The belief that we are phenomenally conscious of the whole scene may be a result of the refrigerator light illusion (Dehaene et al.): Every time attention is directed towards some location, information from that spot may reach consciousness. (This does not say that attention is sufficient for conscious perception.) Similarly, in the phenomenon of IB, individuals are not conscious in any way of the unattended stimuli (Dehaene et al.). The GNWT thus argues against the hypothesis of the RPT that individuals actually are phenomenally conscious of unreported changes in the phenomena of CB and IB. Three forms of processing The GNWT distinguishes three kinds of processing: Subliminal, preconscious and conscious processing (Dehaene et al. 2006). Subliminal processing can be divided into that which is unattended and that which is attended to. Activity in early visual areas only corresponds to unattended subliminal processing. Here, bottom-up stimulus strength is too weak (or is interrupted) to ignite a distributed reverberating activity that would enable global access of visual information. Attended subliminal processing (e.g. when cued) can have some

16 ERP Correlates of Visual Consciousness: A Review 16 effect on the system at an unconscious level. The preconscious is more widespread than subliminal processing. This information is potentially conscious; all that is needed is topdown attention that can bring it into the global workspace so that it can be widely distributed. When attention is directed towards some information it can be held in working memory and be transformed into some kind of motor output. Summary The RPT claims that phenomenal consciousness can exist without the involvement of attention and other higher cognitive functions associated with frontoparietal areas. Phenomenal consciousness can emerge already in the visual system at about 200 ms after stimulus onset by local RP. When the RP reaches a global scale, at about 300 ms, the contents of phenomenal consciousness become available for cognitive operations such as naming or categorization of information. In contrast, the GNWT claims that information in the visual ventral stream must be distributed to parietal and frontal areas for the information to reach the level of consciousness. Consciousness should according to the GNWT arise at about 300 ms when frontoparietal areas are incorporated. Attention is the mechanism that allows visual information to reach the workspace, enabling local unconscious visual information to be globally distributed. Studies investigating the evidence for and against each of these two theories will be presented below. These studies have used the ERP technique together with the various ways to manipulate perception to track the time course of neural activity related to consciousness. In addition, the relationship between attention and the suggested ECVC are described. ERP Correlates of Visual Consciousness The First Positive

17 ERP Correlates of Visual Consciousness: A Review 17 The P1 is usually largest at lateral occipital electrode sites with an onset of ms (Luck, 2005). The P1 has been seen to be abnormal in patients suffering from unilateral visual extinction. Extinction refers to the deficit where patients with damage to parts of the parietal lobe (usually the right) are not conscious of stimuli that are presented at the contralesional side when simultaneously presented with stimuli at the ipsilesional side (Driver & Vuilleumier, 2001). Marzi, Girelli, Miniussi, Smania and Maravita (2000) studied the ERP responses in a patient suffering from unilateral visual extinction. Marzi et al. noted that the P1 component is affected by spatial attention (see, Hillyard & Anllo-Vento, 1998). Because extinction is a failure of attention (Driver & Vuilleumier, 2001) the authors (Marzi et al.) predicted that the P1 should be abnormal during extinction. To test their prediction, a comparison was made between the ERP response to stimuli that were consciously perceived and stimuli that were extinguished. That is, during bilateral trials when contralesional stimuli were extinguished and when both the ipsilesional and contralesional stimuli were correctly reported (i.e. consciously perceived). The authors observed the P1 over the right (damaged) hemisphere when stimuli were consciously seen but not when the contralesional stimuli were extinguished. The conclusion was that an impairment of the mechanisms of spatial attention might have been the cause of extinction, as indicated by the missing P1 component when stimuli were extinguished (Marzi et al.). Binocular rivalry coupled with the ERP technique has been used to study visual consciousness. Binocular rivalry occurs when two dissimilar images are presented one to each eye, leading to an experiential shift between the images instead of being perceived simultaneously (Kim & Blake, 2005). Because the physical stimuli are invariant in binocular rivalry, only the conscious experience of the stimuli changes. Roeber et al. (2008) presented a left-oblique grating to one eye and a right-oblique grating to the other. Sometimes one of the gratings was changed so that both gratings had the same orientation (fusion stimulation).

18 ERP Correlates of Visual Consciousness: A Review 18 Here, participants did not experience any change in orientation. Sometimes it was the grating that was consciously experienced (the currently dominating grating) that changed and sometimes it was the grating not currently consciously perceived that changed. Rivalry occurred when the two gratings had different orientations (alternations in perceived orientation). Roeber et al. measured brain activity related to when participants were conscious of a change in orientation. They observed an enhancement of the P1 for consciously perceived changes in orientation, compared to changes that bypassed consciousness. Another study kept the stimuli near the threshold to consciousness (Pins and ffytche, 2003). The subjective threshold was determined for each individual by presenting the stimuli for varying durations until it was reported as seen about half the times. ERP responses for trials when stimuli information reached the level of consciousness were compared to trials in which stimuli bypassed consciousness. Distributed event-related activity was found. The first ERP waveform elicited as a difference between consciously seen stimuli and stimuli not consciously perceived was the P1 over the occipital lobe. This activity was followed by negative and positive waves extending over parietal and frontal areas. The authors argued that the activity that occurred after the P1 did not directly contribute to conscious visual perception but instead reflected secondary processes. The P1 Amplitude as the Attentional Modulation of Stimuli The P1 amplitude has been shown to increase when stimuli are attended to relative to when stimuli are unattended (Hillyard, Vogel, & Luck, 1998). The ERP waveform for attended and unattended stimuli is the same between conscious and unconscious conditions but they are amplified (or not) depending on if the stimuli are attended to (or not). Railo et al. (2011) propose that the P1 seen in previous studies could be caused by the presence or absence of attentional modulation of stimuli in conscious conditions. For example, the

19 ERP Correlates of Visual Consciousness: A Review 19 enhanced P1 seen by Roeber et al. (2008) in relation to a change in consciously perceived stimulus could be caused by attentional mechanism modulating the perception of the stimulus (Railo et al.). In the study by Marzi et al. (2002) the absence of the P1 during extinction in could be explained by considering that early sensory responses were not modulated when contralesional bilateral stimulus were extinguished because of the damaged parietal lobe of their participant (Railo et al.) as Marzi et al. also noted in their study. The absence of the P1 for undetected stimuli in the study by Pins and ffytche (2003) also indicate that the stimuli failed to be selected by attentional mechanisms (Railo et al.). Visual Consciousness Without the P1 amplitude Further opposition to a P1 correlate of visual consciousness comes from studies where the P1 was absent even though subjects were conscious of the stimuli. Sergent, Baillet, and Dehaene (2005) used the attentional blink (AB) paradigm (Shapiro & Arnell, 1997) with the ERP technique to find out if consciousness is an early or a late process. In rapid serial visual presentation (RSVP) two targets can be presented in a sequence with distractors among them. AB occurs when the identification of the first target (T1) disrupts the conscious perception of the second target (T2) if the two appear within 500 ms (Shapiro & Arnell, 1997). However, T2 may be noticed if participants are told to ignore T1. Sergent et al. compared ERPs evoked by T2s during AB when T2 was consciously seen and when they were not. The P1 (96 ms) wave did not differ between blinked (not consciously perceived) and consciously perceived T2s. Instead, it was evoked both by consciously perceived and unperceived T2s. In a location task experiment conducted by Lamy et al. (2009), participants could correctly locate stimuli both at a conscious level and at an unconscious (above-chance) level. The P1 was found to be unaffected by the stimuli at the conscious and the unconscious level, and to conditions of correct and incorrect location of stimuli. Koivisto et al. (2008) conducted an experiment

20 ERP Correlates of Visual Consciousness: A Review 20 where stimuli were kept near the threshold to visual consciousness. Their study failed to replicate the P1 found by Pins and ffytche (2003), even though participants were conscious of the stimuli. The Visual Awareness Negativity The VAN arises around 200 ms after stimulus presentation over occipitotemporal lobes (Railo et al. 2011). It is a negative difference between conditions in which stimuli enters consciousness and conditions in which it does not (Koivisto et al. 2009). It usually overlaps with the N1, P2, and N2 components (fig. 1) (Koivisto & Revonsuo, 2003; Railo et al. 2011). Experiments utilizing the CB paradigm have correlated the VAN with conscious visual perception. Koivisto and Revonsuo (2003) measured the ERP elicited by conscious change detection and change blindness. Two images of eight rectangles were shown one after the other, separated by a blank screen. Participants instruction was to report if they saw a change in orientation in one of the rectangles in the second image. The authors also compared ERP activity to undetected changes and when no change occurred at all in order to find the ERP correlate of implicit change detection. The VAN was observed around 200 ms for conscious change detection relative to undetected changes or when no change occurred at all. In another CB experiment, Pourtois, De Pretto, Hauert and Vuilleumier (2006) used faces as stimuli, first presenting an image of two faces (S1) then after the presentation of a brief empty screen, either the same faces appeared or there was a change in one of the two faces (S2). Several ERP components differed between the different conditions (conscious change detection - change blindness). During S1, an enhanced P1 was seen when a change was later consciously detected (S2) relative to when the change was not detected. Because the P1 is known to be affected by selective spatial attention (Hillyard & Anllo-Vento, 1998) Pourtois

21 ERP Correlates of Visual Consciousness: A Review 21 et al. reasoned that S1 was processed better because more attentional resources were deployed on the first image. As a result, a change in the subsequent image (S2) was consciously detected more easily. Following the P1, a negative difference around 200 ms was seen when a change in one of the faces was detected in S2. It should be noted that change detection could be seen to be dependent on attention to the object-to-change in S1. For this reason, Koivisto and Revonsuo (2010) suggest that the ERP difference between change detection and change blindness may be different even before the occurrence of the change. If a target stimulus immediately succeeds, precedes or if it spatiotemporally overlaps with another stimulus called the mask, the visibility of the target stimulus may be reduced (Breitmeyer & Ögmen, 2006). Wilenius-Emet, Revonsuo, and Ojanen (2004) masked their stimuli with both backward and forward masks. The stimuli consisted of line drawings of coherent objects and figures whose properties were scrambled, making up meaningless nonobjects. The duration of the masked stimuli were varied in order to reach three perceptual levels: below (27 ms), near (40/53 ms) and above (108 ms) subjective threshold. When stimuli were near subjective threshold, it was sometimes consciously experienced and sometimes not. A comparison was made between ERPs to stimuli above or near subjective threshold and to stimuli below or near-threshold. As predicted, stimuli above and nearthreshold elicited a negative peak around ms and stimuli below-threshold did not. Ojanen et al. (2003) used object and non-object stimuli at three reduced contrast levels: high (14 %), middle (7 %) and low (3 %). The high contrast stimuli were more easily (consciously) recognized than the low contrast stimuli. Stimuli that were consciously recognized elicited a late negative peak (around 400 ms) compared to stimuli that were not recognized. The consciously recognized stimuli were most often the high contrast stimuli, while the low contrast stimuli were almost never consciously recognized. Because the two conditions (above and below subjective threshold) were different in their physical properties

22 ERP Correlates of Visual Consciousness: A Review 22 the results could have been confounded. Noting this, Wilenius-Emet and Revonsuo (2007) used only one reduced-contrast level that was individually determined so that the same stimuli (objects or non objects) were consciously recognized about half the times. In addition, ERP of objects were analyzed separately from ERPs to all stimuli in order to rule out differences in types of stimuli in the ERP results. The results confirmed a delayed VAN in response to above-threshold stimuli. These studies (Ojanen et al., 2003; Wilenius-Emet & Revonsuo, 2007; Wilenius-Emet et al., 2004) may have used too complex tasks in their experiments (Koivisto et al. 2008). Instead of reflecting visual consciousness, the VAN found in these studies could reflect higher-level processing such as categorization of the stimuli as objects or non-objects. For this reason, Koivisto et al. (2008) conducted a simple experiment in order to test the ERPs elicited by the conscious detection of stimulus compared to when the same stimulus were not detected. The simple stimuli in their study appeared between backward and forward masks. A negative peak difference around ms after stimulus onset was found when comparing responses to consciously seen stimuli and stimuli that where not seen, confirming earlier studies. Koivisto et al. reasoned that the masks used in their experiment could have made it difficult to extract the ERP response to visual consciousness from noise. A second experiment was therefore conducted where a stimulus was kept near the threshold to consciousness. The VAN was observed also in this second experiment, thus ruling out the possible effects from the masking procedure. Metacontrast masking refers to a special form of visual backward masking where the mask does not overlap spatially with the preceding target (Breitmeyer & Ögmen, 2006) To be fully effective in suppressing the target, the mask should follow the target after ms. Railo and Koivisto (2009a) used three Stimuli Onset Asynchronies (SOAs) (= 0, 50, or 130 ms) where an effective metacontrast mask or a similar-looking ineffective pseudomask could

23 ERP Correlates of Visual Consciousness: A Review 23 appear. A comparison between the effective mask condition and the pseudomask condition in the 50 ms SOA was predicted to elicit the ECVC. To rule out possible effects of the different mask, control trials were used where only masks were presented. The ERPs elicited by masks (effective and ineffective) and target was compared to the ERPs evoked by masks only. The results showed that recognition accuracy dropped during the 50 ms SOA for both kinds of mask but was larger for the effective mask. In other words, when presented with an effective metacontrast mask at 50 ms, the stimuli was less often (consciously) perceived than when presented with an ineffective pseudomask at 50 ms. ERPs to pseudomasks at 50 ms was found to be more negative in the ms time window than the effective masks at the same SOA over posterior temporal lobes. Control trials showed that the effective mask and the pseudomask had similar ERPs. The response to targets between effective mask and pseudomask conditions should therefore be considered to be unaffected by the masks themselves (Railo & Koivisto, 2009a). Even though the control trials showed similar ERPs to the different masks, Bachmann (2009) argues that their combining with the target may have given two quite different stimuli, which in turn may have given rise to a negative difference that was interpreted as the VAN. Railo and Koivisto (2009b) claim that the effects of the different masks may only reduce or increase the effects of the target but not determine the total difference, which instead is the difference in target visibility (consciousness). Bachmann also argues that the late onset of the VAN (330 ms) in the study by Railo and Koivisto (2009a) shows that it cannot reflect the timing at which consciousness arises. Instead of reflecting phenomenal consciousness it could reflect the post-perceptual processing of the contents of phenomenal consciousness (Bachmann, 2009). Railo and Koivisto (2009b) suggest that the delayed VAN may have been caused by attentional amplification of the pseudomasked targets, a view that is supported by the finding that late VAN is dependent of selective attention (Koivisto & Revonsuo, 2008,

24 ERP Correlates of Visual Consciousness: A Review 24 see below). The VAN and Attention The VAN and selective feature-based attention. Previous ERP studies on attention have shown that attending to relevant (compared to irrelevant) nonspatial features such as colors or shapes elicits a negativity around ms in posterior electrode sites, called the selection negativity (SN) (Hillyard & Anllo-Vento, 1998). The similar timing, polarity and scalp distribution of the SN and the VAN suggest that they work by the same mechanisms (Koivisto & Revonsuo, 2008). Koivisto, Revonsuo and Salminen (2005) set out to test the VAN and the SN with the ERP technique in order to find out which one appears earliest. If the VAN appears earlier than the SN, then it should be independent of selective feature-based attention and vice versa. Koivisto et al. (2005) used backward masking with three letters as stimuli at two SOAs. The letters switched from being targets to nontargets between blocks. At the long SOA (= 133 ms) the stimuli were visible and at the short SOA (= 33 ms) they were invisible. Control trials were performed with a constant stimulus-mask SOA in order to rule out confounds due to differences in the timing of the masks. The constant stimulus-mask SOA was individually determined so that the stimuli reached the level of consciousness about half the times. A negative amplitude was observed from 130 ms, corresponding to the VAN. It was seen to arise independently of selective attention as it emerged similarly between attended and unattended stimuli. Attention started to modulate a later part of the VAN ( ms) as attended features elicited a larger VAN at these later latencies. The difference in timing of the masks in the main experiment cannot explain the observed VAN as it was also found in the control trials. Another backward masking experiment (Koivisto & Revonsuo, 2008) further investigated the relationship between visual consciousness and nonspatial selective attention.

25 ERP Correlates of Visual Consciousness: A Review 25 The time course of the VAN in relation to attended and unattended stimulus features was investigated in order to see if the early VAN ( ms) varied as a function of the level of attention. Long SOA (= 133 ms) and short SOA (= 33 ms) were used so that the participants were conscious of stimulus features and the mask in the long SOA and only of the mask in the short SOA. The stimuli consisted of gratings with high or low spatial frequency, orientated either horizontally or vertically resulting in four different conjunctions of spatial frequency (high, low) and orientation (horizontal, vertical) that were either masked or unmasked. When a conjunction was the target (relevant), all other conjunctions functioned as nontargets (irrelevant). The participants could also be presented with frequency-relevant stimuli alone, or orientation-relevant stimuli alone. This allowed for different levels of attentional relevance to be tested against the ERP related to visual consciousness. The VAN was elicited ms for conscious conditions compared to unconscious conditions similarly for conjunctions, frequency-relevant stimuli, orientation-relevant stimuli, and irrelevant stimuli. The SN was observed at ms, meaning that the VAN first appeared independently from selective feature-based attention and was then modulated by it from 200 ms and onwards. Because the VAN seems to be independent of nonspatial selective attention, Koivisto and Revonsuo (2008) hypothesized that it should emerge also in a passive task not requiring the focusing of attention to stimuli. In this second experiment, the same stimuli as in the first experiment were used with short and long SOAs. The participants task was just to passively fixate on a screen. Control trials were performed where the participants were to respond to a specific, predefined conjunction of features, where the other conjunctions then functioned as irrelevant stimuli. In the trials with passive fixation, the VAN was observed between ms after stimulus onset over occipital and posterior temporal sites. In the control trials, the early VAN was not affected by selective feature-based attention although it started to be

26 ERP Correlates of Visual Consciousness: A Review 26 modulated by it from 200 ms (as in the first experiment). Although the early VAN seems to be independent of selective feature-based attention, it was shown to be dependent on focal or spatial attention because it was attenuated in the trials with passive fixation. The VAN and spatial attention. It has been suggested previously that spatial attention is necessary to support our internal representation of space, which in turn is necessary for any consciousness experience to occur (Driver & Vuilleumier, 2001; Revonsuo, 2006). Koivisto et al. (2009) tested this hypothesis by manipulating spatial attention and conscious perception. Spatial attention was manipulated by keeping it orientated towards only one hemifield (either the left or the right visual field) by presenting task-relevant stimuli there to keep it busy. The stimuli consisted of three letters that were presented one to each eye and each letter shifted from being a target to a non-target between blocks. Masks at SOA (= 33 ms) could follow the letters so that both stimuli became invisible, or only the left visual field or the right visual field was masked. A second mask appeared after a blank screen for both visual fields rendering both images invisible. Control trials were included where stimulus letters were followed by a unilateral mask at SOA (= 33 ms) followed by masks to both visual fields at individually determined SOAs. Roughly half of the stimuli are consciously detected at these SOAs. This was done in order to control for differences in ERP response due to physical differences in stimuli (four or three masks). In both experiments, the VAN was not significant for unmasked stimuli at the unattended visual field. This indicates that the VAN is dependent on spatial attention (at least when there are more than one stimuli present), converging with the results of Koivisto and Revonsuo (2008). Visual Consciousness Without the VAN Some studies where consciousness has been investigated with the ERP technique have not reported the VAN. For example, in their CB experiment, Neideggen, Wichmann and

27 ERP Correlates of Visual Consciousness: A Review 27 Stoerig (2001) reported a positive wave (P3) 400 ms after stimuli onset to correlate with conscious change detection. No earlier activity was report here. In Neideggen et al. s study, ten alphanumeric symbols were presented that differed in font and size. In the second presentation, either the identity (e.g. from e to f) or the position of a character could be altered. Fernandez-Duque, Grossi, Thornton and Neville (2003) suggests that the P3 component in Neideggen et al. s study reflects the detection (outside of consciousness) of low-probability targets, which have previously been shown to elicit a P3 component (see Polich 2007 for a review). Fernandez-Duque et al. addressed this question by presenting the participants with a change repeatedly after the change had first been reported. Firstly, early (100 ms) activity related to the focus of attention was reported, followed by a broadly distributed late (350 ms) positive activity for changes that reached the level of consciousness. Even when the stimuli were no longer novel, a late positive wave was elicited in relation to conscious perception relative to stimuli not consciously seen. In a location task experiment where stimuli were kept near subjective threshold, Lamy et al. (2009) saw that all activity preceding the late positive wave they correlated with conscious perception was not different between conscious and unconscious conditions. Although it has been suggested that the reason for why the VAN was not observed is that the late positive activity is much larger than it and therefore easier to detect (Railo et al. 2011). Moreover, activity preceding the late positive deflection has been identified. Here, only the late positive activity was correlated with conscious perception (see below). The Late Positivity The LP arises typically over parietal and frontal lobes around 300 ms after stimulus onset. It is a positive difference between conscious and unconscious conditions overlapping with the P3 component (fig. 1). The P3 component has been suggested to reflect the updating

28 ERP Correlates of Visual Consciousness: A Review 28 of mental representation of new stimuli, the allocation of attention and the engagement of memory (Polich, 2007). The ERP technique was used to investigate brain activity for when a stimulus crosses the threshold to consciousness (Del Cul et al. 2007). A brief (16 ms) stimulus consisting of a digit was presented in one of four positions around the fixation cross. A mask consisting of letters were presented so that they surrounded the location of the target. There were six SOAs (= 16, 33, 50, 66, 83 and 100 ms) that were varied randomly. Also there were mask-only conditions. ERPs evoked from targets were extracted by subtracting ERPs evoked by the mask-only condition. Del Cul et al. used subjective measures where participants had to rate the visibility of the target (level of visibility from % visible). ERPs to conscious and unconscious conditions were compared to the subjective reports in order to find the ERP component that followed the curve seen in the subjective reports. Also, the ERP component that differed between consciously seen and not seen trials when SOA was kept at 50 ms (threshold to consciousness) was searched for. The subjective measures showed that the percentage of seen trials increased nonlinearly, following a sigmoidal curve: the increase was larger between 33 to 66 ms than for SOAs and ms combined. Several waveforms were extracted by subtracting the mask from the mask-target conditions. The waveforms elicited by the target were all seen to increase in amplitude with an increase in SOA. The ERP followed the same activity until 300 ms when a large positive wave was observed for consciously seen trials. This wave corresponding to the LP was the only wave observed that varied with the SOAs following a curve similar to the subjective reports. Also, when SOA was kept at the subjective threshold, only the LP differed between consciously seen and not seen trials. However, the study by Del Cul et al. rests on the dubious assumption that consciousness is an all-or-none phenomenon (Railo et al. 2011). A contending view is that consciousness is a continuous phenomenon,

29 ERP Correlates of Visual Consciousness: A Review 29 with varying clarity along this continuum (Overgaard, Rote, Mouridsen & Ramsoy, 2006). In the P300 paradigm participants respond to rare and not to frequent stimuli in a successive presentation of stimuli (Polich, 2007). Babiloni, Vecchio, Miriello, Romani and Rossini (2006) argue that the response only to consciously seen rare stimuli could cause a difference in the ERP in such paradigms. The authors (Babiloni et al.) therefore used a paradigm where participants had to respond both after consciously seen trials and not seen trials. In their experiment, white circle cue stimuli could appear either to the left or the right (50/50) at the threshold to consciousness. These stimuli cued the position of the subsequent go stimuli. The cue stimuli were consciously seen roughly half the times. The cue was preceded and succeeded by visual masks consisting of two Xs, one to the left and one to the right of the fixation point. Green circle go stimuli appeared thereafter shortly (around 500 ms) either to the left or the right (50/50). At around 300 ms after stimulus onset, the LP amplitude was seen to be largest for consciously seen trials than for unseen trials. Koivisto and Revonsuo (2010) have argued that the participants in the study by Babiloni et al. may not have been conscious of the masked stimuli that they reported as seen. Rather, the masked stimuli could have caused a faint sensation that would be enough for correct localization of the stimuli. In their location task experiment, Lamy et al. (2009) kept the stimuli near the threshold to consciousness that could appear in one of four possible locations. Participants could be conscious and correctly locate the stimuli (conscious-correct) and unconscious and correctly (above-chance) locate stimuli (unconscious-correct). To distill activity related to conscious perception, conscious-correct and unconscious-correct conditions were compared, thus equating performance on target localization. A larger LP was observed for conscious conditions around 375 ms than for unconscious conditions. The authors concluded that the observed LP can be seen to reflect conscious perception of the target specifically because