The neural mechanisms of figure-ground segregation Matthew W. Self 1 and Pieter R. Roelfsema 1-3

Transcription

1 The neural mechanisms of figure-ground segregation Matthew W. Self 1 and Pieter R. Roelfsema Department of Vision & Cognition, Netherlands Institute for Neurosciences, Meibergdreef 47, 1105 BA, Amsterdam, The Netherlands. 2 Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, VU University, Amsterdam, The Netherlands. 3 Department of Psychiatry, Academic Medical Centre, Amsterdam, The Netherlands. To appear in: Oxford Handbook of Perceptual Organization Oxford University Press Edited by Johan Wagemans Summary We will outline a neural theory of perceptual grouping in which perceptual objects are bound and segregated from their background through an interplay of feed forward, horizontal and feedback connections within the visual system. We will discuss how two key processes, boundary detection and region growing, rapidly and accurately solve a texture-segmentation task. We hypothesize that boundary detection proceeds using a combination of feed forward centre-surround interactions combined with suppressive, horizontal interactions between neurons tuned for the same visual feature. Boundary detection leads to an enhanced representation (firing-rate increase) at the borders of an object, and this enhanced firing-rate could be used to rapidly solve simple object detection tasks. For more complex situations involving novel or overlapping objects it is necessary to invoke the more time consuming incremental grouping processes. We will discuss theoretical and experimental evidence for a region-growing process that is able to enhance the firing-rates of cells in early visual areas that represent the entire surface of an object. We will discuss how these processes interact with attention and how they are implemented by the laminar micro-circuitry of cortex. Finally we will describe how these circuits can be manipulated using different glutamate receptor antagonists to explore their role in perceptual organization. 1. Introduction Vision appears to be simple. We open our eyes and perceive a well-organized world full of recognizable objects without any feeling of effort. The apparent ease with which we perceive the world disguises the immense computational efforts necessary to segregate, localize and recognize objects. The difficulty of this task stems from the fact that (daytime) vision is based on the distributed pattern of activity across the millions of cones in the retina. This point-like representation must be transformed by the neural circuitry of the visual system to produce our coherent percept. The ultimate goal of this circuitry is to localize and recognize objects and to guide visually driven behavior. To achieve this goal it is necessary to group together the activity patterns that are produced by one object (or figure) and to segregate these from patterns produced by other objects or background regions. 1

2 The neuronal mechanisms by which the visual system segregates a figure from its background and groups together the elements belonging to the figure have been studied using a texturesegmentation task. In the original version of this paradigm (Lamme 1995) a macaque monkey was required to fixate on a central fixation dot. Then a full-screen texture composed of thousands of oriented lines was presented. The texture contained a small square region made from lines of the orthogonal orientation (Figure 1a) (a version using motion-defined textures was also used and produced similar results). This region is perceived as a figure in front of, and therefore occluding, the background. The monkey s task was to make an eye-movement towards the figure after the presentation of a go-cue. In some experiments (Self et al. 2012; Supèr et al. 2001) there were also catch-trials with a uniform texture without a figure. On these trials the monkeys were rewarded for maintaining fixation at the centre after the presentation of the go-cue. Monkeys generally perform very well on this task with performance levels of greater than 90% correct. The virtue of this paradigm is that it is possible to vary the position of the figure relative to the receptive field(s) of the neuron(s) under study, while keeping the bottom-up activation of the neurons constant (Figure 1b). If the figure is placed in the receptive field then the response of the neuron to the figure can be tested (red condition in Figure 1b). If the figure is moved elsewhere then the response to the background can be measured (blue condition). Importantly the orientation of the textures is always counterbalanced so that on average exactly the same line elements fall into the RF in both the figure and ground conditions. This creates conditions in which the visual information present in the RF is identical but the visual context is different. On figure trials the RF falls on the behaviorally relevant texture, whereas on ground trials it falls on the irrelevant background region. Figure 1: a. An example stimulus used in the texture-segmentation task. The background texture covers the entire screen and the monkey s task is to make a saccade towards the small square figure region. b. In the figure condition the figure is centered on the RF of the recorded cell (red condition). In the ground condition the figure is moved so that the 2

3 RF falls on the background (blue condition). Note that the orientation is also reversed so that identical line elements are present inside the RF. The graph to the right illustrates the typical response of V1 cells. The early response (<100ms after stimulus onset) is the same regardless of whether the RF was on the figure or background. In the later timeperiod (>100ms) the responses to the figure (red line) are significantly higher than those to the ground (blue line). The shaded grey-region represents the modulation in firing and is referred to as figure-ground modulation (FGM). c. Boundaries can be detected through mutual inhibition between cells tuned for the same orientation. Here cells on either side of the boundary (the pink dashed line) have stronger responses than cells in the middle of the texture as they only receive inhibition (the black bars) from one side. d. Models of region-growing suggest that the figure-region becomes perceptually grouped through excitatory feedback from neurons in higher visual areas tuned to the figural orientation (red cone). This leads to enhanced firing-rates across the entire figure region. The responses of neurons in V1 are modulated by the visual context (Figure 1b). In the previous studies responses for the large majority of neurons were stronger when the RF fell on a figure compared to the background, on average by around 40% of the activity produced by the background. We will refer to this modulation in firing-rate as figure-ground modulation (FGM). Most notably this modulation did not begin until around 100ms after the onset of the texture (40-50ms after the initial visual response in V1). The initial response was identical regardless of the visual context showing that the input into V1 from the thalamus did not discriminate between figure and ground. A follow-up study showed that figures defined by other cues (color, motion, luminance, depth) produced similar levels of FGM in V1 (Zipser et al. 1996). How does the visual system segregate such a texture? Psychophysical studies (Mumford et al. 1987; Wolfson and Landy 1998) have suggested that there are two complementary mechanisms at work to segment the scene. The first is boundary detection, the enhancement of the borders of the object (Figure 1c). We will propose that boundary detection is achieved through a mixture of centresurround interactions mediated by feed forward anatomical connections and mutual inhibition between neurons tuned for similar features mediated by horizontal connections within visual cortex. These processes rapidly enhance neural firing-rates at locations in the visual scene where there are local changes in feature values. The second process is region growing, which groups together regions of the scene with similar features (Figure 1d). We will discuss evidence for a region-growing process in which a surface label (also enhanced neuronal activity) simultaneously arises across regions of similar feature values. We hypothesize that both processes exist in visual cortex and work together to rapidly and accurately segment the visual scene. The neural connection schemes for these processes are however, quite different, and their timing differs too. 2. Boundary detection 2.1. Theory of boundary detection A fundamental processing strategy in the visual system is to contrast feature information from nearby regions of space. This strategy has the dual-effect of making the visual system relatively insensitive to uniform regions of the scene and enhancing the responses to regions in which featurevalues change. A well-known example of the neural implementation of this strategy is the retinal ganglion cell. These cells have a centre-surround receptive field organization; they respond strongly 3

4 to an increase or decrease in luminance restricted in size so that it selectively activates the centre mechanism. They are less driven however by uniform regions of luminance which simultaneously activate the centre and surround mechanism. This organization makes these cells more responsive to luminance defined edges if the edge is correctly aligned with the receptive field. A retinal ganglion cell would not however be able to signal the presence of the boundaries in Figure 1a. These boundaries are defined by orientation and the luminance on each side of the boundary is the same. Such orientation defined edges cannot be detected in the retina or thalamus of primates because these structures lack cells that are selective for orientation; a cortical mechanism is required. In theory, orientation-defined texture boundaries could be detected by orientation-opponent cells driven by one orientation in their centre and the orthogonal orientation in their surround. Such cells have however yet to be found in visual cortex. Instead it has been proposed that these edges are detected through mutual inhibition between neurons tuned for the same orientation (Grossberg and Mingolla 1985; Knierim and Van Essen 1992; Li 1999; Marr and Hildreth 1980; Sillito et al. 1995). In such an iso-orientation inhibition scheme, the activity of neurons that code image regions with a homogeneous orientation is suppressed, whereas the amount of inhibition is smaller for neurons with RFs near a boundary so that their firing rate is higher (Figure 1c). There is a good deal of evidence that iso-orientation suppression exists in visual cortex. Cells in V1 that are well-driven by a line element of their preferred orientation are suppressed by placing line elements with a similar orientation in the nearby surround (Knierim & Van Essen 1992). These surrounding elements do not drive the cell to fire themselves and are therefore demonstrably outside the classical receptive field of the V1 cells, yet they strongly suppress the response of the cell to the centre element. Importantly this suppression is greatly reduced if the line elements outside the RF are rotated so that they are orthogonal to the preferred orientation of the cell. This result supports the idea that V1 neurons receive an orientation-tuned form of suppression coming from regions surrounding the RF (Allman et al. 1985; Jones et al. 2001; Kastner et al. 1999; Levitt and Lund 1997; Nelson and Frost 1978; Sillito, Grieve, Jones, Cudeiro, & Davis 1995). The time-course of this suppression is very rapid. Studies using grating stimuli have determined that iso-orientation suppression can be observed within 25ms of the onset of the visual response (Li et al. 2001; Nothdurft et al. 1999). One study which examined the latency of this effect at the level of individual cells found even shorter latencies of around 7-10ms (Bair et al. 2003). Thus, representations of the boundaries of objects in natural scenes are enhanced and projected forwards to higher visual areas as part of (for luminance defined boundaries), or closely following (for texture-defined boundaries), the initial feed-forward sweep of visual activity. Indeed, studies of the neuronal responses to the boundaries of texture defined figures in V1 (Lamme et al. 1999) and also in higher visual area V4 (Poort et al. 2012) find enhanced activity at around 70ms after stimulus onset Rapid detection performance and the limits of feed forward processing This rapid enhancement of neuronal activity at the edges of the figure may be sufficient to perform rapid detection tasks. The figures used in early studies of texture-segmentation were rather simple square forms (Lamme 1995; Zipser, Lamme, & Schiller 1996) and it is likely that detectors exist in higher visual areas which are activated by such simple and regular forms. The activity of such a detector would signal to the rest of the brain the presence or absence of the square-region in the scene, implicitly grouping together the boundaries of the object. Indeed primates show a remarkable ability to make rapid present or absent judgments when viewing rapidly presented sequences of natural images. For example we are able to very rapidly determine if a stream of images contains an animal or not, even when the presentation time of each image is reduced to 20ms per image (Thorpe et al. 1996). These ultra-rapid abilities may rely on the activation of cells in higher visual areas that 4

5 are tuned for characteristic diagnostic features (e.g. a cell tuned for the presence of an eye in the image would be sufficient to solve the above task). However there are limits to the abilities of cells in higher visual areas to group together the detected boundaries. For example neurons in inferotemporal cortex (IT) have RFs that cover almost an entire hemi-field. This is extremely useful for determining whether a particular grouping of features is present in the visual scene (Brincat and Connor 2004; Kayaert et al. 2005; Tanaka 1993), but information about the precise spatial location of the object is lost. Furthermore the use of specialized feature-detectors at high levels is limited to situations in which familiar objects are presented (Sheinberg and Logothetis 2001). It is highly unlikely that detectors exist for objects that have never been seen. Also, the early studies which examined responses in higher visual areas did so using anesthetized preparations and usually presented one object on the screen at any one time. Studies in awake-behaving animals using multiobject scenes have revealed that there are very strong inhibitory interactions which control the flow of information through this feed-forward network (Miller et al. 1993; Sheinberg & Logothetis 2001). Stimulus representations compete with one another so that at the level of IT there may only be active representations for one or a few objects at a time (Desimone and Duncan 1995). This competition is strongly biased by behavioral relevance so that relevant objects tend to win the representational battle (Luck et al. 1997; Reynolds et al. 1999). In natural images that typically contain many overlapping objects this may mean that very few objects are represented at high levels of the visual system placing a severe limit on the number of objects that can be grouped by fast feed forward processes. In summary feed forward grouping of elements using complex receptive fields has many advantages, such as its speed. It is unlikely, however, that feed forward processing would be able to correctly group scenes containing novel objects and determine their location with high spatial resolution. Furthermore, the inhibitory interactions that curtail the flow of information towards higher visual areas imply that feed forward processes are not sufficient to group scenes containing multiple, overlapping or ambiguous objects. In these situations extra grouping processes are required which are more flexible, but this additional flexibility may come at the cost of taking more time. 3. Region-growing 3.1. What is region-growing? How is the rest of the object grouped together once its boundaries have been detected? One mechanism that has been used in computational models is region-growing. Region-growing is the counterpart to the boundary-detection process described above. Whereas boundary-detection enhances responses at the borders of an object, region-growing has been proposed to begin in regions of uniform feature-values and to spread outwards until encountering a feature-boundary (Grossberg & Mingolla 1985), although we will later suggest that region-growing proceeds simultaneously across large regions of uniform texture. Region-growing relies on statistical similarities between features (Grossberg & Mingolla 1985; Mumford, Kosslyn, Hillger, & Herrnstein 1987; Wolfson & Landy 1998). Regions with similar features are grouped together and thereby segregated from regions with different feature values. Psychophysical studies have demonstrated that the performance of human observers on shape discrimination tasks is best explained by models which use mechanisms for boundary-detection as well as for region-growing (Mumford, Kosslyn, Hillger, & Herrnstein 1987). Indeed, humans can discriminate between textures which are physically separated from one another so that the boundary-detection process cannot be used (Wolfson & Landy 1998). Computational models of texture segmentation stipulated that region-growing requires 5

6 an entirely different connection schemes than boundary-detection (Bhatt et al. 2007; Grossberg & Mingolla 1985; Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012; Roelfsema et al. 2002). Whereas boundary-detection requires iso-orientation inhibition, i.e. cells encoding the same feature should inhibit one another (as was discussed above), region-growing requires iso-orientation excitation, which means that cells that represent similar features enhance each other s activity A computational model of region-growing How is it possible that the visual system implements these opposing connection schemes? One solution would be that the different schemes are implemented during different phases of processing. The boundary-detection process has a relatively short latency of <20ms after the initial visual response in V1 and V4 for texture defined-boundaries 1. In contrast, figure-ground modulation at the centre of a figure-region has a longer latency of >50ms after the initial visual response. However, a difference in timing is unlikely to be the only explanation. It would require that the connectionschemes of visual cortex switch from iso-orientation suppression to iso-orientation enhancement within 20-30ms! Such a dramatic and rapid reorganization of connectivity is highly unlikely. It is more likely that these two processes make use of different sets of cortico-cortical connections. We have previously suggested that boundary-detection algorithms use feed forward and horizontal connections whereas region-growing processes use feedback from higher to lower visual areas (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012; Roelfsema, Lamme, Spekreijse, & Bosch 2002). The implication is that feed forward and horizontal projections can implement centresurround comparisons within the RF and iso-orientation suppression over small spatial scales in early visual areas and over larger scales in higher areas. Feedback connections would then propagate region filling signals from the higher areas back to the lower areas (Figure 2). Figure 2: a. A model of figure-ground segmentation (Roelfsema, Lamme, Spekreijse, & Bosch 2002; Roelfsema and Houtkamp 2011). Neurons encoding the edges of the figure 1 It should be noted that this latency only applies to texture-defined boundaries. Luminance defined boundaries can be detected through centre-surround processes such as the receptive-field of the retinal ganglion cell described above. Enhanced activity at luminance defined boundaries can be seen in the feedforward input into V1 (Sugihara et al. 2011) and does not require the kinds of interactions that we discuss here. 6

7 have enhanced activity as they receive less horizontal inhibition (orange arrows) from their neighbors. b. The input stimulus produces increased activity throughout the visual hierarchy (averaged across orientation maps), the edges of the figure merge together in the large RFs of high-level areas such as TEO. c. Neurons in higher visual areas send feedback back to neurons in lower areas. This feedback is gated by the activity of neurons in lower visual areas and enhances responses throughout the figure (regiongrowing). d. The result of the model is that early responses are enhanced at the boundaries of the figure whereas at later time-points the response enhancement also spreads to the center of the figure. Adapted from (Roelfsema & Houtkamp 2011). This division was made explicit in a computational model of texture-segmentation (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012; Roelfsema, Lamme, Spekreijse, & Bosch 2002). In this model feature-maps were present at multiple spatial scales in a multi-layer visual hierarchy. At each level of the hierarchy there was iso-orientation inhibition for the detection of edges. This architecture has the result that for any given figure-size there will be a level in the model hierarchy at which the figure appears as a singleton amongst distracters, i.e. a form of pop-out (V4 in Figure 2a; TE in Figure 2b). Iso-orientation excitation for region growing is implemented in the feedback pathway. Neurons at the higher level where pop-out occurred then send a feature-specific feedback signal back to earlier visual areas to enhance the response of neurons encoding the same feature and suppress the responses of neurons encoding the opposite feature (Figure 2c). For example, a figure composed of leftwards oriented line-elements strongly activates leftwards-preferring cells in a highlevel area (e.g. IT). These cells send feedback to earlier processing levels and ultimately also to V1 to activate only those cells that prefer leftwards oriented line-elements and to suppress those that prefer rightwards. One further computational rule is required by the model to restrict the enhanced activity to the interior of the figure. The feedback connections have to be gated by feed forward activity, so that only those cells that were well activated by the feed forward sweep of activity are modulated by the feedback signal. This ensures that feedback only excites cells that are activated by an orientation close to their preferred orientation. In the example given here this ensures that feedback does not excite cells that are tuned for the leftward orientation with RFs outside the boundaries of the figure (where the orientation of the line elements is rightwards) and that the region-growing signal stays focused on the representation of the figure. The final result of this model is that the figure-region becomes grouped through enhanced firing-rates in early visual areas compared to the background (Figure 2d). The model is able to reproduce the firing-rate modulations observed in the texture-segmentation tasks described above (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012; Roelfsema, Lamme, Spekreijse, & Bosch 2002). Furthermore the model is able to correctly segregate more complex figures such as N or U shapes or figures with holes which contain potentially confusing interior convex regions which might mistakenly be segregated as figures (Roelfsema, Lamme, Spekreijse, & Bosch 2002). While the model initially incorrectly assigns figure status to the interior of the N/U or the hole, this is later overruled by feedback from higher areas which do not extract the interior of these figures due to the poor spatial resolution of their RFs Alternative explanations for FGM The computational model described above would predict that the enhanced activity observed at the boundaries of the figure relies on mechanisms that differ from those for FGM at the centre of the figure. This prediction has been debated by other groups which have suggested that figure-ground modulation is strongly related to the mechanisms that underlie boundary-detection. Zhaoping Li has presented a model (Li 1999) where FGM mainly arises through iso-orientation inhibition. This mechanism, which according our aforementioned model is responsible for boundary detection, was 7

8 able to reproduce some results of earlier studies of FGM, but it cannot explain the FGM in the centre of larger figures. Another group (Rossi et al. 2001) has suggested that FGM could only be observed with very small figures (up to 2 degrees in diameter) and did not observe FGM in the centre of larger figures. They suggested that FGM is in fact a boundary-detection signal and becomes greatly reduced as one moves away from the boundary. Both of these viewpoints suggest that there is no regiongrowing signal present in V1 and that neural activity in V1 does not reflect surface perception, but rather the presence of nearby boundaries. Poort et al. (2012) reconciled these apparently conflicting findings by showing that region growing is only pronounced for behaviorally relevant objects (see below) A relationship to border ownership? Is the FGM signal observed by Lamme (1995) simply a boundary detection signal? If so, it is unclear why this signal would be restricted to the figure and not also spread out from the boundary into the background. Lamme (1995) showed that FGM is completely absent, or even slightly negative, on background regions close to the figure boundary, whereas the modulation was at a similar level throughout the figure region. This result demonstrates that if boundary-detection signals spread from the borders of an object then this is mediated by a system which has access to which side of the border is object and which side is background. Border-ownership cells provide a possible neural substrate for this mechanism. The concept of border-ownership is dealt with in more detail in chapter 12 (Kogo and van Ee), for our purposes here it is sufficient to know that cells in visual cortex represent border ownership in modulations of their firing-rate (Zhou et al. 2000). For example, a rightwards tuned border-ownership cell will give a greater response when an edge is owned by an object to the right of its RF than when it is owned by an object to the left. In this way borderownership cells can give a spatial signal as to which direction to start spreading a boundary-signal. Border-ownership cells are found in small numbers in V1, and in much greater numbers in V2 and V4. In fact most orientation-selective V2 and V4 neurons are also border-ownership selective highlighting the fundamental nature of border-ownership coding (Zhou, Friedman, & von der 2000). The mechanisms by which border-ownership tuning might arise in these cells were recently discussed by Craft et al ( 2007). Their theory (see also Jehee et al., 2007) relies on the presence of, as-of-yet theoretical, grouping cells in higher visual areas (V4 and above). Grouping cells are activated by the presence of convex, enclosed contours and send feedback to BO-cells in lower areas which are aligned with the contour. This elegant theory can explain how BO-tuning arises, although experimental evidence for grouping cells remains to be found. Computational models suggest that firing-rate modulations shown by BO-tuned cells in V2 could be used as a seed to spread a label in the correct direction within the object, and not outwards into the background (Kogo et al. 2010). The models described above share some similarities with our model in that recurrent processing between neurons with small RFs at low levels of the visual system and those with large RFs at high levels in the visual system is used to determine border ownership. Our model differs in that it specifies a mechanism by which the entire figure region can be labeled simultaneously with enhanced neural firing. The models of Craft et al., ( 2007) and Jehee et al., ( 2007) are concerned with correctly assigning border-ownership and do not make predictions about how FGM arises in V1 and the model of Kogo et al. ( 2010) suggests that FGM would arise first at the boundaries of an object and spread towards the center. Nevertheless these models, and those of Grossberg (Bhatt, Carpenter, & Grossberg 2007; Grossberg & Mingolla 1985), all suggest that feedback to lower visual areas is essential in grouping together the figure region. These models are therefore very different from those of Zhaoping Li who proposes that intra-areal horizontal connections are sufficient to 8

9 assign figure-ground status and that FGM is simply a spreading of boundary-detection signals from the borders of the object (Li 1999; Rossi, Desimone, & Ungerleider 2001; Zhaoping 2005). We have carried out two recent studies which directly investigated the contribution of feed-forward, lateral and feedback connections to boundary detection and region growing. In the first (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012) we studied the effect of task-relevance on the enhanced firing at the boundaries and the center of a figure. We found that FGM at the center of the figure (region filling) depends strongly on the task that the monkey is doing, whereas boundary detection has only a weak dependence. This result indicates that the processes that underlie boundary detection are largely stimulus-driven, in accordance with a strong contribution from lateral and feed-forward inhibition, and that region-filling indeed depends more strongly on feedback connections from higher visual areas. In the second study (Self et al. 2013) we made laminar recordings of activity in V1 while monkeys performed a figure-ground task. Importantly, these laminar recordings provide unique information about the neural circuitry underlying FGM as they allow us to examine the synaptic currents and spiking changes that are produced at the borders and centre of a perceptual figure. We found that boundary-detection engages different laminar circuits than region-filling. Taken together these studies suggest that FGM observed at the center of the figure is not an extension of a boundarydetection signal at the edges. 4. The neural mechanisms of FGM 4.1. The effect of attention on FGM We have hypothesized that the detection of the boundaries of a figure relies on different neural mechanisms than the FGM at the centre of the figure. If this is the case then these two processes may be affected differently by the task-relevance of the figures. In this study (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012) we recorded neural activity from V1 and V4 while monkeys made eye movements towards a texture-defined figure or ignored it. We varied the animals attention by presenting two possible tasks. The upper half of the screen contained two luminance-defined curves for the first curve-tracing task where the monkey was trained to make an eye-movement towards the end of the curve that was connected to the fixation point. In the lower half of the screen a texture-defined figure was present for a texture-segregation task where the animal had to make an eye-movement towards the center of the figure (Figure 3a). The animals performed only one task per day, so that if he was performing the curve-tracing task he would ignore the figure and vice-versa. We shifted the location of the figure so that the neural responses to the figure-edge or figure-center could be recorded along with intermediate locations and responses to the background (Figure 3b). 9

10 Figure 3: a. The paradigm used to study the effect of attention on FGM. The monkeys were always presented with two curves in the upper-half of the screen and a texturedefined figure in the bottom half (shown in plain colors here for simplicity). On different days the monkey performed different tasks. On curve-tracing days the monkey had to make an eye-movement towards the target circle that was connected to the fixationpoint by a curve. On figure-detection days he had to make a saccade towards the figure. b. The position of the figure relative to the RF was varied on each trial to map out responses to the background, edge and center of the figure. c. The 3d color-plot shows the amount of FGM according to position of the figure during the figure-detection task. The plot on the left-hand side shows the response at the edge of the figure (red) vs. the center (blue). d. FGM during the curve-tracing task. When attention is directed to the curve-tracing task the level of FGM is reduced in the center of the figure. The response at the edges was relatively unaffected. Adapted from (Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012). When the animal was performing the figure-detection task we observed that neuronal responses to the figure were enhanced relative to responses evoked by the background, just as in Fig. 1b. We isolated the FGM signal (grey regions in Fig. 1b) by subtracting background responses from responses evoked by the figure. In the figure-detection task, FGM in V1 neurons was similar regardless of whether their RF was located on the figure or on the boundary (Figure 3c). The level of FGM was similar to those obtained in previous studies. However, when the animal was performing the curvetracing task we observed a drop in responses to the figure center whereas responses to the 10

11 boundaries were relatively unaffected (Figure 3d). These results show that the detection of the boundary, which we have linked to iso-orientation suppression, proceeds equally well in presence or absence of attention. Previous studies have also demonstrated enhanced edge-responses when animal ignore a stimulus (Marcus and Van Essen 2002) or even when animals are anesthetized (Kastner et al. 1997; Nothdurft, Gallant, & Van Essen 1999; Nothdurft et al. 2000). In contrast, our results show that the responses at the figure center depend on the task-relevance of the figure. When the figure is behaviorally relevant then responses at the center of the figure are similar to those at the edge, but when attention was directed to the other task the responses fell to approximately halfway between the edge responses and the response to the background. This result leads us to draw two conclusions. Firstly, that the process responsible for boundary-enhancement is different to the process responsible for FGM at the centre of the figure. Secondly, while FGM at the figure-centre is influenced by attention, it still arises in the absence of attention. These results are in good agreement with a study that examined the effect of attention on border-ownership cells (Qiu et al. 2007), which found that border-ownership signal can also be observed outside the focus of attention, but that attention can amplify coding of border ownership. These results are consistent with our hypothesis that boundary detection, which is thought to rely on iso-orientation inhibition, depends on an early process that may rely of feed-forward or lateral connections (Fig. 2a), whereas the FGM at the figure centre depends on iso-orientation excitation, which is mediated by feedback from higher visual areas (Fig. 2c). A process that depends on the activity in higher visual areas is expected to depend more strongly on the task-relevance of the figure. What then is the advantage of enhancing neural activity on figures compared to background? One possibility is that by increasing the responses of neurons in early visual areas, which have small RFs providing excellent spatial resolution, the visual system can more accurately localize the figure to guide behavior. The neuronal processes that are responsible for making a saccade to the centre of the figure might take advantage of the FGM, because it selectively labels all the image elements of the figure. The spatial profile of FGM can therefore be read out by the saccadic system to determine the centre of gravity of the image elements that belong to the figure. We assessed this possibility by examining the relationship between the level of FGM in V1 and the spatial accuracy of the saccade. The animals in this study were required to make very accurate saccades to a 2.5 window centered within the 4 figure. We found that the spatial profile of FGM in V1 indeed predicted the landingpoint of the saccade on the figure. On trials where FGM was strongest on the left-hand side of the figure the animal tended to make saccades that landed to the left of centre. The opposite was observed on trials with strong FGM on the right-hand side. Trials with modulation spread evenly through the figure were associated with the most accurate saccades. This result suggests that the FGM signal in V1 is used by the motor-system to plan saccades to the centre of gravity of the image elements that belong to the figure, possibly through the direct projections from V1 to the superior colliculus (Fries and Distel 1983; Wurtz and Albano 1980). These and previous results, taken together, show that the activity in V1 is closely associated with both the perception of the animal (Supèr, Spekreijse, & Lamme 2001) and the spatial accuracy of the behavioral output The laminar circuitry of figure-ground segregation We have suggested above that increased firing-rates at the boundaries of a figure might be mediated by feed-forward and horizontal connections within V1 whereas FGM at the centre of the figure could be due to feedback projections. These different projections target different layers of V1. Feedforward connections predominantly target layer 4c and layer 6, horizontal connections are present in all layers but are particularly dense in upper layer 4 and the superficial layers (Gilbert and Wiesel 1983; Rockland and Pandya 1979) and feedback connections (from object processing areas of the 11

12 ventral stream) target layer 1 and 5 most strongly (Anderson and Martin 2009; Rockland & Pandya 1979; Rockland and Van Hoesen 1994; Rockland and Virga 1989), and in general tend to avoid layer 4c (Douglas and Martin 2004; Felleman and Van Essen 1991; Nassi and Callaway 2009). We therefore recorded simultaneously from all the layers of V1 while two macaque monkeys performed a texturesegregation task that had been used previously (Supèr, Spekreijse, & Lamme 2001). We used a multicontact laminar electrode (Plexon U-probe ) that allowed us to measure multi-unit spiking activity (MUA) and the local field potential from 24 linearly spaced contacts. The advantage of these electrodes is that they also allow the application of current source density (CSD) analysis to the local field potential (Mitzdorf 1985;Schroeder et al. 1991;Schroeder et al. 1998). This analysis reveals the laminar locations of current sinks (currents flowing into neurons) and current sources (mostly passive current return to the extracellular space). We recorded MUA and CSD responses evoked by the centre and edge of the figure, as well as to the background texture. The results of this study were very revealing. Firstly we found strong laminar variations in the strength of FGM at the center of the figure (Figure 4a). FGM was strongest in the superficial and deep layers and significantly weaker in layer 4. The latency of modulation was relatively constant across the layers, beginning at around 100ms after stimulus onset, so from latency analyses it was difficult to determine the source of this increase in spiking. Even more revealing was the difference in current-flow between the figure and ground conditions. In the figure condition we observed extra current sinks flowing very superficially in layer 1 and/or upper layer 2 as well as in layer 5 (Figure 4b). These layers are well-known to be the targets of feedback projections from V2 to V1 (Anderson & Martin 2009; Rockland & Pandya 1979). These results therefore support the idea that feedback projections, targeting layers 1 and 5, are the source of the increased spiking in V1 for the center of the figure. When we placed the boundary of the figure in the RF we observed an extra component to the FGM signal that started at approximately 70ms after stimulus onset (arrow in Figure 4c). This early boundary-fgm has also been observed in previous studies of texture-segregation (Lamme, Rodriguez-Rodriguez, & Spekreijse 1999; Nothdurft, Gallant, & Van Essen 2000; Poort, Raudies, Wannig, Lamme, Neumann, & Roelfsema 2012), but interestingly in our study the modulation was confined entirely to the superficial layers of cortex. At later time-points (>100ms) this modulation was followed by a pattern of spiking activity very similar to that observed at the figure center. CSD analysis revealed an extra current sink in the edge condition compared to the centre at around 70ms beginning in upper layer 4 and extending into the superficial layers at the same time as the increase in spiking in these layers (arrows in Figure 4d). It is clear from both the pattern of MUA and CSD that the mechanisms underlying early FGM at the edge of the figure differ from the mechanisms responsible for the FGM at the centre. On the other hand, at later time-points (>100ms) the MUA and CSD modulation at the edge resembled quite closely the FGM at the centre. We therefore suggest that the early edge FGM is the result of horizontal projections which are densest in upper layer 4 and superficial layers, whereas the later FGM at the edge might reflect a feedback-signal targeting the entire figure-region. This study therefore provides good evidence that both boundary detection processes (mediated by local connections) and region-filling processes (mediated by feedback connections) play a role in segregating textures and that these processes occur in different layers of cortex, and at different times. 12

13 Figure 4: FGM in the center of the figure (a) and at the edge (c) averaged across a number of penetrations. The color-plots show the laminar profile of FGM the difference in MUA evoked by figure and background. The edge specifically causes early FGM in the superficial layers (white arrow in c). The panels above show the MUAresponse averaged across all laminae; panels to the right show MUA response averaged across time. b. Difference in the CSD evoked by the figure center and background. Warm colors show stronger sinks in the figure condition (and/or stronger sources in the ground condition) and cooler colors stronger sources. The black arrows indicate the first sinks that differentiate between figure and background at a latency of ~100ms in layer 5 and layer 1. d. The difference in CSD between the figure edge and the background. The earliest sinks occur in upper layer 4/layer 3 and then in layer 2 (black-arrows) Feature-specific feedback signals An important requirement for the region-growing signal is that it should respect the boundaries of the figure and should not grow beyond them. In the computational model described above this is partially achieved by using a feature-specific signal. The orientation of the figure is represented by orientation-tuned cells in higher visual areas, which send back a spatially-imprecise, but featureselective signal to lower visual areas. The feature-specificity of the feedback signal ensures that the FGM does not spread onto cells that code the background orientation. This mechanism is effective in the computational model, but the feature-specificity of feedback signal in visual cortex is not yet completely resolved. 13

14 There are several lines of evidence to support feature selective feedback. The first stems from studies of feature based attention. It is well documented that primates can be cued to attend to a particular feature (e.g. the red items in a multi-color display). This can be extremely useful in visual search tasks in which the subject has to locate a target object amongst multiple distracters. Indeed a feature-specific modulation of activity of early visual areas forms a key part of theories of visual search such as feature-integration theory and guided search (Treisman and Gelade 1980; Wolfe et al. 1989). Neurophysiological studies of feature-based attention have found that the responses of neurons encoding the cued feature are enhanced throughout the visual scene (Martinez-Trujillo and Treue 2004; Roelfsema et al. 2003; Treue and Martinez Trujillo 1999; Wannig et al. 2011). These observations suggest that top-down attentional systems can select neurons based on their featuretuning. In spite of these feature-selective feedback effects on neuronal firing rates, the anatomical evidence for feature-specific feedback is mixed. Early studies examined the spatial extent of neurons that send feedback projections back to V1 by injecting retrograde tracers into V1 of cats (Salin et al. 1989; Salin et al. 1995) and monkeys (Perkel et al. 1986). These studies found a good match between the size of the region in V2 that projects to a column in V1 and the size of the region of V2 that receives feedforward projections from that column (Salin, Kennedy, & Bullier 1995). However, as V2 RFs represent much larger regions of space than V1, this means that a V1 column receives feedback from neurons encoding a much larger region of visual space than they themselves represent (Salin and Bullier 1995). These results raised the question of whether feedback projections would be able to provide a signal of sufficient spatial resolution to mediate FGM. Furthermore, these projections were described as producing relatively diffuse patterns of terminal arborisations, suggesting that they would not be able to form the basis for a feature specific signal (Maunsell and Van Essen 1983;Rockland & Pandya 1979). In accordance with this view, Stettler et al. (2002) reported that feedback projections from V2 to V1 in monkey visual cortex are not specific for orientation. However, more recent studies using more specific tracers have found instead than feedback projections are more specific than previously described. The terminal arborisations of feedback-axons have a patchy appearance in V1, suggesting that they target specific orientation columns (Angelucci et al. 2002; Angelucci and Bullier 2003; Shmuel et al. 2005). Thus, although there is clear functional evidence for feature-specific feedback signals in early visual cortex, the anatomical substrate of these effects remains to be fully elucidated Gating of feedback effects by feed-forward activity Feature-specific feedback would ensure that modulation does not spill-over onto neurons activated by the background texture. However this mechanism, by itself, does not prevent that feedback connections activate cells tuned for the orientation of the line-elements inside the figure, but with a RF located on the background. To prevent these cells from becoming modulated it is necessary to gate feedback effects using feed forward activity (Roelfsema 2006). Are feedback effects in visual cortex indeed gated by feed-forward activation? There is substantial evidence that feedback-based effects are strongest for cells that are wellactivated by the visual stimulus (Ekstrom et al. 2008;Treue & Martinez Trujillo 1999) but it is unclear how this arises. Long-range cortico-cortical connections are known to use glutamate as their neurotransmitter (Johnson and Burkhalter 1994) and, in principle, feedback projections might be able to drive their target neurons, even if these neurons are not in an active state. Crick and Koch ( 1998) argued that this would be an undesirable situation because it might lead to strong feed forward-feedback loops which could drive activity towards deleterious, even epileptogenic levels of activity (Crick and Koch 1998). 14

15 The question why feedback only modulates neural activity whereas feed forward projections drive neural responses is not entirely resolved (Sherman and Guillery 1998). One possibility raised by computational models is that feed forward and feedback projections utilize different glutamate receptors (Dehaene et al. 2003; Lumer et al. 1997). A main ionotropic glutamate receptor in cortex is the AMPA receptor (AMPA-R) which is a rapidly activated channel, well-suited to drive a neuron s membrane potential above threshold. The other principle glutamate receptor is the NMDA receptor (NMDA-R) with a more slowly opening channel. The current-passed by this receptor shows a nonlinear relationship with membrane voltage (Daw et al. 1993). At strongly negative membrane potentials the channel does not pass current as it is blocked by the presence of a magnesium ion in the channel pore. At the more depolarized levels that occur if a cell receives other sources of input, the magnesium block is removed and the channel begins to pass current. This mechanism implies that NMDA-Rs can act as coincidence detectors that are only active if the neuron is depolarized by AMPA-R activation (Daw, Stein, & Fox 1993). NMDA-Rs would therefore be well-placed to mediate the gating of a feedback-based modulatory signal, as these receptors are unable to activate neurons that are not receiving synaptic input from other sources. There is some evidence to suggest that NMDA-Rs may be more strongly involved in feedback processing than in feed forward transmission. For example responses in thalamo-cortical recipient layers are unaffected by APV, a drug that blocks all NMDA-Rs (Fox et al. 1990; Hagihara et al. 1988). Furthermore, NMDA has found to produce multiplicative effects on firing in the superficial and deep layers of visual cortex (Fox, Sato, & Daw 1990) and NMDA-Rs therefore provide a possible mechanism for the gating of feedback by feed forward activity. It is unlikely however that feedback connections target synapses that only possess NMDA-Rs as synapses without AMPA-Rs are not functional. It is possible however that feedback connections target synapses that are particularly rich in NMDA-Rs. An alternative possibility has been raised by through the work of Matthew Larkum who has shown that NMDA-Rs are required to integrate the inputs to the apical dendrites layer 5 neurons (Larkum et al. 2009). These dendrites are found in layer 1, the layer which is the predominant target of feedback connections. It may be possible therefore that feedback connections target layer 1, but cannot effectively modulate the firing-rate of cells unless NMDA-Rs are activated The pharmacology of figure-ground modulation We recently investigated the role that different glutamate receptors play in the texturesegmentation task described earlier (Self, Kooijmans, Super, Lamme, & Roelfsema 2012). Our hypothesis was that FGM would predominantly rely on NMDA-R activation and would be blocked by the application NMDA-R antagonists. In contrast we suggested that feed forward processing of the signal would rely on AMPA-R activation, but that these receptors would play no role in producing FGM. To address this hypothesis we made laminar recordings from V1 in the same manner as described above with one slight modification. The laminar electrodes now contained a fluid-line that allowed us to inject pharmacological substances into different layers of cortex. We used CNQX, an AMPA-R antagonist and APV and ifenprodil, which both block NMDA-Rs but with different subunit specificity. APV is a broad-spectrum NMDA-R antagonist which blocks all NMDA-Rs whereas ifenprodil is much more (>100x) specific for NMDA receptors containing the NR2B subunit. In the texture-segregation task, the effects of the AMPA-R antagonist differed markedly from those of the NMDA-R antagonists. CNQX strongly reduced responses in an early response window (50-100ms after stimulus onset). Activity in this time-period is mostly related to feed forward activation. Remarkably though, this drug had little effect on the level of figure-ground modulation (Figure 5a). Indeed the level of modulation measured after injections of CNQX was not significantly different from pre-injection levels. In contrast, both NMDA-R antagonists strongly reduced FGM, whilst having opposing effects on the initial neural response. APV reduced responses during the early time 15