33 Roles of Visual Area MT in Depth Perception

Transcription

1 33 Roles of Visual Area MT in Depth Perception gregory c. deangelis abstract One of the most impressive capacities of the visual system is the ability to infer the three-dimensional structure of the environment from images formed on the two retinas. Although several areas of visual cortex are involved in computing depth, the precise roles of different areas in three-dimensional vision remain unclear. It is important to establish how neural representations of depth in different brain regions are specialized to perform different tasks. This chapter summarizes studies that establish such links between representation and function in visual area MT. The nature of the representation of binocular disparity in MT is first considered, along with the functional roles of MT in coarse and fine depth discrimination. The recently discovered role of area MT in computing depth from motion parallax is then examined. These findings are compared with those from other visual areas to consider possible functional streams of analysis in threedimensional vision. We carry out our daily activities in a three-dimensional (3D) environment. Therefore a fundamental task for the visual system is to construct a 3D representation of our surroundings. This is difficult because the image formed on the retina of each eye is a two-dimensional projection of 3D space hence there is no direct quantitative information about depth in a single retinal image. Rather, the depth structure of the scene must be reconstructed by the brain. The visual system makes use of a wide variety of cues to estimate depth relationships (Howard & Rogers, 1995, 2002). Broadly speaking, these cues can be placed into two categories that I shall label pictorial cues and geometric cues. Pictorial cues to depth are those that are present in a single snapshot of the scene, including occlusion, perspective, shading, relative size, texture gradients, and blur. Together, these cues can be potent, as is evidenced by the fact that we can infer depth relationships in photographs. However, they generally provide only ordinal depth information or require prior knowledge to provide metric depth information. For example, the size of an object in the retinal image can be used to estimate the distance to that object if one knows the true physical size of the object. gregory c. deangelis Department of Brain and Cognitive Sciences, Center for Visual Science, University of Rochester, Rochester, New ork. Geometric depth cues are those that arise when a scene is viewed from multiple vantage points. For species with frontally located eyes, the horizontal separation between the two eyes generates systematic differences known as binocular disparities between the images projected onto the two retinas (figure 33.1A). Thus images from two simultaneous vantage points are available. Binocular disparity (hereafter referred to as disparity) is known to be sufficient to provide precise depth discrimination in the absence of other depth cues, as demonstrated with random-dot stereograms (Howard & Rogers, 1995; Julesz, 1971; Parker, 2007). Combined with an estimate of viewing distance (the distance from the eye to the point of fixation), disparity can provide quantitative estimates of the location of objects in depth. Another geometric cue, motion parallax, arises because of translation of the observer, as illustrated in figure 33.1B. As the observer s head moves from left to right, for example, the vantage point of the left eye changes over time. If the observer s head moves through one interocular distance, then the image that is projected onto the retina of the left eye will vary over time, the endpoint being the same view that would be seen by the right eye at the beginning of the movement (figure 33.1B). Thus there is a formal geometric similarity between disparity and motion parallax cues, at least when the latter arise because of lateral head movements. This means that motion parallax can provide metric depth information when a subject views the scene with one eye, as long as the eye moves relative to the scene. Not surprisingly, then, humans can make judgments of depth from motion parallax that are similar in precision to judgments based on disparity (Rogers & Graham, 1979, 1982). As we shall see later, the similar geometry of these two cues suggests that they might be processed using the same neural mechanisms. Where and how are depth cues processed in the brain? For the pictorial depth cues, very little is known about the neural mechanisms that lead to depth percepts; therefore I shall not consider pictorial cues further here. Until recently, very little was also known about the neural basis of depth from motion parallax, and we shall consider the available physiological information in the last section of this chapter. By comparison, a great deal is known about the neural circuits that process disparity cues for depth perception, as has deangelis: roles of visual area mt in depth perception 483 Gazzaniga_33_Ch33.indd 483 3/5/2009 8:59:56 PM

2 Figure 33.1 Binocular disparity and motion parallax as depth cues. (A) Points falling along the geometric horopter, or Vieth- Muller circle (curved line), have zero binocular disparity. A far object (open symbol) projects to disparate points in the two retinal images (bottom). (B) If the head translates rightward, the image of the far object moves on the retina. If the eye moves through one interocular distance, the position change on the retina due to motion parallax is equivalent to the binocular disparity. Hence depth from motion parallax is often expressed in units of equivalent disparity. also been reviewed elsewhere (Cumming & DeAngelis, 2001; DeAngelis, 2000; Gonzalez & Perez, 1998; Parker, 2007). I shall focus mainly on work that has been performed using macaque monkeys as experimental subjects. However, it should be noted that many important contributions have also been made in other species, particularly cats (for reviews, see Freeman, 2004; Freeman & Ohzawa, 1990; Ohzawa, DeAngelis, & Freeman, 1997). The primary visual cortex (V1) was the major focal point of most early physiological studies of disparity processing (Barlow, Blakemore, & Pettigrews, 1967; Poggio & Fischer, 1977). Neurons in V1 provide the initial encoding of disparity signals (Cumming & DeAngelis, 2001), and the representation of disparities in V1 likely limits the precision of a number of aspects of stereopsis (Nienborg, Bridge, Parker, & Cumming, 2004, 2005; Prince, Pointon, Cumming, & Parker, 2000). However, a series of elegant experiments over the past decade has demonstrated that the representation of disparity in V1 is not sufficient to account for various aspects of our perceptual experience of depth (Cumming & Parker, 1999, 2000, 1997; Nienborg & Cumming, 2006). Therefore it seems clear that disparity processing beyond V1 is critical to account for behavior. During this same period, we have learned that disparity signals are represented in a variety of visual cortical areas that were not previously known to contain disparity-selective neurons, including area V4 (Hegde & Van Essen, 2005; Hinkle & Connor, 2005; Tanabe, Doi, Umeda, & Fujita, 2005), inferotemporal (IT) cortex ( Janssen, Vogels, Liu, & Orban, 2003; Janssen, Vogels, & Orban, 1999, 2000b; Uka, Tanaka, oshiyama, Kato, & Fujita, 2000), and lateral area MST (Eifuku & Wurtz, 1999). Disparity selective-neurons have also been documented in areas not conventionally considered to be predominantly visual, such as the frontal eye fields (Ferraina, Pare, & Wurtz, 2000) and the lateral (Genovesio & Ferraina, 2004; Gnadt & Mays, 1995) and caudal (Taira, Tsutsui, Jiang, ara, & Sakata, 2000; Tsutsui, Jiang, ara, Sakata, & Taira, 2001; Tsutsui, Sakata, Naganuma, & Tanaka, 2002) portions of the intraparietal sulcus. Combined with earlier studies showing the presence of disparity-tuned neurons in areas V1, V2, V3, V3A, MT, and dorsal MST (reviewed in Cumming & DeAngelis, 2001), these studies suggest that disparity signals are much more widely distributed in the primate brain than was suspected a decade ago. The proliferation of disparity signals in visual cortex raises fundamental questions regarding the roles of different cortical areas: How are the representations of disparity specialized in these different areas, and what are the specific roles that particular areas play in 3D vision? These are not simple questions to answer because they require both a detailed understanding of the responses of single neurons to a variety of stimulus configurations and causal tests in which neural activity is manipulated and the consequences on behavior are observed. Over the past several years, considerable progress has been made on these fronts, and there is cause to be optimistic that these experimental approaches will lead to a functional taxonomy that describes how different visual areas contribute to the various perceptual capacities of depth perception. My goal in this chapter is to summarize what we know about how disparity and motion parallax cues are processed in the middle temporal (MT) area of visual cortex and to review experiments that suggest a specialized functional role for area MT in depth perception. Along the way, I shall try to place the findings from area MT into the context of what is known from related studies performed in other parts of primate visual cortex. Binocular disparity processing in area MT Area MT is a relatively small ( 60 mm 2 ) visual area located along the posterior bank of the superior temporal sulcus in macaque monkeys. It receives much of its visual input from areas V1, V2, and V3, and it projects extensively to other occipitoparietal areas, including MST, FST, VIP, and LIP (reviewed in Born & Bradley, 2005). Area MT is well known for its role in processing visual motion, and an extensive body of literature implicates area MT both in the perception of motion and in guiding smooth eye movements that are driven by visual image motion (Born & Bradley, 2005). 484 sensation and perception Gazzaniga_33_Ch33.indd 484 3/5/2009 8:59:56 PM

3 Basic Aspects of Disparity Processing in MT Despite heavy emphasis on the role of MT in motion processing, it has been known for quite some time that MT is rich in neurons that are selective for binocular disparities (Maunsell & Van Essen, 1983b). A more recent quantitative study, using random-dot stereograms (figure 33.2A), reported that 93% of single neurons in MT have statistically significant selectivity for disparity (DeAngelis & Uka, 2003). The strength of disparity selectivity in MT is, on average, stronger than that seen in areas V1, (Prince, Pointon, Cumming, & Parker, 2002) and V4 (Tanabe et al., 2005). As shown in figure 33.2B, disparity tuning curves in MT take on a variety of shapes and have continuously varying preferences over a wide range of disparities (note that zero disparity represents a surface containing the point of ocular fixation). Most frequently, however, MT neurons tend to have tuning that is roughly odd-symmetric around zero disparity, with a welldefined peak response that is either near (e.g., cells 2 and 3 in figure 33.2B) or far (cells 5 and 6). In this regard, MT differs notably from area V1, where disparity tuning curves are much more frequently even-symmetric and tend to peak at disparities closer to zero (Cumming & DeAngelis, 2001). As one ascends the dorsal visual pathway in the macaque, disparity tuning curves progress from largely even-symmetric in V1 to having an odd-symmetric bias in MT to being strongly odd-symmetric in area MST (see Cumming & DeAngelis, 2001, for comparison; Roy, Komatsu, & Wurtz, 1992; Takemura, Inoue, kawano, Quaia, & Miles, 2001). Nearly all neurons in area MT show directionally selective responses to visual motion (DeAngelis & Uka, 2003; Maunsell & Van Essen, 1983a; Zeki, 1974), and nearly all neurons are selective for speed of motion as well (Maunsell & Van Essen, 1983a; Nover, Anderson, & DeAngelis, 2005; Rodman & Albright, 1987). Thus most MT neurons respond substantially more strongly to a moving stimulus than a stationary one. Importantly, however, the majority of MT neurons do produce sustained visual responses to stationary flashed stimuli, and the disparity tuning of these responses is nearly identical to those elicited by moving stimuli (Palanca & DeAngelis, 2003). Thus MT provides reliable disparity signals to the rest of the brain when we look around a stationary scene. Disparity-selective neurons in area MT are organized into a topographic map according to their disparity preferences (DeAngelis & Newsome, 1999), and this disparity map coexists with the well-known map for direction of motion in MT (Albright, Desimone, & Gross, 1984). In contrast to the clear columnar organization for disparity in MT, there is no clear Figure 33.2 Schematic illustration of random-dot stereogram stimulus and example disparity tuning curves measured with this stimulus. (A) A circular patch of moving dots having variable disparity was presented over the receptive field (circle, not present in the actual display) of an MT neuron. Solid and open dots within the receptive field denote the images seen by the left and right eyes; the separation between each pair of open and solid dots is the binocular disparity. The remainder of the screen was filled with stationary dots presented with zero disparity (gray dots). The monkey was required to maintain fixation on the fixation point during each trial. (B) Disparity tuning curves for seven representative MT neurons. Solid symbols and error bars show the mean response to each disparity ± standard error. The smooth curve through each data set is the best-fitting Gabor function. Neurons are presented (from top to bottom) in order of their preferred disparities, from large Near to large Far. The vertical scale bar corresponds to 100 spikes per second. (Adapted from DeAngelis & Uka, 2003.) deangelis: roles of visual area mt in depth perception 485 Gazzaniga_33_Ch33.indd 485 3/5/2009 8:59:56 PM

4 evidence for such a map in area V1 of monkeys (Chen, Lu, & Roe, 2008; Prince et al., 2002). However, there is clear evidence for a map of disparity in both areas V2 (Chen et al., 2008; Nienborg & Cumming, 2006) and V3 (Adams & Zeki, 2001), and it is possible that the map of disparity in MT is at least partially inherited from one or both of these sources. Absolute and Relative Disparity Selectivity Over the past several years, an important distinction to emerge in the cortical processing of binocular disparity involves the difference between absolute and relative disparity coding. Absolute disparity refers to the interocular difference in the angle subtended by a point in space relative to the projection of the fixation point, which lands on the fovea in each eye (figure 33.3A). Thus absolute disparities are defined relative to the point of ocular fixation. In contrast, the relative disparity between two points in space refers to the difference in their absolute disparities (figure 33.3B). This distinction becomes especially important when one considers changes in the fixation distance of a subject and hence the vergence angle of the eyes. When we converge or diverge our eyes to focus on a near or far object, respectively, all of the absolute disparities change. On the other hand, the relative disparity between two points in space is unaffected by changes in vergence angle. This means that uncontrolled errors and A Figure 33.3 Geometric definitions of absolute disparity (Φ, left) and relative disparity (, right). Each panel shows a top-down view in which the eyes are converged on a fixation point (open symbol) directly in front of the subject. (A) The absolute disparity of point P refers to the angle, Φ, subtended by this point relative to the point of fixation. Thus when the subject converges or diverges their eyes to focus at a different distance, the absolute disparity of point P will change. (A) The relative disparity of point P1 with respect to point P2 is given by the angle,, which is the difference between the absolute disparities of these two points. If the eyes converge at a different distance, the relative disparity between P1 and P2 will be unchanged. B variations (i.e., noise) in vergence state will affect absolute disparities much more than relative disparities. For this reason, it has been hypothesized that the visual system might need to contain a neural representation of relative disparities to allow precise discrimination of small differences in disparity (Neri, 2005; Neri, Bridge, & Heeger, 2004; Parker, 2007; Prince et al., 2000; Thomas, Cumming, & Parker, 2002). This hypothesis is supported by behavioral evidence. Both humans and monkeys are able to discriminate much smaller differences in the disparity of a target when that target is located close to a reference disparity (Prince et al., 2000; Westheimer, 1979), suggesting that neurons somewhere in the visual system locally compute relative disparities. To determine whether cortical neurons represent absolute or relative disparities, Thomas and colleagues (2002) devised a test in which two patches of random-dot stereogram are presented in a concentric (center-surround) arrangement and the disparity of both the center and surround are varied in a fully crossed design (figure 33.4A). For a neuron that responds solely to the relative disparity between center and surround, the disparity tuning curve in response to the center disparity should shift with the disparity of the surround (figure 33.4B). In contrast, no such shift would be observed for a neuron tuned to absolute disparity (figure 33.4C ). In their ground-breaking study, Thomas and colleagues (2002) found that neurons in area V1 signal absolute disparity, confirming a previous report (Cumming & Parker, 1999), whereas a subset of neurons in area V2 signal relative disparity. More recently, we have performed similar tests in area MT and have found that MT neurons generally signal absolute disparities, similar to neurons in V1 (Uka & DeAngelis, 2006). This is seen as a distribution of shift ratios (figure 33.4D) that cluster around zero, whereas neurons that are tuned for relative disparity would have a shift ratio near one. Fujita and colleagues have recently performed the same test in area V4 and have reported that the majority of V4 neurons have shift ratios greater than zero, with some neurons representing purely relative disparity while many others show an intermediate representation (shift ratios near 0.5) (Umeda, Tanabe, & Fujita, 2007). These comparative results across areas are summarized in figure 33.4E (modified from Umeda et al., 2007). The finding of relative disparity tuning in area V4 but not MT suggests that the ventral processing stream may emphasize relative disparities, whereas the dorsal stream emphasizes absolute disparities. These results are consistent with the results of a functional magnetic resonance imaging (fmri) study in humans that used an adaptation paradigm to test for sensitivity of different cortical areas to absolute and relative disparities (Neri et al., 2004). This study found that ventral stream areas adapt to both absolute and relative 486 sensation and perception Gazzaniga_33_Ch33.indd 486 3/5/2009 8:59:57 PM

5 Figure 33.4 Stimuli, predicted outcomes, and results of tests for absolute vs. relative disparity tuning. (A) Top-down view of the stimulus configuration, consisting of a center patch of dots and a surrounding annulus. All combinations of nine center disparities and three to five surround disparities were presented in randomly interleaved trials. (B ) If a neuron signals relative disparity, the disparity tuning in response to the center patch should shift horizontally by an amount equal to the change in surround disparity. (C ) If a neuron signals absolute disparity, no shifts should be seen, although some amplitude variations may occur. (D) Distribution of shift ratios for 201 pairings of surround disparities from 45 MT neurons. The shift ratio is computed as the horizontal shift of the tuning curve divided by the difference between the two surround disparities. Thus an idealized relative disparity neuron will have a shift ratio of 1, and an idealized absolute disparity neuron will have a shift ratio of 0. Note that shift ratios from area MT are distributed around zero, with a slight but significant bias toward positive values (sign test, p < ). Solid bars denote shift ratios that were significantly different from zero (sequential F-test, p < 0.05; 52/201 shifts). (Panels A D were adapted from Uka & DeAngelis, 2006.) (E ) Summary of results of the relative disparity test across studies of four different visual areas, adapted from Umeda and colleagues (2007). For each area, the open symbol indicates the median shift ratio, and the error bars represent the range from the 25th percentile to the 75th percentile. Data were compiled by Umeda and colleagues (2007) across four studies, as indicated. Note that relative disparity selectivity increases from V1 to V2 to V4 (presumably reflecting ascension of the ventral stream), whereas neurons in area MT show absolute disparity tuning. deangelis: roles of visual area mt in depth perception 487 Gazzaniga_33_Ch33.indd 487 3/5/2009 8:59:57 PM

6 disparities, whereas dorsal stream areas adapt only to absolute disparities. Together, these findings from monkeys and humans are consistent with the hypothesis that the ventral stream carries out sophisticated disparity computations to represent the 3D shape of objects, whereas the dorsal stream carries out somewhat simpler computations that are aimed at computing the location of objects in 3D space and also perhaps at representing the coarse layout of surfaces in the scene (see also Neri, 2005; Parker, 2007). Along these lines, we might expect area MT to contribute to coarse judgments of depth based on absolute disparities but not to fine judgments of depth based on relative disparities. We shall return to this prediction later. Coding of Three-Dimensional Surface Orientation Most physiological studies of disparity processing in visual cortex have examined responses to frontoparallel planar surfaces that vary in distance from the observer. In natural scenes, however, binocular disparity varies smoothly across surfaces that can have many possible 3D orientations relative to the observer. Spatial gradients of disparity thus provide powerful cues to 3D object shape and 3D surface orientation (Howard & Rogers, 1995, 2002). The simplest form of spatial variation involves monotonic gradients of disparity across space that specify the 3D orientation of planar surfaces. As illustrated in figure 33.5A, the 3D orientation of planar surfaces can be parameterized in terms of tilt and slant. Whereas human perception of tilt and slant defined by gradients of disparity has been studied considerably (Howard & Rogers, 2002; Sedgwick, 1986), only recently have physiologists examined how cortical neurons represent 3D orientation. In area MT, we have tested neurons with random-dot stereograms containing linear gradients of disparity, and we have found that more than half of MT neurons show significant tuning for the tilt of planar surfaces (Nguyenkim & DeAngelis, 2003). This property is shown for an example neuron in figure 33.5C. The neuron shows broad but robust tuning for the tilt of the surface, and this tuning is maintained across variations in the mean disparity of the stimulus. This insensitivity to variations in mean disparity around the peak of the cell s disparity-tuning curve (figure 33.5B) indicates that tilt selectivity cannot be simply explained by miscentering the gradient stimulus on the receptive field or by receptive field inhomogeneities (Nguyenkim & DeAngelis, 2003). Interestingly, this finding of invariant tilt tuning suggests that MT neurons possess some form of relative disparity selectivity when tested with disparity gradients, whereas they do not show this property when tested with concentric edges (figure 33.4D). Although the mechanisms underlying this difference remain unclear, this comparison highlights the important point that there is no single unique test for relative Figure 33.5 Schematic illustration of the 3D orientation of planar surfaces, parameterized by tilt and slant, as well as data from a tilt-selective MT neuron. (A) Tilt refers to the axis around which the plane is rotated away from frontoparallel, and slant defines the amount by which the plane is rotated. Zero slant corresponds to a frontoparallel surface for which the tilt is undefined. In this illustration, tilt and slant are defined by perspective and texture gradient cues. In the MT experiments, surface orientation was defined solely by the direction and magnitude of a linear gradient of horizontal disparity in a random-dot stereogram. (B) A conventional disparity-tuning curve for an MT neuron measured using random-dot stereograms (slant was zero, and different uniform horizontal disparities were applied). Mean responses ± standard error are shown for each stimulus disparity, along with a spline fit. Symbols at the top indicate the three mean disparities used for testing tilt selectivity with disparity gradients. (C ) Tilt-tuning curves for the same MT neuron are show at three different mean disparities (coded by symbol shape). Smooth curves indicate the best fits of the modified sinusoid function. Note that the neuron shows clear tuning for the tilt of a planar random-dot surface and that this tilt tuning is robust to changes in the mean disparity of the stimulus. (Adapted from Nguyenkim & DeAngelis, 2003.) disparity selectivity, and it might depend greatly on stimulus geometry. In MT, tilt tuning is seen only for stimuli that are presented at large slants (generally >45 degrees) (Nguyenkim & DeAngelis, 2003), suggesting that 3D orientation coding in MT involves a rather coarse mechanism that is more likely to be involved in providing the basic layout of surfaces in the scene rather than in analyzing the details of 3D shape (see also Parker, 2007), though this remains to be tested further. Selectivity for 3D surface orientation has also been observed in other cortical areas. In the caudal intraparietal area (CIP), Sakata and colleagues have described neurons that signal the tilt of surfaces defined by disparity gradients 488 sensation and perception Gazzaniga_33_Ch33.indd 488 3/5/2009 8:59:58 PM

7 (Taira et al., 2000; Tsutsui et al., 2001, 2002). Hinkle and Connor (2002) have also described neurons in area V4 that are selective for the 3D orientation of bar stimuli, although this selectivity could be driven more by orientation differences between the eyes because the same neurons did not generally show tuning for tilt and slant in disparity-defined random-dot surfaces. In the inferotemporal (IT) cortex, Janssen, Orban, and colleagues have conducted an impressive series of studies showing that IT neurons have selectivity for 3D shape defined by gradients of disparity as well as boundary cues (Janssen, Vogels, Liu, & Orban, 2001; Janssen et al., 1999, 2000a, 2000b). Thus it is clear that neurons at the upper levels of both the dorsal and ventral processing streams make use of spatial gradients of disparity to extract information about both 3D shape and object/surface orientation. However, much remains to be learned, and the respective roles of these areas in perception of 3D structure are still not well understood. In addition to disparity gradients, the 3D orientation of surfaces may be specified by gradients of texture, velocity, or luminance (shading) (Sedgwick, 1986). Neurons that are closely involved in perception of 3D orientation may thus be expected to signal tilt and slant based on multiple cues. In area CIP, Sakata s group has shown that neurons signal tilt by gradients in both disparity and texture (presented separately), and that their tilt preferences for the two cues are often well matched (Tsutsui et al., 2002). However, no published study has examined how neurons respond to multiple cues to 3D orientation presented simultaneously. In area MT, neurons have previously been shown to exhibit selectivity for tilt defined by velocity gradients (Treue & Andersen, 1996; Xiao, Marcar, Raiguel, & Orban, 1997). We have presented preliminary evidence that individual MT neurons are tuned for tilt defined by both disparity and velocity gradients (Nguyenkim & DeAngelis, 2004). Some neurons have matched tilt tuning for the two cues, whereas others do not. Responses to both cues together appear to be well predicted by weighted linear summation of the individual cue responses (unpublished). It is currently unclear whether MT plays a role in perceptual cue integration for 3D orientation perception or whether it may simply be an early stage at which disparity and velocity gradients begin to interact. A recent fmri study, performed by using behaving macaques, suggests that there is considerable additional processing of 3D surface orientation and shape in regions of the intraparietal sulcus that receive inputs from area MT (Durand et al., 2007). Linking neural representation to function: roles of area MT in coarse and fine depth discrimination As was discussed above, binocular disparity information is now known to be represented across a broad range of visual cortical areas in primates. There seem to be two main possibilities for why this might occur: (1) Disparity processing is highly distributed such that most aspects of depth perception depend on simultaneous activation of many regions of cortex, or (2) different cortical areas have specialized representations of binocular disparity that are well suited to some tasks but not others. In the latter scenario, depth perception in a specific context could depend on only a small subset of visual areas or perhaps only on a subset of neurons within a single visual area. Thus far, we have already discussed evidence that favors the notion of specialized representations, namely, that ventral stream areas appear to represent the precise relative disparity information that is thought to be needed for fine depth discrimination and 3D shape perception, whereas the dorsal stream appears to emphasize absolute disparities (Thomas et al., 2002; Uka & DeAngelis, 2006; Umeda et al., 2007). Moreover, it is likely that we know about only a small fraction of the differences between areas and between visual streams at this time. If different cortical areas are specialized to perform different tasks, then it should be possible to identify experimentally the areas and/or neurons that contribute to performance of a particular task. In recent years, my laboratory has attempted to clarify the functional roles of area MT in stereo vision by performing a series of experiments with monkeys that were trained to perform tasks that were chosen to reveal differences in function that may be linked to absolute versus relative disparity representations. Coarse and Fine Depth Discrimination Tasks To probe depth perception based on coarse absolute disparity information, we trained monkeys to perform the Coarse task illustrated in figure 33.6A. In this task, dots in a stereogram are divided into two groups with adjustable percentages: signal dots are all presented at the same disparity in each trial, which is near or far relative to the plane of fixation; noise dots are given random disparities in each trial such that they form a 3D cloud. The monkey s task is to judge whether the net depth of the stimulus is near or far and to make a saccadic eye movement to signal its choice (Uka & DeAngelis, 2003). Across trials, the relative proportion of signal and noise dots, indexed by a variable called binocular correlation, is varied to manipulate task difficulty. Figure 33.7A (open symbols) shows a psychometric function for one monkey in a typical session. Note that in this task, the monkeys always discriminated between two signal disparities (e.g., 0.4 and +0.4 ) that were on opposite sides of zero disparity and were well above stereoacuity thresholds. To assay the contribution of neurons to depth perception based on fine relative disparities, we also trained monkeys to perform the Fine task depicted in figure 33.6B. In this task, a bipartite center-surround random-dot stereogram is presented, and the monkey is required to report whether deangelis: roles of visual area mt in depth perception 489 Gazzaniga_33_Ch33.indd 489 3/5/2009 8:59:58 PM

8 Figure 33.6 Schematic illustration of two depth discrimination tasks used to study functional contributions of area MT. (A) The Coarse task. A random-dot stereogram was presented over the receptive field (RF), and dots moved at the neuron s preferred velocity (arrow). Solid and open dots represent left and right half-images, respectively. The background was solid with dynamic zero-disparity dots (gray). Saccade targets were located 5 above and below the fixation point, corresponding to far and near choices, respectively. The strength of the depth signal was adjusted by varying binocular correlation. At 50% binocular correlation (right), half of the dots within the receptive field were presented at either the neuron s preferred disparity (horizontal line inside gray oval) or the disparity that elicited a minimal response (null disparity). The remaining dots had random disparities. (B) The Fine task. A bipartite (center-surround) random-dot stereogram was presented. The center patch covered the RF and contained dots moving at the preferred velocity (arrow). The surrounding annulus contained stationary dots presented (in most cases) at a nonzero disparity. A small patch of zero-disparity dots (gray) surrounded the fixation point to help anchor vergence. The monkey reported whether the center patch was in front of or behind the surround patch, and task difficulty was manipulated by finely varying the center disparity around the surround disparity. (Adapted from Uka & DeAngelis, 2006.) the center patch of dots appears near or far relative to the surround (Uka & DeAngelis, 2006). Both center and surround are presented without noise (100% binocular correlation), and the relative disparity between center and surround is varied in fine steps to measure psychophysical threshold (e.g., figure 33.7B). Importantly, the absolute disparities of the center and surround could be both far or both near, such that monkeys are required to judge relative depth to achieve high performance on this task. Given that neurons in area MT have fairly broad disparity tuning and do not represent relative disparities in a centersurround configuration (figure 33.4D), we hypothesized that MT would play a significant role in the Coarse task but not the Fine task. This was assessed by using a variety of approaches, as described below (see also Parker & Newsome, 1998). Neuronal Versus Behavioral Sensitivity From the distributions of firing rates measured during performance of the tasks, we used ROC analysis to compute the ability of an ideal observer to discriminate depth on the basis of the responses of each single MT neuron (Uka & DeAngelis, 2003, 2006). Example neurometric functions for representative MT neurons are shown in figure 33.7A for the Coarse task and figure 33.7B for the Fine task (solid symbols). These neurometric functions describe how the performance of the ideal observer increases as the differences between near and far stimuli become more salient. From each such data set, we computed psychophysical and neuronal thresholds as the stimulus values at which performance reaches 82% correct. Thus each experiment yielded both a psychophysical and a neuronal threshold that could be compared quantitatively. Figure 33.7C shows the distribution of the ratio of neuronal-to-psychophysical thresholds for 104 MT neurons studied during the Coarse task (Uka & DeAngelis, 2003). While threshold ratios span a wide range, the average ratio (geometric mean = 0.98) was close to unity, indicating that the average MT neuron could discriminate coarse disparities in noise with sensitivity comparable to that of the animal. This result is very similar to that found by Newsome and colleagues for direction discrimination in MT (Britten, Newsome, Shadlen, Celebrini, & Movshon, 1992). Thus neuronal sensitivity suggests that area MT could account for coarse depth discrimination. Figure 33.7D shows the analogous distribution of threshold ratios for 98 neurons that were tested during the Fine task. In this case, the average threshold ratio (1.76) is closer to 2, indicating that MT neurons are not as sensitive as the animal is. However, the best neurons could be sufficiently sensitive to account for behavior. Choice Probabilities In a psychophysical task performed around threshold, the same (weak) stimulus gives rise to different perceptual reports, as well as different neural responses, across repeated trials. By testing for a correlation between the trial-to-trial fluctuations in perceptual reports and neural responses (choice probability, or CP), it may be possible to identify neurons that are functionally coupled to perceptual decisions (Britten et al., 1996; Krug, 2004). An advantage of this approach is that it affords single-cell 490 sensation and perception Gazzaniga_33_Ch33.indd 490 3/5/2009 8:59:58 PM

9 Figure 33.7 Summary of single-unit and microstimulation experiments performed in area MT using both the Coarse (left column) and Fine (right column) depth tasks. (A) Example data from a typical experiment using the Coarse task. The psychometric function (open symbols) shows the monkey s percentage of correct responses as a function of binocular correlation. The neurometric function (solid symbols) shows the predicted performance of an ideal observer based on the responses of a single MT neuron. In this example, the neuron has sensitivity nearly identical to that of the animal. (B) Example psychometric and neurometric functions for a typical experiment using the Fine task. In this case, performance is plotted as a function of the (unsigned) relative disparity between center and surround stimuli. In this experiment, the neuron is about half as sensitive as the monkey. (C ) Distribution of the ratio of neuronal/psychophysical threshold ratios across 104 recording sessions involving the Coarse task. The geometric mean ratio was 0.98 (arrowhead). (D) Distribution of neuronal/ psychophysical threshold ratios for the Fine task (N = 98). The geometric mean was (E ) Distribution of choice probabilities for the Coarse task (N = 104). CP values significantly different from 0.5 are indicated by solid bars. Note that most CP values are greater than 0.5. (F ) Distribution of choice probabilities for the Fine task (N = 98). (G ) Distribution of microstimulation effects for the Coarse task (N = 78). Solid bars denote individually significant effects. Positive values indicate biases toward the preferred disparity of the stimulated neurons, as measured in units of percent binocular correlation for the Coarse task. Most experiments produced a significant preferred bias. (H) Distribution of microstimulation effects for the Fine task (N = 46). Shifts are now measured in degrees of relative disparity. Most experiments produced no effect, and the median shift was not significantly different from zero. (Panels A and C were adapted from Uka & DeAngelis, 2003; panel E was adapted from Uka & DeAngelis, 2004; panel G was adapted from DeAngelis et al., 1998, including additional data; panels D and H were adapted from Uka & DeAngelis, Data in panel F are previously unpublished.) deangelis: roles of visual area mt in depth perception 491 Gazzaniga_33_Ch33.indd 491 3/5/2009 8:59:59 PM

10 resolution and allows one to relate the tuning properties of neurons to behavior. Although CPs simply reflect a correlation between neurons and perceptual decisions, there is evidence to suggest that significant CPs reflect a functional contribution of neurons. For example, CPs in area MT and V2 have been shown to vary according to the tuning properties of neurons in a manner that appears to reflect the animal s (suboptimal) strategy for solving the task (Nienborg & Cumming, 2007; Uka & DeAngelis, 2004). We have used CPs as another means to evaluate the role of area MT in the Coarse and Fine depth tasks. As is shown in figure 33.7E, most MT neurons have CPs greater than 0.5 in the Coarse task, which indicates that the neurons tend to fire more strongly when the monkey reports that an ambiguous (e.g., 0% binocular correlation) stimulus matches the preferred depth of the neuron (Uka & DeAngelis, 2004). The average CP is 0.59, which is significantly greater than 0.5 (p << 0.001). This means that an ideal observer could predict the choices of a monkey with 59% correct accuracy by monitoring the activity of an average MT neuron. This finding is consistent with the notion that area MT makes an important contribution to the Coarse depth task. Recently, Nienborg and Cumming (2006, 2007) have examined responses of neurons in area V1 and V2 during performance of a task that is nearly identical to our Coarse task. Interestingly, neurons in V2 show CPs comparable to those we have seen in MT, whereas neurons in V1 do not. Thus trial-to-trial variability in the representation of disparities in V1 does not seem to be linked to depth percepts, whereas similar variability in V2 and MT does correlate with percepts. It is currently unclear whether neurons in V2 with significant CPs in the Coarse task reside in the portions of V2 (the thick stripes) that project heavily to area MT. However, a recent study shows that inactivating areas V2 and V3 by cooling substantially reduces disparity selectivity in MT (Ponce, Lomber, & Born, 2008), consistent with the idea that MT may inherit at least some of its disparity selectivity from V2. It is also interesting to note that CPs for the Coarse task have thus far been seen in areas (V2, MT) that contain a topographic representation of disparity (Chen et al., 2008; DeAngelis & Newsome, 1999) but not in area V1, which lacks such a map for disparity (Chen et al., 2008; Prince et al., 2002). It will be fascinating to see whether a correlation between functional architecture and CPs emerges as similar data are collected from additional tasks in additional areas. Figure 33.7F shows analogous CP data from area MT during performance of the Fine task (unpublished data). In this case, the mean CP is 0.52, which is not significantly different from 0.5 (p > 0.05). This may be taken as evidence that area MT does not contribute to performance of the Fine task. Note, however, that many more neurons than expected by chance have CPs significantly different from 0.5 (solid bars). These neurons are significantly correlated with perceptual decisions about fine relative disparities, but there is no consistent relationship between firing rates and choices across the population. Thus, on average, one cannot reliably predict a monkey s choices by measuring the response of MT neurons in the Fine task. However, this pattern of results remains somewhat puzzling, as it is not clear why MT neurons should show significant CPs that are both lower and higher than 0.5. If one speculates that CPs arise through a top-down signal from decision circuitry to MT, then it might be that these top-down signals cannot correctly target MT neurons because MT does not contain a topographically organized representation of relative disparities. Causal Manipulations Choice probabilities establish a correlation between neural responses and perceptual decisions (independent of the physical stimulus) but do not establish a causal contribution of those neurons to perception. To further link a particular visual area with specific functions in 3D vision, we need to directly manipulate neural activity during performance of relevant tasks. One approach involves electrical microstimulation (Cohen & Newsome, 2004; Salzman, Murasugi, Britten, & Newsome, 1992; Tehovnik, 1996), in which weak biphasic current is passed through a recording electrode to activate a cluster of neurons near the tip of the electrode whose tuning properties are known. By placing an electrode into the midst of one of the disparity columns in area MT (DeAngelis & Newsome, 1999), we have used microstimulation to probe the causal contribution of area MT to the Coarse and Fine tasks. In the Coarse task, microstimulation systematically biases monkeys judgments of depth, as summarized in figure 33.7G (see also DeAngelis, Cummings, & Newsome, 1998; Uka & DeAngelis, 2006). A positive effect of microstimulation means, for example, that electrical stimulation of a cluster of near-tuned neurons causes the monkey to report stimuli as near significantly more often than occurs when microstimulation is withheld. Note that microstimulation frequently produced statistically significant effects in the Coarse task (solid bars in figure 33.7G ) and that the vast majority of these effects were positive. Only one experiment produced a microstimulation effect in the wrong direction, such that stimulation of a cluster of far-preferring neurons produced a bias in favor of perceiving near. This finding, coupled with the sensitivity and CP analyses described above, establishes that area MT contributes to coarse depth perception based on absolute disparities. Figure 33.7H shows comparable microstimulation results for the Fine task. In this case, the median effect of microstimulation is not significantly different from zero (p = 0.88). Moreover, when microstimulation did produce a significant 492 sensation and perception Gazzaniga_33_Ch33.indd 492 3/5/2009 8:59:59 PM

11 effect (solid bars) it was frequently in the wrong direction (Uka & DeAngelis, 2006). Thus we found no clear evidence that area MT contributes to performance of the Fine task, consistent with the hypothesis that MT s role in this task is limited because it does not carry a representation of fine relative disparities. Together, these findings establish a satisfying connection between the neural representation of disparities in area MT (absolute, not relative) and the functional contributions of area MT to depth perception. These findings are consistent with the idea that different visual areas contain specialized representations of binocular disparity signals, and specifically support the notion that the dorsal stream (including MT) mainly processes absolute disparity information to localize objects in 3D space, whereas the ventral stream emphasizes computations of relative disparity for the purpose of 3D shape perception (Neri, 2005; Parker, 2007). Our findings spur hope that similar studies, employing a variety of tasks in a variety of areas, will be capable or revealing a functional taxonomy of visual cortical areas with respect to their roles in 3D vision. This remains to be seen, but there is reason to be optimistic that we can understand the selective contributions of individual areas to this overall process. A weakness of the comparisons that we have made between the Coarse and Fine tasks is that several aspects of the stimuli differ between the two tasks, including absolute versus relative disparities, the range of disparities, inclusion of noise, and presence of segmentation boundaries. Going forward, it will be important to design stimuli that allow us to test perception of depth versus 3D shape while eliminating or minimizing differences in other stimulus variables. Selectivity for depth from motion parallax in area MT As is illustrated in figure 33.1B, self-movement (e.g., moving one s head from side to side) generally causes the image of an object to move on the retina, and both the direction and speed of image motion depend on the location of the object in depth. This depth-dependent image motion is called motion parallax. Motion parallax can also arise because of the movement of objects that have depth structure, but here I shall focus on motion parallax resulting from observer movement. In a ground-breaking series of psychophysical studies, Rogers and Graham placed subjects in an apparatus with a sliding chin rest and asked them to move their heads back and forth while fixating a point on a video display with one eye (Graham & Rogers, 1982; Rogers & Graham, 1979, 1982; Rogers, 1993; Rogers & Rogers, 1992). As the subjects moved their heads (and correspondingly their eyes), the experimenters updated the positions of dots in the display such that their motion was consistent with the presence of a corrugated surface in depth. Despite the lack of disparity or any pictorial depth cues, this arrangement produces a compelling sensation of depth. Studies have shown that depth perception from motion parallax is almost as precise as from disparity (Rogers & Graham, 1982). Moreover, psychophysical studies suggest that disparity and motion parallax processing may share a common neural substrate. Depth percepts from disparity and motion parallax can cross-adapt each other, and combining the cues together can yield substantial improvements in sensitivity over either cue alone (Bradshaw & Rogers, 1996). Whereas perception of depth from motion parallax has been well studied (Nawrot & Joyce, 2006), the neural basis for this behavior has remained unknown. Surely, a neural substrate for depth from motion parallax requires neurons that are selective for the direction and speed of visual motion, and such neurons can be found in many visual areas in primates beginning with V1. It has also been suggested that neurons with relative motion selectivity (Cao & Schiller, 2003; Li, Lei, & ao, 1999) might provide important inputs to such a depth mechanism. However, the presence of visual motion selectivity alone, even relative motion selectivity, does not establish that neurons can provide depth information based on motion parallax. How, then, can we identify neurons that participate in computing depth from motion parallax? Our approach has been to exploit the fact that visual image motion itself can be depth-sign ambiguous, as illustrated in figures 33.8A and 33.8B. In the absence of pictorial depth cues such as occlusion and size, the retinal image motion generated by near and far objects (having equivalent disparities) is identical except for the phase of the motion relative to movement of the subject s head (or eyes). Objects that are nearer than the point of fixation will move in the direction opposite to head motion, whereas far objects will move in the same direction as the head. Thus in the absence of pictorial cues, neurons must combine retinal image motion with extraretinal signals related to head and/or eye movement to determine the sign of depth from motion parallax. We designed random-dot stimuli that were depth-sign ambiguous by removing all pictorial cues to depth (stimulus size, dot size and density, occlusion, etc.) (Nadler, Angelaki, & DeAngelis, 2008). If neurons simply respond to visual image motion, then they cannot differentiate between our near and far stimuli. On the other hand, if neurons receive extraretinal inputs that specify the phase of visual image motion relative to head or eye motion, then they may become selective for depth sign. Figures 33.8C and 33.8D illustrates responses from a neuron recorded from area MT under monocular viewing conditions (ipsilateral eye occluded). In the Retinal Motion condition (figure 33.8C), the visual stimulus simulated a deangelis: roles of visual area mt in depth perception 493 Gazzaniga_33_Ch33.indd 493 3/5/2009 8:59:59 PM