U.S. copyright law (title 17 of U.S. code) governs the reproduction and redistribution of copyrighted material.

Transcription

1 U.S. copyright law (title 17 of U.S. code) governs the reproduction and redistribution of copyrighted material.

2 The Handbook of Cognitive Neuropsychology What Deficits Reveal About the Human Mind Edited by Johns Hopkins University Copyright 2001 Taylor & Francis. O

3 Spatial Representation in Mind and Brain Michael McCloskey The ability to represent and manipulate spatial information is crucial for a wide range of perceptual, cognitive, and motor functions, including object recognition, reading, writing, attention, visually-guided reaching, and navigating through the environment. As a consequence, spatial processing has' attracted considerable interest among cognitive psychologists, and is one of the most active areas of neurophysiological research. However, many of the most fascinating results have come from cognitive neuropsychological research on spatial deficits. My aim in this chapter is not to review the extensive literature on spatial deficits, but rather to explore some fundamental issues about spatial representation through discussion of selected studies. first consider a deceptively simple question: n what sense(s) are spatial representations spatial? Exploring this question proves surprisingly helpful in clarifying basic issues. then examine a concept of central importance in research on spatial representation: that of a reference frame. Here again attempt to clarify the issues and the implications of results. WHAT S SPATAL ABOUT SPATAL REPRESENTATONS? A pervasive assumption in research on spatial perception and cognition is that (at least some) spatial representations in the brain are spatial not merely in the sense of representing spatial information, but also in some other, more fundamental, sense. For example, the psychological, neuropsychological, and neuroscientific literatures are replete with terms like depictive representation, quasi-pictorial representation, internal representational space, cortical map, cognitive map, visuo-spatial scratch pad, mental image, image scanning, and mental rotation. Such terms seem to suggest that internal representations of spatial information are somehow similar to external spatial representations like pictures or maps. Except in the literature on mental imagery (see Kosslyn, 1995, for an overview), this notion has not received much scrutiny, and it remains unclear in what sense(s) internal representations of spatial information are spatial. n the following discussion use neuropsychological evidence from studies of unilateral spatial neglect as a springboard for exploring this issue. first describe several neglect phenomena that have been taken as evidence for internal representations with interesting spatial 101

4 102 Objects and Space properties. Using these phenomena as examples, consider what spatial properties a representation might have, and what the significance of these properties might be. l Examples Figures Viewed through Slits Bisiach, Luzzatti, and Perani (1979) tested 19 participants with right-hemisphere lesions and symptoms of left neglect. Pairs of cloudlike figures were presented (see Figure 5.1 panel A), and the participant judged whether the figures in each pair were the same or different. When the figures differed, the difference was sometimes on the left side (as in Figure 5.1 panel A), and sometimes on the right. n the static condition each figure in a pair was displayed in its entirety for two seconds. n the dynamic condition each figure was presented as if it were moving behind a central vertical slit, with only a narrow slice visible at any given time (Figure 5.1 panel B). The figure made a single pass behind the slit, moving from left to right or right to left over a two-second period. Left neglect was evident in the results from the static condition: The participants as a group showed normal performance in detecting differences on the right sides of figures, but were impaired in detecting left-side differences. Remarkably, the same pattern was observed in the dynamic condition. Even though all parts of a figure were presented in the same central vertical slit, the participants as a group were impaired in detecting differences on the left but not the right sides. (See Ogden, 1985, for a replication in which results from individual participants as well as group averages were reported.) These findings suggest that even when figures were presented bit-by-bit in the central slit, participants constructed internal representations of whole figures. Furthermore, given that the participants neglect affected spatially-defined parts of the figures (i.e., the left sides), it appears that the whole-figure representations were spatial not merely in the sense of representing spatial information about the shapes of the figures, but in some other way as well. n the terms of a metaphor frequently invoked in discussions of neglect (e.g., Bisiach & Luzzatti, 1978), we might say that the representations were spread across the left and right sides of an internal representational space, and that the participants neglected the left side of this internal space. NV and the Piazza del Duomo An elegant demonstration by Bisiach and Luzzatti (1978; see also Bisiach, Capitani, Luzzatti, & Perani, 1981) also points to representations that are spatial in some interesting sense. NV, a 72-year-old lawyer who had suffered a right temporo-parietal hemorrhage, was asked to describe a familiar place-the Piazza del Duomo in Milan-from two imagined perspectives. When told to imagine himself at one end of the square looking toward the cathedral at the other end, he described many landmarks that were on the right side of the square from the imagined viewpoint, but very few landmarks from the left side of the square (Figure 5.2). He was then told to imagine himself at the opposite end of the square, looking toward his initial vantage point (so that the side previously on his left was now on his right, and vice versa). From this perspective NV described many landmarks from the previously-neglected side of the square, and failed to mention any landmarks from the side he had originally described in detail. 1 unilateral spatial neglect is a label applied to individuals with brain damage who show impairment in processing information about the contralesional side of space. For example, a neglect patient with right-hemisphere damage may omit material from the left side when asked to copy a picture or draw an object from memory. Neglect has been observed following lesions to a variety of brain regions, but occurs most commonly as a consequence of parietal damage (e.g., Vallar, 1993).

5 Spatial Representation in Mind and Brain 103 Figure 5.1: Panel A. Example of stimuli from the Bisiach et al. (1979) study. Adapted with permission of Oxford University Press. Panel B. Presentation of stimuli in the dynamic condition. The vertical slit was 1.5 cm wide, whereas the stimulus shapes had a width of 20 cm. After Bisiach (1996), Figure 1, p. 63. Adapted with permission of Blackwell Publishers. Here again the metaphor of an internal representational space seems apt. NV, it appears, represented in some internal space the square as it would look from a specified vantage point, and was impaired in representing or processing information from the left side of the representational space. The temptation is strong to conceive of the internal space as a screen upon which spatial information is displayed, with neglect involving damage to, or impairment in attending to, one side of the screen. NG and Single-Word Reading Caramazza and Hillis (1990a, 1990b; see also Hillis & Caramazza, 1995) reported an extensive study of NG, a 79-year-old woman who had suffered a stroke affecting the left temporo-parietal region. NG showed signs of right-sided neglect in a variety of circumstances (e.g., failing to eat food on the right side of her plate, omitting the right sides when drawing objects from memory). n single-word reading NG evidenced right neglect dyslexia; that is, she made errors that almost always involved the right side of the word (e.g., humid + human, journal + journey). These errors suggested that NG's right-sided neglect affected her ability to represent, attend to, or otherwise process, the right sides of the word stimuli. 2 2 NGs errors did not usually involve simply omitting material from the right side of the word; for example, she did not read humid as hum. Her errors are perhaps best described by saying that when she misread a word, her response was almost always consistent with the left half of the stimulus word, but inconsistent with part or all of the right half. Apparently, NGs word recognition processes usually had information about the presence of letters on the right side of the word, but often did not have information about the identities of these letters (see also Ellis, Flude, & Young, 1987).

6 104 Objects and Space Figure 5.2: NV s performance in describing the Piazza del Duomo in Milan, from two imagined perspectives. NV imagined himself first at location A and then at location B, facing in each case in the direction indicated by the arrow. Landmarks described from perspective A are indicated by a, and those described from perspective B by b. After Bisiach and Luzzatti ( 1978), Figure 1, p Adapted with permission. However, additional results made clear that NG s neglect errors in reading did not result from neglect for the right sides of stimuli. When words were presented with the letters arrayed vertically instead of horizontally (see Figure 5.3), NG s error rate remained the same, and her errors still involved the ends of words (e.g., fraud + frame), even though all of the letters in the vertical stimuli had the same central position on the left-right dimension.

7 Words in Normal Orientation hound house sprinter sprinkle stripe strip Spatial Representation in Mind and Brain 105 Vertically-Oriented Words right r i. g i d comet greenery - r10mmo3 rlzir1331@ F 1 Mirror-Reversed Words - b9f~iu~31 regular Figure 5.3: Exampl es of \ NG s reading errors. Even more remarkable were NG s errors in reading mirror-reversed words (e.g., nommo3). n mirror-reversed stimuli the beginning of the word appears on the right, and the end of the word on the left; therefore, neglect affecting the right sides of stimuli should lead to errors on the beginnings of mirror-reversed words (e.g., nomm03 -+ lemon). n fact, however, NG s errors for mirror-reversed stimuli-like her errors for normal and vertically-oriented stimuli-consistently involved the ends of the words (e.g., nomm03 -+ comet; see Figure 5.3). Caramazza and Hillis (1990a, 1990b) interpreted these results by proposing that regardless of how a word is presented-normally, vertically, mirror-reversed-word recognition processes construct an internal spatial representation in which the letters are arrayed from left to right in their canonical positions, with initial letters on the left and final letters on the right. They argued that NG was impaired in processing the right side of these canonical representations, leading her to err on the ends of words regardless of how the stimuli were oriented. Stimuli, Represented Scenes, and Representations The results reported by Bisiach et al. (1979), Bisiach and Luzzatti (1978), and Caramazza and Hillis (1990a, 1990b) offer compelling glimpses of internal spatial representations, and the metaphor of an internal representational space seems to provide a useful way of thinking about these representations. How, though, should we interpret the notion of a representational space? What does it mean to suggest that participants in the Bisiach et al. (1979) study

8 106 Objects and Space represented the shapes of whole figures in an internal space, and neglected the left side of this space? Also, how should we interpret assertions about the left or right sides of representations, such as the claim that NG neglected the right side of internal word representations? Aspects of Representation n addressing these questions it is useful to distinguish among (a) the stimulus, or other source of information, from which an internal representation is constructed; (b) the state of affairs specified (i.e., represented) by the representation; and (c) the representation itself. For example, in the dynamic condition of the Bisiach et al. (1979) study each stimulus was a continuous succession of slices presented over time in a central vertical strip. However, the internal representation constructed from the stimulus apparently specified the figure as it would have appeared if presented as a whole. n other words the representation apparently described a state of affairs-which will call the represented scene-in which the figure was extended in space in front of the viewer, with the left part to the viewer s left, and the right part to the viewer s right. Finally, the representation itself was a set of brain states or events that stood for the represented scene. n the Bisiach and Luzzatti (1978) study NV generated internal representations of the Piazza del Duomo not from stimuli presented during the study, but rather from knowledge acquired through prior experience. nstructed to imagine the Piazza from two different perspectives, he apparently constructed two different representations. For each representation the represented scene was a view of the Piazza as it would appear from the specified vantage point. Because the represented scenes were different in the two cases, the brain states or events constituting the representations themselves must also have been different. n the Caramazza and Hillis (1990a, 1990b) study NG constructed internal representations from visual word stimuli in normal, vertical, or mirror-reversed orientation. From all three types of stimuli NG apparently generated representations specifying states of affairs (represented scenes) in which letters were arrayed in their canonical positions, with initial letters on the left and final letters on the right. Thus, for any given word (e.g., house) the represented sceneand probably the representation itself-was the same regardless of how the stimulus was oriented. The distinctions among stimuli, represented scenes, and representations can be summarized succinctly: Representations are constructed on the basis of stimuli (or other sources of information), and stand for represented scenes. Two points of clarification may be in order. First, the represented scene can be thought of as the informational content of the representation. Distinguishing represented scenes from stimuli is necessary because the informational content of a representation may not correspond closely to the eliciting stimulus. This is most obvious in cases of misrepresentation (e.g., misrepresenting the lengths of lines when viewing the Müller-Lyer illusion), but is also true when processes constructing a representation substantially elaborate or abstract away from stimulus properties, as in the Bisiach et al. (1979) and Caramazza and Hillis (1990a, b) studies. Second, in characterizing the representations themselves as brain states or events do not intend to imply that representations must be described in neural terms; other vocabularies (e.g., that of symbols, or subsymbolic units) are more appropriate for many purposes. The distinction between represented scenes and representations is therefore not a distinction between cognitive and neural levels of description, but rather a distinction between what is represented (i.e., the informational content), and what is doing the representing. Maintaining the Distinctions There is nothing profound about these distinctions; nevertheless, they can be easy to lose sight of. As we have seen, for example, the findings from some studies of neglect cannot be described by

9 Spatial Representation in Mind and Brain 107 saying that participants neglected the left or right side of the stimuli, and researchers attempting to characterize these findings have typically resorted to saying that neglect affected one side of an internal representation, or one side of an internal representational space. However, interpretations couched in these terms blur the distinction between representations and what they represent. To say that NG neglected the right half of a canonical word representation (e.g., Caramazza & Hillis, 1990a, p. 425) is to conflate the representation (a set of states or events in the brain) with the state of affairs it represents (a sequence of letters in their canonical positions). The claim Caramazza and Hillis intended to make is best stated not by saying that NG neglected the right half of the representations, but rather by saying that she neglected the right half of the represented scenes. Similar points apply to interpretations stated in terms of the internal representational space metaphor. Unless supplemented with more explicit statements, such interpretations are ambiguous: s the intent to say something about representations, represented scenes, or perhaps both? For the Bisiach et al. (1979) and Bisiach and Luzzatti (1978) results a more straightforward characterization is that the participants neglected the left sides of represented scenes (views of whole figures in the former study, and views of the Piazza del Duomo from particular perspectives in the latter). By describing effects of neglect in terms of represented scenes, we accommodate the fact that a neglected region may not be definable in stimulus terms (because the represented scene need not correspond in any simple way to a stimulus); at the same time we avoid the confusion between representations and what they represent (because the represented scene is distinguished from the representation itself). Two Senses of Spatial Representation With this discussion as a foundation, we can begin to sort out the senses in which a spatial representation could be spatial. n this section consider two senses, one involving represented scenes, and the other concerning the representations themselves. Spatial, Representation The most obvious sense in which a representation can be spatial is by representing spatial information. Representations that are spatial in this sense will refer to as spatial, representations. More specifically, will say that a representation is spatial, if it is used within a computational system to stand for spatial information. The representations affected by neglect in the studies have described were apparently spatial, representations. For example, participants in the Bisiach et al. (1979) study presumably had to represent spatial information-the shapes of the cloudlike figures-in making samedifferent judgments for pairs of figures. Claims about spatial, representation have to do with what information is represented, and therefore can be thought of as claims about the nature of the represented scenes. Some of the interesting conclusions suggested by the neglect studies are of this sort (e.g., Bisiach et al. s conclusion that even in the dynamic condition participants represented the figures as they would have appeared if presented as wholes). Spatial, Remesentation A second sense of spatial representation concerns the representations themselves. To introduce this sense need to examine some additional implications of the illustrative neglect studies. n all of the studies, damage to spatially-defined regions of the brain affected spatially-defined regions of represented scenes. For example, in the Bisiach et al. (1979) study the participants right-hemisphere lesions led to neglect for the left sides of the represented figures.

10 108 Objects and Space This pattern could be taken as evidence about the spatial layout of representations in the brain (see, e.g., Bisiach, 1993, 1996; Bisiach & Berti, 1987; Bisiach et al., 1979; Bisiach et al., 1981; Bisiach & Luzzatti, 1978). Assume first that representations of spatial information may extend over portions of both cerebral hemispheres, such that each representation is partially in the left hemisphere and partially in the right. A unilateral brain lesion could then affect a spatially-defined part of a representation (i.e., the left-hemisphere part, or the right-hemisphere part). For instance, in the Bisiach et al. (1979) study the participants right-hemisphere lesions may have affected right-hemisphere parts of representations while leaving left-hemisphere parts intact. n this way we can explain how damage to a spatially-defined brain region could affect a spatially-defined part of a representation. However, it remains to be explained how damage,, affecting a spatially-defined part of a representation (e.g., the right-hemisphere part) could lead to neglect for a spatially-defined part of a represented scene (e.g., the left side). For this purpose we need the additional assumption that spatially-defined parts of the representations were systematically related to spatially-defined parts of the represented scenes. n particular, we need to assume that the right-hemisphere parts of representations specified the left sides of the represented scenes, and vice versa. Damage affecting a representation-part in one hemisphere would then lead to impairment affecting the contralateral side of the represented scene. Thus, in the Bisiach et al. (1979) study the participants right-hemisphere lesions may have affected the left sides of the represented shapes, leading to poor performance in detecting leftside differences between figures. Analogous interpretations can be offered for the Bisiach and Luzzatti (1978) and Caramazza and Hillis (1990a, 1990b) results. 3 Given these assumptions, brain damage could affect representations in two ways (which are not mutually exclusive). First, neural tissue required for instantiating representations could be damaged, so that the parts of representations that should reside in the damaged hemisphere could not be created or maintained normally. For example, NG s left-hemisphere lesion might have damaged neural tissue needed for instantiating the parts of canonical word representations that specified the ends of words. Second, if we assume that each hemisphere contains mechanisms for attending to or otherwise operating upon the parts of representations residing within that hemisphere, then damage to the mechanisms in one hemisphere could selectively impair processing of the representation-parts in that hemisphere. For instance, in the Bisiach and Luzzatti (1978) study NV's right-hemisphere lesion may have impaired his ability to direct attention to parts of representations in that hemisphere, even if the representation-parts themselves were intact.* Could the neglect phenomena be accounted for solely in terms of separate left- and righthemisphere processing mechanisms, without assuming any partitioning of the representations? To develop such an interpretation, one would have to assume that even though representational elements specifying the left side of the represented scene were not separated from elements specifying the right side, right-hemisphere processing mechanisms nevertheless dealt only with the former elements, and left-hemisphere mechanisms only with the latter. This assumption lacks motivation and seems (at least to me) rather implausible. Hence, will continue to assume that the representations underlying the illustrative neglect phenomena were partitioned into left- and right-hemisphere parts, with the left-hemisphere part specifying the right side of the represented scene, and vice versa. However, the possibility that only the 3t may not be necessary to assume that each half of the represented scene is represented solely in the contralateral hemisphere; it may suffice to assume that for all parts of the represented scene the representation is at least predominantly contralateral. For example, Pouget and Sejnowski (1997a, 1997b) proposed a computational model of spatial representation in parietal cortex, and argued that the model can account for neglect phenomena. Their model assumes, on the basis of neurophysiological evidence, that each parietal lobe represents all parts of visual space, but that more cells are dedicated to the contralateral side of the space than to the ipsilateral side. 4The representational assumptions have described are therefore compatible with both representational and attentional theories of neglect. (See, e.g., Humphreys & Riddoch, 1993, for an overview of these theories.)

11 Spatial Representation in Mind and Brain 109 processing mechanisms were partitioned cannot be ruled out definitively; therefore my representational assumptions should be taken not as established facts, but rather as working hypotheses. We are now in a position to consider the second sense in which a representation could be spatial. This sense, spatial,, may be defined as follows: A representation is spatial, if (a) spatially-defined parts of the representation correspond to spatially-defined parts of the represented scene, and (b) at least one spatial property defined over the parts of the representation (e.g., distance between parts) is isomorphic to a spatial property defined over the corresponding parts of the represented scene. The retinotopic map in primary visual cortex ml) is a clear example (Figure 5.4). Each location in V1 corresponds to a location in the (two-dimensional) Fovea Visual field Calcarine fissure Primary visual cortex Calcarine fissure Figure 5.4: The retinotopic map in primary visual cortex (V1 ). After Mason and Kandel (1991), Figure 29-7, p Adapted with permission of The McGraw- Hall Companies.

12 1 10 Objects and Space visual field. Furthermore, several spatial properties defined over locations in V1 are isomorphic to spatial properties defined over locations in the visual field. For instance, distance in V1 mirrors distance in the visual field: locations that are close to one another in V1 correspond to locations that are close to one another in the visual field (although this relationship is modulated by so-called cortical magnification of central visual-field regions relative to peripheral regions). Also, spatial dimensions in V1 are isomorphic to spatial dimensions in the visual field; for example, the ventral-dorsal dimension in V1 is isomorphic to the up-down dimension in the visual field (the more ventral a location is in V1, the higher the corresponding location is in the visual field). n effect, the visual field is spread out topographically (or, to be more precise, topologically) on the cortical surface. Do the representations revealed by the neglect studies meet the criteria for spatial, representations? The answer appears to be a qualified yes. The metaphor of an internal representational space-especially if we think of this space as something like a display screen-encourages us to think of the representations as similar to those in V1, with the represented scenes laid out point-by-point across left- and right-hemisphere parietal regions. However, the neglect phenomena have described do not necessarily imply such a fine-grained correspondence between spatially-defined parts of the representations and spatially-defined parts of the represented scenes, or such a rich isomorphism between spatial properties of the representations and spatial properties of the represented scenes. To account for results showing neglect of the left or right side of a represented scene one need only assume a correspondence between two spatially-defined parts of the representation (a left-hemisphere part and a right-hemisphere part) and two spatially-defined parts of the represented scene (the right and left sides, respectively). No finer-grained correspondence is required; within each hemisphere there need be no correspondence between spatially-defined parts of the representation and spatially-defined parts of the represented scene. A representation of this sort meets the criteria for spatial, representation, in that (a) two spatially-defined parts of the representation (the left and right-hemisphere parts) correspond to two spatially defined parts of the represented scene (the right and left sides, respectively), and (b) a spatial property of the representation (the left-right relation between the left- and right-hemisphere parts) corresponds to a spatial property of the represented scene (the leftright relation between the right and left sides of the scene). However, such a representation is spatial, only in the weakest sense. Of course, the neglect phenomena have discussed do not rule out the possibility of a richer spatial, organization, and various other forms of evidence might be brought to bear. For example, some neglect studies have found a gradient in the severity of neglect within the affected half of the represented scene (e.g., Arguin & Bub, 1993; Baxter & Warrington, 1983; Caramazza & Hillis, 1990a; Ellis, Flude, & Young, 1987; Hillis & Caramazza, 1991). Thus, a person exhibiting left neglect may be more likely to neglect an item the farther left of center it is in the represented scene. Such results may seem to imply that the affected half of the represented scene was laid out systematically over some cortical region within the damaged hemisphere, allowing the brain lesion to affect some parts of the half-scene more than others. However, this interpretation is not compelling. For one thing, it doesn t explain why horizontal gradients of neglect are always-as far as am aware-in the direction of increasing severity from more medial (e.g., slightly left of center) to more lateral (e.g., far left of center) regions in the affected half of the represented scene. (Why doesn t neglect sometimes decrease from medial to lateral regions, or reach a peak somewhere in the middle? Why don t we occasionally observe circumscribed regions of neglect analogous to the scotomas frequently produced by damage to Vl?) t is also worth mentioning that neglect gradients-including their consistent direction-can be interpreted without positing any within-hemisphere spatial organization (see, e.g., Pouget & Sejnowski, 1997a, 1997b). Other types of evidence that might be adduced are similarly equivocal in their implications,

13 Spatial Remesentation in Mind and Brain auditory cortex Figure 5.5: The tonotopic map in primary auditory cortex. After Purves et al. (1997), Figure 12.14, p Adapted with permission of Sinauer Associates, nc. and at present the extent of spatial, organization at levels of representation affected by neglect would appear to be an open question. (Note that many different types of representations can be affected by neglect, and these may differ in the extent to which they have a spatial, organization in the brain.) Spatial, versus Spatial, Representations have defined the concept of a spatial, representation as if it applied only to spatial, representations (i.e., representations of spatial information). However, the concept is also applicable to representations of nonspatial information if the definition is stated in a slightly more general form. n particular, let us say that a representation is spatial, if (a) spatially-defined parts of the representation correspond to spatial or non-spatial parts of the represented material, and (b) at least one spatial property defined over the parts of the representation is isomorphic to a spatial or nonspatial property defined over the corresponding parts of the represented material. Consider the tonotopic map in primary auditory cortex (Figure 5.5). Sounds of different frequencies are represented in different parts of auditory cortex; thus, parts of the representation correspond to parts (i.e., particular frequencies) of the represented material. n addition a spatial dimension in auditory cortex (roughly anterior-posterior) is isomorphic to the frequency dimension in the domain of sounds, such that low frequencies project to more anterior locations, and high frequencies to more posterior sites. Thus, although the representation of frequency in primary auditory cortex is not a spatial, representation, it is a spatial, representation. 5 A Third Sense of Spatial Representation have distinguished representation of spatial information (spatial,) from spatial organization of representations (spatial,), and have noted that some brain representations have a spatial, organization. On the basis of phenomena from the study of neglect, have suggested that 5Both neurophysiological and behavioral evidence suggest that primary auditory cortex represents information about locations as well as frequencies of sounds (see, e.g., Zigmond, Bloom, Landis, Roberts, & Squire, 1999). Nevertheless, the representation of frequency in auditory cortex is a spatial, representation of non-spatial information.

14 1 12 Objects and Space spatial, organization may be present not only at relatively peripheral sensory and motor levels in the brain, but also at some higher levels (although the organization may be less rich at some levels than at others). What, though is the significance of spatial, organization in the brain? What does it tell us about how the brain works as a computational system? Representational and Non-Representational Properties of Representations To answer these questions we need to distinguish between representational and non-representational properties of a representation. This distinction may be illustrated through the examples in Figure 5.6, which involve not internal representations in the brain, but instead external representations on paper. Figure 5.6 panel A is a highly schematic map showing the locations of some major landmarks on the Mall in Washington, D.C. Certain spatial properties of the map stand for spatial properties of the Mall: Distances between dots represent,distances between landmarks; the relative positions of dots on the right-left dimension of the map represents the relative position of landmarks on the east-west dimension of the Mall, and so forth. Figure 5.6 panel B also represents spatial information about landmarks on the Mall. However, in this verbal representation the spatial information is carried by the words and numerals. Spatial properties of the representation-aside from the spatial arrangements of letters and digits-do not play a representational role; these properties do not represent spatial (or other) properties of the Mall. For example, spatial relations among the boxes in the representation do not represent spatial relations among the corresponding landmarks. Whereas the spatial relations among dots in the map are representational, the spatial relations among boxes in the verbal representation are nonrepresentational. (See Palmer, 1978, for related discussion.) A property of a representation is representational (or nonrepresentational) only within a system that uses the property in accordance with its posited representational role (e.g., Gallistel, 1990; Van Gulick, 1980). Thus, the spatial relations among dots in the map would be representational within a system that used them to stand for spatial relations among landmarks on the Figure 5.6: Two external representations of the locations of some landmarks on the Mall in Washington, D.C., illustrating the distinction between representational and non-representational properties. Panel A. A map. Panel B. A verbal representation. t- N lm O Lincoln Memorial O White House O Washington Monument O Capitol From Washington Monument: [ Capitol: 1.4 miles east ( White House: 0.5 miles north [ Lincoln Memorial: 0.8 miles west 1

15 Spatial Representation in Mind and Brain 3 Mall. Among such systems might be a person reading the map, and a computer program that takes the map as input (via a scanner) and produces as output possible routes for walking tours of the Mall. The example of the computer program illustrates an important point: To say that a property is representational in a system is not to say that the system knows, or is aware of, what the property stands for. Any property of a representation that appropriately affects the outcome of the system s computations is a representational property, whether or not the system has knowledge or awareness of the property s meaning. (See, e.g., Dennett, 1981; Fodor, 1980; Haugeland, 1985; Pylyshyn, 1980.) Spatial, Representation The distinction between representational and non-representational properties provides the basis for defining a third sense in which a representation could be spatial. Although this third sense, spatial,, is similar to the second (spatial,), will argue that the difference between the two is important for understanding spatial representation in the brain. The third sense of spatial representation may be defined as follows: A representation is spatial, within a system if (a) spatially-defined parts of the representation correspond to (spatial or nonspatial) parts of the represented material, (b) at least one spatial property defined over the parts of the representation is isomorphic to and is used to represent a (spatial or nonspatial) property defined over the corresponding parts of the represented material. Consider once again the map in Figure 5.6 panel A. Within a system including either the map reader or computer program mentioned above, the map is a spatial, representation, because spatially-defined parts of the representation (e.g., the dots) correspond to spatially-defined parts of the Mall (e.g., the landmarks), and spatial properties defined over parts of the representation (e.g., distances between dots) are isomorphic to, and are used to represent, properties defined over parts of the Mall (e.g., distances between landmarks). Table 5.1 summarizes the criteria for spatial,, spatial,, and spatial, representations. The spatial, criteria include those for a spatial, representation, but require in addition that at least one spatial property of the representation be representational. As a consequence, any spatial, representation is also spatial,, but not all spatial, representations are spatial,. Consider the representation in Figure 5.7, which combines the spatial isomorphisms from the map in Figure 5.6 panel A with the verbal information in Figure 5.6 panel B. This hybrid Table 5.1 Senses in which a Representation Can be Spatial Sense spatial, spatial, spatial, Criteria the represented information is spatial information (a) spatially-defined parts of the representation correspond to (spatial or nonspatial) parts of the represented material, and (b) at least one spatial property defined over the parts of the representation is isomorphic to a (spatial or non-spatial) property defined over the corresponding parts of the represented material (a) spatially-defined parts of the representation correspond to (spatial or nonspatial) parts of the represented material, and (b) at least one spatial property defined over the parts of the representation is isomorphic to, and is used to represent, a (spatial or non-spatial) property defined over the corresponding parts of the represented material

16 1 14 Objects and Space From Washington Monument: Figure 5.7: A hybrid representation, illustrating that spatial, representations may 1 [ or may not be spatial,. - Lincoln Memorial: 0.8 miles west White House: 0.5 miles north representation clearly qualifies as spatial,: Spatially-defined parts of the representation (the boxed areas) correspond to spatially-defined parts of the Mall (the landmarks), and spatial properties defined over parts of the representation (directional and distance relationships among boxes) are isomorphic to properties defined over parts of the Mall (direction-al and distance relationships among landmarks). However, the representation may or may not be spatial,, depending upon how it is used in a representational system. Suppose on the one hand that the representation were used by a person who ignored the positions of the boxes on the page, and relied solely on the words and numerals. Within a system consisting of this individual and the representation, the representation would not qualify as spatial,, because the spatial relations among boxes are non-representational. Even though the directional and distance relationships among boxes are isomorphic to the directional and distance relationships among the corresponding landmarks, the former are not used to represent the latter. On the other hand, in a system involving a person who used the spatial relations among boxes to navigate the mall, the representation would meet the spatial, criteria. Like the spatial, notion, the spatial, concept can be applied to representations of non-spatial as well as spatial information. For example, most graphs (e.g., a plot of the Dow-Jones ndustrial Average over time) are spatial, representations of non-spatial information. Spatial, versus Spatial, Representation in the Brain The distinction between spatial, and spatial, representations raises an obvious question: n brain representations with a spatial, organization, are spatial properties of the representations used to represent the corresponding properties of the represented material? n other words, are spatial, representations in the brain also spatial, representations? The answer appears to be no. Everything we know about the brain implies that the spatial properties of spatial, representations in the brain are non-representational. Consider once again the retinotopic map in V. As we have seen, V1 exhibits a rich and fine-grained spatial, organization, in which spatial properties defined over locations on the cortical surface are isomorphic to spatial properties defined over the corresponding locations in the visual field (see Figure 5.4). Nevertheless, to the best of our knowledge these spatial properties of the representation do not play representational roles (e.g., Bisiach & Berti, 1987). For example, distances between points in visual cortex are presumably not used in the brain s information processing to represent distances among the corresponding locations in the visual field; as far as we know, there is no brain process that determines the distance between cortical locations and uses this distance in performing computations. Spatial properties of the cortical representation are probably correlated with the properties that play representational roles. For example, neurons that are close together may be connected in different ways and to different extents than neurons that are farther apart, and these

17 Spatial Representation in Mind and Brain 1 15 variations in patterns of connectivity presumably have representational significance. However, distances between neurons are not representational in and of themselves. 6 These arguments apply not only to V1, but also to other brain representations with spatial, organization, such as the tonotopic map in primary auditory cortex, the somatotopic maps in somatosensory and motor areas of the brain, and the representations underlying the neglect phenomena have described. Brain representations, although they may be spatial,, are not spatial,. t is important to distinguish the spatial properties of a cortical representation from the receptive field characteristics of the cells making up the representation. The shapes and sizes of receptive fields, and the stimulus properties to which the cells respond (e.g., oriented edges), are certainly relevant for understanding what information is represented, and how it is represented for purposes of computation. However, the spatial relations among cells with particular response characteristics-for example, the fact that cells with receptive fields in the periphery of the visual field are anterior in V1 to those with more central receptive fields-do not speak directly to these representational issues. The inferences we could draw about what information was represented, and how it was represented, would be the same if neurons were scattered apparently at random in V1, such that the location of a neuron in the cortex had no relation to the location of its receptive field in visual space. Why, then, does the brain exhibit spatial, organization? One possibility is that this organization is a by-product of processes that establish connections among neurons during brain development (e.g., axonal guidance mechanisms). That is, connection-making processes may, for their own reasons, work in such a way as to produce spatial, organization in some brain areas, even though this organization plays no functional role in representing information. Perhaps, for example, the most effective process devised by evolution for connecting sensory receptors to cortical receiving areas is one that preserves the topology of the sensory surface throughout the pathway. Such a process could account for a number of spatial, representations in the brain, including the retinotopic map in primary visual cortex and the tonotopic map (which preserves the topology of the cochlea) in primary auditory cortex. mplications have distinguished three senses in which a representation could be spatial, arguing that whereas representations in the brain may be spatial in the first sense and/or in the second sense, they are not spatial in the third sense. n particular, have asserted that even when internal representations of spatial (or nonspatial) information are organized in the brain such that spatial properties of the representation mirror properties of the represented material, the former are not used by the brain to represent the latter. These conclusions have a number of implications, a few of which explore briefly. Questions about Spatial Representation One implication is that there are several distinct questions we can ask about the internal representation of spatial (or other) information, including the following: some instances the lengths of connections between neurons may have representational significance. For example, in certain brainstem circuits concerned with localization of sound sources the lengths of axons are critical to the functioning of the circuits, and could be considered representational. (For an overview of auditory localization mechanisms see Zigmond et al., 1999, pp ). n such cases the connection lengths may be correlated with metric distance between locations in the brain, but it is still the lengths of the connections and not the distances between brain locations that are representational.

18 1 16 Objects and Space What information is represented for purposes of computation? How is the information represented for purposes of computation? How are the representations arranged spatially in the brain? t is especially important to distinguish the second and third questions; answering one of these questions does not answer the other. For instance, the question of how spatial relations among locations in the visual field (e.g., above-below) are represented in V1 for purposes of computation cannot be answered by describing the spatial, organization of V1 (e.g., the isomorphism between the ventral-dorsal dimension in V1 and the above-below dimension in the visual field). n fact the spatial, organization of V1 does not even tell us whether spatial relations among visual-field locations are represented in a way that is accessible to computations. Of course, we know from behavioral evidence that the visual system does represent spatial relations in computationally-relevant ways; for example, people can respond differentially to visual stimuli differing only in the spatial relations among parts (e.g., 29 vs. 92). However, in the absence of such behavioral evidence the spatial, organization alone would not provide sufficient basis for assuming that spatial relations were represented for purposes of computation. The neglect studies presented as examples speak to what was represented (e.g., cloudlike figures as they would appear if presented as wholes), and how the representations were spatially organized in the brain (e.g., left halves of figures represented in the right hemisphere, and vice versa). The studies have less to say, however, about how information was represented for purposes of computation. For instance, it is not clear the anatomical separation between left- and right-hemisphere parts of representations has any significant implications for how the represented information is processed. develop this point more fully when discuss frames of reference in the second part of this chapter. Quasi-Pictorial Spatial Representations? Theorists addressing many aspects of spatial cognition (e.g., mental imagery, memory for visuo-spatial information, navigation) have assumed that the underlying representations are picture- or map-like in ways that are relevant to processing, such that processes carried out on the internal representations are somehow analogous to operations performed on pictures or maps (e.g., rotation, visual scanning). The arguments have developed suggest, however, that this assumption is problematic. Proponents of quasi-pictorial representations have been careful to emphasize that they are not endorsing naïve conceptions in which inner homunculi look at pictures projected on internal screens. Less clear, however, is what substantive theoretical proposals are being advanced. The most characteristic feature of external spatial representations like pictures and maps-the feature that most clearly distinguishes them from non-pictorial representations like words-is that spatial properties of the representation are used to represent spatial properties of the represented material. For example, a photograph of a cat represents the cat because certain spatial properties of the photograph are isomorphic to, and are used to represent, spatial properties of the cat. The external spatial representations are, in other words, spatial, representations. Yet this is exactly what internal representations of spatial information are not. What, then, does it mean to say that internal spatial, representations are quasi-pictorial? 7 The nternal Representational Space Metaphor The arguments have developed also have implications for the internal representational space metaphor frequently invoked in discussions of unilateral spatial neglect (as well as in other 7Note that use the pictorial expression represented scene to refer to what is represented, and not to the representations themselves.