NEURAL FOUNDATIONS OF IMAGERY

Transcription

1 NEURAL FOUNDATIONS OF IMAGERY Stephen M. Kosslyn, Giorgio Ganis and William L. Thompson Mental imagery has, until recently, fallen within the purview of philosophy and cognitive psychology. Both enterprises have raised important questions about imagery, but have not made substantial progress in answering them. With the advent of cognitive neuroscience, these questions have become empirically tractable. Neuroimaging studies, combined with other methods (such as studies of brain-damaged patients and of the effects of transcranial magnetic stimulation), are revealing the ways in which imagery draws on mechanisms used in other activities, such as perception and motor control. Because of its close relation to these basic processes, imagery is now becoming one of the best understood higher cognitive functions. BEHAVIOURISM The school of psychology that focused solely on observable stimuli, responses and the consequences of responses. TRANSCRANIAL MAGNETIC STIMULATION (TMS). A technique used to induce a transient interruption of normal activity in a relatively restricted area of the brain. It is based on the generation of a strong magnetic field near the area of interest, which, if changed rapidly enough, will induce an electric field sufficient to stimulate neurons. Harvard University, 830 William James Hall, 33 Kirkland Street, Cambridge, Massachusetts 02138, USA. Correspondence to S.M.K. smk@wjh.harvard.edu Mental imagery occurs when perceptual information is accessed from memory, giving rise to the experience of seeing with the mind s eye, hearing with the mind s ear and so on. By contrast, perception occurs when information is registered directly from the senses. Mental images need not result simply from the recall of previously perceived objects or events; they can also be created by combining and modifying stored perceptual information in novel ways. Imagery has had a central role in theories of mental function since at least the time of Plato. It has fallen in and out of fashion, in large part because it is inherently a private affair, by definition restricted to the confines of the mind, and so it has been difficult to study. In fact, in 1913, the founder of BEHAVIOURISM, John B. Watson, denied that mental images even existed. Instead, he suggested that thinking consists of subtle movements of the vocal apparatus 1. In spite of the demonstration by Alan Paivio and his colleagues that the use of imagery greatly improves memory 2, many researchers were not convinced that imagery is a distinct form of thought. Indeed, Watson s position was echoed 60 years later by Zenon Pylyshyn, who championed the view that mental images are not images at all, but rather rely on mental descriptions no different in kind from those that underlie language. According to Pylyshyn 3, the pictorial aspects of imagery that are evident to conscious experience are entirely epiphenomenal, like the heat thrown off by a light bulb when you read (which has no role in the reading process). The emergence of cognitive neuroscience has opened a new chapter in the study of imagery. An enormous amount has been learned about the neural underpinnings of visual perception, memory, emotion and motor control. Much of this information has come from the study of animal models. Unlike language and reasoning, these more basic functions have many common features among higher mammals, including humans. In addition, new neuroimaging technologies, especially positron emission tomography (PET) and functional magnetic resonance imaging (fmri), allow theories of imagery to be tested objectively in humans. Researchers have taken advantage of these developments to show that mental imagery draws on much of the same neural machinery as perception in the same modality, and can engage mechanisms used in memory, emotion and motor control. In this article, we draw on results from a variety of methods, including studies of the effects of selective brain damage on behaviour, neuroimaging and TRANS- CRANIAL MAGNETIC STIMULATION (TMS). Each approach has its strengths and weaknesses, but the methods are complementary. So, for example, neuroimaging provides only correlational data (when engaged in a particular task, a particular set of brain areas is activated), but can monitor the entire brain; TMS, by contrast, can be used to establish causal roles of distinct areas (for example, by showing that performance in a task that draws on a specific brain area is impaired following TMS to that area), but must be targeted to a specific location. To the extent that the same conclusions are reached using different methods, the conclusions drawn from these studies can be taken increasingly seriously. NATURE REVIEWS NEUROSCIENCE VOLUME 2 SEPTEMBER

2 We briefly review three main classes of research: evidence that imagery engages brain mechanisms that are used in perception and action; evidence that visual mental imagery engages even the earliest visual cortex (areas 17 and 18); and evidence that imagery engages mechanisms that control physiological processes such as heart rate and breathing, having effects much like those that occur with the corresponding perceptual stimuli. Imagery, perception and action We begin with visual imagery, which is by far the most intensively studied modality, and then turn to auditory and motor imagery. Visual mental imagery. More than 100 years ago, researchers described brain-damaged patients who had lost the ability to form visual mental images after they became blind (for review, see REF. 4; however, see also REF. 5). Methods from cognitive psychology have allowed researchers to characterize such deficits with increasing precision. For example, some patients have perceptual deficits in only one of the two main cortical visual functions. One major visual pathway runs from the occipital lobe down to the inferior temporal lobes (the ventral or object properties processing pathway; see REF. 6); when damaged, the animal or person cannot easily recognize shape. The other main visual pathway runs from the occipital lobe to the posterior parietal lobes (the dorsal or spatial properties processing pathway); when damaged, the animal or person cannot easily register location. The parallel deficits appear in imagery: damage to the ventral pathway disrupts the ability to visualize shape (as used, for example, to determine from memory whether George Washington had a beard), whereas damage to the dorsal pathway disrupts the ability to visualize locations (as used, for example, to indicate the locations of furniture in a room when your eyes are closed 7 ). Indeed, very subtle deficits can occur in imagery that parallel the deficits found in perception. For example, some brain-damaged patients can no longer distinguish colours perceptually or in imagery 8, and others can no longer distinguish faces perceptually or in imagery 9. However, although the deficits in imagery and perception often parallel each other, this is not always the case. In a seminal literature review and analysis, Farah showed that some patients have selective problems in generating images, even though they are able to recognize and identify perceptual stimuli 4. In addition, patients have been reported who could visualize, but had impaired perception (for example, see REFS 10,11). In short, the results from research with brain-damaged patients indicate that visual mental imagery and visual perception share many mechanisms, but do not draw on identical processes. Although shape, location and surface characteristics are represented and interpreted in similar ways during both functions, the two differ in key ways: imagery, unlike perception, does not require lowlevel organizational processing, whereas perception, unlike imagery, does not require us to activate information in memory when the stimulus is not present. For a review of the relationship between imagery and memory, as well as a discussion of intriguing hemispheric effects in imagery, see REF. 12. The results of neuroimaging studies that compare imagery and perception have dovetailed nicely with those from studies of brain-damaged patients. One study, for example, found that of all the brain areas that were activated during perception and during imagery, approximately two-thirds were activated in both cases 13. Presumably, lesions in the areas that are not activated in common produce the dissociations, in which imagery or perception is disrupted independently, whereas lesions in the areas that are activated in both cases produce the more frequently reported parallel deficits in imagery and perception. Finally, studies of deficits following brain damage have underscored the fact that imagery like other cognitive functions is not a single, undifferentiated ability. Rather, it is a collection of abilities, which can be disrupted independently. For example, some patients can make imagery judgements about the shape or colour of objects, but have difficulty imagining an object rotating (for example, when trying to decide whether the letter p would be another letter when rotated through 180 degrees, or whether Z would be another letter when rotated 90 degrees clockwise). Other patients have the reverse pattern of deficits. Indeed, when subjects perform different imagery tasks while their brain activity is monitored, different patterns of activation are observed. For example, when subjects mentally rotate patterns, their parietal lobes (often bilaterally) and right frontal lobes are, typically, strongly activated (for example, see REFS 14 18). By contrast, if they are asked to visualize previously memorized patterns of stripes and judge which are longer, wider and so on (all on the basis of their mental images, with eyes closed), these areas are not activated, but other areas in the occipital lobe and left association cortex are activated 19. Depending on the precise task, different sets of processes are activated 20. Indeed, brain activation during mental imagery might vary according to the type of object that is visualized. Using fmri, O Craven and Kanwisher 20 found activation in the fusiform face area 21 (FFA) when subjects visualized faces. But when subjects visualized indoor or outdoor scenes depicting a spatial layout, these researchers found activation in the parahippocampal place area (PPA). There was no hint of activation of the PPA during face imagery, nor of the FFA during place imagery. These results are similar to what was observed when subjects actually perceived faces and places. The findings indicate that imagery and perception share very specific, specialized mechanisms. Auditory imagery. Do the first three notes of the children s song Three Blind Mice ascend or descend? Most people report that they hear the song in the process of deciding. Remarkably little research has addressed auditory imagery per se. Zatorre and Halpern 22 studied brain-damaged patients to discover whether specific brain areas are crucial for auditory imagery. They compared a group of patients who had had the left or right temporal lobe removed (for the treatment of otherwise intractable 636 SEPTEMBER 2001 VOLUME 2

3 a b Figure 1 Auditory imagery. a A subject lying in a positron emission tomography (PET) scanner, listening to or imagining music. b The patterns of activation detected in auditory imagery (Ima) and perception (Per), compared with a visual baseline (B). Left- and right-hemisphere sagittal slices are shown in both panels. The top panel shows activation in the superior temporal gyrus (STG). Although activation was stronger during perception than in imagery, it was located in similar regions of the temporal lobes in both conditions. In the bottom panel, similar areas of activation between imagery and perception were found in the supramarginal gyrus, and in the middle frontal (Mid F) and inferior frontal (Ant Inf F) cortices. PET data panel reprinted with permission from REF The MIT Press. epilepsy), with otherwise similar control subjects. In one condition, the subjects heard a familiar song while also reading the lyrics, and judged which of two particular words had the higher pitch. In another condition, the subjects saw the lyrics and made the same judgements, but did not actually hear the song, and so had to rely on mental imagery. The patients with right temporal lesions were impaired in both conditions compared with both other groups. These findings show that at least some of the neural structures that are crucial for pitch discrimination during perception have a similar role during imagery. Most neuroimaging research on auditory imagery has focused on imagery for music. Zatorre et al. 23 asked whether auditory imagery draws on the same mechanisms used in auditory perception. Their subjects either listened to songs and judged the relative pitch of pairs of words, or imagined hearing songs and made the same judgements. No auditory stimulation was present during the baseline condition, which required the subjects to judge the relative length of visually presented words. PET revealed that many of the same areas were activated in common in these tasks (FIG. 1), including bilateral associative auditory cortex, or Brodmann area (BA) 21/22 (in spite of the fact that the left temporal lobe has often been identified with the perception of language, and the right with music or environmental sounds), bilateral frontal cortex (BA 45/9 and 10/47), left parietal cortex (BA 40/7) and supplementary motor cortex (BA 6). The bilateral activation in associative auditory cortex observed in this study, in apparent contrast to the patient studies, might indicate that some of the activated areas were not essential to these tasks. Indeed, in a subsequent study, Halpern and Zatorre 24 asked musically trained subjects to listen to the opening notes of familiar (non-verbal) melodies and then to continue hearing the melody with the mind s ear. Again using PET, they found activation in two regions of the right temporal lobe (the superior and inferior temporal cortex), which is consistent with their earlier study of brain-damaged patients; both of these areas are involved in storing and interpreting non-verbal sounds. Moreover, auditory imagery of a melody that required retrieval from memory also activated two right-hemisphere regions, in the frontal lobe and superior temporal gyrus (which is crucial for auditory perception). Finally, the supplementary motor area (SMA) was also activated by auditory imagery, regardless of whether the melody was retrieved or simply rehearsed online. This is interesting, because no overt behaviour was required. Halpern and Zatorre believe that stored movements are used in this sort of imagery; this makes sense for melodies, in which case we can subvocalize the tune as part of the process of retrieving the information. Finally, Griffiths 25 reports a study of patients who became deaf and then hallucinated hearing music. These patients were neither psychotic nor beset with an obvious neurological problem, such as epilepsy. Griffiths was able to perform PET while the patients had such hallucinations, and found that the posterior temporal lobes, in the auditory cortex, were activated as well as the right basal ganglia, the cerebellum and the inferior frontal cortices. In summary, auditory imagery appears to draw on most of the neural structures used in auditory perception. However, in contrast to visual imagery, in which the early visual cortex seems to be activated (see below), there is no evidence that the first auditory cortical area to receive input from the ears area A1 is activated during auditory imagery. NATURE REVIEWS NEUROSCIENCE VOLUME 2 SEPTEMBER

4 Box 1 Mirror neurons A subpopulation of neurons in area F5 of the monkey brain (itself part of premotor cortex) responds selectively not only when the animal performs specific actions with the hand or mouth, but also when the animal observes the same actions being performed by another monkey (or human) 57. Such neurons have been labelled mirror neurons. Neuroimaging and transcranial magnetic stimulation (TMS) studies have shown that the human premotor cortex is activated when humans observe other people s actions (for example, see REFS 58 62), consistent with the existence of mirror neurons in the human brain. The likely homologue of area F5 in humans is Broca s area (typically characterized as being involved in speech production), which has prompted some authors to propose that the mirror neurons in humans might have a crucial role not only in imitation, but also in language acquisition. Mirror neurons might also be involved in motor imagery, consistent with the idea that people often transform images by imagining what they would see if the objects were manipulated in a specific way. For further discussion of mirror neurons, see REF. 63 in this issue of Nature Reviews Neuroscience. MOTOR STRIP Primary motor cortex (area M1). Part of the frontal lobe, which is used to control fine-grained movements. Motor imagery. When people are asked to imagine walking to a specific goal placed in front of them, and to indicate when they would have arrived, their estimates of transit time are remarkably similar to the actual time they subsequently require to walk that distance 26.In such tasks, people report that they imagine moving. Many studies have now been carried out to investigate the neural bases of such motor imagery, and to distinguish motor imagery from purely visual imagery. Although visual imagery can often accompany motor imagery, researchers have documented that motor imagery relies on distinct mechanisms. Specifically, many researchers have shown that areas of cortex used in movement control also have a role in motor imagery. In a classic study, Georgopoulos et al. 27 recorded activity in individual neurons in the MOTOR STRIP of monkeys while the animals were planning to move a lever along a specific arc. They found that these neurons fired in a systematic sequence, depending on their orientation tuning. Specifically, at first, only neurons tuned for orientations near the starting position of the lever fired, followed by those tuned for orientations slightly farther along the trajectory and so on. All of this occurred before the animal actually began to move. However, these findings do not show that mental rotation occurs in the motor strip itself; it is possible that the computation takes place elsewhere in the brain (for example, in the posterior parietal lobes), and that the results of such computation are simply executed in the motor strip. Many neuroimaging studies of mental rotation have been reported, all of which have shown that multiple brain areas are activated during mental rotation. For example, Richter et al. 18 measured brain activation with fmri while subjects mentally rotated the three-dimensional multi-armed angular stimuli invented by Shepard and Metzler 28 (which look as if they had been constructed by gluing small cubes together to form the arms). Subjects were shown pairs of such shapes and asked to report whether the figures in each pair were the same or mirror reversed. Richter et al. 18 report that the superior parietal lobules (in both hemispheres) were activated during this task, as well as the premotor cortex (in both hemispheres), supplementary motor cortex and the left primary motor cortex. Other neuroimaging studies have provided strong support for the role of motor processes in mental transformations. For example, Parsons et al. 29 showed subjects a picture of a hand, which could be rotated to various degrees; the pictures were presented in the left visual field (so the image was registered first by the right hemisphere) or in the right visual field (so the image was registered first by the left hemisphere). The subjects were asked to decide whether each picture showed a left or right hand. Parsons et al. 29 expected motor cortices to be activated in this task if subjects imagined rotating their own hand into congruence with the stimulus. Not only was the supplementary motor cortex activated bilaterally, but also the prefrontal and insular premotor areas were activated in the hemisphere contralateral to the stimulus handedness, indicating that subjects did, in fact, imagine the appropriate movements. Many other areas, including the frontal and parietal lobes, and basal ganglia and cerebellum, were active, as was area 17. Is motor imagery used only to rotate parts of the body? Some researchers have suggested that people often transform images by imagining what they would see if the objects were manipulated in a specific way (which might sometimes involve the operation of socalled mirror neurons ; see BOX 1). One PET study directly compared rotation of hands versus inanimate objects 15, again using Shepard and Metzler s three-dimensional multi-armed stimuli 28. The subjects compared pairs of drawings and decided whether they were identical or mirror images. In the experimental condition, the figures were presented at different relative orientations, and one had to be mentally rotated into congruence with the other; in the baseline condition, the figures were presented at the same orientation, so no mental rotation was necessary. The comparison of the two conditions revealed which areas were activated specifically by mental rotation. The corresponding design was used for drawings of hands, but now the subjects decided whether the two hands in a pair were both left or both right, or whether one was a left hand and one a right hand. In this study 15, several motor areas were activated when subjects mentally rotated hands, including the primary motor cortex (area M1), premotor cortex and the posterior parietal lobe. None of the frontal motor areas was activated when objects were mentally rotated. However, Cohen et al. 14 used fmri to study mental rotation of exactly the same inanimate objects, and found that the premotor cortex was activated in this task, but in only half of the subjects. The fact that only some subjects had activation in a motor area during the mental rotation of inanimate objects indicates that there could be two strategies for performing such rotations. One strategy involves imagining what you would see if you manipulated an object; the other involves imagining what you would see if someone else (or an external force, such as a motor) manipulated an object. To test this idea, Kosslyn et al. 33 asked subjects to perform the same mental rotation task 15, but with a twist: immediately before the task, the subjects either saw a wooden model of that type of stimulus (one not actually used in the task) being rotated by 638 SEPTEMBER 2001 VOLUME 2

5 an electric motor, or physically turned the stimulus themselves. They were told that during the task they should imagine the stimuli being rotated just as they had seen the model rotate at the outset. In this experiment, area M1 was activated when subjects mentally rotated stimuli after having themselves physically rotated the stimulus (and then imagined themselves doing so), but not when they saw the electric motor rotating the stimulus at the outset (FIG. 2). The results show that imagining oneself manipulating an object is one way in which the mental transformation of objects in general (not just body parts) can take place; furthermore, they show that humans can voluntarily adopt this strategy, or use a strategy in which they imagine what they would see if an external force transformed an object. Finally, we can ask whether the primary motor cortex has a functional role in allowing subjects to manipulate objects in images. It is possible that the actual computation is taking place in another area that incidentally activates the primary motor cortex. To test this hypothesis, Ganis et al. 34 disrupted function in the left primary motor cortex by administering TMS while subjects mentally rotated pictures of hands and feet (with the tobe-rotated stimulus appearing in the right visual field). The TMS was time-locked so that it disrupted neural processing only a specific amount of time after the stimulus appeared. Subjects required more time to perform this task if a single magnetic pulse was delivered to the motor strip (roughly over the hand area ) 650 ms after the stimuli were presented (but not at the other temporal delays tested); moreover, rotation of hands was impaired more than rotation of feet, as expected if this area is specialized for controlling the hand. Within the limits of the spatial resolution afforded by the TMS technique, these results indicate that activation in this area reflects processing used to perform the task. However, we cannot say whether this area is the main site of processing, or relays information that is computed elsewhere in the brain. So, mental imagery can engage the motor system. This finding could help to explain why mental practise can improve actual performance In this case, imagining making movements might not only exercise the relevant brain areas, but also build associations among processes implemented in different areas, which in turn facilitate complex performance. Imagery and early visual cortex Much research on the neural bases of visual imagery focuses on whether early visual cortex is activated during imagery. The early visual cortex comprises areas 17 and 18, the first two cortical areas to receive input from the eyes. Researchers have asked whether visual imagery activates these early areas for three main reasons. First, these areas are topographically organized: they preserve (roughly) the local spatial geometry of the retina, so patterns of activation in them depict shape. If these areas are activated during imagery, and such activation has a functional role, this would be evidence that imagery relies on representations that depict information, not a Internal b External Figure 2 Mental rotation. a At the outset of the Kosslyn et al. 33 study, the participants learned to visualize mental rotation by either external action (rotation driven by an electric motor) or internal action (rotation driven by manual turning of the figure). b Using positron emission tomography (PET), an axial brain slice taken 56 mm above the anterior commissure posterior commissure (AC PC) line shows activation in the primary motor cortex (M1) when data from the external-action condition were subtracted from those of the internal-action condition. Depending on the strategy used, motor regions of the brain are recruited during mental rotation. The result also shows that the strategy used to accomplish a given task can vary according to previous training, and can be adopted voluntarily. PET data panel reprinted from REF Lippincott Williams & Wilkins Ltd. describe it. (In other words, this would be evidence that mental imagery relies on actual images.) Second, such findings could not be explained solely by tacit knowledge stored as descriptions, which Pylyshyn 38 used to explain away the findings from earlier behavioural experiments that attempted to show that imagery relies on depictive representations. According to this view, subjects in imagery experiments could have unconsciously tried to imitate what they thought they would have done in the corresponding perceptual situation (for example, by taking more time to scan farther distances across an imaged scene). But such tacit knowledge, NATURE REVIEWS NEUROSCIENCE VOLUME 2 SEPTEMBER

6 Response time (ms) REVIEWS Sham Real Sham Real Figure 3 Area 17 is involved in visual imagery. Before the imagery condition, the participants memorized the stimulus display. They also learned which quadrants were labelled by the numbers 1, 2, 3 and 4. During the imagery task, the participants visualized the entire display, and then listened for the cue to compare two quadrants on a specific dimension. Their task was to decide whether the stripes in the quadrant named first had a pattern that was greater on the named dimension (for example, longer stripes) than the stripes in the quadrant named second; if so, they were to press one button, if not, the other button. The participants were told that they should visualize the entire display and look at the image to make the discrimination. Repetitive transcranial magnetic stimulation (rtms) applied to the medial occipital cortex was performed using a magnetic stimulator and a figure-of-eight coil. During real rtms, the centre of the coil targeted the tip of the calcarine fissure (note that the coil positions in the figure are shown for illustrative purposes only). During sham rtms, the induced magnetic field did not enter the brain, although the touch on the scalp and the sound of the coil being activated were comparable to those in the real rtms condition. Performance on the imagery task after rtms is illustrated for each individual subject (n = 5); performance degraded (response times increased to a comparable degree) in the real rtms condition in both perception and imagery. Behavioural data panel reprinted with permission from REF American Association for the Advancement of Science. SINGLE-PHOTON EMISSION COMPUTED TOMOGRAPHY (SPECT). A method in which images are generated by using radionuclides that emit single photons of a given energy. Images are captured at multiple positions by rotating the sensor around the subject; the threedimensional distribution of radionuclides is then used to reconstruct the images. SPECT can be used to observe biochemical and physiological processes, as well as the size and volume of structures. stored as descriptions, would not explain why early visual cortex would be activated when subjects had their eyes closed during imagery. Third, if imagery can alter the activation of early visual cortex, this indicates that our beliefs and expectations can (at least under some circumstances) modulate what we actually see during perception. This finding would have clear-cut implications for the reliability of eyewitness testimony and, more generally, the veracity of visual memory. More than 50 neuroimaging studies have examined activation in the early visual cortex (for review, see REF. 39). The studies used, in decreasing order of sensitivity, fmri, PET and SINGLE-PHOTON EMISSION COMPUTED TOMOGRAPHY (SPECT). To our knowledge, 21 fmri, 11 PET and two SPECT studies have reported activation in the early visual cortex during visual mental imagery, compared with three fmri, 13 PET and seven SPECT studies that reported no such activation. The following studies seem to provide the strongest support for activation in the early visual cortex during visual mental imagery. The subjects had their eyes closed during all of the neuroimaging tasks, so activation of the early visual cortex could not have been caused by seeing patterns. In one study, subjects were asked to visualize line drawings of objects at different sizes (as if they fit into boxes that were memorized before the PET scan) 40. Not only was area 17 activated when compared with a control condition in which identical auditory cues were provided but no imagery was used, but also the specific locus of activation depended on the size of the imaged object. Even though the subjects eyes were closed, the mere fact of visualizing an object at a larger size shifted the activation to more anterior parts of the calcarine sulcus (the principal anatomical landmark of area 17), just as is found in perception proper 41. This result was replicated by Tootell et al. 42 using fmri with a precise method to localize area 17; there is no doubt that varying the size of objects in mental images shifts the locus of activation along area 17 in a way that resembles what occurs during perception. Second, Klein et al. 43 used event-related fmri to chart activation in area 17 when visual mental images were formed. They found clear activation in every subject, with a clear-cut temporal pattern: activation began about 2 s after an auditory cue to form an image, and peaked around 4 6 s later, before dropping off during the next 8 s or so. But does such activation have a functional role in imagery? In another study, illustrated in FIG. 3, subjects memorized four quadrants, each with black and white stripes (which varied in length, width, orientation and separation), and later had to visualize them and make subtle shape comparisons; for example, in identifying which set had longer or wider stripes 19. PET scanning showed that area 17 was activated during this task. Moreover, when repetitive TMS was applied to area 17 before the task, every subject subsequently required more time to make these judgements than when repetitive TMS was applied so that it did not affect area 17. Indeed, the magnitude of the decrement in performance was the same when subjects had their eyes closed and visualized the stripes, as when they had their eyes open and made judgements based on visible stripes. This makes sense if area 17 is crucial in both the imagery and perceptual versions of the task. Indeed, Farah et al. 44 found that after 640 SEPTEMBER 2001 VOLUME 2

7 HIGH-RESOLUTION DETAIL A feature of a visual percept or image that requires high resolution (operationalized here as 0.5 of visual angle or less, as viewed from the subject s vantage point) to discern. A meta-analysis indicates that the early visual cortex is activated during visual mental imagery when the task requires the extraction of high-resolution details from a visualized stimulus. RECEPTIVE FIELD The area of the sensory space in which stimulus presentation leads to the response of a particular sensory neuron. DIFFUSE OPTICAL TOMOGRAPHY (DOT). A neuroimaging technique that uses arrays of lasers and detectors to measure changes in the absorption of near-infrared light caused by neural activation. The most widely used type of DOT measures changes in blood oxygenation caused by neural activity. one occipital lobe was surgically removed from a patient (as part of a medical treatment), the apparent horizontal extent of objects in images decreased by approximately half, as expected if each occipital lobe represents the contralateral part of space. Finally, in another PET study, subjects closed their eyes and visualized named letters of the alphabet in upper case 45. Four seconds after forming the image, they were asked to judge whether the letter had a specific characteristic (such as any curved lines); the response times and error rates were recorded while their brains were scanned. Not only were variations in the level of activation in area 17 significantly correlated with the time subjects required to make the judgements, but this correlation was present even after all other correlations between variations in regional cerebral blood flow and response time were statistically removed. The results indicate that activation in area 17 is systematically related to spatial properties of the imaged object if area 17 is impaired by TMS or brain damage, so is the use of visual imagery and is not likely to be an artefact of activation in other areas that is transmitted incidentally (via neural connections) to area 17. Given these positive results, why have so many studies failed to find activation in area 17? A meta-analysis that we recently completed (see also REF. 39) indicates that the sensitivity of the technique is important (note the proportion of fmri studies that detected such activation versus those that did not, 21:3, compared with the corresponding proportion for the much less sensitive SPECT technique, 2:7). In addition, the metaanalysis revealed that if a task requires subjects to find a HIGH-RESOLUTION DETAIL in an image (such as by evaluating the shape of an animal s ears or comparing two similar sets of stripes), activation in early visual cortex is likely. If a task requires a spatial judgement (which might be mediated by the parietal lobe), activation is less likely. Many of the studies that did not report activation in early visual cortex used spatial tasks Finally, preliminary results of the meta-analysis indicate that the presence of light might facilitate visual mental imagery. If this result proves to be robust, it could provide important hints about the mechanisms of visual mental imagery; it is possible that light stimulates early visual cortex in a way that facilitates top down processing during imagery. A second puzzle is why some brain-damaged patients continue to have some use of imagery, in spite of the fact that early visual cortex has been severely damaged 5. Probably the most straightforward account of this finding is that the early visual cortex is not necessary for all forms of visual imagery. Indeed, Crick and Koch 49 make a good case for the idea that the experience of visual perception does not arise from the early visual cortex, but from later areas that receive input from the earlier ones. The same is probably true in imagery: if later areas are activated in the absence of the appropriate immediate sensory input, we can experience visual imagery. However, such later areas do not make fine spatial variations accessible to later processes, and so we apparently need to reconstruct the local geometry in earlier areas (which have much smaller RECEPTIVE FIELDS, and so higher resolution) if we are to extract fine-grained details from the imaged object. Imagery: the world within The great behaviourist B. F. Skinner 50 wrote: There is no evidence of the mental construction of images to be looked at or maps to be followed. The body responds to the world, at the point of contact; making copies would be a waste of time. We hope that we have convinced you that the first part of this claim is incorrect: images are in fact internal representations. We now briefly consider the second part: whether having such representations is a waste of time. There is much evidence that imagery of emotional events activates the autonomic nervous system and (as is also evident in single-neuron recordings in humans) the amygdala. That is, visualizing an object has much the same effects on the body as actually seeing the object. For example, skin conductance increases, as do heart rate and breathing rate, when subjects view pictures of threatening objects 51 ; and the same result occurs when they merely visualize the objects. Indeed, mental images of aversive stimuli activate the anterior insula 52, the principal cortical site of feedback from the autonomic nervous system. In addition, Kreiman, Koch and Fried 53 recorded from single cells in the human brain (hippocampus, amygdala, enthorinal cortex and parahippocampal gyrus) while subjects were shown pictures or formed mental images of those same pictures. Some of the cells that responded selectively when subjects viewed specific visual stimuli (for example, faces) also responded selectively when those same stimuli were visualized. Of particular interest, this pattern was seen in the amygdala, which is known to have a key role in certain emotions, especially fear and anger 54,55.So, imagery can engage neural structures that are also engaged in perception, and those neural structures can, in turn, affect events in the body itself. Conclusions Imagery is no longer seen as an awkward leftover from a previous, less rigorous age a topic unfit for polite company. Rather, researchers agree that most of the neural processes that underlie like-modality perception are also used in imagery; and imagery, in many ways, can stand in for (re-present, if you will) a perceptual stimulus or situation. Imagery not only engages the motor system, but also affects the body, much as can actual perceptual experience. Nevertheless, many questions remain. For example, under what circumstances is the early sensory cortex recruited during imagery? Why is the early sensory cortex often recruited during visual mental imagery, but not during auditory imagery? Why do people differ so much in their imagery abilities? Does genetics affect some aspects of imagery more than others? How does semantic content in images engage specific mechanisms? Unlike the situation even 20 years ago, questions such as these can now be answered. Indeed, the advent of additional technologies, such as DIFFUSE OPTICAL TOMOGRAPHY 56 NATURE REVIEWS NEUROSCIENCE VOLUME 2 SEPTEMBER

8 (DOT), promises to facilitate studies of the neural bases of imagery. This technique is portable, very inexpensive, and more forgiving than fmri when subjects move. It is also totally silent. The drawback is that it can only monitor cortical activity, and not even all of that. However, it can assess the lion s share of cortex, and will allow largescale individual-differences studies to be done. Such studies can relate differences in patterns of brain activation to differences in performance, which should, in turn, tell us not only about what the brain is doing, but also about how and why people differ in their modes of thinking. Links MIT ENCYCLOPEDIA OF COGNITIVE SCIENCES Imagery Positron emission tomography Magnetic resonance imaging Single-neuron recording 1. Watson, J. B. Psychology as the behaviorist views it. Psychol. Rev. 20, (1913). 2. Paivio, A. Imagery and Verbal Processes (Holt, Rinehart and Winston, New York, 1971). 3. Pylyshyn, Z. W. What the mind s eye tells the mind s brain: a critique of mental imagery. Psychol. Bull. 80, 1 24 (1973). This is a contemporary critique of visual mental imagery, which focuses on the necessity to have explicit, mechanistic theories. Findings from, and theories in, cognitive neuroscience address many of the concerns voiced here. 4. Farah, M. J. The neurological basis of mental imagery: a componential analysis. Cognition 18, (1984). This is a creative analysis of the perceptual and imagery abilities that are damaged and spared following brain damage. This analysis shows, without question, that imagery is not a single faculty, but is carried out by the joint operation of a set of processes. 5. Chatterjee, A. & Southwood, M. H. Cortical blindness and visual imagery. Neurology 45, (1995). 6. Ungerleider, L. G. & Mishkin, M. in Analysis of Visual Behavior (eds Ingle, D. J., Goodale, M. A. & Mansfield, R. J. W.) (MIT Press, Cambridge, Massachusetts, 1982). 7. Levine, D. N., Warach, J. & Farah, M. J. Two visual systems in mental imagery: dissociation of what and where in imagery disorders due to bilateral posterior cerebral lesions. Neurology 35, (1985). 8. De Vreese, L. P. Two systems for colour-naming defects: verbal disconnection vs colour imagery disorder. Neuropsychologia 29, 1 18 (1991). 9. Young, A. W., Humphreys, G. W., Riddoch, M. J., Hellawell, D. J. & De Haan, E. H. Recognition impairments and face imagery. Neuropsychologia 32, (1994). 10. Behrmann, M., Winocur, G. & Moscovitch, M. Dissociation between mental imagery and object recognition in a braindamaged patient. Nature 359, (1992). 11. Jankowiak, J., Kinsbourne, M., Shalev, R. S. & Bachman, D. L. Preserved visual imagery and categorization in a case of associative visual agnosia. J. Cogn. Neurosci. 4, (1992). 12. Behrmann, M. The mind s eye mapped onto the brain s matter. Curr. Dir. Psychol. Sci. 9, (2000). 13. Kosslyn, S. M., Thompson, W. L. & Alpert, N. M. Neural systems shared by visual imagery and visual perception: a positron emission tomography study. Neuroimage 6, (1997). 14. Cohen, M. S. et al. Changes in cortical activity during mental rotation: a mapping study using functional MRI. Brain 119, (1996). 15. Kosslyn, S. M., DiGirolamo, G., Thompson, W. L. & Alpert, N. M. Mental rotation of objects versus hands: neural mechanisms revealed by positron emission tomography. Psychophysiology 35, (1998). 16. Jordan, K., Heinze, H. J., Lutz, K., Kanowski, M. & Jancke, L. Cortical activations during the mental rotation of different visual objects. Neuroimage 13, (2001). 17. Ng, V. W. et al. Identifying rate-limiting nodes in large-scale cortical networks for visuospatial processing: an illustration using fmri. J. Cogn. Neurosci. 13, (2001). 18. Richter, W. et al. Motor area activity during mental rotation studied by time-resolved single-trial fmri. J. Cogn. Neurosci. 12, (2000). 19. Kosslyn, S. M. et al. The role of area 17 in visual imagery: convergent evidence from PET and rtms. Science 284, (1999). This study demonstrates that area 17 is activated during imagery, and that impairing the functioning of this area, in turn, impairs the performance of an imagery task, providing evidence that this area has a causal role in such processing. 20. O Craven, K. M. & Kanwisher, N. Mental imagery of faces and places activates corresponding stimulus-specific brain regions. J. Cogn. Neurosci. 12, (2000). 21. Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, (1997). 22. Zatorre, R. J. & Halpern, A. R. Effect of unilateral temporallobe excision on perception and imagery of songs. Neuropsychologia 31, (1993). 23. Zatorre, R. J., Halpern, A. R., Perry, D. W., Meyer, E. & Evans, A. C. Hearing in the mind s ear: a PET investigation of musical imagery and perception. J. Cogn. Neurosci. 8, (1996). Auditory imagery is in many ways analogous to visual imagery, but activates only higher level brain areas, not early auditory cortex. The similarities and differences between the different modalities illuminate key facets of the underlying mechanisms of imagery. 24. Halpern, A. R. & Zatorre, R. J. When that tune runs through your head: a PET investigation of auditory imagery for familiar melodies. Cereb. Cortex 9, (1999). 25. Griffiths, T. D. Musical hallucinosis in acquired deafness. Phenomenology and brain substrate. Brain 123, (2000). 26. Decety, J., Jeannerod, J. & Prablanc, C. The timing of mentally represented actions. Behav. Brain Res. 34, (1989). 27. Georgopoulos, A. P., Lurito, J. T., Petrides, M., Schwartz, A. B. & Massey, J. T. Mental rotation of the neuronal population vector. Science 243, (1989). 28. Shepard, R. N. & Metzler, J. Mental rotation of threedimensional objects. Science 171, (1971). This is the classic and in many ways still the best study of mental rotation. It is methodologically tight and reveals that rotation in the picture plane and in depth are equally easy, even though the stimuli are only pictures of three-dimensional objects. 29. Parsons, L. M. et al. Use of implicit motor imagery for visual shape discrimination as revealed by PET. Nature 375, (1995). 30. Decety, J. Neural representation for action. Rev. Neurosci. 7, (1996). 31. Jeannerod, M. & Decety, J. Mental motor imagery: a window into the representational stages of action. Curr. Opin. Neurobiol. 5, (1995). 32. Jeannerod, M. The representing brain: neural correlates of motor intention and imagery. Behav. Brain Sci. 17, (1994). 33. Kosslyn, S. M., Thompson, W. L., Wraga, M. & Alpert, N. M. Imagining rotation by endogenous versus exogenous forces: distinct neural mechanisms. Neuroreport 12, (2001). 34. Ganis, G., Keenan, J. P., Kosslyn, S. M. & Pascual-Leone, A. Transcranial magnetic stimulation of primary motor cortex affects mental rotation. Cereb. Cortex 10, (2000). 35. Driskell, J., Copper, C. & Moran, A. Does mental practice enhance performance? J. Appl. Psychol. 79, (1994). 36. Maring, J. R. Effects of mental practice on rate of skill acquisition. Phys. Ther. 70, (1990). 37. Weiss, T., Hansen, E., Rost, R. & Beyer, L. Mental practice of motor skills used in poststroke rehabilitation has own effects on central nervous activation. Int. J. Neurosci. 78, (1994). 38. Pylyshyn, Z. W. Psychological explanations and knowledgedependent processes. Cognition 10, (1981). 39. Thompson, W. L. & Kosslyn, S. M. in Brain Mapping II: the Systems (eds Toga, A. W. & Mazziotta, J. C.) (Academic, San Diego, 2000). This meta-analysis and review shows that different aspects of imagery tasks predict activation in early visual cortex (for example, the need to extract highresolution details), posterior parietal cortex (used in spatial tasks) and inferotemporal cortex (used in imagery tasks that do not require the extraction of high-resolution details). 40. Kosslyn, S. M., Thompson, W. L., Kim, I. J. & Alpert, N. M. Topographical representations of mental images in primary visual cortex. Nature 378, (1995). This study provides evidence that imagery not only activates early visual cortex, but activates such cortex selectively, depending on the properties of the visualized object; specifically, objects that subtend different visual angles activate different parts of cortex, as is found in perception. 41. Sereno, M. I. et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 268, (1995). This paper is a landmark in the development of techniques to delineate topographically organized areas in the human brain; such techniques will no doubt prove to be increasingly important in the study of mental imagery. 42. Tootell, R. B. H., Hadjikani, N. K., Mendola, J. D., Marrett, S. & Dale, A. M. From retinotopy to recognition: fmri in human visual cortex. Trends Cogn. Sci. 2, (1998). 43. Klein, I., Paradis, A.-L., Poline, J.-B., Kosslyn, S. M. & Le Bihan, D. Transient activity in human calcarine cortex during visual imagery. J. Cogn. Neurosci. 12, (2000). 44. Farah, M. J., Soso, M. J. & Dasheiff, R. M. Visual angle of the mind s eye before and after unilateral occipital lobectomy. J. Exp. Psychol. Hum. Percept. Perform. 18, (1992). 45. Kosslyn, S. M., Thompson, W. L., Kim, I. J., Rauch, S. L. & Alpert, N. M. Individual differences in cerebral blood flow in area 17 predict the time to evaluate visualized letters. J. Cogn. Neurosci. 8, (1996). 46. Mellet, E. et al. Neural correlates of topographic mental exploration: the impact of route versus survey perspective learning. Neuroimage 12, (2000). 47. Mellet, E. et al. Functional anatomy of spatial mental imagery generated from verbal instructions. J. Neurosci. 16, (1996). 48. Mellet, E., Tzourio N., Denis, M. & Mazoyer, B. A positron emission tomography study of visual and mental spatial exploration. J. Cogn. Neurosci. 4, (1995). 49. Crick, F. & Koch, C. Are we aware of neural activity in primary visual cortex? Nature 375, (1995). 50. Skinner, B. F. Why I am not a cognitive psychologist. Behaviorism 5, 1 10 (1977). 51. Lang, P. J., Greenwald, M. K., Bradley, M. M. & Hamm, A. O. Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30, (1993). 52. Kosslyn, S. M. et al. Neural effects of visualizing and perceiving aversive stimuli: a PET investigation. Neuroreport 7, (1996). 53. Kreiman, G., Koch, C. & Fried, I. Imagery neurons in the human brain. Nature 408, (2000). 54. LeDoux, J. E. Emotion: clues from the brain. Annu. Rev. Psychol. 46, (1995). 55. LeDoux, J. E. The Emotional Brain: the Mysterious Underpinnings of Emotional Life (Simon & Schuster, New York, 1996.) 56. Obrig, H. et al. Near-infrared spectroscopy: does it function in functional activation studies of the adult brain? Int. J. Psychophysiol. 35, (2000). 57. Rizzolatti, G., Luppino, G. & Matelli, M. The organization of the cortical motor system: new concepts. Electroencephalogr. Clin. Neurophysiol. 106, (1998). 58. Hari, R. et al. Activation of human primary motor cortex during action observation: a neuromagnetic study. Proc. Natl Acad. Sci. USA 95, (1998). 59. Grafton, S. T., Arbib, M. A., Fadiga, L. & Rizzolatti, G. Localization of grasp representations in humans by positron emission tomography. 2. Observation compared with imagination. Exp. Brain Res. 112, (1996). 60. Rizzolatti, G. et al. Localization of grasp representations in humans by PET: 1. Observation versus execution. Exp. Brain Res. 111, (1996). 61. Fadiga, L., Fogassi, L., Pavesi, G. & Rizzolatti, G. Motor facilitation during action observation: a magnetic stimulation study. J. Neurophysiol. 73, (1995). 62. Gangitano, M., Mottaghy, F. M. & Pascual-Leone, A. Phasespecific modulation of cortical motor output during movement observation. Neuroreport 12, (2001). 63. Rizzolatti, G., Fogassi, L. & Gallese, V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Rev. Neurosci. 2, (2001). 642 SEPTEMBER 2001 VOLUME 2