On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies

Transcription

1 Exp Brain Res (2005) 166: DOI /s RESEARCH ARTICLE Paul J. Laurienti Æ Thomas J. Perrault Terrence R. Stanford Æ Mark T. Wallace Barry E. Stein On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies Received: 6 August 2004 / Accepted: 26 October 2004 / Published online: 30 June 2005 Ó Springer-Verlag 2005 Abstract A growing number of brain imaging studies are being undertaken in order to better understand the contributions of multisensory processes to human behavior and perception. Many of these studies are designed on the basis of the physiological findings from single neurons in animal models, which have shown that multisensory neurons have the capacity for integrating their different sensory inputs and give rise to a product that differs significantly from either of the unisensory responses. At certain points these multisensory interactions can be superadditive, resulting in a neural response that exceeds the sum of the unisensory responses. Because of the difficulties inherent in interpreting the results of imaging large neuronal populations, superadditivity has been put forth as a stringent criterion for identifying potential sites of multisensory integration. In the present manuscript we discuss issues related to using the superadditive model in human brain imaging studies, focusing on population responses to multisensory stimuli and the relationship between single neuron measures and functional brain imaging measures. We suggest that the results of brain imaging studies be interpreted with caution in regards to multisensory integration. Future directions for imaging multisensory integration are discussed in light of the ideas presented. P. J. Laurienti (&) Department of Radiology, Medical Center Boulevard, Wake Forest University School of Medicine, Winston-Salem, NC 27157, USA plaurien@wfubmc.edu Tel.: Fax: T. J. Perrault Department of Neural and Behavioral Sciences, 500 University Drive, Penn State Univeristy College of Medicine, Hershey, PA 17033, USA T. R. Stanford Æ M. T. Wallace Æ B. E. Stein Department of Neurobiology and Anatomy, Medical Center Boulevard, Wake Forest University School of Medicine, Winston-Salem, NC 27157, USA Keywords Multisensory Æ Superadditivity Æ BOLD Æ Imaging Æ FMRI Æ Integration Introduction Information from the different sensory modalities converges on many sites in the nervous system. By nature of receiving such convergent inputs, these sites have the potential for information in one sense (e.g., vision) to shape how information is processed in a different sense (e.g., audition), an alteration that has been collectively referred to as multisensory integration (see Stein and Meredith 1993). A great deal of work has gone into elucidating the manner in which individual neurons integrate multisensory cues, and into relating the products of this integration to observable behavioral and perceptual processes. These studies, which have been carried out in animal models, have shed substantial light on the integrative operations performed by individual multisensory neurons. The ultimate goal of this research is to extend the knowledge base derived from these studies to humans, with the objective being to identify the brain substrates involved in multisensory perceptual processes. Toward this end, recent advances in non-invasive neuroimaging technologies have enabled the first steps to be taken. However, despite the obvious importance of these studies for elucidating the neural bases of multisensory perceptual processes, at present little consensus exists as to the appropriate criteria for identifying sites of multisensory integration in the human cerebral cortex. There are several reasons for this lack of consensus, and these largely revolve around the nature of the signal that can be recorded in functional brain imaging studies. Regardless of whether the technique involved is functional magnetic resonance imaging (fmri), positron emission tomography (PET), magnetoencephalography (MEG), or event-related potentials (ERP), the signals recorded represent the composite activity from large

2 290 neuronal ensembles. Although such measures are undoubtedly advantageous in that they reflect information processing within large neuronal networks, they also pose serious problems in identifying the brain regions involved in both the convergence and integration of multisensory information (Calvert 2001). Recent advances in our understanding of the physiology underlying the fmri signal (Logothetis et al. 2001; Logothetis 2003), as well as new insights into the integrative operations performed by individual multisensory neurons and the prevalence of these operations in a model multisensory structure (Meredith and Stein 1986; Wallace et al. 1993; Nozawa et al. 1997; Quessy et al. 2000; Jiang et al. 2001; Perrault et al. 2003; Stein et al. 2004), suggest that a reexamination of this important issue is in order. In this manuscript we will review current methods used to identify multisensory interactions in human brain imaging studies. In addition, we will consider how methods used to identify multisensory responses using imaging techniques compare to what is known about single neuron activity, as well as discussing the validity of these methods in relation to what is known about the physiology underlying neuroimaging approaches. Although each of the human brain imaging methodologies that have been employed in multisensory studies are likely measuring signals related to the same neurophysiological processes, the primary focus will be on the blood oxygenation level dependent (BOLD) signal measured with fmri with some reference to previous findings using other human brain imaging techniques. Imaging multisensory integration Commonly used methods to identify multisensory brain regions To date there have been a considerable number of human brain imaging studies dedicated to identifying the brain regions involved in multisensory processing (for review see Calvert 2001). Most of the studies have been predicated on electrophysiological studies in animal models looking at the responses of individual multisensory neurons to the presentation of cues from different sensory modalities (Meredith and Stein 1983, 1986; King and Palmer 1985; Stein and Meredith 1993; Benedek et al. 1996; Wallace et al. 1996, 1998, 2004; Jiang et al. 2001; Populin and Yin 2002). Typical outcomes for such combinations include response enhancement, in which the multisensory response exceeds either of the unisensory responses, and response depression, in which the multisensory response is less than the better of the two unisensory responses. These enhancements and depressions of activity have been found to be critically dependent on the spatial and temporal relationships of the combined stimuli, as well as on their effectiveness in inducing a response. For example, response enhancements are the typical result when stimuli that are weakly effective and spatially coincident and temporally coincident are combined (Meredith and Stein 1986, 1996; Meredith et al. 1987, 1992). Recent work has extended these findings to relate the multisensory response to that predicted based on an additive model (Nozawa et al. 1997; Quessy et al. 2000; Perrault et al. 2005; Stein et al. 2004), and has shown that these neurons can produce interactions that exceed this prediction (i.e., superadditive), meet the prediction (i.e., additive) or fall short of the summative prediction (i.e., subadditive). As a consequence of these observations, most imaging studies have been structured with the hypothesis that multisensory brain regions will exhibit enhanced responses to spatially congruent and temporally congruent stimuli and depressed responses to incongruent stimuli. In support of this general framework are data from several brain imaging laboratories (Giard and Peronnet 1999; Calvert et al. 2000, 2001; Macaluso et al. 2000; Bushara et al. 2001; Klucharev et al. 2003; Wright et al. 2003; Molholm et al. 2004). However, although it seems quite sensible to base human imaging studies on findings from animal physiology, it must be remembered that the signal being measured in imaging studies comes not from a single neuron, but rather from a large neuronal population. This population is undoubtedly heterogeneous, most notably in regards to which stimuli will be effective in inducing a response. Thus, in addition to multisensory neurons that receive convergent inputs, it is very likely that the population will also be comprised of many elements that respond exclusively to a single sensory modality. Consequently, the nature of the response to multisensory stimulation in such a large population is ambiguous, in that it could be the result of activation of multisensory elements, independent activation of each of the unisensory elements, or some combination of these two. In short, the finding of enhanced activation in response to multisensory stimulation, even for a single voxel (which is comprised of many thousands of neurons), fails to provide proof of convergence because an enhanced response to two unisensory stimuli might simply reflect the activation of two independent populations of neurons and not the convergence of these inputs onto a common neuronal pool. The inescapable logic of this conclusion has led to the search for more rigorous methods for assessing putative sites of multisensory processing. In reviewing the existing imaging literature, Calvert (2001) noted the following as criteria that have been used as evidence for multisensory integration in human cortex: 1. An area that responds to stimulation in two different sensory modalities (e.g., A \ V). 2. An area that responds more to the multisensory stimulus (AV) than to either of the unisensory stimuli [(AV A) \ (AV V)]. 3. An area that is active under conditions of multisensory stimulation (e.g., AV) but that is not active to either unisensory stimulus.

3 An area that shows greater activation for congruent multisensory stimulation when compared with incongruent stimulation. Using these criteria a number of multisensory cortical regions have been identified using various imaging techniques. These not only include higher-order areas such as the intraparietal sulcus (Lewis et al. 2000; Macaluso et al. 2000; Calvert et al. 2001; Gobbele et al. 2003), superior temporal sulcus (Raij et al. 2000; Macaluso et al. 2001, 2004; Dieterich et al. 2003; Wright et al. 2003; Beauchamp et al. 2004), claustrum/insula (Hadjikhani and Roland 1998; Banati et al. 2000; Calvert 2001), and anterior cingulate cortex (Banati et al. 2000; Calvert 2001; de Zubicaray et al. 2002; Laurienti et al. 2003), but also regions typically considered to be involved only in unisensory processing (Calvert et al. 1997, 1999; Giard and Peronnet 1999; Foxe et al. 2000; Lewis et al. 2000; Macaluso et al. 2000; Shams et al. 2001; Molholm et al. 2004). However, as noted above, meeting these criteria does not constitute de facto proof that the area is involved in the integration of multisensory information. A brain region that responds to both unisensory stimulus conditions could simply be comprised of two separate pools of unisensory neurons (criterion 1). Similarly, an area could show a greater signal under multisensory conditions than under unisensory conditions (criterion 2) because this signal reflects contributions from both unisensory populations (Fig. 1). The meeting of criterion 3 could be the result of a simple thresholding phenomenon in which only when the two unisensory pools are active does the signal rise above background (i.e., the signal in the absence of sensory stimulation). Finally, a multisensory brain region identified using criterion 4 could be sensitive to stimulus congruence regardless of the sensory modality (ies) of stimulation. Each of the criteria presented above are highly susceptible to type I (false positive) errors, whereby regions could be inappropriately identified as multisensory even if they only contain pools of unisensory neurons. unisensory BOLD signals, then it cannot simply be a reflection of the sum of two independent unisensory responses. In this scenario, the superadditive nature of the BOLD signal is interpreted to reflect the integration of multisensory signals by multisensory neurons. It is Superadditivity in imaging studies To limit the possibility of false positives in neuroimaging data, the metric of superadditivity, in which the response to combined stimulation must be greater than that predicted based on a summation of the unisensory responses (i.e., AV > A+V), was proposed for the identification of brain regions actively engaged in multisensory integration (Calvert et al. 2000, 2001; Calvert 2001). Using the conservative criterion of superadditivity, Calvert and colleagues have identified multisensory integration in several brain regions including the superior temporal sulcus and the superior colliculus (Calvert et al. 2001). The rationale for the use of superadditivity as a more rigorous criterion is as follows: if the multisensory BOLD signal is greater than the sum of the Fig. 1 The nature of the BOLD signal in multisensory studies. a The upper portion of the diagram depicts example BOLD signal changes during periods of visual (vis), auditory (aud), and multisensory (ms) stimulation. The stimulus duration is indicated by the colored blocks in the lower portion of the figure. Note that the amplitude of the BOLD response to multisensory stimulation is larger than to either visual or auditory stimulation alone. However, the BOLD signal does not exceed the additive model. b Illustration depicting a single voxel of brain tissue (cube) containing visual (blue), auditory (red), and multisensory (yellow) neurons. Such a mixture of unisensory and multisensory neurons could account for the enhanced BOLD signal shown for multisensory stimulation. c Illustration of a voxel containing only visual (blue) and auditory (red) neurons. Note that the unisensory composition of this voxel could also account for the enhanced BOLD response seen during multisensory stimulation by a simple summation of the BOLD response from the two-neuron populations

4 292 important to point out here that although superadditive interactions are indeed seen in multisensory neurons (Meredith and Stein 1983, 1986; Meredith et al. 1987; Wallace et al. 1992, 1996, 1998), many neurons perform additive or even subadditive operations when exposed to stimuli from multiple sensory modalities (Perrault et al. 2005). These interactions still represent multisensory integration, in that the multisensory response differs from the better of the two unisensory responses. Population model of superadditivity Population impulse calculations In an effort to gain a better handle on the meaning of superadditive interactions in neuroimaging data, we set out to construct a simple model of how multisensory interactions at the level of the single neuron might translate into events that can be seen using neuroimaging (specifically fmri). For our data set we used the unisensory and multisensory response profiles of neurons in the best-characterized model of multisensory interactions the superior colliculus (SC see Meredith and Stein 1983, 1986; Meredith et al. 1987; Wallace et al. 1996, 1998; Perrault et al. 2003, 2005). The SC data was used for two reasons. First, the SC contains one of the largest proportionate representations of multisensory neurons, with anywhere from 25 to 60% of its total population being multisensory. Second, the responses of multisensory SC neurons have been extremely well characterized, with substantial analyses focusing on the integrative operations of these neurons. Although subcortical data may seem less than ideal for calculating the consequences of multisensory integration on the cortical BOLD signal, several additional observations serve to strengthen the use of this dataset. First, and as alluded to above, the SC is likely to have a higher proportionate representation of multisensory neurons than cortex, meaning that estimates based on it will likely provide a best case scenario for cortex. A multisensory area of the cat cortex, the anterior ectosylvian sulcus (AES), is made up of approximately 25% multisensory neurons (Wallace et al. 1992). Second, multisensory neurons in the SC and in cortex appear to be very similar in their response profiles and in their multisensory interactions, with evidence suggesting that these multisensory neurons, regardless of species or brain structure, abide by a very similar set of integrative principles (Wallace and Stein 1996). Furthermore, although no data exists detailing the responses of single multisensory neurons in the human brain, we are reasonably confident that these responses will not deviate dramatically from those recorded in other mammals. The model was based on several assumptions as well as on established data concerning multisensory representations. Population response estimates were calculated for a voxel that is typical size for imaging studies (4 mm 3 ). Based on data from Goldman-Rakic and colleagues (Goldman-Rakic 1995; Selemon et al. 1998), we estimated that each voxel contains approximately 2.5 million neurons. We started with the conservative estimate that 25% of the neurons in this voxel are multisensory (i.e., 625,000 neurons), a number in keeping both with data from the monkey SC (Wallace et al. 1996) and the cat AES (Wallace et al. 1992). In order to simplify the model, we assumed that the remaining sensory-responsive neurons were equally distributed as unisensory visual and unisensory auditory neurons (i.e., 37.5% or 937,500 neurons each). One caveat that should be mentioned is that these population estimates are derived from areas that have three sensory representations (visual, auditory and somatosensory) and generalized to areas (presumably) with two sensory representations. Based on recent data from cat SC (Perrault et al. 2005), we assume that given a readily detectible multisensory stimulus (such as might be presented in the context of an imaging study), 28% of the multisensory neurons (i.e., 175,000 neurons) will exhibit superadditive interactions, with the remaining neurons exhibiting additive or subadditive interactions. Again relying on the SC data set, we generated total spike count estimates for the response to a standard stimulus set (Table 1). These spike count estimates have several embedded assumptions. First, the data represent the averaged responses of a large sample of multisensory neurons to a variety of different stimulus pairings along the intensity continuum. As such, we feel that it is likely to closely reflect the population response to an intermediate intensity stimulus. However, non-linearities in the stimulus-response functions as well as non-equivalent sampling could skew this population response estimate, possibilities that we feel are unlikely to be major contributors to the final product. Second, the model assumes that all of the appropriate elements will be active to the standard stimulus. Since SC neurons are heterogeneous in their response thresholds, it remains possible that certain elements will be unresponsive to this stimulus. Again, given the size of the sample and the fact that the vast majority of SC neurons typically show robust responses to modest intensity stimuli, we feel that unresponsive neurons will have a negligible impact on the population estimate. In examining the sample data set (Table 1), it is readily apparent that superadditive neurons have impulse counts that are less than half those observed in the additive/subadditive population. Consequently, a BOLD response that is driven by the sum of activity of all neurons will be less affected by superadditive neurons with low impulse counts than by additive/subadditive multisensory neurons with higher impulse counts. The preceding numerical example is modeled on neural activity that might be evoked by a typical multisensory stimulus. However, we note that the relative proportion of neurons displaying superadditivity could be much greater for very weak (i.e., near threshold) stimuli, or much smaller for highly salient stimuli (Perrault et al.

5 Table 1 Mean impulse (action potential) counts for typical unisensory and multisensory neurons in response to visual, auditory, and multisensory stimulation Neuron population 293 Multisensory Unisensory Superadditive Non-superadditive Visual Auditory Population numbers 175, , , ,500 Stimulus condition Visual Auditory Multisensor These mean values are based on data collected from 87 multisensory neurons in the cat superior colliculus (Perrault et al. 2005). The total cell count for a mm volume of cortical brain tissue was estimated (population numbers). This total cell count (2.5 million neurons) was then divided into each population category based on the proportion cells expected in a multisensory brain region. The population response was calculated by multiplying the mean impulse number for each stimulus condition by the number of neurons for each population. For example, the response of the superadditive multisensory neuron pool to a visual stimulus = 2.7 impulses 175,000 neurons = 472,500 impulses). Using the data and assumptions described above, we arrive at the following as the total impulse output for our representative voxel in response to a standard stimulus set: total visual response = 12,067,500 impulses [i.e., (937,500 visual neurons 8 spikes/neuron) + (450,000 nonsuperadditive multisensory neurons 9.1 spikes/neuron) + (175,000 superadditive multisensory neurons 2.7 spikes/neuron)]; total auditory response = 10,677,500 impulses; total multisensory response = 20,642,500 impulses 2005). In fact, recent studies have adopted this strategy in the search for sites of multisensory processing in cortex (Callan et al. 2003, 2004). Population BOLD calculations To estimate the BOLD responses that would be generated by these impulse counts the response to the standard visual stimulus was set to be a 2% BOLD signal change. This is an arbitrary value, but one that represents a typical signal change in response to visual stimulation from our experience and that of other investigators (Lewis et al. 2000; Laurienti et al. 2002; Beauchamp et al. 2004; Soltysik et al. 2004). From this predetermined visual response, responses to the auditory and multisensory stimulus conditions were calculated (Fig. 2b). Assuming that the auditory and multisensory responses map similarly onto the BOLD signal (Beauchamp et al. 2004; Soltysik et al. 2004), the model predicts an auditory signal change of 1.77% and a multisensory signal change of 3.42%. Note that the multisensory response does not exceed the sum of the unisensory responses (3.77%), but rather is slightly subadditive. When the model is further simplified by assuming that the voxel is comprised only of multisensory neurons (with the same proportions of superadditive and non-superadditive neurons), the population response diverges even further from the superadditive prediction (2% visual, 2.2% auditory, and 3.29% multisensory). On reflection, it can be seen that the lack of a superadditive BOLD response is due to several factors: multisensory neurons are out numbered by 2:1 by unisensory neurons, superadditive neurons exhibit low impulse counts, make up a relatively small portion of the total multisensory population, and the remainder of the neurons respond in a subadditive manner. Only if multisen- Fig. 2 Predicted unisensory and multisensory responses at the level of both the neural signal (a) and the BOLD signal (b). a Predicted total number of impulses generated per trial from all neurons contained within the model voxel. The sum of the visual (vis) and auditory (aud) responses is depicted as the additive response (add). Note that the modeled multisensory response (ms) is less than the additive response. b BOLD response predicted from the population made up of 25% multisensory neurons. The BOLD response to the visual stimulus is set at 2% (vis). The auditory stimulus generates a 1.77% BOLD signal change. Note again that the modeled multisensory response is less than the additive prediction

6 294 sory neurons were to make up a substantial portion of the active population, were largely superadditive, and exhibited large spike counts would a superadditive BOLD signal be possible from changes in spiking activity. However, as should be evident from above, when constrained by the physiological characteristics of known multisensory representations, the model suggests instead that multisensory stimuli would be very unlikely to give rise to superadditive BOLD signal changes. Interestingly, as has been described earlier, superadditive interactions have been seen in the BOLD signal, raising the question as to the nature of these interactions. Single neuron activity and the BOLD signal The neural processes that underlie the BOLD response have been aggressively sought since the first demonstration that visual stimulation can produce a measurable change in blood oxygenation levels in visual cortex (Ogawa et al. 1990). Initially, this change was attributed to changes in the number of action potentials in the area of interest (Heeger et al. 2000; Rees et al. 2000). However, this interpretation was called into question when both electrophysiological measures and the BOLD signal could be measured in the same subjects. In a landmark study, Logothetis et al. (2001) performed extracellular electrophysiological recordings and fmri simultaneously on the same non-human primate subjects. In this work, they found that the BOLD signal was better correlated with local field potentials (LFP), a measure of local synaptic activity, than with action potentials. Consequently, it was concluded that the BOLD signal is a better reflection of input to a given area (i.e., synaptic events) as opposed to output from that area (i.e., action potentials). This conclusion has been further bolstered by the recent demonstration that blocking action potentials, but not LFPs, in visual cortex has little effect on the BOLD response (Logothetis 2003). Further support has come from work by Lauritzen and colleagues, who have recorded LFPs and cerebral blood flow (CBF) in the rat cerebellar cortex (for review see Lauritzen and Gold 2003). These studies have provided critical support for the relationship between LFP and regional changes in blood flow. Specifically, Lauritzen and colleagues demonstrated that CBF increases can occur in the absence of action potentials (Mathiesen et al. 1998). Stimulation of parallel fiber pathway in the cerebellum decreases or completely inhibits Purkinje cell action potentials but increases the LFP and regional CBF, supporting the concept that the BOLD signal represents input and local processing rather than output from a specified area. Future directions for imaging multisensory integration The concepts presented here suggest that we must be very careful in relating the results of single unit physiology studies to human brain imaging data. If superadditivity is an unlikely outcome in fmri studies designed to look at multisensory interactions like those described in animal studies, why is it frequently seen? One distinct possibility is that because it better reflects changes in synaptic processes rather than changes in action potentials, there may indeed be superadditive interactions at the level of the LFP during multisensory conditions. Possible mechanisms for these non-linearities include changes in synaptic frequency (Gonzalez- Burgos et al. 2004), synchronization of synaptic inputs (Margulis and Tang 1998), or changes in the membrane potential of the postsynaptic neuron (Binder 2002). In fact, these non-linear changes in synaptic processes may well translate into non-linear interactions in the spiking activity of the neurons of interest. However, and as should be clear from the above, since they index different neural processes, one cannot assume a one-to-one relationship between the BOLD signal and action potentials. Although the relevance of superadditive interactions in synaptic responses remains unclear, it represents a potential new form of multisensory interaction not previously described. A configuration can be readily envisioned in which the proper convergence of inputs from different modalities results in a non-linear amplification of the postsynaptic potential. This amplificatory process may be dependent on the spatial pattern of these inputs onto dendritic elements and on their temporal relationship to one another. The validity of this model of synaptic multisensory interactions, as well as its effects on the firing patterns of the target neurons, await studies in which the different sensory inputs can be selectively controlled. One of the primary issues to be considered is to determine what neural process we are trying to identify, and whether fmri is the appropriate tool for identifying it. Take as an example the McGurk effect (McGurk and MacDonald 1976), a perceptual illusion in which the pairing of the sound/ba/with the sight of the lips mouthing the syllable/ga/typically results in the synthetic percept of/da/. Would one expect that the multisensory signal generated under the synthesis conditions to be sufficiently different so as to be resolvable using fmri? Possibly, but in the absence of a difference is it reasonable to say that a given brain region is not involved in the multisensory processes that result in this perceptual synthesis? Obviously not. In fact, based upon the arguments outlined previously, it seems highly unlikely that using current imaging methodologies we will be able to definitively prove convergence of unisensory inputs onto multisensory neurons. It is then incumbent upon us to develop different but complimentary methods by which we can identify brain regions involved in multisensory integration. One methodological change that can be readily implemented now is in the design of the stimulation paradigms used to examine multisensory processes. Currently, most studies use unisensory conditions, a congruent multisensory condition, and frequently some

7 295 form of incongruent multisensory condition. Often these paradigms are passive, in that they necessitate no response from the subject. The addition of a behavioral response to the design (with appropriate controls) has the advantage of providing a window into the subject s behavior and/or perception during unisensory and multisensory stimulation. Using the McGurk example described above, one can identify trials in which the subject perceives the illusory/da/from trials in which they do not, and relate the brain activation patterns to their different perceptual correlates. In fact, this idea has been presented previously (Calvert 2001) and recent multisensory studies have manipulated the sensory stimuli to modify the likelihood of perceptual fusion of combined sensory cues (Callan et al. 2003, 2004; Hayes et al. 2003). The use of a McGurk illusion with stimuli that have varying probabilities of producing a fused percept, such as reported by Hayes et al. (2003), could allow for the identification of brain regions that show similar probabilities of activation. The combination of imaging and behavior may provide a deeper understanding of the relationship between the measured neural processes (typically LFPs) involved in multisensory integration and the associated behavioral or perceptual responses. Recent electrophysiological studies in non-human primates have demonstrated that LFP changes are predictive of sensory perception and/or behavior (Gail et al. 2004) suggesting that imaging measures of synaptic activity may offer a unique view of multisensory processing. Superadditive BOLD responses may very well represent neural integration (not necessarily as indexed by single unit recordings) and the behavioral correlates of these responses will aid greatly in the interpretation of brain imaging data. The use of clinical populations with alterations in some aspects of multisensory integration can also be extremely valuable. If a population exhibits behavioral alterations in multisensory processing then neural activation patterns can be compared between patients and control subjects. Brain regions that differ in their activation patterns between the study populations are likely involved in the processing multisensory stimuli. This type of study design is complicated by multiple changes in neural processing that can occur in patient populations, and one must determine if an identified brain region is actually necessary or sufficient for integration. Nevertheless, the use of clinical populations can add to the battery of study designs available to the imaging scientist investigating multisensory integration. Conclusions The rapid increase in the number of brain imaging studies dedicated to elucidating the neural underpinnings of multisensory integration warrants careful consideration in terms of how to interpret the data. Most importantly, study designs based upon single unit electrophysiology in animal models may not be readily transferable to the human imaging realm. Specifically in this regard, the use of superadditivity as a metric for identifying multisensory integration in fmri is unlikely to map directly onto changes in neuronal activation patterns. We have suggested several study design possibilities and recommend that the interpretation of superadditive interactions in imaging studies be treated with caution. Although we have focused here on increased neural responses and increases in the BOLD signal it should be noted that multisensory-mediated performance enhancements have been associated with decreases in these indices as well (Raij et al. 2000; Schurmann et al. 2002; Gobbele et al. 2003; Klucharev et al. 2003; Wright et al. 2003). Although the arguments detailed above focus on the BOLD signal, it is likely that the concepts outlined apply to all currently available functional brain imaging techniques. Specifically, our model of multisensory integration suggests that superadditivity at the level of neuronal activity (i.e., action potentials) would not be readily identified using other population-based techniques (e.g., PET, MEG, ERP, etc.), in part because the number of neurons capable of producing such responses would not be resolvable at a population level, and more importantly because it is probable that all of these techniques measure processes occurring at the synaptic level. Despite the many caveats presented here, human brain imaging techniques remain valuable tools for studying multisensory processing. In fact, there are many advantages to using these techniques, most notably the ability to study neurophysiological processes in the human brain and to relate these processes to ongoing behavioral and perceptual events. With careful consideration in experimental design, greater knowledge of the relationship of the neuroimaging signal to the underlying neurophysiological processes, and cautious interpretation of the data, great strides will be made in increasing our understanding of how multisensory interactions contribute to the formation of our perceptual gestalt. Acknowledgments Supported in part by Grants NS (PJL), MH (MTW), and NS (BES). References Banati RB, Goerres GW, Tjoa C, Aggleton JP, Grasby P (2000) The functional anatomy of visual-tactile integration in man: a study using positron emission tomography. Neuropsychologia 38: Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41: Benedek G, Fischer-Szatmari L, Kovacs G, Perenyi J, Katoh YY (1996) Visual, somatosensory and auditory modality properties along the feline suprageniculate-anterior ectosylvian sulcus/ insular pathway. Prog Brain Res 112: Binder MD (2002) Integration of synaptic and intrinsic dendritic currents in cat spinal motoneurons. Brain Res Rev 40:1 8

8 296 Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21: Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis- Bateson E (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14: Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis- Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16: Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11: Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997) Activation of auditory cortex during silent lipreading. Science 276: Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10: Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10: Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14: Dieterich M, Bense S, Stephan T, Yousry TA, Brandt T (2003) fmri signal increases and decreases in cortical areas during small-field optokinetic stimulation and central fixation. Exp Brain Res 148: Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77 83 Gail A, Brinksmeyer HJ, Eckhorn R (2004) Perception-related modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry. Cereb Cortex 14: Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11: Gobbele R, Schurmann M, Forss N, Juottonen K, Buchner H, Hari R (2003) Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction. Neuroimage 20: Goldman-Rakic P (1995) Architecture of the prefrontal cortex and the central executive. In: Grafman J, Holyoak K, Boller F (eds) Structure and functions of the human prefrontal cortex. The New York Academy of Sciences, NY USA, pp Gonzalez-Burgos G, Krimer LS, Urban NN, Barrionuevo G, Lewis DA (2004) Synaptic efficacy during repetitive activation of excitatory inputs in primate dorsolateral prefrontal cortex. Cereb Cortex 14: Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18: Hayes EA, Tiippana K, Nicol TG, Sams M, Kraus N (2003) Integration of heard and seen speech: a factor in learning disabilities in children. Neurosci Lett 351:46 50 Heeger DJ, Huk AC, Geisler WS, Albrecht DG (2000) Spikes versus BOLD: what does neuroimaging tell us about neuronal activity?. Nat Neurosci 3: Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85: King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60: Klucharev V, Mottonen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65 75 Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE (2002a) Deactivation of sensory-specific cortex by crossmodal stimuli. J Cogn Neurosci 14: Laurienti PJ, Field AS, Burdette JH, Maldjian JA, Yen YF, Moody DM (2002b) Dietary caffeine consumption modulates fmri measures. Neuroimage 17: Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19: Lauritzen M, Gold L (2003) Brain function and neurophysiological correlates of signals used in functional neuroimaging. J Neurosci 23: Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cereb Cortex 10: Logothetis NK (2003a) The underpinnings of the BOLD functional magnetic resonance imaging signal. J Neurosci 23: Logothetis NK (2003b) MR imaging in the non-human primate: studies of function and of dynamic connectivity. Curr Opin Neurobiol 13: Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fmri signal. Nature 412: Macaluso E, Frith C, Driver J (2000a) Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. J Neurophysiol 83: Macaluso E, Frith CD, Driver J (2000b) Modulation of human visual cortex by crossmodal spatial attention. Science 289: Macaluso E, Frith CD, Driver J (2001) Multimodal mechanisms of attention related to rates of spatial shifting in vision and touch. Exp Brain Res 137: Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21: Margulis M, Tang C-M (1998) Temporal integration can readily switch between sublinear and supralinear summation. J Neurophysiol 79: Mathiesen C, Caesar K, Akgoren N, Lauritzen M (1998) Modification of activity-dependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex. J Physiol 512(Pt 2): McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264: Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221: Meredith MA, Stein BE (1986a) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56: Meredith MA, Stein BE (1986b) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365: Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75: Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I Temporal factors. J Neurosci 7: Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulospinal projection. Exp Brain Res 88: Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:

9 297 Nozawa G, Stanford TR, Vaughan JW, Quessy S, Kadunce D, Stein BE (1997) A factorial approach to modeling multisensory integration in the superior colliculus. Soc Neurosci Abstr 23:451 Ogawa S, Lee TM, Kay AR, Tank DW (1990) Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci USA 87: Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90: Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93: Populin LC, Yin TC (2002) Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22: Quessy S, Sweatt A, Stein BE, Stanford TR (2000) The influence of stimulus intensity and timing on the responses of multisensory neurons in the superior colliculus: comparison to a model s prediction. Soc Neurosci Abstr 26:1221 Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28: Rees G, Friston K, Koch C (2000) A direct quantitative relationship between the functional properties of human and macaque V5. Nat Neurosci 3: Schurmann M, Kolev V, Menzel K, Yordanova J (2002) Spatial coincidence modulates interaction between visual and somatosensory evoked potentials. Neuroreport 13: Selemon LD, Rajkowska G, Goldman-Rakic PS (1998) Elevated neuronal density in prefrontal area 46 in brains from schizophrenic patients: application of a three-dimensional, stereologic counting method. J Comp Neurol 392: Shams L, Kamitani Y, Thompson S, Shimojo S (2001) Sound alters visual evoked potentials in humans. Neuroreport 12: Soltysik DA, Peck KK, White KD, Crosson B, Briggs RW (2004) Comparison of hemodynamic response nonlinearity across primary cortical areas. Neuroimage 22: Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA Stein BE, Jiang W, Stanford TR (2004a) Multisensory integration in single neurons in midbrain and cortex. In: Calvert G, Spence C, Stein BE (eds) A handbook of multisensory processes. MIT Press, Cambridge, MA, pp Stein BE, Stanford TR, Wallace MT, Vaughan JW, Jiang W (2004b) Cross-modal spatial interactions in subcortical and cortical circuits. In: Driver J (ed) Crossmodal and crossmodal attention. Oxford University Press, Oxford, pp Wallace MT, Stein BE (1996) Sensory organization of the superior colliculus in cat and monkey. Prog Brain Res 112: Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91: Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol 69: Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76: Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80: Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci USA 101: Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 13: de Zubicaray GI, McMahon KL, Eastburn MM, Wilson SJ (2002) Orthographic/phonological facilitation of naming responses in the picture-word task: an event-related fmri study using overt vocal responding. NeuroImage 16: