An Account of the Neural Basis and Functional Relevance of Early Cross-Modal Interactions

Transcription

1

2 Student Psychology Journal, 2013, 1-14 An Account of the Neural Basis and Functional Relevance of Early Cross-Modal Interactions Nicholas Murray Trinity College, Dublin Correspondence: Contrary to the previous body of literature, burgeoning research is indicating that cross-modal interactions can occur in brain regions previously considered unisensory and at time frames much earlier that initially thought (Ghazanfar and Schroeder, 2006). This has brought on the suggestion that much of the theoretical frameworks present for perception should be reformulated (Driver and Noesselt, 2008). This article attempts to establish how the perceptual hierarchy may be remodelled to accommodate these early cross-modal interactions, addressing the extent to which such interactions occur, their neural underpinnings and their consequences in perception. Given the vast degree of potential mediators coupled with the wide range of anatomical connections utilized by these interactions, future directions for considered multi-modal research are also discussed. Introduction Long-standing models of perception are currently being challenged in the context of an emerging body of evidence which serves to highlight the occurrence of cross-modal interactions in brain regions previously considered unisensory and at time frames much earlier stages than previously thought (Ghazanfar and Schroeder, 2006). This has prompted the suggestion that many of the present theoretical frameworks of perception should be reformulated (Driver and Noesselt, 2008). In order to assess the validity of such an assertion, it is necessary to establish how the perceptual hierarchy may be remodelled to accommodate these early cross-modal interactions. The establishment of the anatomical substrates of these early cross-modal interactions serves only as a foundation on which greater knowledge can be built. Further questions must be examined in order to provide a more comprehensive picture, such as what stimuli are capable of producing these effects and what is the neural basis for such findings. Only when these questions are

3 2 Student Psychology Journal, Volume IV MURRAY fully addressed will it be possible to integrate early cross-modal interactions into a more comprehensive model of perception. This paper seeks to examine the extent to which processes and their consequences occur in perception through a progressive overview of research to date, assessing the anatomical evidence supporting the nature of these interactions and their neural underpinnings before advancing to the functional implications of early cross-modal interactions. In considering the extent to which cross modal interactions occur early in the human brain and endeavouring to assess their impact in terms of perception, it is important to define what constitutes cross-modal interaction. This is important, particularly given the degree of semantic confusion that has occurred within the discipline (Stein et al., 2010). In addition, it must be recognised that such a definition subsumes three distinct principles which are necessary for investigation. At the level of the single neuron, multisensory integration is defined operationally as: a statistically significant difference between the number of impulses evoked by a cross-modal combination of stimuli and the number evoked by the most effective of these stimuli individually (Stein and Meredith, 1993). This illustrates an important distinction between interaction and integration: interaction occurs in circumstances where one modality might affect another without necessarily always implying a single unified percept. Additionally, it is necessary to recognise that early in this context refers to the time period of neural processes alongside the low-level cortical area in which they occur. On this basis, therefore, both temporal and spatial evidence must be considered. Burgeoning evidence appears to indicate that certain neocortical areas may be involved in such cross-modal interactions. Such research is, however, still in its infancy. In considering the extent to which cross modal interactions occur early in the human brain and attempting to assess their impact in terms of perception, it is imperative that perceptual processes involved are examined on a case by case basis, along with a careful consideration of any potential confounds or interpretative issues in data analysis. Low-Level Multisensory Effects Recent studies have indicated multisensory effects within low-level cortices and at early latencies (Ghazanfar and Schroeder, 2006). In the case of non-human primates, multisensory effects have been identified within primary and adjacent auditory cortices (Lakatos et al., 2007). ). In research involving humans, supra-additive nonlinear responses to multisensory stimuli have been observed to occur within 100ms post-stimulus onset (Giard and Peronnet, 1999) and within primary cortices (Martuzzi et al., 2007). These and similar studies indicate that

4 Literature Review NEURAL BASIS AND RELEVANCE OF CROSS-MODAL INTERACTIONS 3 neural processing in low-level cortices can be modulated by interactions between the senses at specific, early post-stimulus time periods. This structural arrangement has been suggested as being primarily due to the brain s organization, allowing for fast access from all sources (Kotter and Sommer, 2000). In this way, neurons with cross-modal responses serve as information hubs which are capable of modulating multisensory cortical recruitment on receipt of various sensory inputs (Vasconcelos et al., 2011). It would seem reasonable then to conclude that multisensory interactions may be deemed to be specific to each individual area. In evaluating the manner in which neural mechanisms mediate these early multisensory effects, it is important to consider their timing and origin and to assess whether, for example, multisensory interactions coincide with the initial feed-forward inputs in an area or with later feedback activity. The early stages of research in this area have focused on identifying links between primary cortical regions and cross-modal interactions. These findings, however, should now serve as the foundations upon which to build a greater understanding of the neural processes underlying such interactions and the overarching perceptual architecture in which they fit. The prevalent hierarchical models of sensory processing postulate that such early cross-modal interactions in primary cortical regions are modulated through feedback inputs.support for such claims is provided by findings from event-related potential (ERP) based studies which have demonstrated the occurrence of such inputs in monkeys and humans and shown their capacity to cross between sensory modalities (Ghazanfar and Schroeder, 2006). Neural Processes Modulating Interactions A growing body of research indicates, however, that these interactions can additionally be modulated via feed-forward processes. ERPs have shown time effects supporting such postulations, with somatosensory-related activation in A1 region occurring at 9 ms, while auditory activation of the same location was shown to occur at 15 ms (Lakatos, 2007). Similarly, there is evidence of auditory-driven responses at 48 ms within inferior parietal cortex (Schroeder et al., 2004) that represent a potential basis for modulation of early sensory processing within the visual cortex. Although these time effects cannot be considered as conclusive proof for feed-forward mechanisms, the short onset of such effects favours that assumption given that it is considerably unlikely that interactions could be modulated by feedback inputs within such a time frame. Spatial evidence supporting feed-forward mechanisms has been found in monkey studies, with direct lateral projections occ-

5 4 Student Psychology Journal, Volume IV MURRAY urring from primary and adjacent auditory cortices to early visual cortices (V1/V2) (Falchier et al., 2002). If account is taken of this structure, it would appear that auditory inputs are well placed to impact visual processing. Ghazanfar and Schroeder (2006) have boldly posited that the neocortex may be entirely multisensory. The validity of such a postulation rests on the degree to which interactions are due to feedback or feed-forward inputs (Alias et al., 2010). However, a simple dichotomy whereby primary regions are either multisensory or unisensory in nature may not exist. The answer may lie in the middle ground with the neocortex being bi-modal in nature, and perceptual processes influenced by both feed-forward and feedback processes. This assertion may be supported by the differing modulation patterns found in macaque auditory cortex for audio-tactile co-stimulation compared to audio-visual co-stimulation (Kayser et al., 2007). It would appear to emphasise the fact that sensory processing may not represent a serial progression but rather may be mediated by several parallel and recursive loops (Driver and Noesselt, 2007). Future studies should attempt to clarify the nature of such inputs through testing the sources for potential feedback influences from higher cortical regions to sensory-specific regions. Using an efficient methodological design it may be possible to cancel the occurrence of such feedback effects to specific sensory regions through transient lesioning or by engaging in transcranial magnetic stimulation (TMS) of the hypothesized critical higher areas, while leaving the sensory-specific regions intact, indicating whether the cross-modal interaction is still present in those intact regions (Driver and Noesselt, 2007). Thus providing conclusive proof as to whether feedback or feed-forward inputs serve as the crucial mediator of interactions in primary cortical regions. Stimuli Factors Modulating Interactions Nonetheless despite these findings, studies have failed to consider in depth the nature of the stimulus being presented on such interactions. Evidence indicates that cross-modal interactions in primary brain regions may be modulated by factors such as the temporal and spatial congruence of the stimulus presented (Mishra et al., 2007). Additionally an explicit exploration of the effect of stimulus intensity has not been adequately considered within experimentation, which can serve to hinder theoretical advancement due to methodological differences. Addressing such questions is necessary so as to determine whether such cross-modal interactions across sensory systems are general, automatic processes or whether they may be modulated by stimulus-specific processes (Martuzzi et al., 2007). Given the vast array of stim-

6 Literature Review 5 NEURAL BASIS AND RELEVANCE OF CROSS-MODAL INTERACTIONS uli experienced in the environment this serves as a necessary avenue for exploration in order to provide greater clarity on the functional nature of such interactions. Additionally a gap in the research is that most studies have solely considered interactions following the presentation of meaningful stimuli (Martuzzi et al., 2007), an avenue for future research is examining whether such processes may be elicited through the presentation of basic, meaningless stimuli. Despite the growing body of evidence surrounding early cross-modal interactions, several studies have failed to indicate the occurrence of short latency multi-sensory interactions when presented with salient, high intensity visual stimuli (Bolognini et al., 2010). In an explicit test of stimulus intensity Senkowski and colleagues (2011), presented participants with semantically meaningless but behaviourally relevant audiovisual stimuli of high, medium and low intensities, finding an inverse relationship between stimulus intensity and the magnitude of integrative cross-modal processes. Given that such differences may produce contrasts in both the time course and magnitude of early cross-modal interactions, the research field should strive for greater methodological consistency. Through the presentation of low intensity stimuli may allow for greater convergence of findings. The temporal specificity of the stimulus presented was highlighted by Bolognini and colleagues (2010), indicating the presence of a critical temporal window (<50ms) for enhanced cross-modal interactions. The study investigated the effect of auditory stimuli on visual perception of phosphenes induced by TMS of the occipital visual cortex. With the modulation of such phosphine perception being maximised when the auditory stimulus preceded the occipital TMS pulse by 40 ms. In each of these cases, the somatosensory stimulation produced perceptual amplification of auditory input. Lakatos and colleagues (2007) further highlighted the role of timing in accounting for such enhanced responses, indicating that visual or somatosensory input can help to drive the ambient oscillations in auditory cortex into the ideal phase for the auditory input, enhancing auditory cortical response. Functional Relevance Of Interactions Though such anatomical findings provide evidence for early cross-modal interaction and may serve to expand the current theoretical framework of hierarchical sensory processing, it is necessary to consider the consequences of such processes within the perceptual domain. Alongside these underlying neural processes, it is important to establish the functional significance of cross-modal interactions and how they may contribute to perception in a real world environment containing a vast array of stimuli. A considerable degree of spec-

7 6 Student Psychology Journal, Volume IV MURRAY ulation exists around such a question, given that research to date has failed to extend itself fully to this domain and provide considered evidence (Driver and Noesselt, 2008). Kayser and colleagues (2008) have further called into question the extent to which such interactions can at present be labelled as integrative, positing that despite indications of specificity to the temporal or spatial alignment of the stimuli, minimal evidence has been provided so as to establish the extent to which these effects are capable of assisting behavioural responses to stimuli and increasing the capacity of the sensory system to accurately perceive that which surrounds it. Account needs to be taken of sensory combination yielding greater information about the environment as a consequence of combining differing sensory stimuli. In addition, it should be noted that sensory integration reduces the uncertainty in the internal representation of the stimulus, which then enhances the behavioural reaction (Ernst and Bulhoff, 2004). An important starting point from which to address such would be the establishment of the characteristics of these interactions. Research appears to indicate that primary cross-modal interactions lack spatial precision: somatosensory inputs received by the auditory cortex appear scattered in nature (Fu et al., 2003) In addition, lateral connections between primary visual and auditory areas would seem to target the peripheral visual field representations, specifically avoiding the more precise central visual field representations (Rockland et al., 2003). Thus, on the basis of such poor spatial precision, it is open to question as to the extent to which early cross-modal interactions are capable of aiding the construction of precise higher-order multisensory representations. Such a suggestion is supported through findings from Lakatos and colleagues (2007), indicating that the somatosensory modulation occurring in the A1 region is more related to hearing than it is to the unification of sensory inputs into a singular percept. A hypothesis extending from these findings is that the primary function of such non-auditory inputs is to reinforce unisensory auditory processes. Due to the relatively high level of temporal precision in cross-modal interactions, non-auditory inputs may be capable of maximizing auditory perception through the modulation of oscillatory cycles (Schroeder et al., 2004). The behavioural relevance of early cross-modal integration has been highlighted in studies indicating that these interactions are capable of producing faster performance in a simple detection task (Sperdin et al., 2009). Neuroimaging analysis indicated that the trials producing faster reaction times were not engaging distinct brain networks, but rather such effects were due to the modulation of cross-modal integration occurring within areas already being engaged in unisensory conditions. Interactions between 86 and

8 Literature Review 7 NEURAL BASIS AND RELEVANCE OF CROSS-MODAL INTERACTIONS 128ms were recorded for both fast and slow reaction times but further early interactions (40-80ms) were only recorded for fast reaction times. Though such a study only provides a correlational link, it tentatively suggests that early interactions may indeed be capable of enhancing behaviours when attending to stimuli. Given the emergent nature of the topic there is a large degree of speculation as this body of knowledge expands, with this being particularly the case for functional implications. One current direction of research surrounds early cross-modal interaction and the binding problem. Given the vast array of stimuli present, entering perceptual regions on a continual basis, it remains unclear how the many sensory inputs combine to allow for unified object perception. Foxe and Schroeder (2005) have postulated that cross-modal interactions may play a vital role in this process. Such a suggestion stems from the notion that such integration may prove overwhelming if not carried out until sensory inputs reach higher order processing regions. Thus the early and continued association of sensory inputs, close to the sensory input stage before the occurrence of comprehensive sensory integration may be beneficial for such a process. Though at present this is simply a hypothesis to be investigated, if it were indeed found to be the case, it may serve as the most fundamental behavioural benefit of early cross-modal integration. Conclusion It is apparent that presently there are more questions than answers within the domain of cross-modal perceptual processes. Though the picture is presently incomplete, alongside the consolidation of theoretical standpoints many findings have served to signpost necessary future research directions. The indication that processes may be mediated by feedback, feed-forward and lateral inputs highlights that cross modal interactions utilize the full range of anatomical connections. While the neocortex does not appear to be as unisensory as once suggested, neither does it appear to be wholly multisensory as initially postulated (Ghazanfar and Schroeder, 2006). The nature of interactions and their modulators appears somewhat region specific, thus comparative study of neuronal properties within brain areas may allow for greater comprehension of such processes. The vast degree of mediators, such as the temporal and spatial nature of stimuli, indicate the considerable task at hand to provide a comprehensive overview of such interactions, however, considered multi-modal research may bring greater clarity and allow for the integration of these findings into the perceptual framework, bringing about a new perspective on the nature of perceptual processes and the neural pathways mediating them. References

9 8 Student Psychology Journal, Volume IV MURRAY Alais, D., Newell, F. N., & Mamassian, P. (2010). Multisensory processing in review: from physiology to behaviour. Seeing and perceiving, 23(1), Bolognini, N., Olgiati, E., Rossetti, A., & Maravita, A. (2010). Enhancing multisensory spatial orienting by brain polarization of the parietal cortex. European Journal of Neuroscience, 31(10), Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on sensory-specific brain regions, neural responses, and judgments.neuron, 57(1), Ernst, M.O., Bulthoff, H.H (2004) Merging the senses into a robust percept. Trends Cogn Sci 8: Falchier, A. et al. (2002) Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22, Foxe, J. J., & Schroeder, C. E. (2005). The case for feed-forward multisensory convergence during early cortical processing. Neuroreport, 16(5), 419. Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory?. Trends in cognitive sciences, 10(6), Giard, M.H. and Peronnet, F. (1999) Auditory visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. J. Cogn. Neurosci. 11, cross-modal information?. Brain structure and function, 212(2), Kotter R, Sommer F.T., (2000) Global relationship between anatomical connectivity and activity propagation in the cerebral cortex. Philos Trans R Soc Lond B Biol Sci.;355: Martuzzi, R., Murray, M. M., Michel, C. M., Thiran, J. P., Maeder, P. P., Clarke, S., & Meuli, R. A. (2007). Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cerebral Cortex, 17(7), Mishra J., Martinez A., Sejnowski T.J., Hillyard S.A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. J. Neurosci. 2007;27: Rockland, K.S. and Ojima, H. (2003) Multisensory convergence in calcarine visual areas in macaque monkey. Int. J. Psychophysiol. 50, Romei, V., Murray, M. M., Merabet, L. B., & Thut, G. (2007). Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: implications for multisensory interactions. The Journal of neuroscience, 27(43), Kayser, C., & Logothetis, N. K. (2007). Do early sensory cortices integrate

10 Literature Review NEURAL BASIS AND RELEVANCE OF CROSS-MODAL INTERACTIONS 9 Schroeder, C. E., Molhom, S., Lakatos, P., Ritter, W., & Foxe, J. J. (2004). Human simian correspondence in the early cortical processing of multisensory cues. Cognitive Processing, 5(3), Senkowski, D., Saint-Amour, D., Höfle, M., & Foxe, J. J. (2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. Neuroimage, 56(4), Sperdin, H. F., Cappe, C., Foxe, J. J., & Murray, M. M. (2009). Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed.frontiers in integrative neuroscience, 3. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Stein, B. E., Burr, D., Constantinidis, C., Laurienti, P. J., Alex Meredith, M., Perrault, T. J. & Lewkowicz, D. J. (2010). Semantic confusion regarding the development of multisensory integration: a practical solution. European Journal of Neuroscience, 31(10), Vasconcelos, N., Pantoja, J., Belchior, H., Caixeta, F. V., Faber, J., Freire, M. A. M. & Ribeiro, S. (2011). Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination.proceedings of the National Academy of Sciences, 108(37),