SUPPLEMENTARY INFORMATION. Predicting visual stimuli based on activity in auditory cortices

Transcription

1 SUPPLEMENTARY INFORMATION Predicting visual stimuli based on activity in auditory cortices Kaspar Meyer, Jonas T. Kaplan, Ryan Essex, Cecelia Webber, Hanna Damasio & Antonio Damasio Brain and Creativity Institute University of Southern California 3641 Watt Way Los Angeles, California , USA 1

2 Supplementary figures and figure legends 11 s 5 s 2 s 2 s 2 s Video clip Image acquisition Video clip Image acquisition Time Supplementary Figure 1 Sparse-sampling scanning paradigm. All video clips were presented during the silent interval between two image acquisitions. The image acquisition started seven seconds after the beginning and two seconds after the end of a video clip. 2

3 Supplementary Figure 2 Auditory mask. Horizontal (a) and coronal (b) cuts through the brain of Subject 1 showing the bilateral auditory mask on the supratemporal plane. The left hemisphere is displayed on the right side of the images. The horizontal slice does not display the full antero-posterior extent of the right-hemispheric mask. The abrupt anterior cutoff of the left-hemispheric mask is due to the conservative criteria we applied when determining the borders of the mask (see Supplementary methods). 3

4 7 Subjective rating Supplementary Figure 3 Subjective ratings of the video clips. Subjects were asked the following question: Did the video clip you just watched evoke sounds in your mind? They were requested to rate each clip on a scale from 1 ( not at all ) to 7 ( very much so ). The bars represent average ratings across all subjects. The error bars represent the standard error of the mean. 4

5 Classifier performance Supplementary Figure 4 Classifier performance in control regions. The bars represent classifier performance averaged across all subjects and across all three pair-wise categorical discriminations for the target mask in early auditory cortices (green bar) and for control masks in primary visual cortex, anterior cingulate cortex, middle temporal gyrus, and middle frontal gyrus (blue bars). The control masks were adjusted in size to the auditory target mask (see Supplementary methods). 5

6 Classifier performance Tested on novel stimuli Trained / tested on the same stimuli 0 Animals vs. instruments Animals vs. objects Instruments vs. objects Supplementary Figure 5 Classifier performance on novel stimuli. Blue bars represent classifier performance when two stimuli of each category were used as training trials and the remaining stimulus from each category was used as test trial. Performance was significantly above chance for the two discriminations involving the object stimuli. For comparison, green bars represent classifier performance when the classifier was trained and tested on the same two stimuli from each category. 6

7 0.6 Classifier performance Supplementary Figure 6 Classifier performance for scrambled categories. The x-axis represents different ways in which the nine video clips were rearranged from the original categories into novel groups of three. The scores from 3 to 9 indicate how many stimuli remained in their original groups. A score of 9 thus refers to the original categorical arrangement, whereas a score of 7 indicates that two stimuli had been switched between the original groups. A score of 3 indicates that each group contained one stimulus from each of the original categories. The y- axis represents classifier performance on three-way discriminations averaged across all possible arrangements corresponding to a certain score. Prediction power was best for the original arrangement and worst for arrangements in which all three groups contained one stimulus from each of the original categories. Chance level is

8 Classifier performance /2 1/4 1/6 1/8 2/3 2/5 2/7 2/9 3/5 3/7 3/9 4/6 4/8 5/6 5/8 6/7 6/9 7/9 1/3 1/5 1/7 1/9 2/4 2/6 2/8 3/4 3/6 3/8 4/5 4/7 4/9 5/7 5/9 6/8 7/8 8/9 Supplementary Figure 7 Classifier performance on auditory stimuli. Bars represent classifier performance on each of the 36 pair-wise discriminations among individual auditory stimuli (the figure is analogous to Fig. 1a in the main manuscript, which displays the same information for the visual stimuli). Stimulus key: 1 rooster; 2 cow; 3 dog; 4 violin; 5 piano; 6 bass; 7 vase; 8 chainsaw; 9 coins. 8

9 Classifier performance Tested on novel stimuli Trained / tested on the same stimuli 0 Animals vs. instruments Animals vs. objects Instruments vs. objects Supplementary Figure 8 Classifier performance on novel stimuli (auditory condition). Blue bars represent classifier performance when two stimuli of each category were used as training trials and the remaining stimulus from each category was used as test trial. For comparison, green bars represent classifier performance when the classifier was trained and tested on the same two stimuli from each category. 9

10 Supplementary methods Stimuli Please refer to the video files (Supplementary video clips 1 3) provided online for examples of our video clips. We recorded five of the nine clips ourselves; the remaining four were downloaded from the internet, making sure that image quality and resolution were sufficient. Supplementary Fig. 3 displays the subjective ratings of the video clips by the subjects, in terms of how much sound the clips evoked in their mind. Note that the subjects rated the clips after the end of the scanning session; during the experiment, they were naïve to the purpose of the study, i.e. they were not told that the video clips they saw were aimed at evoking sounds in their mind. The rationale behind the categorical selection of the stimuli was two-fold. First, we were interested to see whether the auditory activity patterns induced by visual stimuli pertaining to a conceptual category would share certain features with one another and could be told apart from the patterns induced by stimuli pertaining to different categories. Second (and more importantly with respect to the main purpose of the study), if categorical representations existed, we would be able to use this in our favor from a methodological point of view. The auditory activity patterns we aimed to demonstrate presumably had a very low signal-to-noise ratio (SNR) because they were not induced by direct auditory input (but rather by signals from the visual cortex) and because we employed a sparsesampling scanning paradigm that only permitted us to acquire a single image per stimulus presentation. Given the low SNR, a high number of training trials for the classifier appeared crucial. If the auditory activity patterns induced by stimuli from within a category shared certain features, this would permit us to collapse them to train the classifier and thus increase the number of trials three-fold. Attaining the same number of training trials using a single stimulus would have been extremely monotonous for the subjects and might have resulted in decreased attention and undesired habituation effects. 10

11 Stimulus presentation and image acquisition Timing and presentation of the video clips was controlled with MATLAB (The Mathworks) using the freely available Psychophysics Toolbox Version 3 software 16. The clips were projected onto a rearprojection screen at the end of the scanner bore which subjects viewed through a mirror mounted on the head coil. A sparse-sampling scanning paradigm was used to ensure that the video clips were presented during scanner silence (Supplementary Fig. 1): a single whole-brain volume was acquired starting seven seconds after the beginning (and thus two seconds after the end) of each video clip. This delay was chosen based on observations in a separate set of subjects which had shown that the hemodynamic response recorded from auditory cortex in response to sound stimuli reached its peak relatively early, around four seconds post stimulus presentation. As the sound-implying events occurred towards the end of the five-second video clips, the above stimulus paradigm was aimed at capturing auditory activity at its maximum. During each functional run, the nine video clips were presented three times each in random order, for a total of 27 clips per run. Each subject completed eight functional runs, yielding a total of 216 clip presentations (72 per category, 24 per individual stimulus). Images were acquired with a 3-Tesla Siemens MAGNETON Trio System. Echo-planar volumes were acquired with the following parameters: TR = 11,000 ms, TA = 2,000 ms, TE = 25 ms, flip angle = 90, 64 x 64 matrix, in-plane resolution 3.0 mm x 3.0 mm, 41 transverse slices, each 2.5 mm thick, covering the whole brain. To precisely define the auditory mask, we also acquired a structural T1- weighted MPRAGE in each subject (TR = 2,530 ms, TE = 3.09 ms, flip angle = 10, 256 x 256 matrix, 208 coronal slices, 1 mm isotropic resolution). The functional scans were co-registered to each subject s anatomical scan using FSL s (FMRIB Software Library 17 ) FLIRT linear registration tool 18 to perform a sixdegree-of-freedom rigid-body transformation. The resulting transformation matrix was then used to 11

12 warp the anatomical masks into the functional data space, where all subsequent analyses were performed. Studies applying multivariate pattern analysis often use voxel sizes smaller than the one employed in the present study. However, decreasing voxel size to 1.5 mm x 1.5 mm x 2.0 mm resulted in a decrease, rather than an increase, of classifier performance. This was likely due to the fact that the SNR, presumably low in the first place (see above), was further decreased by the reduction of voxel size. Regions of interest The main objective in defining our target mask in auditory cortices was to assure that it would not include any multimodal cortices, as it was our aim to predict visual stimuli based on the activity of areas not targeted by ascending visual pathways. We therefore opted for a restrictive anatomical mask, rather than a functional localizer, as the latter would have borne the risk of labeling multimodal areas. In the antero-posterior dimension, our auditory mask (Supplementary Fig. 2) was restricted to the extent of Heschl s gyrus or the anterior-most transverse temporal gyrus if there was more than one. We are aware that unimodal auditory cortex is considered to extend beyond Heschl s gyrus to include a larger extent of the planum temporale, especially in the posterior direction. However, the only unambiguous anatomical criterion for the posterior limit of our mask behind Heschl s gyrus would have been the end of the Sylvian fissure; had our mask been defined accordingly, it likely would have included multimodal areas around the temporo-parietal junction. Therefore, we opted for a very restrictive definition of the mask even though this lowered prediction performance with respect to a mask that included more of the anatomical extent of the planum temporale. Heschl s gyrus is located on the supratemporal plane and runs in a postero-medial-to-anterolateral direction. Its postero-medial end is usually an unambiguous eminence that starts to appear at the medial border of the supratemporal plane when one moves through coronal slices in a postero-anterior 12

13 direction. Its antero-lateral end is often less well demarcated. We chose the following conservative definition to determine the anterior border of Heschl s gyrus: we included only slices on which both the first transverse temporal sulcus (bounding Heschl s gyrus anteriorly) and Heschl s sulcus (bounding the gyrus posteriorly) were discernible. In some cases, this meant that we discarded slices on which the gyrus still seemed apparent but was not well-defined in terms of the bordering sulci. Medio-laterally, the mask extended beyond Heschl s gyrus to include the whole width of the supratemporal plane. Medially, the border of the mask was clearly defined by the transition point between the supratemporal plane and the insular cortex; laterally, we made sure to include no more than half of the superior temporal gyrus, as we wanted to avoid any gray matter in the superior temporal sulcus, which is known to contain multimodal areas. To illustrate classifier performance outside unimodal auditory cortex, we established control masks in primary visual cortex, anterior cingulate cortex, middle frontal gyrus, and anterior middle temporal gyrus. These masks were obtained by choosing the corresponding probabilistic masks from the Jülich (primary visual cortex) and Harvard-Oxford (anterior cingulate cortex, middle frontal gyrus, and anterior middle temporal gyrus) cortical atlases included with FSL and then warping them from the standard space into each individual s functional space. For each control mask, we selected the probability threshold so that the average number of voxels it occupied in the subjects functional spaces did not differ significantly from the average number of voxels occupied by the auditory target mask. Multivariate pattern analysis Multivariate pattern analysis (MVPA) of fmri data uses pattern classification algorithms to identify distributed neural representations associated with specific stimuli or classes of stimuli. As opposed to traditional (univariate) fmri analysis techniques, in which each voxel is analyzed separately, MVPA can find information in the spatial pattern of activation across multiple voxels. For example, 13

14 whereas the average activation in a region of interest may not differ significantly between two stimulus conditions, information may still be contained within the spatial profile of the activity pattern in that region. MVPA is often used for decoding, i.e. it aims at identifying a specific perceptual representation based on the pattern of neural activation in a region of interest. For example, MVPA has been used to predict, from the activity pattern in primary visual cortex, which of several visual stimuli a subject was exposed to. Typically, a computer algorithm (referred to as classifier ) is trained for this prediction task using a certain proportion of all trials, and the performance of the algorithm is then tested using the remaining trials. For example, we would train the algorithm by providing it with the auditory activity patterns recorded during dog and chainsaw trials of seven functional runs, telling it each time whether the pattern corresponded to a dog or a chainsaw trial. We would then test the algorithm by providing it with dog and chainsaw trials from the last run and asking it, each time, to guess whether the pattern corresponded to a dog or a chainsaw trial. If the algorithm identified the correct stimulus at higher-than-chance level in the test trials, this would imply that the area within our auditory mask contains information about the video clips presented to the subject. MVPA was performed using the PyMVPA software package 19, in combination with LibSVM s implementation of the linear support vector machine ( Data from the eight functional scans were concatenated and motion corrected to the middle volume of the entire time series using FSL s MCFLIRT tool 20 and then linearly de-trended and converted to z-scores by run. Due to the sparse-sampling scanning paradigm (Supplementary Fig. 1), each trial corresponded to a single functional volume. We performed pair-wise discriminations both among all individual stimuli (n = 36, given there were nine stimuli) and among all categories (n = 3, given there were three categories). In either case, training and testing was performed with a cross-validation approach: for each crossvalidation step, the classifier was trained on seven functional runs and tested on the eighth. This procedure was repeated eight times, using each run as test run once. In each cross-validation step, 14

15 performance was calculated as the number of correct guesses of the classifier divided by the number of test trials. Overall performance was obtained by averaging the results from the eight cross-validation steps. The algorithm classified a total of 48 trials for each individual pair-wise discrimination and a total of 144 trials for each categorical pair-wise discrimination in each subject. Support vector machines were designed for binary classification, i.e. to distinguish between two stimuli only. However, there are various ways to combine multiple binary classifiers to perform multiclass discriminations, i.e. to distinguish among more than two stimuli. LibSVM uses the so-called oneagainst-one method (see reference 21 for further information). Statistical analyses All p-values referred to in the main text are the results of two-tailed t-tests across all eight subjects. Whenever parametric testing was performed, we first verified whether the data points were compatible with the assumption of an underlying normal distribution by performing a Lilliefors test. Among the results of the 36 pair-wise discriminations among individual stimuli (Fig. 1a), this assumption only had to be rejected in a single case (which is to be expected when testing 36 data samples at p = 0.05). The only other set of results that did not pass the normality test was the one for the categorical discrimination between animals and objects (dark gray bars in Fig. 2). However, when the remarkably high performance value of Subject 1 was eliminated from the statistic, the p-value resulting from the remaining seven performance values, which now passed the normality test, was actually even lower than the one indicated in the text (1.2 x 10-5 vs. 6.4 x 10-5 ). 15

16 Supplementary results and discussion Is prediction performance due to differences in the overall activity level evoked by different video clips within the auditory mask? As the subjective ratings of our video clips indicated that they were not identical in terms of how much sound they evoked in the subjects mind (Supplementary Fig. 3), it is conceivable that prediction power would have resulted from differences in the overall level of activity within the auditory mask, rather than from information contained within the spatial activity profile. We assessed this possibility by repeating all pair-wise analyses presented earlier (both among individual stimuli and among categories) but training the classifier only on the averaged activity across the whole auditory mask, thus depriving it of the information contained in the spatial activity profile. Of the 26 pair-wise discriminations among individual stimuli that had yielded prediction performances significantly above chance, only two remained significant when this method was applied (note that at p = 0.05, one would on average expect 1.3 out of the 26 discriminations to fulfill the significance criterion purely by chance). All three pair-wise discriminations among categories were no longer significant. This indicates that the information that allowed the classifier to successfully predict the visual stimuli was indeed contained in the spatial profile, rather than the overall level, of auditory activity. Classifier performance outside early auditory cortices To illustrate classifier performance outside early auditory cortices, we established size-matched control masks in primary visual cortex, anterior cingulate cortex, middle frontal gyrus, and middle temporal gyrus (see Supplementary methods). Within each of these masks, we assessed classifier performance averaged across all subjects and across all three categorical discrimination tasks. Given the visual nature of our stimuli, we expected the analysis of V1 activity to yield high prediction performance. 16

17 Moreover, any brain region targeted by forward projections from the visual cortex might also contain information about the video clips, leading to above-chance prediction performance. The results are in keeping with these predictions (Supplementary Fig. 4 online): performance in primary visual cortex was higher than in the auditory target mask, the averaged performance across all three categorical discriminations approaching 0.8. Prediction performance in the anterior cingulate cortex was modest but significantly different from chance, while performance in the middle frontal gyrus and the middle temporal gyrus were indistinguishable from chance. Thus, classifier performance in the auditory target mask exceeded performance in the latter three control masks even though the control masks were located in areas which are considered multimodal association cortices and potentially receive (indirect) forward projections from the visual cortices. More evidence for categorical representations The fact that the classifier successfully differentiated categories when we collapsed all three stimuli of a group suggests that the auditory activity patterns induced by those stimuli shared certain features. However, it would be conceivable that the classifier could be trained to differentiate any set of three stimuli from any other set of three stimuli. Were the auditory representations induced by video clips from within a category indeed more similar than the representations induced by clips from different categories? We addressed this question in two ways. First, we tested whether the classifier was able to assign entirely novel stimuli to the correct categories. To this end, we again performed the three categorical discriminations described earlier but, in each case, only used two stimuli from each category for training and the remaining stimulus from each category for testing. Thus, the classifier was asked to assign stimuli it had never encountered to categories it had learned based on different stimuli. For each pair-wise categorical discrimination, there were nine ways (3 x 3) to pull out one test stimulus from each 17

18 category. Averaged across all nine permutations and across all subjects, prediction performance was (p = 0.073) for animals vs. instruments, (p = 6.4 x 10-4 ) for animals vs. objects, and (p = 6.5 x 10-3 ) for instruments vs. objects (blue bars in Supplementary Fig. 5). Predictably, these results were lower than those obtained when we used the same two stimuli of each category for training and testing (0.600, 0.693, and 0.608, respectively; green bars in Supplementary Fig. 5). However, they were still significantly above chance for two of the three categorical discriminations and close to significance for the third one. Second, we performed three-way discriminations (see Online methods) both among the three original categories and among scrambled groups of stimuli. If categorical representations existed, prediction performance should be higher when discriminating among the original categories (i.e. [cow, dog, rooster] vs. [bass, piano, violin] vs. [chainsaw, coins, vase]) than when discriminating among randomly arranged groups (e.g. [cow, bass, chainsaw] vs. [dog, piano, coins] vs. [rooster, violin, vase]). We grouped our nine stimuli in all possible ways (n = 280) and then trained and tested the classifier on each of these arrangements. Averaged across subjects, the original arrangement yielded the highest prediction accuracy out of all 280 permutations (0.505 at a chance level of 0.333). Prediction accuracy decreased monotonically as a function of the number of stimuli that were switched with respect to the original categories, the lowest prediction accuracies resulting from arrangements in which each group contained one stimulus of each original category, as in the example given earlier (Supplementary Fig. 6). Both analyses just presented suggest that there were more similarities among the auditory representations induced by video clips from the same category than among the patterns induced by clips from different categories. Does this finding imply that early auditory cortices represent conceptual categories of sound stimuli or does it simply imply that the sounds associated with the clips from a given category share acoustic features, which in turn leads to similar neural representations? A recent study 2 found categorical representations in auditory association cortices for acoustic stimuli (cats, singers, and 18

19 guitars). Notably, those categorical representations were present even though the stimuli from the three categories were closely matched in terms of duration, root-mean-square power, temporal envelope, and temporal profile of their harmonic structure. The authors concluded that the auditory association cortices represent acoustic stimuli categorically irrespective of their acoustic features. It is tempting to speculate that the categorical representations we observe while sounds are perceived in the mind s ear would be of the same nature as those evidenced during auditory perception, i.e. that they would reflect conceptual affiliation rather than acoustic features. However, our study cannot answer this question with certainty because our stimuli were not designed to evoke sounds that would closely match with respect to their acoustic features. Naturally, even if such an attempt had been made, sounds evoked in a subject s mind by visual stimuli could not be controlled as effectively as sounds evoked by auditory stimuli. Classifier performance on auditory stimuli In four subjects (the ones for whom we had obtained the highest average performance during the main experiment, except for one participant who could not partake in a second scanning session due to claustrophobia), we conducted an additional control experiment in which the subjects were exposed to the actual sounds implied by the video clips used in the main experiment (the sounds were presented alone, without the corresponding video clip). In some instances, the audio clips represented directly the audio traces of the video clips; in other instances (when the sound quality of the video clips was insufficient), we recorded separate audio clips or searched for high-quality clips on the internet. In all cases, the auditory stimuli were approximately matched to the corresponding visual stimuli in terms of temporal profile. All clips were matched in terms of root-mean-square (RMS) power and presented at a comfortable listening level adjusted individually for each subject. The clips were presented according to the same sparse-sampling paradigm used in the main experiment (Supplementary Fig. 1). 19

20 We first assessed prediction performance when the classifier algorithm was both trained and tested on audio trials. As in the main experiment, each stimulus was presented three times per run, and each subject completed eight runs, for a total of 24 presentations per stimulus and 72 presentations per category. The same cross-validation paradigm was used to test the classifier as in the video sessions. Classification performance was high for the discriminations both among individual sound clips and among categories. Performance on categorical discriminations, averaged across the four subjects, was for animals vs. instruments, for animals vs. objects, and for instruments vs. objects (dark gray bars in Fig. 3). These values are slightly above those found in a previous study 2, probably due to the higher number of training trials used here and to the fact that a deliberate effort was made in the mentioned study to match the auditory stimuli with respect to a number of acoustic features (see preceding section). We assessed the significance of the categorical discriminations in all four subjects individually using a cumulative binomial distribution function. All results were highly significant (p < 0.01), except for the animals vs. instruments discrimination in Subject 7 (p = 0.075). It is interesting to note that just as on visual trials, the classifier on average performed best on the discrimination between animal and object stimuli (black and dark gray bars in Fig. 3). Performance on pair-wise discriminations among individual clips, averaged across the four subjects, ranged from to 0.889, averaging (Supplementary Fig. 7). An interesting observation could be made from these results. In the main experiment (when video clips were used as stimuli), average classifier performance on discriminations among categories had been higher than on discriminations among individual stimuli. This trend was evident at the group level (0.660 vs ) and in seven of eight individual subjects. In the auditory control experiment, the relationship was reversed both at the group level (0.729 vs ) and in all four individual subjects. A possible explanation for this observation is that the auditory representations of sounds that are perceived are more differentiated than the representations of sounds experienced in the mind s ear. On 20

21 the one hand, this would lead to higher discrimination accuracy at the level of individual stimuli; on the other hand, it would complicate the classifier s task on categorical discrimination tasks, as the activity patterns induced by stimuli from within a category would contain more variability ( noise, from the classifier s perspective). Additional support for this conclusion comes from Supplementary Figs. 5 and 8. Recall that we had assessed the algorithm s ability to classify entirely novel stimuli in order to verify whether the auditory activity patterns induced by video clips from within a certain category were indeed more similar than patterns induced by clips from different categories (see section More evidence for categorical representations and Supplementary Fig. 5). An analogous control experiment was carried out for the auditory stimuli as well (Supplementary Fig. 8). While we found that, just as had been the case for the visual stimuli, the classifier successfully assigned novel auditory stimuli to the correct categories, it can be gleaned by comparing the two figures that the drop in performance when the algorithm was tested on novel stimuli was larger in the auditory condition. This again suggests that, for auditory stimuli, there was more variability among the activity patterns induced by the stimuli from within a category, so that the classifier was less successful at guessing the third stimulus of a category from the first two. Thus, the evidence presented in this and the preceding paragraphs suggests that the auditory activity patterns instantiated while sounds are experienced in the mind s ear, while preserving the core features that permit categorical discrimination, are somewhat degenerated with respect to the features that distinguish the stimuli within the categories. Again, however, the data do not provide a conclusive answer to the question whether these core features characterize conceptual representations or simply reflect acoustic features common to the stimuli from within a category. We also asked whether the classifier, after being trained on video trials, would be able to successfully discriminate the corresponding audio trials, and vice versa. If this were the case, we could conclude that the auditory activity pattern induced by a video clip implying a certain sound would bear 21

22 similarities with the pattern induced by the sound itself. We obtained comparable results when training the algorithm on the visual trials and testing it on the auditory trials as when following the reverse procedure; we therefore only report the results obtained according to the former paradigm. At the categorical level, while the animals vs. instruments and the instruments vs. objects discriminations yielded performances only slightly above chance (0.526 and 0.529, respectively), performance in the animals vs. objects discrimination was (results averaged across all four subjects; light gray bars in Fig. 3). While the animals vs. objects discrimination was significant in two out of four subjects, both the animals vs. instruments and the instruments vs. objects discriminations were significant in one out of four subjects (p < 0.05, cumulative binomial distribution function). A similar pattern was observed for the discriminations among individual stimuli: while the classifier performed above 0.6 in five out of the nine discriminations between animal and object stimuli, no other discrimination reached this value, many results fluctuating around chance level. Several methodological and conceptual issues should be considered when interpreting these results. First, when training and testing the classifier on video trials, performance on the animals vs. instruments and instruments vs. objects discriminations had exceeded 0.6 only by a small margin. It was to be expected that these values would be lower yet when testing the algorithm on audio instead of video trials. Second, it is difficult to decide upon standard audio clips which would match as closely as possible the very personal sound experience each of our subjects had for each of the video clips. In fact, given this dissimilarity in subjective sound experience between the video trials and the standardized audio trials, an analogous dissimilarity would be expected at the level of the respective neural representations in early auditory cortices, and this should inevitably lower prediction performance. A way of avoiding this problem would have been to present the auditory stimuli before the video clips, in order to control more precisely the subjects experience when seeing the sound-implying events. 22

23 However, presenting auditory and visual stimuli in this order would have meant establishing a crossmodal association in the very course of the experiment, and this, in our opinion, would have limited the degree to which our results could be generalized. After all, we intended to show that perceiving soundimplying visual stimuli automatically leads to content-specific representations in early auditory cortices, i.e. without the subjects being aware of the purpose of the study or making a deliberate effort to image sounds. Third, and most importantly, when we averaged the activity level across the whole auditory mask on both the visual training trials and the auditory test trials (in the same way it had been done for the control experiment described earlier), the classifier often performed considerably below chance level (yielding values as low as 0.35). This indicates that the sounds that evoked the highest overall activity level in the auditory condition often were not the ones that had been experienced most vividly in the visual condition. (This occurred even though the audio clips were matched for RMS power and the control experiment described earlier had indicated that the overall activity levels during the visual condition were too similar to yield significant prediction power.) Thus, it appears important to consider at least two dimensions content and vividness when referring both to the subjective experience of sound and its presumed neural counterpart in early auditory cortices. While watching a video clip implying a certain sound and actually hearing that sound may lead to a (somewhat) similar auditory experience in terms of content, the experience may still differ in terms of its vividness; we would thus be looking for similarities in the neural counterparts of mental events that were not so similar in the first place. Specifically, while the sound experience produced by a certain auditory stimulus may be similar, in terms of content, to the experience produced by the corresponding video clip, it may also be similar, in terms of vividness, to the experience produced by a different video clip. Assuming the neural representations in early auditory cortices to reflect the corresponding mental events faithfully, this could be the reason for some of the incorrect categorical attributions we observed. In a more general 23

24 way, it is important to remember that the subjective sound experiences induced when watching a video clip implying a certain sound and when actually perceiving that sound are by no means identical (as evidenced by our awareness of whether we hear a sound in reality or just in our mind s ear). We should not expect the neural activity patterns observed in these two situations to be any more similar than the mental events they presumably reflect. In summary, the present data do not allow the conclusion that the auditory activity patterns evoked by sound-implying video clips are replicas of those evoked by corresponding sounds. However, given the above methodological and conceptual qualifications, it would be no less premature to claim that the patterns are entirely distinct. Also, the finding that the patterns are not identical does not rule out a correlation between neural activity in early auditory cortices and the subjective experience of sound. 24

25 Supplementary References 16. Brainard, D. H. The psychophysics toolbox. Spatial Vision 10, (1997). 17. Smith, S. M. et al. Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage 23 (S1), (2004). 18. Jenkinson, M. & Smith, S. A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5, (2001). 19. Hanke, M. et al. PyMVPA: a Python toolbox for multivariate pattern analysis of fmri data. Neuroinformatics 7, (2009). 20. Jenkinson, M., Bannister, P., Brady, M. & Smith, S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. NeuroImage 17, (2002). 21. Hsu, C.-W. & Lin, C.-J. A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks 13, (2002). 25

26 Supplementary video clips Supplementary video clip 1: This clip displays a howling dog and serves as an example for the animal video clips. Supplementary video clip 2: This clip displays a piano key being struck and serves as an example for the musical instrument video clips. Supplementary video clip 3: This clip displays a handful of coins being dropped into a glass and serves as an example for the object video clips. 26