Figure 1. Excerpt of stimulus presentation paradigm for Study I.

Transition Visual Auditory Tactile Time 14 s Figure 1. Excerpt of stimulus presentation paradigm for Study I. Visual, auditory, and tactile stimuli were presented to sujects simultaneously during imaging. The visual stimulus was either the lue or the red astract figure shown; the auditory stimulus was either the sound of running water or the sound of croaking frogs (actual waveforms shown); the tactile stimulus was either circular rushing or rapid tapping of the lower right leg, using a 6 x 10 cm shower rush. At 14 s intervals, in pseudo-random order, one of the three stimulus modalities underwent a transition from one stimulus type to the other. A total of 44 transitions (15 visual, 15 auditory, 14 tactile) occurred during the course of the experiment. V, visual; A, auditory; T, tactile. 170

Figure 2. Surface rendering of rain regions activated y transitions of the visual, auditory, and tactile stimuli in Study I. Note that unimodal activations are ilateral and correspond predominantly to association cortices, whereas multimodal activations are strongly lateralized to the right hemisphere and correspond to the temporoparietal junction (TPJ), inferior frontal gyrus, and insula. On the left medial wall, the supplementary and cingulate motor areas (SMA and CMA) are also prominently activated, even though sujects were not required to make responses or movements during task performance. Activations represent averaged data from 10 sujects, superimposed on the standardized rain of one suject., left;, right. 175

a -17 +9 c d +21 +21 +9 Figure 3. esponses to visual, auditory, and tactile transitions in Study I. Averaged eventrelated hemodynamic responses to transitions in each stimulus modality are shown for representative unimodally- and multimodally-responsive areas: (a) ilateral fusiform gyrus, () ilateral superior temporal gyrus, (c) secondary somatosensory cortex (S2), (d) ilateral TPJ. Unimodal areas (a), (), and (c) respond only to transition events within visual, auditory, and tactile modalities, respectively, while the TPJ (d) responds ilaterally to transitions in all three modalities. A significant (p = 1) deactivation of the fusiform gyrus in response to auditory and tactile transitions is visile in (a). Multimodal activation in the right anterior insula, not shown in Fig. 2, is visile in (d). The Talairach Z co-ordinate of each slice is indicated in the upper left-hand corner. Error ars indicate S.E.M. for each data point. 176

Temporoparietal Junction Inferior Frontal Gyrus Z +9 Y -43 X -52 Z +6 X +45 Anterior / Posterior Insula SMA / CMA Z +9 Z +6 X -10 Z +42 Z 0 Middle Temporal Gyrus Figure 4. Multimodally-responsive rain regions in Study I. The plane co-ordinate of each slice is indicated in the upper left-hand corner. SMA, supplementary motor area; CMA, cingulate motor area. 177

Change in Signal (%) 0.3 Visual A B Visual B A Auditory A B Auditory B A Tactile A B Tactile B A -4-2 14 16 Time Post-Transition (s) Figure 5. esponses of the right TPJ to each direction of transition in each modality in Study I. Averaged event-related hemodynamic responses to transitions from stimulus A to stimulus B as well as stimulus B to stimulus A are shown for each sensory modality. The appearance of activations for oth transition types excludes the possiility that the eventrelated activations in Figure 3d reflect an increase in stimulus intensity or contrast associated with just one of the two types of transitions. Similar activation for oth types of transitions was verified for all unimodal and multimodal areas reported. 178

pitch pitch a A V A V A V A V Duration (s) Event Visual Stimulus Instruct. 10 12 10 14 12 12 A VI A A VI Auditory Stimulus Suject esponse A Predicted H VI Predicted H c Duration (s) Event Visual Stimulus 12 14 12 10 14 10 Instruct. V AI AI AI V Auditory Stimulus Suject esponse V Predicted H AI Predicted H Figure 6. Task design schematic for Study II. (a) The experimental protocol consisted of 8 locks: 4 visual-relevant and 4 auditory-relevant. A total of 10 visual-relevant (V), 10 visualirrelevant (VI), 10 auditory-relevant (A), and 10 auditory-irrelevant (AI) stimulus changes took place during the course of the experiment. () Example of an A lock. (c) Example of a V lock. Visual and auditory stimuli were presented simultaneously during the experiment. The visual stimulus was a centrally presented square ox containing a diagonal cross. The auditory stimulus was a continuous uzzing sound. At the eginning of each lock, a non-veral instruction stimulus cued the suject to attend to either the visual or the auditory stimulus and report changes in the attended stimulus y raising the right index finger riefly. A total of 5 changes took place in each lock, in a pseudo-random order, at intervals of 10, 12, or 14 s. Changes in the visual stimulus were 10 rotations to the left or right. Changes in the auditory stimulus were 5% increases or decreases in overall pitch. 190

a All Events conjoint p < 001 min. 150 mm 3 Z +12 AntIns Z +22 PCC Tha TPJ TPJ Z +31 X -4 SMA/ CMA IFG Precun ACC Figure 7. Brain regions responding to all changes in Study II. (a) Surface rendering of the common set of regions responsive to V, VI, A, and AI events, superimposed on the standardized rain of one suject. () Slice views of the common set of regions responsive to V, VI, A, and AI events. The plane coordinate of each slice is indicated at upper left. egions responding to all events included right and left TPJ, right IFG, left SMA/CMA, right and left anterior insula, PCC, precuneus right IFG, and right thalamus. This network corresponds closely to the set of regions identified as responsive to changes in the sensory environment across multiple modalities in the asence of a task in Study I. 199

a IPS TPJ IFG Ant Ins TPJ Ant Ins ACC elevant > Irrelevant Precuneus PCC SMA/ CMA elevant ~ Irrelevant % Signal % Signal Change TPJ SMA/ CMA 0Time 2 from 4 Change 6 8 10(s) 12 0 2 from 4 Change 6 8 10 (s) 12 0 Time 2 from 4 6 Change 8 10 (s) 12 Visual elevant Visual Irrelevant Auditory elevant Auditory Irrelevant IFG Visual Visual elevant elevant Visual Visual Irrelevant elevant IrrelevantI Auditory Visual Auditory elevant IrrelevantI elevant Auditory Auditory Visual Irrelevant IrrelevantI elevant Auditory Irrelevant elevant Auditory Irrelevant Visual elevant Visual Irrelevant Auditory elevant Auditory Irrelevant Figure 8. OI-ased analysis of the effect of task-relevance in Study II. (a) The effect of task-relevance on the response to a given sensory event was assessed on a region-y-region asis (OI analysis). All regions responded significantly to all events, ut context-dependent regions also responded significantly more strongly to V versus VI and to A versus AI events. egions showing context-dependent responses included right and left TPJ, left anterior insula, left precuneus, left ACC, and right thalamus. Context-independent regions responded similarly to all events, regardless of task-relevance. arge regions with context-independent responses included right IFG, left CMA/SMA, right anterior insula, left IPS, and precuneus. Smaller context-independent regions included right and left ITG, left MFG, and PCC. () Average BOD responses to task-relevant and task-irrelevant events in right TPJ, right IFG, and left SMA/CMA, indicated arrows in (a). 200

a X -3 SMA/ CMA Y -43 TPJ SMG STG X -35 Ant. Z +17 Insula TPJ TPJ % Signal % Signal Change Time Time from from Change Change (s) (s) elevant > Irrelevant elevant ~ Irrelevant 0 Time 2 from 4 6 Change 8 10 (s) 12 TPJ Visual elevant Visual Irrelevant Auditory elevant Auditory Irrelevant Visual elevant Visual elevant IrrelevantI Visual Auditory IrrelevantI elevant Auditory Visual IrrelevantI elevant Auditory Irrelevant elevant Auditory Irrelevant Figure 9. Voxelwise analysis of the effect of task-relevance in Study II. The effect of taskrelevance on the response to a given sensory event was assessed on a voxelwise asis to identify context-dependent and independent suregions within the OIs responsive to all events. (a) Distinct context-dependent (lack arrows) and context-independent (white arrows) suregions >150 mm 3 were found in four of OIs: left SMA/CMA, left anterior insula, right TPJ, and left TPJ. The context-dependent and independent suregions of the SMA/CMA region correspond to CMA and SMA, respectively. The context-dependent and independent suregions of the right TPJ correspond to the supramarginal gyrus and superior temporal gyrus, respectively. The plane coordinate of each slice is indicated at upper left. () Surfacerendered view of oth context-dependent and context-independent suregions in left and right TPJ. Average BOD responses to task-relevant and task-irrelevant events in suregions of the left TPJ, indicated with arrows on the surface rendering. 202

a % Signal elevant Only Irrelevant Only c % Signal 0.3 M1 / S1 Time Visual from Change elevant (s) Visual elevant IrrelevantI Visual Auditory IrrelevantI elevant Auditory elevant Irrelevant Auditory Irrelevant pre-sma Visual elevant Visual Irrelevant Auditory elevant Auditory Irrelevant 0 2Visual 4 6elevant 8 10 12 Time Time Visual from Change (s) from Change IrrelevantI elevant (s) Auditory Visual IrrelevantI elevant Auditory Irrelevant elevant Auditory Irrelevant Figure 10. Brain regions responding exclusively to task-relevant or task-irrelevant events in Study II. (a) egions responding exclusively during task-relevant stimulus changes included a wide array of sensory and motor areas, including left M1 and S1, ilateral S2 and cereellum, left thalamus, and a large medial region encompassing left SMA and ACC. Some occipital regions, including superior occipital gyrus and left lingual gyrus, also showed responses only to task-relevant changes. Additional activations were found ilaterally in SP and MFG and in right IP. () egions responding exclusively during task-irrelevant stimulus changes included only right IFG and left pre-sma. (c) Average BOD response of the exclusively task-relevant activation encompassing M1 / S1 (shown at left) and of the exclusively task-irrelevant activation in left pre-sma (shown at right). The former area shows a strong response to task-relevant events ut a minimal response to task-irrelevant events, while the latter shows the opposite response profile. Arrows in (a) and () indicate the regions whose average time-courses are plotted in (c). 204

e a Stimulus Pool d S d i h y v l Stimulus Sequence Suject 1 Suject 2 Suject 3... 1 2... 10 11 12 13 14 15 16 17 18... 30 Baseline Fam Fam... Fam Nov Fam Fam Nov Fam Nov Nov Fam... Nov v... d l S y y d d... d S d d v d y l d... d... v y l... S... Novel Predicted HF Familiar Predicted HF Novel - Familiar......... time Figure 11. Schematic of visual stimulus presentation schedule for Study III. (a) The stimulus pool from which aseline, familiar, and novel stimuli were drawn randomly for each suject. () Sample sequences of visual stimulation for 3 sujects, with predicted hemodynamic responses for familiar and novel stimuli. Analogous schedules of presentation were used for stimuli in the auditory and tactile modalities. 221

a Z +23 Z +3 TPJ IFG Ant. Insula IFG X -7 Y -37 Y -52 Ant. Insula ACC TPJ (SMG) p < 001, minimum 150 mm 3 TPJ (STG) TPJa All Novel Novel Stimuli All Familiar Stimuli Novel Novel Familiar Familiar TPJ (SMG) ACC No IFG Fa 0 0 2 2 4 6 4 6 10 12 8 10 12 Time Post-Stimulus from Event (s) Time from Time Event (s) (s) Post-Stimulus Time (s) Time from Event (s) Time Post-Stimulus from Event (s) Time (s) Figure 12. Multimodal novelty-sensitive rain regions in Study III. (a) Activations in the TPJ, IFG, ACC, and anterior insula showed a significantly greater response to novel than familiar stimuli. Activations represent group-average data for all 10 sujects. () Average event-related hemodynamic responses (± SE) to all novel and all familiar stimuli show strong responses to novel stimuli ut nonsignificant responses to familiar stimuli in the TPJ, IFG, and ACC. 224

a TPJ Z +23 Z +7 Novel > Familiar elevant ~ Irrelevant elevant > Irrelevant ACC X -7 Z +31 Novel > Familiar elevant ~ Irrelevant elevant Only c IFG Z +28 Novel > Familiar elevant ~ Irrelevant Irrelevant Only p < 001, minimum 150 mm 3 Figure 13. Comparison of Studies II and III: novelty-related and generalized functions for TPJ, ACC, and IFG. (a) The novelty-sensitive TPJ suregion in the SMG partially overlaps a large right TPJ region sensitive to the task-relevance of non-novel sensory events. The novelty-sensitive TPJ activation in the STG orders a region responding similarly to all sensory events, regardless of relevance. () The novelty-sensitive left ACC region partially overlaps a large medial region spanning ACC, CMA and SMA which was activated exclusively during task-relevant stimulus changes. (c) The novelty-sensitive region in right IFG partially overlaps a region activating exclusively during task-irrelevant sensory events and orders a region activating during oth task-irrelevant and task-relevant sensory events. 228

Predicted BOD signal 1 1 2 3 2 Non-noxious stimulus est Noxious stimulus est Figure 14. Hypothesized response of salience-sensitive cortical regions to prolonged painful and non-painful stimulation. Salience-sensitive regions, such as the TPJ, IFG, and ACC (shown in red), should display transient responses to changes (such as the onset or offset) in a non-painful stimulus, ut prolonged, tonic responses throughout the duration of a painful stimulus. egions showing the hypothesized response were required to fulfil three criteria: 1. A transient response to the onset and offset of the non-painful stimulus; 2. A greater tonic response during painful versus non-painful stimulation; 3. A non-significant (p > 1) level of response during the last 40 s of non-painful stimulation. 239

70 60 Pain Intensity ating (0-100) 50 40 30 20 10 0 0 90 180 270 360 450 540 630 Time (s) Figure 15. Psychophysical ratings of perceived pain in Study IV. Plot of average pain ratings during 12 imaging sessions in 8 sujects undergoing painful TENS and performing continuous on-line pain ratings (see Study IV Methods). Grey shading indicates periods of painful TENS and white shading indicates periods of rest. 246

a Voxelwise Conjoint P 10-12 Conjoint p < 001 min 150 mm 3 10-4 Y -38 Y +1 Y -27 Temporoparietal Junction Inferior Frontal Gyrus S1 and S2 Z 0 Z +12 X -4 Insula and Putamen Insula and Thalamus ACC, CMA/SMA, and PCC Figure 16. egions sensitive to non-painful stimulus changes and painful stimulation in Study IV. (a) Areas showing a significant transient response to oth the onset and the offset of non-painful tingling stimulation, as well as a significantly greater tonic response during the lock of painful versus non-painful stimulation. () Slice view of activations shown in (a). The plane co-ordinate of each slice is indicated in the upper right corner. 247

a Temporoparietal Junction Y -38 From Onset 0.6 0.5 0.4 0.3 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain X +7 Anterior Cingulate Cortex From Onset 0.5 0.4 0.3 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain c Y +1 Inferior Frontal Gyrus From Onset 0.5 0.4 0.3 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain Figure 17. Cortical regions displaying the predicted response profile in Study IV. Plots on the right show the average response of each area on the left to painful and non-painful stimulation. Grey shading on the average response plots indicates the period of stimulation. ight TPJ (a), ACC (), and IFG (c) were among areas showing a significant tonic response during painful ut not non-painful stimuli. During non-painful stimulation, these regions only showed transient responses to the stimulus onset or offset. During painful stimulation, these regions showed sustained responses throughout the duration of the stimulus. 249

a Thalamus Z +12 From Onset 1.0 0.8 0.6 0.4 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain Putamen Z 0 From Onset 0.4 0.3 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain Figure 18. Sucortical regions displaying the predicted response profile in Study IV. Plots on the right show the average response of each area on the left to painful and non-painful stimulation. Grey shading on the average response plots indicates the period of stimulation. ight thalamus (a) and left putamen () were among areas showing a significant tonic response during painful ut not non-painful stimuli. During non-painful stimulation, these regions only showed transient responses to the stimulus onset or offset. During painful stimulation, these regions showed sustained responses throughout the duration of the stimulus. 250

a Z +48 Primary Somatosensory Cortex From Onset 1.0 0.8 0.6 0.4 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain Y -27 Secondary Somatosensory Cortex From Onset 0.6 0.5 0.4 0.3 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain c Anterior Insula Z 0 From Onset 0.8 0.6 0.4 0 15 30 45 60 75 90 Time from Stimulus Onset (s) Tingle Pain Figure 19. egions not displaying the predicted response profile in Study IV. Plots on the right show the average response of each area on the left to painful and non-painful stimulation. Grey shading on the average response plots indicates the period of stimulation. Areas including left S1 (a), left S2 (), and right anterior insula (c) showed a significant tonic response during non-painful as well as painful stimulation. In contrast, the areas shown in Figures 17 and 18 showed a significant tonic response only during painful stimulation. 251