Text to brain: predicting the spatial distribution of neuroimaging observations from text reports (submitted to MICCAI 2018)

Transcription

1 1 / 22 Text to brain: predicting the spatial distribution of neuroimaging observations from text reports (submitted to MICCAI 2018) Jérôme Dockès, ussel Poldrack, Demian Wassermann, Fabian Suchanek, Bertrand Thirion, Gaël Varoquaux INIA, Télécom Paris, Stanford University

2 2 / 22 Text to brain images Many neuroimaging observations are stored in unstructured text. e.g [...] in the anterolateral temporal cortex, especially the temporal pole and inferior and middle temporal gyri It would be good to have them in the form of images and do statistics with them our objective: transform a medical publication into a brain image

3 3 / 22 text to brain image Title: Where sound position influences sound object representations: a 7-T fmi study [5] Abstract: Evidence from human and non-human primate studies supports a dualpathway model of audition, with partially segregated cortical networks for sound recognition and sound localisation, referred to as the What and Where processing streams. In normal subjects, these two networks overlap partially on the supratemporal probability density function (pdf) over the brain (subset of 3 )

4 How we transform text into a brain image (a spatial pdf)

5 5 / 22 Supervised learning approach Simple heuristics don t work Some articles e.g. fmi (functional Magnetic esonance Imaging) studies provide text and coordinates in the brain We extract those coordinates and exploit the pairs (text, coordinates) in a supervised learning setting

6 Example fmi study acroix et al. Neural computations for speech and music perception acroix et al. Neural computations for speech and music perception TABE 2 ocations, peaks and cluster size for significant voxel clusters for each condition s AE and for each contrast of interest. Condition Anatomical locations Peak coordinates Voxels Music passive listening eft inferior frontal gyrus (pars opercularis)* 46, 10, eft medial frontal gyrus*, left subcallosal gyrus 2, 26, eft medial frontal gyrus* 2, 2, eft postcentral gyrus*, left inferior parietal lobule 34, 36, eft superior temporal gyrus*, left transverse temporal gyrus, left middle temporal gyrus, left insula 52, 20, ight inferior frontal gyrus* 48, 10, ight precentral gyrus*, right postcentral gyrus, right middle frontal gyrus 52, 2, ight superior temporal gyrus*, right transverse temporal gyrus, right middle temporal gyrus, right insula 58, 20, ight insula*, right inferior frontal gyrus, right precentral gyrus 42, 14, ight lingual gyrus*, right culmen 16, 54, 2 27 Music discrimination eft medial frontal gyrus*, left middle frontal gyrus 8, 4, eft precentral gyrus*, left postcentral gyrus, left inferior parietal lobule 48, 12, eft precentral gyrus*, left inferior frontal gyrus (pars opercularis) 50, 2, eft superior temporal gyrus*, left transverse temporal gyrus, left precentral gyrus 54, 16, eft superior temporal gyrus*, left middle temporal gyrus 58, 34, 8 92 eft insula*, left inferior frontal gyrus (pars triangularis) 34, 22, 2 48 eft cerebellum* 28, 62, ight inferior frontal gyrus*, right middle frontal gyrus 52, 12, ight precentral gyrus*, right middle frontal gyrus 46, 6, FIGUE 1 (A) epresentative sagittal slices of the AE for passive listening to speech,p<0.05, corrected, overlaid on top of the passive music listening AE. (B) Speech vs. music passive listening contrasts results, p < 0.05 corrected. ight superior temporal gyrus*, right middle temporal gyrus 62, 24, ight superior temporal gyrus*, right precentral gyrus, right insula 50, 6, 2 91 Music error detection eft medial frontal gyrus* 4, 4, Music Tasks vs. Speech Tasks identified bilateral superior temporal sulci regions as well as left The passive listening AE results identify distinct and inferior frontal regions (pars triangularis and pars opercularis). overlapping regions of speech and music processing. We Music memory > speech memory identified a left posterior now turn to the question of how do these distinctions change superior temporal/inferior parietal region and bilateral medial as a function of the type of task employed? First, AEs were frontal regions; speech memory > music memory identified left computed for each music task condition, p < 0.05 FD inferiorfrontalgyrus(parsopercularisandparstriangularis) and corrected (Figure1, Table2). The music task conditions AEs bilateralsuperiorandmiddletemporalgyri. all significantly identified bilateral STG and bilateral precentral In sum, the task pairwise contrasts in many ways mirror gyrus, and inferior parietal regions, overlapping with the the passive listening contrast: music tasks activated more passive listening music AE (Figure2). The tasks also activated dorsal/medial superior temporal and inferior parietal regions, additional inferior frontal and inferior parietal regions not while speech tasks activated superior temporal sulcus regions, identified by the passive listening music AE; these differences particularly in the anterior temporal lobe. In addition, notable arediscussedinasubsequentsection. differences were found in Broca s area and its right hemisphere To compare the brain regions activated by each music homolog: in discrimination tasks music significantly activated task to those activated by speech in similar tasks, pairwise Broca s area (specifically the pars opercularis) more than speech. contrasts of the AEs for each music task vs. its corresponding However,indetectionandmemorytasksspeechactivatedBroca s speech task group were calculated (Figure3, Table2). Music area (pars opercularis and pars triangularis) more than music. discrimination > speech discrimination identified regions The right inferior frontal gyrus responded equally to speech including bilateral inferior frontal gyri (pars opercularis), and music in both detection and memory tasks, but responded bilateral pre and postcentral gyri, bilateral medial frontal more to music than speech in discrimination tasks. Also gyri, left inferior parietal lobule, and left cerebellum, whereas notably, in the memory tasks, music activated a lateral superior speechdiscrimination >musicdiscriminationidentifiedbilateral temporal/inferior parietal cluster (in the vicinity of Hickok and regions in the anterior superior temporal sulci (including both Poeppel s area Spt ) more than speech while an inferior frontal superior and middle temporal gyri). Music detection > speech cluster including the pars opercularis was activated more for detection identified a bilateral group of clusters spanning the speech than music. Both area Spt and the pars opercularis superior temporal gyri, bilateral precentral gyri, bilateral insula previouslyhavebeenimplicatedinavarietyofauditoryworking and bilateral inferior parietal regions, as well as clusters in the memory tasks (including speech and pitch working memory) in right middle frontal gyrus. Speech detection > music detection both lesion patients and control subjects (Koelsch and Siebel, eft superior temporal gyrus*, 50, 18, et transverse temporal gyrus, eft postcentral gyrus, left insula eft inferior parietal lobule*, left supramarginal gyrus, left angular gyrus 40, 48, eft lentiform nucleus*, left putamen 22, 6, ight middle frontal gyrus* 36, 42, ight middle frontal gyrus*, right precentral gyrus 32, 4, ight superior frontal gyrus*, right medial frontal gyrus, left superior frontal gyrus, left medial frontal 2, 10, gyrus ight superior temporal gyrus*, right transverse temporal gyrus, right insula, right precentral gyrus, 50, 18, right middle temporal gyrus, right claustrum ight parahippocampal gyrus* 22, 14, ight inferior parietal lobule*, right supramarginal gyrus 36, 44, ight insula*, right inferior frontal gyrus 32, 22, ight lentiform nucleus*, right putamen, right caudate 18, 6, ight thalamus* 12, 16, 8 33 ight cerebellum* 26, 50, Music memory eft inferior frontal gyrus (pars opercularis)*, left precentral gyrus, left middle frontal gyrus 50, 4, eft inferior frontal gyrus (pars triangularis*, pars orbitalis), left insula 34, 24, 2 57 eft inferior frontal gyrus (pars triangularis)* 44, 26, eft medial frontal gyrus* 4, 52, eft middle frontal gyrus* 32, 4, eft precentral gyrus* 44, 10, eft superior frontal gyrus*, left medial frontal gyrus, right superior frontal gyrus, right medial frontal 0, 12, gyrus eft middle temporal gyrus* 50, 20, eft middle temporal gyrus*, left superior temporal gyrus 46, 4, (Continued) Frontiers in Psychology 5 August 2015 Volume 6 Article 1138 Frontiers in Psychology 6 August 2015 Volume 6 Article / 22

7 Example fmi study or each contrast of interest. Peak coordinates Voxels 46, 10, , 26, , 2, , 36, yrus, left insula 52, 20, , 10, , 2, ral gyrus, right 58, 20, , 14, / 22

8 Predict the distribution of coordinates, given text 8 / 22

9 9 / 22 Details text is vectorized by counting word frequencies functions over the brain are turned into vectors of weights over a set of regions (atlas regions or voxels) { k, k = 1... m} true pdf is estimated from the reported coordinates ˆp = 1 c m k=1 c k I k I k 1, (1) where c k is the number of coordinates in k and c = k c k (possibly ˆp is smoothed with a Gaussian kernel) we fit a linear regression: (either least squares or least deviations) + l 2 penalty on the coefficients

10 10 / 22 Details: regression model We are trying to predict a pdf a common distance between probability measures is Total Variation: TV(P, Q) = sup P(A) Q(A) (2) A the sup is attained by taking A = { k P( k ) > Q( k )} and: TV(P, Q) = 1 2 m P( k ) Q( k ) = 1 p(z) q(z) dz. 2 3 k=1 hence the least-deviations regression we also compare it to least squares because it is popular and easy to solve (3)

11 11 / 22 Details: regression model ( ) ˆβ = argmin Ŷ Xβ 1 + λ β 2 2 β (4) particular case of penalized quantile regression can be solved efficiently as the dual only involves box constraints (and X is sparse)

12 esults: prediction on left-out articles and model inspection

13 13 / 22 Evaluation metric: log-likelihood given coordinates reported in left-out articles if q : 3 [0, 1] is our prediction, and are the coordinates reported in the study, is our score 1 log(q(l)) l

14 14 / 22 esults baseline: human-labelled atlases an atlas is a segmentation of the brain, with labels such as parietal lobe or hippocampus for this baseline, we set the probability of a region to be proportional to the frequency of its label in the text Atlas east-squares east deviations og-likelihood of coordinates in left-out articles

15 egression coefficients link words to voxels 15 / 22

16 esults: regression coefficients anterior cingulate y=-6 y=-6 y=-2 left amygdala, amygdala, right amygdala 16 / 22

17 17 / 22 esults: regression coefficients as word embeddings word2vec: emotion negative emotion emotion expression expressing emotions emotion regulation disturbing emotions emotive face recognition fusiform gyrus inferior temporal gyrus temporal fusiform gyrus occipitotemporal sulcus inferior occipital gyri occipito temporal sulcus middle occipital gyrus brain maps: emotion amygdala basolateral amygdala fear anxiety affection anger fusiform gyrus object recognition face recognition face perception ba37 prosopagnosia identity gender

18 Example (proof-of-concept) application

19 19 / 22 Application we want to know which brain areas are related to Huntington s disease. in our labelled corpus, almost no article (only 21) mention it we learn term-location associations on this small, labelled corpus. we use the learned mapping to encode articles found in a larger, unlabelled corpus (140K articles, over 400 mention huntington), find the putamen and caudate nucleus y=-34 y=2 z=-12 z=0

20 Application parkinson (motor cortex, brain stem, thalamus): y=-34 y=2 z=-12 z=0 aphasia (Broca s and Wernicke s areas): y=14 z=14 20 / 22

21 21 / 22 conclusion framework for translating text into spatial pdfs over the brain quantitative validation; voxel-wise encoding is much better than atlases and least-deviations is better than least-squares. allows pooling together text-only studies and doing statistics on their results in brain space yields interesting embeddings of words related to anatomy, cognition and brain diseases

22 22 / 22 conclusion framework for translating text into spatial pdfs over the brain quantitative validation; voxel-wise encoding is better than atlases and least-deviations is (a bit) better than least-squares. allows pooling together text-only studies and doing statistics on their results in brain space yields interesting embeddings of words related to anatomy, cognition and brain diseases Thank you for your attention!