LETTER. Sensory motor transformations for speech occur bilaterally

Transcription

1 LETTER doi:.38/nture2935 Sensory motor trnsformtions for speech occur bilterlly Gregory B. Cogn, Thoms Thesen 2, Chd Crlson 2 {, Werner Doyle 3, Orrin Devinsky 2,3 & Bijn Pesrn Historiclly, the study of speech processing hs emphsized strong link between uditory perceptul input nd motor production output 4. A kind of prity is essentil, s both perception- nd production-bsed representtions must form unified interfce to fcilitte ccess to higher-order lnguge processes such s syntx nd semntics, believed to be computed in the dominnt, typiclly left hemisphere 5,6. Although vrious theories hve been proposed to unite perception nd production 2,7, the underlying neurl mechnisms re uncler. Erly models of speech nd lnguge processing proposed tht perceptul processing occurred in the left posterior superior temporl gyrus (Wernicke s re) nd motor production processes occurred in the left inferior frontl gyrus (Broc s re) 8,9. Sensory ctivity ws proposed to link to production ctivity through connecting fibre trcts, forming the left lterlized speech sensory motor system. Although recent evidence indictes tht speech perception occurs bilterlly 3, previling models mintin tht the speech sensory motor system is left lterlized,4 8 nd fcilittes the trnsformtion from sensory-bsed uditory representtions to motor-bsed production representtions,5,6. However, evidence for the lterlized computtion of sensory motor speech trnsformtions is indirect nd primrily comes from stroke ptients tht hve speech repetition deficits (conduction phsi) nd studies using covert speech nd hemodynmic functionl imging 6,9. Whether the speech sensory motor system is lterlized, like higher-order lnguge processes, or bilterl, like speech perception, is controversil. Here we use direct neurl recordings in subjects performing sensory motor tsks involving overt speech production to show tht sensory motor trnsformtions occur bilterlly. We demonstrte tht electrodes over bilterl inferior frontl, inferior prietl, superior temporl, premotor nd somtosensory cortices exhibit robust sensory motor neurl responses during both perception nd production in n overt word-repetition tsk. Using non-word trnsformtion tsk, we show tht bilterl sensory motor responses cn perform trnsformtions between speech-perception- nd speech-production-bsed representtions. These results estblish bilterl sublexicl speech sensory motor system. To investigte the sensory motor representtions tht link speech perception nd production, we used electrocorticogrphy (ECoG), in which electricl recordings of neurl ctivity re mde directly from the corticl surfce in group of ptients with phrmcologiclly intrctble epilepsy. ECoG is n importnt electrophysiologicl signl recording modlity tht combines excellent temporl resolution with good sptil locliztion. Criticlly for this study, ECoG dt contin limited rtefcts due to muscle nd movements during speech production compred with non-invsive methods tht suffer rtefcts with jw movement 2. Thus, using ECoG we were ble to investigte directly neurl representtions for sensory motor trnsformtions using overt speech production. Sixteen ptients with subdurl electrodes (see Supplementry Figs nd 2) implnted in the left hemisphere (6 subjects), right hemisphere (7 subjects) or both hemispheres (3 subjects) performed vrints of n overt word repetition tsk designed to elicit sensory motor ctivtions (Fig., Methods nd Supplementry Tble ). We observed increses in neurl ctivity cross the high gmm frequency rnge (6 2 Hz nd bove) with mximl ctivity cross subjects between 7 9 Hz. High gmm ctivity reflects the spiking ctivity of popultions of neurons during tsk performnce 2,2. Individul electrodes showed one of three types of tsk responses: sensory motor (S-M), production (PROD), or uditory (AUD) (Fig. b, see Methods). We found tht AUD ctivity ws generlly loclized to the superior temporl gyrus nd middle temporl gyrus (42 out of 57 electrodes (74%); Fig. 2, b) nd PROD ctivity occurred mostly in the motor nd premotor corticies, somtosensory cortex, nd the inferior prietl lobule (98 out of 24 electrodes (79%); Fig. 2, b), consistent with previous models nd results of speech-perception nd -production studies,2,7. Furthermore, electricl stimultion of PROD electrode loctions resulted in orofcil movements consistent with motor function (see Supplementry b Sensory motor Auditory Production 5 spek Spek Auditory Motor 2 3 Time from uditory onset (s) Frequency (Hz) 2 3 Time from uditory onset (s) mime Mime :=: 2 x Power increse from bseline Figure Behviourl tsks nd exmple neurl ctivtions., 6 Subjects were presented with n uditory consonnt vowel consonnt single-syllble word nd instructed to perform one of three tsks on interleved trils: listen spek (listen to the word, visul prompt, then fter 2-s dely repet the word, visul prompt Spek ); listen mime (listen to the word, visul prompt, then fter 2-s dely, mime speking the word, visul prompt Mime ); listen (pssively listen to the word, visul prompt : 5 : ). Auditory nd motor timelines re shown. b, Exmple time frequency spectrogrms of ECoG ctivity normlized t ech frequency to the bseline power during visul prompt. AUD, significnt ctivity during ech tsk epoch with uditory stimuli; PROD, significnt ctivity during both production epochs; S-M, significnt ctivity during the uditory nd production epochs in listen spek nd listen mime tsks. Center for Neurl Science, New York University, New York, New York 3, USA. 2 Deprtment of Neurology, New York University School of Medicine, New York, New York 6, USA. 3 Deprtment of Neurosurgery, New York University School of Medicine, New York, New York 6, USA. {Present ddress: Medicl College of Wisconsin, Milwukee, Wisconsin 53226, USA. 94 NATURE VOL 57 6 MARCH Mcmilln Publishers Limited. All rights reserved

2 LETTER RESEARCH S2 S3 S4 S5 S6 S7 S8 S9 S S b Left c spek Popultion Right mime Left S-M Individul S S S3 Right S-M S3 S4 Frequency (Hz) S2 x Power increse from bseline Time from uditory onset (s) AUD S5 S6 S6 Sensory motor Sensory motor + uditory Production x Power increse from bseline d Go cue.4 Speech response PROD 3 4 S-M.2 Auditory 2 Time from uditory onset (s) Figure 2 Topogrphy of neurl responses nd bilterl ctivtion., Significnt tsk-relted ctivtions within individul subject brins for left (subjects S3, S4, S7, S8, S, S4), right (S, S2, S5, S6, S9, S2, S5), or both (S, S3, S6) hemispheres. Bilterl coverge is indicted by the light blue box. Electrodes with significnt high gmm ctivity (7 9 Hz) re shown for AUD (green), PROD (blue) nd S-M (red) ctivtions. AUD nd S-M ctivtions (red with green) were often present on the sme electrode. Electrodes without significnt ctivtion re shown in grey. Tringles denote exmple ctivtions from Fig. b, nd squres (S6) denote exmple spectrogrms in Fig. 2c. b, Significnt electrodes projected onto popultion verge left nd right hemispheres, colours s in. Electrode sizes hve been incresed for illustrtive purposes (for ctul sizes see Supplementry Fig. 4). Neurl spectrogrms for exmple S-M electrodes in left nd right hemispheres of S6 during listen spek, listen mime nd listen tsks. d, Popultion verge neurl response profiles for ech clss of electrodes. Shded regions indicte se.m. vlues cross electrodes. Go cue nd verge production response onset re indicted by grey rrows. Fig. 3). Criticlly, contrry to one of the core dogms of brin nd lnguge, S-M ctivity occurred bilterlly in the suprmrginl gyrus, middle temporl gyrus, superior temporl gyrus, somtosensory cortex, motor cortex, premotor cortex nd inferior frontl gyrus (Fig. 2, b, 49 electrodes; see Supplementry Tble 2 nd Supplementry Fig. 4) nd ws observed in ll subjects (Fig. 2). Of the 49 S-M sites, 45 sites showed uditory ctivtion during the listen tsk (Fig. 2, b; Supplementry Figs 4 nd 5; 45 out of 49 electrodes (pproximtely 92%)), suggesting role in speech perception. Hemispheric dominnce s determined by Wd testing did not correlte with the hemisphere of the electrode plcement (x2 (3) 5.92, P 5.34). Importntly, in three subjects with bilterl coverge, S-M ctivity ws present on electrodes in both hemispheres (Fig. 2, c) nd the likelihood of n electrode being S-M site did not differ between hemispheres (Fisher s exct test, P 5.3). These results demonstrte tht S-M ctivity occurs bilterlly. Given the evidence for bilterl S-M ctivity, we performed series of nlyses nd experimentl mnipultions to test the hypothesis tht bilterl S-M ctivity is in fct sensory motor nd represents sensory motor trnsformtions for speech. One concern is tht S-M ctivity is not due to sensory nd motor processes but to sensory ctivtion in both uditory (input) nd production epochs (sound of your own voice). We observed severl convergent lines of evidence tht S-M ctivity reflects both sensory nd motor processing (see Fig. 2d nd Methods). First, S-M sites contin sensory response becuse they responded to uditory stimultion s rpidly s AUD sites (S-M ltency 5 58 ms, AUD 5 64 ms; see Fig. 2d). Second, S-M responses during production re not due to uditory sensory refferent input from hering one s own voice becuse responses were present during the listen mime tsk s well s the listen spek tsk. Third, S-M responses during production re not due to somtosensory refference from moving rticultors becuse S-M ctivity significntly incresed within 248 ms of the production go cue, wheres vocl responses occurred substntilly lter t,2 ms (64 ms s.e.m.). Fourth, S-M production responses contin motor fetures becuse they occurred together with, nd even before, PROD electrode responses (S-M ms, PROD 5 32 ms, Q 5.3; permuttion test; see Methods). Finlly, S-M ctivity ws persistently elevted during the dely period (P 5.; see Fig. 2d, Methods), brodly consistent with plnning ctivity, unlike PROD dely-period ctivity (P 5.64) or AUD dely-period ctivity (P 5.53). These results demonstrte tht S-M ctivity cnnot be simply sensory nd spns both sensory nd motor processes. A relted concern is tht sensory motor trnsformtions re first crried out in the left hemisphere. If so, S-M responses in the right hemisphere could be due to communiction from the left hemisphere. To test this 6 M A R C H 2 4 VO L 5 7 N AT U R E Mcmilln Publishers Limited. All rights reserved

3 RESEARCH LETTER hypothesis, we further exmined ltencies of S-M responses ccording to hemisphere. Response ltencies did not differ significntly in ech hemisphere in either the uditory (right hemisphere, 56 ms; left hemisphere, 82 ms; Q ; permuttion test) or the production epoch (right hemisphere, 272 ms; left hemisphere, 268 ms; Q ; see Methods). Therefore, right hemisphere responses cnnot be due to computtions tht were first crried out in the left hemisphere nd the dt do not support strictly lterlized sensory motor computtions. Another concern is tht S-M ctivity my not reflect speech processing nd my lso be present during simple uditory inputs nd orofcil motor outputs. To test this, we employed tone move tsk in one of the bilterlly implnted subjects (subject 3 (S3); see Methods). We found tht S-M electrodes did not hve significnt sensory motor responses during the tone move tsk (P 5.36, permuttion test; see Supplementry Fig. 6). Thus, S-M ctivity is specific to mpping sounds to gol-directed vocl motor cts nd is likely specific to speech (see Supplementry Discussion.3). Thus fr we hve shown the S-M ctivity is bilterl, sensory motor, nd likely to be specific to speech. However, n importnt open question is whether S-M responses reflect the trnsformtion tht links speech perception nd production nd cn support unified perception production representtionl interfce. A specific concern is tht high gmm ECoG ctivity my pool heterogeneous neurl responses beneth the electrode. S-M responses my combine ctivity from neurons which encode perceptul processes ctive during the uditory cue nd other neurons which encode production processes ctive during the utternce. If this is true, none of the ctivity necessrily reflects sensory motor trnsformtion tht links perception nd production. To be ble to rule out this lterntive nd demonstrte tht S-M responses re involved in sensory motor trnsformtions, we resoned tht two requirements must be met. S-M ctivity must encode informtion bout the content of the underlying speech processes, nd this encoding must reflect trnsformtive coding between the sensory input nd motor output. To test whether S-M ctivity encodes informtion bout speech content, we decoded the neurl ctivity to predict, on ech tril, wht the subjects herd nd sid. We used seven consonnt vowel consonnt words (het, hit, ht, hoot, het, hot nd hut) nd trined sevenwy liner clssifier to decode the neurl responses (see Methods). Individul electrodes only wekly encoded speech content, but when we decoded ctivity pooled cross groups of electrodes, we found tht ll three electrode groups encoded speech tokens (see Fig. 3). AUD electrodes performed best with n verge clssifiction performnce of 42.7% (x 2 () , P ), followed by S-M electrodes, which showed performnce of 33.4% (x 2 () , P ), nd then PROD electrodes, which showed performnce of 27.% (x 2 () 5.5, P ). Furthermore, clssifiction performnce for S-M electrodes did not differ between the two hemispheres (left hemisphere, 29%; right hemisphere, 27%; Fisher s exct test, P 5.5; Fig. 3c). Thus, bilterl S-M ctivity encodes informtion bout the sensory nd motor contents of speech, meeting n importnt requirement for sensory motor trnsformtions. We next sought to test whether S-M ctivity cn link speech perception nd production by trnsforming uditory input into production output. The essentil requirement for trnsformtion is tht neurl encoding of sensory input should depend on subsequent motor output. Previous work hs chrcterized visul motor trnsformtions using trnsformtion tsk in which the sptil loction of visul cue cn instruct motor response to the sme or different sptil loction (the pro nti tsk ) 22,23. Sensory motor neurons in the dorsl visul strem disply different responses to the visul cue depending on the motor contingency, demonstrting role for these neurons in the visul motor trnsformtion 22. Given these predictions from niml neurophysiology, we tested four subjects s they performed n uditory motor trnsformtion tsk (the listen spek trnsformtion tsk) tht employed two non-words (, ) to exmine whether S-M ctivity hs role in trnsformtions for Actul Decoded AUD electrodes PROD electrodes S-M electrodes Clssifiction performnce b Clssifier performnce c Clssifier performnce Cumultive no. of electrodes Cumultive no. of electrodes AUD PROD S-M Chnce Left S-M Right S-M Chnce Figure 3 Neurl decoding of words., Confusion mtrices show proportion identified for seven-wy liner clssifier using neurl responses. AUD electrodes (top), PROD electrodes (middle) nd S-M electrodes (bottom) re shown. The threshold for performnce is t chnce level, P 5.4, for the purposes of displying the electrodes clerly. b, Clssifiction performnce for incresing numbers of electrodes. Chnce performnce is indicted by the dotted line. c, Clssifiction performnce for S-M electrodes in the left nd right hemispheres. Methods present S-M results by response epoch. speech (see Fig. 4A, Supplementry Figs 7 nd 8 nd Methods). This tsk enbled us to hold the sensory nd motor components constnt while mnipulting the trnsformtion process itself in order to mesure how the encoding of this content chnged depending on how perceptul input ws mpped onto production output. The use of non-words insted of words offered other dvntges. Non-words enbled us to exmine sublexicl trnsformtions for speech nd could be designed to differ mximlly in their rticultory dimensions nd their neurl representtions (see Methods nd Supplementry Discussion. nd.2). At lest three models describe how neurl responses encode the tsk vribles. If responses follow strictly sensory model, the encoding will follow the content of the sensory inputs nd confuse tril conditions in which is converted to (R) with trils in which is converted to (R), s well s trils in which is converted to (R) with trils in which is converted to (R) (see Fig. 4B). Conversely, responses tht follow strictly motor model will encode the production outputs, confusing R with R trils nd R with R trils (see Fig. 4Bb). If S-M responses pool responses from sensory nd motor neurons, the encoding will follow the sensory model during sensory input nd the motor model during motor output. In contrst, S-M responses tht reflect the trnsformtion of sensory input into motor output must follow different trnsformtion model nd encode the sensory informtion differently depending on the upcoming motor ct (see Fig. 4Bc). Neurl ctivity displying this property could compute representtionl trnsformtion (see Supplementry Discussion.,.2). If so, responses tht follow trnsformtion model will not confuse tril conditions with either identicl input or identicl output. Consequently, ech of the three models predicted very different ptterns of neurl coding. We constructed liner clssifiers to decode neurl responses. As expected, AUD electrodes in the uditory epoch encoded the uditory input (Fig. 4B, C) nd PROD electrodes encoded the output during NATURE VOL 57 6 MARCH Mcmilln Publishers Limited. All rights reserved

4 LETTER RESEARCH A Mtch Mismtch Mtch Mismtch Spek Spek B Actul High Sensory Motor Trnsformtion Decoded b c Low C Clssifier performnce D K L Divergence (bits) Actul AUD electrodes PROD electrodes S-M electrodes (uditory epoch) (production epoch) (uditory epoch) Decoded b c d Sensory Motor Trnsformtion Chnce b c d S-M electrodes (production epoch) Figure 4 spek trnsformtion tsk. A, In the listen spek trnsformtion tsk, subjects hve to trnsform non-word they her into non-word they spek ccording to simple rule. Subjects were first presented with visul cue: Mtch or Mismtch tht instructed the rule tht determined the non-word to sy in response to the non-word they herd. On mtch trils the rule ws to repet the non-word they herd. On mismtch trils, the rule ws to sy the non-word tht they did not her. The non-words were nd. Subjects then herd one of the two non-words, wited for short dely, then sid the pproprite non-word in response to the Spek cue. There were four tsk conditions: R (her nd sy ); R (her nd sy ); R (her nd sy ); nd R (her nd sy ). B, c, Confusion mtrices predicted by the sensory, motor nd trnsformtion models with high nd low clssifiction scores. C, d, Confusion mtrices during the listen spek trnsformtion tsk. D, d, Model fit quntified using Kullbck Leibler (K L) divergence. the production epoch (utternce; Fig. 4Bb, Cb). However, S-M electrodes chnged their encoding over the course of the tril. During the uditory epoch, S-M electrodes encoded both sensory nd motor conditions concurrently, consistent with the presence of sensory motor trnsformtion (Fig. 4Bc, Cc). Interestingly, during the production epoch, S-M responses no longer encoded the uditory input nd encoded the production output (Fig. 4Cd), suggesting the trnsformtion hs lrgely been computed by tht time. To quntify the comprison of different models, we used the Kullbck Leibler divergence (see Fig. 4D d, Methods). The results were consistent with the response ptterns in the confusion mtrices. We cn lso rule out tht the difference in S-M responses is due to third popultion of neurons tht selectively responds to the cue instructing how perceptul input ws mpped onto production output ( mtch or mismtch ). We rn the sme liner clssifier during cue presenttion nd found tht the S-M responses did not encode the cue (x 2 () 5.8, P 5.78; see Methods). Using direct brin recordings (ECoG) nd overt speech, we demonstrte tht sensory motor system for trnsforming sublexicl speech signls exists bilterlly. Our results re in keeping with models of speech perception tht posit bilterl processing but contrdict models tht posit lterlized sensory motor trnsformtions,6. Our results lso highlight how S-M ctivity during perceptul input reflects the trnsformtion of speech sensory input into motor output. We propose tht the presence of such trnsformtive ctivity demonstrtes unified sensory motor representtionl interfce tht links speech-perception- nd speech-production-bsed representtions. Such n interfce is importnt during speech rticultion, cquisition nd self-monitoring As right hemisphere lesions do not give rise to conduction phsi 9,27 29, our evidence for bilterl sensory motor trnsformtions promotes n interesting distinction between speech nd lnguge: lthough sensory motor trnsformtions re bilterl, the computtionl system for higher-order lnguge is lterlized 5,6 (see Supplementry Fig. 9). This hypothesis invokes strong interfce between sensory-bsed speechperception representtions nd motor-bsed speech-production representtions nd suggests tht deficits for conduction phsi re more bstrct nd linguistic in nture. We propose tht bilterl sublexicl trnsformtions could support unifiction of perception- nd productionbsed representtions into sensory motor interfce 6, drwing distinction between the bilterl perception production functions of speech nd lterlized higher order lnguge processes. METHODS SUMMARY Electrocorticogrphic (ECoG) recordings were obtined from6 ptients ( femles) undergoing tretment for phrmcologiclly resistnt epilepsy. Ech ptient provided informed consent in ccordnce with the Institutionl Review Bord t New York University Lngone Medicl Center. Grid implnttion ws in the left hemisphere (6 subjects), right hemisphere (7 subjects) or both hemispheres (3 subjects). All 6 subjects performed n overt word repetition tsk (listen spek tsk) s well s two control tsks (listen mime tsk 3 nd listen tsk). One subject lso performed tone move tsk. Four subjects lso performed listen spek trnsformtion tsk involving non-words. ECoG recordings were mde using both grid nd strip electrode rrys with 2.3-mm contct size nd -mm spcing. Spectrl nlysis ws performed using 5-ms nlysis windows with 65-Hz frequency smoothing nd stepping size of 5 ms. Neurl responses were defined s high gmm neurl ctivity between 7 nd 9 Hz nd significnce ws ssessed using shuffling procedure. Clssifiction nlyses were crried out using liner discriminnt nlysis with high gmm power spectrl fetures. 24 Mcmilln Publishers Limited. All rights reserved 6 M A R C H 2 4 V O L 5 7 N AT U R E 9 7

5 RESEARCH LETTER Online Content Any dditionl Methods, Extended Dt disply items nd Source Dt re vilble in the online version of the pper; references unique to these sections pper only in the online pper. Received 25 September; ccepted 2 December 23. Published online 5 Jnury 24.. Libermn, A. M., Cooper, F. S., Shnkweiler, D. P. & Studdert-Kennedy, M. Perception of the Speech Code. Psychol. Rev. 74, (967). 2. Libermn, A. M. & Mttingly, I. G. The motor theory of speech perception revised. Cognition 2, 36 (985). 3. Hlle, M. & Stevens, K. N. Speech recognition: model nd progrm for reserch. IEEE Trns. Inf. Theory 8, (962). 4. Hlle, M. & Stevens, K. N. Anlysis by synthesis In (eds Wthen-Dunne, W. & Woods, L. E.) Proceedings of seminr on speech compression nd processing Vol. 2 pper D7 959). 5. Berwick, R. C., Friederici, A. D., Chomsky, N. & Bolhuis, J. J. Evolution, brin, nd the nture of lnguge. Trends Cogn. Sci. 7, (23). 6. Chomsky, N. The Minimlist Progrm (MIT Press, 995). 7. Jkobson, R. Child Lnguge, Aphsi nd Phonologicl Universls (Mouton, 968). 8. Lichtheim, L. On phsi. Brin 7, (885). 9. Wernicke, C. The phsic symptom-complex: psychologicl study on n ntomicl bsis. Arch. Neurol. 22, (97).. Geschwind, N. Disconnexion syndromes in nimls nd mn. I. Brin 88, (965).. Hickok, G. & Poeppel, D. The corticl orgniztion of speech processing. Nture Rev. Neurosci. 8, (27). 2. Price, C. J. The ntomy of lnguge: review of fmri studies published in 29. Ann. NY Acd. Sci. 9, (2). 3. Obleser, J., Eisner, F. & Kotz, S. A. Bilterl speech comprehension reflects differentil sensitivity to spectrl nd temporl fetures. J. Neurosci. 28, (28). 4. Ruschecker, J. P. & Scott, S. K. Mps nd strems in the uditory cortex: nonhumn primtes illuminte humn speech processing. Nture Rev. Neurosci. 2, (29). 5. Hickok, G., Houde, J. & Rong, F. Sensorimotor integrtion in speech processing: computtionl bsis nd neurl orgniztion. Neuron 69, (2). 6. Hickok, G., Okd, K. & Serences, J. T. Are Spt in the humn plnum temporle supports sensory-motor integrtion for speech processing. J. Neurophysiol., (29). 7. Guenther, F. H. Corticl interctions underlying the production of speech sounds. J. Commun. Disord. 39, (26). 8. Wise, R. J. S. et l. Seprte neurl subsystems within Wernicke s re. Brin 24, (2). 9. Crmzz, A., Bsili, A. G., Koller, J. J. & Berndt, R. S. An investigtion of repetition ndlnguge processingin cseofconductionphsi. Brin Lng. 4, (98). 2. Crone, N. E., Sini, A. & Korzeniewsk, A. Event-relted dynmics of brin oscilltions. Prog. Brin Res. 59, (26). 2. Mrkowitz, D. A., Wong, Y. T., Gry, C. M. & Pesrn, B. Optimizing the decoding of movement gols from locl field potentils in mcque cortex. J. Neurosci. 3, (2). 22. Zhng, M. & Brsh, S. Neuronl switching of sensorimotor trnsformtions for ntisccdes. Nture 48, (2). 23. Gil, A. & Andersen, R. Neurl dynmics in monkey prietl rech region reflect context-specific sensorimotor trnsformtions. J. Neurosci. 26, (26). 24. Chng, E. F., Niziolek, C. A., Knight, R. T., Ngrjn, S. S. & Houde, J. F. Humn corticl sensorimotor network underlying feedbck control of vocl pitch. Proc. Ntl Acd. Sci. USA, (23). 25. Oller, D. K., Eilers, R. E. & Oiler, D. K. The role of udition in infnt bbbling the role of udition in infnt bbbling. Child Dev. 59, (988). 26. Agnew, Z. K., McGettign, C., Bnks, B. & Scott, S. K. Articultory movements modulte uditory responses to speech. Neuroimge 73, 9 99 (23). 27. Goodglss, H., Kpln, E. & Brresi, B. Assessment of Aphsi nd Relted Disorders (Lippincott Willims & Wilkins, 2). 28. Dmsio, H. & Dmsio, A. R. The ntomicl bsis of conduction phsi. Brin 3, (98). 29. Benson, D. F. et l. Conduction phsi: clinicopthic study. Arch. Neurol. 28, (973). 3. Murphy, K. et l. Cerebrlres ssocitedwith motorcontrol of speech inhumns cerebrl res ssocited with motor control of speech in humns. J. Appl. Physiol. 83, (997). Supplementry Informtion is vilble in the online version of the pper. Acknowledgements We would like to thnk A. Weiss, J. McArthur nd L. Frnk for developing the dt cquisition hrdwre nd softwre; O. Felsovlyi, E. Londen, P. Purushothmn, L. Melloni, C. Boomhur nd A. Trongnetrpuny for technicl ssistnce; nd D. Poeppel nd C. Brody for comments on the mnuscript. This work ws supported, in prt, by R3-DC475 from the NIDCD, Creer Awrd in the Biomedicl Sciences from the Burroughs Wellcome Fund (B.P.), Wtson Investigtor Progrm Awrd from NYSTAR (B.P.), McKnight Scholr Awrd (B.P.) nd Slon Reserch Fellowship (B.P.). Author Contributions G.B.C. designed the experiment, performed the reserch, nlysed the dt nd wrote the mnuscript. T.T. nd O.D. performed the reserch nd wrote the mnuscript. C.C. nd W.D. performed the reserch. B.P. designed the experiment, performed the reserch, nlysed the dt nd wrote the mnuscript. Author Informtion Reprints nd permissions informtion is vilble t The uthors declre no competing finncil interests. Reders re welcome to comment on the online version of the pper. Correspondence nd requests for mterils should be ddressed to B.P. (bijn@nyu.edu). 98 NATURE VOL 57 6 MARCH Mcmilln Publishers Limited. All rights reserved

6 LETTER RESEARCH METHODS Prticipnts. Electrocorticogrphic (ECoG) recordings were obtined from 6 ptients (6 mles, femles; see Supplementry Tble ) with phrmcologiclly resistnt epilepsy undergoing cliniclly motivted subdurl electrode recordings t the New York University School of Medicine Comprehensive Epilepsy Center. Informed Consent ws obtined from ech ptientin ccordnce with the Institutionl Review Bord t the New York University Lngone Medicl Center. Ptient selection for the present study followed strict criteri: first, cognitive nd lnguge bilities in the verge rnge or bove, including lnguge nd reding bility, s indicted by forml neuropsychologicl testing (see Supplementry Tble ); nd second, norml lnguge orgniztion s indicted by corticl stimultion mpping, when vilble. In ddition, only electrode contcts outside the seizure onset zone nd with norml interictl ctivity were included in the nlysis. Behviourl tsks nd recordings. All prticipnts performed three behviourl tsks: listen spek, listen mime nd listen (Fig. ). Behviourl tsks were performed while prticipnts reclined in hospitl bed. Tsks were controlled by computer plced on the service try on the bed running the Presenttion progrm (NeuroBehviourl Systems). Behviourl udio recordings were either synchronized with the neurl recordings t khz (see below) or recorded on the computer nd referenced to the go cue. For subset of subjects, video cmer with built-in microphone (Sony) ws positioned to monitor subject orofcil movements nd utternces. Video ws stremed to disk (Adobe Premier Pro. (video t frmes per s, nd udio t 44. khz)). Audio-visul nd neurl signls were synchronized video-frme-by-video-frme using n Anlogue-to-Digitl Video Converter (Cnopus). spek, listen mime 3 nd listen tsks were rndomly interleved on tril-by-tril bsis with t lest 4 s between trils. Ech tril begn with visul cue presented, followed by the uditory consonnt vowel consonnt (CVC) token 5 ms lter. We used CVC words composed of the sme consonnts, h nd t, nd different vowels (ht, hit, het, hoot, het, hut, hot). These spoken syllbles spn the vowel spce nd differ in their uditory nd rticultory content. Subjects hd to either listen pssively (listen), repet the syllble fter cue (listen spek) or mime the syllble fter different cue (listen mime, produce the pproprite mouth movements but with no vocl cord vibrtion 3 ; see Supplementry Fig. ). The temporl dely between the uditory cue nd the movement cue ws 2 s. We obtined between 49 nd 66 trils per condition (within subject) nd between 75 nd 334 totl trils per subject. For the tone move tsk (see Supplementry Fig. 6), fter the listen cue ws delivered, 5-ms,,-Hz sinusoidl tone (with -ms on nd off rmps) ws presented. After short, 2-s dely nother visul cue ws presented (move) instructing the subject to move their rticultors (tongue, lips nd jw). For one subject, these trils were rndomly interleved within blocks of the listen spek, listen mime nd listen tsks (see bove). For the listen spek trnsformtion tsk, four subjects (see Supplementry Figs 7 nd 8) were first presented with one of two visul cues: Mtch or Mismtch. After dely, subjects herd one of two non-words: (/kig/) or (/pb/). These non-words were chosen to differ mximlly on their rticultor dimensions: contins velr (bck) voiceless stop consonnt, followed by high front vowel nd finlly velr voiced stop consonnt, nd contins bilbil (front) voiceless stop consonnt followed by bck low vowel nd then bilbil front voiced stop consonnt. The tongue movement therefore goes bck to front to bck for nd front to bck to front for. The reson for choosing mximlly different rticultions ws tht lrger rticultor differences might led to lrger neurl ctivity differences. After short dely (rndomized between.5 nd 2 s), nother visul cue ws presented ( Spek ) to which subjects were to respond by sying the mtch non-word they hd herd if they hd seen the initil mtch cue, or sy the mismtch non-word if they hd seen the mismtch cue. Ech non-word within ech condition ws presented between 63 nd 78 times per subject, with totl trils rnging from 255 to 39 per subject. This control ws crried out in seprte blocks trils tht lternted with blocks of the min listen spek, listen mime nd listen tsks. Surfce reconstruction nd electrode locliztion. To loclize electrode recording sites, pre-surgicl nd post-surgicl T-weighted mgnetic resonnce imging (MRI) scns were obtined for ech ptient nd co-registered with ech other 3. The co-registered imges were then normlized to n MNI-52 templte nd electrode loctions were then extrcted in MNI (Montrel Neurologicl Institute) spce (projected to the surfce) using the co-registered imge, followed by skull stripping 32. A three-dimensionl reconstruction of ech ptient s brin ws computed using FreeSurfer (Fig. 2; S2, S3, S4, S5, S6, S7, S8 nd S (ref. 33). For Supplementry Tble 2, Tlirch coordintes were converted from MNI spce using the EEG/MRI toolbox in Mtlb ( GNU Generl Public License). Neurl recordings nd preprocessing. EEG dt were recorded from intrcrnilly implnted subdurl electrodes (AdTech Medicl Instrument Corp.) in ptients undergoing elective monitoring of phrmcologiclly intrctble seizures. Electrode plcement ws bsed entirely on clinicl grounds for identifiction of seizure foci nd eloquent cortex during stimultion mpping, nd included grid (8 3 8 contcts), depth ( 3 8 contcts) nd strip ( 3 4to3 2 contcts) electrode rrys with -mm inter-electrode spcing centre-to-centre. Subdurl stinless steel recording grid nd strip contcts were 4 mm in dimeter; consequently the distnce between contcts ws 6 mm nd they hd n exposed 2.3-mm dimeter recording contct. For 7 of the 6 subjects, neurl signls from up to 256 chnnels were mplified (3, INA2 Burr-Brown instrumenttion mplifier), bndpss filtered between. 4, Hz nd digitized t khz (NSpike, Hrvrd Instrumenttion Lbortories) before being continuously stremed to disk for off-line nlysis (custom C nd Mtlb code). The front-end mplifier system ws powered by seled led cid btteries (Powersonic) nd opticlly isolted from the subject. After cquisition, neuronl recordings were further low-pss filtered t 8 Hz nd down-smpled offline to 2, Hz for ll subsequent nlysis. For the remining 9 subjects, neurl signls from up to 28 chnnels were recorded on Nicolet One EEG system, bndpss-filtered between.5 25 Hz nd digitized t 52 Hz. In some recordings, modest electricl noise ws removed using line-filters centred on 6, 2 nd 8 Hz (ref. 34). Dt Anlysis. For ctivtion nlysis, time-frequency decomposition ws performed using multi-tper spectrl nlysis 34. The power spectrum ws clculted during 5-ms nlysis window with 6 5 Hz frequency smoothing stepped 5 ms between estimtes. Single trils were removed from the nlysis if the rw voltge exceeded eight stndrd devitions from the cross tril pool, nd noisy chnnels were removed from the nlysis by visul inspection or if they did not contin t lest 6% of the totl trils fter the stndrd devition threshold removl. Sensory motor trnsformtions were defined s ctivity in the gmm rnge (7 9 Hz) tht followed the uditory stimulus s well s the production cue during both listen spek nd listen mime (Fig. b). As the exmple responses illustrte, some electrodes showed consistent increses in ctivity in the high gmm bnd s high s 3 Hz. As the frequency extent vried cross subjects, we chose to focus on the 7 9-Hz frequency rnge s this bnd showed the gretest ctivtion consistently cross ll subjects. Similr results were obtined when broder frequency rnge extending up to 5 Hz ws nlysed. Although the listen mime condition does involve ltering the motor pln (no vocl cord vibrtion), sensory motor ctivtions were bsed on the conjunction of ctivity in both the listen spek nd the listen mime conditions. Any neurl ctivity tht ws specific to the listen mime condition nd not present in norml speking conditions ws therefore excluded (see Supplementry Fig. ). Responses were divided into three types. The first response type, uditory (AUD), ws defined s contining response tht ws seen within ms following the onset of the uditory stimulus in ll three conditions (Fig. b, top). The second response type, production (PROD), ws chrcterized s contining response occurring between 5, ms fter the respond cue in the listen spek nd the listen mime conditions (Fig. b, middle). The lst response type, S-M, contined both post stimulus nd post response cue ctivtion in both the listen spek nd the listen mime conditions (Fig. b bottom). The bseline period ws defined s the 5 ms preceding the uditory stimulus. In Fig. b, the experimentl epoch ws defined s 25 ms (pre) to 3,5 ms post uditory stimulus onset. In Fig. 2c the experimentl epoch ws defined s 25 ms (pre) to 4, ms post uditory stimulus onset. The dditionl 5 ms ws included in Fig. 2c to compenste for slightly lter production responses for tht the subject. Power in ech frequency bnd ws normlized to the power in the bseline period by dividing by the power t ech frequency. As the neurl responses hd vrible onset times but were on verge quite long in durtion, the times were chosen to smple dequtely ll the responses under investigtion. To ssess sttisticl significnce, the verge power cross trils ws tken in two time regions of interest for ech tril within ech condition. For the listen condition, the bseline vlues for ech tril were shuffled with the post uditory vlues, times to crete null distribution. For the listen spek nd the listen mime conditions, both the post-uditory nd the post-production vlues were shuffled, times with the bseline vlues to crete two null distributions. Initil significnce ws ssessed using permuttion test by compring the ctul difference between the post uditory nd post production vlues with the shuffled dt differences 35. To correct for multiple comprisons, for ll subjects, ll three conditions nd both nlysis epochs (listen (post uditory), listen spek (post uditory nd post production) nd listen mime (post uditory nd post production)) were pooled together nd flse discovery rte (FDR) nlysis ws performed with n lph threshold set t.5 (ref. 36). 24 Mcmilln Publishers Limited. All rights reserved

7 RESEARCH LETTER The popultion ltency nlysis ws performed using the bseline-corrected high gmm power response profiles for ech electrode within ech response clss (S-M, AUD nd PROD). The high gmm neurl responses were first bndpss filtered (7 9 Hz) nd then verged within conditions. The listen spek nd listen mime conditions were verged together. As the dt were recorded using two different smpling rtes, the dt were resmpled to 5 Hz smpling rte. To test for ltencies within response clss, the ltencies following either the uditory onset or the go cue were compred ginst the ctivity in the listen condition following the go cue by computing permuted distribution for ech time point. The significnce vlues t ech time point were then corrected for multiple comprisons using FDR set with n lph of.5. The first time point tht ws followed by t lest 2 consecutive significnt time points (4 ms) ws tken to be the ltency of the neurl response. This resulted in four significnt ltency vlues. In the uditory epoch, AUD electrodes hd significnt neurl responses t 64 ms nd S-M electrodes hd significnt responses t 58 ms. During the production epoch, PROD electrodes hd significnt responses strting t 32 ms, wheres S-M electrodes hd significnt responses strting t 248 ms. A similr nlysis ws crried out compring the left S-M electrodes with the right S-M electrodes, which resulted in four more significnt ltency vlues: right hemisphere (uditory 56 ms), left hemisphere (uditory 82 ms), right hemisphere (production 272 ms) nd left hemisphere (production 268 ms). A direct comprison between these ltencies within ech tsk epoch using FDR-corrected shuffle tests (see bove) reveled no significnt results. To ssess whether or not during the uditory nd production epochs, the S-M electrodes disply significntly fster neurl responses thn the AUD nd PROD electrodes, we repeted the permuttion test, except insted of using the comprison of the tsk compred to the condition, we compred the S-M electrodes to the AUD electrodes in the uditory epoch nd the S-M electrodes to the PROD electrodes in the production epoch. The results showed tht wheres S-M nd AUD electrodes did not differ in their ltency vlues during the uditory epoch, S-M electrodes were significntly fster thn PROD electrodes in the production epoch. To test for power differences of the high gmm response (7 9 Hz) cross hemispheres, we performed FDR-corrected permuttion tests. Dt were nlysed by verging 3-ms time window, sliding 5 ms between estimtes. The dt were bseline-corrected (verge 25 ms (pre) to ms pre-stimulus ctivity cross conditions, within electrodes) nd then log-trnsformed before nlysis. For ech condition (listen spek, listen mime nd listen) nd within ech hemisphere (left nd right), we computed the tsk epoch responses by computing the verge of the high gmm response during the uditory epoch (, ms post uditory onset) nd during the production epoch (,5 ms post production cue onset). We then performed series of permuttion tests where we permuted the neurl response cross condition nd/or cross hemisphere, correcting for multiple comprisons using FDR procedure. Only four tests produced significnt results: listen spek versus listen during the production epoch in ech hemisphere, nd listen mime versus listen during the production epoch in ech hemisphere. Furthermore, the neurl responses within ll conditions were not different cross hemispheres (see Supplementry Fig., P..5, FDR corrected). To ssess the significnt dely ctivtion for ech electrode clss, permuttion test ws crried out using filtered dt s listed bove. A permuttion test ws performed for ech electrode clss in which the verge high gmm neurl ctivity of the dely period ( 2 s post uditory onset) ws compred to tht of the bseline period (2 s to 2.5 s pre uditory onset). Although PROD electrodes nd AUD electrodes did not disply elevted popultion neurl ctivity (P 5.64 nd.53, respectively), S-M electrodes hd significntly higher elevted dely ctivity compred to bseline (P 5.; see Fig. 2d). Clssifiction ws performed using the single vlue decomposition (SVD) of the high gmm neurl response (7 6-Hz, 3-ms sliding windows with n overlp of 25 ms) in either the uditory epoch (, ms post uditory onset, AUD electrodes) or the production epoch (,5 ms post go cue, PROD electrodes) or both (S-M electrodes). A liner discriminnt nlysis (LDA) clssifiction ws performed using leve-one-out vlidtion method, in which the trining set consisted of ll the trils of the dt set except the one being tested. Note tht nlysing the different tsk epochs seprtely for the S-M electrodes produced clssifier results tht were lso significntly bove chnce (uditory epoch, 4.2% (x 2 () 5 47, P ), production epoch, 23.2% (x 2 () 5 5.6, P 5.2)). To crete the cumultive curves, the number of electrodes inputted into the clssifier ws incresed linerly. To control for the vribility in tril numbers, the minimum number of trils common to ll subjects nd electrodes ws used. Onehundred itertions for ech number of cumultive electrodes were performed, in which the specific trils nd the specific electrodes were vried rndomly nd the number of SVD components ws equl to the number of electrodes inputted to the clssifier for the AUD nd S-M electrodes, wheres five components were used for the PROD electrodes due to lower number of components present in the PRODelectrode dt. Confusion mtrix scores re simply the proportion of trils clssified s the token on the horizontl xis (decoded) given tht the ctul tril cme from the verticl xis (ctul). Confusion mtrices in Fig. 3 re shown for the lrgest number of cumultive electrodes in ech electrode clss. To nlyse the listen spek trnsformtion tsk responses (Fig. 4), the sme decomposition (SVD) of the neurl signl (7 6 Hz) ws used. Note tht insted of seven-wy clssifier, four-wy clssifier ws used. Confusion mtrices (Fig. 4C) re shown for the lrgest number of cumultive electrodes in ech electrode clss (AUD 5 ; PROD 5 9; S-M 5 8). For the S-M electrodes, ech response epoch (uditory, Fig. 4Cc; production, Fig. 4Cd) ws nlysed seprtely. To mesure the qulity of ech of the models (sensory, motor, sensory motor or chnce; Fig. 4d) we used the Kullbck Leibler divergence, which quntifies the mount of informtion lost in bits when Q (the model) is used to pproximte P (the dt): D KL (PjjQ)~ X P(i) log P(i) i 2 Q(i) where P is the clssifiction percentge for ech ctul or decoded pir (see bove) nd Q is one of the four models: sensory, motor, sensory motor nd chnce. The Kullbck Leibler divergence estimtes the informtion distnce between the pttern of clssifiction errors predicted by ech model, shown in Fig. 4B nd the pttern of clssifiction errors bsed on neurl recordings, shown in Fig. 4C. Smller Kullbck Leibler divergence reflects more informtion bout clssifiction errors nd improved model fit. The sensory model (Fig. 4B) reflects clssifiction scores tht trck the uditory speech input such tht in both the mtch nd the mismtch cses, the sme input will be confused with ech other. Conversely, the motor model (Fig. 4Bb) reflects clssifiction scores tht trck the production output so tht the sme outputs will be confused with one nother. However, the sensory motor model (Fig. 4Bc) will reflect both the input nd output such tht clssifictions for ech of the conditions presented (R, R, R nd R) will be clssified correctly. Finlly, the chnce model will simply reflect chnce performnce in ll cses (.25). Clssifier nlysis of the cue dt ( Mtch versus Mismtch ) in the listen spek trnsformtion tsk ws nlysed on the S-M electrodes for the subjects performing the tsk. The sme liner clssifier ws used s bove, but ws performed during the cue period (, ms post Cue) nd ws two-wy ( Mtch versus Mismtch cues). The results demonstrted tht the clssifiction ws not significnt (men clssifiction %, x 2 () 5.8, P 5.78). Furthermore, using the sme two-wy clssifier between the mtch nd mismtch condition during the uditory epoch ws lso not significnt (men clssifiction %, x 2 () 5.72, P 5.4). Tken together, this indictes tht the sensory motor trnsformtions displyed by these electrodes cnnot be due to third popultion of neurons tht code for the visul cue. 3. Yng, A. I. et l. Locliztion of dense intrcrnil electrode rrys using mgnetic resonnce imging. NeuroImge 63, (22). 32. Kovlev, D. et l. Rpid nd fully utomted visuliztion of subdurl electrodes in the presurgicl evlution of epilepsy ptients. AJNR Am. J. Neurordiol. 26, (25). 33. Dle, A. M., Fischl, B. & Sereno, M. I. Corticl surfce-bsed nlysis. I: segmenttion nd surfce reconstruction. Neuroimge 9, (999). 34. Mitr, P. P. & Pesrn, B. Anlysis of dynmic brin imging dt. Biophys. J. 76, (999). 35. Mris, E., Schoffelen, J.-M. & Fries, P. Nonprmetric sttisticl testing of coherence differences. J. Neurosci. Methods 63, 6 75 (27). 36. Benjmini, Y. & Hochberg, Y. Controlling the flse discovery rte: prcticl nd powerful pproch to multiple testing. J. R. Stt. Soc. B 57, (995). 24 Mcmilln Publishers Limited. All rights reserved