Neural Representations of the Cocktail Party in Human Auditory Cortex

Transcription

1 Neural Representations of the Cocktail Party in Human Auditory Cortex Jonathan Z. Simon Department of Electrical & Computer Engineering Department of Biology Institute for Systems Research University of Maryland ASA May 5, 2014

2 Acknowledgements Grad Students Francisco Cervantes Alex Presacco Krishna Puvvada Past Grad Students Nayef Ahmar Claudia Bonin Maria Chait Marisel Villafane Delgado Kim Drnec Nai Ding Victor Grau-Serrat Ling Ma Raul Rodriguez Juanjuan Xiang Kai Sum Li Jiachen Zhuo Undergraduate Students Abdulaziz Al-Turki Nicholas Asendorf Sonja Bohr Elizabeth Camenga Corinne Cameron Julien Dagenais Katya Dombrowski Kevin Hogan Kevin Kahn Andrea Shome Madeleine Varmer Ben Walsh Collaborators Students Murat Aytekin Julian Jenkins David Klein Huan Luo Past Postdocs Dan Hertz Yadong Wang Collaborators Catherine Carr Monita Chatterjee Alain de Cheveigné Didier Depireux Mounya Elhilali Jonathan Fritz Cindy Moss David Poeppel Shihab Shamma Funding NIH R01 DC NIH R01 DC NIH R01 DC NIH R01 DC NIH R01 AG NIH R01 AG NIH R01 EB NIH R03 DC USDA

3 Introduction Magnetoencephalography (MEG) & Speech Speech as example of Auditory Object Neural Representations of Auditory Objects (e.g., speech) in Auditory Cortex

4 MEG Responses to Speech (up to ~10 Hz) Auditory Model

5 MEG Responses Predicted by STRF Model (up to ~10 Hz) Linear Kernel = STRF Ding & Simon, J Neurophysiol (2012) Spectro-Temporal Response Function

6 Neural Reconstruction of Speech Envelope Speech Envelope Decoder MEG Responses. (up to ~ 10 Hz) stimulus speech envelope reconstructed stimulus speech envelope 2 s Ding & Simon, J Neurophysiol (2012) Zion-Golumbic et al., Neuron (2013) Reconstruction accuracy comparable to single unit & ECoG recordings

7 Auditory Objects What is an auditory object? perceptual (not neural, not acoustic) commonalities with visual objects example: speech stream ( voice ) several formal definitions

8 Auditory Object Definition Griffiths & Warren definition: corresponds with something in the sensory world object information separate from information of rest of sensory world abstracted: object information generalized over particular sensory experiences

9 Auditory Objects at the Cocktail Party Alex Katz, The Cocktail Party

10 Auditory Objects at the Cocktail Party Alex Katz, The Cocktail Party

11 Experiments stationary noise speech noisevocoding competing speech

12 Neural Representation of an Auditory Object neural representation is of something in sensory world when other sounds mixed in, neural representation is of that auditory object, not entire acoustic scene neural representation invariant under broad changes in specific acoustics

13 Selective Neural Encoding

14 Unselective vs. Selective Neural Encoding

15 Selective Neural Encoding

16 Stream-Specific Representation representative subject attending to" speaker 1 attending to" speaker 2 grand average over subjects reconstructed from MEG attended speech envelopes reconstructed from MEG Ding & Simon, PNAS (2013) Identical Stimuli!

17 Single Trial Speech Reconstruction Ding & Simon, PNAS (2013)

18 Invariance Under Acoustic Changes

19 Independence from Acoustic Specifics Neural Response vs. Acoustics Object-Based correlation.2 attended.1 background Speaker Relative Intensity (db) ` correlation.2.1 Neural Results attended background Stimulus- Based correlation.2.1 attended background Speaker Relative Intensity (db) Speaker Relative Intensity (db) Stream-based not stimulus-based" Neural representation is invariant to acoustic changes.

20 Neural Representation of an Auditory Object neural representation is of something in sensory world when other sounds mixed in, neural representation is of auditory object, not entire acoustic scene neural representation invariant under broad changes in specific acoustics

21 Forward STRF Model Spectro-Temporal Response Function (STRF)

22 STRF Results frequency (khz) Attended time (ms) Background time (ms) TRF background STRF separable (time, frequency)" 300 Hz - 2 khz dominant carriers" M50STRF positive peak" M100STRF negative peak attended M100STRF strongly modulated by attention, but not M50STRF

23 Neural Sources M100STRF source near (same as?) M100 source: Planum Temporale" " M50STRF source is anterior and medial to M100 (same as M50?): Heschl s Gyrus anterior posterior Left M50STRF M100STRF M100 medial Right 5 mm

24 Ding & Simon, J Neuroscience (2013) Speech in Noise

25 Cortical Representation of Speech in Noise Neural Reconstruction of Underlying Speech Envelope +6 db 1 s -6 db C Correlation with Intelligiblity B correlation Reconstruction Accuracy Q SNR (db) reconstruction accuracy intelligiblity (%)

26 Noise-Vocoded Speech in noise = +3 db SNR Ding, Chatterjee & Simon, NeuroImage (2014)

27 Noise-Vocoded Speech: Results Cortical entrainment to natural speech robust to noise" Cortical entrainment to vocoded speech is not" Not explainable by passive envelope tracking mechanisms" - noise vocoding does not directly affect the stimulus envelope

28 Noise-Vocoded Speech: Results

29 Summary Cortical representations of speech found here: consistent with being neural representations of auditory perceptual objects very robust to noise (~intelligibility) relies on spectro-temporal fine structure explicitly temporal representation Object representation at 100 ms latency (PT), but not by 50 ms (HG)