Distinguishing concept categories from single-trial electrophysiological activity

Transcription

1 Distinguishing concept categories from single-trial electrophysiological activity Brian Murphy, Michele Dalponte, Massimo Poesio & Lorenzo Bruzzone CogSci08 25 th July 2008

2 Motivation The long-term question: How can we use evidence about concepts from neural activity to evaluate the theories about conceptual representations developed by linguists, psychologists and philosophers? Immediate research aims: Can an algorithm learn how to distinguish between the electrical activity produced by the brain while processing visual and verbal stimuli about broad conceptual categories? Is this language specific? 2

3 Background Semantic deficit studies have revealed selective impairment in the processing of some categories, including animals, plants, tools, musical instruments, food (Warrington & Shallice 1984; Caramazza & Shelton 1998) Imaging studies have located category specific areas of cortex for classes such as tools, animals, faces, places (Martin et al 1996; Haxby et al, 2001) Theories of distributed conceptual processing (e.g. Pulvermüller 2002; Barsalou 2003) would predict synchronous activity among disparate brain regions (e.g. Tallon-Baudry & Bertrand 1999) 3

4 Background EEG reads synchronous neural activity Fine-grained time course allows neural activity associated with semantic representations to be split from that associated with perception and recognition Previously, only group effects seen in EEG/MEG work (Kiefer, 2001; Paz-Caballero et al., 2006; Gilbert et al, 2008) Machine learning techniques from Brain-Computer- Interaction may be able to mine more information from noisy brain data 4

5 Experimental Design Paradigm: 1. Auditory: visualisation of spoken word with self paced response 2. Visual: silent picture naming with self paced response + o /dɒg/ Response ~1s ~1s + 0.5s 2s 2s Stimuli: Fifty animals; fifty small manipulable objects ( tools ); 27 plants; of varying typicality + + Response ~1s 0.5s 2s 2s 5

6 Concept Images 6

7 Experimental Design Participants: Essex: four native speakers of British English Rovereto: four native speakers of Italian Hardware: 64 electrode extended montage with ear reference; Biosemi and BrainVision at 500Hz + 2s o /dɒg/ Response ~1s ~1s + 0.5s 2s Preprocessing: eye and electrical artefacts removed with EEGLAB ICA Analysis: only examine the EEG activity seen after the participant response (i.e. that associated with the steady state representation) 2s + + Response ~1s 0.5s 2s 7

8 Finding sources among EEG data Images: André Kaup; Choi, Cichocki, Park and Lee Sounds: Te-Won Lee 8

9 Classification Techniques CSSD: Common Subspace Spatial Decomposition Akin to a supervised blind source separation: find me the two sources that are most differentially active for the signals associated with categories A and B Time/frequency window optimization by Dalponte and Bruzzone: find me the portions of the time course and of the spectrum that maximises this distinction 64 channels raw data X channels filtered data Tool component Feature vector Filter by Time, Freq and Eelectr. CSSD Decomposition Vector Transform var( tool ), var( animal ) SupVec Machine Answer? Animal component 9

10 Time/Freq Space Italian Participant A, Auditory Stimulus, Animals vs Objects Frequency, Hz Time, Samples 10

11 Classification Techniques Together, can be interpreted as measure of Event-Related- Synchronisation or Desynchronisation Event-related amplitude measures input to support vector machine for learning 64 channels raw data X channels filtered data Tool component Feature vector Filter by Time, Freq and Eelectr. CSSD Decomposition Vector Transform var( tool ), var( animal ) SupVec Machine Answer? Animal component 11

12 Semantic Spaces English Participant D, Auditory Stimulus, Animals vs. Objects English Participant C, Visual Stimulus, Animals vs. Plants 12

13 Results (5-fold Cross Validation) English Speakers Italian Speakers Part. A Part. B Part. C Part. D Part. A Part. B Part. C Part. D Images, Animal vs Plants 69 % 72 % 72 % 65 % Images, Animal vs Plants 73 % 73 % 66 % - Images, Objects vs Plants 56 % 56 % 62 % 69 % Images, Objects vs Plants 66 % 73 % 71 % - Images, Animals vs Objects 64 % 60 % 62 % 59 % Images, Animals vs Objects 62 % 58 % 63 % 62 % Audio, Animal vs Plants 58 % 69 % 62 % 70 % Audio, Animal vs Plants 77 % 65 % 75 % - Audio, Objects vs Plants 62 % 69 % 62 % 73 % Audio, Objects vs Plants 65 % 66 % 61 % - Audio, Animals vs Objects 59 % 59 % 54 % 73 % Audio, Animals vs Objects 70 % 65 % 50 % - Average optimal windows 15-30Hz, s after offset Average optimal windows 20-30Hz, s after offset 13

14 Conclusions & Further Work A cheap and practical methodology to investigate conceptual organisation in the brain Topological information is not necessary Steady state representations can be distinguised from the process of recognition Single instances and stimuli can be investigated 14

15 Conclusions & Further Work Cross participant: can we train on EEG data from participants A, B and C, and then decode data participant D? Across languages also? Cross modality: can we train on recordings of silent image naming, and decode data from word visualisation? Finer grained analysis: e.g. vehicles vs buildings; dogs vs other animals; conceptual attributes (Mitchell et al, 2008) Teasing apart the time-course of categorisation 15

16 Thanks Thanks to Marco Baroni, Stefano Bertamini, Francesca Bovolo, Lisandro Kaunitz, Heba Lakany, Laura Tonelli

17 Concept Images 17