What s wrong with models of ventral temporal cortex?

Size: px

Start display at page:

Download "What s wrong with models of ventral temporal cortex?"

Loraine Atkins
5 years ago
Views:

1 What s wrong with models of ventral temporal cortex? UCLA, July 2016 Jim Haxby Center for Cognitive Neuroscience, Dartmouth College Center for Mind/Brain Sciences (CIMeC), University of Trento

2 Neural representation of faces, bodies, and objects in ventral temporal cortex UCLA, July 2016 Jim Haxby Center for Cognitive Neuroscience, Dartmouth College Center for Mind/Brain Sciences (CIMeC), University of Trento

Students, postdocs, collaborators Analysis of

biological classes Andy Connolly Research

Hyperalignment Swaroop Guntupalli Post-doctoral

Person perception Dylan Wagner Assistant

3 Students, postdocs, collaborators Analysis of similarity structure, representation of biological classes Andy Connolly Research assistant professor (Geisel School of Medicine) Hyperalignment Swaroop Guntupalli Post-doctoral fellow Attention Sam Nastase Graduate student Person perception Dylan Wagner Assistant professor (Ohio State) NeuroDebian Yaroslav Halchenko Research scientist Action representation, computational methods Nick Oosterhof Post-doctoral fellow (CIMeC) 3

4 Neural representation of faces, bodies, and objects in ventral temporal cortex Optimized for person perception Face processing is fast and requires minimal attention and awareness Mediated by a distributed neural system in humans and monkeys A fly in the ointment Stimulus sampling bias Representation of agentic action plays a dominant role Inadequate computational methods for data analysis and modeling Modeling functional architecture as a high-dimensional space captures complexity better than category-specific regions

5 Neural representation of faces, bodies, and objects in ventral temporal cortex Optimized for person perception Face processing is fast and requires minimal attention and awareness Mediated by a distributed neural system in humans and monkeys A fly in the ointment Stimulus sampling bias Representation of agentic action plays a dominant role Inadequate computational methods for data analysis and modeling Modeling functional architecture as a high-dimensional space captures complexity better than category-specific regions

6 Ultrarapid face detection (Crouzet, Kirchner, Thorpe, 2010) Face neurons are faster (Bell et al. 2011) Very rapid detection of familiar faces (Visconti di Oleggio Castello, Gobbini, et al. 2015)

7 Early face-selective neural responses are not affected by attention (Furey et al., 2006) Selective attention to faces (FATTN) or houses (HATTN) 7

8 Upright faces break through interocular suppression faster than inverted faces (Jiang, Costello, He, 2007) 8

9 2350 Faces in full view break through interocular suppression faster than faces in profile (Gobbini et al., Consc Cogn, 2013) Time in ms ± SE profile (averted gaze) profile (direct gaze) full face (averted gaze) full face (direct gaze) and personally familiar faces break through faster than faces of strangers (Gobbini et al. PLoS One, 2013) 9

Distributed neural system for face perception in humans and monkeys Extended System Person knowledge

mental states Top-down modulatory feedback Anterior temporal cortex Biographical knowledge Core System

Superior Temporal Sulcus Posterior STS Dynamic features of facial gesture Inferior occipital & fusiform

Insula Striatum / reward system Lateral Fusiform Gyrus (FFA) Haxby et al.

10 Distributed neural system for face perception in humans and monkeys Extended System Person knowledge Anterior paracingulate Personal traits, aitudes, mental states Temporoparietal juncfon IntenFons, mental states Top-down modulatory feedback Anterior temporal cortex Biographical knowledge Core System Precuneus/Posterior cingulate Episodic memories Visual appearance Inferior Occipital Gyrus (OFA) Superior Temporal Sulcus Posterior STS Dynamic features of facial gesture Inferior occipital & fusiform gyri Invariant features for idenfficafon EmoBon Amygdala. Insula Striatum / reward system Lateral Fusiform Gyrus (FFA) Haxby et al. 2000; Gobbini & Haxby, 2007; Haxby & Gobbini, 2011 Motor simulabon Inferior parietal & frontal operculum Facial Expression IPS & Frontal Eye Fields Eye gaze & spafal atenfon 10 Tsao et al

11 Neural representation of faces, bodies, and objects in ventral temporal cortex? Optimized for person perception Face processing is fast and requires minimal attention and awareness Mediated by a distributed neural system in humans and monkeys A fly in the ointment Stimulus sampling bias Representation of agentic action plays a dominant role Inadequate computational methods for data analysis and modeling Modeling functional architecture as a high-dimensional space captures complexity better than category-specific regions

12 A fly in the ointment

13 A fly in the ointment

14 Neural representation of faces, bodies, and objects in ventral temporal cortex? Optimized for person perception Face processing is fast and requires minimal attention and awareness Mediated by a distributed neural system in humans and monkeys A fly in the ointment Stimulus sampling bias Representation of agentic action plays a dominant role Inadequate computational methods for data analysis and modeling Modeling functional architecture as a high-dimensional space captures complexity better than category-specific regions

15 What s wrong with current models of representation of faces, bodies, and objects in ventral temporal cortex? Serious stimulus sampling bias - Still images - Limited range of categories Inadequate computational methods for data analysis and modeling

16 What s wrong with current models of representation of faces, bodies, and objects in ventral temporal cortex? Serious stimulus sampling bias - Still images - Limited range of categories Inadequate computational methods for data analysis and modeling - Univariate contrasts - Wrong question: What is the function of an area?

17 What s wrong with current models of representation of faces, bodies, and objects in ventral temporal cortex? Serious stimulus sampling bias - Still images - Limited range of categories Inadequate computational methods for data analysis and modeling - Univariate contrasts - Wrong question: What is the function of an area?

18 What s wrong with current models of representation of faces, bodies, and objects in ventral temporal cortex? Serious stimulus sampling bias - Still images - Limited range of categories Inadequate computational methods for data analysis and modeling - Univariate contrasts - Wrong question: What is the function of an area?

19 Social Animations (Heider-Simmel)

20 Social Animations (Castelli et al. 2000) Social vs random animations

21 Extensive activation of perceptual and social brain systems by social animations (Gobbini et al., 2007)

22 Social animations Fusiform activations without faces or bodies (Gobbini et al. 2007) (see also Schultz et al. 2003) Biological motion (point-light displays)

23 Goal-directed actions performed by non-human-like robots activate the FFA (Shultz & McCarthy, 2012) 23

24 Goal-directed actions performed by non-human-like robots activate the FFA (Shultz & McCarthy, 2012) 24

25 Representation of action in the cortex for perception of faces and animate stimuli

26 Study of the effect of attention on the geometry of representations of animal taxa and animal behaviors (Sam Nastase, biorxiv, 2016; under review)

31 Study of the effect of attention on the geometry of representations of animal taxa and animal behaviors (Sam Nastase, biorxiv, 2016; under review)

32 Regional representational geometry was modeled as a weighted sum of RDMs for animal taxa and action categories Nastase et al. biorxiv, 2016; under review

33 Representational geometry is dominated by perceived behavior/action, even in ventral temporal cortex Nastase et al. biorxiv, 2016; under review

34 Representational geometry is dominated by perceived behavior/action, even in ventral temporal cortex Nastase et al. biorxiv, 2016; under review

35 Responses to dynamic natural movies in monkeys is B.E. Russ, D.A. Leopold / NeuroImage 109 (2015) dominated by motion (Russ & Leopold, 2015) B.E. Russ, D.A. Leopold / NeuroImage 109 (2015) Fig. 3. Inﬂated surface maps exhibiting the correlation between a given feature time course and the average brain activity in response to 15 movies maps represent the locations of the fundi of major sulci on the macaque brain. The gray-scale surface map depicts the surface anatomy from subject M labels for the various sulci (LUS: lunate sulcus; ios: inferior occipital sulcus; IPS: intraparietal sulcus; CS: central sulcus; STS: superior temporal sulcus PS: principal sulcus). Black arrows in the faces surface map show the approximate locations of the face patches (see Fig. 2 legend for label names). D signiﬁcant correlation at p =.05, Bonferroni corrected. howing the percent variance explained by the motion (top) and contrast (bottom) feature models to natural movies that contained animals interacting ent. Right column. Surface maps showing the percent variance explained by the motion (top) and contrast (bottom) feature models to natural movies are shown in Supplemental mapping of the facewas patches under natural d explained variance This for motion in the nonsocial movies found in both subjects tested. viewing demon- strates that it is possible to use this approach to gauge aspects of the brain's functional organization despite multiple stimulus features including in the face patches. In the follow- Fig. 2) and i contribution of the different features in o (Subject M1 shown in Supplemental Fig. movement is an important visual feature in the ventral visual pathway.

36 Responses to dynamic natural movies in monkey face patches is dominated by motion (Russ & Leopold, 2015)

37 Responses to dynamic natural movies in monkeys is dominated by animal motion (Russ & Leopold, 2015)

38 Lateral fusiform activity evoked by voice perception in the congenitally blind (Fairhall & Gobbini, under review) EmoFon classificafon Accuracy > chance (25%) lffa rffa Blind Sighted Blind Sighted ATenFon to: emot id emot id emot id emot id

39 Lateral fusiform gyrus activity is evoked or modulated by: Moving triangles Moving points of light Industrial robots Animal behavior more than animal form Voices in the congenitally blind Stimulus sampling bias for still images of faces, bodies, and objects has restricted understanding of the functional architecture of ventral temporal cortex.

40 Lateral fusiform gyrus activity is evoked or modulated by: Moving triangles Moving points of light Industrial robots Animal behavior more than animal form Voices in the congenitally blind So, what is represented in the lateral fusiform gyrus? Animate as compared to inanimate entities? (Kriegeskorte, Mahon, Caramazza, Grill-Spector, and many more) Agency (Gobbini et al. 2007, 2011)

41 The representation of animacy in the ventral temporal lobe Is the animate-inanimate distinction a major large-scale feature?

42 Is the animate-inanimate distinction a major large-scale feature that distinguishes lateral from medial ventral temporal cortex O Toole et al. 2004

43 Is the animate-inanimate distinction a major large-scale feature that distinguishes lateral from medial ventral temporal cortex Kiani et al. 2007

44 Is the animate-inanimate distinction a major large-scale feature that distinguishes lateral from medial ventral temporal cortex Kriegeskorte et al. 2008

45 Is the animate-inanimate distinction a major large-scale feature that distinguishes lateral from medial ventral temporal cortex Grill-Spector & Weiner, 2014

46 But Animate stimuli are usually human, mammal, or avian Broader sampling suggests this distinction does not apply to all animate stimuli Primates vs bugs Connolly et al Faces vs objects

47 The Animacy Continuum: Low animacy animals have neural response vectors that overlap with vectors for bona fide objects Sha et al. 2015

48 The Animacy Continuum: Topography for high versus low animacy mirrors primates versus objects (animate vs inanimate) Uncorrecte Sha et al. 2015

49 Animate-Inanimate dichotomy does not survive broadening stimulus sampling to low animacy animals The animate-inanimate dichotomy is a special case of a broader principle the Animacy Continuum. c.f. Aristotle, De Anima; animacy hierarchies in linguistics; the Great Chain of Being

50 Animate-Inanimate dichotomy does not survive broadening stimulus sampling to low animacy animals The animate-inanimate dichotomy is a special case of a broader principle the Animacy Continuum. c.f. Aristotle, De Anima; animacy hierarchies in linguistics; the Great Chain of Being So Is the Animacy Continuum (varying levels of agentic complexity) the dominant organizing principle in VT cortex?

51 Representation of agents Gobbini et al these results suggest that the fusiform gyrus plays a general role in the representation of visual stimuli that signify agency, independent of visual form. Gobbini et al These results suggest that these areas mediate perception of agency, independently of whether the agents are living or not. Shultz and McCarthy, 2014 While decades of research have demonstrated that a region of the right fusiform gyrus (FG) responds selectively to faces, a second line of research suggests that the FG responds to a range of animacy cues, including biological motion and goal-directed actions, even in the absence of faces or other human-like surface features. These findings raise the question of whether the FG is indeed sensitive to faces or to the more abstract category of animate agents [Our] results suggest that the FG does not respond to all faces in a category-specific way, and is instead especially sensitive to whether an entity is animate.

52 Is the animacy continuum (varying levels of agentic complexity) the dominant organizing principle for representation in ventral temporal cortex? Not exactly Time for a digression to reframe the question Methods for modeling a high-dimensional representational space based on a set of basis functions (tuning profiles, connectivity profiles, and topographic components) that are shared across brains Hyperalignment

53 Subject 1 Subject 2 53

response pattern vectors in individual 3D

54 Broad sampling of a neural representational space with a movie Subject 1 Subject 2 Response patterns in cortex 15 response pattern vectors in individual 3D representational spaces (full exp t has >2600 vectors in >50,000D space)

55 Individual representational spaces Procrustes transformations (improper rotations) Common model representational space S1 = S2 x [ ] =

56 Individual representational spaces Procrustes transformations (improper rotations) Common model representational space S1 = S2 x [ ] s2 = S3 x [ ] s3 =

57 >30 dimensions (42?) are necessary to account for the representational space in VT 80 Between-subject classification accuracy ± SE Number of Principal Components 57

58 Topographies for response patterns are modeled in different brains as weighted sums of individual-specific topographic basis functions using the same weights for common model dimensions PC1 PC2 PC3 PC4 PC5 PC6 PC35 1 => => 2 eights: ß 1 ß 2 ß 3 ß 4 ß 5 ß 6... ß 35 (ß*D)

59 The topographic basis functions for PCs individually show little correspondence to category-selective face and place areas or the domain-specific divisions for animate and inanimate stimuli PC1 PC2 PC3 PC4 PC5 PC6 PC35 1 => 2 => eights: ß 1 ß 2 ß 3 ß 4 ß 5 ß 6... ß 35 (ß*D)

60 Individual variability in the location of the FFA is modeled by VT model topographies for face vs object dimension with high fidelity Subject 1 Subject 2 60

61 But this dimension accounts for only a small portion of variance in responses to the movie 70 Variance accounted for (%) D common model 1D model (face versus object LD) Model dimensions Face vs object LD accounts for 7% of variance, <20% of VAF by first 3 PCs & <13% of VAF by 35 PCs 61

62 The animacy-continuum/face-object dimension accounts for more variance in responses to still images of animals and objects (Sha et al. 2015) ed Proof 62

classification accuracies (Mean ± SE) 80 70 60 50

63 but only carries information about coarse distinctions lower order dimensions carry finer distinctions Pairwise classification accuracies (Mean ± SE) between-class 1st PC within-class 2nd-11th PCs Sha et al

64 Is the animacy continuum (varying levels of agentic complexity) the dominant organizing principle for representation in ventral temporal cortex? Not exactly The animacy continuum and the face-object distinctions account for only a small portion of variance in the responses to a rich, natural, dynamic stimulus The animacy continuum and face-object distinctions account only for coarse, not fine-grained distinctions But the dimensions for these distinctions are derived from responses to still images. Dimensions based on responses to agentic behavior may tell a different story (cf. Nastase et al.)

65 Oversampling of still images from limited variety of categories (faces, bodies, mammals, birds, etc.) Summary 1: Stimulus sampling bias versus

66 Summary 1 Category selective regions have more complex tuning functions that suggest they play more diverse roles in person perception Representational geometry is dominated by representation of behavior, not form The animate-inanimate distinction is not the principal dimension that characterizes the coarse lateral-to-medial topography in ventral temporal cortex

67 Domain specificity! (animate vs inanimate) Summary 2: Inadequate computational methods Animacy continuum! (human to bug) Foveal versus peripheral vision! Categoryselective regions! It s expertise! Stimulus size! 67

68 Summary 2: Inadequate computational methods Univariate statistics - Limiting analysis to univariate contrasts leads to search for the function of an area Multivariate methods (pattern classification, RSA, hyperalignment, etc.) - Model the functional architecture of an area as a high-dimensional representational space - The topographies for dimensions are multiplexed and overlapping - Accounts for category-selective regions and finegrained patterns that carry fine distinctions

69 Thank you

Software for ROI hyperalignment and data are on PyMVPA (www.pymvpa.

release of 7T fmri with natural stimulus and lots more: data website:

70 Software for ROI hyperalignment and data are on PyMVPA ( Yaroslav Halchenko & Michael Hanke, developers New massive data release of 7T fmri with natural stimulus and lots more: data website: paper: Hanke et al. (2014) Nature Scientific Data, 1:

71 Grill-Spector & Weiner, Nature Reviews Neuroscience, 2014

76 Thank you

CISC 3250 Systems Neuroscience

CISC 3250 Systems Neuroscience Levels of organization Central Nervous System 1m 10 11 neurons Neural systems and neuroanatomy Systems 10cm Networks 1mm Neurons 100μm 10 8 neurons Professor Daniel Leeds