Coding of Object Size and Object Category in Human Visual Cortex

Transcription

1 Cerebral Cortex, June 2017;27: doi: /cercor/bhw150 Advance Access Publication Date: 1 June 2016 Original Article ORIGINAL ARTICLE Coding of Object Size and Object Category in Human Visual Cortex Joshua B. Julian, Jack Ryan, and Russell A. Epstein Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA Address correspondence to Russell A. Epstein, 3720 Walnut Street, Philadelphia, PA 19104, USA. epstein@psych.upenn.edu Abstract A salient aspect of objects is their real-world size. Large objects tend to be fixed in the world and can act as navigational barriers and landmarks, whereas small objects tend to be moveable and manipulable. Previous work has identified regions of visual cortex that respond differentially to large versus small objects, but the role of size in organizing representations of object categories has not been fully explored. To address this issue, we scanned subjects while they viewed large and small objects drawn from 20 categories, with retinotopic extent equated across size classes. Univariate analyses replicated previous results showing a greater response to large than small objects in scene-responsive regions and the converse effect in the left occipitotemporal sulcus. Critically, multivariate analyses revealed organization-by-size both within and across functional regions, as evidenced by activation patterns that were more similar for object categories of the same size than for object categories of different size. This effect was observed in both scene- and object-responsive regions and across high-level visual cortex as a whole, but not in early visual cortex. We hypothesize that real-world size is an important dimension for object category organization because of the many ecologically significant differences between large and small objects. Key words: parahippocampal place area, multivoxel pattern analysis, neuroimaging, perception, scene-selective cortex Introduction A central endeavor of cognitive neuroscience is understanding how high-level object representations are organized in the brain. It is well established that some regions of human visual cortex respond preferentially to certain classes of stimuli, such as faces, bodies, and scenes (Kanwisher and Dilks 2013). Moreover, selectivity gradients have been observed across visual cortex, such as an animate inanimate gradient across the medial lateral axis of the ventral pathway (Martin 2007; Grill-Spector and Weiner 2014; Sha et al. 2015). Although the reliability of these results is not in question, their interpretation is a matter of considerable debate. Whereas some researchers argue that category-selective regions act as individual modules dedicated to recognition of specific stimulus classes (Kanwisher 2010), others contend that they are merely hotspots within a single integrated object recognition system (Haxby et al. 2001). Moreover, even among those who accept functional specialization, there is much discussion about the correct characterization of each region s computational role (Tarr and Gauthier 2000; Epstein 2005; Kanwisher and Yovel 2006; Aminoff et al. 2013). The continuance of these debates has led some researchers to explore novel principles that might underlie the organization of high-level representations that mediate visual recognition. Among these principles, one of the most interesting is the idea that object representations might be organized in the occipitotemporal cortex based on the real-world size of the objects. In a recent report, Konkle and Oliva (2012) provided evidence for such macro-scale organization, by showing that large objects (e.g., vehicles, pianos, furniture) tend to activate areas that are more medial in the ventral occipitotemporal cortex, whereas small objects (e.g., coffee cups, coins, paperclips) tend to activate areas that are more lateral. They also observed a similar organization though spatially mirrored on the lateral cortical surface. Notably, the medial areas responding to large objects on the ventral surface include regions that have been previously shown to respond preferentially to scenes, buildings, and other landmarks, while the more lateral areas responding to small objects include The Author Published by Oxford University Press. All rights reserved. For Permissions, please journals.permissions@oup.com

2 3096 Cerebral Cortex, 2017, Vol. 27, No. 6 regions that have been previously shown to respond more to faces, bodies, animals, and artifacts. Thus, the organization-bysize that is observed for special object classes (e.g., scenes vs. faces) is echoed in the responses elicited by common everyday objects. Konkle and Oliva (2012) consider several reasons why size might be an important organizing principle for object representations. Large and small objects tend to have different shapes and material properties (Cant and Goodale 2011), and large objects are typically perceived at greater distances (Cate et al. 2011; Amit et al. 2012) and in more peripheral retinal positions (Levy et al. 2001) than small objects. Moreover, we interact with large objects and small objects in different ways: we walk around large objects, while we pick up small objects with our hands (Bainbridge and Oliva 2015). Related to the latter distinction, real-world size is predictive of whether an object is relevant for spatial navigation or not large objects are more likely to be fixed in space and visually salient, and thus make better landmarks than small objects (Auger et al. 2012; Epstein and Vass 2014). Thus, it is satisfying that brain regions such as the PPA that have been previously implicated in processing of navigationally relevant stimuli such as scenes and buildings respond more strongly to large objects than to small objects. Indeed, in a recent study focusing on scene areas, we replicated and extended Konkle and Oliva s (2012) results by showing that the response in scene regions (i.e., parahippocampal place area PPA, retrosplenial complex, and occipital place area) is strongly modulated by a number of characteristics that are indicative of an object s value as a potential landmark, including not only physical size, but also fixedness, spatial definition (i.e., the extent to which the object defines the space around it), placeness (i.e., the extent to which the object is interpreted as a place rather than a thing), typical viewing distance, and consistency of contextual associations (Troiani et al. 2014) (see also Bar and Aminoff 2003; Mullally and Maguire 2011; Amit et al. 2012). Notably, these characteristics tend to be highly correlated with each other. Thus, physical size might be an organizing principle for object representations because it is a proxy for the salient practical distinction between landmarks and nonlandmarks. Despite these intriguing results, however, there are a number of critical aspects of the relationship between real-world size and object coding that have not yet been investigated. Most notably, all of the studies mentioned above examined univariate rather than multivariate responses. This is an important lacuna because the univariate response of a brain region can provide an incomplete picture of the representational space that it supports, which will likely be multidimensional rather than unidimensional. As an example of this general point, consider that primary visual cortex responds more strongly to high-contrast than lowcontrast edges, but this observation tells us little about how this region codes the orientation of the edges or how differences in contrast affect the orientation code. In the case of size, we know that several brain regions respond differentially to large versus small objects, but this finding tells us little about how (or whether) real-world size affects the object category representations supported by these regions. For example, the fact that scene regions such as the PPA respond more strongly to large than small objects could mean 1) that they code categorical differences for large objects but not for small objects; 2) that they code large objects and small objects with equal fidelity but simply respond more to the large objects; 3) that they code object size but not category, in which case they should distinguish large objects from small objects but not discriminate between object categories of the same size. Moreover, although the idea that size may be an organizing principle for category distinctions at the macro level was implied by Konkle and Oliva s results showing gradients of small-to-large response preferences along the lateral and ventral occipitotemporal surfaces that matched coarse differences between stimulus classes such as faces and scenes, their study did not directly examine how this organization-by-size interacted with organization-by-category for more fine-grain categorical distinctions. To address these kinds of representational questions, we need to know how real-world size affects the distinctions between objects revealed by multivoxel codes, which is not a question that was addressed in previous studies. With these issues in mind, we scanned subjects in a rapid event-related design while they viewed objects drawn from 20 basic object categories. Ten of the categories were objects whose real-world size was large, while 10 were objects whose real-world size was small. Retinal size was matched across the large and small objects, as were other low-level visual features. We then analyzed multivoxel patterns elicited by each object category with the aim of answering 2 questions: 1) to what extent do these patterns reliably distinguish between object categories? 2) to what extent are these object category distinctions organized by real-world size? To anticipate, our data indicate that real-world size is an important organizing principle for object representations throughout high-level human visual cortex, including both scene-responsive and objectresponsive regions. Methods Subjects Sixteen subjects (9 females; mean age, 24.3 ± 3.8 years) were recruited from the University of Pennsylvania community. All subjects were healthy and right-handed, had normal or corrected-to-normal vision, and provided written informed consent in compliance with procedures approved by the University of Pennsylvania Institutional Review Board. MRI Acquisition Scanning was performed at the Hospital of the University of Pennsylvania using a 3T Siemens Trio scanner equipped with a 32-channel head coil. High-resolution T 1 -weighted images for anatomical localization were acquired using a 3-dimensional magnetization-prepared rapid-acquisition gradient-echo pulse sequence (repetition time [TR], 1620 ms; echo time [TE], 3.09 ms; inversion time, 950 ms; voxel size, mm; matrix size, ). T 2 *-weighted images sensitive to blood oxygenation level-dependent contrasts were acquired using a gradient-echo echoplanar pulse sequence (TR, 3000 ms; TE, 30 ms; flip angle 90 ; voxel size, mm; field of view, 192; matrix size, ). Visual stimuli were displayed by rear-projecting them onto a Mylar screen at pixel resolution with an Epson LCD projector equipped with a Buhl long-throw lens. Subjects viewed the images through a mirror attached to the head coil. Images subtended a visual angle of approximately Stimuli Stimuli were 440 digital color photographs drawn from 20 different object categories, with 22 unique exemplars per category (Fig. 1). Ten of the categories were large objects, and 10 categories were small objects. Large objects were roughly body-sized and

3 Representation of Object Size and Category Julian et al Figure 1. Examples of stimuli used for each of the 20 object categories. included arm chairs, benches, cabinets, copiers, pianos, refrigerators, sofas, stoves, treadmills, and washing machines. Small objects were roughly hand-sized and included sports balls, binoculars, coffee makers, small coolers, handheld vacuums, hats, mobile phones, mugs, shoes, and staplers. Each image was pixels and contained a single object presented on a gray background with the longest axis of the object extending to the edge of the image. Large and small object stimuli were matched on the following low-level visual properties: Chrominance and Luminance Chrominance and luminance values were calculated for each image by first converting the RGB images to HSV (hue, saturation, and value). Chrominance was calculated by averaging the pixelwise values of the saturation channel in each image, and luminance was calculated by averaging the pixel-wise values of the value channel in the image. These image-wise values were then averaged across all exemplars for each category, and large and small object categories were compared. Large and small objects did not differ in chrominance (t (18) = 1.03, P = 0.32) or luminance (t (18) = 0.63, P = 0.54). Pixel Count We calculated the number of pixels occupied by the object in each image (i.e., pixels that were assigned to the object rather than to gray background). There was no significant difference in pixel count between large and small object categories (t (18) = 0.01, P = 0.99). This indicates that large objects and small objects had equivalent retinotopic size. Spatial Envelope (GIST) Spatial envelope reflects the coarse scale shape information, or GIST, of an image. To compare GIST between the large and small object categories, we computed the power of different spatial frequencies across the image to construct a GIST descriptor for each stimulus in our set (Oliva and Torralba 2001). We then calculated the distance in GIST space between images corresponding to different object categories. Large object categories were no more dissimilar in GIST than small object categories (t (88) = 1.62, P = 0.11). Note that although large and small objects were matched on each of these low-level properties, object categories of the same sizes were not matched. One-way ANOVAs confirmed significant differences between the object categories of the same size in terms of luminance, chrominance, and pixel count separately for both large and small objects (all Fs > 8.05, all Ps < 0.001). Moreover, as one would expect, stimuli had more similar GIST descriptors when they were of the same object category than when they were of different categories of the same size (both ts > 5.46, both Ps < 10 5). Procedure: fmri Experiment Subjects were scanned while viewing all 440 images of 10 large and 10 small objects, shown one at a time without repetition.

4 3098 Cerebral Cortex, 2017, Vol. 27, No. 6 Each image was presented for 1000 ms followed by a 2000 ms gap before the presentation of the next stimulus. Testing sessions were divided into 4 scan runs, each consisting of 110 stimulus trials and 11 null trials during which the subject viewed a blank screen for 6 s (total length 7 m 18 s per run). Subjects viewed stimuli from 5 large and 5 small categories chosen at random in runs 1 and 3 and stimuli from the other 5 large and 5 small categories in runs 2 and 4. Trials within each scan run were ordered according to a continuous carryover sequence (Aguirre 2007) so that each object category preceded and followed every other object category, including itself, exactly once. A unique carryover sequence was used for each subject and scan run. The specific exemplar used on each trial was randomly chosen, subject to the constraint that exemplars could not repeat (i.e., randomwithout-replacement). Object exemplars were presented from a canonical viewpoint, with the main axis of the object facing to the left for half of the object exemplars in each category, and the other half facing to the right. Thus, the main axis of the objects was balanced across size and category. To ensure attention to the stimuli, subjects were instructed to memorize each object that appeared on screen. Following each scan run, subjects completed 10 trials of a 2-alternative forced-choice (2-AFC) memory test. On each trial, they were presented with an image from one of the 10 object categories shown in the previous run and an image from the same object category but drawn from a separate set of held-out images. They were instructed to select the image that they had seen in the preceding run. Participants correctly recognized marginally more small than large objects (t (15) = 2.11, P = 0.05), although were above chance at recognizing both small objects (79% correct; t (15) = 16.69, P <10 10 ) and large objects (74% correct; t (15) = 9.29, P <10 7 ). In addition to the experimental runs, subjects completed 2 functional localizer scans, which were acquired at the end of each scan session. These scans were 5 min 21 s in length, during which subjects performed a 1-back repetition detection task on scenes, objects, and scrambled objects, presented in 16 s blocks with each stimulus shown for 600 ms, each with a 400 ms interstimulus interval. Procedure: Mechanical Turk Experiment To examine other object properties that typically covary with real-world size, we used Amazon Mechanical Turk (n = 383) to collect ratings of 7 object characteristics for the 440 objects shown in the fmri experiment. The 7 object characteristics were as follows: 1. Distance to object. Large objects are more likely to be perceived from a distance, whereas small objects are more likely to be encountered close-up. Moreover, Amit et al. (2012) found that the PPA has a greater response to distal objects than proximal objects, and Troiani et al. (2014) found that distance was one of several object properties that modulate the univariate response in the scene regions. To quantify distance, we asked subjects to estimatethedistanceoftheobject from the camera that took the picture of the object in feet. 2. Portability. Some objects are easy for a person to move, while others are more difficult. Though portability often relates to size, there are exceptions: some small objects are impossible to move (e.g., a fire hydrant), while some large objects are easy to move (e.g., an automobile). Following Auger et al. (2012), we asked subjects to estimate how easy it would be for them to move the object on a scale of 1 5 (with 1 being very easy and 5 being very difficult). 3. Spatial stability. An important feature of objects is how often they change environmental locations. We assessed spatial stability by asking raters how often they would expect the position of an object to change in everyday life. Subjects rated each object on a scale from 1 to 5: 1 very often; 2 often; 3 occasionally; 4 rarely; and 5 never. This measure was adapted from the measure of permanence of Auger et al. (2012). Note that portability and spatial stability are related to each other, insofar as portable objects are more likely to change their position, but the concepts are not mere inverses of each other. A picture on the wall is portable, but it is also spatially stable. A hippopotamus, on the other hand, changes locations often, but it is not easy to move. 4. Spatial definition. Previous research has shown that certain objects can elicit a sense of the space that surrounds them, even in the absence of the surrounding scene. This property has been termed spatial definition and has been shown to modulate the univariate response to images of objects in the parahippocampal cortex (Mullally and Maguire 2011) and other scene regions (Troiani et al. 2014). We asked subjects to rate the extent to which each object evokes a strong sense of the space surrounding it by indicating whether the object was space defining or spatially ambiguous, meaning that it does or does not elicit a sense of the space that surrounds it, respectively. Participants could also rate an object as not classifiable if they were unsure. 5. Real-world size. To confirm that our distinction between large and small objects reflects how people would typically categorize these objects, participants were asked to estimate how long each object was along its longest axis in feet. 6. Contextual stability. Previous work has suggested that the strength of contextual associations is an important dimension of object processing (Bar and Aminoff 2003). To assess the extent to which each object was associated with a given contextual setting, we asked subjects to label a setting in which they would most often expect to find the object and then rate how often they would expect to encounter the object in that place on a 1 5 scale (1 never; 2 rarely; 3 sometimes; 4 often; 5 all of the time). We label this dimension contextual stability because of its parallel to spatial stability. Note that this method of assessment of contextual stability is similar but not identical to methods used to assess the strength of contextual associations in previous studies (Bar and Aminoff 2003; Troiani et al. 2014). 7. Graspability. Large and small objects differ in terms of how easy they are to grasp and manipulate (Salmon et al. 2010; Bainbridge and Oliva 2015). We asked subjects to rate each object in terms of how easy it is to grasp and manipulate the object with just one hand on a 1 5 scale (with 1 being very easy and 5 being very difficult). One set of surveys was used to collect ratings on the first 6 dimensions and another separate set of surveys was used to collect ratings on the 7th dimension. For each set, the objects were divided up into 22 surveys in which 1 unique exemplar image from each of the 20 object categories was shown. For each survey, participants were shown each object exemplar image, one at a time, and while the image remained on the screen rated it along each of either the first 6 object characteristics or the 7th characteristic before proceeding to the next object. Subjects completed one or more surveys and were required to have a Master Worker rating.

5 Representation of Object Size and Category Julian et al fmri Data Analysis Functional MR images for both the main experiment and functional localizer were preprocessed using the following steps. First, they were corrected for differences in slice timing by resampling slices in time to match the first slice of each volume. Second, they were corrected for subject motion by realigning to the first volume of the scan run using MCFLIRT. Third, the timecourses for each voxel were high-pass filtered to remove low temporal frequency fluctuations in the BOLD signal that exceeded lengths of 100 s. Data from the functional localizer scan were smoothed with a 5 mm full-width at half-maximum Gaussian filter; data from the main experiment were not smoothed. All analyses were performed within the subjects native space. We examined the univariate and multivariate response within several regions of interest (ROIs) known to be involved in visual processing. ROIs were defined individually for each subject using data from the functional localizer scans. Because previous researchers have disputed whether object recognition is best understood in terms of the unified operation of high-level cortex as a whole (Haxby et al. 2001), or division into smaller functional units (Kanwisher 2010), we defined ROIs at both spatial scales, which we refer to as the macro- and meso levels. To examine multivariate patterns across the full range of cortex involved in high-level visual recognition, we defined a temporo-parietooccipital (TPO) ROI consisting of all voxels in these 3 cortical lobes that responded more strongly to scenes, faces, or objects than to scrambled objects at a threshold of P < 0.05, uncorrected. To examine univariate and multivariate responses within smaller ROIs within the TPO region that have been the focus of previous investigation, we defined 6 smaller ROIs, including 3 scene-responsive regions (the parahippocampal place area [PPA], retrosplenial complex [RSC], and occipital place area [OPA]), 2 object-responsive regions (lateral occipital [LO] region and posterior fusiform sulcus [pfs]), and early visual cortex (EVC). These ROIs were defined using a contrast of scenes > objects for PPA, RSC, and OPA, objects > scrambled objects for LO and pfs, and scrambled objects > baseline for EVC, and they were further constrained by a groupbased anatomical map of scene-, object-, or scrambled objectselective activation derived from a large number (42) of localizer subjects that had been previously obtained in our lab (Julian et al. 2012). Specifically, each ROI was defined as the top 100 voxels in each hemisphere that exhibited the defining contrast and fell within the group-parcel mask for that ROI. This method ensured that all ROIs could be defined in both hemispheres in every subject and that all ROIs contained the same number of voxels. All contrasts were performed in the native anatomical space for each subject, and the group-parcel map was mapped into that space using a linear transformation. We then used general linear models (GLMs) implemented in FSL ( to estimate the response of each voxel to each object category in each scan run. Run-wise GLMs contained regressors for each object category presented in those runs, for a total of 10 regressors per run. These response values were used for both univariate and multivariate analyses. For univariate analyses, we calculated the average response of each ROI to large objects and to small objects by averaging response values for the 10 large objects and the 10 small objects across all scan runs and all voxels within each ROI (including both hemispheres). For multivariate analyses, we created run-wise activity patterns for each object by concatenating response values across all voxels in each ROI, including both hemispheres. Multivariate analyses were performed through split-half pattern comparison (Haxby et al. 2001). Specifically, for each ROI, we calculated Pearson correlations between the patterns of response for each of the 20 object categories in the first half of the experiment (runs 1 and 2) with the patterns for each of the 20 object categories in the second half of the experiment (runs 3 and 4). Individual patterns were normalized prior to this computation by subtracting the grand mean pattern (i.e., the cocktail mean) for each half of the data (Walther et al. 2015). We analyzed these correlations between patterns in several ways. First, to visualize the similarities and differences between objects within each ROI, we plotted the correlation values as representational similarity matrices (RSMs). We calculated the correlations between every object category, including correlations between patterns corresponding to the same object category, and arranged the scores alphabetically by category within each size class (i.e., large, small). Then, to understand how the correlations between objects were modulated by category and size, we grouped the correlations into 5 conditions and calculated the average correlation for each group: (i) same category, large; (ii) same category, small; (iii) different categories, all large; (iv) different categories, all small; and (v) different categories, different size. We reasoned that if a region contained information about object category, then patterns corresponding to objects of the same category (groups i and ii) should be more similar than patterns elicited by objects of different category. Similarly, if a region contained information about object size, then patterns corresponding to objects of different category but the same size (groups iii and iv) should be more similar than patterns elicited by objects of different category and different size (group v). To test the first possibility (coding by category), we computed category discrimination indices for large and small objects that were the difference between the average correlation value in the same-category condition and the average correlation value in the corresponding different-category/same-size condition (i.e., group i vs. group iii and group ii vs. group iv). To ensure that the within- and between-category correlations were both calculated based on comparisons between the same scan runs, we only included correlations between runs 1 and 3 and runs 2 and 4 in this computation, excluding the between-category correlations between runs 1 and 4 and runs 2 and 3. To test the second possibility (coding by size), we computed a size discrimination index that was the difference between the average correlation values in the different category/same size conditions (groups iii and iv) and the average correlation value in the different category/different size condition (group v). Note that because we used a correlation metric to assess similarities in the split-half patterns of response across conditions, multivariate coding by object size cannot be induced simply by a univariate main effect of size, as correlation is insensitive to any overall mean response differences across the patterns. In addition, for comparison to earlier studies that examined category coding in TPO ROI without controlling for size, we calculated a category discrimination index which wasthe difference in average correlation between both same category conditions (i.e., groups i and ii) and all different category conditions (i.e., groups iii, iv, and v). We also analyzed the multivariate data in terms of classification performance. For each ROI, we assessed whether the multivoxel patterns in one half of the data were sufficient to classify the size or category of the object being viewed in the other half of the data. A pattern was considered correctly classified by size if the average different-category/same-size correlation was greater than the average different-category/different-size correlation. For example, large object category 1 in the even half of the data was considered correctly classified by size if its average

6 3100 Cerebral Cortex, 2017, Vol. 27, No. 6 correlation with large object categories 2 10 in the odd half of the data was greater than its average correlation with small object categories Object category classification was performed analogously, using pairwise comparisons between categories of the same size (Haxby et al. 2001). For example, object category 1 in the even half of the data was considered correctly classified by category against large object category 2 if the correlation between category 1 (even) and category 1 (odd) was greater than the correlation between category 1 (even) and category 2 (odd). Note that these are essentially the same analyses that we performed when calculating the category and size discrimination indices, but in this case the amount of information is quantified as the proportion of correct classifications rather than as the difference in correlation values. Searchlight Analysis To test for coding of object size and category outside of our ROIs, we implemented a searchlight procedure (Kriegeskorte et al. 2007), which performs the same split-half MVPA calculations, but in small spherical ROIs (radius 5 mm) centered on each voxel of the brain in turn. In particular, we calculated the size discriminability index (same size correlation different size correlation) and the large category and small category discriminability indices (same category correlation different category correlation) in the local neighborhood of each voxel, and assigned the value of these indices to the center voxel of the spherical ROI. This procedure generated 3 whole-brain maps for each subject corresponding to coding for size, large object category, and small object category. The individual subject searchlight maps were then spatially smoothed using a Gaussian smoothing kernel (FWHM = 6 mm) and normalized to Talaraich space. The searchlight maps were then submitted to a second-level, random-effects analysis to identify voxels that reliably exhibited size and category coding across subjects. To find the true type I error rate for each searchlight type (size, large object category, small object category), we performed Monte Carlo simulations, which involved sign permutations of the whole-brain data from individual subjects (Nichols and Holmes 2002). We then report voxels that are significant at P < 0.05 after correcting for multiple comparisons across the entire brain. Results Univariate Results Prior to conducting any multivariate analyses, we attempted to replicate earlier findings by examining the univariate response to large and small objects. We averaged the responses to the large and small objects separately within each ROI and compared the magnitude of these responses. Consistent with the findings of Konkle and Oliva (2012), we observed a significantly greater response to large objects than to small objects in each of the scene regions (PPA: t (15) = 6.84, P <10 5 ;RSC:t (15) = 4.43, P < 0.001; OPA: t (15) = 6.56, P <10 5 ). In contrast, a differential response to large or small objects was not observed in the object-selective ROIs (LO and pfs) (both ts < 0.60, both Ps > 0.56). The latter result diverges from previous findings that a segment of LO responds preferentially to small objects (Konkle and Oliva 2012), possibly because low-level visual features in the present study were more closely matched between the large and small object stimuli. We observed no significant interaction between object size and hemisphere in any ROI (all Fs < 2.75, all Ps > 0.10) except pfs, in which there was a marginally greater response to small than large objects in the left hemisphere compared with the right (F 1,15 =4.12, P = 0.06). Further, no differential response to the large or small object categories was found in the early visual cortex ROI (EVC) (t (15) = 1.54; P = 0.14) (Fig. 2A). Greater response to large than small objects in scene regions was also apparent in the group-level map, where the voxels responding preferentially to large objects almost all fell within the boundaries of the PPA, RSC, or OPA (Fig. 2B). Moreover, consistent with prior reports (Konkle and Oliva 2012), we observed a left-lateralized region near the occipitotemporal sulcus that responded more to small than to large objects. Thus, our results largely replicate earlier findings that certain areas of visual cortex respond differentially to large versus small objects, with the exception that we did not find a locus for small objects in LO. Mulivariate Results: Temporo-Parieto-Occipital Cortex We then moved on to the primary question of the study: to what extent does real-world size act as an organizing principle for the object category representations revealed by multivoxel patterns? We began by examining a large region of interest in temporo parieto occipital (TPO) cortex that encompassed a wide swath of cortical territory that has been implicated in the recognition of meaningful visual stimuli. Previous work has shown that multivoxel patterns in this region contain information about object category, which has been taken as evidence for representation of object category or shape at the macro scale (Haxby et al. 2001; Carlson et al. 2003; Cox and Savoy 2003; O Toole et al. 2005). To replicate these results, we first examined object Figure 2. Univariate responsestolargeand smallobject categories. (A) Average responsein eachfunctionally definedroi (errorbars denote ± 1 SEM). Therewasa significantly greater response to large than small object categories in the PPA, OPA, and RSC, but not LO, pfs, or EVC (***P < 0.001). (B) Group average map of the large versus small object univariate contrast. Voxels in yellow or light blue are significant after correcting for multiple comparisons across the entire brain (P < 0.05, FWE-corrected); other colored voxels show a trend that does not survive correction. There was a significantly greater response to large than small objects in regions corresponding to the PPA (bronze outline), RSC (pink outline), and OPA (green outline), and a greater response to small than large objects in the left occipitotemporal sulcus.

7 Representation of Object Size and Category category coding in TPO without considering size as a possible organizing factor. We used the standard approach of calculating pattern similarity across independent datasets (e.g., different scan runs) for patterns corresponding to the same object category and for patterns corresponding to different object categories. We observed significantly greater similarity for same-category patterns compared with different-category patterns in TPO (t (15) = 4.0, P < 0.001), consistent with previous results indicating that multivoxel patterns in this region contain information about object category. Importantly, however, this analysis does not consider the possible role of size in organizing object categories, and it does not separate out differences in object category from differences in object size. To assess whether object representations in TPO were organized by real-world size, we constructed an across-run RSM of the neural data to visualize the relationship between size and category coding (Fig. 3A). It is immediately evident from inspection of the off-diagonal elements of the RSM that size plays a noticeable role in organizing the object representations: object categories of the same real-world size elicited more similar patterns of fmri response than object categories of different size. To quantify this observation, and to test whether object category distinctions remained after size was controlled, we grouped the cells of the RSM into 5 conditions (Fig. 3B) and then used the average pattern similarity across the cells in each condition to calculate 3 discrimination indices (Fig. 3C). First, we calculated an object category discrimination index for large objects by comparing the on-diagonal elements of the RSM for large objects to the off-diagonal elements corresponding to comparisons between large objects. Second, we calculated the analogous object category discrimination index for small objects. Third, we computed a size discrimination index, by comparing pattern similarity for objects of the same size but different category to pattern similarity for objects of different size and category. Thus, the object discrimination indices measure object category coding controlling for object size, whereas the size discrimination index measures size coding independent of object category. The object category index for large objects was significantly positive (t (15) = 3.16, P < 0.01), whereas the object category index for small objects was positive but only marginally significant (t (15) = 1.82, P = 0.09). These 2 indices were not significantly different (t (15) = 0.36, P = 0.73). Thus, TPO contains category information about large objects when size is controlled, and there is a similar though nonsignificant trend for small objects. Crucially, the size discrimination index was highly significant (t (15) = 5.14; Julian et al P < ). This finding indicates that real-world size organizes the multivoxel codes elicited by objects. Objects of more similar size elicit more similar fmri response patterns in TPO, even when they are drawn from different categories. To put the magnitude of the size coding effect into context, we computed the maximum possible information given any across all possible half splits of the objects. To do this, we split all 20 object categories into every possible set of 2 groups of 10 objects and computed a discrimination index for each of these possible splits for each participant. We then compared the observed size discrimination to the highest possible average discrimination index. There was no significant difference between these quantities (t (15) = 0.13; P > 0.90). This result indicates that real-world size provides close to the best possible (split-half ) coding principle for the objects used in the present study. Mulivariate Results: Individual ROIs Having established that real-world size acts as an organizing principle at the macro level, we then tested whether it has a similar effect on multivoxel patterns within individual cortical regions (which we refer to as the meso level). To do this, we repeated the TPO analyses using patterns observed within scene-selective regions (PPA, RSC, OPA), object-selective regions (LO and pfs), and early visual cortex (EVC). Inspection of the RSMs for each region indicates that the PPA, RSC, OPA, LO, and pfs all showed evidence of size coding, whereas EVC did not. In all 5 high-level visual regions, object categories of the same size elicited more similar response patterns than object categories of different size. To quantify these observations and to test whether object category distinctions were present after size was controlled, we grouped the cells of each RSM into 5 conditions as we did previously (Fig. 4B) and calculated the corresponding category and size discrimination indices (Fig. 4C). Below we consider the results for the category and size indices in turn. In scene and object regions (PPA, RSC, OPA, LO, pfs), there was mixed evidence for object category coding. Multivoxel codes in the PPA distinguished between large objects (t (15) = 3.00, P < 0.01) and exhibited a marginal trend for small objects (t (15) = 1.87, P = 0.08), whereas multivoxel codes in the OPA distinguished between large objects (t (15) = 3.48, P < 0.01) and also between small objects (t (15) = 3.64, P < 0.01). Object category indices in the RSC were not significantly different from zero (large: t (15) = 0.03, P = 0.97; small: t (15) = 0.90, P = 0.38). Among object regions, LO Figure 3. Multivariate results in the temporo-parieto-occipital (TPO) region. (A) RSM depicting the correlations between the patterns of response to each object category in the first and second halves of the data. The RSM is scaled to the range of correlation values observed in the RSM. (B) To test for information about object category and object size in the TPO region, we grouped the correlations into 5 conditions and calculated the average correlation for each group: (i) same object, large; (ii) same object, small; (iii) different objects, both large; (iv) different objects, both small; and (v) different objects, different size. The average pattern similarity (Pearson s r) for each of these 5 groups is shown (error bars denote ± 1 SEM). (C) To test for coding by object category within size, we computed category discrimination indices that were the difference between the same-object and different-object conditions, separately for each size. There was significant information about large object category in the TPO region, with a similar but nonsignificant trend for small objects. To test for coding by size, we computed a size discrimination index that was the difference in average correlation between the different object/same size conditions and the different object/different size condition. There was significant information about object size in TPO (***P < 0.001; **P < 0.01).

8 3102 Cerebral Cortex, 2017, Vol. 27, No. 6 Figure 4. Multivariate results in meso-level ROIs (PPA, RSC, OPA, LO, pfs, and EVC). The left column shows the RSM for each ROI. The RSM for each ROI is uniquely scaled to the range of correlation values observed for that ROI. The middle column shows average pattern similarity (Pearson s r) for each of the 5 object comparison groupings (see Fig. 3 legend for explanation). The right column shows discrimination indices for large and small object category, and object size. There was mixed evidence for large and small object category information across the ROIs. In contrast, in all ROIs except for EVC, there was significant information about object size. (***P < 0.001; **P < 0.01, P < 0.07).

9 Representation of Object Size and Category Julian et al distinguished between large objects (t (15) = 3.83, P < 0.01) and also between small objects (t (15) = 4.08, P < 0.001), whereas pfs did not distinguish between large objects (t (15) = 0.98, P = 0.35) and showed a marginal trend for small objects (t (15) =2.07,P = 0.06). Notably, there was no significant difference between the large object category index and the small index category index in any region (all ts < 0.9, all Ps > 0.34). Consistent with this mixed evidence for category coding across the high-level visual regions, a 1-way ANOVA confirmed that there was a significant difference in category coding across regions (F 4 =7.04,P < 0.001). Thus, the OPA and LO distinguished between object categories when size was controlled, the PPA did so to a lesser extent, pfs showed a trend for small objects only, and the RSC did not distinguish between object categories at all. There was no significant difference in overall category coding between the left and right hemisphere in any ROI (all ts < 1.45, all Ps > 0.17). We also observed significant information about both small and large object categories in EVC (large objects: t (15) =2.82, P <0.01;smallobjects:t (15) =2.95,P < 0.01; difference: t (15) =0.01, P = 0.99). These effects may relate to low-level feature differences between the object categories. Although we controlled for such differences between large objects and small objects, we did not control for such differences between categories within each size. In contrast to the mixed results for size-controlled category discrimination, size discrimination was significant in all 5 highlevel visual regions (PPA: t (15) =7.10;P < ; RSC: t (15) =3.05; P < 0.01; OPA: t (15) =4.10; P <0.001; LO: t (15) = 5.10; P < ; pfs: t (15) = 5.93; P < ). There was no significant difference in size coding across hemispheres in any region (all ts < 1.57, all Ps > 0.14). Thus, in all 5 regions, object categories of similar size elicited more similar activation patterns than object categories of different size. This finding indicates that real-world size not only affects multivoxel patterns at the macro (TPO) scale, but also acts as an organizing principle for object representations within high-level cortical regions. A 1-way ANOVA revealed a significant difference Figure 5. Comparison of the size discrimination index to the maximum possible split-half information present in each meso-level ROI. In the RSC and EVC, the size discrimination index was significantly lower than the maximum possible splithalf coding. In the PPA, OPA, LO, and pfs, there was no significant difference between the observed and optimal split-half coding. in the strength of size coding across these high-level regions (F 4 = 3.7, P < 0.01), with the strongest coding in PPA and OPA, somewhat weaker coding in LO and pfs, and the weakest (though still significant) coding in the RSC. We observed no evidence of organization by size in EVC (t (15) =1.71,P = 0.11), and size coding in EVC was significantly weaker than in each of the high-level visual ROIs (all ts > 3.76, all Ps < 0.002) except for the RSC in which it wasonlymarginallyweaker(t (15) =2.02,P =0.07). We next compared the magnitude of size coding in each ROI to the maximum information possible given any possible split of the objects in each ROI (Fig. 5). The maximum possible information was not numerically different from the observed size coding in the PPA (t (15) =0,P = 1). Thus, real-world size provides the best possible (split-half) coding principle in this region. The maximum information was also not significantly different from observed size coding in the OPA, LO, or pfs (all ts<0.64, all Ps > 0.53); although size coding was numerically weaker than the maximum possible information in these 3 regions, the median percentage of split-half arrangements that yielded more information than size was only 2%, 4%, and 2% in these regions, respectively. Size coding in the RSC was significantly lower than the maximum possible information (t (15) = 3.72; P < 0.01), and 14% of split-half arrangements yielded more information than that observed for size in this region on average. The optimal arrangement of object categories for the RSC involved grouping the coffee maker category with the large objects and the washing machine category with the small objects. Coding was also numerically enhanced in the OPA by switching these 2 object categories. Size coding in EVC was significantly less than the maximum possible information (t (15) = 17.70; P <10 11 ) and 43% of split-half arrangements yielded more information than that observed for size in this region on average. This is unsurprising given that size coding in EVC was not significant. The fact that objects of the same size elicit more similar response patterns than objects of different size in high-level visual regions indicates that these regions contain sufficient information to classify the size of an object category. To more directly assess this, we reanalyzed our data in terms of classification performance. We found that the pattern of response in each high-level region was sufficient to classify the size class (large or small) of the category being viewed with 59 69% accuracy (chance: 50%; all ts > 3.0, all Ps < 0.008; Table 1). In contrast, size could not be significantly classified in EVC (t (15) = 1.73, P =0.1). Pairwise classification of object category controlling for size was also significantly above chance in the PPA, OPA, LO, and EVC (all ts> 2.44, Ps < 0.03; accuracies 55 59%; Table 1), but not in pfs or in the RSC (both ts < 1.78, Ps > 0.1). Although the category classification performance observed here is lower than in some previous reports (Spiridon and Kanwisher 2002; MacEvoy and Epstein 2011), this difference is likely due (at least in part) to the fact that size was not fully controlled in earlier stimulus sets. Indeed, all regions showed significantly stronger category Table 1 Classification accuracy (% correct classification ±1 SEM) in each meso-level region of interest Region PPA RSC OPA LO pfs EVC Object size 69 ± ± ± ± ± ± 2.2 Object category (size controlled) 55 ± ± ± ± ± ± 2.2 Object category (size not controlled) 59 ± ± ± ± ± ± 2.5 Note: Bold values denote significant classification performance (P < 0.05).

10 3104 Cerebral Cortex, 2017, Vol. 27, No. 6 Figure 6. Results of the metric multidimensional scaling (MDS) analysis performed on the RDMs for each functional ROI. The 2 dimensions shown captured 27% of the covariance in the RDMs on average across ROIs. There was clear evidence for grouping of object categories by size in all ROIs except EVC. classification when size was not controlled than when it was (all ts > 3.49, all Ps < 0.004; Table 1), except for RSC, which showed only marginal classification improvement (t (15) = 1.88, P = 0.08). To visualize the between-category neural similarities, we performed classical (metric) Torgerson-Gower multidimensional scaling (MDS) on the neural RSM for each ROI, averaged across participants. This yielded a 2-dimensional plot of the similarity of all of the object categories such that categories placed closer together are more similar in terms of neural coding (Fig. 6). Small and large objects were found to occupy separate clusters in each ROI except EVC. Thus, once again, these results reinforce the importance of real-world size for organizing the structure of object representations within individual brain regions. One exception to a pure distinction between the large and small objects in the MDS plots is the coffee maker, which was closer to the large than small objects in the OPA, RSC, PPA, and LO. This observation reiterates our finding that the maximally informative grouping of object categories in the OPA and the RSC required swapping the coffee maker and washing machine (though this swap only significantly improved size discrimination in the RSC). All the multivariate analyses above were performed using a correlation metric and thus were insensitive to overall activity differences between conditions. To ensure that our results were not specific to the use of a correlation metric, we also analyzed the data using a Euclidean distance metric, which is sensitive both to univariate and to multivariate differences. The overall pattern of results was identical (see Supplementary Fig. 1): there was significant information about category controlling for size in all regions (all ts > 2.21, all Ps < 0.05) except for the RSC (t (15) = 1.01, P = 0.33) and significant size coding in all highlevel regions (all ts > 4.14, all Ps < ) but not in EVC (t (15) = 0.75, P = 0.47). The only notable difference in size coding between the correlation and Euclidean metrics was the ordering of the strength of the effects across high-level regions (Spearman rank correlation: ρ = 0.9, P = 0.08; Euclidean distance rank order: RSC, LO, pfs, OPA, PPA; correlation rank order: PPA, OPA, LO, pfs, RSC). Thus, our results are not specific to the particular metric used to assess representational similarity. Figure 7. Results of the searchlight analysis for object category. (A) Group-averaged searchlight map for large object category information. (B) Group-averaged searchlight map for small object category information. (C) Group-averaged searchlight map for object size information. For each searchlight map, outlines of the PPA (bronze), RSC ( pink), OPA (green), and LO (light blue) parcels are shown. There was significant information about object category in each of these regions in at least 1 category searchlight map. Size coding was most evident in the PPA, RSC, and an occipitoparietal region that extended dorsally from LO and the OPA. Voxels in yellow are significant after correcting for multiple comparisons across the entire brain (P < 0.05, FWE-corrected).

11 Representation of Object Size and Category Mulivariate Results: Searchlight Analyses To complement the results of the ROI analyses, we performed whole-brain searchlight analyses to test for coding of object category and object size throughout the entire brain. Specifically, we calculated the object category and object size indices for the neighborhood surrounding each voxel of the brain. Results for the object category searchlights are shown in Figure 7A,B, and results for the object size searchlight are shown in Figure 7C. The results of the searchlight analyses were generally consistent with the ROI results. The category searchlights showed evidence of large and small object coding in several visual areas, Julian et al including EVC, LO, and OPA (Fig. 7A,B). Contrast of the large and small searchlight maps revealed stronger coding of small than large object category in a bilateral region near the superior temporal gyrus (P < 0.05, corrected; Talaraich coordinates of peak voxel: LH [ 60, 36, 13], RH [58, 20, 5]). No region was found to exhibit significantly stronger coding of large than small objects. The size searchlight also showed evidence of size coding in several regions, with the strongest foci in the PPA, RSC, a lateral occipitoparietal region that encompassed OPA but extended beyond it, and (to a lesser extent) LO (Fig. 7C). Notably, the results of the size searchlight suggest that although size may be an important organizing principle for object representations in many high-level visual regions, the strongest effects are in the classical scene-processing network. Object Properties that Covary with Size Although the above results may reflect coding of size per se, they might alternatively reflect coding of any of the many object characteristics that tend to covary with size. To explore this covariance within our stimulus set, we conducted a separate behavioral experiment in which we used Amazon Mechanical Turk to obtain ratings of each object exemplar along 7 dimensions: distance to the object, portability, spatial stability, spatial definition, real-world size, contextual stability, and graspability. For each dimension, the ratings for large and small objects were significantly different (all ts > 2.82; all Ps < 0.02; Fig. 8A). Thus, many object properties covary with size, consistent with our previous observations (Troiani et al. 2014). Indeed, RSMs for each of the 7 object dimensions based on the similarity in ratings between each object category were highly correlated (see Supplementary Table 1). A principal component analysis performed on these 7 RSMs revealed that a single component explained 87% of the variability in the ratings between object categories. These results suggest that size may be a signature of these multiple object dimensions, rather than critical in its own right. Despite the fact that many object properties covaried with size, there was some variability within the object ratings that are relevant to understanding the structure of neural representations at the meso level (Fig. 8C). Specifically, ratings of spatial stability for the coffee maker were more similar to ratings provided for the large objects than the small objects. The coffee maker did not uniquely differ from the other small objects on any other object properties, although it was also more similar to the large objects on spatial definition than most of the small objects. Recall that in the RSC, neural coding was significantly improved by swapping the coffee maker and the washing machine across size groups. Thus, spatial stability of objects may be a better characterization of the key organizing principle for object representations in the RSC, consistent with prior reports of RSC sensitivity to object stability (Auger et al. 2012). Figure 8. Behavioral ratings for each object category along 7 dimensions: distance to object, portability, spatial stability, spatial definition, size, contextual stability, and graspability. (A) Average ratings (z-scored) for the large (shown in black) and small (shown in gray) object categories (error bars denote ±1 SEM across object categories). Large and small objects were significantly different on each dimension. Note that portability and graspability were rated in terms of how easy it would be to move or grasp the object, respectively, and so higher ratings would be expected for small than large objects. (B) Average ratings for each object category, separately for each of the dimensions rated for each object category. Each dot denotes a different object category (blue for large, red for small). There is a clear clustering of the object categories by size for each dimension. Notably, the coffee maker (CM) category was rated more similarly to the large object categories than the small object categories on spatial stability and spatial definition. Discussion The main finding of this study is that real-world size is an important organizing principle for object representations in human visual cortex, affecting both univariate and multivariate responses. Consistent with earlier studies (Konkle and Oliva 2012; Troiani et al. 2014), we found differential regional sensitivity to large versus small objects, with scene regions (PPA, RSC, and OPA) responding more strongly to large than to small objects, and the OTS responding more strongly to small objects that to large objects. Moreover, in a novel result, we found that objects of similar size elicited more similar multivoxel response patterns, both at

12 3106 Cerebral Cortex, 2017, Vol. 27, No. 6 the macro level (i.e., across TPO cortex) and at the level of individual ROIs. Notably, this organization-by-size was found not only in scene-responsive regions that showed a univariate distinction between large and small objects (PPA, RSC, OPA), but also in object-responsive regions that did not show a univariate effect (LO and pfs). These findings build on earlier results showing that realworld size is an important determinant of fmri response in high-level visual cortex, but move beyond them in several key ways. Most notably, the current results allow us to more directly understand how real-world size affects object category coding. The fact that size may be an organizing principle for category distinctions at the macro level was implied by previous results showing gradients of small-to-large response preferences along the lateral and ventral occipitotemporal surfaces, which mirrored similar distinctions between certain object categories (e.g., faces vs. scenes). However, because these studies did not directly examine how these response gradients related to the multivoxel response patterns elicited by different categories of objects, it was unclear how the observed organization-by-size interacted with organization-by-category. Here we show that multivoxel category distinctions at both the macro- and meso level are strongly organized by size, which was not a hypothesis that was directly tested in earlier studies. Taken as a whole, these results demonstrate that real-world size not only affects the magnitude of the regional response, but it also shapes the underlying objects representations that potentially allow object categories to be distinguished from each other. It was somewhat surprising to us that we observed grouping of object categories by size in so many different high-level visual regions, including not only those known to respond strongly to scenes (PPA, RSC, OPA), but also those known to respond strongly to individual objects (LOC, pfs). Although this may reflect replication of the same organizational principle across many different brain regions, it may alternatively be the case that different brain regions are sensitive to size for different reasons. For example, some regions may code objects by category with size acting as an organizing factor ( coding by size ), as suggested above, but other brain regions may code size directly ( coding of size ). Our data suggest that size effects in the OPA and LO reflect coding by size, because category coding in these regions remained strong even when size was controlled. In contrast, size effects in the RSC might reflect coding of size (or other spatial characteristics; see below), because in this region object categories of the same size were not distinguishable from each other. Previous work suggests that multivoxel patterns in the RSC represent the size of the depicted environment independent of the category or content of the scene (Park et al. 2015) and the spatial stability of objects independent of their identity (Auger and Maguire 2013). Thus our data are consistent with previous research suggesting that the RSC directly represents spatial quantities that can inhere in objects and scenes, in contrast to LO which represents object features that allow categories to be distinguished (Grill-Spector et al. 2001; Eger et al. 2008). However, we must be cautious with the first part of this conclusion, as it relies on a null result in the RSC, and some previous studies have found object category information in this region even when real-world size was partially controlled (Iordan et al. 2015). In pfs and the PPA, we observed an intermediate result. Size could be easily decoded from these regions, whereas object category effects were present but relatively weak. Given that some of the object category effects might reflect low-level visual differences between the object categories, our data leave it unclear whether these regions represent object category distinctions or not. Previous work suggests that pfs likely does code object category even when size is partially controlled (MacEvoy and Epstein 2011), but this code may be at the within-voxel spatial scale and thus not evident in multivoxel patterns (Drucker and Aguirre 2009). In contrast, whether the size effects in the PPA reflect coding of category by size is an open question. This is an important issue for future work to resolve, because theoretical accounts of the PPA make different predictions about whether the region should encode object category irrespective of size. Specifically, accounts that consider the PPA to be part of a general object recognition mechanism (Haxby et al. 2001) or interpret its function in terms of processing of contextual (Bar and Aminoff 2003; Eichenbaum et al. 2007; Diana et al. 2008; Ranganath and Ritchey 2012; Aminoff et al. 2013) or semantic features (Huth et al. 2012; Stansbury et al. 2013) associated with objects require that object categories be distinguishable. In contrast, accounts that consider the PPA to be primarily involved in spatial or scene processing (Epstein and Kanwisher 1998; Walther et al. 2009; Kravitz et al. 2011; Mullally and Maguire 2011; Park et al. 2011) predict that object category distinctions should only be observable insofar as they relate to spatial differences, such as the extent to which certain object categories are space defining, or the extent to which an object can be interpreted as a partial scene. Finally, a third set of accounts considers the PPA to be involved in recognition of objects, but only when these objects are navigational landmarks (Janzen and van Turennout 2004; Epstein and Vass 2014; Marchette et al. 2015). At first glance, one might think that this third set of accounts would predict that object patterns should be more distinguishable for large objects than for small objects, which would be inconsistent with our data. However, given that landmarks have navigational relevance only by virtue of their unique identity, we believe that this theory only makes predictions about the distinguishability of individual object exemplars, not about the distinguishability of object categories. In sum, the current data do not clearly adjudicate between these accounts and may in fact be consistent with an intermediate view whereby the PPA encodes both object and spatial information (Harel et al. 2012) with the exact items coded determined in part by navigational relevance. Turning back to the size effects, which were highly robust in all of our ROIs except EVC, an important question is whether these are driven by real-world size per se or by one of the several other stimulus dimensions that are typically correlated with realworld size. Consistent with previous observations, we found that the large and small objects in our experiment differed not only on real-world size, but also on typical viewing distance, portability, spatial stability, spatial definition, contextual stability, and graspability. We previously suggested that all of these stimulus dimensions might be taken as proxies for a single underlying object characteristic, which we labeled landmark suitability, which in our previous data was most closely related to the measures of spatial definition (the extent to which an object defines the space around it) and placeness (the extent to which an object is treated as a place rather than a thing) (Troiani et al. 2014). Interestingly, in the current dataset, there was 1 object category that appeared to be misclassified : the coffee maker. Though physically small, it elicited activity patterns in several regions that were more similar to the patterns elicited by the large objects than to the patterns elicited by the other small objects. This similarity to large objects was observed in the MDS plots for the RSC, OPA, PPA, and LO, and it was strong enough to lead to suboptimal size classification in the RSC (with a similar trend in the OPA). Notably, the behavioral ratings indicated that the coffee maker also had a higher degree of spatial stability than the other small objects,

13 Representation of Object Size and Category Julian et al and it also had a high value for spatial definition. Although we must be careful not to overinterpret this single observation, it does suggest the possibility that spatial stability and/or spatial definition both of which are intimately related to landmark suitability might be a better description of the underlying stimulus dimension than real-world size. The fact that the neural effects of coffee maker misclassification were strongest in the RSC is consistent with this view, as previous work has shown that multivoxel codes in this region are sensitive to spatial stability (Auger and Maguire 2013). More generally, our results suggest that the distinction between large stable objects and small unstable objects is a fundamental one, affecting not only whether a region responds strongly or weakly to an object, but also how objects are represented within multiple brain regions. A possible argument against this conclusion is that it assumes that the representational similarities that we observe here reflect fixed aspects of neural organization. In contrast to this view, recent studies suggest that the representational spaces revealed by multivoxel fmri pattern analyses can be affected by task (Harel et al. 2014; Erez and Duncan 2015) and attentional set (Peelen et al. 2009; Cukur et al. 2013). In the current case, the task did not require subjects to think about the navigational valence or landmark suitability of the items, so it is unlikely that this dimension was especially enhanced. However, the presence of both large and small objects within the same scan run might have sufficed to draw attention to size differences. Moreover, the memory encoding task that we used to ensure attention to the stimuli might have elicited semantic representations in high-level visual cortex which might not be elicited by simple perceptual tasks such as 1-back repetition detection, dot detection, or passive viewing. Indeed, previous work has found that the elicitation of semantic codes can be affected by factors such as task and processing time (MacEvoy and Epstein 2011; Harel et al. 2014). Thus, an interesting question for future work is whether or not the size effects observed here in multivoxel patterns are malleable by task and attention. We should note, however, that previous work suggests that differences in the univariate response are quite robust, as they have now been observed in several different experiments utilizing different tasks. Assuming that our results do reflect a permanent or semipermanent aspect of the neural representation of objects, we are left with several important questions. First, how does the neural organization by size observed here relate to previous results showing organization based on other distinctions, such as living versus nonliving (Chao et al. 1999; Kriegeskorte et al. 2008). In the current experiment, all of the objects were inanimate artifacts, so it is possible that size organization only applies to such nonliving objects (Konkle and Caramazza 2013). Second, how do the univariate distinctions relate to the multivariate distinctions? Regional preferences to large versus small objects may reflect the nature of the cognitive mechanism supported by each region, such as spatial navigation in the PPA, RSC, and OPA, and physical manipulation in the OTS, whereas multivariate distinctions within each region might reflect the representational structure supported by each mechanism. Thus, the size distinctions at the macro and meso levels, though ascertained similarly in our analyses, might in fact reflect fundamentally different levels of structure within the underlying cognitive architecture. Third, does neural organization by size reflect differential sensitivity to some visual features that distinguish large and small objects, uncontrolled in the present study, such as boxiness or rectilinearlity (Nasr and Tootell 2012; Nasr et al. 2014; Long et al. 2016)? Alternatively, does this organization arise through experience with object size? All of these questions are ripe targets for future investigation. In sum, our results confirm and extend previous reports that real-world size is an important organizing principle for object representations throughout high-level human visual cortex. Not only do certain brain regions respond preferentially to large versus small objects, but also object category representations within these regions are grouped by size. We suggest that size is important for neural representations, because it is important in the real world: it determines how we interact with objects and thus their functional meaning. Supplementary Material Supplementary material can be found at: oxfordjournals.org/. Funding This work was supported by National Institutes of Health (R01 EY , R21 EY ), National Science Foundation (SBE ) grants to R.A.E., and a National Science Foundation Graduate Research Fellowship to J.B.J. Notes Conflict of Interest: None declared. References Aguirre GK Continuous carry-over designs for fmri. Neuroimage. 35: Aminoff EM, Kveraga K, Bar M The role of the parahippocampal cortex in cognition. Trends Cogn Sci. 17: Amit E, Mehoudar E, Trope Y, Yovel G Do object-category selective regions in the ventral visual stream represent perceived distance information? Brain Cogn. 80: Auger SD, Maguire EA Assessing the mechanism of response in the retrosplenial cortex of good and poor navigators. Cortex. 49: Auger SD, Mullally SL, Maguire EA Retrosplenial cortex codes for permanent landmarks. PLoS One. 7: e Bainbridge WA, Oliva A Interaction envelope: local spatial representations of objects at all scales in scene-selective regions. Neuroimage. 122: Bar M, Aminoff E Cortical analysis of visual context. Neuron. 38: Cant JS, Goodale MA Scratching beneath the surface: new insights into the functional properties of the lateral occipital area and parahippocampal place area. J Neurosci. 31: Carlson TA, Schrater P, He S Patterns of activity in the categorical representations of objects. J Cogn Neurosci. 15: Cate AD, Goodale MA, Kohler S The role of apparent size in building- and object-specific regions of ventral visual cortex. Brain Res. 1388: Chao LL, Haxby JV, Martin A Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat Neurosci. 2: Cox DD, Savoy RL Functional magnetic resonance imaging (fmri) brain reading : detecting and classifying distributed patterns of fmri activity in human visual cortex. Neuroimage. 19:

14 3108 Cerebral Cortex, 2017, Vol. 27, No. 6 Cukur T, Nishimoto S, Huth AG, Gallant JL Attention during natural vision warps semantic representation across the human brain. Nat Neurosci. 16: Diana RA, Yonelinas AP, Ranganath C High-resolution multi-voxel pattern analysis of category selectivity in the medial temporal lobes. Hippocampus. 18: Drucker DM, Aguirre GK Different spatial scales of shape similarity representation in lateral and ventral LOC. Cereb Cortex. 19: Eger E, Ashburner J, Haynes JD, Dolan RJ, Rees G fmri activity patterns in human LOC carry information about object exemplars within category. J Cogn Neurosci. 20: Eichenbaum H, Yonelinas AP, Ranganath C The medial temporal lobe and recognition memory. Annu Rev Neurosci. 30: Epstein R, Kanwisher N A cortical representation of the local visual environment. Nature. 392: Epstein RA The cortical basis of visual scene processing. Vis Cogn. 12: Epstein RA, Vass LK Neural systems for landmark-based wayfinding in humans. Philos Trans R Soc Lond B Biol Sci. 369: Erez Y, Duncan J Discrimination of visual categories based on behavioral relevance in widespread regions of frontoparietal cortex. J Neurosci. 35: Grill-Spector K, Kourtzi Z, Kanwisher N The lateral occipital complex and its role in object recognition. Vision Res. 41: Grill-Spector K, Weiner KS The functional architecture of the ventral temporal cortex and its role in categorization. Nat Rev Neurosci. 15: Harel A, Kravitz DJ, Baker CI Deconstructing visual scenes in cortex: gradients of object and spatial layout information. Cereb Cortex. 48:S50. Harel A, Kravitz DJ, Baker CI Task context impacts visual object processing differentially across the cortex. Proc Natl Acad Sci USA. 111:E962 E971. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 293: Huth AG, Nishimoto S, Vu AT, Gallant JL A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron. 76: Iordan MC, Greene MR, Beck DM, Fei-Fei L Basic level category structure emerges gradually across human ventral visual cortex. J Cogn Neurosci. 27: Janzen G, van Turennout M Selective neural representation of objects relevant for navigation. Nat Neurosci. 7: Julian JB, Fedorenko E, Webster J, Kanwisher N An algorithmic method for functionally defining regions of interest in the ventral visual pathway. Neuroimage. 60: Kanwisher N Functional specificity in the human brain: a window into the functional architecture of the mind. Proc Natl Acad Sci USA. 107: Kanwisher N, Dilks DD The functional organization of the ventral visual pathway in humans. In: Chalupa L, Werner J editors. The New Visual Neurosciences. Cambridge: MIT Press, pp Kanwisher N, Yovel G The fusiform face area: a cortical region specialized for the perception of faces. Philos Trans R Soc Lond B Biol Sci. 361: Konkle T, Caramazza A Tripartite organization of the ventral stream by animacy and object size. J Neurosci. 33: Konkle T, Oliva A A real-world size organization of object responses in occipitotemporal cortex. Neuron. 74: Kravitz DJ, Peng CS, Baker CI Real-world scene representations in high-level visual cortex: it s the spaces more than the places. J Neurosci. 31: Kriegeskorte N, Formisano E, Sorger B, Goebel R Individual faces elicit distinct response patterns in human anterior temporal cortex. Proc Natl Acad Sci USA. 104: Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, Esteky H, Tanaka K, Bandettini PA Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron. 60: Levy I, Hasson U, Avidan G, Hendler T, Malach R Centerperiphery organization of human object areas. Nat Neurosci. 4: Long B, Konkle T, Cohen MA, Alvarez GA Mid-level perceptual features distinguish objects of different real-world sizes. J Exp Psychol Gen. 145:95. MacEvoy SP, Epstein RA Constructing scenes from objects in human occipitotemporal cortex. Nat Neurosci. 14: Marchette SA, Vass LK, Ryan J, Epstein RA Outside looking in: landmark generalization in the human navigational system. J Neurosci. 35: Martin A The representation of object concepts in the brain. Annu Rev Psychol. 58: Mullally SL, Maguire EA A new role for the parahippocampal cortex in representing space. J Neurosci. 31: Nasr S, Echavarria CE, Tootell RBH Thinking outside the box: rectilinear shapes selectively activate scene-selective cortex. J Neurosci. 34: Nasr S, Tootell RB A cardinal orientation bias in scene-selective visual cortex. J Neurosci. 32: Nichols TE, Holmes AP Nonparametric permutation tests for functional neuroimaging: a primer with examples. Hum Brain Mapp. 15:1 25. Oliva A, Torralba A Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis. 42: O Toole AJ, Jiang F, Abdi H, Haxby JV Partially distributed representations of objects and faces in ventral temporal cortex. J Cogn Neurosci. 17: Park S, Brady TF, Greene MR, Oliva A Disentangling scene content from spatial boundary: complementary roles for the parahippocampal place area and lateral occipital complex in representing real-world scenes. J Neurosci. 31: Park S, Konkle T, Oliva A Parametric coding of the size and clutter of natural scenes in the human brain. Cereb Cortex. 25: Peelen MV, Fei-Fei L, Kastner S Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature. 460: Ranganath C, Ritchey M Two cortical systems for memoryguided behaviour. Nat Rev Neurosci. 13: Salmon J, McMullen P, Filliter J Norms for two types of manipulability (graspability and functional usage), familiarity, and age of acquisition for 320 photographs of objects. Behav Res Methods. 42: Sha L, Haxby JV, Abdi H, Guntupalli JS, Oosterhof NN, Halchenko YO, Connolly AC The animacy continuum in the human ventral vision pathway. J Cogn Neurosci. 27:

15 Representation of Object Size and Category Julian et al Spiridon M, Kanwisher N How distributed is visual category information in human occipito-temporal cortex? An fmri study. Neuron. 35: Stansbury DE, Naselaris T, Gallant JL Natural scene statistics account for the representation of scene categories in human visual cortex. Neuron. 79: Tarr MJ, Gauthier I FFA: a flexible fusiform area for subordinate-level visual processing automatized by expertise. Nat Neurosci. 3: Troiani V, Stigliani A, Smith ME, Epstein RA Multiple object properties drive scene-selective regions. Cereb Cortex. 24: Walther A, Nili H, Ejaz N, Alink A, Kriegeskorte N, Diedrichsen J Reliability of dissimilarity measures for multi-voxel pattern analysis. NeuroImage. pii: S (15) Walther DB, Caddigan E, Fei-Fei L, Beck DM Natural scene categories revealed in distributed patterns of activity in the human brain. J Neurosci. 29:

16