Integrated Contextual Representation for Objects Identities and their Locations. Nurit Gronau, Maital Neta, and Moshe Bar

Transcription

1 Integrated Contextual Representation for Objects Identities and their Locations Nurit Gronau, Maital Neta, and Moshe Bar Martinos Center for Biomedical Imaging at MGH, Harvard Medical School 16 March 2007 Running title: Integrated representation for identities and locations Correspondence should be addressed to: Dr. Moshe Bar Martinos Center for Biomedical Imaging at MGH Harvard Medical School 149 Thirteenth St. Rm 2301 Charlestown Massachusetts, Phone: (617)

2 Integrated representation for identities and locations Gronau et al., 2 Abstract Visual context plays a prominent role in every day perception. Contextual information can facilitate recognition of objects within scenes by providing predictions about objects that are most likely to appear in a specific setting, along with the locations that are most likely to contain objects in the scene. Is such identity-related ( semantic ) and location-related ( spatial ) contextual knowledge represented separately or jointly as a bound representation? We conducted an fmri priming experiment whereby semantic and spatial relations between prime and target object pictures were independently manipulated. This method allowed us to determine whether the two contextual factors affect object recognition with or without interacting, supporting a unified vs. independent representations, respectively. Results revealed a semantic spatial interaction in reaction times for target object recognition. Namely, significant semantic priming was obtained when targets were positioned in expected (congruent), but not in unexpected (incongruent) locations. fmri results showed corresponding interactive effects in brain regions associated with semantic processing (inferior prefrontal cortex), visual contextual processing (parahippocampal cortex), and object-related processing (lateral occipital complex). In addition, activation in fronto-parietal areas suggests that attention and memory-related processes might also contribute to the contextual effects observed. These findings indicate that object recognition benefits from associative representations that integrate information about objects identities and their locations, and directly modulate activation in object-processing cortical regions. Such context frames are useful in maintaining a coherent and meaningful representation of the visual world, and in providing a platform from which predictions can be generated to facilitate perception and action.

3 Integrated representation for identities and locations Gronau et al., 3 Introduction Objects are typically seen embedded in specific contextual settings, such as a desk in an office or a car on a street. These environmental regularities can be useful in facilitating object identification by increasing predictability in the sensory input, and thus streamlining the process of recognition. Indeed, studies using behavioral (Bar & Ullman, 1996; Biederman, 1972; Biederman, Mezzanotte, & Rabinowitz, 1982; Chun & Jiang, 1998; Davenport & Potter, 2004; Friedman, 1979; Mandler & Johnson, 1976; Palmer, 1975), as well as functional magnetic resonance imaging (fmri) (Bar & Aminoff, 2003; Cox, Meyers, & Sinha, 2004) approaches have documented the importance of contextual information in the facilitation and enhancement of visual processing and perceptual memory (for a review, see Bar, 2004). The influence of contextual processes on object recognition may stem from multiple sources of information, including knowledge about the expected identity, size, position and relative depth of an object within a scene (e.g., Biederman et al., 1982; Mandler & Johnson, 1976). Among these factors, two important sources of contextual knowledge are the focus of the present study. One comprises information about which object identities are most likely to appear within a specific visual setting (e.g., an office typically contains a desk, a desk lamp, and a computer); and the second comprises information about which locations within a visual setting are most likely to contain objects (e.g., objects are typically placed above, and not below desks). While both identity-based and location-based associative knowledge have been shown to enhance object detection and recognition (Bar & Ullman, 1996; Biederman et al., 1982; Chun & Jiang, 1998, 1999; Sanocki & Epstein, 1997), the exact nature of the relationship between these contextual factors remains largely unclear. Do identity- and location-related associations have distinct effects on visual perception, suggesting independent underlying representations for the two sources of knowledge? Or do these contextual factors interact in the course of object recognition, supporting a joint representation of identities and locations in visual associative processing?

4 Integrated representation for identities and locations Gronau et al., 4 Recently, we have proposed that contextual knowledge is represented within memory structures, or context frames, that contain knowledge about specific objects that are typical for a given visual setting, as well as about the probable spatial relations between these objects (Bar, 2004; Bar & Ullman, 1996). The notion of context frames is reminiscent of earlier concepts such as schemata (Biederman et al., 1982; Hock, Romanski, Galie, & Williams, 1978; Mandler & Johnson, 1976), scripts (Schank, 1975), and frames (Minsky, 1975), which all strongly imply a unified, global representation for identity- and location-based associative information. Such a joint representation is appealing as a theoretical construct since it suggests that objects are organized in memory within structures that depict typical scenes. However, not all empirical findings are necessarily in agreement with this concept. For instance, studies examining associative processing during visual object recognition have shown that a prime object (desk) can facilitate recognition of a successive, semantically related object identity (lamp), even when both pictures appear at the same physical location (e.g., at the screen center) where there is often a violation of the real-world spatial relations between the two items (i.e., a lamp typically appearing above a desk) (Carr, McCauley, Sperber, & Parmelee, 1982; McPherson & Holcomb, 1999; Sperber, McCauley, Ragain, & Weil 1979). Similar priming effects are obtained for target picture objects when using semantically related words as primes (e.g., the word DESK ), implying that the meaning of both verbal and pictorial stimuli is encoded in a common, amodal representation (e.g., Carr et al., 1982; Sperber et al, 1979). Such an amodal representation for visual objects is presumably abstracted of specific sensory details and of metric coordinates, suggesting that associative links between object identities are not necessarily encoded in a point-to-point scene-like representation (Potter & Faulconer, 1975; Pylyshyn, 1973; Sperber et al., 1973). Furthermore, studies directly investigating processing of objects within scenes have shown that immediate memory for object identities and their visual details is unaffected by the spatial relations between the objects within the scene, i.e., whether items are organized in a coherent (scene-like) or incoherent (jumbled) spatial organization (e.g.,

5 Integrated representation for identities and locations Gronau et al., 5 Mandler & Johnson, 1976; Mandler & Ritchey, 1977). In accordance with these findings, the representation of object identity appears to be largely independent of spatial layout and object location in online visual scene processing, as well as in visual short-term memory (e.g., Rensink, 2000; Simons, 1996). Taken together, these findings suggest that contextual associations between visual objects can be independent of spatial location representations. Visual contextual knowledge, then, may be conceptual, or abstract in nature, rather than encoded in a strictly picture-like representation. Additional evidence for the independence of identity- and location-related contextual knowledge comes from studies showing that the spatial properties of a scene or an object can be processed independently of knowledge of an associated object identity. For instance, rapid extraction of the spatial layout of a scene can constrain the probable location(s) of a target object within the scene, regardless of the object s identity (Chun & Jiang, 1998; Sanocki & Epstein, 1997). This rapid extraction of spatial properties may rely on basic visual features of the scene, such as global structure, perceived depth, and surface configuration (e.g., Chun & Jiang, 1998; Sanocki & Epstein, 1997; Schyns & Oliva, 1994). Alternatively, location-related associative knowledge may rely on semantic comprehension of a visual stimulus, such as when a static picture of an object in action (e.g., a tool pointing down) produces a sense of motion (Freyd, 1983; Kourtzi & Kanwisher, 2000; Reed & Vinson, 1996) and guides visual attention to the direction of the implied action (Gronau & Bar, submitted; Riddoch, Humphreys, Edwards, Baker, & Willson, 2003). Likewise, a picture of a table can guide attention to an implied, upper, location, based on one s knowledge that tables generally serve as surfaces above which objects are placed. Note that in the last two examples, knowledge of the coarse semantic and spatial properties of a context does not necessarily dictate which particular object identities are associated with it. For example, although all desks and tables are used for placing objects on, location-based contextual knowledge does not differentiate between specific objects that are more likely to appear on an office desk (e.g., desk lamp, computer), on a picnic table (e.g.,

6 Integrated representation for identities and locations Gronau et al., 6 picnic basket, paper plates), or on a coffee table (e.g., vase, decorative bowl). Thus, spatial guidance based on an object s direction of action, or on its basic function, does not necessarily provide the detailed knowledge required to activate an associated object identity. Locationbased knowledge, then, may generate expectations about a position of an upcoming object, in the absence of specific information regarding the object s identity. While the evidence above suggests that contextual knowledge of object identities and locations can be represented separately, there is nevertheless a firm basis to postulate that the two types of information are integrated at some level of representation. In every-day life, objects locations often correlate with their identities, such that, certain objects are more expected to be positioned at certain locations in a scene (e.g., sofas and carpets are more expected to be seen at floor level of a room, whereas computer screens and curtains are more expected to be seen at hand, or eye level, of the room). Given the potential usefulness of such environmental regularities to visual recognition and to action planning, it is reasonable to expect that the underlying contextual representations will reflect these regularities, and that the brain will exploit them. Consequently, knowledge of an object s identity should constrain the possible locations in which the object is to be searched, and vice versa. Thus, while one may maintain some form of an abstract, conceptual association between objects (e.g., a desk lamp is related to a desk), along with some general knowledge about the spatial relations between objects in the world (e.g., objects are typically placed above, not below, desks), we propose that a unified context frame encompasses these two subtypes of contextual associations (e.g., a desk lamp is necessarily placed on a desk, by virtue of its function). Such a global contextual representation can provide a highly coherent and meaningful representation of the visual environment, since it depicts with greater accuracy the relations between objects within a certain context. Behavioral studies investigating the effects of visual contextual information on object recognition have not allowed a systematic investigation of the relations between identity-based and location-based associative factors because these factors were typically examined in

7 Integrated representation for identities and locations Gronau et al., 7 isolation (Bar & Ullman, 1996; Biederman, 1972; Biederman et al., 1982; Davenport & Potter, 2004; Palmer, 1975; De Graef, Christiaens, & d Ydewalle, 1990). For instance, the effects of the relatedness between an object identity and a scene were examined only within expected, but not unexpected, locations in the scene (Biederman et al., 1982; Davenport & Potter, 2004; De Graef et al., 1990). Similarly, effects of spatial consistency between objects were examined only among conceptually related, and not unrelated, identities (e.g., Bar & Ullman, 1996; Biederman, 1972; Biederman et al., 1982; De Graef et al., 1990; Hollingworth, 2006). In addition, visual search studies using real-world objects and scenes have typically manipulated targets location within conceptually related scenes only (Neider & Zelinsky, 2006; Torralba, Oliva, Castelhano, & Henderson, 2006). Thus, in the absence of a simultaneous manipulation of both identity and location contextual factors, the dependency between the two types of associative knowledge could not be explicitly assessed. Similar limitations underlie fmri studies (Aminoff, Gronau & Bar, 2006; Bar & Aminoff, 2003; Cox et al., 2004; Goh et al., 2004) and ERP studies (Ganis & Kutas, 2003; McPherson & Holcomb, 1999) investigating the neural correlates of contextual effects on visual recognition. The goal of the present study, therefore, was to determine whether identity- and locationrelated associative knowledge is represented separately or jointly, examining both the behavioral and the neural levels. To this end, we used an event-related fmri priming task in which object primes served as contextual cues for recognition of target objects. Most importantly, identity and location relations between prime and target were independently manipulated, allowing a direct examination of the dependency of the two associative factors in visual object recognition. If the two factors are represented independently, we predicted that additive effects would be found in behavioral measures (e.g., reaction time) as well as in corresponding fmri measures (BOLD) for target object recognition. If, however, identity- and location-related associative knowledge is combined at some level of processing to form a unified contextual representation, we should obtain interactive effects of the two factors

8 Integrated representation for identities and locations Gronau et al., 8 (possibly in addition to each of the factors separate effects). Thus, identity-based contextual effects should be larger when targets appear in spatially congruent (i.e., plausible, or expected) locations than in incongruent (i.e., implausible, or unexpected) locations, suggesting that an object s position in space is tightly related to its perceived identity within a particular visual setting. Note that while both location- and identity-associative factors may rely on semantic analysis of a contextual prime, they nevertheless have different associated outputs under an assumption of their independency: the former guides attention to a specific spatial location (regardless of the target s expected identity), whereas the latter primes a target s identity (regardless of its specific location in space). We use here the term semantic knowledge to denote the representation of object relations that are abstracted of specific visual details and spatial coordinates. This term is adopted from the semantic priming literature, in which both verbal and pictorial priming effects are typically accounted for by conceptual (e.g., categorical) relatedness. Likewise, we use the term spatial knowledge to denote information about an expected location of an upcoming target object (independent of its specific identity). In effect, we considered an object to be spatially congruent with its contextual environment if it appeared in a plausible, rather than an implausible, location, as predicted from the context (see Methods). Importantly, the use of semantic and spatial associative terms is somewhat arbitrary and should be considered merely as working definitions for identity- and location- based contextual factors, under the null hypothesis of independent representations. Both definitions are based on statistical cooccurrences of specific object properties in the natural environment. As for the neural manifestation of visual contextual representation, we were particularly interested in the interactive effects of identity- and location-associative factors within three levels of cortical representation. First, we aimed to investigate whether brain regions typically associated with semantic processing would be modulated by spatial contextual information. Therefore, we examined activity in the inferior prefrontal cortex (IPC), a region previously shown

9 Integrated representation for identities and locations Gronau et al., 9 to be associated with a wide variety of semantic and conceptual tasks (both verbal and nonverbal), and specifically associated with the phenomenon of semantic priming (for a recent review, see Van Petten & Luka, 2006). To the extent that semantic (or conceptual) priming effects are influenced by spatial factors, IPC should be sensitive to visual-global properties of a contextual setting, exhibiting increased differences between semantically related and unrelated targets when these appear in an expected (congruent) location. Second, we examined activation in medial temporal lobe regions (hippocampus, parahippocampal cortex) that have been strongly implicated in associative processing (see Bar, 2004; Eichenbaum, 2004, for reviews). The parahippocampal cortex (PHC), in particular, was found to play a key role in a visual cortical network for analyzing contextual associations (Aminoff et al., 2006; Bar & Aminoff, 2003). We have previously proposed that this region serves as a switchboard of associations between visual items (Bar, 2004), and thus it is a natural candidate for exploring the interactive effects of identity- and location- contextual factors on object processing. Third, we investigated the effects of associative knowledge on perceptual regions directly involved in visual object perception. Specifically, we examined brain activity in object-selective regions within the ventrolateral occipital-temporal lobes, typically known as the lateral occipital complex (LOC, see, e.g., Grill-Spector, Kourtzi, & Kanwisher, 2001; Kanwisher, Chun, McDermott, & Ledden, 1996; Malach et al., 1995). We sought to explore whether evidence for an integrated contextual representation would be manifested in LOC, indicating that top-down contextual influences may directly affect visual object processing. Finally, it should be noted that by proposing a unified context frame for identity- and location- related associative information, we are not suggesting that such a representation necessarily resides within a specific brain region. Rather, we employ a more global approach according to which rich information about object identities and their spatial relations can reside within and/or exert influence on a network of brain regions. Such a network potentially encompasses areas subserving semantic processing, visual-contextual analysis, and possibly other high order functions. The concept of a context

10 Integrated representation for identities and locations Gronau et al., 10 frame, then, should be viewed as a general framework for investigation of high-level associative processing, rather than a construct represented within a highly localized cortical region. Methods Participants. Twenty volunteers (ten females; mean age = 25, range = 23 to 28 years) participated in the experiment. Nineteen of the subjects were right-handed. All participants gave written, informed consent before participation in the study (all procedures were approved by Massachusetts General Hospital Human Studies Protocol number 2001P ). Materials and task procedures. Subjects underwent fmri scanning while performing a fast event-related priming task. Prime stimuli consisted of real world objects and target stimuli consisted of either real objects (50%) or nonsense objects (50%). Subjects task was to classify rapidly whether the target in each trial was a real or a nonsense object by using a twoalternative forced-choice key press. The real/nonsense object task was adopted from the lexical decision task in the verbal semantic priming domain, in which subjects typically judge whether a target letter string is a real or a nonsense word. Real-object stimuli (including both primes and targets) consisted of everyday objects, such as furniture, tools, and clothing. Nonsense-object targets consisted of artificial, colorful shapes that were manually designed with Adobe Photoshop. Primes always appeared at the center of the screen, whereas targets either appeared at an upper (50%) or lower (50%) screen location, regardless of their identity. Critically, primes served as spatial cues, as they implied that a target would appear, under real world conditions, in one of the two task-relevant locations (e.g., a picture of a dresser implied that the target would appear in an upper location, since an object would typically be placed above, and not below the dresser; whereas a picture of a drill pointing down suggested that the target would appear in a lower location, since the drill was directed toward an object potentially located below it; see Figure 1a). Accordingly, a target was defined as spatially congruent if it

11 Integrated representation for identities and locations Gronau et al., 11 appeared in a plausible (expected) location, and it was defined as spatially incongruent if it appeared in an implausible (unexpected) location, with respect to the prime. In effect, half of the targets appeared in a spatially congruent location and half appeared in a spatially incongruent location, with an equal proportion of congruent and incongruent targets in upper and lower screen positions. In addition to manipulation of target location, within the real object target trials, half of prime and target pairs were semantically related (e.g., dresser and mirror), and half were semantically unrelated (dresser and pot), forming an orthogonal two-by-two factorial design with four equally probable conditions (see Figure 1b). [please bring Figure 1 about here] Notice that the plausibility, or congruency, of the target s position relative to the prime was independent of whether the prime and target were semantically related or unrelated. In the example shown in Figure 1b, for instance, a prime dresser would typically rest on the floor (as is the case with tables, beds, sofas, etc), and thus any target object would potentially be placed above, and not below, the dresser. Prime and target object pictures appeared on a gray background square, corresponding to a visual angle of approximately 14.6! The gray square appeared at the center of a black screen and was divided into 3 imaginative horizontal sections in which a prime or a target object could be presented (the former occupying the central section, whereas the latter occupying either the upper or the lower sections; prime and target stimuli were centered within each horizontal plane). Object pictures spanned approximately 4.9! 4.9, with a 4.9 center-to-center distance between prime and target stimuli. There were altogether 144 real-object target pairs (36 in each condition) and 144 nonsenseobject target pairs (for which a separate set of primes than that of the real-object target pairs was used). In addition, 156 fixation trials were randomly interleaved between the prime-target trials, to allow optimal deconvolution of the BOLD signal (stimuli randomization was determined by the Optseq program within the FS-FAST software tools, see

12 Integrated representation for identities and locations Gronau et al., 12 Upon presentation of all 444 trials (288 real and nonsense target pairs, plus 156 fixation trials), stimuli were repeated in a new randomized order, such that each prime-target pair was presented twice in the course of the experiment. Importantly, each subject was exposed to a prime stimulus in only one of the four experimental conditions (counterbalanced across subjects), and the same prime-target pair seen at initial presentation was also seen upon repetition. The temporal difference between the first and second stimulus presentation was approximately 15 minutes, and it was equal across all four conditions. Trials in which subjects made errors (5%) or exhibited extreme reaction times (above or below three standard deviations, 3%) were excluded from further statistical analyses. Prior to participation in the study, subjects performed a practice session of 52 trials that differed from the experimental trials. Localizer task for defining object-processing brain regions. In addition to the main priming task, a functional localizer task was administered to define object-selective regions within LOC. Subjects performed a 1-back task in which they viewed alternating blocks of faces, houses, outdoor scenes, and everyday objects (excluding objects presented in the main priming task). Within each block subjects were required to indicate via key press whether a picture was presented twice successively during the block (10% of trials). There were 12 presentation blocks for the object picture category and 6 presentation blocks for each of the other picture categories. Pictures within a block were presented for 700ms with a 300ms blank interstimulus interval (ISI), for a total of 20 pictures presented at a duration of 20 seconds. During rest periods between blocks (20 seconds) subjects passively fixated on the screen. FMRI data acquisition. Stimuli were presented with a Matlab software and were back-projected onto a translucent screen for viewing in the MRI scanner via a mirror attached to a custom-built

13 Integrated representation for identities and locations Gronau et al., 13 head-coil. All images were acquired in a 3T Siemens-Allegra scanner at the MGH-Martinos Center in Charlestown, Massachusetts. For each subject, two high-resolution structural images were acquired for spatial normalization and cortical surface reconstruction using a 3D MPRAGE (T1 weighted) sequence (128 sagittal slices, TR = 2.53 s, TE = 3.25 ms, flip angle = 7, FOV = 256, in-plane resolution 1 1 mm, slice thickness = 1.33 mm). Subsequently, a series of functional images was collected using a T2*-weighted EPI sequence of 33 interleaved slices oriented along the AC-PC line (TR = 2 s, TE = 25 ms, flip angle = 90, FOV = matrix, inplane resolution mm, slice thickness = 3 mm + 1 mm skip). Functional imaging parameters were identical for both the priming and the localizer task. Statistical image analysis. Functional data were analyzed using the FS-FAST analysis tools (see detailed description in Bar & Aminoff, 2003). Functional images were motion corrected (using the AFNI package, Cox, 1996) and spatially smoothed with a 8 mm Gaussian full-width, half-maximum (FWHM) filter. The intensities for all runs were then normalized to correct for signal intensity changes and temporal drift, with global rescaling for each run to a mean intensity of In order to obtain whole brain group activation maps in the event-related priming task, a Finite Impulse Response (FIR) model was used. This model does not make any a priory assumptions about the shape of the hemodynamic response (HDR). Linear models are used to estimate the response amplitude at each time point of the HDR (see Burock & Dale, 2000, for details of this technique). Since the BOLD response typically begins 2 s after stimulus onset and peaks between 4 and 7 s (e.g., Dale & Buckner, 1997), and in order to obtain a reliable estimation of the HDR that is based on an optimal time window of the BOLD activation, the signal intensity in each condition was averaged across 2-8 seconds from trial onset (see, e.g, Manoach, Greve, Lindgren, & Dale, 2003). Motion parameters derived from realignment correction were also entered to the model as covariates of no interest. The data were then tested for statistical significance and activation maps were constructed for specific contrasts of

14 Integrated representation for identities and locations Gronau et al., 14 interest. Average group results were obtained using a random-effect statistical model, and were projected onto an inflated brain (Fischl, Sereno, & Dale, 1999) with an average curvature of 80 different brains. Cortical activation maps were corrected for multiple comparisons by employing a Monte Carlo simulation and obtaining significance levels with a cluster-wise threshold. Specifically, we ran 10,000 Monte Carlo simulations of the smoothing, resampling, averaging and thresholding procedures using synthesized white Gaussian noise data. This allowed us to derive a p-value for each of the clusters in our analysis which indicated how likely it was that a cluster of a certain size would be found by chance given an initial voxel-wise threshold of p= Only clusters significant at a cluster-wise p-value of 0.01 were presented in the final whole-brain analysis. Region of interest (ROI) analysis. ROIs for the inferior prefrontal cortex (IPC), parahippocampal cortex (PHC), hippocampus, and lateral occipital complex (LOC), were constrained both structurally and functionally. The structural constraint was based on a hand labeling of these brain structures for each subject. In the IPC the anatomical label was restricted to posterior IPC (i.e., frontal operculum and inferior precentral sulcus), as only little activation was obtained for the functional mask (see below) within the anterior IPC. In the PHC, the anatomical label consisted of the collateral sulcus and the parahippocampal gyrus, and in the LOC it consisted of the fusiform gyrus, the occipital-temporal sulcus, and the more dorsal lateral occipital (LO) region. The additional functional constraint for the ROI analysis in each subject was based on an unbiased mask selecting only the subset of the voxels within each anatomical label that were activated in a positive direction by any component of the priming task (i.e., all conditions > fixation-baseline). For LOC, the functional mask was defined by comparing activation for objects versus faces, houses, and scenes within the localizer task (see e.g., Hasson, Harel, Levy, & Malach, 2003). All functional masks were obtained by using an estimated hemodynamic response that was defined by a gamma function of 2.25 sec

15 Integrated representation for identities and locations Gronau et al., 15 hemodynamic delay and 1.25 sec dispersion. For each region of interest a minimum of 20 voxels, active at a threshold of p<0.01 (uncorrected), was required in order to conduct the ROI analysis. For subjects who did not comply with these criteria (ranging from 0 to 4 subjects, depending on the specific brain region), a less stringent threshold of p<0.05 was used for the functional mask. Subjects who still showed no activation under these conditions were discarded altogether from the ROI analysis (1 subject in left IPC, 2 subjects in left PHC, 3 subjects in left and right hippocampus, and 2 subjects in right LOC; altogether 6 subjects were excluded from one or more analyses, with a maximal exclusion of 3 analyses for one of the subjects). Voxels from the ROIs were then averaged for each time point in each condition, within each anatomical structure. Percent signal changes from baseline were computed and averaged across 2-8 seconds from stimulus onset, allowing to conduct statistical tests for specific contrasts of interest across subjects. Results Behavioral Results The primary goal of the study was to assess whether identity-based and location-based contextual factors affect target object recognition with or without interacting, implying a unified vs. independent representations, respectively. Each prime-target pair was presented twice during the course of the experiment, and thus a secondary goal was to investigate whether effects of repetition (e.g., repetition priming) were modulated by the different contextual factors. Because semantic relations could be manipulated only between prime objects and real target objects, and to simplify results presentation, we discarded all nonsense target object trials from the statistical analyses. The results, therefore, focus on the four contextual conditions within the real target object trials (Figure 1b). Overall accuracy level in the target classification task was high (95%), with no trend for speed-accuracy tradeoff. A repeated measures ANOVA of reaction time (RT) for the real target

16 Integrated representation for identities and locations Gronau et al., 16 objects, with Semantic (related, unrelated) and Spatial (congruent, incongruent) relations as within-subject factors, revealed shorter latencies for the semantically related than the unrelated condition (main effect of the semantic factor, F 1,19 = 15.35, p<0.001). This finding replicates typical semantic priming effects obtained with both verbal and pictorial stimuli (e.g., Ganis & Kutas, 2003; McPherson & Holcomb, 1999; Van Petten & Luka, 2006). There was no main effect for the spatial factor (F 1,19 = 0.32), however a (marginally) significant interaction emerged between the semantic and spatial factors (F 1,19 = 4.17, p<0.055), suggesting that the former was modulated by prime-target spatial relations. Namely, shorter RTs were obtained for semantically related than for unrelated targets in the spatially congruent (t 19 = 4.42, p<0.001, two-tailed), but not in the incongruent condition (t 19 = 0.09, see Figure 2a). All other simple main effects were not significant (p>0.05). [please bring Figure 2 about here] A closer inspection of the results revealed that the interaction above was further qualified by differences between the first and second presentation of the stimuli (Figure 2b). Specifically, whereas a semantic main effect was obtained at first presentation (F 1,19 = 4.52, p<0.05), semantic and spatial factors interacted only upon stimulus repetition (i.e., in the second presentation, F 1,19 = 9.48, p<0.006). A three-way ANOVA confirmed that the Semantic (related, unrelated) Spatial (congruent, incongruent) Presentation (first, second) interaction was statistically significant (F 1,19 = 5.77, p<0.03). This interaction could also be described as a Semantic Spatial interaction in repetition-priming (RP) effects (i.e., the difference in RT between the first and second presentation, see Figure 2b), where a greater RP effect was found for semantically related than for unrelated targets within the spatially congruent condition (t 19 = 2.36, p<0.03), but not in the incongruent condition (t 19 = -0.94). Repetition priming effects are often considered to reflect a form of learning that improves perceptual processing and/or response selection for a repeated target (see Schacter, Dobbins, & Schnyer, 2004, for a review). The Semantic Spatial interaction in RP indicates that semantic relatedness affected

17 Integrated representation for identities and locations Gronau et al., 17 efficiency processing of a repeated target, however, this effect was dependent on the target s location (whether congruent or incongruent with respect to the prime). Note that we describe the interaction effects obtained as a modulation of semantic effects by spatial condition. However, in some cases (such as the Semantic Spatial interaction in the second presentation), spatial contextual effects were also modulated by semantic factors, resulting in significantly faster RTs for spatially congruent than incongruent targets in the semantically related condition; along with a reversed pattern of results in the semantically unrelated condition. These findings indicate that spatial consistency can facilitate recognition of a semantically related object, yet it may also hinder recognition of a semantically unrelated object (possibly due to violation of expectations regarding the most probable object identity to appear in a spatially congruent location). Interestingly, and in contrast to expectations, nonsense object targets (not shown in Figure 2) displayed a similar pattern of results to that of the semantically unrelated objects, i.e., slower RTs for the spatially congruent than the incongruent locations, suggesting that these nonsense patterns were actually perceived as unrelated (rather than neutral) objects within the specific task context (we discuss possible implications of this outcome in the Discussion section). Despite the spatial effects mentioned above, we refer throughout the manuscript mainly to the modulation of semantic contextual effects by spatial location since it was most consistent across analyses, both in the RT and in the fmri data below. Our behavioral findings suggest, therefore, that semantic and spatial associations can affect object recognition in an interactive manner. Although there was no Semantic Spatial interaction in the first stimulus presentation, subjects were nevertheless sensitive to the global configuration of prime and target, as reflected by the interactive results in the second experimental exposure. Furthermore, a semantic main effect of target identity was obtained during first presentation, reflected by shorter latencies to semantically related than unrelated objects, regardless of target location. These findings suggest that subjects may have used

18 Integrated representation for identities and locations Gronau et al., 18 different types of schematic knowledge during initial and repeated presentations. Specifically, while responses to first presentation were likely mediated by a pure semantic (or conceptual) contextual schema that is abstracted of specific spatial relations, responses during the second presentation were dominated by a combined semantic-spatial contextual representation, as reflected by the Semantic Spatial interaction in RTs. We propose that activation of such a rich visual contextual representation was triggered by the exposure to the prime-target pairs and the encoding of their spatial relations during the initial presentation. Whereas subjects were equally exposed to stimuli in all contextual conditions, however, our results imply that encoding of prime-target relations was most efficient for stimuli pairs that formed a coherent percept, i.e., objects that were consistent with each other in both semantic and spatial dimensions. We will further elaborate on the differences in results between the initial and repeated experimental presentations in the Discussion section. fmri Results Semantic processing in the inferior prefrontal cortex (IPC). To account for the differences between the first and second presentations obtained in the behavioral (RT) results, we analyzed the BOLD responses from the two presentations separately. Results for the first presentation within the posterior IPC showed a clear interaction of Semantic Spatial conditions in both hemispheres (LH: F 1,18 = 12.48, P<0.002; RH: F 1,19 = 6.27, p<0.03 ), reflecting a larger percent signal change for the semantically related than for the unrelated targets within the spatially congruent (LH: t 18 = 3.82, p<0.001; RH: t 19 = 2.12, p<0.05), but not the incongruent (p>0.05 in both hemispheres) condition. Interestingly, this pattern of results was reversed in the second presentation, where semantically related targets showed reduced percent signal change compared with unrelated targets. Once again, this difference was significant only in the spatially congruent condition (LH: t 18 = -3.31, p<0.004; RH: t 19 = -2.42, p<0.03; see Figure 3a). [please bring Figure 3 about here]

19 Integrated representation for identities and locations Gronau et al., 19 Note that the interaction obtained in the BOLD responses for the first presentation was in contrast to the lack of interaction found in the behavioral results, whereas both measures (BOLD signal and RT) showed a similar pattern of responses in the second presentation (i.e., reduced/faster responses for semantically related than for unrelated targets, in the spatially congruent, but not the incongruent, condition). The fmri findings for the repeated items (in the second presentation) mirror typical semantic priming effects, in which response reductions are usually obtained for semantically related, relative to unrelated items (for a review, see Van Petten & Luka, 2006). In contrast, the response enhancement found for the semantically related targets in the first presentation (within the spatially congruent condition) suggests that rather than being primed and thus activated to a lesser degree, these items may have benefited from deeper encoding than the unrelated targets, resulting in significantly enhanced activation. This account is in accordance with our prior suggestion that subjects encoded more efficiently primetarget pairs that constructed a coherent percept, than an incoherent one, during the first experimental presentation. The fact that prime and target stimuli were presented within very short temporal intervals, and at adjacent spatial locations, further supports a post-recognition associative account for the increased activation. Namely, subjects actively associated, or linked, prime and target into a global visual percept subsequent to the presentation and recognition of prime-target pairs (as is the case with post-lexical matching in the verbal semantic priming domain, e.g., Neely, 1991). Indeed, studies investigating associative encoding of words and pictures (e.g., using word pair associates, or face-name associates) have typically shown increased activation in the prefrontal cortex during successful encoding processing (Dolan & Fletcher, 1997; Montaldi et al., 1998; Sperling et al., 2001). Most importantly, the unique encoding-related activity found for meaningful contextual configurations suggests that the latter benefited from rich pre-existing long-term memory representations. As mentioned previously, the differences between the fmri data in the first and second presentations can also be portrayed as a Semantic Spatial interaction in repetition-priming

20 Integrated representation for identities and locations Gronau et al., 20 (RP) effects (LH: F 1,18 = 16.77, p<0.001; RH: F 1,19 = 11.11, p<0.003), reflecting greater responsereduction for the semantically related than for the unrelated targets in the spatially congruent (LH: t 18 = 4.48, p<0.001; RH: t 19 = 2.84, p<0.01), but not the incongruent condition. These results replicate our behavioral findings, in emphasizing that the efficiency at which targets were processed during their repeated presentation was strongly affected by a joint influence of semantic and spatial factors. Interestingly, the fmri pattern for RP also mirrored the fmri results for the first presentation, revealing maximal response-reduction for targets that initially showed maximal encoding-related activation (in the semantically related, spatially congruent, condition), and relatively little, if any, response-reduction for targets that initially showed low levels of activation (in all other conditions). Note that the little response-reduction mentioned cannot be attributed to a floor effect, as activation levels in the second presentation for these conditions exceeded that of the semantically related, spatially congruent, condition (in fact, in the semantically unrelated, spatially congruent, condition there was a response-enhancement upon stimulus repetition, suggesting that greater neural resources were required to process an unexpected object identity appearing in an expected location). The selectivity of responses to prime-target pairs that constituted a coherent percept is in agreement with classical RP findings obtained with single cell recordings, in which greatest neural reduction was found for repeated stimuli among neurons that were most active at first presentation (Li, Miller, & Desimone, 1993). Thus, a differential pattern of enhanced activation, followed by a significant drop in the BOLD signal, was obtained for targets consistent with primes in both semantic and spatial dimensions, further supporting our hypothesis that these two associative factors are linked within a unified contextual representation. Visual contextual processing in the parahippocampal cortex (PHC). The overall pattern of results in the PHC resembled that of the IPC (see Figure 3b), however only the left hemisphere showed statistically significant Semantic Spatial interaction effects. Namely, a significant

21 Integrated representation for identities and locations Gronau et al., 21 interaction was found for the first presentation in the left PHC (F 1,17 = 7.74, p<0.02), indicating a larger activation for the semantically related than for the unrelated targets in the spatially congruent (t 17 = 2.76, p<0.02), but not in the incongruent condition. Similar to findings in the IPC, a reversed pattern of activation was seen in the second presentation (i.e., stronger activation for the semantically unrelated than for the related targets, only in the spatially congruent condition), resulting in a three-way Semantic Spatial Presentation interaction (F 1,17 = 9.15, p<0.008). These findings replicate our previous behavioral and neural results of greater responsereduction for the semantically related than for the unrelated items appearing in a congruent (t 17 = 2.79, P<0.02), but not in an incongruent location. Paralleling findings were obtained within the left hippocampus, reflecting a significant Semantic Spatial interaction in the first presentation (F 1,16 = 7.76, p<0.02), as well as in the repetition-priming effects for the second presentation (F 1,16 = 5.66, p<0.03). Results from the right hemisphere of both the PHC and the hippocampus showed a similar trend to those of the left hemisphere, though none of the interactions reached significance (p>0.05). Object-related processing in the lateral occipital complex (LOC). Analysis of activation from the LOC replicated our findings from previous ROI analyses, however, similar to the PHC and hippocampus, the Semantic Spatial interactions obtained were restricted to the left hemisphere (see Figure 3c). Specifically, a semantic effect, depicting increased activation for semantically related items in the first presentation, was found for the spatially congruent (t 19 = 2.47, p<0.03), but not the incongruent condition (F 1,19 = 5.21, p<0.04, for the Semantic Spatial interaction). In addition, a reversed trend in the second presentation resulted in greater RP effects for the semantically related than for the unrelated targets. This difference was evident only in the spatially congruent condition (t 19 = 3.06, p<0.006; F 1,19 = 5.43, p<0.04, for the threeway interaction). A similar pattern of activation was found in the right hemisphere, however all

22 Integrated representation for identities and locations Gronau et al., 22 interaction effects failed to reach significance (p> 0.05). Whole brain analysis of Semantic Spatial interaction effects. In addition to the ROI analyses, we conducted a whole-brain group analysis examining contextual-dependent activation for specific contrasts of interest. The contrasts of semantic and spatial main-effects (e.g., semantically related versus unrelated, across spatial congruency; and spatially congruent versus incongruent, across semantic relatedness) revealed very little, if any, brain activation, both within and across experimental presentations. We therefore focus on the statistical activation maps evoked by the two main interaction effects obtained in the study (see Figure 4ab): 1) a Semantic Spatial interaction during the first presentation (defined as [semantically related semantically unrelated, within the spatially congruent location] - [semantically related semantically unrelated, within the spatially incongruent location]); and 2) the three-way (Semantic Spatial Presentation) interaction, corresponding to the Semantic Spatial interaction for the repetition-priming (RP) effect (defined as [RP for semantically related RP for semantically unrelated, within the spatially congruent location] - [RP for semantically related RP for semantically unrelated, within the spatially incongruent location]). Recall that these two interaction contrasts showed consistently robust effects in previous ROI analyses, indicating a differential semantic effect within the spatially congruent condition, during both initial stimuli presentation and upon prime-target repetition. [please bring Figure 4 about here] The statistical activation map for the Semantic Spatial interaction effect in the first stimulus presentation (Figure 4a) revealed significant activations in regions previously subjected to a ROI analysis, namely, bilateral posterior IPC (~BA 44/45), and left lateral occipito-temporal sulcus (within the LOC). Although demonstrating a significant effect in the ROI analysis, activation in the left PHC and the left hippocampus did not reach significance in the whole-brain analysis.

23 Integrated representation for identities and locations Gronau et al., 23 Other significant clusters of activation for the interaction contrast were found in bilateral premotor cortex (precentral sulcus and gyrus), sub-central gyrus, superior temporal lobe, cingulate cortex, and precuneus and cuneus gyrus (extending to the calcarine sulcus and the lingual gyrus). In addition, activation was found in the right anterior frontal cortex (~BA 10) (see Table 1). Notably, a very similar map of activation was obtained for the interaction contrast of the RP effect (or the Semantic Spatial Presentation interaction), albeit this RP contrast produced overall stronger activity spanning over larger brain regions (see Figure 4b). The relative similarity between the two activation maps indicates once again that maximal response reduction for repeated stimuli was obtained in regions that were initially most reactive to the joint effect of semantic and spatial associative factors. Among those regions were the IPC (spanning both posterior and anterior inferior frontal cortex), and the left LOC (including lateral occipitaltemporal sulcus and fusiform gyrus), as well as other medial and lateral regions originally activated by the Semantic Spatial interaction in the first presentation (see Table 2). [please bring Tables 1 & 2 about here] In addition, lateral parietal activation was observed bilaterally in both dorsal (e.g., intraparietal sulcus, straddling the post-central sulcus) and ventral (e.g., inferior supramarginal parietal gyrus) foci, suggesting a potential role for attention in the priming effect for the repeated items (see e.g., Corbetta & Shulman, 2002, for the role of parietal cortex in attentional allocation). One possible explanation for such attentional involvement is that perceptual processes, or processes associated with response-selection, are less attentionally demanding for (repeated) semantically related than unrelated targets. This difference in processing efficiency, however, accounts only for targets appearing in spatially congruent, but not incongruent, locations. Alternatively, the parietal activations (in conjunction with the robust frontal activations) may reflect memory retrieval processes, occurring either implicitly or explicitly upon stimuli repetition (see reviews in Buckner & Wheeler, 2001; Wagner, Shannon,