Rapid Onset Relational Memory Effects Are Evident in Eye Movement Behavior, but Not in Hippocampal Amnesia

Transcription

1 Rapid Onset Relational Memory Effects Are Evident in Eye Movement Behavior, but Not in Hippocampal Amnesia Deborah E. Hannula 1, Jennifer D. Ryan 2, Daniel Tranel 3, and Neal J. Cohen 4 Abstract & Little is known about the mechanisms by which memory for relations is accomplished, or about the time course of the critical processes. Here, eye movement measures were used to examine the time course of subjects access to and use of relational memory. In four experiments, participants studied faces superimposed on scenic backgrounds and were tested with threeface displays superimposed on the scenes viewed earlier. Participants exhibited disproportionate viewing of the face originally studied with the scene, compared to other equally familiar faces in the test display. When a preview of a previously viewed scene was provided, permitting expectancies about the to-be-presented face to emerge, disproportionate viewing was manifested within msec after test display onset, more than a full second in advance of explicit behavioral responses, and occurred even when overt responses were not required. In the absence of preview, the viewing effects were delayed by approximately 1 sec. Relational memory effects were absent in the eye movement behavior of amnesic patients with hippocampal damage, suggesting that these effects depend critically on the hippocampal system. The results provide an index of memory for face scene relations, indicate the time by which retrieval and identification of these relations occur, and suggest that retrieval and use of relational memory depends critically on the hippocampus and occurs obligatorily, regardless of response requirements. & INTRODUCTION Many of the current investigations of memory focus on processes and representations that support memory for previously experienced (as compared to novel) items and the changes in brain activity that accompany them. For example, studies have shown differences in eventrelated potential (ERP) waveforms and functional magnetic resonance imaging (fmri) measures of cortical activity (e.g., Buckner & Wheeler, 2001; Ranganath & Paller, 2000; Rugg & Allan, 2000), as well as changes in eye movement behavior (Ryan, Althoff, Whitlow, & Cohen, 2000; Althoff & Cohen, 1999), for novel versus repeated words, faces, objects, and scenes. In addition, localized electrical and hemodynamic changes associated with successful retrieval of memory for previously experienced items have been reported (see McDermott & Buckner, 2002; Buckner & Wheeler, 2001), and recent investigations have explored the behavioral and neural correlates of recollective processes as distinct from familiarity-based processes presumed to be involved in recognition memory for recently studied items (see Yonelinas, 2002 for a review). 1 University of California, Davis, 2 The Rotman Research Institute, Toronto, Canada, 3 University of Iowa, 4 University of Illinois, Urbana-Champaign Considerably less research has addressed mechanisms of memory for relations among items, such as relations among the constituent elements of recently viewed scenes or recently experienced events. Accomplishing memory for relations, above and beyond memory for the items themselves, would seem a significant challenge, particularly for items that were only arbitrarily or accidentally associated by co-occurrence. Consistent with this intuition, it has been shown, in the context of response signal paradigms, that memory for interitem relations becomes available later in time than memory for individual items. In one of these investigations, at least 200 msec more time was required to distinguish repeated pairings from re-pairings (of old words from different pairs) than to distinguish repeated pairings from entirely novel ones (Gronlund & Ratcliff, 1989). Similar results have been reported in ERP investigations in which differential brain activity to repeated pairs of items, as compared to re-pairs and novel pairs, begins approximately 600 msec after stimulus onset; activity distinguishing novel from studied items, irrespective of the pairings, is evident just 400 msec after stimulus onset (Donaldson & Rugg, 1998, 1999; Weyerts, Tendolkar, Smid, & Heinze, 1997; but see Tsivilis, Otten, & Rugg, 2001). It has been suggested that the later-emerging component (the P600), sensitive to the integrity of interitem D 2007 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 19:10, pp

2 relationships, reflects processing in medial-temporal lobe (MTL) structures (e.g., Dietl et al., 2005; Trautner, Dietl, Staedtgen, et al., 2004), and consistent with this proposal, the P600 component is absent from scalp-recorded ERPs in amnesic patients with MTL damage (Duzel, Vargha- Khadem, Heinze, & Mishkin, 2001; Olichney et al., 2000). These results are in line with the view that relational memory performance selectively engages a particular brain system, the hippocampal system, and that successful performance on tasks demanding relational memory requires an intact hippocampus (cf. Cohen & Eichenbaum, 1993). Another relational property attributed to the hippocampus is the ability to support pattern completion, by which exposure to a subset of items from an earlier learning event can cause reactivation of memory for the entire original event (Norman & O Reilly, 2003; Eichenbaum & Cohen, 2001; O Reilly & Rudy, 2001; Cohen & Eichenbaum, 1993; Halgren, 1984; Squire, Cohen, & Nadel, 1984). On our view, pattern completion entails (and the term will be used in this article to refer to) the retrieval, in response to partial cues, of relational memory capturing previously bound information about various relations among constituent elements of scenes, events, or experiences. Here, in four different experiments, we use eye movements to examine the time course of subjects access to and use of relational memory. Patterns of eye movements elicited by stimuli of various kinds can provide information about aspects of memory representation and knowledge without appealing to subjective (and possibly unavailable) verbal reports, and can provide information about the time course of processing, as documented in studies of reading (e.g., Yang & McConkie, 2001; Rayner, 1998), language (e.g., Griffin & Bock, 2000), scene perception (e.g., Henderson & Hollingworth, 1999; Rayner, 1998; McConkie & Currie, 1996; Loftus & Mackworth, 1978), and memory (Ryan et al., 2000; Althoff & Cohen, 1999). The present work extends a paradigm in which we assessed memory for previous exposure to studied faces by examining viewing of three-face test displays (Ryan, Hannula, & Cohen, 2007). In that work, patterns of viewing changed as a function of the test block instructions. When participants were required to identify the studied face from among the three alternatives, they directed a disproportionate amount of viewing to that face; however, when free-viewing instructions were given, subjects spent more time viewing novel faces. This latter result is consistent with reports of novelty preferences that have been observed in both animal and human memory investigations (e.g., Manns, Stark, & Squire, 2000; Mishkin & Delacour, 1975; Fagan, 1970; Fantz, 1964), but critically, both effects are indicative of memory for the previously viewed faces. What distinguishes the current work from our own previous research, and from the body of work in which novelty preferences have been documented, is the emphasis on memory for relations among items, rather than memory for individual items. Participants in the experiments reported here viewed arbitrary (i.e., preexperimentally unrelated) face scene pairings, and were then tested with three-face displays superimposed on the previously studied scenes. All three faces in a test display were seen equally often during the study blocks and, therefore, were not distinguishable on the basis of differential previous viewing history. However, on some of the trials, one of the faces matched (i.e., had been studied with) the background scene. Because all of the faces were equally familiar, any disproportionate viewing of the matching face would be a function of its relation to the scene with which it was presented at study that is, a reflection of retrieval and use of relational memory for specific face scene pairings studied earlier. The notion that eye movement behavior to previously studied faces might be influenced by representation of the scenes with which the faces were studied (i.e., by context reinstatement) has intuitive appeal. Behavioral research has shown that face processing in context influences the speed and accuracy with which face recognition decisions are made (Rainis, 2001; Beales & Parkin, 1984; Memon & Bruce, 1983; Davies & Milne, 1982; Klee, Leseaux, Malai, & Tiberghien, 1982), and it has been suggested that memory for unfamiliar faces, in particular, may be bound up in the contexts with which they have been associated (Bruce & Young, 1986). All the experiments here required participants to learn arbitrary pairings between previously unfamiliar faces and unfamiliar scenes. Manipulations of experimental conditions and subject populations then permitted several questions to be addressed about relational memory. Information about the time course of retrieval and use of memory for face scene relations was sought by manipulating the timing of presentation at test of the scene with respect to the three-face test display. In Experiments 1, 2, and 4, each test trial began with a 3-sec preview of a previously viewed scene; thereafter, the test faces were superimposed on that scene. The scene preview encouraged retrieval of the associated face (i.e., pattern completion of the original pairing) prior to the subsequent appearance of the test display. Successfully retrieved relational memory during the preview period would promote expectancies about the to-be-presented face, which in turn could then guide viewing among the three faces of the test display, permitting differential viewing of the matching face to emerge very early in viewing of the test faces. In Experiment 3, no scene preview was provided, thereby restricting the retrieval of relational memory about face scene pairings until after the onset of the scene face test displays. Under these conditions, disproportionate viewing of the matching face would be expected to emerge more slowly than in the other experiments. Manipulation of instructions permitted us to address another issue about relational memory. Experiments 1 and 2 differed only in whether or not subjects were asked to make explicit memory judgments at test time. Participants were either given recognition instructions Hannula et al. 1691

3 or were instructed just to study the three faces along with the associated scene. Finding very comparable eye movement effects, with viewing directed disproportionately to the matching face regardless of response requirements, would suggest that the disproportionate viewing effect and the retrieval and use of relational memory on which it depends can emerge spontaneously (automatically). Finally, in order to determine whether these viewing effects depend upon the integrity of the hippocampal system, participants in Experiment 4 were amnesic patients with hippocampal damage and a matched comparison group. Intact disproportionate viewing of the matching face might be expected, even in hippocampal amnesia, on an account that emphasizes the role of the hippocampus (and the deficit in amnesia) in explicit as opposed to implicit memory (e.g., Schacter, 1987; Graf & Schacter, 1985), as eye movements can be considered an indirect (or implicit) measure of memory for previous exposure. By contrast, the absence of these viewing effects in amnesia is predicted by the view that hippocampus is critical in memory for arbitrary relations (e.g., Hannula, Tranel, & Cohen, 2006; Moses & Ryan, 2006; Eichenbaum & Cohen, 2001; Cohen & Eichenbaum, 1993). EXPERIMENTS 1 3 Methods Participants Participants were 108 students from the University of Illinois (36 different participants in each of 3 experiments) who were compensated with course credit or payment. Apparatus Eye position was monitored at a rate of 60 Hz using an Applied Science Laboratories (ASL) 4250R remote eye tracker in Experiments 1 and 2, and with an ASL model 504 remote eye tracker in Experiment 3. Stimulus presentation and data collection were controlled by Windows-based computers with 21-in. color displays. Stimuli and Design The stimuli used in all three experiments were 36 male and 36 female full-color face images selected from our faces database (see Althoff & Cohen, 1999), and 36 fullcolor images of real-world scenes acquired from Brand X photography. Each face was sized to pixels and placed upon a pixel uniform gray background; scenes measured pixels. Thirty-six faces, from the set of 72, were paired with the scenic backgrounds, resulting in 36 face scene pairs that were presented in the study blocks; the remaining 36 faces were used in the test block only. Each face scene pair was presented once per block in each of five study blocks, with presentation order independently randomized across blocks. In a sixth test block, 24 displays were shown, each consisting of three faces superimposed on one of the scenes that had been presented during study. Three different display types were used in the test block: (6) match, (6) re-pair, and (12) novel displays. In each match display, all three faces were previously studied; one was studied with (and thus matched) the scene on which it was superimposed. In each re-pair display, all three faces were previously studied, but none had been studied with the scene. In each novel display, none of the three faces had been seen before (see Figure 1). Faces and scenes were randomly assigned to lists, and counterbalancing was conducted such that each list of faces was seen equally often with each list of scenes during the study blocks; these lists also rotated across experimental conditions. In addition, any matching face in match displays appeared equally often in each spatial position (i.e., left, right, and bottom) across trials. Procedure After each participant provided informed consent, the eye tracker was calibrated using a 3 3 spatial array, in a process repeated prior to the initiation of each block. Each experimental trial began contingent upon the subject fixating a centrally located cross-hair. Participants took breaks between blocks as necessary, and were fully debriefed upon completion of the test block. Study blocks. Study block trials in all experiments began with the presentation of a scene for 3 sec, followed by the presentation of a single face superimposed on the center of that scene; the face scene pair remained on the screen for 5 sec. Subjects were simply instructed to study them, committing each pair to memory so that they would be able to identify the match between a particular face and the associated background scene when a recognition test was administered. Test block. The test blocks differed across experiments, as follows (also see Figure 1): EXPERIMENT 1. Test block trials began with presentation of a studied scene for 3 sec (scene preview), followed by presentation of the three test faces superimposed on the scene for 10 sec. Participants were to identify the matching face in each three-face display, even when they felt that none of the faces had been presented with that scene at study, by pressing a key corresponding to the spatial position of the face. In this way, a response was made for all trial types, including novel and re-pair trials. Because only match trials had a face that matched the scene, whereas all three trial types required responding, differences in eye movement behavior between match trials and novel or re-pair trials can be interpreted as 1692 Journal of Cognitive Neuroscience Volume 19, Number 10

4 Figure 1. (Top) Examples of single-face displays presented during the study blocks. On every trial, a scene was presented for 3 sec, followed by presentation of the face scene pair for 5 sec. A fixation screen was presented in between subsequent trials and remained in view until the participant was looking at the centrally located cross-hair. (Bottom) Examples of three-face displays presented during the test block. In each match display, all three faces had been studied previously; one had been studied with (and thus matched) that scene. In each re-pair display, all three faces had been studied previously, but none had been studied with that scene. In each novel display, none of the three faces had been seen before. A 3-sec scene preview was provided in Experiments 1, 2, and 4, followed by presentation of the three-face display superimposed on that scene for 10 sec. The scene preview was not provided in Experiment 3. Subsequent trials were separated by the presentation of a fixation screen; the trial was advanced as soon as the participant was looking at the centrally located cross-hair. being due to memory effects (i.e., related to a match between the test face and a representation of the relevant face scene pairing) as opposed to being due to response effects (related to the response or intention to respond) (see Results section). It was expected that eye movements would be drawn disproportionately to the matching face in match displays very early in viewing of the subsequent test display. EXPERIMENT 2. Test trials were presented as in Experiment 1. But rather than giving participants recognition instructions, they were instead instructed just to study the three faces along with the associated scene, for a recognition test that would be administered at the end of the experiment. However, there was no subsequent memory task; instead, eye movements elicited by the three-face test displays were used to measure memory indirectly. It was expected that the same early-emerging preferential viewing effects that were evident in Experiment 1 would occur here, indicating that such effects can emerge spontaneously or automatically. EXPERIMENT 3. The three-face test display was presented simultaneously with the scene (i.e., no scene preview), and remained on the screen for 10 sec. As in Experiment 2, eye movements elicited to three-face displays presented during the test block provided an indirect measure of memory. In the absence of scene preview, and the associated opportunity for pattern completion, it was expected that preferential viewing effects elicited by the matching face would be significantly delayed. Eye Movement Measures Evidence of retrieval and use of memory for face scene relations was taken from four classes of eye movement measures to the three-face displays in the test block: (1) Between-display measures included the number of fixations, and number of transitions among regions of the display, throughout the 10-sec viewing period. Differences between match and re-pair displays versus novel displays would indicate an effect of memory for items (studied vs. novel faces), whereas differences in sampling between match versus re-pair displays would indicate an effect of memory for face scene relations (co-occurrence). (2) Within-display measures included the proportion of fixations and proportion of viewing time directed to the matching face, within a match display throughout the 10-sec viewing period. Disproportionate viewing of the matching face from among the other, equally familiar, faces (i.e., significantly greater than the 33% that would be expected if viewing was equally distributed among the three faces in the display) would be attributable to the match between face and scene, indicating an effect of memory for face scene co-occurrence. (3) Time-course measures determined how early in time disproportionate viewing of the matching face actually emerged. Proportion of viewing time directed to the matching face was examined for each successive 1000-msec time bin ( msec, msec, etc.). Analyses focused particularly on the first 2 sec of viewing, aided by a more fine-grained analysis Hannula et al. 1693

5 involving each successive 250 msec bin within that time window (0 250 msec to msec). (4) Response-locked measures, calculated only for Experiment 1, determined how much in advance of the behavioral response disproportionate viewing of the matching face emerged. Eye movement data for each trial in Experiment 1 were aligned with respect to when the response was made, and the proportion of viewing time to the various faces was examined in each successive 500-msec time bin. The analysis was conducted on all trials associated with a response, for the 2 sec of viewing prior to and following the response, accumulated across all participants. Critically, if disproportionate viewing emerged differentially for the matching face in match displays than for the selected face in novel or re-pair displays, even though all trial types required a response, we could attribute that to an effect of memory for face scene co-occurrence. Statistical Analyses Omnibus analysis of variance (ANOVA) tests were performed, with corrections using the Greenhouse Geisser adjustment to the degrees of freedom (df ). Accordingly, for all F tests with more than 1 df in the numerator, both the corrected p value and the Greenhouse Geisser epsilon value (> ) are reported. The correction was applied because the Greenhouse Geisser epsilon values deviated substantially from 1.0 for several of the repeated-measures ANOVAs that were calculated. As this can be an indication that sphericity has been violated, we choose to report the more conservative p values. Effect size was calculated using Cohen s d with the standard deviation of the differences between groups as the standardizing unit. All post hoc comparisons were Bonferroni corrected. 1 Results and Discussion Results are reported for the test block only, for viewing directed to the faces during the 10 sec in which the faces and scene were presented together. 2 Participants who completed Experiment 1, in which behavioral responses were made, accurately identified the matching face in match displays on 95.2% of the trials, and a face was selected on 99.1% and 99.5% of the trials for re-pair and novel displays, respectively. Eye movement analyses were performed for these trials only. 3 Between-display Comparisons Significant between-display differences were observed in all three experiments. In Experiment 1, significant differences between match, re-pair, and novel displays were evident in the number of fixations made to a display [F(2, 68) = 15.04, p <.001, > =.83] and in the number of transitions in a display [F(2, 68) = 29.93, p <.001, > =.99]. Planned comparisons revealed that fewer fixations were made to match than to novel [t(34) = 4.02, p <.001, d = 0.68] or re-pair displays [t(34) = 4.97, p <.001, d = 0.84], and fewer transitions were made for match than for novel, [t(34) = 6.74, p <.001, d = 1.14] or re-pair displays [t(34) = 6.59, p <.001, d = 1.11]. In Experiment 2, eye movements continued to distinguish among the three display types despite the absence of explicit response requirements. As in Experiment 1, differences in patterns of viewing were evident both in the number of fixations [F(2, 70) = 3.92, p <.05, > =.95] and the number of transitions [F(2, 70) = 6.30, p <.01, > =.89]. Planned comparisons showed fewer fixations for match than for novel displays [t(35) = 2.61, p <.05, d = 0.44] and fewer transitions for match than for novel [t(35) = 3.70, p =.001, d = 0.62] or re-pair displays [t(35) = 2.38, p <.05, d = 0.40]. Finally, when the 3-sec scene preview was not provided (i.e., Experiment 3), significant differences across display types were evident for the number of fixations [F(2, 70) = 4.82, p <.05, > =.84]. Fewer fixations were made to both match and re-pair displays than to novel displays [t(35) = 2.63, p =.01, d = 0.44 and t(35) = 2.43, p <.05, d = 0.41, respectively]. Observed differences in viewing of match versus novel displays in Experiments 1 and 2, and in viewing of both match and re-pair versus novel displays in Experiment 3, with more exploratory eye movement behavior elicited by displays with novel faces than by displays with previously viewed faces, are entirely consistent with our earlier findings of eye movement effects reflecting memory for previously viewed items (Ryan et al., 2007, 2000; Althoff & Cohen, 1999). Critically, the viewing differences observed between match and re-pair displays in Experiments 1 and 2 cannot be due to memory for individual faces or scenes because all items in both display types were equated for previous exposure. Instead, these differences must be a reflection of retrieving and using memory for specific face scene co-occurrences. These preferential viewing effects do not appear to be a consequence of explicit response requirements because behavioral responses were required only in Experiment 1 and not in Experiment 2, but do appear to depend upon the 3-sec scene preview, as they were absent in Experiment 3. Within-display Comparisons Significantly disproportionate viewing of the matching face compared to the other equally familiar faces in match displays was observed only when a 3-sec scene preview was provided (i.e., in Experiments 1 and 2). In Experiment 1, both the proportion of fixations and the proportion of total viewing time directed to the matching face were significantly greater than the 33% (equally distributed) viewing level [t(34) = 3.31, p <.01, d = 0.55 and t(34) = 3.77, p =.001, d = 0.64, respectively]. In Experiment 2, the proportion of total viewing time 1694 Journal of Cognitive Neuroscience Volume 19, Number 10

6 directed to the matching face was significantly greater than the 33% viewing level [t(35) = 2.33, p <.05,d =0.39] although the comparable effect was not evident for proportion of total fixations. The disproportionate viewing effect for the matching face over other, equally well-studied faces within the match display, in Experiments 1 and 2, suggests an effect of memory for the relation between that face and the matching scene. It remains possible, though, that increased viewing of the matching face in Experiment 1 is an effect of response intention or execution rather than an effect of memory. That the effect in Experiment 1 can be attributed specifically to memory is evident in the results of time-course and response-locked analyses, described next. The time-course analyses also permit us to explore the possibility that the absence of a relational memory effect in Experiment 3 is attributable to using a measure of viewing collapsed across time. Time-course Measures These results are shown in Figures 2 (top), 3 (top), and 4 (top). Significantly disproportionate viewing of the matching face in match displays emerged very early in the viewing period when a 3-sec scene preview was provided; greater than chance viewing of the matching face was evident within the first 1000 msec after the faces were presented, both in Experiment 1 [t(34) = 5.95, p <.001, d = 1.00] and Experiment 2 [t(35) = 4.12, p <.001, d = 0.69]. Critically, in Experiment 1, where there was responding for all display types, planned comparisons revealed that preferential viewing of the matching faces in match displays exceeded that for selected faces from both re-pair [t(34) = 3.87, p <.001, d = 0.65] and novel displays [t(34) = 3.34, p <.01, d = 0.56] within the first 1000 msec. That the effect was seen specifically for matches rather than for all responded-to faces indicates that it is an effect of memory for face scene pairings rather than any effect of responding. In the absence of scene preview (Experiment 3), preferential viewing of the matching face was evident msec after test display onset [t(35) = 2.70, p =.01,d = 0.45], a full second later than the effects observed in the previous experiments. The results, further partitioned into 250-msec time bins within the first 2 sec of viewing, are shown in Figure 2. Proportion of total viewing time allocated to the correctly identified matching faces and to faces that were selected from re-pair and novel displays in Experiment 1. (Top) The mean proportion of viewing time to matching and selected faces is shown for the final block, in 1000-msec time bins, with standard error bars plotted around the means. Significantly disproportionate viewing of the correctly identified matching face relative to faces selected from re-pair and novel displays occurred within the very first 1000-msec time bin. (Bottom) Eye movement data from the first 2 sec of each trial in the final block partitioned into 250-msec time bins. Significantly disproportionate viewing of the correctly identified matching face compared to faces that were selected appeared within msec after the faces were presented. Hannula et al. 1695

7 Figure 3. Proportion of total viewing time allocated to matching faces in Experiment 2. (Top) The mean proportion of viewing time to matching faces is shown for the final block, in 1000-msec time bins, with Bonferroni-corrected 99% confidence intervals plotted around the means. The dashed line indicates equally distributed viewing. Within the very first 1000-msec time bin, the proportion of time spent viewing the matching face was significantly greater than the level that would be expected if viewing was equally distributed. (Bottom) Eye movement data from the first 2 sec of each trial in the final block partitioned in 250-msec time bins. Significantly disproportionate viewing of the matching face appeared within msec after the faces have been presented. Figures 2 (bottom), 3 (bottom), and 4 (bottom). As can be seen in Figure 2B, viewing of the matching face in Experiment 1 was significantly greater than chance, reaching nearly 57% of total viewing time, just msec after the three-face display was presented [t(34) = 5.87, p <.001, d = 0.99]. This early viewing effect was specific to matching faces; it was not seen for selected faces in novel or re-pair displays. 4 Post hoc tests revealed disproportionate viewing of matching faces from match displays relative to selected faces from re-pair [t(34) = 3.94, p <.01,d =0.66]andnovel displays [t(34) = 4.26, p <.01,d = 0.72] within this same time bin. Despite the difference in response requirements, early emergence of disproportionate viewing of the matching face in Experiment 2 was virtually identical to that seen in Experiment 1. Viewing of the matching face reached nearly 55% of total viewing time, again significantly above chance [t(35) = 5.88, p <.001, d = 0.98], within msec of the three-face display presentation. The onset of the preferential viewing effect in Experiments 1 and 2 was very rapid; it begins to emerge as early as comparable effects have been reported in ERP waveforms (i.e., the P600). In Experiment 3, the absence of a 3-sec scene preview delayed preferential viewing of the matching face by approximately 1 sec, consistent with the results of the 1000 msec time-course analysis described above. Greater than chance viewing of the matching face was not evident until msec after the onset of the threeface display [t(35) = 3.25, p <.05, d = 0.54], and the effect was relatively modest, reaching just 46% of total viewing time. Together, the results of the time-course analyses suggest that relational memory effects develop rapidly and occur spontaneously when conditions encourage pattern completion. In the absence of a scene preview, and thus, of expectancies about the to-be-presented face, this relational memory effect is of reduced magnitude and is slower to emerge. Response-locked Measures As can be seen in Figure 5, disproportionate viewing of the matching face in match displays occurred well before responding in Experiment 1; 5 viewing was already significantly disproportionate, reaching fully 55% of total 1696 Journal of Cognitive Neuroscience Volume 19, Number 10

8 Figure 4. Proportion of total viewing time allocated to matching faces in Experiment 3, when the three-face display was presented at the same time as the scene. (Top) The mean proportion of viewing time to matching faces is shown for the final block, in 1000-msec time bins, with 95% confidence intervals plotted around the mean. The dashed line indicates equally distributed viewing. Greater than chance viewing of the matching face was delayed by approximately 1 sec in the absence of the scene preview. (Bottom) Eye movement data from the first 2 sec of each trial in the final block partitioned in 250-msec time bins. Again, viewing of the matching face was delayed by approximately 1 sec in the absence of the scene preview. viewing time, by the period sec prior to response [t(34) = 6.19, p <.001, d = 1.05]. By the next time bin ( sec prior to response), nearly 80% of total viewing time was directed to the matching face. In this timeframe prior to responding, viewing of the matching face was significantly greater than viewing of selected faces from re-pair [t(34) = 7.46, p <.001, d = 1.26] or novel displays [t(34) = 10.18, p <.001, d = 1.72], although these display types also required a response. 6 Thus, long before any response was made, the eyes are Figure 5. Mean proportion of viewing time directed to matching and selected faces in the final block was aligned in time with respect to when a response was made on a trial-by-trial basis. Standard error bars are plotted around the means. Significantly disproportionate viewing of the correctly identified matching face was evident msec before a response was made. Hannula et al. 1697

9 drawn disproportionately to the matching face, revealing the retrieval and use of memory for the face scene pairing. EXPERIMENT 4 The results from the earlier experiments provided an index of memory for face scene relations, indicated the time course by which retrieval and use of such relational memory must occur, and suggested that relational memory retrieval and use occurs spontaneously. This experiment examined whether these effects depend critically on the hippocampal system. Methods Participants Participants were six patients (4 men) with amnesia, and six neurologically intact comparison participants each matched to one of the patients individually with respect to sex, age, education, and IQ. Patients were drawn from the Patient Registry of the Division of Cognitive Neuroscience at the University of Iowa, and comparison participants were recruited from the Champaign Urbana community. Experimental procedures were approved by the ethics committees at the Universities of Iowa and Illinois, and informed consent was obtained from each participant. For five of the patients, amnesia was secondary to an anoxic/hypoxic episode, and MTL damage was thought to be limited to the hippocampus. In three patients, this was due to cardiac or cardiopulmonary arrest; in the other two, the cause was an episode of seizures (leading in one case to status epilepticus). Structural MRI examinations completed on four of the patients showed that hippocampal volumes were significantly decreased, with the studentized residual differences in hippocampal volume relative to a matched comparison group down by at least 2.6 and as much as 4.2 z-scores (see Table 1; see also Allen, Tranel, Bruss, & Damasio, 2006; Hannula, Tranel, & Cohen, 2006). The impairment of the hippocampus in all four cases is disproportionate to other MTL regions (namely, the amygdala) and the remainder of the temporal lobe (gray and white matter), although the possibility of perirhinal cortex damage cannot be ruled out in patient Patient 2563 wears a pacemaker and cannot undergo MRI examination; the anatomical analysis for this patient was based on a computed tomography scan, and the only visible damage was in the hippocampal region. For the remaining patient (2308), amnesia was secondary to herpes simplex encephalitis, with extensive MTL damage. This individual has a bilateral lesion, but it is asymmetric with much more left hemisphere damage. In the left hemisphere, there is damage to the entire medial-temporal region, including the amygdala, hippocampus proper, entorhinal cortex, and perirhinal cortex. The temporal pole is damaged, and the damage extends posteriorly to include the anterior one-fifth to one-third of the superior, middle, inferior, and fourth temporal gyri. On the right, the damage is restricted to the anterior medial-temporal region, including the amygdala and the anterior part of the hippocampus. The pole and all lateral parts of the temporal lobe are spared. All patients had memory impairments that were sufficiently severe to interfere with the activities of daily life, including preventing them from being employed since the onset of their amnesia; neuropsychological testing confirmed selective memory impairment disproportionate to deficits in general cognitive or intellectual functioning (see Table 1). Performance on the Wechsler Memory Scale-III was at least 25 points lower than performance on the Wechsler Adult Intelligence Scale- III (mean Full Scale IQ minus General Memory Index difference = 32.17), and all patients were also severely Table 1. Patient Characteristics and Neuropsychological Test Scores Hippocampus WAIS-III WMS-III Subject Handedness Etiology z-score VIQ PIQ FSIQ GMI CFT 1606 R Anoxia R Anoxia R Anoxia R Anoxia L Anoxia N/A L Encephalitis N/A Hippocampal z-scores represent the combined (left and right hemisphere) studentized residuals of hippocampal volume relative to a group of 43 comparison subjects (see Allen et al., 2006). WAIS-III = Wechsler Adult Intelligence Scale-III (VIQ: Verbal IQ; PIQ: Performance IQ; FSIQ: Full Scale IQ); WMS-III = Wechsler Memory Scale-III (GMI: General Memory Index). These tests yield mean scores in the normal population of 100 with a standard deviation of 15. CFT = Complex Figure Test (delayed recall raw score [total possible correct = 36]) Journal of Cognitive Neuroscience Volume 19, Number 10

10 impaired in delayed recall on standardized tests like the Complex Figure Task. Apparatus As in Experiment 3, an ASL model 504 remote eye tracker was used to monitor eye position. Stimulus presentation and data collection were as described for the previous experiments. Procedure Participants completed four sessions, each consisting of three study blocks and an associated test block. At the start of each session, participants were told that face scene pairings would be presented and that they should attempt to commit each pair to memory in anticipation of a later recognition memory test. The eye tracker was then calibrated for the viewer using a 3 3 spatial array, a process repeated prior to the initiation of each block. Each trial began contingent upon the subject fixating a centrally located cross-hair. Study trials were as in Experiments 1 3: A scene was presented for 3 sec, a single face was then superimposed on the center of the scene, and the face scene pair remained on the screen for 5 sec. Test block trials were as in Experiments 1 and 2, each consisting of a 3-sec scene preview, followed by presentation of the three-face display for 10 sec. For the test block, participants were told that three previously viewed faces would be superimposed on a studied scene and that they were to select the face that had been paired with that scene earlier. As in Experiment 1, it was emphasized that participants should respond even when they felt that none of the faces matched the scene, that is, when re-pair displays were presented. Patients and comparison participants made their responses verbally (saying left, right, or bottom) and the experimenter recorded these responses via button press. The instructions were repeated between experimental blocks and sessions, and participants took breaks between sessions as necessary; each participant was debriefed after the test block of the final session. Stimuli and Design There were 72 male and 72 female full-color face images from our faces database (Althoff & Cohen, 1999), and 144 fullcolor images of real-world scenes (Brand X photography). Each face was superimposed on one of the scenes to create 144 face scene pairs. These pairs were presented in one of four experimental sessions. Thirty-six pairs were presented per session, once in each of three study blocks, with presentation order independently randomized across blocks. A fourth, test block was run at the end of each session. Each test block consisted of 12 displays in which three faces were superimposed on the already-viewed set of scenes. Two different display types, described above (see Figure 1), were used in this block: (6) match and (6) re-pair displays. Faces and scenes were randomly assigned to lists, and counterbalancing was conducted such that each list of faces was seen equally often with each list of scenes; lists rotated across experimental sessions and conditions. In addition, matching faces in match displays appeared equally often in each spatial position (i.e., left, right, and bottom) across trials. Eye Movement Measures Evidence for the retrieval and use of memory for relations was taken from three classes of eye movement measures described earlier: (1) between-display, (2) within-display, and (3) time-course measures. Although behavioral responses were collected, response-locked measures are not reported because the participants provided verbal, rather than keypress, responses. Results and Discussion Results are reported for the test block only, for viewing directed to the faces during the 10 sec in which the faces and scene were presented together. 7 Both groups of participants complied with the behavioral response instructions by selecting a face when re-pair displays were presented; amnesic patients and comparison participants made responses on 99.3% and 98.6% of the repair trials, respectively [t(10) = 0.62, p >.05]. Relative to the comparison group, amnesic patients were severely impaired on the test of relational memory [mean performance: 36.8% and 93.1% correct for patients and comparison participants, respectively; t(10) = 8.47, p <.001]; further, patients were no better than chance at identifying the matching face from among the three alternatives in match displays [t(5) = 0.704, p >.05], thereby demonstrating the hippocampal-dependence of this task. Between-display Comparisons Consistent with the results of Experiments 1 and 2, comparison participants made fewer transitions when viewing match displays than re-pair displays [t(5) = 4.19, p <.01], reflecting memory for the studied face scene pairing. This relational memory effect was absent in amnesic patients, even on those trials when they correctly identified the matching face. There were no differences in number of fixations to match versus repair displays for either group. Within-display Comparisons Significantly disproportionate viewing of the matching face compared to the other equally familiar faces in match displays was observed in both groups of participants. The proportion of fixations and the proportion of total viewing Hannula et al. 1699

11 time directed to the correctly identified matching face were both significantly greater than the 33% viewing level for the comparison participants [t(5) = 7.42, p =.001 and t(5) = 9.19, p <.001, respectively] and for the amnesic patients [t(5) = 4.59, p <.01andt(5) = 6.21, p <.01]. In order to determine whether the preferential viewing of the matching face on correct trials shown by amnesic patients reflected memory for previously established face scene relationships, or instead was tied to making a behavioral response, eye movement behavior was also examined for incorrect trials. Amnesic patients showed above-chance viewing of faces that were incorrectly selected, that is, had not been studied with the background context [proportion of fixations: t(5) = 9.03, p <.001; proportion of time: t(5) = 8.04, p <.001]. This result suggests that their preferential viewing effect was associated with making a behavioral response, and was not a relational memory effect. Of course, it is possible that preferential viewing of the matching face in the comparison group was also a response-related effect, but there were too few error trials to examine that possibility directly. Instead, as in Experiment 1, we explored the response issue by comparing patterns of viewing elicited by correctly identified matching faces with those elicited by selected faces from re-pair displays in the time-course analyses, described next. A true relational memory effect would manifest itself in eye movements that distinguish matching faces from selected faces very early in viewing, as was observed in Experiment 1. Time-course Measures Based on results from Experiments 1 and 2 that preferential viewing of the matching face is evident just msec after face display onset, analyses here were limited to the first 2 sec of each test trial, broken down into 250-msec time bins (see Figure 6). Differences in patterns of eye movements were evident across groups. A2 2 8 repeated-measures ANOVA with the factors group (amnesic patients; comparison group), face trial type (correctly identified match; selected re-pair), and time bin (0 250, msec, etc.) was calculated. There were main effects of group [F(1, 10) = 6.62, p <.05] and face trial type [F(1, 10) = 12.73, p <.01], as well as significant Group Face type [F(1, 10) = 16.11, p <.01] and Group Time bin [F(7, 70) = 2.97, p =.05, > =.40] interactions. As predicted, by msec, viewing of the correctly identified matching face was significantly greater among comparison participants than amnesic patients [t(10) = 2.28, p <.05]. The comparison participants here showed abovechance viewing of the correctly identified matching face just msec after face display onset [t(5) = 4.29, p <.01], much like the college-age participants from Experiments 1 and 2, despite having just three (rather than five) study exposures to the face scene pairs. Viewing of the matching faces reached nearly 60% of total viewing time during this time bin. That this was an effect of memory, rather than response related, was confirmed by planned comparisons, which revealed disproportionate viewing of matching faces in match displays relative to selected faces from re-pair displays [t(5) = 2.49, p =.05] by msec after face display onset. Amnesic patients did not exhibit above-chance viewing of the correctly identified matching face from match displays at any time point within the first 2 sec, and viewing of the correctly identified matching face also failed to exceed viewing of the selected face from re-pair displays. GENERAL DISCUSSION In four experiments, viewers eye movements were drawn disproportionately to faces at test that matched Figure 6. Mean proportion of viewing time directed to correctly identified matching faces and selected faces from match and re-pair displays, respectively, for the amnesic patients and the matched comparison group. Comparison participants showed preferential viewing of the matching face just msec after the threeface display was presented; amnesic patients failed to show this relational memory effect. Standard error bars are plotted around the means Journal of Cognitive Neuroscience Volume 19, Number 10

12 the scenes with which they had co-occurred during study. In Experiments 1 and 2, disproportionate viewing emerged significantly just msec after presentation of the test face displays, and viewing favored that face regardless of response requirements. When responses were required, the eye movement effect occurred far (>1 sec) in advance of behavioral responding. The same eye-movement-based memory effect was evident among (the older) comparison participants in Experiment 4, despite a reduction in the number of study exposures to each face scene pair. Across Experiments 1, 2, and 4, college-age subjects and comparison participants directed approximately 57%, 55%, and 60% of total viewing time to the matching face within msec of presentation of the three-face display, greatly above chance levels. In Experiment 3, without scene preview, disproportionate viewing of the matching face was also seen, but was delayed by a full second and was of reduced magnitude (46% of total viewing time). The fact that the disproportionate viewing was for the matching face might be considered surprising in light of novelty preferences consistently reported in the literature (e.g., Manns et al., 2000; Mishkin & Delacour, 1975; Fagan, 1970; Fantz, 1964). In the visual paired comparison task, for example, viewers spend more time looking at a novel picture than at a previously viewed picture presented side-by-side (e.g., Manns et al., 2000; Fagan, 1970; Fantz, 1964). However, the current experiments differ from the previous literature in that, here, item novelty was controlled; the three faces in each test display had been studied equally often, preventing differential viewing patterns from emerging based on differences in item memory. Instead, the controlling variable here was memory for relations between items. Taken together, the findings suggest that, for neurologically intact participants, memory for specific face scene pairings shaped viewing of the test face displays, permitting differential viewing of the face matching that scene. In the experiments with scene preview (Experiments 1, 2, and 4), under conditions in which a preview of the scene can promote expectancies about the to-bepresented face, the differential viewing effect emerged very rapidly. Additionally, the results from Experiment 4, in which amnesic patients showed impaired behavioral performance and no relational memory effect in their viewing of the test face displays, provide strong support for the view that memory for relations among items depends critically on the hippocampal system (e.g., Hannula, Tranel, & Cohen, 2006; Moses & Ryan, 2006; Eichenbaum & Cohen, 2001; Eichenbaum, Otto, & Cohen, 1994; Cohen & Eichenbaum, 1993). Hippocampus and Relational Memory The absence of relational memory effects in eye movement behavior in amnesia has been reported in our previous work (Ryan et al., 2000), when memory for spatial relations was tested. Critically, correct identification or differential viewing of the matching face in the experiments reported here could not be guided by memory for spatial position because the matching face appeared equally often, and unpredictably, in all three spatial locations across test trials, never in the central position in which all faces are initially studied. Rather, performance here depended upon memory for arbitrary co-occurrence of faces and scenes during the study blocks. Accordingly, the current findings, along with those reported by Ryan et al. (2000; see also Ryan & Cohen, 2004), document the critical role of hippocampus in supporting relational memory, generalizable across different kinds of relations (spatial and nonspatial), even when memory is measured indirectly. The dependence of relational memory on the hippocampus is also supported by findings from a patient with limited hippocampal damage (Mayes, Holdstock, Isaac, et al., 2004), who showed impairment on several tests of relational (or associative) memory, but whose memory for single items remains largely intact (for comparable findings also see Hannula, Tranel, & Cohen, 2006; Olson, Page, Moore, Chatterjee, & Verfaellie, 2006; Giovanello, Verfaellie, & Keane, 2003). However, not all investigators have found such clear specialization of hippocampus for relational memory (e.g., Stark & Squire, 2003; Stark, Bayley, & Squire, 2002), or generalization of the hippocampal role to nonspatial relational learning (Henke et al., 1999). The cause of such discrepancies is not clear, although it is possible that differences in the size and extent of MTL lesions, or the ability to create blended or unified stimulus representations, which would obviate the need for relational memory processing (see Moses & Ryan, 2006), might contribute. At the same time, however, a burgeoning body of fmri results also supports the claim of a critical and selective role of the hippocampus in relational memory (for reviews, see Davachi, 2006; Cohen et al., 1999). Time Course of Relational Memory The current results also index the time by which retrieval and use of (face scene) relational memory must occur. The emergence of disproportionate viewing in neurologically intact participants in Experiments 1, 2, and 4 was very rapid, considering that subjects could not predict where in a three-face display the matching face would appear, and that, in a typical trial, the msec period is only enough time to permit two or three fixations. Accordingly, what we have is a very conservative estimate of the time course of relational memory. There must be, first, successful retrieval of information about the face scene pairing (presumably during the period of scene preview), followed by identification of the match, and finally, the influence of this information on the allocation of viewing among the test faces, an influence that Hannula et al. 1701