DISENTANGLING FACTORS CONTRIBUTING TO HUMAN MEMORY FOR SPATIAL AND NON-SPATIAL MATERIALS. Dorian Pustina

Transcription

1 DISENTANGLING FACTORS CONTRIBUTING TO HUMAN MEMORY FOR SPATIAL AND NON-SPATIAL MATERIALS by Dorian Pustina A thesis submitted in partial fulfilment of the requirements for the degree of Philosophiae Doctoris (PhD) in Neuroscience from the International Graduate School of Neuroscience Ruhr University Bochum September 30 th 2010 This research was conducted at the Institute of Cognitive Neuroscience, within the Faculty of Psychology of the Ruhr University under the supervision of Prof. Dr. Irene Daum Printed with the permission of the International Graduate School of Neuroscience, Ruhr University Bochum 1

2 Statement I certify herewith that the dissertation at hand was completed and written independently and without outside assistance. The "Guidelines for Good Scientific Practice" according to 9, Sec. 3 were adhered to. This work has never been submitted in this or a similar form at this or any other domestic or foreign institution of higher learning as a dissertation. Dorian Pustina Bochum,

3 PhD Commission Chair: 1 st Internal Examiner: PD. Boris Suchan 2 nd Internal Examiner: Prof. Dr. Klaus-Peter Hoffmann External Examiner: Prof. Dr. Tim Shallice Non-Specialist: Prof. Dr. Martin Brüne Date of Final Examination: PhD Grade Assigned: Magna Cum Laude 3

4 Table of Contents I. INTRODUCTION The dual-process framework Separating processes experimentally Anatomical evidence Neuroimaging studies Event-related potentials (ERPs) The role of MTL in perceptual processing Aims of the current project II. PROCEDURE SETUP Step 1: morphing Step 2: stimuli calibration Step 3: presentation time calibration III. EXPERIMENT 1: BEHAVIOURAL Methods Results Discussion IV. EXPERIMENT 2: FMRI Participants Experimental task Imaging parameters Data analysis Encoding Retrieval Results Behavioural results Neural correlates: Encoding Neural correlates: Retrieval Whole brain analysis Discussion Activity at Encoding Activity at Retrieval MTL activity related to reaction times General considerations on MTL activity

5 V. EXPERIMENT 3: ERPs Participants Experimental Task EEG recording and analysis Results Behavioural results Analysis of Event-Related Potentials (ERPs) Discussion VI. CONCLUSIONS AND REMARKS Perceptual-processing areas and memory failures Questions for future studies VII. REFERENCES VIII. APPENDICES IX. CURRICULUM VITAE

6 List of Figures Figure I-1: Figure II-1: Figure II-2: Figure II-3: Figure III-1: Figure IV-1: Figure IV-2: Figure IV-3: Figure IV-4: Figure IV-5: Figure IV-6: Figure IV-7: Figure V-1: Figure V-2: Figure V-3: Figure V-4: Figure V-5: Figure V-6:

7 Table of Acronyms Acronym Definition ± Alternative sign denoting standard deviation Ag Chemical element: silver AgCl Chemical component: silver chloride ANOVA Analysis of Variance BOLD Blood Oxygenation Level Dependent CRs Correct Rejections EEG Electroencephalography ERPs Event-Related Potentials FAs False Alarms fmri Functional Magnetic Resonance Imaging ISI Interstimulus Interval MNI Montreal Neurological Institute k Extent threshold in fmri analysis LPC Linear Polynomial Contrast MRI Magnetic Resonance Imaging MTL Medial Temporal Lobe PRc Perirhinal cortex RTs Reaction Times SOA Stimulus-Onset Asynchrony SPM Statistical Parametric Mapping software STD Standard Deviation η² Partial-Eta Squared, statistical measure of effect size 7

8 Abstract Memory processes are typically studied with subjective rating procedures. This project uses a morphing procedure to objectively manipulate the similarity of target stimuli. The objective manipulation of the stimulus is analyzed both separately and in combination with the subjective response (confidence rating) to evaluate the contribution of each factor on the neural correlates investigated through fmri and EEG. Two types of stimuli were investigated, spatial and non-spatial (scenes and faces). Calibrated manipulations were created for both materials to produce similar shifts in memory performance. Results show that medial temporal lobe (MTL) activity can be best explained by a combination of subjective and objective factors. Memory success is associated with activity modulation in the hippocampus both for faces and for scenes. Memory failures correlate with lower hippocampal activity for scenes, but not for faces. There is a considerable impact of reaction times (RTs) on memory-related areas. RTs are distributed as Λ-shape on the confidence scale, suggesting that MTL activity is heterogeneous, showing both linear and non-linear trends, depending on the scale used for analysis. When investigating event-related potentials (ERPs), we could not find the late parietal ERP component described in the literature. The 'old/new effect' emerged at N400 centrally, and continued on right frontal areas persisting until the end of the analyzed segment (1500 ms), both for faces and for scenes. ERP results are concordant about a memory strength signal over the right frontal areas at ms. These findings explain some inconsistent findings in the field and add new interpretations for various neurocognitive patterns observed in the literature. 8

9 I. INTRODUCTION The study of human memory has been of major interest since the discovery of a localized area in the medial temporal lobe (MTL) whose lesion causes a dramatic incapacity to form new memories. Patient H.M. developed a specific deficit in memory-related tasks after bilateral removal of MTL portions to treat drug resistant epilepsy (Scoville and Milner 1957). Later studies with magnetic resonance imaging (MRI; Corkin, Amaral et al. 1997) as well as the post-mortem anatomical sectioning showed that the lesion affected particularly the anterior portion of MTL, confirming the initial claim of the investigators that anterior hippocampus and hippocampal gyrus, either separately or together, are critically concerned in the retention of current experience. Other patients have been recently described with an impairment more severe than H.M. when the damage includes the structures damaged in H.M. but also extends far enough posteriorly to involve the parahippocampal cortex (patients E.P. and G.P.; Kirwan, Bayley et al. 2008). These findings, along with models proposed in the literature in the past 50 years, advance specific hypotheses on the interrelations between the different medial temporal lobe structures. One theoretical framework has particularly stimulated the recent scientific debate and has caused the development of special procedures for the separation of two putative cognitive processes: recollection and familiarity. 1. The dual-process framework The development of dual-process models of recognition memory began in the 1970s. Several authors contributed on this field commonly assuming that recognition judgements can be based on (1) recollection of details about previous events or on (2) assessment of stimulus familiarity (Mandler 1981; Gardiner 1988; Jacoby 1991). In the conditional search model of Atkinson (1974), familiarity is considered perceptual-based information, while recollection is considered meaning-based, semantic, access that occurs when familiarity produces ambiguous responses. Extending the hypothesis of Atkinson, Jacoby and colleagues (1992) associate 9

10 familiarity to processing fluency. According to these authors, an item is familiar when it is processed more fluently, e.g. with more ease. Subjects might attribute processing fluency to past experience with those items and therefore judge them to be familiar (Jacoby, Lindsay et al. 1992). In contrast with Atkinson s affirmations, Tulving (1985) associated the experience of knowing with semantic memory, whereas remembering is supposed to be related to episodic memory. Despite theoretical discrepancies, the properties of these memory processes have been intensely studied. The following list provides the main distinctions provided by the current literature: 1. In speeded recognition experiments, familiarity appears to be faster and is available earlier than recollection (Yonelinas and Jacoby 1994). 2. Divided attention tasks affect recollection estimates far more than familiarity, indicating an active process during recollection encoding different from familiarity (Gardiner and Parkin 1990; Yonelinas 2001). 3. The data suggest that the hippocampus is critical for recollection but not familiarity, whereas the perirhinal cortex contributes to and is necessary for familiarity (Eichenbaum, Yonelinas et al. 2007; see anatomical and neuroimaging evidence below). 4. Both recollection and familiarity are improved by tasks that require deep semantic manipulation, but recollection benefits to a greater extent than does familiarity (Donaldson, Mackenzie et al. 1996). 5. Forgetting rates: across intermediate-term delays (i.e., 10 seconds or 8 to 32 intervening items) familiarity decreases rapidly while recollection is relatively unaffected. In contrast, across long-term delays (i.e., minutes to months) both recollection and familiarity exhibit pronounced forgetting effects (Yonelinas and Levy 2002). 6. False alarms are typically higher for familiarity responses than recollection. 10

11 7. Recollection can form novel associations between items and/or context, but familiarity cannot (Perfect, Mayes et al. 1996). However, under some conditions association processes have been shown also for familiarity. Familiar associations can occur between similar materials (voice-voice, face-face) or when different aspects of an event are perceived as a whole item. 1.1 Separating processes experimentally Two classes of experimental paradigms, objective and subjective, have been used to separate recollection and familiarity. Objective recollection paradigms (also referred to as relationalrecognition tests ; Eichenbaum, Yonelinas et al. 2007) include direct tests of memory for associations or contextual features. During these tests, the subject is invited to recall an associating feature of the stimulus, i.e. source memory (colour, associated voice, accompanying picture, etc.). Tests of source memory are commonly used to obtain objective measures of recollection (for a review, see Mitchell and Johnson 2009). Nevertheless, the assumption that source retrieval is equivalent to recollection may not be true. Occasionally, the cue and the target may create a unitized representation, which memory relies upon for familiarity rather than recollection judgements. Behavioural, neuroimaging, and patient studies suggest that source memory can, in fact, elicit neural correlates typical of familiarity, depending on the degree of unitization (Quamme, Yonelinas et al. 2007; Diana, Yonelinas et al. 2008; Diana, Yonelinas et al. 2009). Subjective recollection paradigms rely on self assessment of memory representations (also called meta-memory judgements). The most widely used paradigm is the Remember Know procedure (Tulving 1985). In this task subjects are required to rate their memories on the basis of introspection. They have to judge whether they recognize items on the basis of remembering (i.e. recollection of contextual information about the study event) or knowing (i.e. the item is familiar in the absence of recollection of contextual information). 11

12 The major criticism is that remembering and knowing do not reflect the direct contributions of recollection and familiarity, but rather subjective states of awareness (Gardiner, Ramponi et al. 1998). In support to these criticisms, recent studies show that slight changes in the Remember-Know instructions may produce different results, confirming a labile correlation between memory processes and meta-memory judgments (Geraci, McCabe et al. 2009; McCabe and Geraci 2009). Furthermore, the massive analysis of previously published experiments using Remember-Know judgements suggests that remember and know responses rely on a single underlying process (Dunn 2004; Dunn 2008) 1. Another approach for estimating measures of recollection and familiarity involves asking the participant to express the subjective confidence in a scale from new to old. Subjects answer with various degrees of confidence on the new side or on the old side. High confidence responses are situated at the far extremes of the scale, while less confident responses are located towards the middle area. A Receiver Operating Characteristic curve (ROC) is then obtained by plotting hits and false alarms as a function of each response confidence. The more options are available in the graded response scale, the more accurate ROC estimates are obtained. In order to define a ROC curve, at least three points are mathematically required. Consequently, the minimum number of options in the graded response scale is four. After the estimation of the ROC, its shape is analyzed to provide information about the contributions of recollection and familiarity to the recognition performance (Yonelinas 1994). Differently from the Remember-Know procedure, such confidence rating procedure does not attribute each stimulus as recognized through recollection or familiarity. Instead, a value is obtained for the overall contribution of familiarity and recollection during the entire recognition performance. Typically, confidence scales are used when two or more sets of stimuli are compared respectively for recollection and familiarity shifts. Given the demands of this research project, confidence rating has been used in all memory tasks. 1 State-trace analysis. For an overview see Newell, B. R. and J. C. Dunn (2008). "Dimensions in data: testing psychological models using state-trace analysis." Trends Cogn Sci 12(8):

13 2. Anatomical evidence Considerable effort has been devoted to understand the precise role of the MTL in memory function, as well as the involvement of different MTL substructures in memory formation and retrieval. These findings suggest an anatomically guided hypothesis of the functional organization of MTL. Anatomical studies leading to the following considerations are based on the research of major MTL subregions and connections in humans, monkeys and rodents. At the broadest level, MTL can be subdivided into the perirhinal cortex (anteriorly), parahippocampal cortex (posteriorly), entorhinal cortex, and hippocampus (comprising dentate gyrus, Ammons s horn and subiculum). The perirhinal, entorhinal and parahippocampal cortex are often grouped together under the term of parahippocampal region, while the hippocampus is referred to as hippocampus proper. Afferent connections to the perirhinal cortex originate at unimodal areas, presumably processing the quality of objects (i.e. the what pathway), while the parahippocampal cortex receives afferences mainly from polymodal areas which process spatial information (the where pathway; Suzuki and Amaral 1994; Lavenex and Amaral 2000). The two streams continue partially in separate paths as the perirhinal cortex projects mainly to the lateral entorhinal area and the parahippocampal cortex projects mainly to the medial entorhinal area (Suzuki and Amaral 1994). Subsequently, entorhinal efferences converge toward the hippocampus, where the information is presumably integrated and sent back to entorhinal, perirhinal, parahippocampal cortices, and then to neocortex again 2 (Suzuki and Amaral 1994; Burwell 2000; Duvernoy 2005). This anatomical evidence suggests specific hypotheses about how information flows in the medial temporal lobe during memory encoding or retrieval. Being the final destination of the what and where pathways, the hippocampus may play an associative role, which corresponds to what is defined as recollection. Information 2 In humans, there is no agreement to which areas correspond to perirhinal and lateral/medial entorhinal of rodents and monkeys. A recent review Eichenbaum, H., A. R. Yonelinas, et al. (2007). "The Medial Temporal Lobe and Recognition Memory." Annu Rev Neurosci. Considers the activation in the anterior parahippocampal gyrus to reflect the perirhinal and lateral entorhinal area, while activation in posterior parahippocampal to reflect parahippocampal cortex with or without medial entorhinal area activation. See Figure I-1. 13

14 from different cortical areas is relayed to this common structure, which presumably integrates different items to a coherent episodic representation of what, where and when. Figure I-1: Lateral surface of the brain of a rat (left), ventral surface of the brain of a rhesus monkey (middle), and a human (right) depicting the location of selected structures. (Murray, Bussey et al. 2007) It should be noted that the localization of the perirhinal and entorhinal cortices in human subjects does not follow precise landmarks and is prone to inter-subject variability. To resolve this issue, a recent tool has been developed to create probability maps for the entire population based on individual post-mortem cytoarchitectonic maps (Eickhoff, Heim et al. 2006). Many authors have relied, instead, on the use of traditional anatomical landmarks described in two important studies (Insausti, Juottonen et al. 1998; and Pruessner, Kohler et al. 2002). For a better cross-validation of anatomical labelling all above resources has been used in the analysis of the functional magnetic resonance imaging (fmri) data. 3. Neuroimaging studies Structures within MTL are in close proximity, near the resolution of current fmri methodology, and they are highly interconnected. Therefore neural processing originating in one region can activate neighbouring voxels, resulting in uncertain interpretation of the fmri 14

15 data. Despite the technical issues, there are relatively consistent findings, which show specific patterns of activation across studies. A review of 25 fmri studies (Eichenbaum, Yonelinas et al. 2007), suggests that hippocampal activation during both encoding and retrieval is consistently higher for recollected items and is not related to changes in familiarity strength (but see also Squire, Stark et al. 2004). This is true particularly when the task involves associations of items and requires context recollection. Conversely, an anterior parahippocampal signal (i.e. perirhinal cortex) is generally correlated with familiarity and rarely with item recollection. Familiarity, however, elicits a different pattern of activation in encoding and retrieval. Activation during encoding is higher for items that are subsequently rated as highly familiar compared to later forgotten items, whereas during retrieval activity decreases progressively for more familiar items. Such pattern is consistent with results from physiological studies on monkeys, which identified novelty detection neurons in the perirhinal cortex. Indeed, nearly 25% of neurons in this area fire exclusively upon presentation of new as opposed to old presented stimuli (Miller, Li et al. 1993; Brown and Xiang 1998). Beside the concordant findings of the above review, there is growing evidence of inconsistent or incompatible activity patterns in the MTL. For example, activity in the hippocampus has been found to increase (Daselaar, Fleck et al. 2006) or decrease (Johnson, Muftuler et al. 2008) depending on the task. Activity in perirhinal cortex has been found to be specific for faces, for objects, for perceptual fluency, related to conscious recognition, or generally elicited by various types of material, including scenes. In a recent experiment Johnson et al. (2008) showed that small areas in the hippocampus can contain both voxels expressing linear graded activity and voxels with non-linear categorical activity. The nature of the stimuli employed in the memory tasks constitute another source of contamination on the current debate of single vs. dual process models (for recent reviews, see Diana, Yonelinas et al. 2007; Eichenbaum, Yonelinas et al. 2007; Spaniol, Davidson et al. 15

16 2009). For example, a recent meta-analysis of 36 fmri studies by Spaniol et al. (2009) produced left-lateralized results on many contrasts. As the authors point out, this result may reflect a semantic component since words are used as stimuli in many memory experiments. MTL sub-specialization for faces and scenes has also been suggested, such that hippocampus and perirhinal cortex subserve scene and face processing, respectively (Lee, Buckley et al. 2006; Taylor, Henson et al. 2007; Lee, Scahill et al. 2008). This pattern was not confirmed in other studies, where similar activity for faces and scenes was observed in perirhinal cortex (Buffalo, Bellgowan et al. 2006; Preston, Bornstein et al. 2009). The distinction between spatial and non-spatial material has been assigned a differential contribution of the right and the left hemispheres, too. For example, Bellgowan et al. (see also Buffalo, Bellgowan et al. 2006; 2009) found higher laterality indices in right MTL for spatial encoding, and higher laterality indices in left MTL for object encoding. More recently, a study by Litman et al. (2009) partially confirmed this tendency. These authors showed a gradual pattern for spatialnonspatial stimuli in the anterior-posterior axis of the parahippocampal gyrus, but faces elicit above baseline activity particularly in the left hemisphere (Litman, Awipi et al. 2009). In the same study words and pseudowords elicited no significant activity in the MTL substructures (Litman, Awipi et al. 2009). These findings suggest that the type of material is an important factor affecting MTL activity in mnemonic tasks. Another possibility frequently mentioned in fmri studies of human recognition memory is that the blood oxygenation level dependent (BOLD) signal could follow a non-linear relationship with the manipulation of interest, such that graded experimental manipulations could lead to abrupt and unpredictable vascular responses (Soltysik, Peck et al. 2004; Squire, Wixted et al. 2007; Johnson, Muftuler et al. 2008; Tendolkar, Arnold et al. 2008; Qin, Rijpkema et al. 2009). While this hypothesis cannot be ruled out without a profound understanding of the neural coupling between brain activity and vascular response, some authors have relied upon it to explain inconsistent findings. 16

17 4. Event-related potentials (ERPs) The first reports of a difference between correctly classified old items and correctly classified new items were reported 30 years ago (Sanquist, Rohrbaugh et al. 1980; Warren 1980). In general, ERPs are more positive 300 ms after the presentation of the correctly recognized studied items compared to correctly rejected non-studied items. This old/new effect has been found in spatially and temporally distinct ERP modulations for which has been assigned to specific memory processes. Two ERP components have been debated in the literature for their attribution to familiarity and recollection: 1. An early component (~ ms) maximal over the anterior portions of the scalp called early frontal positivity. This component is frequently assigned to familiarity, as opposed to recollection. 2. A late component (~ ms) maximal over parietal portions of the scalp presumably indexing recollection as opposed to familiarity. This component is often found in left portions of the scalp. Although various authors have assigned these components to recollection and familiarity (Wilding and Rugg 1996; Allan, Wilding et al. 1998; Donaldson and Rugg 1998; Rugg, Mark et al. 1998; Curran 2000; Curran and Cleary 2003; Rugg and Yonelinas 2003; Curran 2004), the majority of such ERP evidence comes from paradigms using lexical stimuli, i.e. words. It has been argued that words have a baseline of familiarity because they have been encountered outside the experimental context. The presence of a baseline has brought the concept of relative familiarity (as opposed to absolute familiarity) to prevent a clear interpretation of memory correlates on the basis of reprocessed as opposed to newly created mnemonic traces. Moreover, it is a well known fact that lexical material is processed and activates the left brain hemisphere in most right-handed subjects. Studies using pictorial stimuli have eventually found inconsistent activity with the familiarity/recollection assignment of these ERP 17

18 components (Paller, Gonsalves et al. 2000; Yovel and Paller 2004; Mackenzie and Donaldson 2007; but see Curran and Cleary 2003). Thus, the attribution of ERP components to certain cognitive processes is precarious and far from conclusive. A third ERP component has been described to start later (~ ms) and to be prominent over right frontal scalp electrodes. The functional interpretation of this component has been assigned to post-retrieval processing, evaluation of memory for source or item-specific features, or the start of strategies related to the task. Another interpretation concerning the specific interest of this project is that such component arises due to effortful memory decisions involving weak familiarity or impoverished elements for a judgement (Rugg, Allan et al. 2000; Wolk, Schacter et al. 2004). 5. The role of MTL in perceptual processing Though the role of MTL in mnemonic processes is broadly accepted, recent studies have challenged this view by noticing a specific role of MTL in perceptual processes. This new view has added another factor of interest to future studies willing to investigate the functional properties of MTL activity. In particular, these studies have shown complex perceptual deficits in MTL lesioned patients which traditionally were diagnosed as amnesic (Barense, Bussey et al. 2005; Lee, Bussey et al. 2005; for a review Murray, Bussey et al. 2007). Consequently, these authors propose a broader role of MTL at higher-level perceptual processing in concert or beside its mnemonic function. Such interpretation is helped by traditional or new considerations of memory in terms of perceptual fluency (Jacoby, Lindsay et al. 1992), repetition priming/suppression (Brozinsky, Yonelinas et al. 2005; Voss, Hauner et al. 2009), or by finding effects of repetition suppression in MTL with or without conscious experience (Henson, Shallice et al. 2002; Gonsalves, Kahn et al. 2005; Voss, Hauner et al. 2009). While MTL has been initially related only to episodic memory, the above mentioned findings suggest a broader role of MTL in a variety of tasks. In this context, the distinction between explicit and 18

19 implicit memory, or automatic and effortful retrieval, may be far less pronounced as previously thought. 6. Aims of the current project A deep understanding of human memory requires cognitive and neural descriptions of memory processes along with a conception of how memory processing drives behavioural responses and subjective experiences. The challenge to this goal is that memory processes are operative within a mix of various processes, which should be experimentally dissociated and operationalized. Current literature offers a wide spectrum of interpretations for distinct processes, though not conclusive and occasionally contrasting. Extending these findings, an objective measure of memory manipulation may be of high importance for the elucidation of the relationship between status of the stimulus (old / new) and subjective quality of memory. Stimuli manipulation has been used in early psychophysical experiments to study the relation between objective stimuli and subjective perception. These studies were among the first to gradually manipulate the stimuli and find a determinate relation with perceived experience (i.e. Weber Fechner law). A similar rationale is adopted here in order to variably manipulate the target-distractor similarity and obtain a controlled effect on human memory. The choice to extend the simple old-new manipulation to a parametric manipulation relies on the consideration that memory representation relies on other memory representations; i.e. a new face is somehow similar to a known face, a new place is similar to other visited places. The investigation of how internal representations are compared to gradually more similar external targets may provide a controlled environment for the investigation of memory processes. Previous studies have used mainly one type of stimulus to investigate recognition memory. This choice may create a serious confound in the attempt to interpret controversial results from studies with different types of stimuli. Therefore, a combination of more than one stimulus 19

20 type during the same memory task can provide important insights in the similarities and distinctions of memory processes for each material. The present project focuses on visual memory for faces and scenes, with parametric manipulation of stimulus similarity and calibrated morphing scales for each stimulus type. The questions these project attempts to answer are: Can we gradually manipulate human memory? Can we create comparable manipulations for faces and scenes? What is the relationship between subjective experience or objective identity and MTL activity? How is memory for different types of stimuli reflected in brain activity and scalp potentials? What is the relationship between MTL activity at encoding and at retrieval? Which ERP components correlate with subjective and objective memory for faces and scenes? What is the role of ERP components previously assigned to familiarity and recollection in the objective/subjective distinction? 20

21 II. PROCEDURE SETUP Memory manipulation requires a set of physically manipulated stimuli to induce a parametrically controlled change in the perception of oldness. This chapter describes the procedure with which manipulated stimuli have been created (Step 1: morphing) and the morphing scales have been calibrated for faces and scenes (Step 2: stimuli calibration). A final section describes the attempts to carefully select appropriate presentation times in order to balance the performance between faces and scenes (Step 3: presentation time calibration). 1. Step 1: morphing Two sets of stimuli have been created separately, one for scenes and one for faces. Faces consist of 204 black and white pictures previously used in other experiments (Endl, Walla et al. 1998; Kipp, Jager et al. 2006). They were coupled in 102 pairs (51 male and 51 female pairs). Each pair was newly morphed in our lab (Abrosoft Fantamorph v3 software) by placing corresponding points to the facial features (eyes, eyebrows, hair, etc.). The resulting morphed pair produced a gradual change from 0% (Face 1) to 100% (Face 2) in steps of 5% (Figure II-1) 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Figure II-1: Two faces morphed to each other. Only 10% steps are shown, however, 5% steps were used for the experiments. 21

22 Scenes consist of 102 high resolution pictures of landscapes and buildings downloaded from the internet and from personal albums. Photos of known places, famous cities and pictures of resolution less then pixels were excluded. To create the morphing steps, a virtual window was placed on the left side of the panorama and considered 0%. The window was moved to the right to create the other frames by removing 5% of the pixels on the left and adding 5% new pixels on the right. The last frame (100%) was exactly outside the starting one (for a similar procedure, see Park and Chun 2009). A Matlab script (Mathworks Inc., MA, USA) was created to inspect the moving window, create the frames, normalize the contrast between frames, and convert them into grayscale 3. The 102 resulting sets were selected among ~500 downloaded pictures to obtain manipulations which look as differently as possible between the 0% frame and the 100% frame. 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Figure II-2: Manipulation of scenes through lateral shifting. Only 10% steps are shown, however, 5% steps were used for the experiments. Another set of unmorphed face and scene stimuli was used as distractors. They were used first during the calibration of the stimuli and then during the test phase of the memory experiments. 3 Script available at 22

23 2. Step 2: stimuli calibration The two morphing techniques are hardly comparable considering the different procedures they follow. Different degrees of face morphing reflect a linear change of pixel value, while different degrees of scene manipulation reflect a spatial shift laterally. These physical changes may be related differently to human perceived changes. I.e. a 20% manipulated picture can correspond to a human perception of 60% difference for faces and 35% difference for scenes. In order to guarantee comparable manipulations for scenes and faces, the perception of change should be balanced for both materials. On this purpose, a calibration procedure was performed to fit them on the same scale. Twenty-four subjects were asked to rate the similarity between pairs of pictures. The pairs could contain morphed stimuli (e.g. steps 0% and 40%; 0% and 80%) or two unrelated stimuli (both chosen from the distractor set) 4. One of the pictures in the morphed couples was always the 0% step, while the other picture could be one of the following morphing steps: 0%, 20%, 40% 60%, or 80%. Both faces and scenes were rated by the same subjects. They were told to focus on the personal feeling of similarity instead of the small physical details changing between pictures. In each trial, two pictures were presented in sequence for 1000 ms each with an interval of 400 ms between the first and the second picture. They were preceded by a fixation cross for 1000 ms. After the second stimulus had disappeared, a scale appeared on the screen and the subject was asked to rate the similarity by moving the mouse between 0% and 100% in steps of 1%. The scale remained on the screen until the subject clicked on the desired level of percentage. Each subject was presented with an equal number of stimuli for each morphing step and saw only one step from each morphed pair. Six subjects were needed to complete all rating steps for a pair. A randomization procedure was employed to administer all steps to all 24 subjects (four full ratings for each morphed pair). 4 Distractor pairs were introduced to avoid subjects focusing on narrow differences between stimuli. 23

24 Regression was applied to group ratings with Datafit software (Oakdale Engineering, PA, USA). The best regression equation was chosen among the five best fitting curves after using a non-linear solver for multiple order polynomials. The procedure was repeated separately for faces and scenes yielding two different equations (Figure II-3). A. B. Figure II-3: Calibration curves (blue lines) for faces (A) and scenes (B) interpolated on the subjects ratings (circles). The best fitting equation is shown at the bottom. The corresponding real difference perceived by the subjects as 20%, 40%, and 60% different was calculated from the inverse solution of the resulting equation. These levels represent the values of real morphing necessary to elicit a perceived 20%, 40% or 60% difference. Exact values for perceived 20%, 40% and 60% change resulted in 35%, 55% and 75% real face morphing, and 25%, 45% and 90% real scene shifting, respectively (Table II-1). In the following paragraphs, the values 20%, 40%, and 60% are mentioned to refer to perceived 24

25 morphing, implying that the corresponding real values of matched calibration have been used during the experiments. Table II-1: Calibration values between perceived similarity and real morphing for faces and scenes. Calibrated (perceived) Face (real) Scene (real) 0% 0% 0% 20% 35% 25% 40% 55% 45% 60% 75% 90% After finishing the calibration, six pairs from the face set and six pairs from the scene set were removed because only 96 morphed pairs were needed for the following memory experiments 5. To select the six pairs for removal, face and scene manipulations were screened for variability ratings amongst subjects. From each 102 pair-set six most discordant morphs were removed which had the lowest correlation between subject ratings. 3. Step 3: presentation time calibration A pilot experiment was needed to calibrate the stimulus onset asynchrony (SOA) and check the validity of our calibration. This experiment was designed to ressemble the upcoming memory studies. The task comprised two phases: encoding and retrieval. Due to the large amount of pictures, the experiment was divided into four independent blocks, each having the same amount of stimuli and the encoding and retrieval phases. This design is the same as in the behavioural (Chapter III), fmri (Chapter IV), and ERP (Chapter V) experiments, except that three stimulus onset asynchronies (SOAs) were used during encoding to determine the best option for the remaining project. The three presentation times were 1500 ms, 1700 ms, or 2500 ms. In sum, for each block, 48 pictures (24 faces and 24 scenes; 6 from each morphing step) 5 A number divisible by 16 was needed to split each of the four morphing steps (0%, 20%, 40%, 60%) in equal number to four study-test blocks. 25

26 were presented during the encoding phase, preceded by a fixation cross for 1000 ms with an interstimulus interval (ISI) of 300 ms. Five subjects performed the experiment for each SOA. Subjects were instructed to fixate the cross and look carefully at each picture to memorize it for a recognition test. They were encouraged to use personal strategies in order to better memorize the stimuli. The retrieval phase started at least 90 seconds after the end of the encoding phase. During this pause, the subject was distracted by the experimenter entering the room and talking to him/her. The retrieval phase consisted of pictures presented one by one on the screen for an unlimited time until the subject answered. A 6 point confidence scale was located underneath the picture ( ; 1: sure new; 6: sure old). The responses were given by placing six fingers on six different buttons, three from each hand, and the scale was inverted for half of the subjects so they could answer old with the other hand than the other half of the subjects. Each picture presented in the retrieval phase, could be either a previously studied picture (0% difference in morphing steps), a different morphing step of a previously studied picture (20%, 40% or 60% difference in morphing steps) or a new distractor (100% difference in morphing steps). Subjects were unaware of the physical manipulation of the stimuli. Whenever a subject mentioned a possible manipulation to the experimenter, he denied any modification on purpose. During the sessions has been noted, however, nearly half of the subjects detected one or two scenes shifting their point of view. The number of conscious detections was nevertheless small compared to the total amount of stimuli. No subject detected a manipulation of the faces, even though a couple of subjects advanced the hypothesis that determinate faces could have single parts changed (i.e. the nose). Pictures subtended a visual angle of for faces and for scenes both during encoding and retrieval. Results showed an advantage of scenes over faces for 2500 ms presentation time, probably due to a longer study time allowing more saccades and a full representation of spatial relationships between scene elements. The shortest SOA (1500 ms presentation) produced an overall drop in 26

27 memory performance. The best compromise of acceptable performance and symmetrical performance between scenes and faces was obtained when pictures were presented for 1700 ms during the encoding phase. This SOA was used for the following memory experiments. 27

28 III. EXPERIMENT 1: BEHAVIOURAL The aim of this experiment was to investigate the validity of the morphing paradigm as a tool to parametrically manipulate human memory. A gradual change in recollection and familiarity estimates is expected to follow the covert manipulation scale. Given the use of calibrated scales, comparable performance shifts were expected for faces and scenes. Particular interest was posed on the relationship that familiarity and recollection might have with morphed stimuli. In particular, recollection is expected to be an all-or-none process occurring especially for identical targets (0% morphed), while familiarity might follow a linear trend for the gradual covert manipulation. 1. Methods The first experiment to fully test the paradigm used the same design explain before ( Step 3: presentation time calibration above). The experiment was split into four independent blocks and the SOA of 1700 ms was used during encoding. Eighteen subjects participated in the experiment in exchange for credit points (VP-stunde) or economical reimbursement. All participants gave informed consent. The experiment was run using Presentation software 6. Participants were told that the experiment was about memory and importance of performance accuracy was stressed. They were invited to use personal strategies to enhance memory performance. During the pauses between experimental blocks and between encoding-retrieval phases the experimenter entered the room and talked to the participants. If the subject felt tired between one block and the other, the pause was extended until he/she was rested and ready to start the next encoding-retrieval block. Face and scene stimuli were intermixed randomly during encoding and retrieval. Regarding the morphing procedure, although 0% stimuli was considered as the step presented during study 6 All experiments and pilots in this project were programmed in Presentation software ( 28

29 phase, the actual order of presentation between encoding and retrieval phases was random. For example, a 0% morphed stimulus could be presented at study and the corresponding 20% could be presented during retrieval, or vice versa. The morphing distance was the same in both cases, but the possibility has been taken into account that the morphed stimulus could contain unnoticeable perceptual alterations which may lead to perceptual-based categorizations of old stimuli. This eventuality would undermine the interpretation of the results, for which it has been randomized across study-test phases. 2. Results The comparison of reaction times (RTs) between experimental blocks was analyzed with a 4 1 repeated measures ANOVA. Results showed a tendency toward faster responses in successive blocks (F (3,54) = 3.25, p =.054, η² =.13). A repeated measures ANOVA on response button and stimulus type showed slower RTs for scenes compared to faces (F (1,18) = 67.7, p <.001, η² =.79), possibly resulting from more saccades needed for exploration of scenes. Reaction times were faster for more confident responses, creating an inverted U shaped quadratic function (F (1,18) = 103.8, p <.001, η² =.85). All other polynomial contrasts (linear, cubic, 4 th order, and 5 th order) were non significant (p >.15). Response 6-sure old was the fastest of all responses with the smallest variability between subjects (mean RT = 3775 ms; SD = 119). An response button stimulus type interaction was observed (F (5,90) = 4.88, p <.005, η² =.21), indicating that RT shifts between response buttons are different for faces and scenes. Nevertheless, the effect size for the interaction term is small (η² =.21) compared to the main effect of quadratic RT distribution (η² =.85) and scene slowness (η² =.79), indicating robust main effects. Estimates of recollection and familiarity derived from the ROC analysis have been calculated for faces and scenes separately, and for each morphing step, using the algorithm published 29

30 online by Yonelinas (1996) 7. The comparison between blocks showed no difference in performance (F (3,54) = 0.538, p =.658, η² =.03) and no interaction of blocks with recollection/familiarity estimates (F (3,54) = 0.598, p =.619, η² =.03), suggesting no carry over or fatigue effects. Performance was then analyzed with a repeated measures ANOVA with the factors stimulus (face/scene) process (recollection/familiarity) morphing (0%, 20%, 40%, 60%). Results showed that scenes were better remembered than faces despite the initial calibration (F (1,18) = 18.24, p <.001, η² =.50). Gradual morphing produced a downfall in performance in a linear manner (F (1,18) = 31.80, p <.001, η² =.78), but the quadratic contrast was significant too (F (1,18) = 7.72, p <.05, η² =.30). Post-hoc analysis revealed that the quadratic term was significant for recollection (F (1,18) = 5.48, p <.05, η² =.23) but not for familiarity (F (1,18) = 3.61, p =.07, η² =.17). The crucial point of the analysis was to understand if the calibration performed was appropriate and the two material types changed comparably between 0% and 60%. In this regard, no interaction was observed between morphing and stimulus type (F (3,54) = 1.37, p =.264, η² =.07), showing that the manipulation is well calibrated and produces the same performance changes for faces and scenes, which are illustrated in Figure III-1. Please note the linear drop for both materials. Coherently, a significant interaction morphing process type was found (F (3,54) = , p <.001, η² =.44) given that the downfall for familiarity is steeper than for recollection (Figure III-1 below). Moreover, faces and scenes showed an interaction with recollection / familiarity processes (F (1,18) = 13.56, p <.005, η² =.43). Detailed analysis revealed higher familiarity estimates for scenes compared to faces, but this difference was lower for recollection, which 7 Original algorithm available at [ Improved and translated into Matlab code for this project [ 30

31 caused the interaction. Finally, no three-way interaction was observed between morphing process type stimulus type (F (3,54) = 0.22, p =.707, η² =.01). Figure III-1: Familiarity and recollection estimates for faces (blue) and scenes (green) 3. Discussion Today s research on human memory is largely based on self-examination of memory feelings. Both remember/know and confidence ratings require the active participation of the individual. The only input provided by the experimenter is the alternation of old and new stimuli. This paradigm was extended to increase the control on subjects feelings of memory. The first most 31

32 important result of our experiment is that human memory can be linearly manipulated without self awareness. Our morphing paradigm can be used as a tool to parametrically manipulate memory with experimental precision. Secondly, faces and scenes obtained similar performance changes, though the manipulation methods were substantially different. Future results of faces and scenes can be compared with the confidence of having produced the same amount of memory manipulation. Thirdly, the existence of two linear scales, one objective (morphing) and one subjective (response confidence), gives us the possibility to investigate the relation between objective manipulation and subjective memory. As will be described later, subjective memory failures are related to determinate neural correlates, which has been investigated with parametric combinations of the subjective and objective scales. The morphing paradigm has, nevertheless, space for improvement. Future developments of the technique may focus on calibrating each pair individually to obtain more precise results and less stimulus-related variability. Despite our attempt to obtain balanced performances, scenes were remembered better than faces. This superiority for spatial material is consistent with previous literature. Notably, our research relies on the intra-item memory shifts and not on the direct comparison between materials. With respect to this, memory shifts were similar between materials. 32

33 IV. EXPERIMENT 2: FMRI The aim of this experiment was to investigate factors contributing to MTL activity during a memory task. Additional investigation of the whole brain activity may provide insights on the involvement of off-mtl areas in memory processes and their functional relationship with MTL areas. The major factors under investigation were: subjective response, objective status of the stimulus, and material type (face/scene). Reaction times (RTs) were included in the analysis to covary out the activity explained by temporal differences of motor responses. Given that subjective and objective factors consist in graded scales, parametric modulations are an appropriate tool for their analysis. Parametric modulations are a numerical representation of the different levels of a factor (or continuous variable), such that corresponding shifts in brain activity can be statistically evaluated through regression procedures. In contrast to normal factorial analysis, which is particularly tuned to test a single difference between categorical conditions, parametric modulations are tuned to test complex stimuli with a number of stimulus dimensions which can be modelled by a set of parametric regressors (e.g. faces can be described in terms of attractiveness, masculinity, symmetry, etc). In turn, the experimenter can look at the contribution of each stimulus dimension independently. Parametric modulators have been demonstrated quite useful to capture various aspects of multi-feature processes in fmri (Wood, Nuerk et al. 2008). Memory studies using parametric modulators have differentiated between trends of activation or were used to predict a specific pattern of activity (Maguire, Henson et al. 2001; Daselaar, Fleck et al. 2006; 2006; Montaldi, Spencer et al. 2006; Tendolkar, Arnold et al. 2007). 1. Participants Twenty-three subjects, all right-handed according to the Edinburgh Handedness Inventory (Oldfield 1971), participated in the experiment. They reported no history of psychiatric or neurologic disorders and had normal or corrected-to-normal vision. The data from four 33

34 participants were discarded because of excessive head motion or failures of the technical equipment. The remaining 19 participants (mean age 24.4 ± 2.4, 10 female) were included in the statistical analysis. All subjects were reimbursed for participation and gave informed consent. The study was approved by the local ethical committee of the medical faculty of the Ruhr University Bochum. 2. Experimental task Stimuli were projected in the fmri room on a back projecting screen and subtended a visual angle of for scenes and for faces. The experiment was run using Presentation software ( All stimuli (during both encoding and retrieval) were preceded by a fixation cross for 1000 ms, with a variable interstimulus interval of ms. Given the high number of stimuli, the experiment was split into four blocks of encodingretrieval alternations. For the study phase of each block, 48 pictures (24 scenes, 24 faces, 12 male) were randomly presented for 1700 ms each. A pause of 90 seconds followed and the retrieval phase continued with 72 pictures (48 old + 24 new) randomly presented for 5000 ms each. Four morphing steps were used during the retrieval phase: 0%, 20%, 40%, and 60%. For every face or scene used during encoding, one corresponding morph was presented during retrieval. Therefore, in contrast to traditional experimental designs, not all old stimuli were truly old, but comprised different morphing steps of the pictures presented in the study phase. Of the 48 stimuli used for recognition, only 12 were truly old (0% morphing). The remainder 36 pictures comprised 20%, 40% and 60% morphs (12 items each) and had thus varying degrees of similarity with the original 0% morph viewed during encoding. Participants were instructed to memorize the pictures during the encoding phase and give the best possible memory performance during recognition. They were not informed about the manipulation. Participants were invited to use the whole range of response buttons to express their memory evaluation. At the end of the retrieval phase, subjects were invited to relax and a pause of about 150 s was introduced before the next block started. 34

35 Memory judgements could be made on a four point confidence scale (1: sure new; 4: sure old) while the stimulus stayed on the screen (5000 ms). In case of no response during the 5000 ms, the trial was dropped from further analysis and the subject was asked to respond faster. Responses were given using four fmri compatible buttons positioned on the index and middle fingers of each hand. Nine subjects answered the old-new scale from left to right and ten subjects answered from right to left. 3. Imaging parameters Imaging data were collected using a 1.5 T Siemens Sonata MRI scanner with a spin echo sequence pulse, 30 ms echo time (TE), 2000 ms repetition time (TR) and 90 flip angle. Volumes contained 28 transversal slices acquired in sequential order (ventral to dorsal, 1 to 28) of voxels each, with 3 mm slice thickness and 0.96 mm slice gap. Additionally, a high resolution T1-weighted structural image was acquired (flip angle 30, TR 9.2 ms, echo time 4.46 ms, sagittal orientation, 128 slices, 1.5 m slice thickness). Subjects were scanned during both encoding and retrieval phases. The four experimental blocks were divided in two fmri runs with two blocks each (mean: volumes/run; standard deviation [STD] 3.55). A pause of two minutes between scanning sessions corresponded to the pause between block two and block three of the experimental task. 4. Data analysis FMRI data preprocessing and analyses were performed with Statistical Parametric Mapping software (SPM5, Wellcome Department of Imaging Neuroscience, London, UK). Images were slice-time corrected (reference slice 14), realigned, normalized to the EPI template of the Montreal Neurological Institute provided by SPM5, resliced in mm voxels, and smoothed with a Gaussian kernel of 8 mm full-width half-maximum. Parametric modulations were used for further analysis and realignment parameters were added as nuisance factors. 35

36 4.1 Encoding Differences in activation during encoding can distinguish between trials later remembered and trials later forgotten (Paller, Kutas et al. 1987; "subsequent memory effect"; Brewer, Zhao et al. 1998; Wagner, Schacter et al. 1998). Our task contained morphed stimuli which cannot be considered completely old during retrieval. Recognition of 0% morphs should be easier than recognition of 20% morphs, which in turn should be easier than recognition of 40% morphs and so on. In an abstract sense, 0% morphed items permit a strong prediction on subsequent memory, while other items imply gradually less confidence in prediction of later memory. Based on this rationale, a weighting scale was created for predicting the parametric modulation of brain activity during encoding based on responses during retrieval. In sum, different modulation weights were assigned to different morphing steps so that weights for 0% morphs received the highest predictive values, while weights for 60% morphs were assigned the lowest predictive values (see Figure IV-1:A). Besides considering the objective morphing manipulation, the traditional subsequent memory effect was considered as well. Items, which were better encoded and thus later remembered (responses 3 or 4 ) received positive values (lower rows in Figure IV-1:A), while items later forgotten (responses 1 or 2 ) received negative values (upper rows). By combining the subsequent memory prediction with the variable prediction due to morphing, the following scale has been created: confidence ratings for trials morphed 60% were weighted -1.5, -0.5, 0.5 and 1.5, respectively, predicting only slight BOLD changes around the mean activity. The next morphing step (40%) received higher absolute weights than the 60% morphing step (-3, -1, 1, 3) because items were more similar to previously studied items and subsequent memory can therefore be predicted more confidently. The last morphing step (0%) involved the strongest prediction (-6, -2, 2, 6) since this was equivalent to a real subsequent memory effect. Given that predictions vary around zero and differ in magnitude depending on prediction confidence, the stimuli completely 36

37 identical for encoding and retrieval (0%) have a major impact on the regressor estimate, while the other steps have gradually less impact. In other words, a subsequently remembered 0% morphed face (response 4 during retrieval) is associated with a strong prediction of 6. If the data deviate from this prediction, the error term for estimating the beta regressor will be higher than for a value of 1.5 produced by a subsequently remembered 60% morphed stimulus. The expansion of confidence in prediction is illustrated with arrows in Figure IV-1:A. The shortfall of this scale is the possibility to miss strong activations for items well memorized during encoding, which are later presented morphed (i.e. 60%) and not recognized at all. Nevertheless, the direction of the scale (positive vs. negative) together with a decrease in magnitude for morphed values should minimize the error. The values represented in Figure IV-1:A were added as a parametric modulator after collapsing all trials of a stimulus category (i.e. faces) into one condition. The model was convolved with a 1.7s haemodynamic delta response function (HRF) for each trial. No time and dispersion derivatives were added. 4.2 Retrieval Previous studies have employed more than one general linear model (GLM) on the same data to investigate different aspects (i.e. Daselaar, Fleck et al. 2006). Similarly, two separate GLMs were used to analyze different aspects of the retrieval phase. In addition to the two study conditions (faces and scenes), each GLM contained two retrieval conditions (faces and scenes) with all trials of a stimulus category collapsed together and convolved with the HRF delta function of 5s duration. The two GLMs differed only with respect to the parametric modulations added to retrieval conditions. The first GLM (GLM1) contained two parametric modulations for retrieval conditions, one with the vector of confidence ratings ( ) and another with the morphing values ( ; Figure IV-1:B). 37

38 Figure IV-1: (A) Modulation values to code the subsequent memory effect during encoding. Arrows indicate the expansion of prediction confidence. (B) Schematic representation of parametric increase for objective morphing and subjective respone. (C) The combination of morphing and response scales in virtual planes of memory success and failure. In the upper part, a schematic representation of the correlations between virtual planes after orthogonalization with RTs. A custom scheme of orthogonalization was used such that parametric modulations for responses and morphing were orthogonalized with RTs (present at retrieval conditions in both GLMs), but not with each other. This procedure permits the unique attribution of activity to the morphing or the confidence rating scale, while the common effect is lost (see Appendix B: orthogonalization of regressors during retrieval, page 115). After orthogonalization with RTs the correlation between morphing and response vectors was 0.46 (STD ± 0.16) for faces and 0.58 (STD ± 0.13) for scenes. 38

39 The other GLM (GLM2) focused on the combination of responses and morphing to create hits, misses, false alarms (FAs) and correct rejections (CRs). Given the parametric nature of both factors, virtual planes has been created for Hits, Misses, CRs and FAs (Figure IV-1:C). Each plane was obtained by multiplication of stimulus values with response values, i.e. Hits carry the peak where a stimulus is completely old (considered 5 ) and response confidence is 4 (5 4 = 20). The lowest value on the Hits plane occurs when a new stimulus (considered 1 ) is rated as 1 (1 1 = 1). The opposite is true for correct rejections by simply inverting stimulus and response scales before multiplication. The other planes were obtained by inverting one of the factors prior to multiplication, i.e. Misses have a peak when the stimulus is identical to the previously studied one (again, considered 5 ), and is rated 1 (inverted to 4; 4 5 = 20). Figure IV-1:C also shows the correlation coefficients between planes after orthogonalization with RTs. Notably, there is a high correlation between Hits and CRs (r = 0.89) in the main diagonal compared to the off diagonal. This pattern confirms that more old stimuli obtain gradually more old responses, reflecting a performance along the main diagonal. Wrong answers ( Misses and FAs ) produce a lower correlation because they do not represent the main response strategy (represented by the main diagonal). CRs were removed to permit model estimability and the main diagonal was considered a single continuum of activation / deactivation, i.e. activation for Hits = deactivation for CRs and vice versa. The other two planes ( Misses and FAs ) were added in the model despite their correlation. 5. Results 5.1 Behavioural results Given the actual interest of many studies on familiarity and recollection processes, behavioural results were analyzed with a dual-process dissociation algorithm (Yonelinas, Dobbins et al. 39

40 1996) 8. Figure I-1 shows familiarity and recollection estimates in relation to morphing steps. Using these estimates as dependent variable, a repeated measures ANOVA was applied with the factors morphing (0%, 20%, 40%, 60%), material type (faces, scenes) and process type (familiarity, recollection). Results yielded evidence for a significant effect of morphing manipulation (F (3,54) = 86.3, p < 0.001, η² =.83) with a significant linear polynomial contrast (F (1,18) = 186.8, p < 0.001, η² = 0.91). The interaction between material type and morphing did not reach significance (F (3,54) = 0.82, p = 0.487, η² =.04), suggesting a comparable effect of morphing on faces and scenes. There was a tendency to remember scenes better than faces (F (1,18) = 3.62, p = 0.07, η² =.17). The interaction between material type and familiarity / recollection processes was not significant (F (1,18) = 0.52, p = 0.481, η² =.03), suggesting a uniform distribution of memory processes across material types in general. Finally, the interaction between morphing and familiarity / recollection estimates was significant (F (3,54) = 10.73, p < 0.001, η² =.37), because the decrease for familiarity is steeper than recollection (Figure IV-2). The three-way interaction was not significant (F (3,54) = 0.66, p = 0.581, η² =.04). 8 A revised algorithm has been created in Matlab, available at: 40

41 Figure IV-2: Estimates of recollection and familiarity from the ROC curves of faces and scenes. Carry over and fatigue effects were analyzed by a 4 2 repeated measures ANOVA involving experimental block (four blocks) and familiarity/recollection. Results show no significant change of performance between blocks (F (3,54) = 0.41, p = 0.748, η² =.02) and no interaction between blocks and familiarity/recollection (F (3,54) = 0.63, p = 0.6, η² =.03). It is important to stress the lack of any interaction with experimental blocks, which confirms a stable performance across time and similar brain activity between fmri sessions. FMRI analysis was performed in two steps. First, subject contrasts were obtained for each parametric modulation. Second, random effects for a parametric modulation were analyzed at the group level. Given the exploratory nature of our study, the analysis was confined to a region of interest (ROI) and an uncorrected threshold of p < was used at the group level. A minimum of eight consecutive significant voxels constituted the minimal cluster considered statistically significant (k = 8). The ROI was defined as a mask of hippocampal, parahippocampal and amygdala regions from the WFUpick atlas (Maldjian, Laurienti et al. 41

42 2003) and the Anatomy Toolbox (Eickhoff, Stephan et al. 2005) merged together. Each of the atlases contains areas not covered by the other atlas alone, thus masks needed to be merged. Replicated regressors (encoding and RTs) in both GLMs were analyzed with paired t-tests (equal variance, dependent samples) to verify that they did not change from GLM1 to GLM2. None of the tests on replicated regressors yielded a significant difference between the GLMs at p < (uncorrected, k = 5). In conclusion, encoding and RTs did not change significantly between GLM1 and GLM2. Reported results were arbitrarily taken from GLM2. The next step involved the comparison of GLMs for explaining registered BOLD activity. To investigate which GLM is more effective on explaining MTL activity, an F-contrast was run on both faces and scenes for every factor of interest. Because the direction of activity modulation can be either increasing or decreasing, the original threshold was lowered for F-tests (p < ). The number of significant voxels for each modulation of interest is shown in Table IV-1. Table IV-1: Comparison between GLMs. GLM1 GLM2 # voxels mean Z mean F (2,18) Response Morphing Hits Misses FAs The combination of subjective and objective factors (GLM2) produced more significant voxels compared to the contribution of each factor separately (GLM1). In total, 370 significant voxels were found in GLM1, mostly due to activity for subjective responses, while 2012 voxels were active in GLM2 for the combination of objective and subjective factors in hits, misses and false alarms. 42

43 5.2 Neural correlates: Encoding As expected, better encoding was associated with increasing BOLD signal for both types of material. For faces, this increase was located bilaterally in the amygdala (Figure IV-5:A, y = - 6), bilaterally in the hippocampi (Figure IV-5:A, y = -20), and bilaterally in medial parahippocampal cortex (Figure IV-5A, y = -30). For scenes, activity increase was observed bilaterally in the posterior and medial parahippocampal cortex (Figure IV-5:A, y = -42, y = - 30), bilaterally in the body of the hippocampus (Figure IV-5:A, y = -30) and anteriorly in the right amygdala and right perirhinal cortex (Figure IV-5:A, y = -6, y = 6). Importantly, activity in some areas was not exclusive for one stimulus type but overlapped with activity for the other stimulus type as well (i.e. right middle parahippocampal cortex; Figure IV-5:A, y = -30). 5.3 Neural correlates: Retrieval Morphing, i.e. objective similarity of faces or scenes to previously studied ones, modulated activity in several regions independently of subjective memory strength. Activity in the left amygdala increased for more similar faces and decreased for more similar scenes. However, these areas of opposing activity did not overlap (Figure IV-5:B, y = 0). Gradually more similar scenes also elicited gradually decreasing activity in the right hippocampus (Figure IV-5:B, y = -18), right entorhinal/perirhinal cortex (Figure IV-5:B, y = 0), and left parahippocampal gyrus extending into the subiculum (Figure IV-5:B, y = -18). Subjective memory strength modulated activity independently of morphing. Activity increased with increasing memory strength both for faces and for scenes. Specifically, subjective memory for scenes correlated with increasing activity in the amygdala bilaterally, with the right cluster extending into the entorhinal cortex. Subjective memory for faces increased activity only in the right amygdala. Again, activity modulation for each stimulus type showed nearly no overlap (Figure IV-5:C, y = 3, y = -3). Additional clusters in bilateral hippocampi, 43

44 mostly in the subiculum probability maps (Figure IV-5:C, y = -30, y = -42), increased activity with subjective memory strength for faces. The Hits modulation produced the most extensive significant activation in the MTL, revealing activity changes in the virtual continuum Hits<->CRs. Specifically, Hits for faces correlated with increasing activity bilaterally in the amygdala, extending into the hippocampus and entorhinal/perirhinal areas (Figure IV-5:D, y = 0), in the right entorhinal/perirhinal cortex (Figure IV-5:D, y = -10), in the right hippocampus (Figure IV-5:D, y = -20), and posteriorly in bilateral hippocampi, mostly in the subiculum probability maps (Figure IV-5:D, y = -30, y = - 40). The only cluster to increase activity for scenes overlapped with the cluster exhibiting an activity increase for faces in posterior MTL, falling within the probability maps of the subiculum (Figure IV-5:D, y = -40). However, as opposed to faces, scenes led to additional activity decreases in the body of the left hippocampus (Figure IV-5:D, y = -20) and the head of the right hippocampus extending into amygdala and entorhinal/perirhinal areas (Figure IV-5:D, y = -10). Notably, the activity decrease for scenes and the increase for faces were adjacent but not overlapping in right anterior MTL. The FAs modulation revealed activity changes for increasingly erroneous attribution of new stimuli as old. Faces produced no significant activity change. Scenes caused activity decreases in bilateral entorhinal/perirhinal areas, bilateral hippocampi extending into the amygdalae (Figure IV-5:E, y = -14), bilateral body of the hippocampi (Figure IV-5:E, y = -24, y = -30), left parahippocampal cortex (Figure IV-5:E, y = -24), and the tail of the right hippocampus extending into subiculum probability maps (Figure IV-5:E, y = -40). The Misses modulation revealed an activity change for increasingly erroneous evaluation of old stimuli as new. Both missed faces and missed scenes were associated with decreasing 44

45 activity. Specifically, for faces activity decreased only in right inferior temporal cortex (fusiform gyrus), adjacent to parahippocampal gyrus (Figure IV-5:F, y = -32), while scenes led to decreasing activity bilaterally in the heads of the hippocampi (Figure IV-5:F, y = -14), bilateral bodies of the hippocampi on the subiculum probability maps (Figure IV-5:F, y = -26), left amygdala (Figure IV-5:F, y = -4), left entorhinal/perhirhinal areas, left parahippocampal gyrus, and tail of the hippocampus extending into subiculum probability maps (Figure IV-5:F, y = -40). RTs modulation revealed BOLD changes for increasingly longer RTs. For both faces and scenes, strong negative correlations between MTL activity and RTs were observed, i.e. longer RTs correlated with gradually decreased activity in MTL. This pattern was observed for faces bilaterally along the whole axis of the hippocampus extending into the amygdala anteriorly (Figure IV-5:G) and into the left parahippocampal cortex posteriorly (Figure IV-5:G, y = -40). For scenes, decreasing activity was observed in the right entorhinal/perirhinal area (Figure IV-5:G, y = -10), bilaterally in the amygdalae (Figure IV-5:G, y = -2), bilaterally in the heads of the hippocampi (Figure IV-5:G, y = -10), right subiculum (Figure IV-5:G, y = -22) and left body of the hippocampus (Figure IV-5:G, y = -30). Face and scene related activity was overlapping in anterior MTL (Figure IV-5:G, y = -2, y = -10), but also in the left hippocampus (Figure IV-5:G, y = -30). Increasing activity for increasing RTs was only found for scenes, primarily in right posterior parahippocampal cortex (Figure IV-5:G, y = -40). Additionally, small clusters in left posterior parahippocampal cortex and subiculum increased activity for longer RTs in response to scenes (see Table A-1 in Appendix A, page 107, for a full presentation of the results). 45

46 5.4 Whole brain analysis Parietal activations The importance of the parietal lobes for mnemonic processes is well known (for a review, see Wagner, Shannon et al. 2005). Parietal activity is not simply related to motor preparation and attentional processes (Shannon and Buckner 2004), it shows an increasing gradient for items perceived as old (Kahn, Davachi et al. 2004; Wheeler and Buckner 2004) and there is evidence for a functional relationship between hippocampus and the inferior parietal lobule (Vincent, Snyder et al. 2006). Moreover, parietal areas whose activity correlates with hippocampal activity are spatially distinct from parietal areas whose activity correlates with that in visuospatial attention areas (Vincent, Snyder et al. 2006). Various models have been proposed to explain the functional distinction between inferior and superior parietal activity. For example, recollection-specific activity has been found in the inferior parietal lobule, while familiarityrelated activity has been found in more superior areas (Wagner, Shannon et al. 2005). More recently, the attention-to-memory model defined a distinction between effortful retrieval (topdown) and fast automatic retrieval (bottom-up) mirrored in superior parietal activity and inferior parietal activity, respectively (the AtoM model; bottom-up vs. top-down attention; Cabeza, Ciaramelli et al. 2008). In our study, memory success ( Hits ) elicited higher activity in superior parietal areas, both for faces and for scenes, while memory failures ( Misses and FAs ) correlated with decreased activity in inferior parietal areas. Additionally reaction times played an important role in dissociating superior from inferior parietal activity. Higher activity is observed for long reaction times in the superior parietal lobule and the opposite pattern is observed in the inferior parietal lobule, i.e. lower activity for long reaction times. The ventral parietal area observed in our experiment is similar to the area correlating with hippocampus activity (Vincent, Snyder et al. 2006) or the areas found to be related with recollection responses (see meta-analysis, Fig. 4 in Cabeza, Ciaramelli et al. 2008). Curiously, in our experiment the inferior parietal lobule exhibited the same relationship to RTs as the 46

47 hippocampus, higher activity for fast RTs, possibly fostered by the functional relationship described in Vincent et al. (2006). While our experiment cannot clearly distinguish between recollected and familiar items, it is important to note a dorsal-ventral axis for RTs. Given that RTs differ between confidence levels or remember / know responses, it is likely that fmri contrasts for high vs. low (or remember vs. know) responses compare fast RT areas with slow RT areas. Following this rationale, the studies cited by the authors in support of the AtoM model were investigated. In four of the five studies referred to in Cabeza et al. (2008), faster RTs were found for high confidence and remember responses (Wheeler and Buckner 2004; Yonelinas, Otten et al. 2005; Daselaar, Fleck et al. 2006; Kim and Cabeza 2007) and one study used a special procedure which does not allow firm conclusions concerning RTs (Moritz, Glascher et al. 2006). The first author of the latter study reports a pattern similar with the other studies (Moritz, Glascher et al. 2006, personal communication). From the five up mentioned studies only one controlled explicitly for RTs (self-paced fmri analysis; Wheeler and Buckner 2004), even though RTs were not used as regressors. Therefore, the possibility of RTs affecting different contrasts cannot be excluded. Likewise, the same problem arises in many remember/know and confidence rating experiments, whereby remember and high confidence responses are typically faster than know and low confidence ones. The distinction between effortful retrieval (top-down) and fast automatic retrieval (bottom-up) is temporal overall, which may fit with finding different RTs in superior and inferior parietal areas. From this perspective, RTs reflect differences in processing time, and not motor processes per se. Nevertheless, it was possible to separate the activity related to RTs from the activity related to memory processes. This is a step in advance toward the dissociation of the factors involved in a memory task. Overall, our results support the dissociation between ventral and dorsal parietal areas, but the functional relationship with memory processes emerges in terms of failures and successes, respectively. It was not found the frequently mentioned pattern of higher activity for Hits in 47

48 the ventral parietal cortex, which is attributed to recollection in dual-process accounts (Wagner, Shannon et al. 2005). Instead, decreased activity was observed for Misses and FAs, as well as for slow RTs. The incongruence with previous studies can be explained by the different approach of the data analysis. Differently from previous studies, the present study employed parametric modulations to investigate activity patterns independently from each other. Traditional contrasts, instead, explore the relative difference between conditions. In this case, higher activity for a certain condition can be observed when the activity is lower for the opposite condition. For example, the contrast Hits > Misses may reveal higher activity for Hits when the activity is effectively decreasing for Misses. Consequently, previous experiments may have misinterpreted BOLD activity in two possible directions: 1. By contrasting Hits vs. Misses / FAs or 2. By contrasting fast RTs with slow RTs, since high confidence (or remember ) responses are typically answered faster than low confidence (or know ) responses. Given the results obtained from our analysis, it is plausible that any traditional contrast would produce results similar to those published in the literature, although the pattern of underlying activity could be the same as in our experiment. While these conclusions are based on the activity of the left hemisphere only, it is important to note that parietal activity during memory tasks is often found lateralized to the left (Cabeza, Ciaramelli et al. 2008; Vilberg and Rugg 2008; Spaniol, Davidson et al. 2009). 48

49 Figure IV-3: Activitiy in parietal cortex (p < 0.001). Arrows indicate decreasing activity in the inferior parietal cortex. 49

50 5.4.2 Occipitotemporal activity Activity in ventral occipitotemporal areas was significantly higher when items were encoded better based on subsequent performance ( subsequent memory effect ; see Figure IV-4, Encoding). Increasing activity was observed in similar areas for Hits during the retrieval phase, particularly for face stimuli (Figure IV-4, Hits). The opposite pattern, decreasing activity, was observed during retrieval for memory errors (Figure IV-4, Misses and False Alarms). These clusters overlap with those of encoding-related contrasts. In a recent comparison of memory correlates for faces and places, Prince et al. (2009) found a similar pattern of activity in posterior perceptual processing areas, suggesting a reactivation of stimulus-specific regions necessary for successful memory retrieval. The fusiform face area ("FFA"; Kanwisher, McDermott et al. 1997) has been previously shown to change its activity for familiar compared to unfamiliar stimuli (Rossion, Schiltz et al. 2003; Large, Cavina-Pratesi et al. 2008; Minnebusch, Suchan et al. 2009). Although a functional localizer was not used to verify the functional specialization for faces, our results confirm these findings and support the hypothesis that higher order perceptual processing is required in order to make successful memory decisions. The role of perceptual processing areas in memory decisions could be limited to retrieve or replay perceptual features, but their involvement in all response categories suggests a specific relationship with encoding-retrieval processes. Increasing activity was observed for successful memory encoding and retrieval, while memory failures during retrieval were accompanied exclusively by an activity decrease. Although our results confirm and extend those of Prince et al. (2009), an important question is raised by other conflicting results in the field (see conclusive discussion, Perceptual-processing areas and memory failures, page X). 50

51 Figure IV-4: Whole brain analysis at p < 0.001, in focus the lower occipital and temporal areas. 6. Discussion The aim of the current study was to elucidate MTL mechanisms involved in memory for faces and scenes. Graded morphing was used to create distractors from targets and their effect on memory quality was studied. Face morphing has been extensively used to study perception of emotions, races, and memory processes (Steyvers 1999; Walker and Tanaka 2003; Balconi and 51

52 Lucchiari 2005; Jager, Mecklinger et al. 2006). Scene morphing, on the other hand, is limited by the impossibility to match spatial elements of different natural landscapes onto each other. In a novel approach, the manipulation procedures were extended to scenes and morphing scales for faces and scenes were calibrated to a common function of perceived morphing level. Previous studies have shown that faces morphed at 35% are still perceived as representing the same person if no calibration is performed (Jager, Seiler et al. 2005; Rosler, Ranganath et al. 2009, pp 363). The calibration, therefore, has a double effect of (1) equating the scales for faces and scenes and (2) removing non linear perceptual biases inside each morphing scale. Behavioural results confirmed the validity of such calibration, with both stimulus categories producing parallel lines of estimated familiarity (Figure IV-2). The familiarity / recollection manipulation provided important insights into dynamics of these processes during graded memory shifts. Familiarity followed the covert stimulus manipulation almost linearly, while recollection exhibited sudden shifts at different steps for faces and scenes. Face recollection dropped immediately between 0% and 20% morphing, while scene recollection persisted at 20% and dropped only at 40% morphing (Figure IV-2). These results can be explained by the different morphing procedures. Faces morphed 20% are already changed in every facial feature to create a person overall different from the previously studied person. The effect seemed detrimental to recollection, but not to familiarity. Scenes morphed 20%, on the other hand, are not manipulated in all features, but only shifted to one side. Many objects and elements of the original scene are still present at 20% morphing. These unvaried elements can elicit recollection, despite the overall spatial configuration shift. What the results suggest are two important properties of recollection. First, scene recollection does not require the preservation of the whole spatial configuration. A few identical elements with preserved local configuration might be enough for the full recollection of a scene. Second, recollection cannot be graded for holistic stimuli (i.e. faces), but requires the presentation of the identical stimulus that has been encoded. Imaging studies on repetition suppression support such 52

53 conclusion, since manipulated or degraded stimuli produce a reversal of the repetition effect, i.e. repetition enhancement (Dolan, Fink et al. 1997; Grill-Spector, Kushnir et al. 2000; Kourtzi, Betts et al. 2005; Turk-Browne, Yi et al. 2007). Activity in the MTL The precise nature of memory retrieval processes associated with MTL activation is still under debate. In contrast to other studies (i.e. Prince, Dennis et al. 2009), in the present study faces and scenes has been analyzed separately and has not been contrasted against each other. The memory effect for each material type was analyzed by means of a parametric modulator permitting the investigation of subtle shifts in MTL activity in parallel for faces and scenes. 6.1 Activity at Encoding Both materials increased activity for better encoded stimuli in MTL areas (subsequent memory effect). The posterior parahippocampal gyrus showed increased activity bilaterally during encoding of scenes but not faces (Figure IV-5:A, y = 42). This finding is consistent with the specialization of this area in contextual associations (Aminoff, Gronau et al. 2007; Bar, Aminoff et al. 2008) and spatial information encoding (Epstein and Kanwisher 1998; Epstein, Harris et al. 1999). Importantly, this pattern emerged spontaneously from single parametric modulations without the need for a functional localizer to isolate the parahippocampal place area (Epstein and Kanwisher 1998). Our results support the specialization of the posterior parahippocampal cortex in encoding spatial content independently of a spatial vs. non-spatial comparison. 53

54 Figure IV-5: Activity correlates in MTL (p < 0.001, k = 8) overlaid on the mean anatomical scans of all 19 normalized subject brains. 54

55 More anteriorly, increasing activity for faces and scenes was contiguous, while overlapping clusters for faces and scenes were found in right parahippocampal gyrus (Figure IV-5:A, y = 30). Adjacent clusters emerged even more anterior, but only in the right hemisphere (Figure IV-5:A, y = 20). Anterior MTL clearly supports faces and scenes differently. Left MTL correlates with face encoding, while right MTL shows adjacent clusters of activity for faces and scenes (Figure IV-5:A, y = -6). Increasing activity for scenes in anterior MTL is particularly interesting, since this area exhibited higher activity for faces in main condition contrasts (Figure IV-6). These results suggest that not only anterior MTL / PRc exhibit heterogeneous responsiveness to spatial and non-spatial stimuli, but that anterior activity may be especially tuned to scenes in separate clusters irrespective of any comparison with other stimuli. Disentangling stimulus-specific activity with parametric modulations is beneficiary in this context. Figure IV-6: Main condition contrasts during retrieval (p < 0.001, k = 8) and BOLD signal time courses extracted for the highest peak of activity evoked by faces (left) and scenes (right). Dashed lines represent 90% confidence intervals. Data extracted with RFXplot ver 36 [Glascher J Visualization of group inference data in functional neuroimaging. Neuroinformatics 7(1):73-82] 55

56 Our results confirm previous findings of common activity in anterior MTL for both spatial and non-spatial materials (Buffalo, Bellgowan et al. 2006; Bellgowan, Buffalo et al. 2009; Preston, Bornstein et al. 2009). The lateralization of face processing was shown in Litman et al. (2009), who reported that responses for faces were above baseline only in anterior MTL, particularly in the left hemisphere. Consistent with their study, a clear anterior-posterior differentiation was found during encoding in the left hemisphere, where anterior MTL activity correlated exclusively with face encoding. Less clear differentiation was observed in right anterior MTL, where face and scene clusters arose in adjacent portions. A preference for spatial stimuli in the right MTL is demonstrated in two other studies, which showed higher laterality indices in right MTL for spatial encoding, and higher laterality in left MTL for object encoding (Buffalo, Bellgowan et al. 2006; Bellgowan, Buffalo et al. 2009). However, rather than complete lateralization, our results and those of the afore-mentioned studies indicate a biased laterality. The lack of a complete lateralization suggests that MTL functionality may be temporary plastic in lateralizing some processes due to the task properties, but existing bilateral connections are present for holistic and spatial processing. 6.2 Activity at Retrieval Various causes influence activity during retrieval. First of all, retrieval implies a response from the subject, which can take the form of graded confidence strength, remember/know, or new/old responses. Obtaining a response requires some motor action and decision making. This picture obscures the interpretation of neural activity on the basis of memory processes alone. Motor actions create different reaction times between confidence levels or remember/know judgments. Typically, highly confident and remember responses are found to be faster, and these responses are associated with recollection (Dewhurst and Conway 1994; Dewhurst, Holmes et al. 2006; Rotello and Zeng 2008). 56

57 Second, the repetition of the stimulus may trigger automatic processes, i.e. repetition suppression, perceptual fluency, etc (Grill-Spector and Malach 2001; Voss, Hauner et al. 2009). Third, retrieval should not be considered as opposed to encoding. These processes may complement, compete or alternate each other, but their progression is tightly connected. The retrieval of old material is typically accompanied by encoding of new items, a process called incidental encoding, which suggests that during retrieval there are encoding processes which should not be ignored (Buckner, Wheeler et al. 2001; Stark and Okado 2003). In addition, retrieval and encoding do not temporally serialize. Rather they seem to influence each other during the test phase, suggesting that general cognitive modes apply. For example, incidental encoding improves when retrieval is tested in conditions of full attention (Dudukovic, Dubrow et al. 2009) and when new items are presented along with old items previously studied in deep compared to shallow encoding (Jacoby, Shimizu et al. 2005; Shimizu and Jacoby 2005). These findings suggest a re-entry in the original encoding mode or a reinstatement of the cognitive state experienced during encoding. In the present study, we attempted to elucidate the contribution of three different factors: objective stimulus identity, subjective memory strength, and reaction times. The possibility to find encoding activity during retrieval was considered as well. The main question that our study might answer is whether the contribution of awareness is an important factor explaining haemodynamic activity during retrieval (Henke, Mondadori et al. 2003; Daselaar, Fleck et al. 2006; Danckert, Gati et al. 2007). One possibility is that MTL elaborates perceptual identity automatically and this could be independent of self-awareness. Consistent with this hypothesis, previous studies observed suppression of the BOLD signal in anterior MTL for repeated stimuli observed passively (Henson, Cansino et al. 2003; Gonsalves, Kahn et al. 2005; Furl, van Rijsbergen et al. 2007). Similarly, in this study it was expected to 57

58 find a corresponding decrease for stimuli gradually more similar to those previously studied. Indeed, a gradual signal decrease was found in bilateral amygdala, right perirhinal cortex and right hippocampus for increasingly similar scenes (Figure IV-5:B). However, no such decrease was observed for faces. Instead, gradually more similar faces correlated with increasing activity in the left amygdala (Figure IV-5:B, y = 0), a similar area as activated during face encoding (Figure IV-5:A, y = -6). While the role of the amygdala in memory processes has been previously stressed (Kensinger and Schacter 2006), it is interesting to note that activation of the left amygdala correlated with objective recollection 9 in many studies (Spaniol, Davidson et al. 2009) and with objective similarity in our analysis, suggesting a role in automatic retrieval. Opposite activity patterns, increasing activity for more similar faces and decreasing activity for more similar scenes (Figure IV-5:B, y = 0), suggest that this region might, nevertheless, be involved in objective processing of both stimuli. Increasing activity for faces and decreasing activity for scenes may be explained by the different nature of the morphing procedures. Specifically, the original elements of a scene are preserved and shifted, while faces are morphed in all recognizable elements. Previous studies have stressed the importance of stimulus invariability for repetition suppression. The exact persistence of visual features is so important that degraded or manipulated repetitions produce enhancement rather then suppression of activity (Dolan, Fink et al. 1997; Grill-Spector, Kushnir et al. 2000; Kourtzi, Betts et al. 2005; Turk-Browne, Yi et al. 2007). Remarkably, even identical stimuli may not produce repetition suppression if they are illuminated from a different angle (Tootell, Devaney et al. 2008). As a result, repetition suppression may occur for identical objects still present in the shifted scene, but not for morphed faces, which rather elicit repetition enhancement. 9 Objective recollection paradigms consist of direct tests of memory for associations or contextual features, i.e. source memory. (Spaniol et al., 2009) 58

59 On the other hand, subjective feelings of memory correlated with activity increase for both faces and scenes. Based on previous studies (Henson, Cansino et al. 2003; Weis, Specht et al. 2004; Gonsalves, Kahn et al. 2005), it was expected to find an activity decrease for increasing memory confidence, especially in perirhinal cortex, which was not the case. Counterintuitive results may derive from the full separation of subjective and objective factors (i) and the linear parametrization of both factors under investigation (ii). In support of this explanation, previous studies have shown that perirhinal activity is more suppressed for correct answers ( hits ) than for incorrect answers ( false alarms ; Weis, Specht et al. 2004, see fig 2; Danckert, Gati et al. 2007, see fig 2), suggesting that both subjective feeling and objective status contribute to perirhinal suppression. The joint contribution of separate factors investigated through unorthogonalized parametric modulations may cancelled, leaving weak neural correlates for each factor separately (Andrade, Paradis et al. 1999). In turn, activity attributed uniquely to subjective memory can be weak or perhaps non-linear after the removal of the concomitant objectively explained variability. Subjective memory strength (responses 1 to 4 ) correlated with increasing activity in bilateral posterior parahippocampal gyrus and right amygdala for faces. In a similar region of the right amygdala, partially overlapping, activity increased for increasing memory strength for scenes (Figure IV-5; y = 3, y = -3), suggesting the involvement of the right amygdala in subjective awareness for both material types. Based on the common activity found in adjacent areas for both materials, the conclusion drawn is that left anterior MTL is tuned more to automatic perceptual processes, while right anterior MTL is involved more in subjective memory experience. Results from GLM2 indicated that our previous interpretation about concurrent objectivesubjective factors was correct. Linear modulations of Hits, Misses and FAs convey important information to the model, producing significant activity in broader clusters with a 59

60 better fit. Higher z-scores in group contrasts indicated that, in general, activity in MTL is better explained by a combination of subjective and objective factors than by either of the two alone. Beside the previously mentioned fmri studies, mixed properties of MTL neurons have been observed in physiological recordings. Using single-neuron electrodes in humans, Reddy et al. (2006) found MTL neurons firing preferentially for certain stimuli (perceptual component), but their response was stronger for correct than for incorrect answers during a change detection paradigm (awareness component). Similarly, in a recent experiment, Voss et al. (2009) observed suppressed activity for primed words in bilateral perirhinal cortex, but left PRc correlated with the behavioural outcome as well. Results from the latter study indicate that the combination of objective and subjective factors may arise in locally distinct areas of MTL. At a broader view, this interpretation supports a recent reconsideration of MTL activity in terms of perceptual processes (Barense, Bussey et al. 2005; Lee, Bussey et al. 2005; for a review Murray, Bussey et al. 2007; Graham, Barense et al. 2010). These authors attribute a perceptual function to MTL areas, supported by subtle discrimination deficits in patients with MTL damage. Supporting this view, it has been shown that perirhinal activity can be modulated by perceptual tasks without a proper mnemonic component (Furl, van Rijsbergen et al. 2007). That MTL is a converging point for the what and where pathways is widely known in the literature (Eichenbaum, Yonelinas et al. 2007; Litman, Awipi et al. 2009), but the extent to which perceptual processes cooperate with memory oriented processes is not yet clear. According to Voss et al. (2009), the interaction of implicit and explicit factors during memory tasks may arise as a joint contribution of facilitated perceptual processing (i.e. perceptual fluency or priming) and familiarity-based recognition. These processes have been previously investigated separately and thought to be unrelated with respect to MTL activity. Our study fills this gap by demonstrating a concurrent contribution of objective and subjective factors in MTL activity during memory tasks. Yet, the relationship between these factors has to be established and is probably right key to understand memory processes. 60

61 MTL activity for memory failures Activity correlated with memory errors was only observed in response to scenes (Figure IV-5:E/F) 10. Scenes caused gradually lower activity for Misses and FAs along all the anterior-posterior axis, particularly in the hippocampus proper. Given that both errors correlated with decreased activity, it is possible that memory failures arise from insufficient activity in the hippocampus, which given the exclusive relationship with errors for scenes, may be crucial in correctly processing relational properties of spatial stimuli (O'Keefe and Nadel 1978; Burgess and O'Keefe 2003; Bird and Burgess 2008). MTL activity for memory successes Memory success modulated activity both for faces and for scenes. In particular, an area in left posterior MTL increased activity for Hits, for spatial and non-spatial stimuli (Figure IV-5:D, y = -40). For scenes, this was the only cluster exhibiting an activity increase, the other two clusters decreased activity for Hits. An activity decrease in MTL for Hits is atypical for memory tasks, but can be explained by the nature of our analysis. Given that Hits and CRs are in the same linear continuum, decreasing activity for Hits could be attributed to an activity increase for new stimuli recognized as new, possibly associated with encoding processes. Confirming this hypothesis, one of the two clusters showing decreasing activity for Hits during retrieval phase overlap nearly completely with one of the clusters increasing activity during encoding, suggesting a specific role of this area (right anterior hippocampus) in encoding novel scenes, and possibly signalling the upcoming novelty during retrieval to help memory success. The replication of the same activity in retrieval phase provides strong evidence of encoding processes occurring during retrieval. 10 A cluster was found to correlate with memory errors for faces, but this was small and fell out of the areas known to be involved in memory processes (fusiform gyrus; see Table 2). 61

62 Encoding and retrieval clusters for faces overlap in the opposite way. They increase activity both during encoding and during retrieval. Specifically, three areas, two in bilateral anterior hippocampus/amygdala and one in the right body of the hippocampus, increased activity both during the encoding phase and during Hits in the retrieval phase, suggesting once again the involvement of encoding mechanisms in the correct categorization of stimuli during retrieval. It is unclear why faces and scenes are characterized by different coupling patterns, though one reason could rely on the different processing mechanisms of spatial vs. non-spatial stimuli, which may follow different encoding-retrieval couplings. Another reason could be the different morphing procedures. Faces are morphed in every aspect, while scenes preserve the same elements shifted aside. An interaction of the above reasons is possible as well. Figure IV-7: Mean reaction times accross subjects distributed on the four response buttons for faces (yellow) and scenes (blue). Error bars denote standart deviations. 6.3 MTL activity related to reaction times Reaction times were included in our analysis to remove variability explained by motor actions. Though this was not the main focus of research, a striking impact of RTs on MTL activity was surprisingly found during retrieval. Preliminary analysis showed that this outcome did not 62

63 depend on the order of parametric modulations or on orthogonalization. The distribution of RTs across the four response options confirmed that high confidence responses ( 1 and 4 ) were answered faster than low confidence responses ( 3 and 4 ; Figure IV-7). The Λ-shape function does not follow the linear morphing and subjective memory scales. For this reason, it is possible that RTs explain signal changes in MTL occurring in parallel and independently of the other factors. In one cluster in right posterior parahippocampal cortex, activity increased for increasing RTs of scenes (Figure 4-G). Notably, this area increased activity also during successful encoding, despite no motor action was required in encoding phase. All other clusters showed decreased MTL activity with increasing RTs, extending along the whole anterior-posterior axis, both for faces and for scenes, eventually overlapping in anterior MTL (Figure IV-5:G). Lower activity for slow RTs indicate higher activity for fast RTs. Figure IV-7 shows that fast RTs peaked on responses 1 and 4. Consequently, maximum activity is observed for encoding sure new and retrieving sure old items. This result is consistent with previous research showing high activity both for retrieved (hits) and for encoded stimuli (CRs) in the retrieval phase (Buckner, Wheeler et al. 2001; Stark and Okado 2003; Danckert, Gati et al. 2007; Johnson, Muftuler et al. 2008). The RT regressors, therefore, reflect the Λ-shape activity of the confidence scale. The co-occurrence of short RTs and high confidence (or remember ) responses is a standard result in many studies (Dewhurst and Conway 1994; Dewhurst, Holmes et al. 2006; Rotello and Zeng 2008). Given that RTs differ between confidence levels or remember / know responses, it is likely that fmri contrasts for high vs. low confidence levels (or remember vs. know) have actually compared fast RTs with slow RTs. These contrasts would produce inevitably higher activity for the fastest responses, typically remember and high confidence. The pattern of activity predicted by this assumption is actually observed in the literature, where remember vs. know show higher activity in the hippocampus. Therefore, most studies on 63

64 human recognition memory may actually have measured a difference in MTL activity relying simply on temporal differences between responses, rather than pure cognitive processes. Despite the intention to separate memory mechanisms from behavioural responses (after all we do not press a button in everyday life), it may be difficult to spoil the activity related purely to cognitive processes from the temporal dimension of their resulting behavioural response. Nevertheless, given their strong correlation with MTL activity, RTs may be an important aspect to control in recognition memory studies and investigate in the future. 6.4 General considerations on MTL activity This study identified voxels linearly related to objective perceptual identity, subjective memory strength, and reaction times. Non-linear patterns of activity may exist which are not captured by scalar modulations. The simple finding of different functional shapes (linear for response confidence and Λ-shaped for RTs) suggest that activity in MTL is heterogeneous. Multiple factors might contribute separately or in concert to create a number of patterns of activity. In a recent experiment Johnson et al. (2008) showed that small areas in the hippocampus can contain both voxels expressing linear graded activity and voxels with non-linear categorical activity. Daselaar et al. (2006) used two separate GLMs to investigate the role of linear and non-linear activities presumably reflecting familiarity and recollection, respectively. Our study further confirms the existence of different types of activity revealed by different parametric modulations. Given that MTL regions show both linear and non-linear activity, depending on the scale used for analysis, it can be concluded that MTL specialization occurs in different dimensions. Slight modifications of the experimental design can lead to different results, hence to misleading generalized conclusions. In support to this conclusion, some studies have shown opposing results, i.e. hippocampal activity increase (Daselaar, Fleck et al. 2006) or decrease (Johnson, 64

65 Muftuler et al. 2008) in response to old items. As pointed out by Diana et al. (2007), the involvement of certain MTL regions in memory processes may depend on the specific task demands and the type of information involved. MTL heterogeneity together with a dynamic assignment of local processes may create a delicate scenario for the interpretation of studies of human recognition memory. 65

66 V. EXPERIMENT 3: ERPs The aim of this experiment was to investigate ERP components that correlate with subjective and objective memory for faces and scenes. Particular attention was posed to investigate the role of ERP components previously assigned to familiarity and recollection in the objective/subjective distinction. 1. Participants Twenty-seven subjects participated in the experiment. Three subjects were excluded from the analysis (1 because of technical failures, 1 because of poor performance, and 1 for balancing the left-right response order across participants). The remaining 24 participants (mean age 23.8 ± 2.3, 12 female, all right handed) had normal or corrected-to-normal vision. All subjects were reimbursed for participation and gave informed consent. 2. Experimental Task Stimuli were projected on a LCD monitor and subtended a visual angle of for scenes and for faces. Similar to the behavioural experiment (Chapter III, page 28), participants viewed a series of intermixed face and scene stimuli (study phase). They were instructed to memorize the stimuli as good as possible for a later recognition test. Four blocks of study-test alternation were completed. During each block, 48 stimuli were presented in the study phase (24 faces) and 72 stimuli were rated during the recognition phase (24 completely new). Each stimulus remained on the screen for 1700 ms during the study phase and for an unlimited time during recognition until the subject responded. Six response buttons were positioned on the index, middle and ring fingers of each hand corresponding to a 6-point scale ranging from sure new to sure old ( ). The scale was displayed underneath the stimulus. In order to balance the neural correlates of motor responses, half of the subjects used the scale in reversed left-right order ( ). The subjects received feedback which 66

67 button they had pressed on the screen before the next trial started, but no feedback was given about the accuracy of his/her choice. During the study phase, the presented stimuli were considered 0% morphed. The 48 old stimuli presented during recognition comprised truly old items (i.e. the same 0% morphed stimuli presented in the study phase) as well as different morphing steps of the studied stimuli (20%, 40%, and 60%). The overall number of morphed faces or scenes for a certain morphing step in the entire experiment was 24 (i.e. 24 morphed faces at 0%, 24 morphed faces at 20%, etc.). 3. EEG recording and analysis EEG was recorded with 30 Ag/AgCl electrodes mounted in an elastic cap according to the extended system (Chatrian, Lettich et al. 1985). Registered electrodes comprised F7, F3, FZ, F4, F8, FT7, FC3, FCZ, FC4, FT8, T7, C3, CZ, C4, T8, TP7, CP3, CPz, CP4, TP8, P7, P3, PZ, P4, P8, PO7, PO3, POz, PO4, and PO8. All scalp electrodes were referenced to linked mastoids and impedance was kept below 10 kohm. A NeuroScan amplifier was used to record direct current EEG, amplify the data, and digitize with a 500 Hz sample rate. To control for ocular artefacts, eye movements were also recorded by placing electrodes at the outer canthi of both eyes as well as above and below the right eye. Analysis was performed using BrainVision Analyzer v1.05 software (Brainproducts, Munich, Germany). EEG data were filtered ( Hz with 24 db), corrected for ocular movements (Gratton, Coles et al. 1983), and corrected for DC drifts with the algorithm implemented in the Analyzer software. Data were segmented from 100 ms pre-presentation to 1500 ms postpresentation of the stimulus. The 100 ms pre-presentation were used for baseline correction. Segments with artefacts were removed An artifact was considered a shift of 50 µv or more between consecutive time points, a general amplitude shift more than 300 µv or less than 0.5 µv in the entire segment, or amplitude values exceeding the -75 µv to 75 µv range. These criteria were applied to all except occipital and ocular electrodes. 67

68 For analysis, eight electrodes were selected and pooled in four groups. The groups were anterior-left (F3-F7), anterior-right (F4-F8), posterior-left (P3-P7), and posterior-right (P4-P8). Each pool contained the averaged values of the two electrodes at any time point. Event-related potentials for each subject were obtained by averaging all the segments of an experimental condition. Grand averages were calculated over all subjects for inspection purposes (plots shown in Appendix A, page 107). 4. Results 4.1 Behavioural results Reaction times were analyzed by two separate ANOVAs. First, differences between experimental blocks were controlled in a 4 1 ANOVA and a tendency to answer faster in later blocks was observed (F (3,69) = 2.44, p =.07, η² =.10). Second, we analyzed the distribution of RTs for faces and scenes for each of the six buttons (2 6 ANOVA). Results showed that responses to faces were faster than responses to scenes (F (1,23) = 75.25, p <.001, η² =.77), and there were significant RTs differences between response buttons (F (5,115) = 54.97, p <.001, η² =.71). Polynomial contrasts showed the most significant component to be of quadratic shape (F (1,23) = , p <.001, η² =.82), though the cubic component was also significant (F (1,23) = 6.52, p <.05, η² =.22). In other words, more confident answers (i.e. 1 and 6 ) were faster than less confident ones, creating an inverted U-shaped function (Figure V-1 below). The interaction material type response (F (5,115) = 4.96, p <.01, η² =.18) was significant, indicating that RTs varied differently for faces and scenes across buttons. Figure V-1 indicates that RTs for central/unsure responses 3 and 4 differed more between faces and scenes than RTs for side/sure responses 1 and 6. Polynomial contrasts of the interaction showed that the quadratic component was the only significant one (F (1,23) = 24.53, p <.001, η² 68

69 =.52), confirming the pronounced difference between materials for centrally located responses. Figure V-1: Distribution of RTs for responses to faces and scenes across the six response buttons. The effect of fatigue was investigated by analyzing recollection and familiarity estimates (Yonelinas, Dobbins et al. 1996) 12 for changes between experimental blocks. No significant difference between blocks (F (3,69) = 0.10, p =.96, η² = 0) and no interaction between recollection / familiarity and blocks were observed (F (3,69) = 0.43, p =.73, η² =.02). These results indicate that no fatigue or carry over effects modified subjects performance during the experiment. The effect of morphing on behavioural performance was analyzed with a repeated measures ANOVA with the factors stimulus (face/scene), process (familiarity/recollection), and morphing (0%, 20%, 40%, 60%). A main effect of morphing was significant (F (3,69) = , p <.001, η² =.87) and polynomial contrasts revealed a linear performance change 12 A revised algorithm has been created in Matlab, available at: 69

70 (F (1,23) = , p <.001, η² =.92) 13. The interaction between material type and morphing was significant (F (3,69) = 7.41, p <.001, η² =.24), suggesting that the morphing procedure produced different effects on faces and scenes. A closer inspection of the interaction revealed that it was likely to arise from differences in recollection estimates, which fell rapidly for faces between 0% and 20%, but not for scenes (see Figure V-2 below). A separate 2 4 ANOVA with factors morphing (0%, 20%, 40%, 60%) and stimulus type (face/scene) on familiarity estimates confirmed this hypothesis producing no significant interaction between morphing and stimulus type (F (3,69) = 2.46, p =.07, η² =.10). Removing recollection also impacted the quadratic polynomial contrast, which was not significant any more (familiarity alone: F (1,23) = 1.89, p =.18, η² =.08). These results suggest that the relation between familiarity estimates and morphing steps is linear, while recollection exhibits a significant portion of non-linearity in performance shifts. Figure V-2: Familiarity and recollection estimates for each morphing level for faces (blue) and scenes (green) The persistence of recollection for scenes, but not for faces, may be related to the different morphing procedures. In contrast to faces, scenes contain the same objects shifted in one 13 The quadratic term was significant as well, but the effect size was considerably smaller (F(1,23) = 5.53, p <.05, η² =.19) 70

71 direction, suggesting the possibility that object identity helps high confidence scene recognition. In contrast with the behavioural pilot and fmri experiments, the present experiment suggested that morphing affects faces and scenes differently. Though this was shown to be due to recollection, not familiarity, two reasons could contribute to such discrepancy. First, the EEG experiment was conducted in a strongly controlled environment, whereas the behavioural experiment involved more distraction and spatial interaction in the laboratory. Second, the EEG experiment gave the subjects unlimited time for answering, whereas the fmri forced a maximum time of five seconds for answering. The quiet controlled environment together with a more relaxed experimental pace may provide the necessary conditions for a better scrutiny of scenes which leads to higher performance. This possibility evidences once more how different experimental contexts could undermine the replication of the results and occasionally lead to false conclusions (Geraci, McCabe et al. 2009; McCabe and Geraci 2009; Voss and Paller 2010). 4.2 Analysis of Event-Related Potentials (ERPs) In order to establish the correct temporal windows of ERP components reported in the literature, grand averages were screened to identify the components of interest. Starting from the P200, temporal windows were created by centring the window on the component peak and extending laterally until the start of the next ERP component. Following this procedure and the time windows suggested in published studies, the following early time windows were established: P200: ms P300: ms N400: ms 71

72 Late components (up to 1500 ms) have not been systematically considered in the literature and do not have a specific shape. Given that recollection-specific activity has been suggested to reside in the ms temporal window, we split the ms activity into these time windows: Late 1: ms Late 2: ms Analyses were performed by calculating the average voltage value for each of the above-listed time windows and by submitting the data to repeated measures ANOVAs. Each time window was analyzed separately, therefore each factor of interest was analyzed by five separate ANOVAs for the consecutive time windows. The factors analyzed were: Traditional old-new ERP difference, Objective Stimulus Identity (Morphing), and Subjective Response Traditional old-new ERP difference Previous studies have compared the difference between correctly recognized old stimuli (Hits) and correctly rejected new stimuli (Correct Rejections, CRs). Given that morphed stimuli cannot be considered completely old or completely new, the traditional difference was compared between Hits for old unmorphed stimuli and CRs for completely new stimuli. All intermediate morphing steps (20%, 40%, and 60%) were ignored. Hit 0% was considered an unmorphed stimulus answered 5 or 6, while CR was considered a new stimulus answered 1 or 2. Twenty-four unmorphed stimuli and 48 new stimuli were presented during the recognition phase for each material type, producing a maximal number of 24 and 48 segments, respectively. The resulting numbers of segments after artefact removal are given in Table V-1. Table V-1: Subjects average segment counts for Hits and CRs ± standard deviation 14 CONDITION FACE SCENE Hits 0% morphed 17 ± 4 16 ± 4 Correct Rejections 27 ± 9 39 ± 5 14 Segment counts rounded to the nearest integral number. 72

73 Figure V-3 shows the general topographical distribution of the voltage difference between Hits 0% and CRs for each time window. A four-way repeated measures ANOVA was conducted for every time window with the factors: Hits-CRs (2 levels), Stimulus (face/scene), Laterality (left/right) and Anterior/Posterior. We were specifically interested in the main effect of Hits-CRs and its interaction with the other factors. Figure V-3: Voltage maps of the amplitude difference Hits 0% - CRs. Significant differences between activity for Hit 0% and CR trials started to emerge in the N400 time window (~ ms). Faces and scenes did not differ significantly but a trend toward significance for the four-way interaction suggests slight differences in the topographic distribution in the first four time windows. P200 No main effect of Hits-CRs was observed in this time window (F (1,23) =.00, p =.98, η² =.00), neither any two-way (minimal p =.39, maximal η² =.03), or three-way (minimal p =.210, maximal η² =.07) interaction. The four-way interaction was below significance threshold, but a tendency toward significance was noticed (F (1,23) = 3.28, p =.08, η² =.13). P300 No significant main effect of Hits-CRs was observed (F (1,23) =.69, p =.42, η² =.03) and no two-way (minimal p =.41, maximal η² =.03) or three-way interaction was significant 73

74 (minimal p =.60, maximal η² =.01), although there was a slight tendency toward significance for Hits-CRs Laterality interaction (F (1,23) = 3.25, p =.09, η² =.12). The four-way interaction was again just below the threshold of significance (F (1,23) = 3.61, p =.07, η² =.14). Overall, no systematic pattern emerged for the difference Hits-CRs in this time window, but the voltage maps (Figure V-3 above) and the four-way interaction showed a slight difference in topography distributions between faces and scenes. N400 The main effect of Hits-CRs was significant for this time window (F (1,23) = 5.70, p <.05, η² =.20), but no significant two-way (minimal p =.13, maximal η² =.09) or three-way interaction (minimal p =.41, maximal η² =.03) influenced its interpretation. However, the four-way interaction was just below significance (F (1,23) 3.95, p =.06, η² =.15), suggesting that, despite a consistent Hits-CRs effect, its topographic distribution was inconsistent between faces and scenes. Figure V-3 shows a more anterior effect for faces compared to a more central effect for scenes, which may be responsible for the accumulation of evidence in favour of the four-way interaction. Late 1 The main effect of Hits-CRs was well above significance threshold (F (1,23) = 10.30, p <.005, η² =.31), but this result was challenged by other interactions. In particular, the two-way interaction Hits-CRs Anterior/Posterior was significant (F (1,23) = 9.35, p <.01, η² =.29), due to a stronger memory effect in anterior portions of the scalp. Additionally, a three-way interaction Hits-CRs Anterior/Posterior Laterality emerged (F (1,23) = 5.79, p <.05, η² =.25) indicating a bias of the memory effect on the anterior right portion of the scalp. The fourway interaction was not significant, though a tendency toward significance emerged (F (1,23) = 74

75 3.14, p =.09, η² =.12). The ERP differences in this time window showed a clearer picture of the Hits-CRs bias toward the anterior right electrodes, which appear to be slightly modulated by stimulus category. It should be noticed that voltage maps shown in Figure V-3 suggest a substantial difference between faces and scenes. Yet, this difference was not significant statistically (Hits-CR Stimulus: F (1,23) = 1.63, p =.22, η² =.07). Late 2 The Late 2 time window showed a significant main effect of Hits-CRs (F (1,23) = 4.36, p <.05, η² =.16) and a significant interaction Hits-CRs Anterior/Posterior (F (1,23) = 14.93, p <.001, η² =.39). The three-way interaction Hits-CRs Anterior/Posterior Laterality was significant, too (F (1,23) = 11.14, p <.005, η² =.33), suggesting that the memory effect was particularly high over anterior and right portions of the scalp. All other two-way (minimal p =.22, maximal η² =.06) and three-way (minimal p =.34, maximal η² =.04) interactions were not significant, neither was the four-way interaction (F (1,23) = 2.32, p =.14, η² =.09). This pattern of results closely resembles that of the previous time window, underlined by similar voltage maps in Figure V-3, too. Again, the interaction Hits-CRs Stimulus was not significant (F (1,23) =.75, p =.39, η² =.03), which would have otherwise confirmed the difference between faces and scenes visible in Figure V-3. Preliminary remarks about the traditional old-new ERP differences This analysis aimed at finding neural correlates of the traditional comparison between Hits and CRs. No difference was observed between Hits and CRs in P200 and P300 time windows. The first significant difference appeared at N400 and persisted until the end of the investigated time range. The topographic distribution of the old-new effect was centrally located for the N400 time window, but shifted in the anterior right portion of the scalp for the later temporal 75

76 windows. No significant difference was observed between faces and scenes, though the four way interaction showed a trend toward significance in the first four time windows. This pattern of results suggests that slight faces and scenes could have slightly different topographical distributions which could not be revealed. Nevertheless, the late ERP component follow a similar topographic distribution for faces and scenes over the right anterior frontal areas, suggesting category-independent neurophysiological processes Objective Stimulus Identity (Morphing) Twenty-four stimuli were presented for each morphing step and each material type during the recognition phase. Other 48 stimuli were completely new and were considered 100% morphed for this analysis. A maximal number of 24 and 48 segments could have been obtained for each morphed step or new stimulus, respectively. The average number of segments analyzed following artefact removal and preprocessing is given in Table V-2. Table V-2: Subjects average segment counts for the morphing scale ± standard deviation 15 CONDITION FACE SCENE 0% morphed (identical) 23 ± 1 23 ± 1 20% morphed 23 ± 1 23 ± 1 40% morphed 23 ± 1 23 ± 1 60% morphed 23 ± 1 23 ± 1 100% morphed (new) 45 ± 3 46 ± 2 For each time window, a four-way repeated measures ANOVA was run with the factors: Morphing (5 levels: 0%, 20%, 40%, 60%, NEW), Stimulus (face/scene), Laterality (left/right) and Anterior/Posterior. We were specifically interested in the main effect of Morphing and its interaction with the other factors. Given that the Morphing condition contained more than two levels of analysis, voltage maps were created both for the linear difference and for the U- 15 Segment counts rounded to the nearest integral number. 76

77 shaped quadratic difference of the morphing scale (Figure V-4). Grand averages are presented in Appendix A, page 107. P200 Figure V-4: Voltage differences for the morphing steps for the scalar difference (0% - 20% - 40% - 60% - 100%; upper rows) and for the quadratic U-shaped difference [(0% - 20% - 40%) + (100% - 60% - 40%); lower rows] The P200 time window showed no effect of Morphing (F (4,92) =.42, p =.79, η² =.02), no twoway interaction (minimal p =.09, maximal η² =.09), and no three-way interactions (minimal p =.48, maximal η² =.04). The four-way interaction was not significant, too (F (4,92) = 1.28, p =.28, η² =.05). A tendency toward significance was noticed for the two-way interaction 77