Neural correlates of recognition memory for emotional faces and scenes

Transcription

1 doi: /scan/nsq003 Neural correlates of recognition memory for emotional faces and scenes SCAN (2011) 6, 24 ^37 Michelle L. Keightley, 1,2,3,4,5 Kimberly S. Chiew, 6 John A. E. Anderson, 5,6 and Cheryl L. Grady 5,6,7 1 Department of Occupational Science and Occupational Therapy, 2 Graduate Department of Rehabilitation Science, University of Toronto, 3 Toronto Rehabilitation Institute, 4 Bloorview Kids Rehabilitation, 5 Department of Psychology, University of Toronto, 6 Rotman Research Institute at Baycrest, and 7 Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada We examined the influence of emotional valence and type of item to be remembered on brain activity during recognition, using faces and scenes. We used multivariate analyses of event-related fmri data to identify whole-brain patterns, or networks of activity. Participants demonstrated better recognition for scenes vs faces and for negative vs neutral and positive items. Activity was increased in extrastriate cortex and inferior frontal gyri for emotional scenes, relative to neutral scenes and all face types. Increased activity in these regions also was seen for negative faces relative to positive faces. Correct recognition of negative faces and scenes (hits vs correct rejections) was associated with increased activity in amygdala, hippocampus, extrastriate, frontal and parietal cortices. Activity specific to correctly recognized emotional faces, but not scenes, was found in sensorimotor areas and rostral prefrontal cortex. These results suggest that emotional valence and type of visual stimulus both modulate brain activity at recognition, and influence multiple networks mediating visual, memory and emotion processing. The contextual information in emotional scenes may facilitate memory via additional visual processing, whereas memory for emotional faces may rely more on cognitive control mediated by rostrolateral prefrontal regions. Keywords: emotional memory; faces; scenes; amygdala; hippocampus; fmri INTRODUCTION Emotion is thought to facilitate memory via arousal or enhanced attentional processes that occur during an emotional event (Ledoux, 2000; McGaugh, 2004; Anderson et al., 2006; Talmi et al., 2007). Evidence from patients with lesions of the amygdala implicate it as a key region underlying the enhanced memory effect, which lies primarily in modulating long-term memory for emotional stimuli (Adolphs et al., 1994, 1997, 2000; Anderson and Phelps, 2000; Siebert, Markowitsch, & Bartel, 2003). Functional neuroimaging experiments also have examined emotional memory, and found that increased amygdala activity during encoding enhances long-term memory for emotional relative to nonemotional material (Cahill et al., 1996; Canli et al., 2000). In addition, there is evidence that arousal-related activity in the amygdala, and other brain regions, during learning is chiefly responsible for memory facilitation (Cahill et al., 1996; Canli et al., 2000), although valence per se also appears to have a unique influence on memory (Dolcos et al., 2004; Kensinger and Corkin, 2004). Importantly, these studies have shown that enhanced memory for emotional Received 14 November 2009; Accepted 23 December 2009 Advance Access publication 1 March 2010 The authors thank the MRI technologists at Sunnybrook Health Science Centre for their assistance in carrying out this experiment. This work was supported by a grant from the Canadian Institutes of Health Research (MOP14036 to C.L.G.); and by infrastructure support from the Ontario Research Fund and the Canadian Foundation for Innovation. C.L.G. is also supported by the Canada Research Chairs program. Correspondence should be addressed to Michelle Keightley, PhD, CPsych, Department of Occupational Science and Occupational Therapy, Rehabilitation Sciences Building, University of Toronto, University Ave., 9th Floor Toronto, ON, M5G 1V7, Canada. michelle.keightley@utoronto.ca material is associated with activity during encoding in areas beyond the amygdala. These include the hippocampus and various regions in prefrontal cortex (Hamann et al., 1999; Dolcos et al., 2004; Kensinger and Corkin, 2004). Neuroimaging studies also have examined brain activity during emotional memory retrieval, primarily recognition (Buchanan, 2007). For example, recognition of emotional vs neutral scenes is associated with activity in a number of regions, including the amygdala, temporal pole, occipital cortex and hippocampus (Dolan et al., 2000; Sharot et al., 2004; Strange and Dolan, 2004; Dolcos et al., 2005; Smith et al., 2006). Increased amygdala activation also has been reported during retrieval of visual details for emotional vs neutral items (Kensinger and Schacter, 2007), and emotional vs neutral context (Maratos et al., 2001; Erk et al., 2003; Smith et al., 2004; Fenker et al., 2005; Kensinger and Schacter, 2005). Other brain regions showing increased activity for emotional vs neutral context include the hippocampus, orbitofrontal and inferior frontal regions, and parietal cortex (Dolan et al., 2000; Maratos et al., 2001; Erk et al., 2003; Smith et al., 2004; Fenker et al., 2005; Kensinger and Schacter, 2005). The magnitude of emotion associated with autobiographical memories also influences activity in the amygdala and frontal cortex (Daselaar et al., 2008; Kross et al., 2009) and hippocampus (Addis et al., 2004) when these memories are retrieved. Some evidence also exists that the specific emotional valence associated with an autobiographical memory influences the brain regions active during retrieval (Piefke et al., 2003). ß The Author (2010). Published by Oxford University Press. For Permissions, please journals.permissions@oup.com

2 Emotional recognition memory SCAN (2011) 25 Taken together, these studies suggest that the effect of emotion on brain activity at retrieval would be to increase activity in areas that process emotion, such as the amygdala, and areas that participate in memory for non-emotional material, e.g. the hippocampus and parietal cortex. That is, multiple brain networks might participate in retrieval of emotional items. Although there is still debate in the literature as to how to define a network (Horwitz, 2003), here we define it as a group of regions where activity co-varies in a similar fashion across a set of cognitive tasks. No study to date has attempted to identify such distributed patterns of activity across the whole brain reflecting the engagement of these multiple networks during emotional recognition. In addition, a number of studies of emotional memory have used scenes as stimuli (Sharot et al., 2004) and a few have used faces (Sergerie et al., 2005; Sterpenich et al., 2006), but there has been no direct comparison of activity to these different types of stimuli. It is important to compare faces and scenes directly because emotion and/or memory-related activity might differ depending on whether the item to be remembered is a single face or a more complex scene involving people in some sort of environmental context. Brain activity may be sensitive to changing task demands as well as stimulus features, and it is important to understand their differential influences on brain activity. We previously identified how emotion-related activity during encoding could be attenuated depending on the cognitive task employed (Keightley et al., 2003), which highlighted the effect that different attention-demanding cognitive effects can have on brain activity. Real-world information processing that requires emotional recognition rarely considers an emotional stimulus in isolation of its environmental context, yet to date, research continues to examine these features independently. Therefore, the purpose of the current experiment was to assess memory-related brain activity for emotional faces and scenes that contained people in emotional contexts, so that we could compare directly the brain activity for these two types of emotional stimuli. In this experiment, we identified patterns of activity across the whole brain, thus characterizing distributed sets of regions, or networks. We examined modulations of brain activity during recognition of emotional and neutral faces and scenes to determine how type of stimulus and emotional content affect brain activity during recognition. We expected that there would be similarities in brain activity for recognizing emotional scenes and faces reflecting the influence of emotional valence per se, as well as differences due to the presence of context in the scenes, but not the faces. Although it was difficult to predict the exact patterns of network activity that would be related to these influences, we expected that there would be more activity in extrastriate areas for scenes, relative to faces, as scenes are more visually complex and recognition of emotional scenes can elicit increased activity in the lingual gyrus (Taylor et al., 1998). Also, as scenes contain more contextual information than faces presented alone, we expected to see more activity in the hippocampus for scenes, given the role of this region in relational memory and binding of context (Eldridge et al., 2000; Ryan and Cohen, 2004; Fenker et al., 2005). Finally, based on the literature reviewed above, we hypothesized that activity in a distributed set of regions involved in emotional processing, including the amygdala, and those important for recognition memory per se (e.g. inferior parietal, prefrontal), would be increased for emotional stimuli, consistent with enhanced memory for these items. METHODS Participants Participants were 18 young adults (10 men, 8 women, mean age ¼ 26 5 years). Seventeen participants were right-handed and one was left-handed. We included the data from the one left-handed participant in the analyses because he was not an outlier on any measure that we assessed. All were screened to ensure there was no history of psychiatric, neurological or other medical illness, or a history of substance abuse that might compromise cognitive function. We also assessed emotional awareness using the 20-item Toronto Alexithymia Scale (TAS-20, Parker et al., 1999) and personality with the NEO-Five Factor Inventory (NEO-FFI, McCrae and Costa, 2003). All participants scored within normal limits (i.e. average T scores) on these measures. That is, no one had a score that would reflect an extreme emotion-related personality trait, such as extremely high or low Alexithymia (which reflects level of emotional awareness), or extremely high or low Neuroticism. Each participant gave informed consent in accordance with the Baycrest Research Ethics Board. Stimuli Stimuli were black and white photographs of faces and scenes, half of which were emotional (positive or negative) and half were neutral. Faces were obtained from the set developed by Matsumoto and Ekman (Biehl et al., 1997) as well as from other sources (websites, magazines). Scenes were obtained from the International Affective Picture System (Lang et al., 1995) and from other sources such as websites and magazines. All face stimuli were cropped so that only the face was visible (e.g. the hair was removed from the image). All scenes contained at least one person so that all stimuli would contain information relevant to people. Scenes displayed people in an emotionally positive, negative or neutral context (for examples of stimuli see Keightley et al., 2003). In the majority of scene stimuli, the faces of the people were shown, but in a third of the scenes, the faces were partially obscured or seen at a distance, and so were not readily visible. For all but a few scenes, the faces comprised only a small proportion of the scene (much <50%). Prior to scanning, participants saw a list of 72 faces and 72 scenes (in separate blocks) and were instructed to rate each in terms of valence, i.e. as positive, negative or

3 26 SCAN (2011) M. L. Keightley et al. neutral. For each stimulus category there were 18 positive, 18 negative and 36 neutral stimuli. Half the participants rated faces first, and half rated scenes first. Participants were not told that there would be a memory test. After the encoding phase, participants filled out the TAS-20 and the NEO-FFI, and then were placed in the scanner. During fmri scanning, participants carried out a series of recognition tasks for faces and scenes (with stimulus type assigned to different runs) in which they were instructed to indicate if each stimulus was old or new. For each stimulus type, all previously presented (i.e. old ) items for each valence were presented (18 positive/negative or 36 neutral) and there were 42 new positive, 42 new negative and 84 new neutral items (a total of 240 faces and 240 scenes). We included more neutral items because we expected memory to be lower for these items, and we wanted to have sufficient trials for the fmri analysis. Task order was balanced so that half the participants received faces first and half scenes first. The task order was further balanced so that of the half receiving faces first, half of those also rated faces first during encoding, while half rated scenes first. The same task order strategy was used for scene recognition. Old and new faces and scenes in each valence category were matched for valence intensity and arousal based on ratings obtained for these stimuli by a separate but comparable group of individuals (16 young adults, see Supplementary Table). Negative scenes were rated as somewhat more arousing than negative faces, but otherwise there were no differences due to stimulus type or old/new status in either valence intensity or arousal ratings. In addition, there were no differences in arousal ratings between positive and negative items. During scanning each stimulus was presented for 2.5 s with an inter-stimulus interval (a fixation cross) of 1.5 s. New and old stimuli for each valence, and null events (fixation crosses presented for 2.5 s) were presented randomly during 8 runs of 540 seconds each. Given the random presentation of null events, the effective inter-stimulus interval was 5.5 s on average. Image acquisition Anatomical and functional images were collected using a 3T GE scanner with a standard head coil. For each participant, we acquired a T1-weighted volumetric anatomical MRI (124 axial slices, 1.4 mm thick, FOV ¼ 22 cm). Brain activation was assessed using the blood oxygenation level-dependent (BOLD) effect. For functional imaging, 26, 5 mm thick axial slices were obtained utilizing a T2*-weighted pulse sequence with spiral in-out readout (TR ¼ 2000 ms, TE ¼ 30 ms, FOV ¼ 20, matrix). Stimuli were presented using fmri-compatible AVOTEC goggles mounted on the head coil. Responses were collected with the Rowland USB Response Box (RURB). Images were reconstructed and preprocessed utilizing the Analysis of Functional Neuroimages software (AFNI) (Cox, 1996) and Statistical Parametric Mapping (SPM99) software. The images were co-registered to account for head motion of the participants (head motion did not exceed 1.2 mm). Furthermore, the images were normalized to a standard space, using a linear transformation with sinc interpolation. Lastly, each participant s images were smoothed with an 8-mm Gaussian filter. The resulting voxel size after processing was mm. Statistical analyses Behavior data. Recognition accuracy (d 0 ) was analyzed using a repeated measures ANOVA. Median reaction times (RTs) for correct responses were analyzed with an ANOVA as well, with outlier RTs removed for each subject where applicable. Outliers were considered to be data points that were three standard deviations above or below the mean value. Where appropriate, post hoc comparisons were corrected for multiple comparisons using the Bonferroni correction. Functional neuroimaging data. For statistical analysis, we used a multivariate approach, Spatiotemporal Partial Least Squares, or PLS (McIntosh, 1999; McIntosh et al., 1996, 2004), in order to identify whole brain patterns of activity. PLS operates on the covariance between brain voxels and the experimental design to identify a new set of variables (so-called latent variables or LVs) that optimally relate the two sets of measurements. PLS is similar to other multivariate techniques, such as principal component analysis, in that contrasts across conditions or groups typically are not specified in advance; rather, the algorithm extracts LVs in order of the amount of covariance explained between conditions and brain activity (with the LV accounting for the most covariance extracted first). Each LV contains a spatial activity pattern depicting the brain regions that show the strongest relation to (e.g. are covariant with) the task contrast identified by the LV. Two analyses were carried out. The first examined the effects of stimulus type, and averaged all events for six trial types, i.e. stimulus type (face and scene) and valence (positive, negative, neutral), across correct responses (hits and correct rejections). Because we averaged across all correct responses (i.e. including old and new stimuli), all participants had at least 20 trials included for each condition (on average there were more than 30 trials for each condition in this analysis). The second analysis examined memory-related activity for faces and scenes by contrasting hits and correct rejections (i.e. items eliciting old vs new responses) for positive, negative and neutral items. For this analysis, there were fewer trials per event type, so to maximize power as best we could, we included only the 14 participants who had at least seven hits for each stimulus/valence condition (there were at least 22 trials for correct rejections in all conditions). We were not able to analyze misses and false alarms in this way due to even smaller numbers of events in these cells in some participants.

4 Emotional recognition memory SCAN (2011) 27 Each analysis included eight post-stimulus TRs for each event (i.e. 16 s) and activity at each time point was normalized to activity in the first TR of the trial. The first TR was designated TR0 and the following TRs 1 7. In event-related PLS, there is no baseline condition per se; rather, because data from all time points in each event are normalized to the first time point in the event, the changes in signal represent either increases or decreases of activity relative to the beginning of each trial. PLS as applied to event-related data results in a set of brain regions that are reliably related to the task contrasts for each TR on each LV, thus providing temporal as well as spatial information (McIntosh et al., 2004). Each brain voxel has a weight, known as a salience, which is proportional to the covariance of activity with the task contrast at each time point on each LV. Multiplying the BOLD signal value in each brain voxel for each subject by the salience for that voxel, and summing across all voxels, gives a brain score for each subject on a given LV. To characterize brain activity across the conditions, we plotted the mean brain score at each TR for each condition (referred to here as the temporal brain scores, which are analogous to a hemodynamic response function for a given region). The significance for each LV as a whole was determined by using a permutation test (McIntosh et al., 1996). As 500 permutations were used, the smallest P-value obtainable for each LV was P < In addition to the permutation test, a second and independent step was used to determine the reliability of the saliences for the brain voxels characterizing each pattern identified by the LVs. To do this, all saliences for each TR were submitted to a bootstrap estimation of the standard errors (Efron and Tibshirani, 1986). Reliability for each voxel was determined from the ratio of its salience value to the standard error for that voxel, and clusters of at least 20 contiguous voxels with a bootstrap ratio >3.0 were identified. A ratio of 3.0 approximates P < (Sampson et al., 1989). The local maximum for each cluster was defined as the voxel with a bootstrap ratio higher than any other voxel in a 2-cm cube centered on that voxel. Locations of these maxima are reported in terms of coordinates in MNI space. Confidence intervals (95%) for the mean brain scores (collapsed across all eight time points) in each condition also were calculated from the bootstrap, and the reliability of differences in activity between conditions was determined via a lack of overlap in these confidence intervals. RESULTS Behavioral data Table 1 reports the performance data on the memory tasks. Recognition accuracy (d 0 ) was analyzed using a two (faces and scenes) three (positive, negative and neutral) repeated measures ANOVA. This analysis revealed a significant main effect of stimulus type, F(1,17) ¼ 109.3, P < 0.001, with participants showing better recognition for scenes than for faces. Although face recognition was significantly poorer Table 1 Performance on recognition tasks Condition Positive Negative Neutral Faces Hits 0.54 (0.16) 0.54 (0.20) 0.56 (0.24) False alarms 0.40 (0.21) 0.32 (0.17) 0.46 (0.20) D prime 0.42 (0.46) 0.69 (0.45) 0.34 (0.41) Reaction time 1365 (195) 1392 (187) 1317 (180) Scenes Hits 0.70 (0.17) 0.62 (0.15) 0.61 (0.14) False alarms 0.27 (0.19) 0.18 (0.11) 0.20 (0.12) D prime 1.36 (0.60) 1.37 (0.67) 1.23 (0.52) Reaction time 1482 (193) 1492 (201) 1451 (194) RTs expressed in milliseconds. Values are means with standard deviations in parentheses. than scene recognition, face recognition accuracy for positive, negative and neutral faces was still significantly better than chance performance (assessed via t-tests, all P s < 0.01, corrected for multiple comparisons). There also was a significant main effect of valence, F(2,34) ¼ 4.1, P < Using post hoc contrasts, we found that recognition of positive stimuli did not differ from neutral stimuli, F < 1, but recognition of negative stimuli exceeded that for both positive and neutral stimuli, F(1,17) ¼ 6.0, P < The interaction of stimulus type valence was not significant, F(2,34) ¼ 1.4, P > A two (faces and scenes) three (positive, negative and neutral) repeated measures ANOVA on RTs for correct responses during recognition showed a significant main effect of stimulus type, F(1, 17) ¼ 16.8, P < 0.005, indicating faster RTs to faces than to scenes (Table 1). The main effect of valence also was significant, F(2,34) ¼ 4.2, P < RTs for recognition of positive stimuli did not differ from RTs to negative stimuli (F < 1), but recognition of neutral stimuli was faster than that for positive and negative stimuli, F(1,17) ¼ 6.7, P < The interaction of valence and stimulus type was not significant, F <1. fmri data: effects of stimulus type and valence To determine how stimulus type and valence influenced brain networks during successful recognition, we compared activity across all valence conditions for both scenes and faces (hits and correct rejections). The LV accounting for the most covariance in this contrast (P ¼ 0.002) appeared to differentiate scenes from faces, and to show an influence of valence (Figure 1A). Examination of non-overlapping confidence intervals (CIs, see Supplementary Figure 1a) confirmed this pattern and showed the following reliable differences: (i) activity for positive faces was lower than all other conditions; (ii) activity for neutral faces differed from all scene conditions; (iii) negative faces differed from emotional, but not neutral scenes; (iv) neutral scenes showed lower activity than positive scenes.

5 28 SCAN (2011) M. L. Keightley et al. Fig.1 Results of the analysis of stimulus and valence, LV1. (A) Temporal brain scores (averaged over participants) for each face and scene condition are plotted across the seven TRs in the analysis period. Brain scores are summary scores of activity across the entire brain and the graph shows how the pattern of activity across the brain is expressed over the 16 s period. These time courses may be thought of as whole-brain hemodynamic response functions. (B) Areas making a significant contribution to this pattern are shown on the average MRI of the 18 participants in MNI space (the same average MRI also is used in subsequent figures). Data shown are from TR2 and the Z level of each slice is shown beneath it. Activity in the red areas was greater for scenes and negative faces. Blue areas showed less activity for scenes and negative faces. Table 2 Modulations of activity by stimulus type and valence (LV1) Region BA X Y Z Ratio Peak (TR) R inferior frontal gyrus L inferior frontal gyrus R fusiform gyrus L fusiform gyrus L middle occipital gyrus R precuneus/spl R parahippocampal gyrus R inferior frontal gyrus R middle occipital gyrus R inferior occipital gyrus R inferior parietal lobule R superior temporal gyrus L superior frontal gyrus R, right; L, left; BA, Brodmann s area; Ratio, bootstrap ratio indicating reliability of each voxel (the ratio reported is for the peak TR). Peak (TR) is time point (TR) where the bootstrap ratio for each region is maximal. X (right/left): negative values are in the left hemisphere; Y (anterior/posterior): negative values are posterior to the zero point (located at the anterior commissure); Z (superior/inferior): negative values are inferior to the plane defined by the anterior and posterior commissures. Coordinates are in MNI space. inferior frontal gyrus bilaterally. Regions with the opposite pattern of activity, i.e. more activity for positive and neutral faces, included right inferior parietal and temporal regions, and left superior frontal cortex (Figure 1B and Table 2, negative ratios). In all of these latter regions, the difference was due to a decrease in activity for scenes and negative faces rather than to increased activity for positive/neutral faces. A second significant LV (P ¼ 0.02) identified areas that differentiated negative faces from the other conditions (Figure 2A). Examination of non-overlapping CIs confirmed a reliable difference between negative faces and all other conditions, as well as a difference between neutral and emotional scenes (see Supplementary Figure 1b). This LV identified several regions with lower activity for negative faces, including bilateral extrastriate cortex, right parahippocampal gyrus and hippocampus, superior temporal cortex and thalamus (Figure 2B and Table 3, positive ratios); some of these differences persisted into the late phase of the trial. More activity for negative faces was found in the cerebellum, left inferior temporal gyrus and left sensorimotor cortex (Figure 2B and Table 3, negative ratios). Increased activity for emotional scenes, relative to neutral scenes, and negative faces, relative to positive faces, was seen in bilateral fusiform gyri (extending into the parahippocampal gyrus), bilateral middle occipital gyri and precuneus (Figure 1B and Table 2, positive ratios). Greater activity for emotional scenes and negative faces also was seen in the fmri data: effects of memory success and valence To examine the neural correlates of memory for scenes and faces we contrasted activity for correct old vs new responses (hits vs correct rejections). There were three significant patterns showing a memory effect. The first (P ¼ 0.002,

6 Emotional recognition memory SCAN (2011) 29 Fig. 2 Results of the analysis of stimulus and valence, LV2. (A) Temporal brain scores (averaged over participants) for each face and scene condition are plotted across the analysis period. (B) Areas making a significant contribution to this pattern. Data shown are from TR2, TR3 and TR5, and the Z-level of each slice is shown beneath it. Activity across the TRs distinguished negative faces from the other conditions, with red areas showing less activity for negative faces and blue areas more activity for negative faces. Table 3 Modulations of activity by stimulus type and valence (LV2) Region BA X Y Z Ratio Peak (TR) L middle occipital gyrus R superior temporal gyrus R parahippocampal gyrus L superior occipital gyrus R thalamus L superior temporal gyrus R hippocampus R middle cingulate R superior temporal gyrus R caudate nucleus L superior frontal gyrus R cerebellum L inferior temporal gyrus L postcentral gyrus 2/ Coordinates are in MNI space. See Table 2 for abbreviations. Figure 3A) differentiated hits from correct rejections for both negative faces and scenes, and negative hits from positive and neutral hits (see Supplementary Figure 2); positive and neutral hits and correct rejections did not differ from each other. Increased activity for correctly recognized negative stimuli was seen in inferior frontal gyrus, fusiform gyrus and inferior parietal lobe bilaterally (Figure 3B and Table 4). Increased activity for negative hits also was seen in the right amygdala and hippocampus, and left thalamus. A second LV (P ¼ 0.004, Figure 4) showed increased activity for correctly recognized negative and positive faces, relative to correct rejections of emotional face stimuli (see Supplementary Figure 3). In addition, activity for hits to positive faces was reliably greater than that for hits to positive scenes. Activity was increased for hits to negative scenes, although across the entire trial the difference between hits and correct rejections for negative scenes was not reliable (see Supplementary Figure 3). More activity for correctly recognized emotional faces was seen in bilateral inferior/middle temporal regions and sensorimotor cortex, anterior and posterior cingulate, inferior/superior parietal lobes (more extensive in the left hemisphere), and rostral regions of prefrontal cortex (Figure 4B and Table 4, positive ratios). Less activity for emotional faces was seen in left temporal cortex (Table 4, negative ratios). Finally, a third pattern (P ¼ 0.04, Figure 5A) showed more activity for hits to positive faces and scenes, relative to correct rejections of positive items (see Supplementary Figure 4). No reliable differences in this pattern were found between hits and correct rejections for negative or neutral items. Areas with increased activity for correctly recognized positive stimuli included right amygdala, fusiform gyrus, caudate nucleus and inferior frontal gyrus (Figure 5 and Table 4, positive ratios). Decreased activity for positive hits was seen in left parahippocampal gyrus, superior frontal cortex and inferior parietal lobe (Table 4, negative ratios). Interestingly, the decreases in activity occurred somewhat earlier than the increases in activity. DISCUSSION The present study examined how brain activity during recognition memory varied as a function of stimulus type

7 30 SCAN (2011) M. L. Keightley et al. Fig. 3 Results of the contrast of hits vs correct rejections for faces and scenes LV1. (A) Temporal brain scores (averaged over participants) are shown for faces and scenes separately for clarity, although these scores come from the same analysis. (B) Regions contributing to the pattern (from TR3) are shown (images are from Z ¼ 24 to Z ¼þ36 in 4 mm steps). This LV differentiated negative hits from correct rejections and was characterized by increased activity in orange/red areas for correctly recognized negative faces and scenes. and emotional valence. To our knowledge, this represents the first study to compare directly these influences on the neural correlates of recognition memory. We found that scenes were remembered better than faces, consistent with previous research reporting a superiority of picture memory compared to other types of stimuli, such as words (Paivio, 1971; Park et al., 1983). We also found enhanced memory for negative items, which is in line with numerous other studies showing that negative emotion confers a larger enhancement of memory compared to positive emotion in young adults (Levine and Bluck, 1997; Storbeck and Clore, 2005; Kensinger and Schacter, 2006; Grady et al., 2007). The fmri data showed similarities, as well as differences between activity for emotional faces and scenes, consistent with the idea that the effects of emotional valence during recognition are modulated by the presence of contextual information in the scenes. In terms of the stimulus effect, we found greater activity in widespread areas of extrastriate cortex and bilateral inferior frontal regions for scenes, especially emotional scenes, relative to faces. Activity in these regions also was greater for negative faces compared to positive faces. In addition, negative faces were characterized by a unique pattern of activity, relative to other face and all scene conditions, consisting of more activity in sensorimotor cortex and cerebellum and less activity in hippocampus, thalamus and caudate. Second, in terms of memory effects, there was more activity for correctly recognized negative faces and scenes, relative to correct rejections, in amygdala and hippocampus, as well as bilateral extrastriate cortex, inferior frontal gyri and left inferior parietal cortex. Correctly recognized positive faces and scenes also were associated with increased activity in the amygdala, but this effect occurred relatively late in the trial. The major difference in memory effects due to stimulus type was a robust enhancement of activity for correctly recognized emotional faces, but not for emotional scenes, in superior parietal, cingulate and sensorimotor regions. These data indicate that emotional valence was processed somewhat differently depending on whether the viewed stimulus was a single face or a more complex scene involving people, and that

8 Emotional recognition memory SCAN (2011) 31 Table 4 Modulations of activity for hits vs correct rejections Region BA X Y Z Ratio Peak (TR) Negative Hits for Faces and Scenes (LV1) R inferior frontal gyrus R inferior/middle frontal gyrus L inferior frontal gyrus Medial frontal gyrus L caudate L posterior insula R middle temporal gyrus R fusiform gyrus L superior parietal lobe R precuneus/posterior cingulate L inferior frontal gyrus L thalamus R hippocampus R amygdala R inferior parietal lobe L inferior parietal lobe L fusiform gyrus L middle occipital gyrus L middle occipital gyrus R inferior occipital gyrus Positive and Negative Hits for Faces (LV2) R precentral/inferior frontal gyrus 6/ R superior frontal gyrus L superior frontal gyrus L precentral gyrus R paracentral lobule L middle frontal gyrus L middle frontal gyrus L postcentral gyrus L anterior cingulate gyrus L middle cingulate gyrus R middle temporal gyrus R inferior parietal lobe L inferior parietal lobe R insula L superior temporal gyrus L superior temporal gyrus Positive Hits for Faces and Scenes (LV3) R fusiform gyrus R superior frontal gyrus R inferior frontal gyrus R caudate/putamen R middle occipital gyrus L cuneus R fusiform gyrus R amygdala L middle cingulate gyrus L superior frontal gyrus L middle frontal gyrus 8/ L parahippocampal gyrus R inferior temporal gyrus 21/ L angular gyrus L angular gyrus Coordinates are in MNI space. See Table 2 for abbreviations. negative valence had the largest influence on brain activity when participants were correctly recognizing both faces and scenes, consistent with better memory for negative items. The influence of stimulus type on brain activity It is interesting that we did not find any simple main effects of stimulus type on brain activity when participants were viewing scenes and faces. Instead, we found a pattern that was expressed the most during viewing of emotional scenes, to a medium degree during viewing of neutral scenes and negative faces, and least during viewing of positive and neutral faces. This pattern consisted of broadly enhanced extrastriate activity, including the ventral processing streams that mediate the processing of faces and scenes, respectively (Ungerleider and Haxby, 1994; Aguirre et al., 1998; Epstein and Kanwisher, 1998). This extrastriate activity may reflect greater visual processing or analysis of scenes and negative faces, which enhances the ability to recognize them. This finding is in line with previous work (Taylor et al., 1998; Fenker et al., 2005) suggesting that emotional modulation of activity in those visual areas representing specific stimulus properties occurs during recognition and may facilitate correct recognition judgments. Enhanced extrastriate activity may also be related to the fact that responses to scenes overall were slower than those to faces, and RTs to negative faces also tended to be slower, perhaps reflecting longer visual processing times or greater capture of attentional processes (Talmi et al., 2008). There also was more activity for emotional scenes and negative faces in ventrolateral prefrontal cortex bilaterally. These ventral prefrontal regions are similar in location to the regions reported in numerous studies of episodic and working memory (Owen et al., 2005; Dove et al., 2006), and are thought to provide top-down attentional control and represent stimulus salience (Dove et al., 2006; Seeley et al., 2007). In our study, ventral prefrontal activity also may be related to its anatomical connections with the ventral visual stream (Ungerleider et al., 1989), as this frontal activity was seen in conjunction with activation of widely distributed regions of ventral extrastriate cortex. In addition to increased brain activity, emotional scenes and negative faces also showed decreased activity in some areas. Regions with reduced activity during cognitive tasks are sometimes referred to as default mode areas that represent monitoring of both internal states and the external environment (Raichle et al., 2001). Given that the areas with decreased activity seen here are similar to regions considered to be part of the default mode network (Fox et al., 2005; Toro et al., 2008), and that default deactivations are related to task demands (McKiernan et al., 2003; Persson et al., 2007), our result may indicate greater cognitive engagement during processing of emotional scenes and negative faces. This interpretation would be consistent with the slower RTs shown by our participants when processing these stimuli, as noted above. Taking into account the increases and decreases of activity seen in this pattern, our results suggest that there is more engagement of the ventral visual pathway (both extrastriate and ventral frontal cortex) and more extensive suppression of some default mode

9 32 SCAN (2011) M. L. Keightley et al. Fig. 4 Results of the contrast of hits vs correct rejections for faces and scenes LV2. (A) Temporal brain scores (averaged over participants) are shown for faces and scenes separately for clarity, although these scores come from the same analysis. (B) Regions contributing to the pattern (from TR 2) are shown (images are from Z ¼ 20 to Z ¼þ40 in 4 mm steps). This LV-differentiated emotional hits for faces from correct rejections and was characterized by increased activity in orange/red areas for correctly recognized emotional faces. activity when carrying out recognition tasks on emotional scenes and negative faces, both of which may contribute to carrying out enhanced visual analysis. A second pattern of activity from the analysis of stimulus effects specifically differentiated scenes from negative faces. Scenes elicited more activity, compared to negative faces, in bilateral anterior temporal lobes, the thalamus, caudate, parahippocampal gyrus and hippocampus. Hippocampal activity for scenes was expected, considering that scenes contain contextual information and the hippocampus mediates relational processing in memory (Cohen et al., 1999; Henke et al., 1999; Bunge et al., 2004; Moses and Ryan, 2006). In addition, the anterior regions of the temporal lobes are involved in semantic processing (Mummery et al., 1999; Graham et al., 2003), and the thalamus participates in episodic memory (Harding et al., 2000; Van Der Werf et al., 2003; Kishiyama et al., 2005; Burianova and Grady, 2007). The two patterns revealed by this analysis, taken together, suggest that recognition of scenes is mediated in the brain by increased activity in multiple brain networks that involve visual processing (mediated by extrastriate and ventral prefrontal regions), semantic processing (via anterior temporal cortex) and memory processing (via the hippocampus and thalamus). The cooperative activity in these networks may facilitate correct recognition judgments for scenes, both old and new judgments. In contrast, negative faces elicited increased activity in left sensorimotor regions, relative to scenes. This sensorimotor activity is consistent with the idea that identification of facial emotional expressions is facilitated by generating somatosensory representations of how that person might feel (Adolphs et al., 2000). Our results suggest that this simulation process is engaged to a greater degree when faces with negative expressions are viewed. Also, notable in the pattern that differentiated negative faces from scenes, was the finding that a number of regions showed their most reliable contribution to this pattern near the end of the trial, including the hippocampus and caudate. Since these later time points were influenced by activity returning to baseline, or even decreased below baseline, this finding suggests that viewing of negative faces was

10 Emotional recognition memory SCAN (2011) 33 Fig. 5 Results of the contrast of hits vs correct rejections for faces and scenes LV3. (A) Temporal brain scores (averaged over participants) are shown for faces and scenes separately for clarity, although these scores come from the same analysis. (B) Regions contributing to the pattern are shown. Data shown are from TR3, TR4 and TR5, and the Z-level of each slice is shown beneath it. This LV differentiated positive hits from correct rejections and was characterized by increased activity in orange/red areas and decreased activity in blue areas for correctly recognized positive faces and scenes. accompanied by a faster fall-off of activity or sustained reductions of activity in these areas. The functional significance of activity reductions is not always clear, but reduced activity in some medial temporal regions has been associated with familiarity responses to visual stimuli (Henson et al., 2003; Montaldi et al., 2006), suggesting a role for familiarity in negative face recognition. The influence of emotion during memory The strongest effects of emotion on brain activity were seen for negative valence when correctly recognizing old faces and scenes. Increased activity in both the right amygdala and hippocampus characterized correct recognition of negative items. Amygdala activity is in line with a large number of studies showing activation of this region for negatively valenced stimuli both at encoding and recognition (Cahill et al., 1996; Canli et al., 2000; Kensinger and Schacter, 2007), and a particular affinity of this region for negative faces (Adolphs et al., 1999; Morris et al., 1996). Hippocampal activity is found with recollection and strong memories (Eldridge et al., 2000; Maguire et al., 2000; Ranganath et al., 2004; Dolcos et al., 2005; Daselaar et al., 2006), consistent with the better memory that we found for negative items. We also found that these medial temporal regions co-activated with visual areas (fusiform, middle occipital), consistent with a recent model of face representation in the brain (Gobbini and Haxby, 2007). In addition, recognizing negative faces and scenes was characterized by increased activity in frontal and parietal cortices bilaterally. These areas have shown retrieval-related effects in a number of experiments (Cabeza and Nyberg, 2000; Rugg et al., 2002; Wagner et al., 2005; Stevens and Grady, 2007; Schaefer et al., 2009), and left parietal activity in particular has been associated with successful memory retrieval (Wheeler and Buckner, 2004; Wagner et al., 2005; Cabeza et al., 2008). The frontal and parietal activity seen here indicates that activity in these memory-related areas is particularly enhanced by negative valence during recognition. Thus, memory for negative visual stimuli, both faces and scenes that contain people, may be supported by an integrated enhancement of multiple distributed brain networks that mediate sensory, emotional and memory processes. In addition, this enhancement of activity for hits to negative faces and scenes, relative to correct rejections, does not appear to

11 34 SCAN (2011) M. L. Keightley et al. be driven by arousal, as old and new stimuli were equated for arousal, but rather to negative emotion per se (for a similar result see Talmi et al., 2007). Positive emotion also influenced activity for correctly recognized scenes and faces, and was associated with increased activity in right amygdala and fusiform, similar to that seen for negative items. However, this activity was not associated with widespread co-activation of other areas of cortex, but more limited activity in occipital and right frontal cortices. Amygdala activity for positive hits also occurred later (10 12 s post-stimulus onset) than for negative hits (6 8 s post-stimulus onset), suggesting that this activity occurs after some cognitive processing of the positive stimulus has taken place. This would be consistent with the idea that the amygdala responds somewhat preferentially and rapidly to negative stimuli (Morris et al., 1996; Anderson et al., 2003), but this type of delayed response to positive stimuli has not been shown previously during recognition. In addition, activity in left inferior parietal cortex was decreased for positive hits. As this region has been associated with high confidence in memory retrieval (for a review, see Cabeza et al., 2008), this would suggest that recognition of positive stimuli is associated with less confidence, although we did not measure this directly in our study. The one activity pattern that showed a difference in memory effects for faces and scenes was the one that distinguished correct recognition of old emotional faces from correct rejections of new emotional faces, but did not differentiate hits and correct rejections of scenes. This pattern included increased activity in rostral prefrontal cortex and anterior inferior parietal lobes, as well as sensorimotor regions. It has been suggested that the rostral prefrontal cortex represents abstract concepts and mediates higher order control of task demands (Gilbert et al., 2006; Badre and D Esposito, 2007; Christoff et al., 2009). Together with the anterior inferior parietal lobes it may form part of a cognitive control network that facilitates the operations of other brain networks (Vincent et al., 2008). Activity in these regions for recognition of emotional faces, in conjunction with sensorimotor activity that may reflect simulation (suggested above), indicates that emotional face recognition may require simulation and control processes to a greater degree than recognition of emotional scenes. It may be that the additional contextual information in scenes facilitates recognition without the need for these control processes. Limitations of the study There are several limitations of our study that should be noted. Because our choice of scene stimuli was motivated to maintain social relevance for comparison to the face stimuli, all of the scenes contained people with more or less visible faces. Therefore, the results of this experiment may not generalize to scene stimuli that do not contain people (e.g. a negatively valenced picture of a snake or gun). Nevertheless, they do indicate that there is an influence of context on emotional face processing during memory, as has been shown for labeling face emotions (Aviezer et al., 2008). Another limitation is that the power for determining the effects of memory, i.e. in the analysis of hits vs correct rejections, was somewhat limited. Fortunately, we were able to identify reliable effects of emotional valence on successful memory, as well as differences in brain activity between face and scene memory, despite this limitation. That is, despite there being fewer hits in general than correct rejections, we nevertheless were able to show reliable differences in brain activity patterns between positive and negative hits and their corresponding correct rejections. However, it is possible that with greater power we might have been able to find additional differences between face and scene memory, beyond the difference in activity associated with correctly recognized emotional items that we observed. CONCLUSIONS The present study builds upon previous research investigating the relationship between emotional recognition and regional brain activity. Our findings suggest that activity at recognition can be differentially influenced according to stimulus and valence. Successful recognition of both negative and positive items recruits the amygdala, although the influence of positive emotion is seen somewhat later in time. In addition, recognition of negative faces and scenes is associated with the engagement of multiple cognitive processes and brain networks during recognition, including those thought to be involved in visual and memory processes in general. The contextual information in emotional scenes may facilitate memory via additional visual processing, whereas memory for emotional faces may rely more on cognitive control mediated by rostrolateral prefrontal regions and simulation of others emotions. These data add to our knowledge about the role of distributed brain systems in successful recognition of emotionally valenced complex visual material. Understanding the intricate nature of the interplay among various brain regions under differing cognitive/emotional conditions has clinical significance with respect to mood and memory disorders where the nature of these networks and their interactions may be compromised. REFERENCES Addis, D.R., Moscovitch, M., Crawley, A.P., McAndrews, M.P. (2004). Recollective qualities modulate hippocampal activation during autobiographical memory retrieval. Hippocampus, 14, Adolphs, R., Cahill, L., Schul, R., Babinsky, R. (1997). Impaired declarative memory for emotional material following bilateral amygdala damage in humans. Learning and Memory, 4, Adolphs, R., Damasio, H., Tranel, D., Cooper, G., Damasio, A.R. (2000). A role for somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping. Journal of Neuroscience, 20,