Event-related potentials and EEG oscillations link attention and serial order effects in working memory

Transcription

1 1 Event-related potentials and EEG oscillations link attention and serial order effects in working memory Yigal Agam and Robert Sekuler Volen Center for Complex Systems Brandeis University Waltham, MA 02454, USA Abstract. Working memory, the focus of intensive behavioral, neural, and computational study, is integral to a wide range of human intellectual achievements(1, 2). One of working memory's notable attributes, its limited capacity, has been linked to limitations on attentional resources(2-13). We asked whether attention might also govern another key attribute of working memory, the serial ordering of the reliability with which successive items are recalled. We recorded the electroencephalogram (EEG) of human subjects who were viewing and memorizing novel motion sequences, each comprising five successivelypresented segments, which were later to be imitated(14). From the EEG, we extracted two known electrophysiological hallmarks of attention, the gain modulation of event-related potentials (ERPs)(15-18) and the power of high-frequency (>20 Hz) oscillations(19-21). Results with both measures suggest a declining level of attention as each additional segment is encoded into working memory: Both ERP amplitude and the power in highfrequency, beta and gamma, bands of the EEG grew weaker with each additional motion segment that was viewed and remembered for later imitation. Such sequential, ordered changes were absent when subjects viewed the exact same stimuli, but did not have to encode the stimuli for later retrieval. Thus, it seems that serial ordering of items strength in working memory arises from an uneven distribution of attentional resources, and that primacy effects, the advantage enjoyed by first-in items, may be due to a greater amount of attention available for encoding the earliest elements in stimulus sequences.

2 2 We recorded scalp EEG from human adults who performed a sequential imitation task, known to generate strong effects of serial order(14). Fig. 1a shows a schematic diagram of the experimental paradigm. On each trial, subjects viewed a moving disc, whose trajectory comprised five randomly oriented, connected linear segments. Then, several seconds later, subjects used a stylus and a graphic tablet to reproduce the trajectory from working memory (see also supplementary video clips). Performance was defined by the accuracy with which the orientation of each segment in the original trajectory was reproduced. Behaviorally, the results demonstrated a pronounced primacy effect and a modest, one-item recency effect (Fig. 1b), confirming previous findings with this paradigm(14). In a control condition, meant to distinguish between sensory- and memory-related effects, subjects viewed identical stimulus trajectories, but did not have to encode the trajectory s multiple segments. Instead, they were asked to detect changes in the speed of the disc or matches between the direction of motion to a predefined directional cue. Fig. 1c shows event-related potentials (ERPs) at five midline electrode locations, timelocked to the onset of the first motion segment of the disc, and encompassing the entire 3.75-second period in which subjects viewed the moving disc. The figure shows that every motion segment elicits a distinct response. As our primary goal was to examine changes across successive segments, we superimposed traces corresponding to individual segments, and evaluated the ERPs and power spectra for differences between segments in the memory and control conditions. We chose to omit the first segment from further analysis for two reasons: First, in both the memory and control conditions, the ERP evoked by the first segment was visibly different in form from the ERPs evoked by all other segments (Fig. 1c), probably due to the transition from a stationary to a moving disc. Second, oscillatory activity in the alpha band is known to be attenuated by increased alertness and expectancy, and this transient attenuation process has been shown to last approximately the duration of one motion segment in our experiment(22). As we expected, then, the onset of disc movement elicited a strong decrease in alpha power between the first a second segment in both experimental conditions, a decrease which would have occluded other, memoryrelated effects. Consequently, our analyses included only the second to fifth segments. Analysis of segment-by-segment ERPs focused on epochs of interest identified by visual inspection of the data; We chose epochs that encompassed the most notable features of the ERPs (Fig. 1c): The early, brief occipital peak at around 200 ms after the onset of each segment, the large frontal peak at 300 ms, the smaller peak following it at central electrodes, and the slower, final phase ( ms). The ERPs within each epoch were then averaged and subjected to a repeated measures ANOVA at each of 27 electrode sites. A random permutation procedure corrected for multiple comparisons by providing an empirical estimate of the appropriate significance threshold for each individual comparison(23-25). Fig. 2 demonstrates the segment-by-segment differences between ERPs in the memory and control conditions. The memory condition produced widely-distributed, significant differences between successive segments of the trajectory. Specifically, as each successive segment was seen, ERP amplitude decreased. No significant differences whatsoever

3 3 appeared in the perceptual control condition. Therefore, differences among segments in the memory condition are not caused by visual processes alone, but arise from the requirement to encode and maintain successive components of the seen trajectory in working memory. Oscillatory power in high-frequency bands of the EEG demonstrated effects analogous to those seen in ERP. To compare oscillatory power associated with successive segments, we divided the frequency spectrum into four standard frequency bands (theta, alpha, beta and gamma)(23, 24), and computed the average power across frequencies within each band. We then performed a repeated measures ANOVA (corrected for multiple comparisons) for every frequency band at every electrode site. As with the ERPs, significant differences between segments were widespread across the scalp, and were expressed exclusively in the memory condition. As Fig. 3 shows, segmentwise differences in the EEG reveal a loss of power in the high frequencies (beta and gamma bands) as a function of serial position. This is consistent with a previous study using free recall of word lists, which found stronger gamma activity aggregated across the presentation of the early words in the list, compared to the words in the middle(23). As mentioned earlier, enhanced ERP amplitude and high-frequency oscillations in the EEG are electrophysiological hallmarks of selective attention. Classic ERP studies of attention have shown that attention modulates evoked responses multiplicatively, i.e., by modulating the ERP s gain: When some stimulus is attended rather than ignored, the same ERP components are present, but their relative amplitude is increased(15-18). Similarly, oscillatory high-frequency (>20 Hz) power is correlated with visual attention: EEG(19, 20), intracranial EEG(26), MEG(21) and monkey single-unit(27) experiments have shown that stimuli drive stronger beta and gamma activity when they are attended compared to when they are ignored. The serial ordering we observed in both measures, then, probably reflects the allocation of diminishing amounts of attentional resources as successive segments are encoded into memory. Why do the early motion segments enjoy such an encoding advantage? A likely possibility is that the effort needed to hold past information in memory impairs the processing of presently perceived material. One theoretical framework which could help explain the link between short-term storage and the encoding of incoming information is the Attentionbased Rehearsal(2, 8, 10, 12) acocunt of working memory, whereby covert attention mediates the maintenance of visuospatial information in working memory, much like the proposed role of verbal articulation in keeping information in the phonological store(28). According to this view, the same selective attention mechanism that modulates incoming sensory information also refreshes the representations of information in working memory, which themselves depend upon the same neural structures that mediate perception(2). Empirically, this account has been supported mainly in the context of working memory for spatial locations: Several studies, measuring ERP amplitude(4, 7) and hemodynamic changes(5, 9), showed that when spatial locations are held in memory, stimuli presented at those locations (or the same hemifield) elicit greater responses than when these locations are not memorized, just as stimuli at attended locations are followed by an enhanced response compared to stimuli at those locations when they are ignored. The parallels

4 4 between the effects have been taken as an indication of a common mechanism supporting spatial attention and rehearsal in spatial working memory. In this light, the reduction we observed in both ERP amplitude and high-frequency power between successive motion segments can be interpreted as less efficient processing of incoming visual information as more and more attention has to be directed at keeping previous information alive in working memory. Our study shows that attention influences types of working memory other than memory for spatial locations: In our procedure, subjects encoded each successive direction of motion(14); by measuring the response to each successive motion segment as it was being encoded, we provide direct evidence linking attention to working memory. Whereas the studies mentioned above measured responses to behaviorally irrelevant visual stimuli while spatial locations were already held in memory, we show that the response evoked by the very presentation of the to-be-remembered stimulus is down-modulated if prior, taskrelevant information already occupies working memory. The serial ordering in the results suggests a compelling, attention-based explanation of the primacy effect; Attention decrement has long been thought to constitute a possible explanation of earlier encoded items superior recall(29), but empirical support for that speculation has been scarce(23), particularly with serial recall. We show that earlier segments may benefit from the greater attentional resources available to encode them, leading to more reliable later recall. Much more, however, is yet to be understood about the neural and cognitive mechanisms underlying working memory. We have focused on primacy effects, i.e., superior recall of early information, but not on how serial order per se is coded, itself a formidable theoretical and experimental challenge(14, 30-33). Our electrophysiological results do not explain the one-item recency effect seen in our behavioral data and in most other serial recall tasks: The electrophysiological trends continue to develop between the fourth and the fifth segments, even though the fidelity of imitation levels off (Fig. 1b). However, we and others(14, 32) have previously proposed that our recency result may be due to reduced interference and fewer order errors during retrieval, so it is plausible that it would not be reflected in the activity during encoding. Using two different EEG indicia of attention, we demonstrate that variation in attention levels likely underlies not only capacity limits but also primacy effects in working memory. Although an unequivocal, causal relationship between the two processes is yet to be established, future research could undoubtedly benefit from a more unified approach to working memory and attention, rather than conceptualizing the two functions as independent modules. Materials and Methods Subjects and Procedure. Seventeen right-handed, neurologically healthy subjects (8 male, 9 female, age range 18-26) participated after providing written informed consent. Each of the observers performed between 200 and 240 trials of the imitation task (memory

5 5 condition). As this task has been described in detail previously(14), only a brief outline is given here: Each stimulus model comprised a set of five directed motion segments, each 1.5 degrees visual angle long, whose orientations varied randomly. The angular difference between the orientations of each pair of adjacent segments was between 30 and 150 degrees. The moving disc took 525 ms to traverse each segment; Successive segments were separated by a 225 ms pause, in which the disc remained stationary. Subjects had to knit together the directed components in their mind s eye, and hold the trajectory in memory for 3.75 seconds. They then tried to reproduce it with a stylus on a graphic tablet (Wacom, Vancouver, WA). The accuracy of the imitation was assessed by an automatic segmentation algorithm, which decomposed the imitation into individual segments, and then compared the orientations of reproduced segments to segment orientations in the stimulus model. All seventeen subjects performed a control task, in which stimuli were visually identical to those in the imitation task. For nine subjects, on one third of 240 trials (which were subsequently excluded from the analysis), the speed of the disc through one motion segment changed relative to the normal speed by up to 25%. Subjects had to indicate whether such a speed change happened on a given trial. The eight remaining subjects performed a different control task: At the onset of every one of 100 trials, an arrow pointed to a random direction. Subjects were asked to count on how many, out of fifteen segments, the direction of motion matched the direction of the arrow. Only the first five segments, which were visually identical to the stimuli in the memory condition, were included in the analysis. Electrophysiological Recordings and Analysis. We recorded from 129 electrode sites at 250 Hz using an Electrical Geodesics (Eugene, OR) system. Data were cleaned of bad channels, re-referenced to the grand average and reduced to a standard 27-electrode montage using BESA (MEGIS Software GmbH, Munich), then analyzed using MATLAB (The Mathworks, Natick, MA). Blink artifacts were eliminated by rejecting trials in which the difference between the maximum and minimum voltage at any electrode exceeded 100 V. Data were notch-filtered at 60 Hz and high-pass filtered at 1 Hz. For ERP analysis, an additional low-pass filter was applied at 30 Hz. For EEG analysis, power spectra were computed using a 512-point (Hann-windowed) Fourier transform on each segment.. Oscillatory power was integrated across evenly-spaced points within each frequency band (0.1 octave frequency steps), and the total power in each band was log-transform to correct for non-normality due to lower bounding. Correcting for multiple comparisons. To correct for the inflated risk of false positive statistical inferences due to multiple comparisons, we used a random permutation procedure, which provides an empirical estimate, given existing correlations in the data, of the Type I error rate(23-25). To determine the appropriate significance threshold for the ANOVA tests, we randomized the order of segments within each trial in every condition, so that correlations between nearby electrodes and time points within a segment remained intact, but the effect of serial position was abolished. This process was repeated 1,000 times, to produce a set of 1,000 random permutations. We then averaged the trials in each permutation and performed the same statistical tests as we would perform on the real, unshuffled data. The number of significant differences seen for a particular significance

6 6 threshold, divided by 1,000, provides the probability of a Type I error. Finally, we chose a significance threshold so that the number of permutations that produced any false positives did not exceed 50, i.e., a global 0.05 probability of a Type I error. Acknowledgements This study was supported by in part by National Science Foundation grant SBE and National Institute of Health grant R01MH

7 7 References 1. Jonides, J. & Nee, D. E. (2006) Neuroscience 139, Jonides, J., Lacey, S. C. & Nee, D. E. (2005) Curr Direct Psychol Sci 14, Cowan, N. (2001) Behav Brain Sci 24, ; discussion Awh, E., Anllo-Vento, L. & Hillyard, S. A. (2000) J Cogn Neurosci 12, Awh, E., Jonides, J., Smith, E. E., Buxton, R. B., Frank, L. R., Love, T., Wong, E. C. & Gmeindl, L. (1999) Psychol Sci 10, Engle, R. W., Kane, M. J. & Tuholski, S. W. (1999) in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds. Miyake, A. & Shah, P. (Cambridge University Press, Cambridge). 7. Jha, A. P. (2002) Brain Res Cogn Brain Res 15, Postle, B. R. (2006) Neuroscience 139, Postle, B. R., Awh, E., Jonides, J., Smith, E. E. & D'Esposito, M. (2004) Brain Res Cogn Brain Res 20, Smyth, M. M. & Scholey, K. A. (1994) Mem Cognit 22, Theeuwes, J., Olivers, C. N. & Chizk, C. L. (2005) Psychol Sci 16, Awh, E., Vogel, E. K. & Oh, S. H. (2006) Neuroscience 139, Cowan, N. (1999) in Models of Working Memory: Mechanisms of Active Maintenance and Executive Control, eds. Miyake, A. & Shah, P. (Cambridge University Press, Cambridge), pp Agam, Y., Bullock, D. & Sekuler, R. (2005) J Neurophysiol 94, Van Voorhis, S. & Hillyard, S. A. (1977) Percept Psychopys 22, Hillyard, S. A., Hink, R. F., Schwent, V. L. & Picton, T. W. (1973) Science 182, Hillyard, S. A. & Muente, T. F. (1984) Percept Psychophys 36, Hillyard, S. A., Vogel, E. K. & Luck, S. J. (1998) Philos Trans R Soc Lond B Biol Sci 353, Gruber, T., Muller, M. M., Keil, A. & Elbert, T. (1999) Clin Neurophysiol 110, Muller, M. M., Gruber, T. & Keil, A. (2000) Int J Psychophysiol 38, Sokolov, A., Lutzenberger, W., Pavlova, M., Preissl, H., Braun, C. & Birbaumer, N. (1999) Neuroreport 10, Klimesch, W. (1996) Int J Psychophysiol 24, Sederberg, P. B., Gauthier, L. V., Terushkin, V., Miller, J. F., Barnathan, J. A. & Kahana, M. J. (In press) Neuroimage. 24. Hwang, G., Jacobs, J., Geller, A., Danker, J., Sekuler, R. & Kahana, M. J. (2005) Behav Brain Funct 1, Ludbrook, J. (1998) Clin Exp Pharmacol Physiol 25, Tallon-Baudry, C., Bertrand, O., Henaff, M. A., Isnard, J. & Fischer, C. (2005) Cereb Cortex 15, Womelsdorf, T., Fries, P., Mitra, P. P. & Desimone, R. (2006) Nature 439, Baddeley, A. D. & Hitch, G. (1974) in The Psychology of Learning and Motivation, ed. Bower, G. A. (Academic Press, New York), Vol. 8, pp

8 29. Atkinson, R. C. & Shiffrin, R. M. (1968) in The Psychology of Learning and Motivation, eds. Spence, K. W. & Spence, J. T. (Academic Press, New York), Vol. 2, pp Farrell, S. & Lewandowsky, S. (2004) J Mem Lang 51, Henson, R. N. (1998) Cognit Psychol 36, Page, M. P. & Norris, D. (1998) Psychol Rev 105, Smyth, M. M. & Scholey, K. A. (1996) Q J Exp Psychol A 49, Loftus, G. R. & Masson, M. E. J. (1994) Psychon Bull Rev 1,

9 Figure 1: Stimuli and results. a, Example of a memory trial. Note that while viewing and reproducing the stimulus, the disc did not leave a trace, so subjects only saw its instantaneous position; the dashed lines are for illustration purposes. b, Behavioral results. The average segment orientation error is plotted against segment serial position. Error bars are within-subject SEM(34). c, ERPs at the midline electrode locations for the memory condition, time-locked to the onset of the disc s motion. Vertical dashed lines denote the onset of individual motion segments. The 3.75 seconds for which ERPs are displayed correspond to the period during which subjects viewed the stimulus models. 9

10 10 Figure 2: Modulation of ERP amplitude. a-d, ERPs time-locked to the onset of segments 2-5 at several midline electrode locations. The left and middle panels show ERPs for the memory and control conditions, respectively. Each color corresponds to a different serial position, and the vertical dashed lines represent averaging epoch boundaries. Colored stripes above each panel denote significance levels for each time epoch (P values above 0.05 are unmarked). An asterisk indicates a P value exceeding the significance threshold after correction for multiple comparisons. Note the ordering of ERP amplitudes in the memory condition: Amplitude is systematically largest for segment 2 and smallest for segment 5. The right panel shows the mean values across each time epoch for the memory and control conditions (left and right sides of each pair, respectively), normalized relative to segment 2 in each condition and time epoch. The vertical scale is expanded for clarity. Light gray shading marks epochs in which there was a statistically significant ordering of ERPs to segments in the memory condition.

11 Figure 3: Modulation of high-frequency oscillations. a-f, Power in four standard frequency bands in the memory and control conditions at various electrode locations. Each group of bars corresponds to a different frequency band, and shows the power in segments 3, 4 and 5 (left, middle and right bars, respectively) relative to the power in segment 2. Significance markers are similar to Fig. 2. Note the significant decrease in the high frequencies during memory trials, but not in control trials. 11