Frontal oscillatory dynamics predict feedback learning and action adjustment van de Vijver, I.; Ridderinkhof, K.R.; Cohen, M.S.

Transcription

1 UvA-DARE (Digital Academic Repository) Frontal oscillatory dynamics predict feedback learning and action adjustment van de Vijver, I.; Ridderinkhof, K.R.; Cohen, M.S. Published in: Journal of Cognitive Neuroscience DOI: /jocn_a_00110 Link to publication Citation for published version (APA): van de Vijver, I., Ridderinkhof, K. R., & Cohen, M. X. (2011). Frontal oscillatory dynamics predict feedback learning and action adjustment. Journal of Cognitive Neuroscience, 23(12), DOI: /jocn_a_00110 General rights It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons). Disclaimer/Complaints regulations If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. UvA-DARE is a service provided by the library of the University of Amsterdam ( Download date: 09 Mar 2019

2 Frontal Oscillatory Dynamics Predict Feedback Learning and Action Adjustment Irene van de Vijver 1, K. Richard Ridderinkhof 1, and Michael X Cohen 1,2 Abstract Frontal oscillatory dynamics in the theta (4 8 Hz) and beta (20 30 Hz) frequency bands have been implicated in cognitive control processes. Here we investigated the changes in coordinated activity within and between frontal brain areas during feedback-based response learning. In a time estimation task, participants learned to press a button after specific, randomly selected time intervals ( msec) using the feedback after each button press (correct, too fast, too slow). Consistent with previous findings, theta-band activity over medial frontal scalp sites (presumably reflecting medial frontal cortex activity) was stronger after negative feedback, whereas beta-band activity was stronger after positive feedback. Theta-band power predicted learning only after negative feedback, and beta-band power predicted learning after positive and negative feedback. Furthermore, negative feedback increased theta-band intersite phase synchrony (a millisecond resolution measure of functional connectivity) among right lateral prefrontal, medial frontal, and sensorimotor sites. These results demonstrate the importance of frontal theta- and beta-band oscillations and intersite communication in the realization of reinforcement learning. INTRODUCTION 1 University of Amsterdam, 2 University of Arizona Medial frontal cortex (MFC) monitors ongoing actions and their outcomes to adjust behavior adaptively (Ridderinkhof, Ullsperger, Crone, & Nieuwenhuis, 2004). More specifically, MFC activity has been related to conflict monitoring (Botvinick, Cohen, & Carter, 2004), cost benefit analyses (Rushworth, Behrens, Rudebeck, & Walton, 2007; Kennerley, Walton, Behrens, Buckley, & Rushworth, 2006), and the evaluation of outcome history and modification of action values and selection (Nieuwenhuis, Holroyd, Mol, & Coles, 2004; Gehring & Willoughby, 2002; Holroyd & Coles, 2002). Although often considered separate and conflicting, these different functions may be part of a general action optimization learning function (Botvinick, 2007). When an action or outcome is suboptimal and MFC signals a need for adjustment, this also appears to lead to an increase in cognitive control, possibly via the additional recruitment of lateral pfc (Kerns, 2006; Kerns et al., 2004; Ridderinkhof et al., 2004). Lateral pfc is assumed to adjust higher-level decision-making strategies to changing contexts and demands and to integrate information over time (Lee & Seo, 2007; Tanaka et al., 2006; McClure, Laibson, Loewenstein, & Cohen, 2004). Although the general knowledge about the roles of MFC and lateral pfc in action outcome learning or feedbackbased learning seems considerable, the question remains how these areas communicate the necessity and implementation of cognitive control and behavioral adjustment. Changes in oscillatory activity within brain regions and synchrony of oscillations between brain regions may be important measures of underlying mechanisms (Akam & Kullmann, 2010; Wang, Spencer, Fellous, & Sejnowski, 2010; Gregoriou, Gotts, Zhou, & Desimone, 2009; Womelsdorf et al., 2007; Fries, 2005). Several characteristics of brain oscillations constitute differing and valuable sources of information about the dynamics within and between brain areas: (1) Oscillatory power (amplitude) represents the activation magnitude in a brain area, (2) intertrial phase coherence signifies the timing of oscillations within a brain area over trials, and (3) intersite phase synchrony is the similarity in timing of oscillations in different brain areas. Studies into the role of brain oscillations in conflict and reward situations demonstrate the relevance of oscillations in the theta band (4 8 Hz) for processing errors and losses. MFC theta power and intertrial phase coherence increase more after errors or negative performance feedback than after successful trials or positive feedback (Cohen, 2011; Cavanagh, Frank, Klein, & Allen, 2010; Cavanagh, Cohen, & Allen, 2009; Christie & Tata, 2009; Marco-Pallares et al., 2008, 2009; Cohen, Ridderinkhof, Haupt, Elger, & Fell, 2008; Cohen, Elger, & Ranganath, 2007; Trujillo & Allen, 2007; Luu, Tucker, & Makeig, 2004; Luu & Tucker, 2001). Lateral pfc theta power also increases after errors compared with successful trials (Cavanagh et al., 2009, 2010; Luu et al., 2004). Moreover, theta-band intersite phase synchrony between MFC and lateral pfc increases after errors (Cavanagh et al., 2009, 2011 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 23:12, pp

3 2010), suggesting increased communication between these areas. Frontal oscillatory theta-band activity also predicts posterror slowing and postcorrect speeding (Cavanagh et al., 2009, 2010) and model-derived prediction errors (Cavanagh et al., 2010). Thus, when feedback implies an unfavorable outcome and a possible need to adjust behavior, theta-band oscillations in the frontal network appear to be involved in communicating that message and implementing both task-specific and task-general adjustments. Whereas theta-band oscillations are involved in processing negative feedback, positive feedback can induce an increase in higher beta or lower gamma power (20 30 Hz; Marco-Pallares et al., 2008, 2009; Cohen et al., 2007). This beta-band activity is often but not always observed, whereas the theta-band oscillations elicited by negative feedback are more consistently observed. The increase in beta-band activity appears to be induced specifically when positive feedback contains valuable information for the subject, for example, a monetary win (Marco-Pallares et al., 2008, 2009) or a correct response in a learning situation (Cohen et al., 2007). This is in line with the recent suggestion by Engel and Fries that beta-band oscillations signal the tendency to maintain the status quo (Engel & Fries, 2010), such that increases in beta-band activity promote the existing motor or cognitive set through endogenous top down influence. If MFC and pfc oscillations are indeed underlying action outcome learning, they should be related not only to alterations of attention and predictions but also to subsequent behavioral changes. Only a small number of studies investigated the influence of activations in MFC and lateral pfc on next-trial performance (Chase, Swainson, Durham, Benham, & Cools, 2011; Cavanagh et al., 2010; Philiastides, Biele, Vavatzanidis, Kazzer, & Heekeren, 2010; Hester, Barre, Murphy, Silk, & Mattingley, 2008; Cohen & Ranganath, 2007; Yasuda, Sato, Miyawaki, Kumano, & Kuboki, 2004). Although most of these studies suggest that the level of MFC activity scales with the prediction error and predicts behavioral adjustment, a direct link between frontal oscillations and behavioral learning has not yet been reported. Therefore, in the current study, we investigated how oscillatory dynamics in the theta and beta bands over MFC and lateral pfc (measured through surface EEG) were related to response learning and behavioral adjustment. Consistent with the observation that the level of MFC hemodynamic activity after errors is larger when the next trial is successful compared with when the next trial is another error (Hester et al., 2008), we hypothesized that larger increases in both MFC theta-band power and intertrial phase coherence after errors would also predict learning. Moreover, we expected larger increases in lateral pfc theta-band power and intertrial phase coherence and in theta-band oscillatory intersite phase synchrony between MFC and lateral pfc after errors as an implementation of cognitive control (Cavanagh et al., 2009, 2010). Because beta-band power was hypothesized to signal the importance of the continuation of the current situation, we expected increases in beta power to predict learning from positive feedback. Results from EEG and MEG source-imaging studies suggest that the theta-band oscillations measured at medial frontal scalp sites might originate from MFC (i.e., anterior cingulate or supplemental motor area; Cohen, 2011; Keil, Weisz, Paul-Jordanov, & Wienbruch, 2010; Christie & Tata, 2009; Luu et al., 2004; Luu & Tucker, 2001). This is confirmed by intracranial measurements from the human MFC (Cohen et al., 2008; Wang, Ulbert, Schomer, Marinkovic, & Halgren, 2005). In the current study, participants performed a time estimation task in which they had to use feedback to learn the correct time delay after which to press a button. This enabled us to study learning between consecutive trials. Besides MFC and lateral pfc, we added the contralateral cortical motor system as an area of interest because it implements the selected action and may be involved in processing feedback signals (Cohen & Ranganath, 2007). Motor control over the hand muscles induces changes in beta- and gamma-band oscillations (15 80 Hz) within the sensorimotor system and in cortico-muscular coupling (Schoffelen, Oostenveld, & Fries, 2005; Kilner, Baker, Salenius, Hari, & Lemon, 2000; Mima & Hallett, 1999). Cortico-muscular coupling with frontal brain areas varies with the required kind of control (Babiloni et al., 2008). Therefore, we explored whether cortico-muscular intersite phase synchrony might also reflect feedback learning. For comparability with previous studies, the feedback-related negativity (FRN) was also analyzed, an ERP elicited by valenced performance feedback (Holroyd & Coles, 2002; Miltner, Braun, & Coles, 1997). METHODS Subjects Twenty-four right-handed first-year psychology students at the University of Amsterdam (11 men; mean age = 21.7 years, range = years) participated in this experiment for course credits. They gave informed consent before participation. All procedures were executed in compliance with relevant laws and institutional guidelines and approved by the local ethics committee. Task Participants performed a reinforcement learning task (Figure 1A), in which they had to learn when to press a button with the right thumb. The computer randomly selected a target RT between 300 and 2000 msec. Participants were instructed to learn to press the button at this target time by trial-and-error using feedback. A dot was presented until the response was made (maximum 2500 msec). Then 500 msec after the button press, feedback van de Vijver, Ridderinkhof, and Cohen 4107

4 Figure 1. Overview of the task and behavioral performance. (A) Sequence of events in an example trial. (B) Behavioral performance of one subject in the first 100 trials. Gray bars represent target windows; black dots represent actual RTs. (C) Average behavioral performance represented as the adjustment of the difference between the actual response time and the target window over time. was presented for 1000 msec. A varying intertrial interval of msec separated trials. Responses in a 400 msec window surrounding the target time (±200 msec) were taken as correct. When a new target time was selected, this window decreased by 5% if participants were correct more than 70% of the last 15 trials and increased by 5% if they were incorrect more than 70% of the last 15 trials. Feedback consisted of a smiley face (correct), an upward arrow (speed up), or a downward arrow (slow down), and the number of points gained or lost. On correct trials, participants were rewarded two points. This amount increased one point per consecutive correct trial, with a maximum of six points. On incorrect trials participants lost one point. If participants did not respond while the dot was on screen, a message was presented that no response was detected. Two counters determined when a new target time was selected. A success counter increased on every correct trial and decreased on every incorrect trial but never went below zero. An error counter increased on incorrect trials if the success counter was at zero and decreased on correct trials if its value was larger than two. If one of the counters reached five, a new target time was selected. Effectively, this meant that participants had to be either correct or incorrect at least five times for a new target time to be selected. During the task subjects were not notified when the target time changed. New target times were selected at random with the exception that consecutive target windows could not overlap. We refer to a series of trials with the same target time as a block. We categorized blocks as learned or non-learned depending on whether the success or error counter reached five. The task consisted of 600 trials. Figure 1B displays behavior from one subject on the first 100 trials to illustrate the dynamics of the task. Self-paced rest breaks were given between target times and at least 50 trials apart. Participants performed 50 trials of training. The entire experiment, including time to set up, lasted approximately 2hr. Behavioral Analyses and Experimental Conditions The last trial of each block was excluded because feedback on the following trial was based on a new target time. Differences in average RT adjustments after each possible feedback (positive, speed up or slow down) were tested with a one-way ANOVA using Statistical Package for the Social Sciences for Windows (Version 15.0; SPSS, Inc., Chicago, IL) software. To confirm that learning only took place in the blocks that were classified as learned blocks, for each block we separated the trials into four bins according to the order of presentation (first quarter of trials in first bin, second quarter of trials in second bin, etc.). For each bin, we computed the difference between the average RT of the trials in the bin and the target window (zero if the RT was within the window). The average difference scores per bin were subjected to a 4 (Bin: 1 4) 2 (Learning: yes, no) repeated measures ANOVA. Because we expected participants to need some time (and thus, trials) to learn a new target time, we predicted that the difference between average RTs and target window in the learned blocks but not in the nonlearned blocks would decrease over time (bins). Only trials from learned blocks were included in ERP and oscillation analyses. These trials were separated into four conditions: incorrect trials followed by incorrect trials, incorrect trials followed by correct trials, correct trials followed by incorrect trials, and correct trials followed by correct trials. Signal Recording and Processing Electrophysiological data were recorded with a sampling rate of 2048 Hz from 64 scalp electrodes (EEG), 2 ocular electrodes (VEOG), 2 electrodes on the right abductor pollicis brevis (EMG), and 2 reference electrodes on the earlobes, using a BioSemi Active Two system. Data were preprocessed with the EEGLAB toolbox (Delorme & Makeig, 2004) in Matlab (The MathWorks, Natick, 4108 Journal of Cognitive Neuroscience Volume 23, Number 12

5 MA). Data were downsampled to 512 Hz, and a 1.0-Hz high-pass filter was applied. All data were re-referenced to the average of the earlobe electrodes. Cue-locked epochs of 1500 msec precue until 4200 msec postcue were extracted (epoch end at least 1200 msec after feedback onset). Baseline correction was applied by aligning the time series to the average amplitude in a msec precue interval. Trials with movement or other artifacts in the EEG signal were manually removed. After independent component analysis, blink and noise components were manually selected and removed. The resulting data were converted to current source density (CSD; Kayser & Tenke, 2006) to increase spatial selectivity and minimize volume conduction. CSD acts as a spatial high-pass filter, significantly improves topographical localization of surface EEG (Srinivasan, Winter, Ding, & Nunez, 2007), and is appropriate for intersite phase synchronization analyses (Cavanagh et al., 2009). Nonetheless, it does not solve the inverse problem or allow us to unambiguously determine the origin of the topographical dynamics. After exclusion of nonlearned blocks and trials with blinks orartifacts,thenumberof trials (SD) per condition was (19.24) for incorrect followed by incorrect trials, (14.87) for incorrect followed by correct trials, (10.08) for correct followed by incorrect trials, and (32.78) for correct followed by correct trials. The smallest number of trials in any condition in any subject was 27. ERP Analyses To investigate the FRN, the data per epoch were responselocked (this also means feedback-locking, because the response-feedback delay was a fixed 500 msec). ERP values per condition from channel FCz were averaged over a msec postfeedback window and entered into a 2 (Current success: correct, incorrect) 2 (Next-trial success: correct, incorrect) repeated measures ANOVA. The time window selectionwasbasedonexistingliterature(holroyd& Coles, 2002; Miltner et al., 1997) and visual inspection of FCz ERPs. Greenhouse Geisser corrections were applied. Oscillation Analyses Time Frequency Decomposition Time frequency characteristics were extracted from the data using custom-written Matlab routines (Cavanagh et al., 2009; Cohen et al., 2008). To study the oscillatory dynamics of the electrophysiological data, single-trial cue-locked data were convolved with a family of complex Morlet wavelets, defined as Gaussian-windowed complex sine waves: e i2πft e t2 =ð2σ 2Þ ; where i is the complex operator, t is time, f is frequency, whichincreasedfrom2to40hzin40logarithmically spaced steps, and σ defines the width of each frequency band and was set to 4/(2πf ). From the resulting complex signals of every epoch, we extracted estimates of power and phase angles twice: once from the cue-locked complex signal and once from the response-locked complex signal. Power is defined as the modulus of the resulting complex signal Z(t) (power time series: p(t) =real[z(t)] 2 + imag[z(t)] 2 ). Because power decreases with increasing frequencies (power law), response-locked power was normalized with a conversion to decibel scale (10 log 10 [power/baseline], with a cue-locked 500 to 200 msec precue baseline). Phase angle is defined as Φ t = arctan (imag[z(t)]/real[z(t)]). Phase angle values were used to compute intertrial phase coherence and intersite phase synchrony. Intertrial phase coherence represents the extent to which phase angles take on similar values across trials at each time frequency point. It is defined as 1 n Xn t¼1 e iϕ t where n is the number of time frequency points, t is trial, i is the complex operator, and Φ is the phase angle at each time point. Intersite phase synchrony represents the extent to which phase angle differences between electrodes are consistent over trials at each time frequency point. It is defined as 1 n Xn t¼1 ; e iðϕ jt ϕ kt Þ where j and k are two electrodes. Baseline intertrial phase coherence and intersite phase synchrony (cuelocked 500 to 200 msec precue condition-specific baseline) were subtracted from the response-locked intertrial phase coherence and intersite phase synchrony. Because any tonic differences in signal between the conditions (e.g., any differences caused by differing numbers of trials) would also influence the baselines of the different conditions, by subtracting a condition-specific baseline removed tonic differences between conditions and were able to focus on phasic differences. Statistical analyses were performed on response-locked data. Final response-locked epochs ranged from 500 msec preresponse (1000 msec prefeedback) to 1500 msec postresponse (1000 msec postfeedback). Statistical Analyses To assess the role of theta-band oscillations in learning, statistical analyses were performed on theta-band power (activation magnitude) and intertrial phase coherence (activation timing) at medial frontal (FCz), left lateral prefrontal (F5), and right lateral prefrontal (F6) electrodes. ; van de Vijver, Ridderinkhof, and Cohen 4109

6 Figure 2. Topographical voltage maps (±50 msec around response) demonstrate a relatively posterior peak during the response. For both measures, we averaged the signal per condition in the theta band (4 8 Hz). Participant- and conditionspecific peaks were detected in a msec postfeedback window. This peak-finding procedure allows for individual-specific patterns of brain activity. Average peak values from a 100-msec window (±50 msec from the peak), and peak latencies were entered into 2 (Current success) 2 (Next-trial success) repeated measures ANOVAs. On the basis of visual inspection of the time frequency plots of FCz, we selected a band of Hz to assess the role of beta power. Because sharp peaks were not observed in this frequency band, based on visual inspection of average beta power we entered the averaged activity at and msec postfeedback windows into ANOVAs to investigate both immediate and sustained effects. Because all responses were made with the right hand and topographical ERP maps (Figure 2) demonstrated a relatively posterior peak during the response, we selected electrode CP3 to represent response-related sensorimotor activity (for analyses with C3, see supplementary material). Theta-band intersite phase synchrony in a msec postfeedback window was investigated between medial frontal and left lateral prefrontal (FCz F5), medial frontal and right lateral prefrontal (FCz F6), medial frontal and sensorimotor (FCz CP3), left lateral prefrontal and sensorimotor (F5 CP3), and right lateral prefrontal and sensorimotor electrodes (F6 CP3). Intersite phase synchrony between CP3 and muscle activity was examined in a msec periresponse window. Peaks were detected and statistics performed as described for theta power and intertrial phase coherence. RESULTS Behavioral Results: Learning Postfeedback RT adjustments differed significantly with feedback content (F(1, 23) = , p <.001): A downward arrow led to an average RT increase of 360 msec (SD = 68), an upward arrow led to a decrease of 427 msec (SD =83),andafterpositivefeedback,therewasaminor although significant increase of 22 msec (SD =25;t(23) = 4.40, p <.001). Of the average of 55.9 blocks (SD =6.1), more than half (M =35.3,SD = 6.1) were learned blocks. The difference-score between RTs and target windows of the blocks (Figure 1C) demonstrated a main effect of Learning (F(1, 23) = 18.68, p <.001), a main effect of Bin (F(1, 23) = , p <.001), and an interaction between Learning and Bin (F(1, 23) = , p <.001). Whereas in the nonlearned blocks improvement was limited to the first bins and the target window was not learned consistently, in the learned blocks the difference between RTs and target windows approached zero over time (and thus, bins) and learning was achieved. ERP Analyses Topographical CSD maps and condition-specific line plots (Figure 3) confirmed the presence of an FRN between 200 and 300 msec postfeedback, signifying an effect of Current success (FCz: F(1, 23) = 78.29, p <.001). There were no significant effect of Next-trial success (F(1, 23) = 1.73, p =.200) and no significant interaction effect (F(1, 23) = 0.09, p =.772) in this window. In a later time window of msec postfeedback, there remained a significant main effect of Current success (F(1, 23) = 88.36, p <.001), although the direction of the effect reversed. In the later time window no effect of Next-trial success (F(1, 23) = 2.13, p =.158) or interaction effect (F(1, 23) = 2.49, p =.129) were found. Thus, time domain average ERPs did not predict learning. Oscillations at Medial Frontal Scalp Sites Learning from Negative Feedback and Activation Magnitude: Theta Power FCz theta power increased following feedback in all conditions, particularly between 200 and 500 msec (Figure 4). This increase was larger for incorrect than correct trials (F(1, 23) = 39.34, p <.001; Figure 4A) and was larger for next-correct than next-incorrect trials (F(1, 23) = 5.53, p =.028; Figure 4C). A significant interaction effect indicated that the power increase predicted learning only on current incorrect trials (F(1, 23) = 15.76, p =.001). Follow-up t tests confirmed that the increase in 4110 Journal of Cognitive Neuroscience Volume 23, Number 12

7 theta power predicted next-trial accuracy during current incorrect trials (t(23) = 5.20, p <.001) but not during current correct trials (t(23) = 1.38, p =.180). The latency of the peak of the increase in theta power also differentiated between current incorrect and correct trials: Power peaked earlier for correct than incorrect trials (F(1, 23) = 9.59, p =.005; Figure 4C). There was no effect of Next-trial success on peak latency (F(1, 23) = 0.34, p =.567) and no interaction effect (F(1, 23) = 3.16, p =.089). Thus, postfeedback FCz theta power is stronger and peaks later for current incorrect compared with correct trials and predicts learning from negative feedback. Learning from Positive Feedback and Beta Power FCz beta power msec postfeedback was significantly larger for next-correct trials than next-incorrect trials (F(1, 23) = 8.00, p =.010; Figure 5). There was no influence of Current success in this time window (F(1, 23) = 0.08, p =.780) and no interaction effect (F(1, 23) = 0.02, p =.881). However, beta power msec postfeedback demonstrated effects of both current- and next-trial success: Beta power was larger for current correct than incorrect trials (F(1, 23) = 13.18, p =.001) and larger for next-correct than next-incorrect trials (F(1, 23) = 7.19, p =.013). Again no interaction effect was found (F(1, 23) = 0.02, p =.887). Thus, both early and sustained postfeedback beta power were predictive of next-trial accuracy, but current success only affected the sustained increase. Activation Timing: Theta Intertrial Phase Coherence Theta-band intertrial phase coherence at FCz was observed after feedback (Figure 6) and was larger on incorrect than correct trials (F(1, 23) = 12.72, p <.001; Figure 6A). There was a main effect of Next-trial success such that intertrial Figure 3. Postfeedback FCz ERPs varied with success on the current trial. (A) Topographical CSD maps of the differences between correct and incorrect trials. (B) Difference between current and next correct and incorrect trials at FCz. van de Vijver, Ridderinkhof, and Cohen 4111

8 Figure 4. Medial frontal theta power was predictive of learning from negative feedback. (A) Time frequency representations of postfeedback power at FCz on incorrect and correct trials. (B) Topographical plots of theta power (4 8 Hz) msec postfeedback on incorrect and correct trials. (C) Average postfeedback FCz theta power (4 8 Hz) for current and next incorrect and correct trials. phase coherence was larger for next-correct than nextincorrect trials (F(1, 23) = 13.42, p =.001; Figure 6C), but no interaction with Current success (F(1, 23) = 0.23, p =.640) as observed for power. The latency of theta intertrial phase coherence differed between current correct and incorrect trials as well: The increase peaked later for incorrect than correct trials (F(1, 23) = 25.67, p <.001). The latency did not predict next-trial behavior (F(1, 23) = 0.61, p =.445), and the interaction between current and next-trial success was only at trend level (F(1, 23) = 2.98, p =.098). Thus, theta intertrial phase coherence was larger and peaked later for current incorrect than correct trials, and a larger increase in intertrial phase coherence predicted learning. In summary, FCz theta-band oscillations reflected current valence and predicted behavioral adjustments based on negative feedback. Beta-band oscillations were more pronounced after positive feedback and predicted learning as well. Oscillations at Lateral Prefrontal Scalp Sites Feedback Processing and Prefrontal Theta Power Theta power at electrodes F5 and F6 increased after feedback (Figure 7). Theta power at F6 (right side of the head) was significantly larger for incorrect than correct trials (F(1, 23) = 6.09, p =.021; Figure 7A) but did not predict nexttrial performance (Next trial: F(1, 23) = 1.63, p =.214; Current trial Next trial: F(1, 23) = 0.70, p =.413).The latency of the increase in theta power at F6 did not demonstrate significant main effects of Current success (F(1, 23) = 0.77, p =.390) or Next trial success (F(1, 23) = 0.69, p =.417) and no interaction effect (F(1, 23) = 2.14, p =.157). Although the same effect of Current success seems visible in theta power at F5 (left side of the head; Figure 7B), it did not significantly differ across conditions (Current trial: F(1, 23) = 0.91, p =.350; Next trial: F(1, 23) = 0.06, p =.817; Current trial Next trial: F(1, 23) = 0.00, p =.988) or latency (Current trial: F(1, 23) = 0.22, p =.641; Next trial: F(1, 23) = 0.13, p =.725; Current trial Next trial: F(1, 23) = 0.05, p =.833). Differences in Prefrontal Theta Intertrial Phase Coherence Theta-band intertrial phase coherence was observed at F5 and F6 after feedback (Figure 8). Intertrial phase coherence at F5 predicted next-trial accuracy (F(1, 23) = 6.17, p =.021; Figure 8B) such that intertrial phase coherence was higher on next-correct compared with next-incorrect trials. Intertrial phase coherence at F5 did not reflect current success (F(1, 23) = 2.18, p =.153;Figure 8A) and the interaction between current and next-trial success was only marginally significant (F(1, 23) = 3.20, p =.087).Nomain effects of Current success (F(1, 23) = 0.93, p =.346) or Next-trial success (F(1, 23) = 0.30, p =.588) on the latency of intertrial phase coherence at F5 were observed. There was a significant interaction effect between Current and Next-trial success (F(1, 23) = 6.54, p =.018). Intertrial phase coherence at F6 reflected current success: Intertrial phase coherence was larger on incorrect than correct trials (F(1, 23) = 7.76, p =.011; Figure 8C). There were no significant relation with Next-trial behavior (F(1, 23) = 0.00, p =.972; Figure 8D) and no interaction effect (F(1, 23) = 2.43, p =.132). The latency of intertrial phase coherence at F6 demonstrated a marginally 4112 Journal of Cognitive Neuroscience Volume 23, Number 12

9 significant effect of Next-trial success (F(1, 23) = 3.97, p =.058) with a later peak for next-correct than next-incorrect trials. The latency did not reflect Current accuracy (F(1, 23) = 0.30, p =.590), and there was no interaction effect (F(1, 23) = 0.08, p =.778). In summary, theta-band oscillatory activity at electrode F6 only responded strongly to current valence. Oscillations in the same band at F5 seemed more related to next-trial success, but the effects in this area were less coherent. Communication between Areas: Theta-band Intersite Phase Synchrony Increasing Control after Errors: Intersite Phase Synchrony Theta-band intersite phase synchrony increased significantly for current incorrect compared with correct trials between FCz and F6 (F(1, 23) = 4.75, p =.040; Figure 9A), between F6 and CP3 (F(1, 23) = 13.18, p =.001; Figure 9B), and between FCz and CP3, although the latter effect was only marginally significant (F(1, 23) = 3.08, p =.093; Figure 9C). No significant differences in intersite phase synchronization were observed between FCz and F5 (F(1, 23) = 1.48, p =.236) or between F5 and CP3 (F(1, 23) = 0.16, p =.694). None of the connections between brain areas predicted behavior on the next trial, and there were no effects on the latency of the intersite phase synchrony. This overall pattern of results was similar when using C3 (see supplementary material). Thus, a larger increase in oscillatory intersite phase synchrony after current incorrect trials was seen only in a network of medial frontal, right lateral prefrontal and left sensorimotor sites. A single cortical generator projecting to multiple scalp sites could inflate intersite phase synchrony between separate electrodes. If the synchrony dynamics between separate electrodes were caused by a single generator, the absolutedifferenceinphaseanglebetweenthesignals measured at the separate locations should be either zero (electrodes measure same side of the dipole) or π (electrodes measure opposite sides of the dipole). Therefore, we also computed the absolute phase angle differences between FCz and F6, between FCz and CP3, and between F6 and CP3 at the time and frequency of the participantand electrode pair-specific peaks in theta-band intersite phase synchrony in a msec postfeedback window. t Tests confirmed that the phase angle difference between FCz and F6 was not zero (t(23) = 19.49, p <.001) Figure 5. Early and sustained medial frontal beta power were larger after positive feedback and predictive of learning. (A) Topographical plots of beta-band power (18 24 Hz) msec postfeedback on incorrect and correct trials. (B) Average postfeedback FCz beta power (18 24 Hz) for current and next incorrect and correct trials. van de Vijver, Ridderinkhof, and Cohen 4113

10 Figure 6. Medial frontal theta-band intertrial phase coherence increases were larger and peaked later after negative feedback and were predictive of learning. (A) Time frequency representations of postfeedback intertrial phase coherence at FCz on incorrect and correct trials. (B) Topographical plots of theta-band intertrial phase coherence (4 8 Hz) msec postfeedback on incorrect and correct trials. (C) Average postfeedback FCz theta-band intertrial phase coherence (4 8 Hz) for current and next incorrect and correct trials. Figure 7. Lateral prefrontal theta power (4 8 Hz) increased after feedback at (A) F6 and (B) F Journal of Cognitive Neuroscience Volume 23, Number 12

11 Figure 8. Left and right lateral prefrontal sites were differently affected by feedback. (A and C) Time frequency representations of postfeedback intertrial phase coherence at F5 and F6 on incorrect and correct trials. (B and D) Average postfeedback theta-band intertrial phase coherence (4 8 Hz) at F5 and F6 for current and next incorrect and correct trials. or π (t(23) = 4.63, p <.001), the phase angle difference betweenfczandcp3wasnotzero(t(23) = 7.45, p <.001) or π (t(23) = 6.28, p <.001), and the phase angle difference between F6 and CP3 was not zero (t(23) = 11.60, p <.001) or π (t(23) = 6.64, p <.001). Moreover, we also investigated the peak frequencies of power at the separate electrodes and intersite phase synchrony between the electrodes. Peak frequencies over time in the theta range of power at FCz, power at F6, and intersite phase synchrony between FCz and F6 are presented in Figure 10. Visual inspection clearly demonstrates that the peak frequency of the intersite phase synchrony between the electrodes differs from the peak frequencies of the oscillatory power at the separate electrodes. The fact that the phase angles deviate from zero and π, the differences in peak frequencies between the separate sites and intersite phase synchrony, and the differences in learning-related effects between the separate sites and intersite phase synchrony, together strongly suggest that intersite phase synchrony is the result of functional connectivity between separate brain areas rather than a single generator. Response Adjustment: Intersite Phase Synchrony between Brain and Muscle During the response, theta intersite phase synchrony between CP3 and the right thumb muscle increased, and was stronger for correct trials than incorrect trials (F(1, 23) = 19.77, p <.001; Figure 11). When using C3 this effect was numerically in the same direction though not statistically significant (see supplementary material). There was no influence of Next-trial accuracy on the intersite phase synchrony between CP3 and muscle (F(1, 23) = 1.20, p =.285) and no interaction effect (F(1, 23) = 0.11, p =.740). In summary, theta-band intersite phase synchrony between medial frontal, right lateral prefrontal, and sensorimotor sites was larger after current incorrect trials, van de Vijver, Ridderinkhof, and Cohen 4115

12 whereas intersite phase synchrony between sensorimotor sites and muscle was larger during current correct trials. Low-level Processing of Feedback in Visual Cortex Positive feedback (face) and negative feedback (arrow) pictures differed in their visual features. We therefore analyzed the effect of current feedback on theta power, theta intertrial phase coherence and early (peak) beta power over low-level visual areas (time windows and peak/averaging procedures as described for frontal areas, electrodes Oz, O1, O2, PO7, PO8). Theta-band power (F(1, 23) = 6.39, p =.019) and intertrial phase coherence (F(1, 23) = 9.63, p =.005) were only significantly larger after positive than negative feedback at electrode O2. However, the direction of these effects was contrary to the effects of feedback valence on theta-band oscillations over frontal scalp locations. Moreover, because feedback signals in different modalities induce similar medial frontal ERPs (Miltner et al., 1997), differences in low-level feedback features do not seem to affect higher-level processing of the feedback content. Therefore, we ignored Figure 9. Theta-band intersite phase synchrony (4 8 Hz)was larger for incorrect than correct trials between (A) FCz and F6, (B) F6 and CP3, and (C) FCz and CP Journal of Cognitive Neuroscience Volume 23, Number 12

13 Figure 10. Differences between peak frequencies of FCz theta-band power, F6 theta-band power, and FCz-F6 theta-band intersite phase synchrony over time in a msec postfeedback window. Separate plots represent individual participants. the effects of feedback differences on low-level visual processing. DISCUSSION In the current study, we examinedtheroleofmedial frontal and lateral prefrontal theta (4 8 Hz)andbeta (18 24 Hz) band oscillations in feedback learning. We demonstrated that MFC oscillations were systematically relatedtothesuccessofpostfeedbackactionadjustments. As hypothesized, theta- and beta-band oscillations over medial frontal scalp sites, respectively, reflected current negative and positive feedback valence. Theta-band oscillations predicted learning from negative feedback, whereas beta-band oscillations predicted learning from positive and negative feedback. Moreover, negative feedback Figure 11. Theta-band intersite phase synchrony (4 8 Hz) between CP3 and muscle activity (EMG) was larger during correct than incorrect responses. van de Vijver, Ridderinkhof, and Cohen 4117

14 Figure 12. Schematic representation of effects (A) in the theta band (4 8 Hz) during the response, (B) in the theta band (4 8 Hz) after the feedback, and (C) in the beta band (18 24 Hz) after the feedback. induced an increase in theta-band oscillations in a larger network consisting of right lateral prefrontal, sensorimotor, and medial frontal sites, suggesting that more brain regions are recruited when behavioral adjustments are required (see Figure 12 for schematic overview of results). Theta-band Oscillations in MFC Our finding that theta-band activity over medial frontal sites increased more after negative than after positive feedback is in line with previous reports on theta-band oscillations in this area. MFC theta-band oscillations are larger after errors compared with correct responses in conflict tasks (Cohen, 2011; Cavanagh et al., 2009; Cohen et al., 2008; Trujillo & Allen, 2007; Luu et al., 2004) and reinforcement learning (Cavanagh et al., 2010) and after losses compared with wins in gambling tasks (Marco- Pallares et al., 2008, 2009; Cohen et al., 2007). Increases in theta-band oscillations have also been proposed as underlying mechanism of the error-related negativity (ERN; Trujillo & Allen, 2007; Luu et al., 2004), an ERP over MFC after an erroneous response that may share similar neurobiological features with the FRN (Holroyd & Coles, 2002). Thus, increases in theta-band activity seem to be an important mechanism for the signaling of undesirable outcomes and a possible need to adjust behavior. In the current study, theta-band activity over medial frontal sites not only signaled whether an adjustment was needed but predicted the success of the behavioral adjustment as well. To our knowledge this is the first report of a direct relationship between medial frontal theta-band oscillations and learning success, providing a possible neural mechanism for the adjustment process. The likelihood of MFC being involved in the evaluation of outcome history and the modification of action values and selection has widely been recognized (Botvinick, 2007; Nieuwenhuis et al., 2004; Ridderinkhof et al., 2004; Gehring & Willoughby, 2002; Holroyd & Coles, 2002). The importance of MFC for action outcome learning is supported by single-cell recordings in monkey MFC. For example, MFC neurons seem to represent both desired outcomes and the responses to acquire them (Luk & Wallis, 2009; Matsumoto, Suzuki, & Tanaka, 2003). Activity of MFC neurons also differentiates to-be-learned from already-learned responses (Quilodran, Rothe, & Procyk, 2008; Matsumoto, Matsumoto, Abe, & Tanaka, 2007). Finally, monkeys with MFC lesions are unable to take into account a longer history of actions and outcomes in reinforcement-guided action selection, despite intact immediate behavioral adaptation (Rudebeck et al., 2008; Kennerley et al., 2006). Imaging studies in humans also support the notion that MFC is an important node in the reinforcement learning network. MFC activity represents the prediction error necessary to compute the appropriate adjustment (Chase et al., 2011; Jessup, Busemeyer, & Brown, 2010; Philiastides et al., 2010; Bellebaum & Daum, 2008; Holroyd & Coles, 2008; Behrens, Woolrich, Walton, & Rushworth, 2007; Cohen, 2007; Brown & Braver, 2005). Furthermore, the sizes of the ERN and FRN not only represent the prediction error but, in some cases, also predict subsequent behavioral change (Philiastides et al., 2010; Cohen & Ranganath, 2007; Yasuda et al., 2004). When feedback is used to correct an error, MFC hemodynamic activity is increased compared with when the 4118 Journal of Cognitive Neuroscience Volume 23, Number 12

15 error is not corrected (Hester et al., 2008), and individual differences in MFC activation after errors are correlated with learning success. Hence, studies in both monkeys and humans imply that MFC activity is directly relevant for the ability to use feedback for learning. The only study that investigated the role of frontal theta-band oscillations in learning (Cavanagh et al., 2010) reported correlations of medial frontal theta-band oscillations with posterror slowing but not with behavioral accuracy. However, MFC activity has been related to response outcome learning, whereas stimulus outcome learning is commonly attributed to pfc or OFC (Rudebeck et al., 2008; Lee & Seo, 2007; Rushworth et al., 2007). In the current study, the response rather than the stimulus was informative and determined the outcome of the action. Moreover, the design of the current task allowed subjects to use the acquired information for immediate behavioral adjustment. Therefore, accuracy may have relied more on processes in MFC in the current task than in other (probabilistic) learning tasks such as the one used by Cavanagh and colleagues, where the stimulus contains reward information and learned stimulus outcome associations have to be kept in mind for a longer period. Influence of Theta-band Oscillations in Other Brain Areas and Motor Output We also found larger increases in theta activity over lateral prefrontal scalp sites and in theta-band intersite phase synchrony between medial frontal, lateral prefrontal, and sensorimotor sites after negative compared with positive feedback. The increases in lateral prefrontal thetaband activity and in intersite phase synchrony between lateral prefrontal and medial frontal sites mirror the effects after errors that have been reported in a flankers task (Cavanagh et al., 2009) and probabilistic learning task (Cavanagh et al., 2010), and seem to signal the need to increase cognitive control. Because we found no learningpredictive differences in intersite phase synchronization, this synchrony may be more important for signaling that behavior must be adjusted, whereas the actual adjustment of behavior may be carried out by other mechanisms. The increase in intersite phase synchrony between medial frontal and lateral prefrontal sites on the one hand and sensorimotor sites on the other hand has not been reported before. In the current paradigm the representation of the correct action needs to be established and updated to optimize behavior. Because the amount of intersite phase synchrony between medial frontal and sensorimotor sites did not predict learning whereas medial frontal theta activity did, we suggest that differences in theta oscillations in MFC (and maybe also subcortical areas) shape motor plans to improve selection on the next encounter. The increase in theta-band intersite phase synchrony with sensorimotor cortex may then be part of the communication of the adjusted action plan for the next trial (Cohen & Ranganath, 2007), but whether the adjustment is effective and successful depends critically on the input from MFC to the motor system. Oscillatory communication between areas is more effective at specific phase relationships between areas when input arrives at an optimal phase of the receiving neurons (Gregoriou et al., 2009; Sauseng & Klimesch, 2008; Womelsdorf et al., 2007). Different patterns of MFC thetaband oscillations can represent different action reward associations (Womelsdorf, Johnston, Vinck, & Everling, 2010) and theta-band intersite phase synchrony in networks of relevant brain areas changes flexibly with task demands (Mizuhara & Yamaguchi, 2007). The fact that not only medial frontal theta power but also intertrial phase coherence predicted learning suggests that coherence in the phase of the oscillations strengthens specific connections and determines the effectiveness of the output of MFC to other areas. Intersite phase synchrony between sensorimotor sites and thumb muscles increased more during a correct than an incorrect response. Interestingly, this difference in intersite phase synchrony also resided in the theta frequency band rather than the beta or gamma band. The fact that the differences were found in the theta band corroborates the idea that oscillations in this specific frequency band increase effective communication via a chain of phasedependent synchronies between successive brain areas from pfc to the motor system and finally to the muscle. Beta-band Oscillations and Learning from Positive Feedback Positive feedback induced a larger increase in beta-band oscillations than negative feedback. This is in accordance with the increase in beta-band oscillations after positive outcomes that has been reported in gambling tasks (Marco- Pallares et al., 2008, 2009) and probabilistic reinforcement learning (Cohen et al., 2007). Because beta-band oscillations were sensitive to both the magnitude and the probability of gain, Marco-Pallares and colleagues (2008) suggest that oscillations in the beta range may be a neural marker of reward that originates from ventromedial OFC rather than MFC. A possible role of beta-band oscillations might be the synchronization of neural populations over long distances to couple frontal and striatal structures involved in reward processing. In the current study, beta-band activity was not only responsive to positive feedback but functioned as a learning signal as well. In line with the idea of beta-band oscillations signaling the tendency to maintain the status quo (Engel & Fries, 2010), the increase in beta-band oscillations after positive feedback in our study can function as a mechanism to strengthen the current response set in favor of other options, thereby influencing future behavior. This explanation seems even more plausible because the current task required subjects to make the same movement at the same time on the next trial, which may have engaged top down endogenous control. van de Vijver, Ridderinkhof, and Cohen 4119

16 Conclusion We found learning-related increases in both theta and beta frequency bands over medial frontal sites. We suggest that differences in theta-band oscillations are underlying learning from errors and guide the adjustment of motor plans and their communication to other brain areas such as the motor system. The increase in control via the synchronization of oscillations in a chain of brain areas to an underlying theta rhythm seems to continue even into the response muscle. Increases in beta-band oscillations appear to signal the necessity of the continuation of the status quo in reward and motor networks after positive feedback. Acknowledgments This study was funded by a Vici grant from the Netherlands Organization for Scientific Research to K. R. R. M. X C. was supported by a Human Frontiers Science Program Grant. Reprint requests should be sent to Irene van de Vijver, Department of Psychology, University of Amsterdam, Roetersstraat 15, 1018 WB, Amsterdam, the Netherlands, or via i.vandevijver@uva.nl. REFERENCES Akam, T., & Kullmann, D. M. (2010). Oscillations and filtering networks support flexible routing of information. Neuron, 67, Babiloni, C., Vecchio, F., Bares, M., Brazdil, M., Nestrasil, I., Eusebi, F., et al. (2008). Functional coupling between anterior prefrontal cortex (BA 10) and hand muscle contraction during intentional and imitative motor acts. Neuroimage, 39, Behrens, T. E., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10, Bellebaum, C., & Daum, I. (2008). Learning-related changes in reward expectancy are reflected in the feedback-related negativity. European Journal of Neuroscience, 27, Botvinick, M. M. (2007). Conflict monitoring and decision making: Reconciling two perspectives on anterior cingulate function. Cognitive, Affective, & Behavioral Neuroscience, 7, Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: An update. Trends in Cognitive Sciences, 8, Brown, J. W., & Braver, T. S. (2005). Learned predictions of error likelihood in the anterior cingulate cortex. Science, 307, Cavanagh, J. F., Cohen, M. X, & Allen, J. J. (2009). Prelude to and resolution of an error: EEG phase synchrony reveals cognitive control dynamics during action monitoring. Journal of Neuroscience, 29, Cavanagh, J. F., Frank, M. J., Klein, T. J., & Allen, J. J. (2010). Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage, 49, Chase, H. W., Swainson, R., Durham, L., Benham, L., & Cools, R. (2011). Feedback-related negativity codes prediction error but not behavioral adjustment during probabilistic reversal learning. Journal of Cognitive Neuroscience, 23, Christie, G. J., & Tata, M. S. (2009). Right frontal cortex generates reward-related theta-band oscillatory activity. Neuroimage, 48, Cohen, M. X. (2007). Individual differences and the neural representations of reward expectation and reward prediction error. Social Cognitive & Affective Neuroscience, 2, Cohen, M. X. (2011). Error-related medial frontal theta activity predicts cingulate-related structural connectivity. Neuroimage, 55, Cohen, M. X, Elger, C. E., & Ranganath, C. (2007). Reward expectation modulates feedback-related negativity and EEG spectra. Neuroimage, 35, Cohen, M. X, & Ranganath, C. (2007). Reinforcement learning signals predict future decisions. Journal of Neuroscience, 27, Cohen, M. X, Ridderinkhof, K. R., Haupt, S., Elger, C. E., & Fell, J. (2008). Medial frontal cortex and response conflict: Evidence from human intracranial EEG and medial frontal cortex lesion. Brain Research, 1238, Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, Engel, A. K., & Fries, P. (2010). Beta-band oscillations signalling the status quo? Current Opinion in Neurobiology, 20, Fries, P. (2005). A mechanism for cognitive dynamics: Neuronal communication through neuronal coherence. Trends in Cognitive Sciences, 9, Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295, Gregoriou, G. G., Gotts, S. J., Zhou, H., & Desimone, R. (2009). High-frequency, long-range coupling between prefrontal and visual cortex during attention. Science, 324, Hester, R., Barre, N., Murphy, K., Silk, T. J., & Mattingley, J. B. (2008). Human medial frontal cortex activity predicts learning from errors. Cerebral Cortex, 18, Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, Holroyd, C. B., & Coles, M. G. (2008). Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior. Cortex, 44, Jessup, R. K., Busemeyer, J. R., & Brown, J. W. (2010). Error effects in anterior cingulate cortex reverse when error likelihood is high. Journal of Neuroscience, 30, Kayser, J., & Tenke, C. E. (2006). Principal components analysis of Laplacian waveforms as a generic method for identifying ERP generator patterns: I. Evaluation with auditory oddball tasks. Clinical Neurophysiology, 117, Keil, J., Weisz, N., Paul-Jordanov, I., & Wienbruch, C. (2010). Localization of the magnetic equivalent of the ERN and induced oscillatory brain activity. Neuroimage, 51, Kennerley, S. W., Walton, M. E., Behrens, T. E., Buckley, M. J., & Rushworth, M. F. (2006). Optimal decision making and the anterior cingulate cortex. Nature Neuroscience, 9, Kerns, J. G. (2006). Anterior cingulate and prefrontal cortex activity in an fmri study of trial-to-trial adjustments on the Simon task. Neuroimage, 33, Kerns, J. G., Cohen, J. D., MacDonald, A. W., III, Cho, R. Y., Stenger, V. A., & Carter, C. S. (2004). Anterior cingulate conflict monitoring and adjustments in control. Science, 303, Journal of Cognitive Neuroscience Volume 23, Number 12