NeuroImage 54 (2011) Contents lists available at ScienceDirect. NeuroImage. journal homepage:

Transcription

1 NeuroImage 54 (2011) Contents lists available at ScienceDirect NeuroImage journal homepage: Parsing decision making processes in prefrontal cortex: Response inhibition, overcoming learned avoidance, and reversal learning Steven G. Greening b, Elizabeth C. Finger c,d, Derek G.V. Mitchell a,b,d, a Department of Psychiatry, The University of Western Ontario, London, Ontario, N6A 5A5, Canada b Department of Anatomy and Cell Biology, The University of Western Ontario, London, Ontario, N6A 5A5, Canada c Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, The University of Western Ontario, London, Ontario, N6A 5A5, Canada d Department of Psychology, The University of Western Ontario, London, Ontario, N6A 5A5, Canada article info abstract Article history: Received 1 April 2010 Revised 18 August 2010 Accepted 8 September 2010 Available online 17 September 2010 Reversal learning refers to the ability to inhibit or switch responding to an object when the object-reward contingency changes. Deficits in this process are related to social abnormalities, impulsiveness, and a number of psychiatric disorders. A range of neural regions play a role in this process, including dorsolateral prefrontal cortex (dlpfc), dorsomedial prefrontal cortex (dmpfc), and inferior frontal gyrus (IFG). However, determining the specific functional contribution of each region has proved difficult, in part because reversal learning involves multiple cognitive subprocesses such as error detection, inhibiting responding to formerly rewarded stimuli, and overcoming avoidance of previously punished stimuli. We used fmri and an experimental task adapted from a recent neurochemical study in marmosets to parse neural responding to subprocesses of reversal learning during choice and feedback trial components. Error-feedback processing was associated with increased activity in dmpfc, dlpfc, and IFG whether participants were overcoming avoidance, inhibiting responding, or performing classic response reversal. Reduced activity in medial prefrontal cortex (mpfc) was associated with error-feedback processing for response inhibition but not overcoming avoidance. Conversely, there was significantly greater activity in anterior dmpfc during error-feedback processing in overcoming avoidance compared to response inhibition. A conjunction analysis confirmed that a striking overlap in activity was observed across the three conditions in IFG, dlpfc, and dmpfc. The results are consistent with conceptualizations of IFG function that emphasize modulating stimulus response maps rather than purely response inhibition. The approach has implications for models of prefrontal function and neurocognitive perspectives on a range of behavioural abnormalities associated with impairments in decision making Elsevier Inc. All rights reserved. Introduction Decision making is a flexible cognitive process that involves selecting response options that will ultimately maximize gains while minimizing losses. Reversal learning is a critical component of decision making by which behavioural alterations are made when the reinforcement value of available response options change. Difficulties with reversal learning, as occurs following lesions to the orbitofrontal cortex (OFC), are associated with behavioural abnormalities including impulsiveness, disinhibition, and social impropriety (Hornak et al., 2004; Rolls et al., 1994). Reversal learning deficits are also associated with reactive aggression (Mitchell et al., 2006), and a range of psychiatric disorders featuring social impairment including psychopathy (Budhani et al., 2006; Mitchell et al., 2002), severe conduct disorder Corresponding author. University Hospital, University of Western Ontario, 339 Windermere Road, London, Ontario, Canada. Fax: address: dmitch8@uwo.ca (D.G.V. Mitchell). (Budhani and Blair, 2005), frontotemporal dementia (Rahman et al., 1999), and bipolar disorder (Dickstein et al., 2009). Lesion studies from both human and non-human primates suggest OFC is critical for reversal learning (Dias et al., 1996; Fellows and Farah, 2005; Izquierdo and Murray, 2004; O'Doherty, 2007; Roberts, 2006; Rolls et al., 1994). However, functional neuroimaging studies implicate multiple areas of prefrontal cortex in this process including OFC (O'Doherty et al., 2001), inferior frontal gyrus (IFG; Cools et al., 2002; Nagahama et al., 2001), dorsomedial prefrontal cortex (dmpfc; Budhani et al., 2007; Mitchell et al., 2009), dorsolateral prefrontal cortex (dlpfc; Mitchell et al., 2009; Remijnse et al., 2005), and medial prefrontal cortex (mpfc; Budhani et al., 2007; O'Doherty et al., 2003a). Studies have also implicated posterior parietal cortex (Glascher et al., 2009; Hampshire and Owen, 2006), and subcortical structures including the amygdala (Budhani et al., 2007; Elliott et al., 2004), and striatum (Hampton and O'Doherty, 2007; Mitchell et al., 2008; Tanaka et al., 2008) in reversal learning. Although a number of neural regions have been implicated, the specific functional contribution of each in this context remains unclear /$ see front matter 2010 Elsevier Inc. All rights reserved. doi: /j.neuroimage

2 S.G. Greening et al. / NeuroImage 54 (2011) One reason for this lack of functional specificity is that reversal learning likely involves multiple components including error detection, inhibiting a response to a previously rewarding stimulus, and overcoming avoidance of a previously punished stimulus. Additionally, in the majority of reversal learning studies to date, the blood oxygenation level dependent (BOLD) response during response selection and feedback processing have been confounded, making it difficult to distinguish response control processes from contingencychange detection or stimulus response mapping reconfiguration. Recently, two components of reversal learning have been dissociated on a neuropharmacological level. In an elegant operant learning experiment, Clarke et al. (2007) found that depleting prefrontal levels of serotonin in marmosets selectively disrupted the capacity to inhibit responding to a previously rewarded response. The manipulation left intact the ability to overcome avoidance of a previously punished stimulus. In the same study, depleting prefrontal levels of dopamine had no effect on either facet of reversal learning. The study was the first to fractionate these key reversal components in the same task; however, diffusion of the pharmacological agent made it difficult to determine the relative contribution of lateral and medial regions of prefrontal cortex in this context. This raises the intriguing possibility that hitherto confounded processes in reversal learning (response inhibition versus overcoming avoidance) may map onto distinct subregions of prefrontal cortex. Fractionating these processes in humans with neuroimaging may have important implications not only for refining models of prefrontal cortex function, but also for improving our understanding of the neurocognitive basis for the range of disorders associated with reversal learning deficits. In the present study we used a variant of Clarke et al.'s (2007) object discrimination task in conjunction with fmri to delineate the functional neuroanatomy of two neurochemically distinct subprocesses of reversal learning. Specifically, we sought to uncover neural activity associated with two distinct components of reversal learning: 1) inhibiting responding to a previously rewarded stimulus in favour of a novel one (IR); and 2) overcoming avoidance of a previously punished stimulus to obtain reward (OA). In addition, we designed the experimental trial structure in a manner that enabled us to distinguish the BOLD response to the choice interval (i.e., response selection) from that associated with the feedback interval. This allowed us to address two specific questions. First, do the conditions of IR and OA engage distinct regions of prefrontal cortex during the choice phase of decision making before feedback is received? Second, do distinct regions of prefrontal cortex reconfigure inappropriate stimulus response mappings across conditions in reaction to negative feedback that precedes response change (cf. Mitchell et al., 2009; Nagahama et al., 2001; O'Doherty et al., 2003a)? Methods Subjects Twenty-five healthy adults participated in the study. Three participants were excluded from the analysis due to technical difficulties (user-interface or computer malfunction). One participant did not perform above chance during the task and was excluded from the analysis, leaving data from 21 participants (10 women, age range 20 32, mean age 24.5, SD=2.8). All subjects granted informed consent, were in good health, and had no past history of psychiatric problems, neurological disease or head injury as determined by screening and interview using the Structured Clinical Interview for DSM-IV (First et al., 1997). All subjects had normal or corrected-to-normal vision, and were right-handed as determined by the Edinburgh handedness inventory (Oldfield, 1971). The study was approved by the Health Sciences Research Ethics Board at the University of Western Ontario, London, Ontario, Canada. Experimental task We adapted an object discrimination reversal learning task used in a recent neuropharmacological lesion study (Clarke et al., 2007) and incorporated design features applied in our previous neuroimaging studies (Mitchell et al., 2009; 2008). On each trial, participants were asked to select one object (fractal image) from a pair displayed against a white computer screen (see Fig. 1). Participants were explicitly told that within each of the object pairs one was associated with positive reinforcement ( You WIN 100 points! ), and the other with negative reinforcement ( You LOSE 100 points! ). Participants were also told that at some point during the task, the object associated with positive reinforcement could change, and that they should change their response accordingly. Each trial consisted of four sequential intervals (Fig. 1) beginning with a fixation interval (250, 550, or 850 ms), followed by a choice interval depicting a pair of objects (1200, 1500, or 1800 ms), a blank screen (500, 800, or 1100 ms), and finally, a feedback interval (750, 1050, or 1350 ms). The duration of each interval was pseudo-randomized so that all durations were equally represented within each run. Additionally, sixteen jitter fixation-only trials (8 x2000 ms and 8 x3000 ms) were presented randomly throughout each run to serve as baseline. The use of variable duration trial elements and jitter trials is an effective means for distinguishing between events that occur within a trial (Grinband et al., 2008; Serences, 2004), and similar designs have been used effectively in decision making research (Hampshire et al., 2009; Hester et al., 2007; Tobler et al., 2009). Participants responded by making a left (index finger) or right (middle finger) button press using the right hand. Within each object pair, the relative position was counterbalanced so that they appeared equally on the left and right side of the screen. The task was programmed in E-Prime (Psychology Software Tools, 2002). During each run, four distinct object pairs were used corresponding to the four experimental conditions. Following an acquisition phase that varied between 6 and 10 trials (mean of 8), three of the four pairs underwent a value reversal (relearning stage), forming three experimental conditions (varying between 6 and 10 trials per pair per run). The number of relearning trials and the onset time of the reversal for each pair varied across runs for each condition (i.e., within a run, pairs underwent reversals at different times) in a counterbalanced fashion. This manipulation was used to pre-empt the anticipation of specific contingency changes across runs. In the classic response reversal (RR) condition, the value of each object in a pair reversed (i.e., the object associated with 100 points was associated with a loss of 100 points and vice versa). In the inhibit responding (IR) condition, the value of the previously rewarded stimulus was reversed (i.e., the object associated with 100 points became associated with losing 100 points) and a novel rewarded object was introduced (associated with winning 100 points) to replace the previously punishing stimulus. In the overcoming avoidance (OA) condition, the value of the punished object was reversed so that it became associated with winning 100 points; at the same time, a novel punishing object was introduced, which replaced the previously rewarding stimulus. The fourth experimental condition was a control (CTL) in which the object associations remained unchanged. The trial structure and sample conditions are presented in Fig. 1. Subjects completed six runs that were each 5 min and 18 s in duration. Each run involved new stimuli to insure that participants had to learn new object reward contingencies. Consequently, participants received 48 acquisition phase and 48 relearning phase trials for each condition. Multiple versions of the same task were produced so that the object pairs that represented each relearning condition were counterbalanced across participants. MRI data acquisition Subjects were scanned during task performance using a 3 T Siemens MRI scanner with a 32 channel head coil. Each session began with a

3 1434 S.G. Greening et al. / NeuroImage 54 (2011) Fig. 1. Object discrimination task. The top panel illustrates the trial structure and variable duration trial intervals. Participants were instructed to select one object from a pair as quickly and as accurately as they could. They subsequently received positive feedback ( you WIN 100 points ) or negative feedback ( you LOSE 100 points ). The bottom panels depict the object-reward contingency changes for each pair when switching from the acquisition phase to the relearning phase in a single trial ( + represents the rewarded object; represents the punished object). high resolution, T1 weighted, anatomical scan covering the whole brain (repetition time=2300 ms, echo time=4.25 ms; FOV=25.6 cm; 192 axial slices; voxel size=1 mm isovoxels; 256 X 256 matrix). Next, six functional MRI runs were completed in which BOLD changes were measured. The fmri images were taken with a T2*-gradient echo-planar imaging sequence (repetition time=3000 ms; echo time=30 ms; 120x120 matrix; FOV=24 cm). Complete brain coverage was obtained with 45 slices of 2 X 2 mm in plane with a slice thickness of 2.5 mm, which formed voxels of 2 X 2 X 2.5 mm. Slices were acquired in an interleaved fashion. fmri analysis Individual and group analyses were conducted using Analysis of Functional NeuroImages software (AFNI; Cox, 1996). The first four dummy volumes of each of the six runs were discarded in order to insure that magnetization equilibrium was reached. Motion correction was completed by registering all BOLD data in each run of the task to the first volume of the first experimental run, which immediately followed the anatomical image. The dataset for each subject was spatially smoothed (using an isotropic 4 mm Gaussian kernel) and the time series data were normalized by dividing the signal intensity of a voxel at each time point by the mean signal intensity of that voxel for each run and multiplying the result by 100. The resultant regression coefficients represented the percent signal change from the mean activity. To account for voxel-wise correlated drifting, a baseline plus linear drift and quadratic trend were modelled to the time series of each voxel. This produced a beta coefficient and t-value for each voxel and regressor. To perform the group analyses, each individual's data was transformed into the standard space of Talairach and Tournoux. This individual subject analysis was followed by the group analyses described below. Two separate analyses were conducted for the choice interval and the feedback interval (described below). However, in order to first demonstrate that our experimental manipulation successfully dissociated the BOLD response during the choice interval from the BOLD response during the feedback intervals, we performed a 4 (Condition) by 2 (Trial Interval: Choice, Feedback) ANOVA using the BOLD data of the early relearning stage. We examined the effect of trial phase across conditions in the precentral gyrus using an anatomical region of interest mask. If our manipulation to separate the BOLD response during responding versus feedback was effective, robust activity in the precentral gyrus should be observed during the choice interval compared to the feedback interval across conditions. Separate regressor models for choice and feedback intervals were developed in order to address our hypothesis regarding IR and OA. Resulting regressors were convolved with a gamma-variate basis function to account for the slow hemodynamic response. The BOLD response was then fit to each of the regressors to perform linear regression modeling. Choice interval analysis For the choice interval, regressors were created to capture the decision making process for the RR, IR, OA, and CTL conditions, which began with the onset of the object pair choice screen and ended once an object selection was made via button press. Correct responses and errors were modelled separately. Correct responses were separated into early and late trials of the relearning stage for each experimental condition. We were particularly interested in examining early relearning trials given evidence that greater demands are placed on neural systems involved in associative learning and response control during early compared to late exposures to new contingencies (Finger et al., 2008b; Marschner et al., 2008; Milad et al., 2007; Phelps et al., 2004). This resulted in eight regressors being modelled using a variable duration (response time locked) epoch design (Grinband et al., 2008). The early relearning phase included the first 4 correct choices that the participant made in each run (24 total, representing half of the mean number of total relearning stage trials), and the late relearning stage was comprised of the remaining correct relearning stage trials. All acquisition trials were modelled separately as regressors of no interest. This allowed us to first perform a 4 (condition: RR, IR, OA, CTL) by 2 (Relearning Stage: early, late) ANOVA on the regression coefficients during the choice interval. Follow-up analyses using paired t-tests were performed to delineate the nature of the main effects and interaction. Correction for multiple comparisons was performed using AlphaSim with 1000 Monte Carlo iterations on a whole brain EPI matrix to pb0.05 (Ward, 2000). We

4 S.G. Greening et al. / NeuroImage 54 (2011) also performed an anatomically derived small volume correction (SVC) for multiple comparisons using AlphaSim with 1000 Monte Carlo iterations on a bilateral amygdala and a bilateral orbitofrontal cortex EPI matrix to pb0.05. Feedback interval analysis During the feedback interval, we modelled the BOLD response during both correct and error-feedback across all four conditions in the relearning stage. Specifically, we modelled only those errorfeedback events that were followed by response change the next time that particular object pair was encountered. Error trials in which participants did not change their response on subsequent trials, or error trials that occurred following the performance of a correct reversal trial occurred infrequently and so were modelled as regressors of no interest. This resulted in eight regressors of interest modeling the feedback interval for each condition using a variable duration epoch design (i.e., the duration of the feedback screen). These regressors of interest were then used to perform key contrasts for the feedback interval. Correction for multiple comparisons was performed using AlphaSim with 1000 Monte Carlo iterations on a whole brain EPI matrix to pb0.05 (Ward, 2000). The current experimental design enabled us to dissociate the feedback interval from the choice interval, and also the response inhibition from overcoming avoidance components of decision making. In order to determine the impact of feedback on distinct regions of prefrontal cortex previously implicated in reversal learning, we followed previous reversal learning studies in contrasting the BOLD response to our error trials versus correct trials (cf. Budhani et al., 2007; Kringelbach and Rolls, 2003; Mitchell et al., 2009; O'Doherty et al., 2001). To address the question of whether distinct regions of prefrontal cortex reconfigure stimulus response mappings in reaction to negative feedback, we contrasted the BOLD response to errorfeedback for each condition with the BOLD response to correct-feedback during the same condition. Thus, four contrasts were performed: 1) an IR error-feedback versus IR correct-feedback contrast; 2) an OA errorfeedback versus OA correct-feedback contrast; 3) an IR error-feedback versus OA error-feedback contrast; and 4) an RR error-feedback versus RR correct-feedback contrast. All four contrasts were thresholded at pb0.005 and corrected to pb0.05 for multiple comparisons. Additionally, we used the erode function in AFNI to break up small bridges of active voxels that were connecting disparate regions of the cortex to facilitate interpretation of the results. significant difference in accuracy between OA and IR (pn0.8). In addition, a 4 (Condition: IR, OA, RR, CTL) X 2 (Relearning Stage: Early vs. Late) ANOVA was performed to examine the impact of the experimental conditions on response latency during early and late stage correct trials. This revealed a main effect of condition (F (3,60) =4.02; pb0.05). Collapsing across relearning stage, participants were significantly slower to make their response on the OA condition compared to the CTL (pb0.01) condition. There were no significant differences between any of the other conditions. There was also a main effect of relearning stage (F (1,20) =73.94; pb0.001). Pairwise comparisons revealed that participants were significantly slower to respond in the early stage of reversal learning compared to the late stage for all conditions (pb0.001). A significant condition by relearning stage interaction was also present (F (3,60) =3.22; pb0.05). During the early relearning stage participants responded significantly slower in the OA condition compared to the IR (pb0.001), RR (pb0.05), and CTL (pb0.001) conditions. There were no significant differences between the response latencies across any of the other conditions during the early stages of reversal. There were also no significant differences in response latency across any of the conditions during the late reversal relearning stage. fmri results We examined the main effect of interval (choice versus feedback processing) as a positive control to insure that our manipulation distinguished BOLD responding associated with motor responding versus feedback processing. In line with our expected effect, we observed significantly greater activity in the precentral gyrus during Conjunction analysis We performed a conjunction analyses to determine the extent to which the brain regions sensitive to error-feedback in the IR, OA, and RR conditions were overlapping and distinct. This involved constructing a composite map representing areas that were commonly and uniquely activated in the IR, OA, and RR error-feedback contrasts (threshold at pb0.005). Specifically, we investigated four combinations of conjunction, IR+OA+RR, IR+OA, IR+RR, and OA+RR. Results Behavioural results We performed a repeated-measures ANOVA comparing response accuracy across the four conditions of the relearning stage. This revealed a main effect of condition (F (3,60) =44.12, pb0.001). Followup pairwise comparisons showed that response accuracy was significantly higher in the control condition (M=93.8%) relative to all other conditions (pb0.001). Participants were also significantly more accurate in the OA (M=81.1%) and IR (M=81.5%) conditions relative to the RR condition (M=75.6%; pb0.005). There was no Fig. 2. Areas modulated by condition during the choice interval. Images depicting the main effect of condition during the choice phase revealed greater activity in the (a) right frontal pole and (b) right mofc during RR, OA, and CTL conditions compared to IR. There was greater activity in the (c) left amygdala during RR, IR, and OA conditions compared to CTL. Mean percent change of the BOLD signal within the entire cluster is represented along the y-axis; bars represent the main effect of condition (collapsed across early and late trials) and error bars depict the SEM between subjects. Images are thresholded at pb0.005, (a) was whole-brain corrected to pb0.05, (b) and (c) were small volume corrected to pb0.05.

5 1436 S.G. Greening et al. / NeuroImage 54 (2011) Table 1 Results of the choice interval 4 (condition)x 2 (relearning stage) ANOVA. Location R/L BA X Y Z Volume F value Main effect of condition OANIR, RR, and CTL Inferior parietal lobe L 40/ Cuneus R Superior occipital gyrus L 7/ Cerebellum L Cuneus R OA, RR, CTLNIR Frontal pole R Medial orbital frontal cortex a R 10/ OA, IR, RRNCTL Amygdala a L Interaction of condition and relearning stage Dorsolateral PFC L All clusters are thresholded at pb0.005 and pb0.05 corrected. a Small volume corrected (SVC) to pb0.05. the choice interval compared to the feedback interval (see Supplementary Materials Fig. S1). BOLD changes during the choice interval In order to determine the impact of condition (RR, IR, OA, and CTL) on the brain's response, we performed a 4 (Condition: RR, IR, OA, CTL) by 2 (Relearning Stage: Early versus Late) ANOVA on the whole brain BOLD data from the choice interval, thresholded at pb0.005 and corrected for multiple comparisons to pb0.05. This revealed a main effect of relearning stage as well as the two results of interest, a main effect of condition and a condition-by-relearning-stage interaction (see Table 1). Follow-up pairwise comparisons were performed to determine the nature of the main effect and interaction (p-values for pairwise comparisons are displayed throughout this section in parentheses). A significant main effect of condition was observed in regions previously implicated in reversal learning including OFC, inferior parietal cortex, and amygdala (OFC and amygdala were small volume corrected pb0.05). Follow-up paired t-tests revealed greater activity for the RR, OA, and CTL conditions compared to the IR condition in the right frontal pole (BA 10; pb0.05; Fig. 2a) and right mofc (BA 10/13; pb0.001; Fig. 2b). Additionally, greater activity was observed in the left inferior parietal lobe during OA relative to the RR, IR, and the CTL conditions (pb0.01). Consistent with its role in encoding stimulus values (Paton et al., 2006), enhanced activity in the amygdala was observed during all conditions that featured a change in stimulus value relative to the control condition (RR, IR, and OANCTL condition; pb0.05; Fig. 2c), and there were no significant differences between the three conditions that underwent reversal. A significant condition by relearning stage interaction was observed only in dlpfc (Fig. 3). Within this region of dlpfc, significantly enhanced activity was observed to early versus late trials during RR and IR (earlynlate; pb0.05), but not during OA (pn0.4) or CTL conditions (pn0.09). Interestingly, the condition that elicited greatest activity in dlpfc during early trials (RR), showed the least amount of activity in late trials. Indeed, activity elicited in dlpfc to the late stage of RR was significantly less than late-stage activity in any other condition (RR late bir late, pb0.05; RR late boa late, pb0.001; RR late b CTL late, pb0.05). BOLD changes during feedback interval To determine the impact of feedback across conditions we examined brain activity associated with the processing of errorfeedback on the final trial before a correct reversal within a given condition as per previous studies (Cools et al., 2002; O'Doherty et al., 2003a; Remijnse et al., 2005). First, we examined error-related activity during the IR condition (IR error-feedback versus IR correctfeedback; thresholded at pb0.005, corrected to pb0.05; Fig. 4a and Supplemental Table S1). Notably, this contrast revealed increased activity during error-feedback in neural regions associated with response inhibition and control (Aron et al., 2003; Casey et al., 2001), including bilateral IFG. In addition, significant activity during errorfeedback was observed in regions implicated in resolving decisional and perceptual conflict (Botvinick et al., 2004; Mitchell et al., 2009), including bilateral dmpfc, bilateral dlpfc, and bilateral inferior parietal lobe (BA 39). There was also enhanced activity bilaterally in the frontal pole (BA 10) in response to error-feedback. Conversely, there was reduced activity in a region of ventromedial PFC (Fig. 5a) implicated in the representation of stimulus-outcome associations during error-feedback relative to correct-feedback (Budhani et al., 2007; Finger et al., 2008a), and the bilateral inferior parietal lobe (BA 40) (Fig. 5b). Second, we examined error-related activity during the OA condition (OA error-feedback versus OA correct-feedback; thresholded at pb0.005, corrected to pb0.05; Fig. 4b and Supplemental Table S1), which also revealed enhanced activity in bilateral dmpfc, bilateral dlpfc, bilateral inferior parietal lobe (BA 39), and the bilateral IFG. In addition, there was significantly greater activity in the bilateral frontal pole (BA 10). Conversely, the left hippocampus displayed significantly reduced activity during error-relative to correct-feedback. Last, we contrasted the brain activity associated with processing error-related feedback between the two subprocesses of reversal learning (IR error-feedback versus OA error-feedback, thresholded at pb0.005, corrected to pb0.05). This contrast revealed enhanced activity in a region of the left anterior dmpfc and left Fig. 3. Interaction reveals an effect of relearning stage and condition on dlpfc activity during the choice interval. The condition x relearning stage interaction (threshold pb0.005, corrected to pb0.05) demonstrated differential activity in dorsolateral PFC. Follow-up paired t-tests revealed significantly enhanced activity in early versus late trials during RR and IR (pb0.05), and significantly less activity in this area during late trials of RR compared to all other conditions. Mean percent change of the BOLD signal within the entire cluster is represented along the y-axis; error bars depict the SEM between subjects.

6 S.G. Greening et al. / NeuroImage 54 (2011) Table 2 Contrast of IR and OA error-feedback precipitating behavioural change. Location R/L BA X Y Z Volume T value OA error-feedbacknir error-feedback Anterior dmpfc L Angular gyrus L Precuneus L Mid temporal gyrus L All clusters are thresholded at pb0.005 and pb0.05 corrected. Fig. 4. Contrasts of error-feedback versus correct-feedback with IR (a) and OA (b) conditions. Images depict regions with greater activity in response to errorfeedback (threshold pb0.005, corrected to pb0.05). This revealed greater activity for both contrasts in bilateral dlpfc, dmpfc, and inferior parietal cortex (left); greater activity in bilateral IFG (middle); greater bilateral activity in the frontal pole (right). angular gyrus (BA39) during OA error-feedback relative to IR errorfeedback (Figs. 5c,d; Table 2). Conjunction analysis In order to further determine the extent to which the IR, OA, and RR components of reversal learning activate similar neural regions, we conducted a conjunction analysis in which we quantified contiguous volumes of activity generated from the error-feedback contrasts (error-feedback versus correct-feedback). Thus we have reported cluster volumes for four conjunctions below: IR+OA+RR, IR+OA, IR+RR, and OA +RR. We included all clusters that survived correction at the level of the within condition contrasts and have reported clusters of 400 or more overlapping voxels. All four conjunction combinations indicated a striking overlap in neural regions that respond with greater activity to error-feedback (Fig. 6). The full results of the conjunction analysis are presented in Table 3. Of note, the conjunction of IR+OA+RR revealed enhanced activity within the bilateral dmpfc (7856 mm 3 ), bilateral dlpfc (right = 9536 mm 3, left = 4232 mm 3 ), bilateral frontal pole (right=3200 mm 3, left=784 mm 3 ), bilateral IFG (right=536 mm 3, left=1784 mm 3 ), and bilateral inferior parietal lobe (right=5504 mm 3, left=5496 mm 3 ). Similarly, the conjunction of IR+OA revealed enhanced activity within the bilateral dmpfc (8456 mm 3 ), bilateral dlpfc (right=12,360 mm 3, left=5592 mm 3 ), bilateral frontal pole (right= 3880 mm 3, left = 1136 mm 3 ), bilateral IFG (right = 584 mm 3,left= 1792 mm 3 ), and bilateral inferior parietal lobe (right=10,256 mm 3, left=6224 mm 3 ). Overlapping activity during IR+RR was observed within the bilateral dmpfc (10,672 mm 3 ), bilateral dlpfc (right= 11,400 mm 3, left=4952 mm 3 ), bilateral frontal pole (right=4136 mm 3, left=784 mm 3 ), bilateral IFG (right=1560 mm 3, left=2368 mm 3 ), and bilateral inferior parietal lobe (right=6760 mm 3, left=5624 mm 3 ). In addition, the conjunction of OA+RR revealed enhanced activity within the bilateral dmpfc (10,360 mm 3 ), bilateral dlpfc (right= 14,968 mm 3, left=9416 mm 3 ), bilateral frontal pole (right=5616 mm 3, left= 3408 mm 3 ), bilateral IFG (right=1296 mm 3, left=3432 mm 3 ), and bilateral inferior parietal lobe (right=7776 mm 3, left=19,576 mm 3, this cluster spans to include the precuneus). We then performed a conjunction analysis on regions that displayed a decrease in activity during error-feedback relative to correct- Fig. 5. Contrast of IR error-feedback versus IR correct-feedback (top) and between OA error-feedback versus IR error-feedback (bottom). The contrast involving IR errorfeedback versus IR correct-feedback (threshold pb0.005, corrected to pb0.05) revealed significantly reduced activity during error-feedback relative to correct-feedback processing in mpfc (a) and bilateral inferior parietal lobe (b). The contrast involving OA error-feedback versus IR error-feedback (threshold pb0.005, corrected to pb0.05) revealed greater activity during OA in anterior dmpfc (c) and angular gyrus (d). Fig. 6. Conjunction of areas activated by error-feedback during IR, OA, and classic reversal learning. Images of the conjunction analysis of the error-feedback versus correct-feedback contrasts (threshold at pb0.005, correct to pb0.05) reveal a high degree of overlap in the bilateral dlpfc and dmpfc (left), bilateral IFG (middle), and bilateral frontal pole (right).

7 1438 S.G. Greening et al. / NeuroImage 54 (2011) Table 3 Conjunction of RR, IR, and OA error-feedback versus correct-feedback contrasts. Location R/L BA X Y Z IR+OA+RR IR+OA IR+RR OA+RR Error-feedback correct-feedback Frontal pole R Frontal pole L Dorsolateral PFC R 8/ ,360 11,400 14,968 Dorsolateral PFC L IFG/anterior insula R 47/ IFG/anterior insula L 47/ Dorsomedial PFC R/L 32/ ,672 10,360 Precentral gyrus L Inferior parietal lobe R 39/ , Inferior parietal lobe L ,576 a Precuneus R/L ,576 a Cuneus L Cuneus R/L Cuneus R Fusiform gyrus L Fusiform gyrus L Fusiform gyrus R Lingual gyrus R Lingual gyrus R/L Lingual gyrus R Thalamus R Thalamus L Correct-feedback Nerror-feedback Medial PFC R/L 32/ Conjunction of contrasts thresholded at pb0.005 and corrected to pb0.05. Numbers in the last four columns depict volume of overlap in cubic millimeters. a This is one contiguous cluster spanning the inferior parietal lobe and precuneus. feedback. Notably, we observed overlapping activity in only the IR+ RR combination, in a region of mpfc (BA 10/32; 400 mm 3 ). Discussion Recently, it has been demonstrated that two subprocesses of reversal learning are dissociable at the neurochemical level (Clarke et al., 2007). Marmosets with selective depletions of prefrontal levels of serotonin showed an impaired ability to inhibit responding to a previously rewarded stimulus. In contrast, the same animals retained the ability to overcome avoidance to a formerly punishing stimulus. This neurochemical manipulation likely influenced both medial and lateral regions of prefrontal cortex. A remaining question therefore was whether two subprocesses of reversal learning, inhibition of responding (IR) and overcoming avoidance (OA), are neuroanatomically dissociable processes in humans. In the current study, we adapted a similar behavioural manipulation for use fmri to address this unknown during the choice and feedback intervals of decision making. During the choice interval, differential effects of IR versus OA were observed. Relative to early response inhibition trials, early successful attempts to overcome avoidance were associated with significantly enhanced activity in right medial OFC, right frontal pole, cuneus, cerebellum, and inferior parietal cortex. Consistent with its role in encoding stimulus values, enhanced activity during the relearning phase was observed in left amygdala to all stimuli that underwent a contingency change relative to the control condition. Conjunction analysis was performed to quantify the extent of overlap in significantly active voxels across conditions. This confirmed that during the feedback interval, similar patterns of activity were observed across trials featuring classic reversals, response inhibition, and overcoming avoidance in dmpfc, dlpfc, IFG, frontal pole, and inferior parietal cortex. Conversely, the contrast of error-feedback versus correct-feedback displayed reduced activity during errorfeedback in the mpfc during the IR and not the OA condition. A final contrast uncovered enhanced activity in anterior dmpfc and angular gyrus during OA compared to IR error-feedback processing. The results indicate that overlapping regions of dmpfc, dlpfc, IFG, frontal pole, and parietal cortex are involved in both overcoming avoidance and response inhibition. However, they indicate that during the choice phase mofc and lateral frontal pole may play a greater role in overcoming the avoidance of a previously punishing stimulus than in inhibiting a response to a formerly rewarding object. Neuroimaging studies implicate multiple areas of prefrontal cortex in reversal learning including OFC (O'Doherty et al., 2001), IFG (Cools et al., 2002), dmpfc (Budhani et al., 2007; Mitchell et al., 2009), and dlpfc (Mitchell et al., 2009; Remijnse et al., 2005). Although it is increasingly apparent that these regions play some role in rewardrelated decision making, dissociating the function of each has proved difficult, particularly because even a relatively confined and controlled experimental task such as reversal learning includes several dissociable cognitive functions (Fellows, 2007). Perhaps the region of prefrontal cortex with the clearest link to reversal learning is OFC. Lesion data from human (Berlin et al., 2004; Fellows and Farah, 2003; Hornak et al., 2004; Rolls et al., 1994) and non-human (Dias et al., 1996) primates are consistent with suggestions that OFC plays a critical role in reversal learning. The OFC is thought to contribute to reversal learning and decision making more broadly by representing expected reward values or behavioural outcomes (O'Doherty, 2007), and updating the incentive value of stimuli when they change (Izquierdo et al., 2004; Rudebeck and Murray, 2008). A recent neuroimaging study has implicated OFC in modulating both approach and avoidant operant learning as a function of reinforcement change (Finger et al., 2008b). In the current study, we observed greater activity in a similar area of mofc (BA 10/ 32) during the choice interval of the OA condition relative to the IR condition. Deactivation in mofc in response to unexpected feedback has been interpreted in some cases as evidence of prediction error signaling, when expected reinforcement is not received (Budhani et al., 2007; Finger et al., 2008b; Tobler et al., 2006). However, in the current study, the effect concerns the choice phase of responses before feedback was obtained. One interpretation of this result is that the activity in the choice phase reflects the neuronal representation of the expected reward (Schoenbaum and Roesch, 2005; Tanaka et al., 2008). This is consistent with prediction error studies demonstrating a temporal shift in neuronal reward-related firing during learning from the feedback phase to the time that the cue associated with

8 S.G. Greening et al. / NeuroImage 54 (2011) reward is presented (O'Doherty et al., 2003b; Schultz et al., 2000). One might predict that such an outcome expectancy shift from feedback to cue would be least pronounced for the IR condition, which is the only condition that involves selecting a novel stimulus (i.e., novel stimuli have the least reward history and are therefore least likely to generate a strong outcome expectancy). In addition to activity in mofc during choice trials, and in line with previous studies of reversal learning (Budhani et al., 2007; Finger et al., 2008b), we observed significantly reduced activity in the mpfc (BA 32/10) in response to error-feedback compared to correct-feedback in the IR and RR conditions. The conjunction of the error-related contrast tests revealed that this effect was overlapping and significant only for error-feedback in the IR and RR conditions (Table 3). Consistent with a prediction error conceptualization, these are the only conditions involving errors to stimuli that were previously associated with reward (i.e., the conditions for which an expected reward was not obtained). Future work involving formal prediction error modeling will be required to determine the relative contribution of mpfc to inhibiting responding to previously rewarding stimuli versus overcoming avoidance to previously punishing stimuli. We also observed activity in superior lateral regions of frontal pole (BA 10). Although similar regions have been implicated in response change in previous studies (Finger et al., 2008b; Mitchell et al., 2009; 2008; O'Doherty et al., 2003a), this area of frontal pole has received less consideration than other regions of prefrontal cortex. Based on anatomical data, it has been noted that the frontal pole, with connections to dlpfc and sensory regions of temporal lobe, is well positioned to be involved in multi-tasking and complex planning (cf. Petrides, 2005). Functional studies have implicated similar or adjacent areas in exploration behaviour (Daw et al., 2006), monitoring or appraising feedback (Tsujimoto et al., 2010), and representing future alternative choices on the basis of past experience (Boorman et al., 2009). In this study, enhanced activity was revealed in the frontal pole during the RR and OA relative to IR conditions, particularly during the choice interval. In contrast to IR trials, which involve selecting a novel stimulus (i.e., one without a significant reward history), the RR, OA and CTL conditions all feature correct responses to stimuli with an established record of reinforcement values. Perhaps choices involving previously encountered stimuli with a significant reinforcement history involve greater participation from a system that evaluates the long-term evidence collected in favour of adapting future behaviour (cf. Boorman et al., 2009). In addition, we also observed enhanced activity to error-feedback in anterior dmpfc and angular gyrus during overcoming avoidance relative to response inhibition trials. Activity in these areas of prefrontal cortex and parietal cortex has been given less consideration in reversal learning. The region of dmpfc is anterior to that typically seen in decision making studies, but it has been observed when making decisions during a reward-related gambling task (Rogers et al., 2004). It is also noteworthy that areas of parietal cortex have been implicated in response change in other studies (Hampshire and Owen, 2006). The current study raises the possibility that these regions may play a particular role in processing feedback that signifies the need to overcome avoidance of a particular stimulus. Further study will be required to evaluate this possibility. Neuroimaging studies have also consistently linked IFG to reversal learning (Budhani et al., 2007; Cools et al., 2002; Mitchell et al., 2009; 2008; Nagahama et al., 2001). However, there are little data concerning the impact of focal lesions to this region on components of decision making, and so the relative importance and precise functional role of IFG in this context is less clear. Aron and his colleagues present compelling lesion data linking right IFG with deficits inhibiting a motor response on a go/no-go task (Aron et al., 2003). Nevertheless, IFG function has not only been conceptualized in terms of motor response inhibition (Aron et al., 2003; Casey et al., 2001), but also in selecting among competing response options (Budhani et al., 2007; Hampshire et al., 2009; Mitchell et al., 2009; Nagahama et al., 2001; Rushworth et al., 2005), and processing punishment information (O'Doherty et al., 2001). In the current study, there was enhanced IFG activity bilaterally to correct selections during the choice interval across early relearning stage trials. We also observed enhanced IFG activity following a sub-optimal response (i.e., during error-feedback) in both the IR and OA condition, as well as the classic response reversal condition. Together, these results are consistent with the idea that the IFG is involved in modulating the competitive weights of different stimulus response-mappings, rather than being involved in response inhibition or punishment processing per se (Hampshire et al., 2010; Mitchell et al., 2009; 2008; Nagahama et al., 2001). Indeed, such a conceptualization of function would account for results involving not only the current data, but also data gleaned from the stop signal task, reversal learning paradigms, and experiments examining punishment processing. Imaging studies also consistently implicate dorsal regions of the PFC, notably dmpfc and dlpfc, in reversal learning. Although lesions to these dorsal regions of prefrontal cortex do not affect reversal learning (Fellows and Farah, 2003), they do disrupt more complex, but related, forms of reward-related decision making such as risk aversion learning (Fellows and Farah, 2005; Manes et al., 2002). Recently, it has been shown that activity in overlapping areas of dlpfc and dmpfc are modulated by both reversal errors, and when correct selections are made between two response options of similar as opposed to widely differing values (Mitchell et al., 2009). These data are consistent with the idea that dorsal regions of prefrontal cortex may help resolve conflict whether it is perceptual conflict driven by opposing stimulus features (cf. Botvinick et al., 2004; Carter et al., 1998; Liu et al., 2006), or decision conflict generated by a similarity of reward values among competing response options (Blair et al., 2006; Mitchell et al., 2009; Pochon et al., 2008). In the current study, we observed enhanced activity in the dlpfc and dmpfc in response to early response conflict at the start of the relearning phase (i.e., during the choice interval) and to errorfeedback (i.e., during the feedback interval). A significant interaction revealed that, within dlpfc, this activity was greatest during the early stages of the RR condition, and least to the same condition during late trials relative to all other conditions. The results are consistent with the idea that activation in this area early on during periods of high decision conflict not only facilitates the resolution of conflict during early trials, but also that it does so in a manner that may lead to enhanced learning of contingencies, and greater ease of decision making in later trials. It is important to note that if this were the case, it could be predicted that the effect of stage (early versus late) would be greater on the RT to the RR condition relative to the other conditions. Although participants were significantly slower to respond correctly during the early versus late trials across conditions, the effect was not any greater for the RR condition. As a consequence, this potential interpretation concerning the impact of dlpfc activity on contingency learning remains speculative pending additional empirical consideration. Finally, in the context of reversal learning, the amygdala is thought to represent the learned incentive value of a reward and bias behaviour accordingly (Elliott et al., 2004; Izquierdo and Murray, 2007; Rudebeck and Murray, 2008). In line with this, an electrophysiology experiment involving rhesus monkeys revealed that cellular activity within the amygdala following stimulus-reward contingency change predicted subsequent behavioural change (Paton et al., 2006). Furthermore, Blair and colleagues (Budhani et al., 2007) observed enhanced amygdala activity to correct-feedback during reversal learning in a recent neuroimaging study. In line with these results, we observed enhanced amygdala activity during the early stages of relearning when reinforcement contingencies changed. This effect of reinforcement change was not modulated by whether the decision