Dissociable but inter-related systems of cognitive control and reward during decision making: Evidence from pupillometry and event-related fmri

Transcription

1 NeuroImage 37 (2007) Dissociable but inter-related systems of cognitive control and reward during decision making: Evidence from pupillometry and event-related fmri Theodore D. Satterthwaite, a Leonard Green, b Joel Myerson, b Jamie Parker, c Mohana Ramaratnam, b and Randy L. Buckner c,d, a Department of Psychiatry, University of Pennsylvania, Philadelphia, PA 19104, USA b Department of Psychology, Washington University, St. Louis, MO 63130, USA c Department of Psychology and Center for Brain Science, Howard Hughes Medical Institute, Harvard University, Cambridge, MA 02138, USA d Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA Received 3 January 2007; revised 13 April 2007; accepted 16 April 2007 Available online 24 May 2007 Decision making involves the allocation of cognitive resources in response to expectations and feedback. Here we explored how frontal networks respond in a gambling paradigm in which uncertainty was manipulated to increase demands for cognitive control. In one experiment, pupil diameter covaried with uncertainty during decision making and with the degree to which subsequent outcomes violated reward expectations. In a second experiment, fmri showed that both uncertainty and unexpected outcomes modulated activation in a network of frontal regions. Thus, the frontal network supports multiple phases of the decision-making process including information regarding reward uncertainty and reward outcome. In contrast, striatal activation only tracked reward delivery, suggesting a distinct reward pathway that might, under certain circumstances, oppose the frontal network. These results are consistent with the interpretation that reward signals may bias recruitment of frontal networks that are linked to allocation of cognitive resources Published by Elsevier Inc. Keywords: Cognitive control; Reward; Decision making; Prefrontal; fmri; Pupillometry Introduction Decision making under uncertainty recruits a distributed network of frontal regions (Botvinick et al., 2001; Huettel et al., 2005; Volz et al., 2005). Lateral and midline frontal regions found to be active in neuroimaging studies of decision making are similar to those observed in other paradigms that require controlled processing. Activation in these frontal regions typically increases Corresponding author. HHMI/MGH Martinos Center, th St., Rm. 1115C, Charlestown, MA 02129, USA. address: rbuckner@wjh.harvard.edu (R.L. Buckner). Available online on ScienceDirect ( with task difficulty (Duncan and Owen, 2000; Paus et al., 1998). Moreover, frontal networks involved in cognitive control may be biased by dopamine (DA) neuromodulatory signals that may direct the allocation of cognitive resources (Cohen et al., 2002; Holroyd and Coles, 2002; Schultz, 2002). In an elegant series of studies, Schultz and colleagues demonstrated that midbrain DA neurons respond to rewards in a variety of contexts, exhibiting phasic firing patterns that code for prediction errors in response to discrepancies between reward expectation and delivery (Fiorillo et al., 2003; Hollerman and Schultz, 1998; Schultz et al., 1997). The finding of phasic DA firing in animal studies has motivated several neuroimaging studies in humans. These imaging studies have consistently observed reward-related activity in the striatum (for reviews, see Knutson and Cooper, 2005; McClure et al., 2004), which are known to receive significant DA projections from the midbrain (Lidow et al., 1989; Lynd-Balta and Haber, 1994). Responding for rewards increases [11C] raclopride (a dopamine marker) binding in the striatum (Koepp et al., 1998; Zald et al., 2004). Moreover, striatal activity increases in response to rewarding stimuli such as cocaine and amphetamine infusion (Breiter et al., 1997; Knutson et al., 2004), beautiful faces (Aharon et al., 2001), juice (Berns et al., 2001; McClure et al., 2003; O'Doherty et al., 2002) and monetary rewards (Breiter et al., 2001; Delgado et al., 2003; Dreher et al., 2006; Elliott et al., 2000; Haruno et al., 2004; Knutson et al., 2000). Consistent with Schultz et al. (1997), imaging experiments have found reward prediction error signals in the putamen (McClure et al., 2003) and nucleus accumbens (O'Doherty et al., 2003). However, there are little data concerning how networks involved in cognitive control interact with reward prediction and error signals during decision making. Understanding this interaction is important because it has been proposed that dopaminergic reward pathways may serve as motivating and learning signals in the dynamic allocation of resources during decision making /$ - see front matter 2007 Published by Elsevier Inc. doi: /j.neuroimage

2 1018 T.D. Satterthwaite et al. / NeuroImage 37 (2007) (Cohen et al., 2002; Holroyd and Coles, 2002; Montague et al., 2004; Schultz, 2002). Here we used pupillometry and functional MRI (fmri) to examine the response of control networks in a gambling paradigm similar to that employed by Critchley et al. (2001). Gambling paradigms have been used to investigate human decision making (Breiter et al., 2001; Critchley et al., 2001; Delgado et al., 2000; Elliott et al., 2000; Rogers et al., 1999), in part because they enable researchers to manipulate both the demands for cognitive control and the level of reward expectation. In the current study, subjects made decisions under varying levels of reward uncertainty and received monetary feedback that confirmed or violated these expectations. This design has the advantage of being able to dissociate processes of cognitive control that are associated with decision uncertainty from reward processes associated with trial outcome (Maunsell, 2004). We hypothesized that both the uncertainty of decisions and the degree to which outcomes violate expectations would be associated with increased activity in frontal control networks. In contrast, as has been well established in the literature, we expected to confirm that activity in subcortical reward pathways would depend primarily on reward outcome. Dissociation of the effects of reward and increased demands for cognitive control should be most apparent on trials involving an unexpected loss, which would lead to augmented frontal recruitment. Interpretation of the results of the present study may be complicated by the fact that uncertainty and expected value covaried because the level of reward expectation was manipulated while holding the amount of reward constant. Because the expected value of a reward is its amount times its probability, it follows that increasing uncertainty by decreasing reward probability lowers expected value. However, Knutson et al. (2005) have shown that activity in medial prefrontal regions is positively correlated with overall expected value. Therefore, one would expect that if expected value is the dominant factor, then activity in these regions should be depressed on uncertain (low expected value) trials, whereas if uncertainty is the critical factor in this paradigm, as we hypothesize, then activity should increase. To gain insight into the temporal dynamics of resource allocation during our gambling paradigm, in a separate group of subjects we monitored changes in pupil diameter over the course of each trial. Pupil diameter has long been known to provide a nonspecific measure of cognitive effort with high temporal resolution (for a review, see Beatty and Lucero-Wagoner, 2000). The temporal resolution of pupillometry was important for this study because it allowed examination of the temporal dynamics of the gambling task in a way that is not possible using fmri alone. Although the mechanism by which pupil diameter tracks task demands remains uncertain (but see Aston-Jones and Cohen, 2005), neuroimaging studies have demonstrated that changes in pupil diameter correlate with activity in prefrontal regions involved in cognitive control (Critchley et al., 2005; Siegle et al., 2003). In the present study, pupillometry demonstrated effects of control during temporally separate decision and feedback epochs, while event-related fmri revealed neural correlates of cognitive control that were distinct from regions that respond to reward outcomes. These results reveal a dissociation between a striatal reward system and a frontal control network, consistent with the theory that dopaminergic reward pathways provide a modulatory gating signal to a frontal network of cognitive control (Holroyd and Coles, 2002; Montague et al., 2004). Materials and methods Subjects Thirty-three subjects participated in the pupillometry experiment. All had normal or contact-lens-corrected vision. One subject was excluded from the analysis due to an error in data acquisition, leaving 32 subjects (mean age 20.1 years; 16 males). Twenty-seven subjects participated in the fmri experiment. All were right handed, had no history of neurologic injury, and had normal or corrected vision. One subject was excluded from the analysis due to an error in data acquisition, leaving 26 subjects (mean age 23.8 years; 15 males). Task and stimuli Subjects in both the pupillometry and fmri experiments performed a gambling task (see Fig. 1). Stimuli were color images of standard playing cards from all four suits. Each trial began with one card presented face-up and one card presented face-down. Subjects had to predict which of the two cards would turn out to be higher when the face-down card was turned over. Each trial lasted 4.72 s and was composed of two 2.36-s epochs. In Epoch 1, the decision epoch, the two cards appeared on the screen and subjects made their choice. Card values ranged from 2 to Ace (Ace high). Subjects were told that 8 was the middle card and that suit was of no consequence. The face-up card randomly appeared on either the left or right. At the bottom of the screen were two boxes indicating the amount of money (50 cents) to be won or lost. A fixation crosshair was positioned midway between the two cards and stayed on for the duration of the task. Subjects had 2 s to select the card they predicted would be higher by pressing a button corresponding to the side of the chosen card. Subjects were instructed to respond quickly and accurately and were told that they would be penalized $1 for not responding within 2 s. Following its selection, the chosen card was highlighted with a black border. At the start of Epoch 2, the feedback epoch (2.36 s into the trial), the face-down card was turned over and the outcome was displayed. Either +50 in the top box was highlighted in green, indicating a win, or 50 in the lower box was highlighted in red, indicating a loss. The outcome stimuli remained on the screen for 1.7 s, after which the cards and outcome boxes were removed, leaving only the fixation cross-hair for an inter-trial interval of 0.66 s. The subject's total score was not displayed on the screen at any time. There were three levels of reward uncertainty, determined by the value of the face-up card: uncertain (cards 6, 7, 8, 9, and 10; averaged across these cards, probability of reward = 0.60); probable (cards 3, 4 5, J, Q, K; averaged across these cards, probability of reward =0.83); and certain (for cards 2 and Ace, probability of reward = 1.0). It should be noted that whereas reward probability ranged from 0.5 (if the face-up card was an 8) to 1.0 (if the face-up card was a 2 or an Ace), reward uncertainty ranged from complete uncertainty (when the probability of a win was equal to chance) to complete certainty (when the probability of a win was equal to 1.0). To ensure adequate power, trials were distributed among the levels of reward uncertainty so that there was a minimum of 20 trials per trial type and outcome (e.g., probable trials that ended in wins, probable trials that ended in losses, etc.). Thus, there were more probable trials so that there would be an adequate number of trials that ended in a loss and violated reward expectation. There were 60 uncertain, 120 probable, and 20 certain trials, for an

3 T.D. Satterthwaite et al. / NeuroImage 37 (2007) Fig. 1. Experimental paradigm. Each trial consisted of two 2.36-s epochs. In the decision epoch, two cards were displayed on the screen, one face-up and the other face-down, and subjects selected the card they believed would turn out to be of higher value. Cards were grouped into three trial types based on the uncertainty of reward as determined by the value of the face-up card: uncertain trials (average probability of reward=60%), probable trials (average probability of reward=83%), and certain trials (100% probability of reward). Subjects had 2 s to make their selection. The card they selected then was surrounded by a black border. In the feedback epoch, the face-down card was turned over and the outcome revealed. Subjects won $0.50 for each correct choice and lost $0.50 for each incorrect choice. average probability of reward of There were also 40 fixationonly trials in which no cards appeared on the screen (which were used to induce jitter; Buckner et al., 1998a), resulting in 240 trials in all. Subjects completed the 240 trials in four runs of 60 trials each, with a short break between each run. The order of trial types was arranged pseudorandomly to allow for hemodynamic signal extraction (Buckner et al., 1998a; Dale and Buckner, 1997). The pairs of cards that were presented on each trial were randomized with the following constraints: no card pairing occurred more than four times, and no ties between the face-up and face-down card were allowed. Prior to beginning, subjects were given detailed instructions including three sample and 10 practice trials. Subjects in the pupillometry experiment were told that they would receive their winnings; subjects in the fmri experiment were told that they would receive their winnings and an additional $25 for participation. For pupillometry, the experiment was run in a dim, quiet room, and subjects wore headphones to attenuate noise. Stimuli were presented using an Apple Power Macintosh G4 (Apple Computers, Cupertino, CA) running PsyScope software (Cohen et al., 1993) and displayed on a 21-in. monitor. Subjects were seated 34 in. from the monitor with their head position stabilized by an adjustable headrest so that stimuli subtended 12 of visual angle. The background of the monitor was white to minimize relative luminance changes across the trial. Subjects responded via a PsyScope button box (Carnegie Mellon University, Pittsburgh, PA). For fmri, procedures were similar to those above, except that stimuli were projected onto a screen (Sharp LCD PG-C20XU; Sharp, Mahwah, NJ) positioned at the head of the magnet bore. Subjects viewed the screen using a mirror attached to the head coil. Subjects responded via fiber-optic, light-sensitive keys interfaced with the PsyScope button box. For both the pupillometry and fmri experiments, the percentage of trials on which the face-down card was chosen was calculated for each value of the face-up card. In addition, response times were recorded and subjected to repeated measure ANOVAs with post hoc t-tests in order to evaluate the effect of level of uncertainty (i.e., uncertain, probable, certain). Pupillometry The subject's left pupil diameter was monitored with a tablemounted infrared eye-tracking system (Model 504, Applied Science Laboratories, Bedford, MA) using a sampling rate of 60 Hz. Pupillary measures were routed through a general physiologic interface (MP150, Biopac Systems, Goleta, CA) to allow for time locking of stimulus, response, and pupillometry data. Measures of pupil diameter were processed with software written in-house using Matlab 7 (Mathworks, Natick, MA). First, values falling below a threshold of 1 mm, as in the case of blinks, were linearly interpolated. In the linear interpolation, three samples on either side of each below-threshold value were skipped, and the

4 1020 T.D. Satterthwaite et al. / NeuroImage 37 (2007) slope for the linear interpolation was calculated using a window of 10 samples (167 ms) both before and after the interpolated window. Second, trials were averaged for each subject to produce an average waveform for each of the 5 trial types: wins and losses on uncertain trials, wins and losses on probable trials, and wins on certain trials. Third, differences in baseline among subjects were normalized via linear subtraction of the difference between the individual and across-subjects average pupil diameter over the first 30 samples (500 ms) of each trial type. This normalization corrected for differences in pupil sizes among subjects. Normalized individual subject waveforms then were averaged across all subjects to produce an average waveform for each trial type. As a final step, in order to remove baseline differences among trial types that presumably were due to slow drifts, data were normalized based on the 30 samples (500 ms) following the beginning of the epoch of interest (i.e., decision or feedback) using a linear subtraction of the difference between the average for the trial type and the overall (grand) mean. Differences between pupil diameters on uncertain and certain trials, probable and certain trials, and between wins and losses on both probable and uncertain trials, were calculated at each time point and subjected to paired t-tests (α = 0.05). This analysis has the advantage of not assuming a specific shape for the pupillary response waveform. The results indicated that these difference waveforms were very stable. fmri Functional imaging was conducted using a 1.5-T Magnetom Vision MRI system (Siemens, Erlangen, Germany). Foam pillows and a thermoplastic facemask were used to minimize head movement. Headphones dampened scanner noise and allowed communication with subjects. Structural images were acquired using a sagittal magnetization preparation rapid acquisition gradient echo (MP-RAGE) T1-weighted sequence [repetition time (TR), 9.7 ms; echo time (TE), 4 ms; flip angle, 10 ; inversion time, 20 ms; delay time, 200 ms]. Whole-brain functional images (sixteen 8-mm thick transverse slices) were collected in four functional runs using an asymmetric spin-echo echo-planar sequence sensitive to blood oxygen level-dependent (BOLD) contrast [T2 ; TR, 2.36 s; TE, 37 ms; mm in-plane resolution (Kwong et al., 1992; Ogawa et al., 1992). The first four images in each run were discarded from functional analysis to allow for magnetization to stabilize. The first image, because of its T1 weighting, was used to align the data to the high-resolution T1-weighted anatomical image. fmri data analysis A series of processing steps, similar to those used in previous studies in our laboratory (Shannon and Buckner, 2004; Velanova et al., 2003), was performed. The fmri data were first corrected for odd even slice intensity differences and then motion corrected using a rigid-body rotation and translation correction (Snyder, 1996). Between-slice timing differences caused by slice acquisition order were adjusted using sync interpolation. Linear slope was removed on a voxel-by-voxel basis to correct for drift. Data were normalized using a mean magnitude value of 1000 to facilitate comparisons across subjects. Data were transformed into atlas space (Talairach and Tournoux, 1988) using 2-mm isotropic voxels (Maccotta et al., 2001), and smoothed with a Gaussian spatial filter (2-mm FWHM). Preprocessed data were analyzed using the general linear model (Friston et al., 1995; Miezin et al., 2000; Worsley and Friston, 1995) implemented using an in-house analysis and display package. For each subject, the BOLD response on each trial type was estimated by coding a different regressor for each of the eight TRs (16.52 s total) including and immediately after stimulus onset. This estimation produced one time course estimate per voxel per trial type. Generation of activation maps Contrasts of interest were regressed against temporally delayed boxcars convolved with a gamma function (Boynton et al., 1996) that approximated the range of hemodynamic responses typically encountered (Buckner et al., 1998b; Schacter et al., 1997). Four delays were modeled: 0 s, 1 s, 2 s, and 3 s. Statistical activation maps then were constructed for each contrast on a voxel-by-voxel basis using a t statistic. Whole-brain activation maps were generated by converting the t statistics to z statistics and then plotting them across the brain. Hypothesis-driven a priori regional analyses Our primary hypotheses in the present study concerned how control networks respond under conditions of uncertainty and unexpected feedback. To focus our analyses and increase statistical power, we initially targeted a small number of a priori regions. Hypothesis-driven analyses using a priori regions are particularly effective because they increase power by decreasing the number of multiple comparisons and also because of averaging across many voxels within each region. In a second stage of analysis (described below), we carried out voxel-wise exploration of the whole brain. Whole-brain analyses allow full exploration of the functional data but are less powerful than directed region-based analyses because they require correction for multiple comparisons and do not benefit (beyond smoothing) from signal averaging across the voxels. This form of two-step analysis has proven useful in prior studies (e.g., Buckner et al., 1996; Gold and Buckner, 2002; Velanova et al., 2003; Wheeler and Buckner, 2004). As the results will show, the effects concerning the control network's role in multiple components of decision making are converged upon by both the a priori regional analyses and the whole-brain exploratory analyses, providing considerable confidence in their robustness. Specific regions of interest associated with cognitive control were selected a priori on the basis of literature reviews and recent work in our laboratory. We defined regions of interest based on activation peaks from a whole-brain map comparing all task trials versus fixation trials. This contrast was orthogonal to our hypotheses (i.e., differences between levels of uncertainty and outcome) in that it created an unbiased activation map of voxels that responded significantly to any aspect of the task. In turn, this allowed the creation of a priori ROIs that contained only taskresponsive voxels, while avoiding the confound of selecting regions based on a map that was biased by differences between trial types. Regions of interest consisted of all voxels above a threshold of z=3.00 within a radius of 10 mm of the peaks of activation selected. One potential limitation of the present method of regional definition is that it might miss regions that showed complex response properties that go below baseline in one or more of the conditions. However, as described below, such regions did not emerge in the whole-brain exploratory analyses.

5 T.D. Satterthwaite et al. / NeuroImage 37 (2007) Fig. 2. Pupil diameter responds to level of uncertainty and to violations of expectations. (A) Mean percentage choice of the face-down card as a function of the value of the face-up card. (B) Mean response times (i.e., decision latencies) for each value of the face-up card. Color bars on the x-axis indicate the three levels of uncertainty based on the value of the face-up card: uncertain (blue), probable (black), and certain (red). In panels A and B, the error bars indicate the standard error of the mean (SEM). (C) Mean pupil diameter on uncertain, probable, and certain trials. (D) Difference in pupil diameter between the uncertain and certain waveforms and the difference between the probable and certain waveforms. (E) Mean pupil diameter on uncertain trials that resulted in wins and uncertain trials that resulted in losses. (F) Mean pupil diameter on probable trials that resulted in wins and probable trials that resulted in losses. (G) Difference in pupil diameter between the win and loss waveforms for probable trials and for uncertain trials. In panels C through G, the plots depict mean pupil diameter as a function of time since the beginning of a trial. Dashed vertical lines at 2.36 s indicate the start of the feedback epoch, and grey envelopes indicate the SEM. Grey bars with stars along the x-axis in panels D and G indicate significant differences between the trial types.

6 1022 T.D. Satterthwaite et al. / NeuroImage 37 (2007) above a z = 3.00 threshold. Although magnitudes were estimated for exploratory regions in an identical fashion to that described above, statistical analyses are not possible for these exploratory regions because they already had been selected on the basis of significant differences using whole-brain contrasts. Results Choice proportions and response times Fig. 3. Behavioral results from the fmri experiment. (A) Mean proportion of trials on which the face-down card was chosen as a function of the value of the face-up card. (B) Mean response times (i.e., decision latencies) as a function of the value of the face-up card. Color bars on the x-axis indicate the three levels of uncertainty based on the value of the face-up card: uncertain (blue), probable (black), and certain (red). The error bars indicate the standard error of the mean (SEM). For a priori regional analysis, magnitude estimates, averaged over the region volume, were calculated for each subject and for each trial type. This approach affords considerable power by reducing the number of comparisons and by averaging the many voxels within each region, thereby increasing the signal-to-noise ratio. More specifically, estimates of response magnitude were calculated by subtracting the mean signal at times 0 s and s (representing the baseline) from the mean signal at 7.08 s (representing the peak). The magnitude estimates for each subject were entered into a random-effects model, and specific comparisons were made using a 2 2 repeated measures ANOVA (uncertain and probable win and loss) with post hoc t tests. Because this analysis makes strong a priori assumptions, we also performed exploratory analyses that surveyed the entire brain but which could not yield unbiased estimates of regional effects. Exploratory whole-brain analyses Maps of voxel-wise activity change were constructed in an exploratory manner in order to investigate other potential cognitive control and reward-related areas. A whole-brain contrast of uncertain win trials minus probable win trials was created to identify further regions recruited by uncertainty. This contrast was limited to trials involving wins in order to avoid the potential confound of including differential proportion of wins and losses under different levels of uncertainty. As with those regions defined a priori, frontal exploratory regions consisted of all voxels within 10 mm of the peak that surpassed a threshold of z=3.00. In order to confirm previously observed reward-responsive regions in the striatum that were not explicitly explored in the a priori analyses that focused on control processes, the whole-brain contrast of probable win trials versus probable loss trials was created. Regions within the striatum were subsequently defined using voxels within an 8-mm radius around these peaks that were In both the pupillometry and fmri experiments, subjects choosing between the face-up and face-down cards nearly always selected the card with the higher probability of reward (Figs. 2A and 3A). The amount of time it took them to make their selection increased as a function of the uncertainty of the outcome (Figs. 2B and 3B, and Table 1). Response times (RTs) on uncertain trials were slower than RTs on probable trials [pupillometry: t(31) = 17.63, p b 0.001; fmri: t(25) = 11.84, p b 0.001], which in turn were slower than RTs on certain trials [pupillometry: t(31) = 2.28, p b 0.05; fmri: t(25) = 5.16, p b 0.001]. On average, subjects in the pupillometry experiment won $55.02, and subjects in the fmri experiment won $ Pupil diameter Pupil diameter tracked uncertainty in the decision epoch and violations of expectations in the feedback epoch. Paralleling the RT data, greater pupil dilation was observed on uncertain trials than on certain trials (Fig. 2C). Examination of the plot of the difference between the certain waveform and the uncertain and probable waveforms (shown in Fig. 2D) reveals that this divergence began in the decision epoch, before the outcome was revealed. Paired t- tests at each time point detected a difference between the uncertain and certain waveforms from 1.45 s onward. During the feedback epoch, a difference between dilation on win trials and loss trials emerged that was modulated by the uncertainty of the outcome: losses provoked a greater dilation than wins on both uncertain and probable trials (Figs. 2E and F), but this difference was greater on probable trials when the loss was relatively unexpected. This may be seen most clearly in the plot of the difference between losses and wins in probable and uncertain trials (shown in Fig. 2G). The divergence between win and loss waveforms did not begin until after the outcome was displayed. As revealed by paired t-tests, the difference between wins and losses was greater on probable trials than on uncertain trials beginning at 3.55 s. Note that the effect of uncertainty in the decision epoch and the effect of violating expectations in the feedback epoch are temporally distinct but interact: in the decision epoch, greater uncertainty led to greater dilation, but it also presumably resulted in weaker expectations. These weaker expectations, in turn, resulted in less dilation in the feedback epoch of uncertain trials on which expectations were violated. The high temporal resolution Table 1 Mean response times (RTs) Uncertainty of reward Pupillometry RT (ms) fmri RT (ms) Certain Probable Uncertain

7 T.D. Satterthwaite et al. / NeuroImage 37 (2007) afforded by pupillometry makes it possible to observe this interaction. Imaging does not provide such temporal resolution, but fmri does permit the examination of neural correlates of the underlying decision processes. Hypothesis-driven a priori regional analyses A priori regions of interest were created from a whole-brain map contrasting all trials with fixation trials (see Fig. 4). From this map, regions were defined based on expectations of a frontal network previously implicated in cognitive control by a broad array of studies. The studies motivating our a priori selections evaluated control in the context of memory paradigms (Buckner, 2003; Velanova et al., 2003; Wheeler and Buckner, 2003), semantic judgments (Gold and Buckner, 2002), task-switching or Stroop paradigms (Derrfuss et al., 2004), prediction tasks (Paulus et al., 2002; Volz et al., 2003), working memory tasks (Jansma et al., 2001), and response inhibition tasks (Wager et al., 2005). As noted by Duncan and Owen (2000), these studies and others demonstrate a network of frontal regions that respond to task difficulty. We hypothesized that uncertainty in the decision epoch (Huettel et al., 2005; Volz et al., 2003) and unexpected outcomes in the feedback epoch (Brown and Braver, 2005; Holroyd et al., 2004) would engage this network. Specifically, we predicted that such effects would be seen in the bilateral prefrontal region (LPFC 45, 3, 28; RPFC at 43, 7, 30). This region, which sits at the junction of BA 6, 9, and 44 and is alternately referred to as posterior inferior prefrontal cortex (Gold and Buckner, 2002) or inferior frontal junction (Brass et al., 2005), has been shown to be of particular importance across a wide range of cognitive control paradigms in a recent meta-analysis (Derrfuss et al., 2004). A different meta-analysis (Ridderinkhof et al., 2004) has demonstrated robust effects of control in medial frontal regions in a variety of performance monitoring tasks; we predicted a region corresponding to the pre-sma at dorsal BA 6 (3, 15, 46) would display similar effects of control to those of bilateral prefrontal regions. Other nearby regions, such as the anterior cingulate, were not initially selected in order to reduce the number of a priori comparisons (see exploratory analyses below). We hypothesized that these frontal regions (shown in Fig. 5A) would respond to uncertainty and to violations of expectations. The imaging results confirmed our hypotheses (Figs. 5B and C). As was the case with the pupillometry data, there was a greater response on uncertain trials than on probable or certain trials in all three regions (Fig. 5B). Furthermore, there was an effect of trial outcome, with losses producing a greater response than wins (Fig. 5C). These main effects were statistically significant for all three a priori regions (Table 2). Moreover, even though the present study was underpowered for detecting an interaction between uncertainty (probable versus uncertain) and outcome (win versus loss), this interaction (Fig. 5D) reached the trend level for the RPFC as well as for the average of the three regions of interest. This trend echoes the findings in pupillometry, with unexpected losses in the probable reward condition producing a greater response than losses in the uncertain reward condition. Exploratory whole-brain analyses Exploratory analyses were conducted to identify additional regions involved in cognitive control and the processing of reward information (Fig. 6). Three regions that have been frequently cited in the literature as being involved in control processes were selected for further examination (Fig. 7A): dacc (Bush et al., 2002; Holroyd et al., 2004; Kerns et al., 2004; Ridderinkhof et al., 2004) and the right and left anterior insula (Derrfuss et al., 2004; Huettel et al., 2005; Ullsperger and von Cramon, 2003). The pattern of response in these regions (Fig. 7) was remarkably similar to the patterns of response in the a priori regions and in the pupillometry data. All three regions explored demonstrated a greater response on uncertain trials compared to probable and certain trials (Fig. 7B), and losses provoked a greater response than wins (Fig. 7C). As with the pupillometry data, the difference between wins and losses in the imaging data suggests that the uncertainty of the outcome modulates this response in that there appears to be a larger difference between wins and losses on probable trials than on uncertain trials (Fig. 7D). The dorsal striatum has been shown to be reward-responsive in previous studies (McClure et al., 2003; O'Doherty et al., 2002). Consistent with these studies, two regions in the dorsal striatum were significant in the present analysis (Fig. 8). Regions defined about these activations were found to have response properties quite distinct from frontal regions involved in cognitive control. Neither region showed an initial response to uncertainty (Fig. 8B), but both regions showed the expected effect of wins N losses (Fig. 8C). Wins resulted in sustained activation, whereas losses led, if anything, to a decrease below baseline following an initial peak (Figs. 8C and D). Full listings of coordinates of activation in the exploratory images are available upon request. Discussion The present study investigated the neural processes underlying decision making in a gambling paradigm. In one experiment, the Fig. 4. Statistical activation maps based on the contrast of all trials versus fixation trials. Left is displayed on the left. Transverse sections represent sections within the atlas coordinate framework of Talairach and Tournoux (1988).

8 Fig. 5. Frontal regions selected a priori respond to uncertainty and violation of expectations. (A) A priori regions of interest defined based on the contrast of all trials versus fixation trials (see Fig. 4): bilateral pre- SMA (top row), right PFC (middle row), and left PFC (bottom row). (B) BOLD response as a function of time for uncertain, probable, and certain trials that resulted in wins (left column) and uncertain and probable trials that resulted in losses (right column). (C) BOLD response as a function of time for probable (left column) and uncertain (right column) trials that resulted in wins and losses. (D) Estimates of response magnitude for different trial types. Error bars indicate the SEM. Dashed horizontal lines indicate the magnitude estimates for wins on certain trials T.D. Satterthwaite et al. / NeuroImage 37 (2007)

9 T.D. Satterthwaite et al. / NeuroImage 37 (2007) Table 2 A priori regions of interest Region Pre-SMA Uncertainty Outcome Uncertainty* Outcome R prefrontal Uncertainty Outcome Uncertainty* Outcome L prefrontal Uncertainty Outcome Uncertainty* Outcome Average of a priori regions Uncertainty Outcome Uncertainty* Outcome Magnitude ANOVA F(25,1) =43.90; pb F(25,1) =14.43; pb0.001 F(25,1) =1.83; p=0.18 F(25,1) =15.04; pb0.001 F(25,1) =7.07; pb0.01 F(25,1) =3.74; p=0.07 F(25,1) =11.40; pb0.01 F(25,1) =5.31; p=0.03 F(25,1) =2.04; p=0.17 F(25,1) =31.93; pb F(25,1) =12.98; pb0.001 F(25,1) =3.36; p=0.08 temporal dynamics of cognitive resource allocation were studied using pupillometry. We found separate phasic responses during the decision and feedback epochs. Robust pupil dilation, indicative of increased cognitive effort, occurred in response to uncertainty early during the decision epoch and in response to unexpected losses during the feedback epoch. The fmri results from a second experiment using the same gambling paradigm revealed a similar pattern, with a common frontal network that was activated by both uncertainty and unexpected losses. In contrast, a reward system involving the dorsal striatum showed increased activity only when rewards were delivered during the feedback epoch, regardless of the uncertainty of the reward. As hypothesized, this dissociation between networks of control and reward was most apparent on trials where a surprising loss produced a reduction of activity in the dorsal striatum in contrast to robust recruitment of frontal regions. These findings suggest a system whereby striatal reward pathways may bias recruitment of a network of frontal regions involved in cognitive control. We elaborate on these results and their implications below. Evidence for temporally distinct stages of increases in cognitive effort Pupil diameter, used as a measure of cognitive effort, showed temporally distinct increases in response to uncertainty during the decision epoch and in response to outcomes during the feedback epoch. In the decision epoch, pupillary dilation was greatest when the outcome was uncertain (see Fig. 2). The difficulty of decision making increased with uncertainty, as evidenced by the slower response times for the more uncertain decisions (see Figs. 2B and 3B, Table 1). Thus, decision difficulty and decision uncertainty were highly related. The finding that phasic pupillary dilation varied with uncertainty (Figs. 2C and D) is consistent with the view that pupil dilation is a nonspecific indicator of cognitive effort (Beatty, 1982; Kahneman, 1973). In the current paradigm, decision uncertainty and low expected value were correlated, and thus one cannot exclude the possibility that low expected value influenced pupillary dilation in the decision epoch. However, the present results are consistent with the ample literature demonstrating pupillary responses to cognitive demands across a wide range of tasks (for review, see Beatty and Lucero-Wagoner, 2000). In the feedback epoch, there was an interaction between uncertainty and outcome (see Figs. 2E, F): dilation following an unexpected loss (i.e., a loss on probable trials where mean reward probability was 0.83) was greater than dilation following a more likely loss (i.e., a loss on uncertain trials where mean reward probability was 0.60). Prior studies have found that error trials in a Stroop task lead to greater pupillary response (Critchley et al., Fig. 6. Results of whole-brain exploratory analyses. (A) Statistical activation maps based on the contrast of uncertain wins versus probable wins. Exploratory regions of interest in dacc and bilateral anterior insula were defined from these maps. (B) Statistical activation maps based on the contrast of probable wins versus probable losses. Exploratory regions of interest in bilateral putamen were defined from these maps. Full coordinate listings are available upon request.

10 Fig. 7. Exploratory regions also respond to uncertainty and violation of expectations. (A) Exploratory regions of interest defined based on the contrast of uncertain wins versus probable wins (see Fig. 6A): dacc (top row), right anterior insula (middle row), and left anterior insula (bottom row). (B) BOLD response as a function of time for uncertain, probable, and certain trials that resulted in wins (left column) and uncertain and probable trials that resulted in losses (right column). (C) BOLD response as a function of time for probable (left column) and uncertain (right column) trials that resulted in wins and losses. (D) Estimates of response magnitude for different trial types. Error bars indicate the SEM. Dashed horizontal lines indicate the magnitude estimates for wins on certain trials T.D. Satterthwaite et al. / NeuroImage 37 (2007)

11 Fig. 8. Dorsal striatum responds to reward outcome only. (A) Exploratory regions of interest defined based on the contrast of probable wins versus probable losses (see Fig. 6B): right putamen (top row) and left putamen (bottom row). (B) BOLD response as a function of time for uncertain, probable, and certain trials that resulted in wins (left column) and uncertain and probable trials that resulted in losses (right column). (C) BOLD response as a function of time for probable (left column) and uncertain (right column) trials that resulted in wins and losses. (D) Estimates of response magnitude for different trial types. Error bars indicate the SEM. Dashed horizontal lines indicate the magnitude estimates for wins on certain trials. Note the difference in response pattern compared to that seen in regions displayed in Figs. 5 and 7. T.D. Satterthwaite et al. / NeuroImage 37 (2007)

12 1028 T.D. Satterthwaite et al. / NeuroImage 37 (2007) ). The current study extends such results by suggesting that cognitive effort in the feedback epoch tracks the difference between expectations and outcomes (i.e., a prediction error) and thus may reflect the occurrence of reevaluation or other forms of attentional increase following unexpected feedback. Pupillary diameter is known to increase in response to highly arousing stimuli (e.g., pain; Ellermeier and Westphal, 1995). According to prospect theory (Kahneman and Tversky, 1979), losses have a steeper valuation curve than wins. It may be that a loss is more arousing than a gain of equivalent amount, which may account for the greater pupillary dilation we observed following losses. However, this effect alone is unlikely to account for the observed modulation of the response to losses by uncertainty, in which unexpected losses produced a significantly greater pupillary response (Figs. 2E G). Nonetheless, further study will be required to fully exclude such effects in experiments specifically designed to investigate predictions of prospect theory. The physiological process by which pupil diameter comes to reflect cognitive effort is unclear. However, prior studies have shown that pupillary changes are correlated with prefrontal cortex activity (Critchley et al., 2005; Siegle et al., 2003). It has been proposed that pupil diameter may index locus coeruleus activity (Aston-Jones and Cohen, 2005), but the mechanism relating pupillary control and prefrontal cortex activity has not been established. The present study demonstrates the utility of pupillometry for dynamically tracking task-induced demands for cognitive processing, highlighting its value as a non-invasive and relatively inexpensive method for assessing cognitive effort. In particular, it is a valuable complement to fmri, which provides anatomic localization but lacks temporal resolution. A frontal network tracks cognitive effort In the imaging experiment, fmri revealed a pattern of frontal activation in which a convergent network responded to both reward uncertainty and reward outcome, paralleling the responses observed in the pupillometry experiment. Bilateral prefrontal cortex (BA 6/9/44) and pre-sma (dorsal BA 6) showed activation that increased as a function of decision uncertainty and also as a function of the outcome. Losses, which for the most part would have been unexpected (the average probability of reward across all trials was 0.78), produced a greater response than wins, which were usually expected. The interaction between uncertainty and outcome seen in pupillometry also was apparent in the fmri data as a trend in the average activation of the three a priori regions (LPFC, RPFC, and pre-sma) as well as in the RPFC alone. In addition, the same pattern was observed in exploratory analyses of the dacc (BA 32) and bilateral anterior insula. Considered in the context of the pupillary response, the fmri results suggest that the frontal network is recruited in response to different task demands. Studies have consistently shown that the above regions are components of a frontal network that is activated during tasks requiring cognitive control in response to task difficulty. For example, in a meta-analysis, Derrfuss et al. (2005) identified a bilateral region near BA 6/9/44 (which they refer to as the inferior frontal junction) that responds to demands for control in Stroop and task-switching paradigms. In another meta-analysis, Ridderinkhof et al. (2004) reported that activations on tasks requiring cognitive control were clustered in midline regions that included both the dacc and pre-sma. Consistent activation of the dacc by tasks involving cognitive control also was found in a meta-analysis of more than 100 PET studies by Paus et al. (1998). These midline and lateral regions, as well as a ventrolateral region including the anterior insula, were identified in a review of fmri studies by Duncan and Owen (2000), leading them to propose a distributed frontal network that is recruited in cognitively demanding situations. The current results are consistent with this literature and suggest a distributed frontal network that responds to task difficulty in the context of decision making under uncertainty. One interpretation of the current results is that at higher levels of uncertainty, the decision becomes more difficult because one response must be suppressed, and the frontal network is engaged to mediate this conflict between responses (Kerns et al., 2004). As already noted, our design cannot exclude the possibility that expected value, which in our paradigm was correlated with uncertainty, plays a role in the observed responses. However, Knutson et al. (2005) recently reported that the level of activity in medial prefrontal regions near to those examined in the current study is positively correlated with expected value. In the current study, therefore, one might have expected that activity in prefrontal regions would be depressed in response to decreased expected value (or increased uncertainty). This was not the case, however, suggesting that the cognitive demands of decision uncertainty in this paradigm were more important than expected value in producing the observed responses. This is consistent with the finding by Knutson et al. that the calculation of expected value is distributed, with cortical and subcortical regions being responsible for different components of the calculation. Medial frontal regions within the frontal network, most notably the dacc, have also been implicated in detection of errors in both imaging and event-related potential (ERP) studies (Brown and Braver, 2005; Carter et al., 1998; Holroyd et al., 2004; Ullsperger and von Cramon, 2003). Unexpected outcomes have been shown to lead to greater dacc activation than expected outcomes in fmri studies (Brown and Braver, 2005; Holroyd et al., 2004) and to produce a greater amplitude error-related negativity in ERP studies (Holroyd et al., 2003), leading authors to posit that processing unexpected outcomes requires more cognitive control. The present results are consistent with these findings and suggest further that such error signals may not be restricted to the dacc but may be present in other frontal regions as well. The observation that the same regions respond to both decision uncertainty and unexpected feedback suggests that these two different demands recruit the same frontal network. It is important to note that the hemodynamic responses to decision uncertainty and reward outcome could not be temporally separated in this rapid event-related fmri study. When taken in the context of the pupillometry data (where such a distinction was possible), however, it would appear that the frontal network may be recruited at temporally discrete phases of the decision-making process by functionally divergent demands. Reward activity in the dorsal striatum is dissociated from frontal activation In contrast to the activity in the network of frontal regions, dorsal striatal regions responded only to outcomes (Fig. 8), with wins producing a greater response in these reward-related regions. Unlike regions in the frontal network, whose activity was modulated by uncertainty, activity in dorsal striatal regions was not so modulated. This dissociation between frontal and striatal regions was most apparent on loss trials, where frontal regions

13 T.D. Satterthwaite et al. / NeuroImage 37 (2007) responded with increased activity, whereas activity in dorsal striatal regions was reduced. Several neuroimaging studies have reported striatal responses to reward in humans (Delgado et al., 2003; Knutson et al., 2001b; McClure et al., 2003). As in McClure et al. (2003), the striatal regions which were observed to track outcomes in the present study were located in the dorsal putamen. Data from studies by McClure et al. and O'Doherty et al. (2003) suggest that reward-related activity in the striatum may reflect the phasic firing of DA neurons that project from the ventral tegmentum to the striatum (Schultz et al., 2000). The results of the present study are consistent with such an account. Not only was there a striatal response to reward, but there were also indications of a response decrement following reward omission on loss trials (Fiorillo et al., 2003; Schultz et al., 1997). It should be noted that the current results differ from certain findings in prior imaging studies of reward. In previous studies, the nucleus accumbens in the ventral striatum has been found to respond to reward (Knutson et al., 2001a; O'Doherty et al., 2004; Pagnoni et al., 2002), and one recent study demonstrated uncertainty-modulated reward responses in the midbrain (Dreher et al., 2006). We did not observe such nucleus accumbens or midbrain responses, but this may be because of differences in task designs and our use of thicker (8 mm) slices in image acquisition, thereby decreasing resolution of small subcortical structures. However, our results are consistent with the findings of Knutson et al. (2005), who suggested that subcortical reward pathways may primarily represent reward valence and magnitude, but do not track reward probability. Striatal reward activity may bias frontal recruitment Taken together, the results from the pupillometry and imaging experiments suggest two primary conclusions. First, activity in a frontal network correlates both with initial uncertainty and with losses that violate expectations. Second, regions in the dorsal striatum respond only to reward, showing a clear dissociation from the response of the frontal network. The finding that activity in the frontal network increases in response to losses that violate reward expectations, whereas dorsal striatal activity is unaffected or even suppressed by such events, is consistent with the suggestion that striatal reward pathways may modulate the recruitment of frontal cognitive resources. A link between these two systems has been proposed by Holroyd and Coles (2002), who theorized that recruitment of frontal resources may in part be controlled by phasic DA firing. In this account, unexpected negative outcomes produce phasic suppression of firing by DA neurons (Fiorillo et al., 2003), which in turn promote the recruitment of a network of frontal regions involved in cognitive control. Indeed, some authors have proposed that such a dopaminergic gating mechanism could be the fundamental teaching signal for reinforcement learning (e.g., Montague et al., 2004). Our results are consistent with such a link between these systems of reward and control, whereby a distributed frontal network that responds to demands for cognitive control may be modulated by a dissociable reward system that responds to outcomes. Acknowledgments We thank Neal Cohen and John Stern for their invaluable assistance with the cognitive pupillometry. Avi Snyder and Mark McAvoy provided support with imaging and analysis tools. We also thank Ben Shannon for discussion. This work was supported in by NIH grants MH55308, AG05886 and the Howard Hughes Medical Institute. References Aharon, I., Etcoff, N., Ariely, D., Chabris, C.F., O'Connor, E., Breiter, H.C., Beautiful faces have variable reward value: fmri and behavioral evidence. Neuron 32, Aston-Jones, G., Cohen, J.D., An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, Beatty, J., Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91, Beatty, J., Lucero-Wagoner, B., The Pupillary System. In Handbook of Psychophysiology. In: Cacioppo, J., Tassinary, L., Bernston, G. (Eds.), Cambridge University Press, London. Berns, G.S., McClure, S.M., Pagnoni, G., Montague, P.R., Predictability modulates human brain response to reward. J. Neurosci. 21, Botvinick, M.M., Braver, T.S., Barch, D.M., Carter, C.S., Cohen, J.D., Conflict monitoring and cognitive control. Psychol. Rev. 108, Boynton, G.M., Engel, S.A., Glover, G.H., Heeger, D.J., Linear systems analysis of functional magnetic resonance imaging in human V1. J. Neurosci. 16, Brass, M., Derrfuss, J., Forstmann, B., von Cramon, D.Y., The role of the inferior frontal junction area in cognitive control. Trends Cogn. Sci. 9, Breiter, H.C., Gollub, R.L., Weisskoff, R.M., Kennedy, D.N., Makris, N., Berke, J.D., Goodman, J.M., Kantor, H.L., Gastfriend, D.R., Riorden, J.P., et al., Acute effects of cocaine on human brain activity and emotion. Neuron 19, Breiter, H.C., Aharon, I., Kahneman, D., Dale, A., Shizgal, P., Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30, Brown, J.W., Braver, T.S., Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, Buckner, R.L., Functional anatomic correlates of control processes in memory. J. Neurosci. 23, Buckner, R.L., Raichle, M.E., Miezin, F.M., Petersen, S.E., Functional anatomic studies of memory retrieval for auditory words and visual pictures. J. Neurosci. 16, Buckner, R.L., Goodman, J., Burock, M., Rotte, M., Koutstaal, W., Schacter, D.L., Rosen, B., Dale, A.M., 1998a. Functional anatomic correlates of object priming in humans revealed by rapid presentation event-related fmri. Neuron 20, Buckner, R.L., Koutstaal, W., Schacter, D.L., Dale, A.M., Rotte, M., Rosen, B.R., 1998b. Functional anatomic study of episodic retrieval: II. Selective averaging of event-related fmri trials to test the retrieval success hypothesis. NeuroImage 7, Bush, G., Vogt, B.A., Holmes, J., Dale, A.M., Greve, D., Jenike, M.A., Rosen, B.R., Dorsal anterior cingulate cortex: a role in rewardbased decision making. Proc. Natl. Acad. Sci. U. S. A. 99, Carter, C.S., Braver, T.S., Barch, D.M., Botvinick, M.M., Noll, D., Cohen, J.D., Anterior cingulate cortex, error detection, and the online monitoring of performance. Science 280, Cohen, J.D., MacWhinney, R.C., Flatt, M., Provost, J., Psyscope: an interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers. Behav. Res. Methods Instrum. Comput. 25, Cohen, J.D., Braver, T.S., Brown, J.W., Computational perspectives on dopamine function in prefrontal cortex. Curr. Opin. Neurobiol. 12, Critchley, H.D., Mathias, C.J., Dolan, R.J., Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29,