Frontal Cortex and the Discovery of Abstract Action Rules

Transcription

1 Article Frontal Cortex and the Discovery of Abstract Action Rules David Badre, 1,4, * Andrew S. Kayser, 2,4, * and Mark D Esposito 3 1 Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI 02906, USA 2 Department of Neurology, University of California at San Francisco, Ernest Gallo Clinic & Research Center, Emeryville, CA 94608, USA 3 Helen Wills Neuroscience Institute and Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA 4 These authors contributed equally to this work *Correspondence: david_badre@brown.edu (D.B.), akayser@gallo.ucsf.edu (A.S.K.) DOI /j.neuron SUMMARY Although we often encounter circumstances with which we have no prior experience, we rapidly learn how to behave in these novel situations. Such adaptive behavior relies on abstract behavioral rules that are generalizable, rather than concrete rules mapping specific cues to specific responses. Although the frontal cortex is known to support concrete rule learning, less well understood are the neural mechanisms supporting the acquisition of abstract rules. Here, we use a reinforcement learning paradigm to demonstrate that more anterior regions along the rostro-caudal axis of frontal cortex support rule learning at higher levels of abstraction. Moreover, these results indicate that when humans confront new rule learning problems, this rostro-caudal division of labor supports the search for relationships between context and action at multiple levels of abstraction simultaneously. INTRODUCTION Rapid adaptation to novel circumstances is a hallmark of intelligent behavior. This ability partially requires the representation of rules that can associate a context with a specific behavioral response. However, adaptive behavior further requires the ability to generalize rules to novel circumstances. For example, we can rapidly work out an unusual mechanism by which a door is opened, such as pulling a cord rather than turning a knob, even if we have no prior experience using that mechanism to open a door. Such rule generalization depends on the discovery of abstract relationships between context and classes of action that are not dependent on a one-to-one mapping between a stimulus and a particular motor response. Lateral frontal cortex, and the prefrontal cortex (PFC) in particular, has an established role in representing rules for action and supporting adaptive behavior (Badre and Wagner, 2004; Bunge, 2004; D Esposito et al., 1995; Duncan, 2001; Miller and Cohen, 2001; Passingham, 1993; Petrides, 2005; Stuss and Benson, 1987; Wallis et al., 2001). Current models of PFC conceptualize this function in terms of cognitive control, defined as the ability of frontal neurons to represent contextual information in order to bias selection of appropriate action pathways over competitors (Badre and Wagner, 2006; Botvinick et al., 2004; Braver et al., 2003; Cohen et al., 1990; Miller and Cohen, 2001; O Reilly and Frank, 2006). Moreover, frontal cortex, in concert with striatum, is critical for the acquisition of behavioral rules (Asaad et al., 1998; Passingham, 1989; Petrides, 1987; White and Wise, 1999). However, previous studies of learning have primarily focused on the formation of concrete conditional associations between specific stimuli and responses, rather than on the learning of abstract rules. The functional organization of frontal cortex may provide clues as to the mechanisms by which abstract rules are acquired. Growing evidence indicates that the rostro-caudal axis of frontal cortex may be organized hierarchically such that neurons in more anterior regions of frontal cortex process progressively more abstract representations in the service of cognitive control (Badre, 2008; Badre and D Esposito, 2007; Badre et al., 2009; Botvinick, 2007, 2008; Buckner, 2003; Bunge and Zelazo, 2006; Christoff and Keramatian, 2007; Koechlin and Jubault, 2006; Koechlin et al., 2003; Koechlin and Summerfield, 2007; Petrides, 2006; Race et al., 2008). In general, hierarchies facilitate learning and adapting to novel circumstances because they have the ability to represent information at multiple, increasingly abstract levels (Chase and Simon, 1973; Estes, 1972; Gick and Holyoak, 1983; Greeno and Simon, 1974; Lashley, 1951; Miller et al., 1960; Newell, 1990; Paine and Tani, 2005). Such abstracted representations are more easily analogized to novel circumstances, thereby facilitating transfer of knowledge gained in one context to a new one. Hence, it is reasonable to hypothesize that the capacity for rule abstraction afforded by a putative frontal lobe hierarchy might support rapid adaptive behavior. However, previous demonstrations of hierarchically arrayed processors in the PFC have only tested the execution of welllearned rules, acquired through explicit instruction, and so these studies have not been designed to address how this functional organization might be leveraged to facilitate rule discovery. The current study seeks to fill this gap by investigating reinforcement learning of abstract versus concrete behavioral rules. A rule can be defined as abstract to the extent that it determines a set of simpler rules based on contextual information. This type of abstraction is termed policy abstraction (Badre et al., 2009; Botvinick, 2008). For example, consider two simple rules: a circle cues a left-hand response and a triangle cues Neuron 66, , April 29, 2010 ª2010 Elsevier Inc. 315

2 Figure 1. Schematic Depiction of Trial Events, Example Stimulus-to-Response Mappings, and Policy for Hierarchical and Flat Rule Sets (A) Trials began with presentation of a stimulus followed by a green fixation cross. Participants could respond with a button press at any time while the stimulus or green fixation cross was present. After a variable delay after the response, participants received auditory feedback indicating whether the response they had chosen was correct given the presented stimulus. Trials were separated by a variable null interval. (B) Example stimulus-to-response mappings for the Flat set. The arrangement of mappings for the Flat set was such that no higher-order relationship was present; thus, each rule had to be learned individually. (C) This set of many first-order rules can be represented as a large, Flat policy structure with only one level and eighteen alternatives. (D) Example stimulus-to-response mappings for the Hierarchical set. Response mappings are grouped such that in the presence of a red square, only shape determines the response, whereas in the presence of a blue square, only orientation determines the response. (E) The Hierarchical set can be represented as a two-level policy structure with a second-order rule selecting between the shape or orientation mapping sets, and a set of first-order rules then relating specific shapes or orientations to responses. a right-hand response. This first-order policy specifies a one-toone relationship between a specific stimulus (i.e., a shape) and a response. However, consider an independent set of first-order policy, based on size, in which a large stimulus cues a left-hand response and a small stimulus cues a right-hand response. Because the shape and size rule sets are independent, both cannot simultaneously govern responding. For instance, if the relevant set of first-order policy is unknown, a stimulus that is both circular and small cues opposing responses. Consequently, a more abstract rule (second-order policy) is required in order to specify which set of first-order rules (shape or size) should govern responding in the current context. For example, framing the stimulus with a red border might indicate that shape is the appropriate first-order policy, whereas green might indicate size. Because this second-order policy based on color specifies a class of simpler rule sets (shape or size) rather than a specific response, it is more abstract. Using the above definition of abstraction, we designed a reinforcement learning task that provides participants an opportunity to acquire an abstract rule (second-order policy). During fmri scanning, participants were required to learn two sets of rules, in separate epochs, that linked each of 18 different stimuli uniquely and deterministically to one of three buttonpress responses (Figure 1). For each rule set, an individual stimulus consisted of one of three shapes, at one of three orientations, inside a box that was one of two colors for a total of 18 unique stimuli (3 shapes 3 3 orientations 3 2 colors; Figure 1). Participants were instructed to learn the correct response for each stimulus on the basis of auditory feedback (Figure 1A). For one of the two rule sets, termed the Flat set, each of the 18 rules had to be learned individually as one-to-one mappings (first-order policy) between a conjunction of color, shape, and orientation and a response (Figures 1B and 1C). In the other set, termed the Hierarchical set, stimulus display parameters and instructions were identical to the Flat set. And, indeed, the Hierarchical set could also be learned as 18 individual first-order rules. However, the arrangement of response mappings was such that a second-order relationship could be learned instead, thereby reducing the number of first-order rules to be learned (Figures 1D and 1E). Specifically, in the context of one colored box, only the shape dimension was relevant to the response, with each of the three unique shapes mapping to one of the three button responses regardless of orientation. Conversely, in the context of the other colored box, only the orientation dimension was relevant to the response. Thus, the Hierarchical rule set permitted learning of abstract, second-order rules mapping color-to-dimension along with two sets of first-order rules (i.e., specific shape-to-response and orientation-to-response mappings; Figure 1E). Critically, all instructions, stimulus presentation parameters, and between-subject stimulus orderings were identical between the two rule sets. The Flat and Hierarchical rule sets only differed in that the organization of mappings in the Hierarchical set permitted learning of a second-order rule. Hence, these two sets contrast a learning context in which abstract rules can be 316 Neuron 66, , April 29, 2010 ª2010 Elsevier Inc.

3 A B Figure 2. Behavioral Data (A) Shown are the learning curve estimates, bounded by a 90% confidence interval, for the single subject whose learning trials for the Hierarchical and Flat sets were closest to the group means for each condition (Hierarchical = 64; Flat = 91). Black arrows illustrate the learning trial, at which the lower confidence bound rose above chance performance (33%). Gray arrows highlight the terminal accuracy. (B) Subsequent panels depict the correlates of learning ± SEM across the 20 subjects for the Hierarchical and Flat sets: the terminal accuracy; the maximal first derivative of the learning curve, representing the speed of learning; the maximal second derivative of the learning curve, representing the rate of change in the speed of learning; and the learning trial (i.e., the value depicted by the black arrows in A). For three subjects, learning for the Flat set never rose above chance; these subjects were excluded from the calculation for the mean Flat learning trial (n = 17). (See also Figure S1 for further behavioral data.) discovered with an analogous context in which no such rules can be learned. Thus, this design provides a means of studying the neural mechanisms of abstract rule learning. RESULTS Behavioral Results Learning curves were generated based on the estimated probability of a correct response on each trial along with a 90% confidence interval (see Experimental Procedures and Figure 2A). Differences in these estimates between the Hierarchical and Flat rule sets were consistent with the acquisition of generalizable, second-order rules for the Hierarchical set. First, generalization should make the learning task easier, in that more of the specific mappings between stimuli and responses should be acquired. Terminal accuracy was significantly higher for the Hierarchical (84%) than the Flat (58%) rule set [F(1,19) = 26.3, p <.0001; Figure 2B, leftmost panel]. Moreover, a significantly higher proportion of individual rules were learned in the Hierarchical (72%) than Flat (43%) set [F(1,19) = 14.6, p <.005]. It should be noted that neither these effects, nor any others reported here, changed as a function of which rule set was learned first. Second, generalization should make learning more efficient, in that once an abstract rule is acquired for one stimulus, it is applicable to all others like it. Indeed, learning trial estimates defined as the number of presentations of a specific stimulus before the response associated with that stimulus is known came earlier for individual rules in the Hierarchical versus Flat set [t(19) = 2.1, p =.05; Figure 2B, rightmost panel]. Crucially, the facilitation of learning associated with generalization should be specific to the first-order rules entailed by a learned second-order rule. To test this prediction, we identified all first-order rules for each subject that were learned above chance, termed learned first-order rules. We then assumed that a subject knew a second-order rule associating a color with either shape or orientation if, in the Hierarchical case, all nine rules sharing that color were learned first-order rules. We defined such sets as known second-order sets. We note there was never a case in which all nine rules associated with a given color were known by the end of learning for the Flat rule set. The average learning trial for learned first-order rules that were members of known second-order sets in the Hierarchical condition was reliably earlier in learning than the average learning trial for learned first-order rules in the Flat condition [t(19) = 3.8, p <.005]. Importantly, this effect was not driven by the fact that Hierarchical rules were learned more quickly, on average, than were Flat rules. Within the Hierarchical condition itself, the average learning trial for first-order rules that were members of known second-order sets was also reliably earlier than the learning trial for learned first-order rules that were not members of a known second-order set [t(19) = 2.5, p <.05]. Moreover, there was no reliable difference in the learning trial for learned first-order rules in the Flat set and learned first-order rules in the Hierarchical set that were not members of a known second-order set [t(19) = 1.5, p = 0.2]. Thus, consistent with generalization, the faster learning rate for the Hierarchical set was specific to those learned first-order rules in the Hierarchical set for which the second-order rule had been acquired. Third, once acquired, the generalization of a second-order rule to unknown first-order rules should be reflected in an abrupt gain in accuracy. Across subjects, Hierarchical curves consistently showed step-wise increases, presumably reflecting acquisition and generalization of a second-order rule (e.g., Figure 2A; see also Figure S1A available online). By contrast, a gradual increase was evident for the Flat curves. This qualitative difference in the shape of the learning curves was reflected in a greater maximum first derivative (maximum learning rate) and second derivative (maximum rate of change in the learning rate) for the Hierarchical than Flat rule sets [Fs > 9.0, ps <.01; Figure 2B]. We further tested the tendency for the Hierarchical learning curves to be step-wise relative to the Flat learning curves by explicitly fitting a sigmoid function to each participant s learning curves. The sigmoid is defined by two parameters, a and b, that represent the slope and offset of the sigmoid, respectively. A larger value of a indicates a steeper step, and a smaller value of b indicates that the step occurred earlier in learning. Critically, a was significantly larger (Wilcoxon s rank sum test: Z = 2.5, p < 0.05) and b was significantly smaller (Z = 2.9, p < 0.005) for the Hierarchical than Flat curves. Goodness of fit did not differ (Z = 0.75, p =.46). To ensure that these differences were not Neuron 66, , April 29, 2010 ª2010 Elsevier Inc. 317

4 A C B Figure 3. Basic Imaging Results (A) Inflated representation of the left hemisphere showing areas that demonstrated a positive main effect of task (T-values indicated by the color bar), thresholded by a false discovery rate < The locations of regions of interest (ROIs) determined independently from a previous data set (Badre and D Esposito, 2007) are overlaid (from posterior to anterior, 1 = dorsal premotor cortex [PMd], 2 = pre-premotor cortex [prepmd], 3 = mid dorsolateral prefrontal cortex [mid- DLPFC], and 4 = rostro-polar cortex [RPC]). (B) For prepmd but not for PMd, total activity as measured by the integrated percent signal changes (ipscs) for correct trials only differed across learning for the Hierarchical and Flat sets (*p < 0.05). (C) Dividing the learning curve into three temporal epochs of 120 trials each (Begin, Middle, and End) reveals that these differences in PMd emerged after the initial phase of learning for the Hierarchical set (*p < 0.05; p < 0.10). (D) Dividing the learning curve by estimated performance (see text) confirms the temporal differences seen in prepmd. (See also Figure S2.) All error bars indicate the standard error of the mean. D driven by model assumptions that produced the parametric learning curve estimates, we also fit sigmoid functions, via Bernoulli assumptions, directly to each subject s responses. Consistent with the above analysis, a was again significantly larger (Z = 2.7, p <.01) and b was significantly smaller (Z = 2.7, p <.01) for the Hierarchical than Flat curves. FMRI Activation during the Learning Task A whole-brain, voxel-wise contrast of all conditions versus baseline identified regions that were reliably activated relative to baseline during the learning task (Figure 3A). This contrast yielded a characteristic frontoparietal and subcortical network consistent with both prior studies of rule learning and studies of hierarchical cognitive control. Bilateral bands of activation extended rostral to caudal in frontal cortex, including dorsal premotor cortex (PMd; ; ; BA 6), dorsal anterior premotor cortex (prepmd; ; ; BA 6/44), middorsolateral prefrontal cortex (mid- DLPFC; ; ; BA 9/ 46), and frontal polar cortex (FPC; ; ; BA 10/46). These rostro-caudal frontal activations corresponded closely to regions previously associated with progressively abstract levels of policy selection (Badre and D Esposito, 2007). Additional frontal activations were observed in the supplementary motor area (SMA; ; ) and anterior insula ( ; ). Beyond frontal cortex, task-related activation was evident in bilateral superior ( ; ) and inferior parietal lobules (IPL; ; ). Subcortically, task-related activation was observed in bilateral striatum, including the body of the caudate ( ; ) and anterior putamen ( ; ). To test our predictions regarding learning at different levels of abstraction and their relationship to the rostro-caudal axis of frontal cortex, we defined regions of interest (ROI) in PMd ( ), prepmd ( ), mid-dlpfc ( ), and FPC ( ) by using coordinates that were previously in associated with parametric increases in first through fourth order control, respectively (Badre and D Esposito, 2007). Analysis of the effects of learning manipulations focused initially on these ROIs. 318 Neuron 66, , April 29, 2010 ª2010 Elsevier Inc.

5 Hierarchical versus Flat Rule Sets in the Frontal Cortex We assessed differences in frontal cortex due to the rule set, restricting this and all subsequent analyses to stimulus-related activity (i.e., prior to feedback) for correct trials only. Both the Flat and Hierarchical rule sets involve the learning and execution of simple stimulus-response mappings (first-order policy). However, only the Hierarchical rule set includes rules at a second order of policy abstraction. In prior work (Badre and D Esposito, 2007), we demonstrated that tasks requiring only first-order policy activated PMd, whereas second-order policy additionally engaged the more rostral prepmd. Consistent with this previous study, differences between Flat and Hierarchical rule sets were evident in prepmd [F(1,19) = 5.0, p < 0.05], but not in the more caudal PMd region (F = 0.4; Figure 3B). Even more rostral mid- DLPFC and FPC, at the highest levels of abstraction, did not show a reliable difference between the Hierarchical and Flat sets (Fs < 0.9). Hence, despite the fact that subjects engaged in a task in which no explicit instructions were provided about a second-order rule, prepmd was the only region to reliably distinguish between Hierarchical and Flat rule sets. Time-Dependent Analysis of Learning in the Frontal Cortex The overall difference in prepmd activation between Hierarchical versus Flat conditions is consistent with previous work demonstrating a hierarchical organization along the rostro-caudal axis of the frontal cortex (Koechlin et al., 2003; Badre and D Esposito, 2007), although here the higher-order rules were acquired through reinforcement rather than explicit instruction. However, because the present study is primarily concerned with understanding the mechanisms of abstract rule learning, determining at what point in time and in what way this difference in prepmd emerges during learning is of central importance. In particular, if the discovery and execution of second-order rules only occurs after first-order rules are successfully learned, one might predict that activation in prepmd would remain at baseline until late in learning and increase only in the Hierarchical condition after sufficient numbers of first-order rules have been acquired. Alternatively, if the search for higher-order rules occurs in parallel with the search for first-order rules, one would anticipate that activation in prepmd would be above baseline for both the Hierarchical and Flat sets from the outset of learning, remaining at that level throughout the block in the Hierarchical condition but declining to baseline by the end of learning in the Flat condition. In order to test these predictions, we divided learning sets into three phases: Begin, Middle, and End. The crossing of learning set (Hierachical/Flat) with learning phase (Begin/ Middle/End) was assessed in the PMd and prepmd. During the Begin phase of learning, both regions were reliably active relative to baseline [t(19) > 3.9, ps <.001], and no difference was evident between the learning sets in either region (Fs < 1.9; Figure 3C). However, by the Middle phase of learning, a reliable difference emerged between the Flat and Hierarchical sets in prepmd [F(1,19) = 4.2, p < 0.05] but not in PMd (F = 0.5). The End phase again showed a reliable difference between Flat and Hierarchical in prepmd [F(1,19) = 6.4, p < 0.05]. This difference between Flat and Hierarchical in prepmd was due to a reliable decline in activation for the Flat [F(1,19) = 4.3, p < 0.05] rule set at the Middle and End phases of learning, whereas no such decline was evident for the Hierarchical learning set (F = 0.05). During the End phase, PMd also revealed a trend difference between Hierarchical and Flat rule sets [F(1,18) = 3.8, p =.06]. However, this difference was due to a reliable increase in activation for the Hierarchical [F(1,18) = 4.9, p <.05] but not Flat (F =.8) learning set. In summation: (1) At the beginning of learning, prepmd and PMd were active above baseline for both the Hierarchical and Flat sets. (2) By the Middle phase of learning, activation had declined reliably for the Flat but not the Hierarchical set in prepmd. (3) At the end of learning, there was a reliable increase in activity for the Hierarchical but not Flat set in PMd. These results are initially consistent with the hypothesis that the search for rules occurs at multiple levels of abstraction from the outset of learning. However, although the region by phase interaction was reliable [F(2,36) = 3.4, p <.05), individual differences in the learning curves (see Figure S1A) could introduce variability and so reduce our sensitivity to regional differences. Additionally, due to superior accuracy in the Hierarchical set, processes related to improved performance, but unrelated to policy abstraction, could diminish the interpretability of our effects. For addressing these issues, learning analysis was performed on performance-aligned curves. Performance-Equated Changes in Learning in the Frontal Cortex To evaluate differences in performance between subjects, we divided the learning curves into performance epochs based on accuracy, rather than temporal epochs, using three anchor points: (1) the division between the lowest level of accuracy (PE-1) and the next highest (PE-2) was defined by the median across-subject accuracy (0.42) at the learning trial; (2) the division between the highest level of accuracy (PE-4) and the next lowest (PE-3) was defined by the median terminal accuracy (0.70); and (3) the division between performance levels two and three (PE-2 and PE-3) was defined as one-third of the difference in accuracy between the two extreme anchor points (0.51). With this approach, accuracies were equated for the first three bins (ts < 1.2, ps > 0.12). Because many fewer subjects reached the highest level of accuracy in the Flat (5) as compared to the Hierarchical (15) condition, accuracies were necessarily different for PE-4 (Figure S1B). Consequently, PE-4 was not included in the statistical analyses. Consistent with the results of the time-based analysis, activity in prefrontal cortex strongly differentiated the Hierarchical and Flat conditions (Figure 3D). A repeated-measures ANOVA inclusive of PMd and prepmd demonstrated both ROI 3 learning set [F(1,19) = 12.4, p = 0.002] and ROI 3 performance epoch [F(2,38) = 12.6, p = ] interactions. These differences were confirmed by direct post-hoc comparison of Hierarchical and Flat activity in prepmd, which was significant for PE-2 and PE-3 (ts > 1.8, ps < 0.05) but not for PE-1 [t(32) = 0.5, p = 0.33]. By contrast, activation in the more caudal PMd did not reliably differ between learning sets for any performance epoch (ts < 1.5). Thus, results from the performance based analysis corroborated those from the time-based analysis: Hierarchical versus Flat differences in prepmd emerged late in learning Neuron 66, , April 29, 2010 ª2010 Elsevier Inc. 319

6 [R =.51; t(16) = 2.3, p < 0.05], terminal accuracy [R =.56; t(19) = 2.9, p <.05], and max first derivative [R =.51; t(19) = 2.5, p < 0.05]. A positive trend was also evident for the fourth marker, the max second derivative [R =.39; t(19) = 1.8, p = 0.09]. However, no such correlations were significant for PMd (Rs <.3, ps >.21). Thus, these data provide evidence that early activation in prepmd and not PMd reflects search for higher order rules. Figure 4. Scatter Plots Demonstrating Brain-Behavior Correlations The x axis of each plot shows the integrated percent signal change (ipsc) for correct trials only versus baseline for the beginning phase of learning collapsed across rule set (Hierarchical/Flat) and accuracy (correct/error) for PMd (left plots) and prepmd (right plots). This early learning activation across rule sets is plotted against the difference in learning trial (row 1), terminal accuracy (row 2), max first derivative (3rd row), and max second derivative (row 4) between Hierarchical and Flat rule sets. because of a decline in activation for the Flat relative to the Hierarchical rule set. Correlation of Activation in prepmd and Behavior during Hierarchical Learning Does the activation early in learning for both Flat and Hierarchical sets relate to successful higher order rule learning? The previous two hypotheses make opposite predictions concerning the locus of such a relationship. If first-order rules must first be learned in order for second-order rules to be acquired, early activity in areas supporting first-order rule acquisition (PMd) should be predictive of subsequent second-order rule learning. Conversely, if first-order and second-order rules are explored in parallel, early activity in areas supporting second-order rule learning (prepmd), but not first-order rule learning (PMd), should be predictive of successful second-order rule acquisition. In order to address this question, we conducted between-subjects correlations of the mean activation at the Begin phase of learning, across Hierarchical and Flat sets, with the behavioral differences between Hierarchical and Flat learning sets (i.e., learning trial, terminal accuracy, max first, and max second derivatives) that mark the successful acquisition of the higher order rules. As depicted in Figure 4, Begin phase activation in prepmd correlated reliably with the difference in learning trial Time- and Performance-Dependent Analysis of Learning in Striatum Consistent with past work on reinforcement learning (Cohen and Frank, 2009; Cools et al., 2002; Dayan and Balleine, 2002; Frank and Claus, 2006; Hadj-Bouziane et al., 2003; Murray et al., 2000; Packard and Knowlton, 2002; Schönberg et al., 2007; Seger, 2008; Sutton and Barto, 1998; Toni et al., 1998; Tremblay et al., 1998), the whole-brain analysis reliably identified regions in the striatum both caudate and anterior putamen that were active during learning relative to baseline (Figure 5). Unlike PMd and prepmd, stimulus-related activity (prior to feedback) in striatum increased with learning of the Hierarchical but not Flat set, [F(1,19) = 6.9, p < 0.05) without significant interactions between conditions (Fs < 1.5, ps > 0.2). Post-hoc comparison revealed a difference between rule sets by the end of learning in the left putamen [t(19) = 2.2, p < 0.05] and right caudate [t(19) = 2.4, p < 0.05], and a trend difference in the left caudate (t(19) = 1.9, p = 0.07; Figure 5A). Performance-based analysis was consistent with these effects (see Figure S3). Effective Connectivity Analysis of the Frontostriatal Network Learning of first-order stimulus-response associations has been consistently shown to depend on dynamic interactions between striatum and cortex (Hadj-Bouziane et al., 2003; Murray et al., 2000; Packard and Knowlton, 2002; Seger, 2008; Toni et al., 1998; Tremblay et al., 1998). Thus, we evaluated effective connectivity between them using Granger causality (GC), a method for determining whether the BOLD time series in one region helps to predict the time series in another (Goebel et al., 2003; Kayser et al., 2009; Roebroeck et al., 2005) (see Supplemental Information for additional analysis). PMd and prepmd were Granger causal for the bilateral caudate (ps <.0005; Figure 5B). Conversely, activity in bilateral putamen was Granger causal for both PMd and prepmd (ps < 0.05). Importantly, none of the above effects differed significantly between rule sets (ps > 0.18). DISCUSSION In the present study, we contrasted learning of two rule sets in which only the Hierarchical set afforded the opportunity to learn an abstract, second-order rule. Broadly, results from this experiment provide fundamental insights into the way that humans approach novel learning problems. In summary: (1) Participants were capable of rapidly acquiring abstract rules when they were available. (2) Activation was evident in both PMd and prepmd early in learning but declined in the more rostral prepmd by the end of learning of the Flat set, which contained no secondorder rules. (3) Activation early in learning in prepmd across 320 Neuron 66, , April 29, 2010 ª2010 Elsevier Inc.

7 Figure 5. Striatum/GC Analyses (A) Areas within the striatum demonstrating a positive main effect of task were identified in both caudate and putamen. Across time, the integrated percent signal change (ipsc) in both areas tended to be greater in the Hierarchical than the Flat case (*p < 0.05; p < 0.10, where error bars represent the standard error of the mean). (B) Despite parallel striatal univariate changes, Granger causality analysis demonstrated that BOLD signal in putamen (Pt) was reliably Granger causal (*p < 0.05; **p < ) for activity within PMd and prepmd, which was in turn reliably Granger causal for activity in the caudate (Cd). (To PMd and prepmd from left putamen: GC = and GC = 0.003, respectively; from right putamen: GC = and GC = 0.007, respectively. From PMd and prepmd to left caudate: GC = and GC = 0.013, respectively; to right caudate: GC = and GC = 0.013, respectively.) Slices show the main effect of task (T values indicated by the color bar), with Pt and Cd regions of interest designated by the small circles at the origins/terminations of the GC arrows. See also Figures S3 and S4 and Table S1 for further details. Hierarchical and Flat sets, but not in PMd, was correlated with behavioral differences between the Hierarchical and Flat learning curves. (4) Striatum showed greater activation by the end of learning for the Hierarchical relative to the Flat set, but the dynamics of frontostriatal interactions did not differ between sets i.e., for both, the putamen influenced, whereas caudate was influenced by, activation in PMd and prepmd. These results suggest that from the outset of learning the search for relationships between context and action may occur at multiple levels of abstraction simultaneously and that this process differentially relies on systematically more rostral portions of frontal cortex for the discovery of more abstract relationships. However, dynamic interactions between striatum and frontal cortex that support reinforcement learning appear common across levels of abstraction. Growing evidence suggests that the frontal cortex may possess a rostro-caudal organization whereby more rostral regions support cognitive control involving progressively more abstract representations (Badre, 2008; Badre and D Esposito, 2007; Badre et al., 2009; Botvinick, 2007, 2008; Buckner, 2003; Bunge and Zelazo, 2006; Christoff and Keramatian, 2007; Koechlin and Jubault, 2006; Koechlin et al., 2003; Koechlin and Summerfield, 2007; Petrides, 2006; Race et al., 2008). An important question left open by these previous experiments is the extent to which this rostro-caudal organization can be leveraged to facilitate learning of abstract rules. Consistent with past work, our results demonstrate that a differentiation does indeed emerge rostrally, in prepmd, when a second-order rule must be learned through reinforcement rather than explicit instruction. However, critical to understanding the mechanisms of abstract rule learning is understanding how this difference arises. In particular, there are at least two qualitatively distinct ways to account for the emergence late in learning of a difference in activation between the Hierarchical and Flat rule sets in prepmd: (1) PrePMd might be recruited to search for and execute a second-order rule only after first-order rules have been learned. (2) PrePMd might be directly involved in the search for, as well as the execution of, second-order policy from the outset of learning and decrease its involvement to the extent that such rules are not rewarded. Our data are consistent with the second of these proposals and inconsistent with the first. Specifically, activation is evident early in prepmd, before all first- or second-order rules are known and, in the case of the Flat set, even when no second-order rules can be known. Moreover, this early activation in prepmd, across learning conditions (i.e., Flat or Hierarchical), correlates with discovery of second-order rules when they are available, indicating that this activation reflects neural processes related to the early search for abstract rules. Thus, the decline in activation in prepmd during the Flat set, when second-order rules are not available, may reflect the attenuation of higher-level search when higher-order rules are not rewarded. Following from this account, this result provides potential insight into another fundamental question concerning a putative rostro-caudal hierarchical organization of the frontal cortex; namely, to the extent that the brain does possess such an architecture, what advantages might it convey over other schemes? In particular, it has been demonstrated that though complex action may be represented hierarchically (i.e., in terms of goals, subgoals, etc.), the existence of hierarchical representations does not require that the action system itself segregate these representations among spatially separate pools of neurons (Botvinick and Plaut, 2004; Botvinick, 2007). One possible advantage of having such an organization, then, is that structural hierarchies can facilitate learning of tasks that require acquisition of abstract policy relationships (Paine and Tani, 2005). One reason for such efficiency could be the capability of hierarchical structures to search independently for rules at multiple levels of abstraction (i.e., in parallel). The present results are consistent with this Neuron 66, , April 29, 2010 ª2010 Elsevier Inc. 321

8 perspective in that frontal cortex appears to leverage its hierarchical organization in order to engage in search at multiple levels of abstraction from the outset of learning. Interestingly, these results also provide a potential account of the classical learning versus execution dissociation between PFC and PMd during rule learning. In particular, it has been widely noted that with substantial training, activity in PFC declines and activity in PMd is sustained (Brasted and Wise, 2004; di Pellegrino and Wise, 1993; Hadj-Bouziane et al., 2003; Hoshi and Tanji, 2006, 2007; Lucchetti and Bon, 2001; Mitz et al., 1991; Passingham, 1988, 1989; Petrides, 1985a, b, 1987; Boettiger and D Esposito, 2005). Indeed, lesioning PFC after learning does not impair subsequent execution of the rules (Bussey et al., 2001; Petrides, 1985b). In the present study, learning the Flat rule set is analogous to these past studies of rule learning, given that it involves learning of arbitrary first-order rules. Indeed, perhaps consistent with these past experiments, activation declines over the course of learning in prepmd but not in PMd. However, during learning of the Hierarchical set, activation does not decline but is sustained in prepmd throughout learning. Thus, past distinctions between learning and execution of rules in the frontal cortex may also reflect the fact that most rules in these studies are first-order policy, by our definition, and so may not have been abstract enough to require sustained involvement of the PFC. Finally, these results have implications for the study of changes in the frontal cortex and striatum during reinforcement learning (Brasted and Wise, 2004; Fujii and Graybiel, 2005; Loh et al., 2008; Pasupathy and Miller, 2005). At least two alternative models have been proposed with respect to frontostriatal dynamics during learning. In the first, frontal cortex serves to uncover patterns in the environment that are subsequently consolidated in the basal ganglia (Graybiel, 1998). This hypothesis predicts that cortical activity should precede that of the striatum. Alternatively, the striatum may uncover stimulusreward contingencies that merit more dedicated cortical processing (Houk and Wise, 1995). This alternative hypothesis appears to predict the reverse, namely that basal ganglia activity should precede that of the frontal cortex. These timing differences have been suggested not only to occur across the course of learning, with either cortex or basal ganglia instructing the other across this longer time scale, but also to potentially reflect (and possibly to result from) moment-by-moment precedence of activity (Houk and Wise, 1995). Previous results in the nonhuman primate have supported both sides of this controversy. Pasupathy and Miller (2005) found that recordings from both area 9/46 of the prefrontal cortex and the head and body of the caudate were consistent with the latter hypothesis in macaques performing a well-learned serial reversal task. In their study, caudate activity reliably preceded that of the PFC throughout learning, but moved relatively earlier in time as learning proceeded. However, Fujii and Graybiel (2005) found that local field potentials in prefrontal cortex peaked earlier than LFPs in the striatum on single trials as macaques performed a well-learned serial saccade task (also see Brasted and Wise, 2004). Our effective connectivity results point to a potentially more complex system in which the frontal cortex both influences and is influenced by the striatum during rule learning. Moreover, we demonstrate that this temporal relationship between BOLD signal in the putamen, cortex, and caudate is consistent across the duration of learning. Such a dynamical system is broadly consistent with a range of proposals in the reinforcement learning literature that assume functional divisions both within the striatum and within cortex itself and that acknowledge dynamic interactions between them such that the striatum can influence cortical representations e.g., through updating/ gating and can likewise be influenced by what the cortex represents e.g., for the purposes of learning and action selection (Alexander et al., 1986; Daw et al., 2005; Frank et al., 2004; Grahn et al., 2008, 2009; Hazy et al., 2007; Houk and Wise, 1995; O Reilly and Frank, 2006; O Reilly et al., 2007; Seger and Cincotta, 2005, 2006). In conclusion, our findings suggest that the rostro-caudal architecture of the frontal cortex may support rapid learning of action rules at multiple levels of abstraction. When encountering a novel behavioral context, we may search for relationships between context and action at multiple levels of abstraction simultaneously, a capability that underlies our remarkable behavioral adaptability and our capacity to generalize our past learning to new problems. Hence, how we address novel problems in reasoning, decision-making, and selecting actions under uncertainty may very well reflect both the adaptability and the constraints conferred by the basic functional organization of the frontal cortex. EXPERIMENTAL PROCEDURES Participants Twenty right-handed, native English speakers (eight female; ages years) with normal or corrected-to-normal vision were enrolled in the study. Data from an additional six participants was collected but excluded because of excessive head motion (>3 mm: four subjects) or an inability to learn above chance in either condition (two subjects). All participants underwent prescreening for neurological or psychological disorders, use of medications with potential vascular or CNS effects, and any contraindications for MRI. Normal color vision was verified for all subjects as assessed by the Ishihara test for color deficiency. Participants received a base payment of approximately $56 and an average bonus of $20.57 for correct responses during the task (see Behavioral Procedures). Informed consent was obtained from subjects in accordance with procedures approved by the Committees for Protection of Human Subjects at the University of California, Berkeley and University of California, San Francisco. Logic and Design For investigating the discovery of abstract rules, a reinforcement learning task was designed that required the learning of two rule sets, one of which contained a higher-order rule structure (Hierarchical rule set) and one that could only be learned as one-to-one mappings between stimuli and responses (Flat rule set). Participants were not given an indication through an instruction or any other cue that a higher-order structure existed in one of the rule sets. Moreover, trials for both rule sets were identical in terms of all stimulus presentation parameters, instructions, and response-reward contingencies. Each rule set was learned over the course of 360 individual learning trials divided equally into six fmri scan runs. Each trial commenced with the presentation of a stimulus display consisting of a nonsense object (i.e., without a real-world counterpart) appearing in one of three orientations (up [0 ], left [ 90 ], or oblique [23 ]) and bordered by a colored square. The stimulus display subtended 10 of visual angle. Two colors, three object shapes, and three orientations were used for each rule set, the conjunction of which resulted in 18 unique stimulus displays (i.e., three shapes 3 three orientations 3 two colors). Each of the 18 unique displays occurred 20 times across the six 322 Neuron 66, , April 29, 2010 ª2010 Elsevier Inc.

9 fmri runs for a given rule set. The specific colors and shapes differed across the two rule sets within subject and were counterbalanced for rule set across subjects. The object and square appeared together for 1 s and were then replaced by a green fixation cross that appeared for up to an additional 2 s. While the stimulus display or green fixation cross was present, the participant could respond with one of three buttons using the index, middle, or ring fingers of his right hand. Once a response was made or 3 s had passed without a response, the fixation cross became red and no further responding was allowed. If a participant had not responded by the 3 s deadline, that trial was scored as incorrect. The red fixation cross following a response or upon reaching the response deadline was presented for either 0, 1, or 2 s, after which feedback was provided in the form of an auditory tone. The variable interval permitted estimation of the BOLD response to feedback independent from that to the stimulus display. A pure high tone (750 Hz) indicated a correct response, and a buzzing tone (combination of 300 and 400 Hz pure tones) indicated an incorrect response. Participants were given a $0.05 bonus reward for each correct response. A running total bonus was provided at the end of each run. After feedback, the red fixation cross remained on the screen for a variable null intertrial interval (mean 1.5 s). The order of trials and duration of intertrial intervals within a block was determined by optimizing the efficiency of the design matrix so as to permit estimation of the event-related response (Dale, 1999). Efficiency was equated across rule sets, and the order of rule set learning (i.e., whether Hierarchical or Flat was learned first) was counterbalanced across participants. For both rule sets, participants were given the same instruction. No indication was given that a higher-order relationship existed or that they should search for an abstract rule. Participants did not practice the task but they were allowed to fully familiarize themselves with all 18 stimuli they would encounter for a given rule set prior to conducting the learning trials for that rule set. Hence, there were no differences in any stimulus presentation parameters or instructions between the rule sets. Where the two rule sets differed was simply in the arrangement of mappings between stimulus displays and responses (Figure 1). For the Hierarchical set, the mappings between the 18 stimulus displays and 3 responses were ordered such that in the context of one colored box, shape fully determined the response. In other words, each of the three shapes corresponded to one of the three buttons regardless of the orientation of the object. Conversely, in the context of the other color, orientation fully determined the response. Thus, a second-order rule, linking color and dimension, determined the relevant set of first-order rules that linked shape and response or orientation and response. For the Flat set, the arrangement of responses was such that no such higher-order relationship existed. Thus, each of the 18 first-order rules linking a unique stimulus display with one of the three responses had to be learned individually. Critical to the logic of the experiment, the Hierarchical set could be learned as 18 first-order rules, if participants could not discover the higher-order relationship. By contrast, the Flat structure did not afford the opportunity to acquire second-order rules and so had to be learned as a set of one-to-one mappings between stimulus and response. In counterbalancing the specific mappings between stimulus displays and responses across subjects, two additional constraints were applied beyond those listed above. First, all responses were represented equally across the entire set. Second, given that three of the specific object-orientation combinations in the Hierarchical learning set had the same response regardless of colored box (i.e., those cases in which the orientation and shape cued congruent responses), we ensured that three object-orientation combinations also shared a response across colored boxes in the Flat set, equating this feature of the rule sets. Behavioral Analysis Learning curves were calculated with a state-space modeling procedure (Smith et al., 2004) that estimates the probability of a correct response on each trial as a function of a latent Gaussian state process (i.e., the state of knowledge the subject) and an observable Bernoulli response process (i.e., the responses of the subject). In other words, the model uses the learner s trial-by-trial responses (either correct or incorrect) to estimate his knowledge about the task over time. In contrast with sliding average or other methods of computing learning curves, this approach allows one to define a confidence interval associated with the estimate of learning on each trial. Thus, this method produces a learning trial, or the trial at which the confidence interval no longer encompasses chance performance. We note that because this method estimates a single value for the variance of the Gaussian state process across learning, it does not incorporate details of the task or make assumptions about hierarchical learning. Learning curves with this procedure were calculated both for the entire rule set and also for each of the 18 rules individually based on the 20 encounters with a particular stimulus display. In addition to the behavioral analyses described below, these curves were used for the fmri analysis (see below). On the basis of learning estimates calculated with this approach, we focused our behavioral analysis on four components of the curve: (1) the learning trial, as described in the preceding paragraph; (2) the terminal accuracy (i.e., the probability of a correct response on the final trial), which is related to the degree of learning at the conclusion of the session; (3) the maximal first derivative of the learning curve, which serves as an index of the maximal speed of learning over the session; and (4) the maximal second derivative of the curve, which defines the maximal rate of change in the learning rate over the session. We further conducted a model-based analysis in order to explicitly assess the shape of the learning curves for the Hierarchical and Flat learning sets. In particular, we fit a sigmoid function defined as follows: 1 by = ð1 + e aðx bþ Þ (Equation 1) both to the learning curves, and directly to the subject s binary responses based on Bernoulli assumptions. In this function, a reflects the slope of the sigmoid and b defines the temporal offset relative to the start of the learning session. A step-wise function will have a steep slope (large a), and faster learning will have a shorter offset (smaller b). Parameters (a and b) were estimated with a nonlinear least-squares data fit (the Matlab function nlinfit.m ; Goodness of fit was assessed using a c 2 criterion: c 2 = X i ðy i by i Þ 2 (Equation 2) by i where y i represents the probability of a correct response at time i, as determined by the learning curve, and yˆi represents the sigmoid-derived estimate of this value. We did not include an additional parameter to allow for variation in asymptotic performance. Because the rules are deterministic and occupy a finite space, there is not an a priori reason to believe that learning in the Flat case should asymptote below perfect performance, rather than simply taking longer because of the larger number of individual stimulus-response relationships that must be learned. MRI Procedures Whole-brain imaging was performed on a Siemens 3T TIM Trio MRI system with a standard 12-channel head coil. Functional data were acquired with a gradient-echo echo-planar pulse sequence (TR = 2 s, TE = 28 ms, flip angle = 90 ; 29 axial slices, matrix = , FOV = mm, slice thickness = 3 mm, 203 volume acquisitions per run). High-resolution T1-weighted (MP-RAGE) anatomical images were collected for anatomical visualization. Head motion was restricted with firm padding that surrounded the head. Visual stimuli projected onto the screen were viewed through a mirror attached to the head coil. Auditory feedback was presented through Siemens headphones provided as a stock component with the Trio scanner. All experimental scripts were programmed and run on a Macintosh computer with the Psychophysics Toolbox in MATLAB ( fmri Analysis Functional imaging data were processed with SPM2 (Wellcome Dept. of Cognitive Neurology, London). After quality assurance procedures for assessment of outliers or artifacts in volume and slice-to-slice variance in the global signal, functional images were corrected for differences in slice acquisition Neuron 66, , April 29, 2010 ª2010 Elsevier Inc. 323