Oxford Handbooks Online

Transcription

1 Oxford Handbooks Online Hierarchical Cognitive Control and the Functional Organization of the Frontal Cortex David Badre The Oxford Handbook of Cognitive Neuroscience: Volume 2: The Cutting Edges Edited by Kevin N. Ochsner and Stephen Kosslyn Print Publication Date: Dec 2013 Online Publication Date: Dec 2013 Subject: Psychology, Cognitive Neuroscience DOI: /oxfordhb/ Abstract and Keywords Cognitive control refers to the ability of humans and other primates to internally guide behavior in concert with goals, plans, and broader contextual knowledge. However, in everyday life, we often must manage multiple goals at once over different time scales and at different levels of abstraction with respect to an overt response. Recent cognitive neuroscience research has suggested that cognitive control processing in the frontal lobes of the brain may be hierarchically organized along their rostrocaudal axis in order to deal with this problem. This chapter provides a brief overview of current research on hierarchical cognitive control and how this function may emerge from the functional organization of the frontal lobes. Keywords: cognitive control, executive function, prefrontal cortex, learning and generalization, planning Introduction Human behavior is marked by its flexibility. We choose the courses of action that are likely to achieve our goals from a vast space of candidate actions. Moreover, as circumstances change and previous plans become obsolete, we can rapidly change our behavior to accommodate the new situation and still reach our goals. However, having such a large behavioral repertoire comes with a cost, namely, the problem of choice. With so many ways to act, how do we navigate this broad space and maintain coherent, goal-directed behavior? Part of the answer to this question lies in our capacity for cognitive control, or the mechanisms by which we use plans, goals, or features of our environment to constrain action selection (Badre & Wagner, 2004; Bunge, 2004; Miller & Cohen, 2001; O Reilly & Frank, 2006). However, another important consideration is the way that we structure action representations and the cognitive control system itself in order to reduce the number of alternatives from which a choice must be made (Badre, 2008; Botvinick, Niv, & Barto, 2009; Cooper & Shallice, 2006; Lashley, 1951; Schank & Abelson, 1977). Consider, for example, the task of making a sandwich. Even if one eliminates the problem of deciding which type of sandwich to make, there are many different ways of constructing a particular sandwich, such as spreading mayonnaise before or after the mustard. Moreover, consider how difficult this problem becomes if we could plan only in terms of sequences of motor-effector movements. It is clear that the spread of potential options would be vast, and comparison among them in order to choose one plan over another by a cognitive control system would be intractable. Hierarchies are useful in dealing with large spaces of options, such as in the problem of action, because they permit a divide-and-conquer approach (Newell, 1990; Rosenbaum, 1987). Returning to our sandwich example, one can represent this task as a very abstract goal: make a sandwich. This abstract goal can be broken down into a more specific series of subgoals like slicing bread, spreading mayonnaise, etc. These subgoals can be broken down further into more specific sub-subgoals and so forth until the task is decomposed into a highly specific Page 1 of 20

2 sequence of neuromuscular outputs. Thus, choices about which actions to take can be made at multiple levels of abstraction. This has the benefit of separating updating, monitoring, and maintenance of contextual information relevant to each level independently. And, importantly, choices at the higher levels will constrain the space of possible actions at lower levels, reducing complexity and the demands on choice. Recent evidence has suggested that the cognitive control of action is hierarchical in nature and emerges from the functional organization of the frontal cortex (Badre, 2008; Badre & D Esposito, 2009; Botvinick, 2008; Koechlin & Summerfield, 2007; O Reilly, 2010). In this chapter, we will consider evidence for hierarchical control as it relates to frontal lobe function, how this type of architecture may support flexibility and abstraction in action, and, finally, a model for how hierarchical control mechanisms may operate in the brain. Policy Abstraction: Rules that Govern Rules Cognitive control refers to our ability to guide our behavior on the basis of internal representations of goals, plans, and context (Cohen, Dunbar, & McClelland, 1990; Miller & Cohen, 2001). Consider the everyday example of entering a colleague s office and finding a place to sit down. On a daily basis, in one s own office, the chair behind one s desk is the appropriate seat. However, in another s office, even one with which we have no prior experience, we easily alter our behavior and look for the chair in front of the desk. Our ability to flexibly shift our behavior without multiple trials of reinforcement and based on an abstract social rule depends on cognitive control. In the brain, the frontal lobes are known to be necessary for cognitive control function (Stuss & Benson, 1987). Cognitive control mechanisms are generally thought to operate through a process of biased competition, whereby maintenance of a distributed neural representation of the task context ( colleague s office ) by frontal neurons biases processing throughout the action system in favor of an appropriate course of action ( sit in the chair in front of desk ) over competing ones ( sit in the chair behind the desk ) (Cohen et al., 1990; Cohen & Servan-Schreiber, 1992; Desimone & Duncan, 1995; Miller & Cohen, 2001). Thus, cognitive control permits a state of the system (i.e., context ) to influence selection of an operator, such as a particular action, that is likely to achieve a desired outcome state (the goal ). A rule that relates a state, an action, and a desired outcome has been termed policy in the reinforcement learning literature (Botvinick, 2008; Botvinick et al., 2009). Thus, from this perspective, the investigation of cognitive control is centrally concerned with how the brain acquires, selects, and implements action policy. Click to view larger Figure 19.1 Schematic example of a second-order policy structure. a. First-order policy mappings based on shape. Circles cue a left (L)-hand response and rectangles cue a right (R)-hand response. b. First-order policy mappings based on size. Large shapes cue a left-hand response and small shapes cue a right-hand response. c. In this example, the color of the stimulus determines whether to use the shape or size rules. The rule relating color to first-order policy sets is second-order policy. Expressed as a tree structure, the order of policy abstraction can be determined by the number of branch points that must be traversed in order to determine a response. Policy provides a useful way of conceptualizing hierarchical abstraction in a cognitive control system. In particular, a policy can be considered abstract to the extent that it determines a class of simpler policy based on a state. For example, consider two simple rules: a circle cues a left-hand response and a rectangle a right-hand response (Figure 19.1a). This first-order policy is concrete because the state (the identity of the shape) fully specifies what action to take (which hand to move). However, consider an independent set of first-order policy, based on size, in which a large stimulus cues a left-hand response and a small stimulus a right-hand response (Figure 19.1b). Because the shape and size rule sets are independent, both cannot simultaneously govern responding. For Page 2 of 20

3 instance, if the relevant set of first-order policy is unknown, a stimulus that is both circular and small cues opposing responses. Consequently, a more abstract rule (second-order policy) is required in order to specify which set of first-order rules (shape or size) should govern responding in the current context. For example, coloring the stimulus red might indicate that the shape set is the appropriate first-order policy, while blue might indicate size (Figure 19.1c). Because this second-order policy based on color specifies a class of simpler rule sets (shape or size) but not a specific response, it is more abstract. Considerable research in both humans and animals has implicated the frontal cortex in the acquisition and implementation of simple behavioral rules (Asaad, Rainer, & Miller, 1998, 2000; Bunge, 2004; Bunge, Kahn, Wallis, Miller, & Wagner, 2003; Wallis, Anderson, & Miller, 2001; Wallis & Miller, 2003; White & Wise, 1999), particularly those involving first-order policy. Recently, our lab and others have become interested in how the brain s cognitive control system is organized to processes more complex policy structures that are likely critical in accounting for the massive flexibility and vast behavioral repertoire evident in human and nonhuman primates. Policy Abstraction and the Rostrocaudal Axis of Frontal Cortex Fuster s perception action cycle theory was the first to associate a concept of abstraction in action control with the functional organization of frontal cortex (Fuster, 1997, 2001, 2004). Fuster proposed a series of loops between perceptual hierarchy in posterior neocortex and an action hierarchy along the rostrocaudal axis of the frontal lobes. According to the theory, as abstract plans are translated into concrete responses, progressively posterior regions of lateral frontal cortex are responsible for integrating more concrete inputs from sensory systems over more proximate time intervals. Thus, in the perception action cycle, the hierarchical structure of action may be represented in the organization of cortical areas from rostral to caudal frontal cortex. The first empirical evidence regarding a rostrocaudal organization of the frontal cortex came from an fmri study in which participants were tested while performing response and task-set selection tasks (Koechlin, Ody, & Kouneiher, 2003). In the response selection task, participants selected a finger response based on a colored cue. In the task-set selection task, participants chose a letter classification task based on a colored cue. Selection of a finger response in the response selection task was associated with dorsal premotor cortex (PMd), whereas selection of the task set for letter classification was associated with the more rostral anterior premotor cortex (prepmd). In addition, across both experiments, prior to a block of trials, an instruction indicated what the mappings between color, and task and response were going to be in the upcoming block. Importantly, on blocks when the mappings were infrequent, this instruction had to be maintained or refreshed in some way in order to interpret the color and select the right rule on each trial. The low-frequency conditions elicited activation inclusive of a further anterior region, mid-dorsolateral PFC (area 9/46). Thus, because the contextual information required to make a choice became more abstract and temporally remote, more rostral regions of the frontal cortex were required to engage in control. To what extent did this rostrocaudal gradient in frontal cortex reflect differences in policy abstraction, as defined above? In order to test this, we developed a paradigm that permitted manipulation of cognitive control demands over four levels of policy abstraction (Badre & D Esposito, 2007). Importantly, we parametrically taxed policy at one level at a time while keeping demands on higher-level policy minimal and lower-level policy constant. Thus, we were able to distinguish which regions of the brain were selectively sensitive to each level of policy independently from correlated changes in control demands at lower levels and from overall changes in difficulty. In order to understand the logic of the experiment, it is helpful to return to our example of second-order policy from the preceding section. Mapping out the decisions (Figure 19.1c) results in a two-tiered decision tree with each branch point representing a decision, or a point at which some contextual information will bias selection of one action path over another. The depth of the decision tree remaining to be traversed from any branch point in order to reach a response determines the order of policy abstraction. Thus, in our example, deciding which rule set, color or shape, is appropriate requires traversing two branch points and so represents a decision at a second order of policy abstraction. Deciding what response to make based on shape, by contrast, requires traversing only one branch point and so is at a first order of policy abstraction. In an experiment, one can continue to add layers of contingency and thus additional branch points to the tree, thereby requiring decisions to be made at higher orders of policy abstraction. The order of policy abstraction, then, may be determined by the depth of the tree Page 3 of 20

4 (number of branch points) to be traversed to determine a response. Click to view larger Figure 19.2 Task used for testing four levels of policy abstraction in study by Badre and D Esposito (2007). a. Task schematics showing trial events for the response, feature, dimension, and context tasks. In the response task, the color of the box on every trial determined what response to make. In the feature task, the color determined the target feature (indicated as a particular texture above each trial). When the target texture appeared, this resulted in a positive response; other textures received a negative response. In the dimension and context tasks, the color determined the relevant dimension (shape, size, orientation, or texture) along which to compare two presented objects. Participants decided if the objects matched or not along the relevant dimension. In the context task, the frequency of mapping between the color and the relevant dimension was also varied to add an additional level of selection. b. Policy structures associated with each of the four tasks. The mid- and high-competition conditions of each experiment require selection at first (response), second (feature), third (dimension), and fourth (context) orders of policy abstraction, respectively. P, positive; N, negative; M, match; NM, non-match. Of course, making a choice more abstract will arguably increase complexity of the rule and, in some cases, its difficulty. Thus, one wants to be sure that differences in fmri activation can be attributed to changes in abstraction and not to task difficulty in general. Importantly, the decisions at any branch point can also be made more difficult without increasing abstractness by increasing the number of alternatives at that level (or reducing the frequency of any one alternative). For example, at a second order of policy abstraction, choosing among four orthogonal rule sets (e.g., defined by shape, size, texture, and orientation) will be more difficult than choosing among two rule sets. Increasing the number of alternatives increases competition and thus will demand greater control at a second level of policy abstraction. Increasing competition puts greater demands on cognitive control mechanisms to make a choice, but adding alternatives does not produce a deeper tree. Although difficulty increases, the order of policy abstraction does not. Thus, policy abstraction and competition may be unconfounded by separately varying the depth and width of the tree. Using this logic, Badre and D Esposito (2007) independently tested the response of frontal cortex to increasing cognitive control demands at four levels of policy abstraction during fmri scanning (Figures 19.2 and 19.3). Page 4 of 20

5 Click to view larger Figure 19.3 Results from fmri from parametric manipulations of control at four levels of policy abstraction. Surface-rendered activation (right) and selectively averaged time courses from regions of interest (left) show that increasing the competition from low to mid to high in the response, feature, dimension, and context experiments resulted in selective, parametric increases in fmri activation in dorsal premotor cortex (PMd; BA 6), anterior PMd (prepmd; BA 8), inferior frontal sulcus (IFS; BA 9/46), and frontal polar cortex (FPC; BA 46 or 10). Response Task During the response task (first-order control), participants were required to select a response on each trial based on the color of a presented square (Figure 19.2a). Competition at a first order of policy abstraction was manipulating by requiring a choice from among one, two, or four alternative responses. Increasing competition at a first order of policy abstraction (Figure 19.2b) resulted in a parametric increase in activation in PMd (Figure 19.3). Feature Task On each trial of the feature task (second-order control), participants were presented with a single object inside a colored box (Figure 19.2a). From trial to trial, the object varied along one dimension, such as texture. The participant looked for a particular target texture and made a positive response if the target texture was present and a negative response to any other texture. On any trial, only one texture entailed a positive response and other textures entailed a negative response. However, on the next trial, the target could change (as cued by a different color), and so a different texture could require a positive response. Knowledge of the texture alone was insufficient to determine which response mapping (positive or negative) was required on that trial. Rather, the participant knew the set of mappings between textures and responses based on the colored box. Thus, control at a second level was required in that one set of texture-to-response mappings had to be selected over competing sets (Figure 19.2b). As the number of sets of feature-defined mapping sets increased from one to two to four, activation increased in caudal PFC/anterior PMd (prepmd) (Figure 19.3). Dimension Task On each trial of the dimension task (third-order control), participants were presented with two objects appearing inside a colored box (Figure 19.2a). From trial to trial, objects varied along four dimensions: texture, orientation, shape, and size. The participant decided whether the objects matched or mismatched along one of the dimensions. The color of the surrounding box was required in order to decide which dimension was relevant to the decision. Control at a third order of abstraction was necessary to the extent that the relevant dimension had to be selected from two or more competitors (Figure 19.2b). As the number of candidate dimensions increased from one to two to four, activation increased in dorsolateral PFC (DLPFC), specifically along the upper bank of the inferior frontal sulcus (IFS) (Figure 19.3). Context Task Page 5 of 20

6 Finally, during the context task (fourth-order control), participants were again required to make a match decision between two objects (Figure 19.2a). They also had to choose which of two dimensions was relevant based on the color of the surrounding square. However, in the context task, a given color could map to different dimensions on different blocks. To the extent that a given color-to-dimension mapping is infrequent across blocks, fourth-order control is required to select the relevant mapping (Figure 19.2b). Thus, as the frequency of a given color to dimension mapping decreased from 100% to 50% to 25%, activation increased in frontal polar cortex (FPC) (Figure 19.3). To summarize, across these four separate tasks, competition and policy abstraction were independently varied. Policy abstraction affected the locus of activation, with progressively anterior regions being associated with added orders of abstraction. By contrast, competition at any level of policy abstraction was associated with greater BOLD activation but not increases in more rostral regions. This finding is important because it indicates that difficulty alone does not account for the caudal-to-rostral organization. Rather, increases in policy abstraction elicit activation of progressively rostral frontal cortex. Alternatives to Policy Abstraction These data provide motivation for characterizing the rostrocaudal axis in terms of policy abstraction, but this experiment does leave open important alternatives regarding how to rank the levels of hierarchy in frontal cortex. Indeed, abstraction has been defined in different ways across separate bodies of evidence concerning rostrocaudal functional organization (Badre, 2008). Most notable among the alternatives to policy abstraction are temporal abstraction and relational complexity. Temporal Abstraction Temporal abstraction is a concept highly related to policy abstraction and refers to a time scale over which a contextual representation must influence action. It has also been hypothesized that regions along the rostrocaudal axis are differentiated on the basis of temporal abstraction (Botvinick, 2007, 2008; Botvinick et al., 2009; Fuster, 2001; Koechlin & Summerfield, 2007). Simply put, separable pools of neurons along the rostrocaudal axis of the frontal cortex may differ in their ability to maintain information and resolve contingencies over longer time intervals. As action representations become more concrete, going from abstract goals down to a specific sequence of neuromuscular outputs, goal and subgoal information must be updated more frequently and sustained over shorter intervals and so temporal abstraction decreases. The relationship between temporal abstraction and the rostrocaudal axis of the frontal lobe has received support from a number of convergent sources. In particular, electrophysiological evidence has distinguished PFC from premotor regions on the basis of neurons in PFC mediating longer cross-temporal contingencies (Fuster, Bodner, & Kroger, 2000). FPC, at the most anterior extent of the rostrocaudal axis, has been associated across a range of experiments with a sustained response relative to higher-frequency event-related responses in posterior PFC (Braver, Reynolds, & Donaldson, 2003; Donaldson, Petersen, Ollinger, & Buckner, 2001; Visscher et al., 2003). Most directly, however, the data from the response and task-set selection experiments by Koechlin et al. (Koechlin & Summerfield, 2007) described above have been argued to demonstrate a rostrocaudal organization in the integration of control signals arising from increasingly temporally remote contexts. In their model, from premotor to posterior DLPFC, control relies on information available in the immediate environment. Within this posterior zone, regions are distinguished from one another on the basis of demand to select longer sequences or sets of responses (Koechlin & Jubault, 2006). Anterior DLPFC and FPC engage in control relying on information that is not present in the current environment but occurred either in the current temporal frame, such as an ongoing task condition (anterior DLPFC) (Koechlin et al., 2003), or during a previous context (FPC) (Koechlin, Basso, Pietrini, Panzer, & Grafman, 1999; Koechlin, Corrado, Pietrini, & Grafman, 2000), such as is required during subgoaling (Braver & Bongiolatti, 2002). Relational Integration Abstraction through relational integration suggests that regions along the rostrocaudal axis are differentially Page 6 of 20

7 required given the number of variable dimensions that must be integrated in order to determine a response (Bunge, 2004; Bunge, Wendelken, Badre, & Wagner, 2005; Christoff & Gabrieli, 2000; Christoff & Keramatian, 2007; Christoff, Ream, Geddes, & Gabrieli, 2003; Halford, 1993; Ramnani & Owen, 2004; Robin & Holyoak, 1995). A first order of relational complexity requires only a single dimension be considered, as in assigning a simple property to a specific item ( What is the color? ), and has been associated with ventrolateral PFC (VLPFC) (Bunge et al., 2003, 2005; Christoff & Keramatian, 2007; Kostopoulos & Petrides, 2003; Wagner, Paré-Blagoev, Clark, & Poldrack, 2001). By contrast, a second order of complexity requires drawing simple relations between concrete properties ( Do the colors match? ) and is associated with DLPFC according to neuroimaging and neurophysiological evidence (Bunge et al., 2003; Christoff & Keramatian, 2007; Wallis et al., 2001; Wallis & Miller, 2003). A secondorder relation is more abstract because it does not depend on a specific property of an item and may be generalized to novel items. A higher, third order of relational complexity, with a hypothesized association with FPC, entails evaluation of relations among relations. For example, deciding whether the mismatching dimension (texture or shape) of a target pair matches the mismatching dimension of a subsequently presented target pair is associated with activation in FPC (Christoff et al., 2003). Thus, from the perspective of relational complexity, abstraction can be varied experimentally to the extent that a task requires generation of relationships that become increasingly removed from the specific properties of a stimulus. Increasing the number of relations that must be integrated should result in processing progressing from VLPFC (first order) to DLPFC (second order) to FPC (third order). Of course, these types of abstraction are not necessarily mutually exclusive with respect to each other or policy abstraction and may share a number of common properties. For example, temporal and policy abstraction are highly correlated. Representations that are more abstract in policy terms are also likely to be relevant over longer time scales. Similarly, policy abstraction and relational integration both require monitoring and updating of increasing numbers of independent contextual dimensions as abstraction increases. Thus, although precisely defining abstraction with respect to rostrocaudal frontal organization is a key theoretical point to be clarified in future research, there is a common theme that more rostral regions of the frontal cortex are involved in more abstract, higher-order control operations. Consequently, in the interest of conceptual clarity, the remainder of the chapter will use policy abstraction as its working framework for discussing frontal hierarchy. Evidence for a Processing Hierarchy in the Frontal Cortex In light of the previous discussion, there is now fairly strong convergent evidence for a gradient of abstraction along the rostrocaudal axis of frontal cortex that could support hierarchical cognitive control. However, a key feature of a hierarchy is that influence and inheritance are asymmetrical from superordinate to subordinate levels relative to the reverse order. Thus, a key question is whether this gradient of abstraction in cognitive control actually reflects a processing hierarchy in lateral frontal cortex, whereby rostral neurons influence activity in caudal neurons more than the reverse situation. In a review of the corticocortical anatomy of frontal cortex, Badre and D Esposito (2009) suggested that rostrocaudal frontal connectivity displays two features that are prerequisites for a processing hierarchy such as that described above (Figure 19.4a). In particular, adjacent regions along the rostrocaudal axis are directly connected to one another. Notably, it is not the case that every frontal region shares connections with all regions adjacent to it. For example, the dorsal or ventral adjacent regions may not share direct connections (such as ventral vs. dorsal 9/46). Thus, this contiguity principle is a specific property of the rostrocaudal connectivity of the frontal lobe. More importantly, frontal subregions do not project to more rostral regions beyond those regions immediately adjacent, whereas rostral regions do project to caudal regions beyond those immediately adjacent. This asymmetry of input from rostral to caudal is consistent with a processing hierarchy. Page 7 of 20

8 Click to view larger Figure 19.4 Connectivity along the rostrocaudal axis of the frontal cortex. a. Intrinsic connections of the lateral prefrontal cortex (PFC; top) and a schematic summary of the connections of the principal frontal regions (area 10, shown in orange; area 9/46, shown in green; and area 6, shown in blue) that are proposed to be part of a rostrocaudal functional gradient based on functional studies (bottom). Area 4 depicts the primary motor cortex. b. Results from Vincent et al. (2008) showing that spontaneous activity in regions along the rostrocaudal axis of the PFC and in parietal and medial frontal cortex is correlated with activity in the frontopolar cortex (shown in light green). Also depicted in the figure is the spatial relationship of these regions to two other networks: the dorsal attention system (DAS) and the hippocampal-cortical memory system (HCMS), which were identified using visual motion area MT+ and the hippocampus as seeds, respectively. FPCS, frontoparietal control system. Reproduced with permission from Badre and D Esposito (2009). In line with the anatomy, evidence from functional connectivity analysis at rest has provided evidence that the regions along the rostrocaudal axis of frontal cortex (46/10, 9/46, 8, and 6) couple as a coherent functional network (Vincent, Kahn, Snyder, Raichle, & Buckner, 2008), along with regions of parietal cortex and the posterior temporal lobe (Figure 19.4b). Owing to its general association with cognitive control, this network has been termed the frontoparietal control system, and is distinguishable from at least two other coherent networks: a more dorsal attentional network and a ventral hippocampal network. Importantly, effective connectivity values, which provide a direction of influence as estimated with structural equation modeling of fmri data, have been shown to differentially flow from front to back in the frontal cortex within this network (Koechlin et al., 2003; Kouneiher, Charron, & Koechlin, 2009). However, the neuroimaging data cannot be conclusive on this point. Indeed, some perspectives can account for a rostrocaudal functional gradient without a requirement that the processing architecture be hierarchical (Christoff & Gabrieli, 2000; Christoff & Keramatian, 2007; Christoff et al., 2003; Petrides, 2005). Thus, a fundamental issue to resolve is whether the observed rostrocaudal gradient reflects a hierarchical or nonhierarchical organization of function (Badre, 2008). To test the asymmetry hypothesis, we asked 12 individuals with focal frontal lobe lesions and 24 age-matched controls to perform the four response selection tasks (response, feature, dimension, and context) performed previously and for which we had fmri data (see Figure 19.2) (Badre, Hoffman, Cooney, & D Esposito, 2009). An anterior-to-posterior flow of control processing in the frontal lobes predicts that performance on tasks involving higher-order control should be impaired by disruptions to lower-order processors, even when the higher-order processors are intact. However, the reverse prediction should not hold. Performance should be unaffected for tasks involving only intact lower-order processors when higher-order processors are impaired. Put more concretely, a deficit in third-order control would affect performance on the dimension task but not the feature or response tasks. Conversely, a deficit in second-order control would impact the feature and dimension tasks. Page 8 of 20

9 The results from this experiment provided evidence for this type of asymmetrical deficit pattern. On this task, patients showed a greater deficit compared to controls as the task rules became more abstract. There can be two explanations for this finding: (1) higher-order control demands could increasingly challenge all patients, regardless of the site of their lesion, thus their performance would become differentially impaired as the task complexity increases, or (2) because of the asymmetrical dependencies predicted by a hierarchy, deficits in higher-level tasks would be more likely across patients, regardless of the site of their lesion, resulting in a greater aggregate likelihood of a deficit. If the aggregation account is the case, then the presence of an impairment at any level should increase the likelihood of an impairment at all higher levels but should not increase the odds of an impairment at a lower level. The probability of a deficit on any task, p(d), was 62% across the patients. Critically, however, the probability of a deficit at any level given a deficit at a lower level, p(d L), was 91% across patients, a significant change over p(d) on any task. By contrast, the probability of a deficit at any level given a deficit at a higher level, p(d H), was only 76%, a weak change over the prior probability of a deficit on any task. This asymmetry provides initial support for the hierarchical dependencies among deficits at the different levels and the aggregation account of the group data. We then used an observer-independent method to assign patients to lesion-overlap groups based on their behavioral performance across the four tasks. Vectors were created that corresponded to the idealized behavior of a patient with a selective deficit at a particular hierarchical level. These vectors served as regressors in a multiple regression on each patient s performance differences from age-matched controls across all conditions of all experiments. Based on this regression, a patient was assigned to a particular lesion-overlap group. Two groups of patients were found (Figure 19.5). Figure 19.5a shows the lesion-overlap map of patients that had a behavioral pattern consistent with a deficit at the feature level. In other words, they were impaired on the second-, third-, and fourth-level tasks but not at the firstlevel task. The site of their maximal lesion shown in dark red was within area 8, pre-pmd, in an almost identical location to that identified in our prior fmri study, which is shown directly above the lesion overlap. Figure 19.5b shows the lesion overlap of patients that had a behavioral pattern consistent with a deficit at the dimension level. In other words, they were impaired on the third- and fourth-level tasks but not at the lower first and second levels. The site of maximal lesion was within IFS, area 9/46, in an almost identical location to that identified in our prior fmri study. Taken together, these results provide evidence of a rostral-to-caudal asymmetry in the flow of cognitive control, which is consistent with a hierarchical processing architecture in the frontal cortex. Hierarchical Learning and Abstraction Flexible behavior requires the ability to rapidly adapt to novel circumstances and apply abstract rules to concrete actions. In this regard, hierarchies are convenient structures. Because a hierarchy represents a task at multiple levels of abstraction (Estes, 1982; Lashley, 1951; Miller, Galanter, & Pribram, 1960; Newell, 1990), analogy and the application of existing knowledge is easier at higher levels when encountering a novel task. Thus, hierarchical structures can support chunking, whereby a known higher-order relationship eases the learning of specific examples (Chase & Simon, 1973). Hierarchies can support transfer of a strategy or rule set to a novel task because analogies can be drawn at abstract levels of task structure (Gick & Holyoak, 1980, 1983; Speed, 2010). And, failures to perceive these higher-order relationships may underlie deficits in abstract reasoning seen in various patient groups (Butterfield, Wambold, & Belmont, 1973; Solomon, Ozonoff, Cummings, & Carter, 2008). Page 9 of 20

10 Click to view larger Figure 19.5 Patient overlap maps showing selective, asymmetrical deficits at second and third orders of policy abstraction. Patients were assigned using an observer-independent method to feature or dimension deficit groups depending on whether their behavior showed an asymmetrical deficit at second or third levels of policy abstraction. Sites of highest overlap are shown in red and correspond to prepmd and IFS for the feature and dimension groups, respectively. Results associated with second- and third-order control from fmri (Badre & D Esposito, 2007) are shown for comparison. Reproduced with permission from Badre et al. (2009). It remains an open question whether these putative advantages conveyed by hierarchies are supported by a hierarchical control system along the rostrocaudal axis of the frontal lobe. However, some evidence supports the role of the frontal lobe in this type of higher-order cognition. For example, more anterior regions of the PFC have been associated with the ability to draw analogies and higher-order relations. Explicit tests of analogy, such as verifying the statement sailor is to boat as pilot is to plane, are associated with FPC (Bunge et al., 2005). By contrast, activation is observed in VLPFC during simple retrieval of semantic information, as in who operates a boat? (Badre, Poldrack, Pare-Blagoev, Insler, & Wagner, 2005; Badre & Wagner, 2007; Bunge et al., 2005; Wagner et al., 2001). Recently, increasing numbers of studies have tied the rostrocaudal organization of frontal cortex to relational and fluid reasoning and analogy (Badre, 2010; Christoff, Keramatian, Gordon, Smith, & Madler, 2009; Golde, von Cramon, & Schubotz, 2010; Krawczyk, McClelland, Donovan, Tillman, & Maguire, 2010; Krawczyk, Michelle McClelland, & Donovan, 2011; Speed, 2010; Volle, Gilbert, Benoit, & Burgess, 2010). So, in order to more directly connect policy abstraction to learning and generalization, we conducted an fmri experiment to assess whether the rostrocaudal axis of frontal cortex helps in the discovery of abstract rules when they are available in a novel learning context (Badre, Kayser, & D Esposito, 2010). Specifically, using the policy abstraction definition, we designed a novel reinforcement learning task that contrasted a learning context when participants had an opportunity to acquire an abstract rule (second-order policy) against one in which only concrete (first-order policy) rules were available. During fmri scanning, participants were required to learn two sets of rules, in separate epochs, that linked each of 18 different stimuli uniquely and deterministically to one of three button-press responses. For each rule set, an individual stimulus consisted of one of three shapes, at one of three orientations, inside a box that was one of two colors, for a total of 18 unique stimuli (3 shapes 3 orientations x 2 colors). Participants were instructed to learn the correct response for each stimulus based on auditory feedback (Figure 19.6a). Page 10 of 20

11 Click to view larger Figure 19.6 Schematics of the hierarchical learning task. a. Trial events during all epochs of the hierarchical learning task. On each trial, participants received a shape, at a particular orientation, surrounded by a colored box. Based on these features they selected one of three responses on the keypad. This was followed by feedback indicating whether the response was correct or not. ITI, Inter-trial-interval. b. Policy structure for the flat condition. The arrangement of mappings was such that 18 unique mappings had to be learned between each conjunction of shape, orientation, and color and a response, yielding a very wide flat first-order structure. c. Policy structure for the hierarchical condition. The arrangement of mappings was such that participants could select either a limited set of shape or orientation rules based on color. This results in a second-order policy structure. For one of the two rule sets, termed the flat set, each of the 18 rules had to be learned individually as one-to-one mappings (first-order policy) between a conjunction of color, shape, and orientation and a response (Figure 19.6b). In the other set, termed the hierarchical set, stimulus display parameters and instructions were identical to those for the flat set. Indeed, the hierarchical set could also be learned as 18 individual first-order rules. However, the arrangement of response mappings was such that a second-order relationship could be learned instead, thereby reducing the number of first-order rules to be learned (Figure 19.6c). Specifically, in the context of one colored box, only the shape dimension was relevant to the response, with each of the three unique shapes mapping to one of the three button responses regardless of orientation. Conversely, in the context of the other colored box, only the orientation dimension was relevant to the response. Thus, the hierarchical rule set permitted learning of abstract, second-order rules mapping color to dimension along with two sets of first-order rules (i.e., specific shape-to-response and orientation-to-response mappings). Critically, all instructions, stimulus presentation parameters, and between-subject stimulus orderings were identical between the two rule sets. The flat and hierarchical rule sets only differed in that the organization of mappings in the hierarchical set permitted learning of a second-order rule. Hence, these two sets contrast a learning context in which abstract rules can be discovered with an analogous context in which no such rules can be learned. Results from this experiment provide fundamental insights into the way that humans approach novel learning problems. First, participants were clearly capable of rapidly acquiring abstract rules when they were available. Relative to flat-set learning, which showed a slow, monotonic increase over the course of learning, learning curves during the hierarchical set were associated with rapid, step-function increases (Figure 19.7a). Behavioral analysis indicated that this rapid increase was due to generalization of an abstract rule to multiple specific instances. Page 11 of 20

12 Second, consistent with prior evidence, activation was greater in prepmd for hierarchical- than for flat-set learning epochs. Importantly, however, this pattern emerged because of a decline in prepmd over learning (Figure 19.7b). In other words, activation was evident in both PMd and prepmd early in learning but declined in the more rostral prepmd by the end of learning of the flat set, which contained no second-order rules. This pattern suggests that participants were searching, early on, for any higher-order rules a search process that depended on prepmd. However, during flat set learning, when such rules were not rewarded, this activation declined progressively. Consistent with the search hypothesis, behavioral differences between the hierarchical- and flat-set learning curves correlated with individual differences in early activation in prepmd. In other words, to the extent that an individual activated prepmd (though not PMd) early in learning, this individual was more likely to discover the abstract rule when it was available. Click to view larger Figure 19.7 Results from fmri study of hierarchical learning. a. Learning curves from a representative participant from the task. For the flat condition, learning increased gradually over the course of the trial. For the hierarchical condition, learning demonstrated a step function rapidly reaching ceiling. The learning trial, or point when learning is better than chance (light arrow), was reliably earlier, and the terminal accuracy (dark arrow) was reliably greater in the group between the hierarchical and flat conditions. b. Integrated percent signal change (ipsc) (y-axis) across early, middle, and late phases of learning (x-axis) for flat (dark bar) and hierarchical (light bar) conditions for PMd (left) and prepmd (right) regions of interest. PrePMd showed activation early for both hierarchical and flat conditions. In contrast to PMd, activation in prepmd declined by the middle phase of learning. Overall, these results provide potential insight into the advantage that a hierarchical architecture conveys over other schemes, particularly where the ability to make abstractions with regard to action selection is of central benefit. It has been demonstrated that though complex action may be represented hierarchically (i.e., in terms of goals, subgoals, etc.), the existence of hierarchical representations does not require that the action system itself segregate these representations among spatially separate pools of neurons (Botvinick, 2007; Botvinick & Plaut, 2004). However, one advantage of having such an organization is that structural hierarchies can facilitate learning of tasks that require acquisition of abstract policy relationships (Paine & Tani, 2005). One reason for this efficiency could be the capability of hierarchical structures to search independently for rules at multiple levels of abstraction in parallel. The results from the learning experiment are consistent with this perspective in that frontal cortex appears to leverage its hierarchical organization in order to engage in search at multiple levels of abstraction from the outset of learning. Mechanisms of Hierarchical Control A fundamental open question concerns the mechanisms by which the brain carries out hierarchical control. Moreover, what mechanisms support higher-order rule discovery, such as that described above? To begin to address these questions, we sought to model the Badre et al. (2010) learning task by adapting an established neural model of cognitive control (Frank & O Reilly, 2006; O Reilly & Frank, 2006) for hierarchical control (Badre & Frank, 2012; Frank & Badre, 2012). However, before describing the hierarchical model, we need to first introduce the concept of adaptive gating. As described at the outset of the chapter, cognitive control is hypothesized to operate through a system of biased Page 12 of 20

13 competition whereby relevant contextual information is maintained in the PFC to bias selection of relevant over irrelevant action pathways. However, we often encounter many aspects of our world that are simply irrelevant to our present purposes. Thus, for biased competition to work, it is important to selectively input relevant contextual information into working memory while keeping irrelevant information out. This function is termed input gating. Moreover, even when relevant information is gleaned from the environment and maintained in working memory, one may want to hang onto it for a period of time without its mere maintenance in the PFC actually influencing our behavior. Consider, for example, in a task with many subgoal steps, some information may need to be maintained but not influence the task until a specific time in the sequence of steps. Thus, one also needs a mechanism of output gating that determines which items currently maintained in working memory can currently influence behavior. O Reilly and Frank (2006) proposed a neural network architecture in which both input and output gating are achieved through interactions between the PFC and the striatum. From this perspective, the striatum disinhibits thalamic units to permit certain components of the sensory input to pass into the PFC (i.e., input gating). The striatum also determines which representations maintained in PFC influence selection of a motor response (i.e., output gating) via similar mechanism. However, in this case, the striatum disinhibits thalamic units that interact with the output cortical layers of PFC (corresponding to lamina 5/6). Importantly, the selection of which representations to input and output gate is learned via a common dopaminergic reward prediction error (RPE) signal that modulates activity in go and no-go striatal neuronal populations expressing D1 and D2 dopamine receptors, respectively (Frank, 2005; O Reilly & Frank, 2006; Shen, Flajolet, Greengard, & Surmeier, 2008). Click to view larger Figure Schematic of the neural net model of hierarchical control in frontal corticostriatal loops, from Frank and Badre (2012). Circles represent labeled regions of frontal cortex or striatum. Arrows indicate direction of influence, with a recursive arrow indicating a recurrent maintenance loop. From the model, rostral frontal regions influence output-gating dynamics between striatum and caudal regions of frontal cortex. So, in the schematic, maintenance of context by prepmd, influences the output gating loop between striatum and deep output layers of PMd, which in turn affects motor response gating by the motor loop. To adapt this model for hierarchical control and learning, we proposed that information maintained in rostral regions of PFC influences the striatal units that output-gate more caudal regions of PFC (Figure 19.8). To be concrete, consider our example second-order policy (see Figure 19.1) in which color determines which dimensions, shape or size, of the stimulus determine a response. The model would learn to input-gate color for maintenance by prepmd, and shape and orientation by PMd. Importantly, the maintenance of color by prepmd would influence the striatal output-gating units for the PMd. So, depending on the color maintained by prepmd, these units would gate the appropriate maintained stimulus feature (shape or orientation) into the output layer of PMd. And maintenance of this feature by PMd would influence selection of the motor response by the motor basal ganglia circuit. Thus, in this model, hierarchical control, and the observed rostrocaudal organization of the frontal cortex, emerges from a series of nested corticostriatal loops. This proposed architecture is consistent with evidence from monkey tracing studies and probabilistic connectivity in humans showing a rostrocaudal organization of inputs from premotor/prefrontal cortex to corresponding regions of striatum (Draganski et al., 2008; Inase, Tokuno, Nambu, Akazawa, & Takada, 1999; Lehericy, Ducros, Krainik, et al., 2004; Lehericy, Ducros, Van de Moortele, et al., 2004; Postuma & Dagher, 2006). It also follows from the general anatomical property that inputs to striatum are strongest from cortical areas of closest proximity (Kemp & Powell, 1970). A central feature of this nested architecture is that it permits rapid learning of hierarchical structure, as was Page 13 of 20

14 observed in Badre et al. (2010). By contrast, in other versions of the model, which do not include a hierarchical architecture, learning is not as efficient or successful. Moreover, it suggests a mechanistic account for the decline in prepmd activation observed by Badre et al. (2010). Specifically, in the model, the representation of contextual information in prepmd, which constrains attention to one of the dimensions in PMd, is not adaptive during flat blocks in which no hierarchical structure is present. As such, prepmd layer activity comes to be associated with negative value and generates a negative RPE, which in turn, allows the striatum to learn not to gate contextual representations into this layer. In order to test predictions of the neural net model with fmri, we developed a second probabilistic mixture of experts model that abstracts the key computations of the neural net circuitry. However, this model can be fitted to the behavioral data of each participant in Badre et al. (2010) and so can provide estimates of the latent states of the participant at each point during the task. In other words, for a given trial, are they attending to the hierarchical rule? And, which rule are they using to determine their response? Based on these estimates, we were able to analyze two key components of the model in our fmri data. First, we could assess which regions of the brain changed depending on the extent to which a participant attended to hierarchical rules vs. other rules. Second, because we would have an estimate of which rule and thus what outcome a participant anticipated for each, we could compute RPE. The results from this analysis confirmed key predictions of the model and provided novel evidence for the nested corticostriatal structure it proposes. Specifically, we found greater prepmd activation in participants who searched for hierarchical rules more during learning than in those who did not. RPE estimates were generally associated with activation in striatum and frontal cortex (Figure 19.9a). However, when RPE was weighted by attention to hierarchical rules, only prepmd and a focal subregion of the striatum at the same rostrocaudal extent were activated (Figure 19.9b d). Moreover, the hierarchical RPE signal in this striatal subregion (but not RPE in others) correlated with the decline in prepmd activity when no hierarchical structure existed (i.e., in the flat condition). This provided empirical support for the mechanistic explanation of this decline that emerged from the neural net model. Thus, our results suggest that a specific corticostriatal circuit is involved in learning second-order hierarchical structure. This is in line with the broader hypothesis that each level of the hierarchy is associated with a specific corticostriatal loop and that hierarchical control emerges from this nested structure. Put another way, the observed rostrocaudal functional organization of the frontal cortex may emerge as a consequence of the way that frontal cortex interacts with the striatum, as a series of nested gating loops. Conclusions Our cognitive control system is hierarchical in nature. We can update and maintain information relevant to decisions at multiple levels of abstraction. This capacity for abstraction in control is likely a source of our remarkable cognitive flexibility. Hierarchical control emerges from the functional organization of the frontal cortex. Progressively rostral regions of the frontal cortex appear to support cognitive control representations at increasing levels of abstraction. Moreover, the relationship of these subregions to each other is hierarchical, in the sense that neural activity in rostral regions appears to influence neural activity in caudal regions more than the reverse. Finally, at least in the context of hierarchical learning, it appears that this hierarchical architecture may emerge from a series of nested corticostriatal loops that permit rostral regions of PFC to differentially affect gating in caudal regions of the PFC. Given the ability of only a few such loops to support control over large policy structures, this simple nested gating architecture may be one key to the remarkable flexibility in our thought and action. Future Directions Following are questions to pursue further in future research: What is the relationship between the rostrocaudal organization of frontal cortex as observed in cognitive control of action and analogical and relational reasoning and intelligence? How does this organization develop, and how does its development affect learning and cognitive control in Page 14 of 20

15 children? How does hierarchical cognitive control in goal-directed behavior relate to habitual behavior, which can also be expressed hierarchically? Click to view larger Figure 19.9 Results from tests of model predictions with fmri (Badre & Frank, 2012). a. BOLD response to brain areas that track reward prediction error (RPE). Significant activations were observed across the entire striatum and prepmd. b. Functionally defined regions of interest (ROIs) for PMd, prepmd, and areas within caudate that are posterior to, at the same level as, and anterior to prepmd. c. Within cortical ROIs, prepmd tracked RPE specifically when the model-derived attentional weight to hierarchical rule (RPE_Hmod) was high, but not the flat rule (RPE_Fmod). PMd did not distinguish between hierarchical and flat rules in its sensitivity to RPE. d. Within caudate, areas at the same anterior-toposterior level as prepmd tracked RPE modulated by attention to hierarchical relative to flat rule. Caudate areas more posterior and more anterior to prepmd were not sensitive to this distinction. How does this frontal organization fit within a broader network for cognitive control in the brain beyond local interactions in frontal cortex? Some evidence exists for a similar rostrocaudal organization along ventrolateral prefrontal cortex. What distinguishes this pathway from the dorsal gradient that has been the focus of research to date? Acknowledgments This work was supported by the National Institutes of Health (NS065046). I also wish to acknowledge my collaborators, including M. J. Frank, A. Kayser, and M. D Esposito, on the work described in this chapter. References Asaad, W. F., Rainer, G., & Miller, E. K. (1998). Neural activity in the primate prefrontal cortex during associative learning. Neuron, 21 (6), Asaad, W. F., Rainer, G., & Miller, E. K. (2000). Task-specific neural activity in the primate prefrontal cortex. Journal of Neurophysiology, 84 (1), Badre, D. (2008). Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes. Trends in Cognitive Sciences, 12 (5), Page 15 of 20

16 Badre, D. (2010). Is prefrontal cortex necessary for the storage and acquisition of relational concepts? Cognitive Neuroscience, 1 (2), Badre, D., & D Esposito, M. (2007). Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex. Journal of Cognitive Neuroscience, 19 (12), Badre, D., & D Esposito, M. (2009). Is the rostro-caudal axis of the frontal lobe hierarchical? Nature Reviews Neuroscience, 10 (9), Badre, D., & Frank, M. J. (2012). Mechanisms of hierarchical reinforcement learning in corticostriatal circuits, Part II: Evidence from fmri. Cerebral Cortex, 22 (3), Badre, D., Hoffman, J., Cooney, J. W., & D Esposito, M. (2009). Hierarchical cognitive control deficits following damage to the human frontal lobe. Nature Neuroscience, 12 (4), Badre, D., Kayser, A. S., & D Esposito, M. (2010). Frontal cortex and the discovery of abstract action rules. Neuron, 66 (2), Badre, D., Poldrack, R. A., Pare-Blagoev, E. J., Insler, R. Z., & Wagner, A. D. (2005). Dissociable controlled retrieval and generalized selection mechanisms in ventrolateral prefrontal cortex. Neuron, 47 (6), Badre, D., & Wagner, A. D. (2004). Selection, integration, and conflict monitoring; Assessing the nature and generality of prefrontal cognitive control mechanisms. Neuron, 41 (3), Badre, D., & Wagner, A. D. (2007). Left ventrolateral prefrontal cortex and the cognitive control of memory. Neuropsychologia, 45 (13), Botvinick, M., & Plaut, D. C. (2004). Doing without schema hierarchies: A recurrent connectionist approach to normal and impaired routine sequential action. Psychological Review, 111 (2), Botvinick, M. M. (2007). Multilevel structure in behaviour and in the brain: A model of Fuster s hierarchy. Philosophical Transactions of the Royal Society of London Series B: Biological Science, 362 (1485), Botvinick, M. M. (2008). Hierarchical models of behavior and prefrontal function. Trends in Cognitive Sciences, 12 (5), Botvinick, M. M., Niv, Y., & Barto, A. C. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113 (3), Braver, T. S., & Bongiolatti, S. R. (2002). The role of frontopolar cortex in subgoal processing during working memory. Neuroimage, 15 (3), Braver, T. S., Reynolds, J. R., & Donaldson, D. I. (2003). Neural mechanisms of transient and sustained cognitive control during task switching. Neuron, 39 (4), Bunge, S. A. (2004). How we use rules to select actions: A review of evidence from cognitive neuroscience. Cognitive, Affective & Behavioral Neuroscience, 4 (4), Bunge, S. A., Kahn, I., Wallis, J. D., Miller, E. K., & Wagner, A. D. (2003). Neural circuits subserving the retrieval and maintenance of abstract rules. Journal of Neurophysiology, 90 (5), Bunge, S. A., Wendelken, C., Badre, D., & Wagner, A. D. (2005). Analogical reasoning and prefrontal cortex: Evidence for separable retrieval and integration mechanisms. Cerebral Cortex, 15 (3), Butterfield, E. C., Wambold, C., & Belmont, J. M. (1973). On the theory and practice of improving short-term memory. American Journal of Mental Deficiency, 77, Chase, W. G., & Simon, H. A. (1973). The mind s eye in chess. In W. G. Chase (Ed.), Visual information processing (pp ). New York: Academic Press. Christoff, K., & Gabrieli, J. D. E. (2000). The frontopolar cortex and human cognition: Evidence for a rostro-caudal Page 16 of 20

17 hierarchal organization within the human prefrontal cortex. Psychobiology, 28, Christoff, K., & Keramatian, K. (2007). Abstraction of mental representations: Theoretical considerations and neuroscientific evidence. In S. A. Bunge & J. D. Wallis (Eds.), Perspectives on rule-guided behavior (pp ). New York: Oxford University Press. Christoff, K., Keramatian, K., Gordon, A. M., Smith, R., & Madler, B. (2009). Prefrontal organization of cognitive control according to levels of abstraction. Brain Research, 1286, Christoff, K., Ream, J. M., Geddes, L. P. T., & Gabrieli, J. D. E. (2003). Evaluating self-generated information: Anterior prefrontal contributions to human cognition. Behavioral Neuroscience, 117 (6), Cohen, J. D., Dunbar, K., & McClelland, J. L. (1990). On the control of automatic processes: A parallel distributed processing account of the Stroop effect. Psychological Review, 97, Cohen, J. D., & Servan-Schreiber, D. (1992). Context, cortex, and dopamine: A connectionist approach to behavior and biology in schizophrenia. Psychological Review, 99, Cooper, R. P., & Shallice, T. (2006). Hierarchical schemas and goals in the control of sequential behavior. Psychological Review, 113, Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, Donaldson, D. I., Petersen, S. E., Ollinger, J. M., & Buckner, R. L. (2001). Dissociating state and item components of recognition memory using fmri. Neuroimage, 13 (1), Draganski, B., Kherif, F., Kloppel, S., Cook, P. A., Alexander, D. C., Parker, G. J., et al. (2008). Evidence for segregated and integrative connectivity patterns in the human basal ganglia. Journal of Neuroscience, 28 (28), Estes, W. K. (1982). Learning, memory, and intelligence. In R. J. Sternberg (Ed.), Handbook of intelligence (pp ). New York: Cambridge University Press. Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17 (1), Frank, M. J., & Badre, D. (2012). Mechanisms of hierarchical reinforcement learning in corticostriatal circuits, Part I: Theory. Cerebral Cortex, 22 (3), Frank, M. J., & O Reilly, R. C. (2006). A mechanistic account of striatal dopamine function in human cognition: Psychopharmacological studies with cabergoline and haloperidol. Behavioral Neuroscience, 120 (3), Fuster, J. M. (1997). The prefrontal cortex: Anatomy, physiology, and neuropsychology of the frontal lobe (3rd ed.). Philadelphia: Lippincott-Raven. Fuster, J. M. (2001). The prefrontal cortex an update: Time is of the essence. Neuron, 30 (2), Fuster, J. M. (2004). Upper processing stages of the perception-action cycle. Trends in Cognitive Sciences, 8 (4), Fuster, J. M., Bodner, M., & Kroger, J. (2000). Cross-modal and cross-temporal association in neurons of frontal cortex. Nature, 405, Gick, M. L., & Holyoak, K. J. (1980). Analogical problem solving. Cognitive Psychology, 12, Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15, Golde, M., von Cramon, D. Y., & Schubotz, R. I. (2010). Differential role of anterior prefrontal and premotor cortex in the processing of relational information. Neuroimage, 49 (3), Page 17 of 20

18 Halford, G. S. (1993). Children s understanding: The development of mental models. Hillsdale, NJ: Lawrence Erlbaum. Inase, M., Tokuno, H., Nambu, A., Akazawa, T., & Takada, M. (1999). Corticostriatal and corticosubthalamic input zones from the presupplementary motor area in the macaque monkey: Comparison with the input zones from the supplementary motor area. Brain Research, 833 (2), Kemp, J. M., & Powell, T. P. (1970). The cortico-striate projection in the monkey. Brain, 93 (3), Koechlin, E., Basso, G., Pietrini, P., Panzer, S., & Grafman, J. (1999). The role of the anterior prefrontal cortex in human cognition. Nature, 399 (6732), Koechlin, E., Corrado, G., Pietrini, P., & Grafman, J. (2000). Dissociating the role of the medial and lateral anterior prefrontal cortex in human planning. Proceedings of the National Academy of Sciences U S A, 97 (13), Koechlin, E., & Jubault, T. (2006). Broca s area and the hierarchical organization of human behavior. Neuron, 50 (6), Koechlin, E., Ody, C., & Kouneiher, F. (2003). The architecture of cognitive control in the human prefrontal cortex. Science, 302 (5648), Koechlin, E., & Summerfield, C. (2007). An information theoretical approach to prefrontal executive function. Trends in Cognitive Sciences, 11 (6), Kostopoulos, P., & Petrides, M. (2003). The mid-ventrolateral prefrontal cortex: Insights into its role in memory retrieval. European Journal of Neuroscience, 17 (7), Kouneiher, F., Charron, S., & Koechlin, E. (2009). Motivation and cognitive control in the human prefrontal cortex. Nature Neuroscience, 12 (7), Krawczyk, D. C., McClelland, M. M., Donovan, C. M., Tillman, G. D., & Maguire, M. J. (2010). An fmri investigation of cognitive stages in reasoning by analogy. Brain Research, 1342, Krawczyk, D. C., Michelle McClelland, M., & Donovan, C. M. (2011). A hierarchy for relational reasoning in the prefrontal cortex. Cortex, 47 (5), Lashley, K. S. (1951). The problem of serial order in behavior. In L. A. Jeffress (Ed.), Cerebral mechanisms in behavior (pp ). New York: Wiley. Lehericy, S., Ducros, M., Krainik, A., Francois, C., Van de Moortele, P. F., Ugurbil, K., et al. (2004). 3-D diffusion tensor axonal tracking shows distinct SMA and pre-sma projections to the human striatum. Cerebral Cortex, 14 (12), Lehericy, S., Ducros, M., Van de Moortele, P. F., Francois, C., Thivard, L., Poupon, C., et al. (2004). Diffusion tensor fiber tracking shows distinct corticostriatal circuits in humans. Annals of Neurology, 55 (4), Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24, Miller, G. A., Galanter, E., & Pribram, K. H. (1960). Plans and the structure of behavior. New York: Holt, Rinehart and Winston. Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press. O Reilly, R. C. (2010). The what and how of prefrontal cortical organization. Trends in Neuroscience, 33 (8), O Reilly, R. C., & Frank, M. J. (2006). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18 (2), Paine, R. W., & Tani, J. (2005). How hierarchical control self-organizes in artificial adaptive systems. Adaptive Page 18 of 20

19 Behavior, 13, Passingham, R. E., & Rowe, J. B. (2002). Dorsal prefrontal cortex: Maintenance in memory or attentional selection? In D. T. Stuss & R. T. Knight (Eds.), Principles of frontal lobe function (pp ). Oxford: Oxford University Press. Petrides, M. (2005). Lateral prefrontal cortex: Architectonic and functional organization. Philosophical Transactions of the Royal Society of London, Series B: Biological Science, 360 (1456), Postuma, R. B., & Dagher, A. (2006). Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cerebral Cortex, 16 (10), Ramnani, N., & Owen, A. M. (2004). Anterior prefrontal cortex: Insights into function from anatomy and neuroimaging. Nature Reviews Neuroscience, 5 (3), Robin, N., & Holyoak, K. J. (1995). Relational complexity and the functions of the prefrontal cortex. In M. S. Gazzaniga (Ed.), The cognitive neurosciences (pp ). Cambridge, MA: MIT Press. Rosenbaum, D. A. (1987). Hierarchical organization of motor programs. In S. Wise (Ed.), Neural and behavioral approaches to higher brain functions (pp ). New York: Wiley. Schank, R. C., & Abelson, R. (1977). Scripts, plans, goals, and understanding. Hove, UK: Lawrence Erlbaum Associates. Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science, 321 (5890), Solomon, M., Ozonoff, S. J., Cummings, N., & Carter, C. S. (2008). Cognitive control in autism spectrum disorders. International Journal of Developmental Neuroscience, 26 (2), Speed, A. (2010). Abstract relational categories, graded persistence, and prefrontal cortical representation. Cognitive Neuroscience, 1 (2), Stuss, D. T., & Benson, D. F. (1987). The frontal lobes and control of cognition and memory. In E. Perecman (Ed.), The frontal lobes revisited (pp ). New York: IRBN Press. Vincent, J. L., Kahn, I., Snyder, A. Z., Raichle, M. E., & Buckner, R. L. (2008). Evidence for a frontoparietal control system revealed by intrinsic functional connectivity. Journal of Neurophysiology, 100 (6), Visscher, K. M., Miezin, F. M., Kelly, J. E., Buckner, R. L., Donaldson, D. I., McAvoy, M. P., et al. (2003). Mixed blocked/event-related designs separate transient and sustained activity in fmri. Neuroimage, 19 (4), Volle, E., Gilbert, S. J., Benoit, R. G., & Burgess, P. W. (2010). Specialization of the rostral prefrontal cortex for distinct analogy processes. Cerebral Cortex, 20 (11), Wagner, A. D., Paré-Blagoev, E. J., Clark, J., & Poldrack, R. A. (2001). Recovering meaning: Left prefrontal cortex guides controlled semantic retrieval. Neuron, 31, Wallis, J. D., Anderson, K. C., & Miller, E. K. (2001). Single neurons in prefrontal cortex encode abstract rules. Nature, 411, Wallis, J. D., & Miller, E. K. (2003). From rule to response: Neuronal processes in the premotor and prefrontal cortex. Journal of Neurophysiology, 90 (3), White, I. M., & Wise, S. P. (1999). Rule-dependent neuronal activity in the prefrontal cortex. Experimental Brain Research, 126 (3), David Badre David Badre, Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI Page 19 of 20

20 Page 20 of 20