Kathryn Cain Dickerson ALL RIGHTS RESERVED

Transcription

1 2011 Kathryn Cain Dickerson ALL RIGHTS RESERVED

2 Multiple Memory Systems Involved in Human Probabilistic Learning By: Kathryn Cain Dickerson A Dissertation submitted to the Graduate School Newark Rutgers, The State University of New Jersey in partial fulfillment of the requirements for the degree of Doctor of Philosophy Graduate Program in Behavioral and Neural Science Written under the direction of Professor Mauricio R. Delgado and approved by: Dr. Lila Davachi Dr. Mauricio R. Delgado Dr. Bart Krekelberg Dr. Catherine Myers Dr. Elizabeth Tricomi Newark, New Jersey October 2011

3 ABSTRACT OF THE DISSERTATION Multiple Memory Systems Involved in Human Probabilistic Learning By: Kathryn Cain Dickerson Dissertation Director: Professor Mauricio R. Delgado This dissertation investigated the nature of interactions between multiple memory systems (MMS) in the human brain during probabilistic learning. How MMS interact in the brain is highly debated in the literature, with evidence supporting competition, cooperation, and parallel engagement observed across several species and various experimental paradigms (e.g., Poldrack et al., 2001; Poldrack and Rodriguez, 2004; Voermans et al., 2004; White and McDonald, 2002). In this dissertation, three functional neuroimaging experiments investigated the relationship between the medial temporal lobes (MTL), implicated in declarative learning, and the basal ganglia (BG), involved in nondeclarative learning, during probabilistic learning. Specifically, tasks which varied with respect to learning type (feedback, akin to nondeclarative learning and observation akin to declarative learning) and cue difficulty (e.g., easy, hard) were employed. Based on the evolutionary theory of MMS, neuroanatomical connections between these regions, as well as connectivity with midbrain dopaminergic centers involved in reward, memory, and reinforcement learning, it ii

4 was hypothesized that rather than one specific relationship (e.g., competitive or cooperative), MMS operate in parallel during probabilistic learning at times exhibiting parallel engagement, and at times interacting directly via cooperation or competition depending on the context of the learning situation. Interactions between memory systems were measured by the relative engagement of each system during learning (as measured by BOLD responses), simple correlation analyses, as well as functional and effective connectivity measures. Lastly, to examine putative dopaminergic influences during learning, reinforcement learning models were employed. Across all experiments, it was observed that a) patterns of activation in MMS during learning varied depending on the learning context; b) functional and effective connectivity existed across regions (MTL, BG); and c) a dopaminergic learning signal correlated with activity in both the BG and MTL. To conclude, the results of this dissertation support the hypothesis that multiple memory systems operate in parallel during probabilistic learning in humans. Understanding how such systems communicate during learning may inform treatment of several neuropsychiatric disorders which affect learning and memory in humans (e.g., Parkinson s disease, schizophrenia, and MTL amnesia). iii

5 Dedication To my mother, Christine Josephine Dickerson, for her constant support and love To my father, Kenneth Gerard Dickerson, for his quiet and steady guidance I would not be here without the two of you I am eternally grateful for all you have given me & To my grandparents, Dorothea Marie and Raymond James Cain thank you for teaching me the power of knowledge, hard work, and determination You are deeply loved and missed iv

6 Acknowledgments There are many people who I would like to thank for their support and assistance on my journey towards achieving my graduate degree. First I would like to thank my advisor, Dr. Mauricio Delgado. He has been an excellent mentor and I am so happy to have had the opportunity and pleasure of working with him. He has taught me a great deal and I am very appreciative of his kindness and patience over the years. I would also like to thank all the members of my dissertation committee, both past and present, Dr. Elizabeth Tricomi, Dr. Bart Krekelberg, Dr. Mark Gluck, Dr. Catherine Myers, and Dr. Lila Davachi. They have all been extremely helpful and thoughtful in guiding me through my dissertation experience. I am also very grateful for the advice and mentoring I received at the University of Rochester, especially from Dr. Mark Mapstone, which helped me decide to attend graduate school. My time at Rutgers would not have been the same without my dear friends and colleagues in the Delgado lab. I am especially thankful to Dominic Fareri, Laura Martin, Mike Niznikiewicz, Tony Porcelli, and Swati Battacharya-Sharma with whom I began my journey as well as Kamila Sip, Meredith Johnson, Andrea Lewis, Lauren Leotti and Anastasia Rigney, with whom I am ending it. It has been such a joy and a pleasure knowing you all, and I value your friendships enormously. Mike, Dominic, Tony and Laura especially taught me so much and were wonderfully patient teachers. It was also my pleasure to work with Victoria Lee, Martina Henderson, and Stephanie Herrera, who were undergraduate students in the lab. Vicki especially has been such a wonderful support, helping me with many aspects of my projects over the years. v

7 I am very grateful to the Neuroscience Department, especially Dr. Ian Creese and Dr. Steve Levison, who were the directors of the program for the majority of my time at Rutgers, as well as the all of the faculty who taught me a great deal about neuroscience. I am so happy I was accepted here and have very much enjoyed being a part of the department as well as being affiliated with the Psychology department. I am very grateful to all of my fellow neuroscience students, especially Amber Ziegler, Kate Seip, Arielle Schmidt, Anushree Karnik, Jessica Wright, Alex Kohl, Josh Callahan, Till Hartmann, Bengi and Tim Unal, Julia Basso, and Liad and Alfonso Renart, for listening and providing excellent comments on practice talks and for their general support and encouragement. I would also like to thank the staff of the Neuroscience Department, Ann Kutyla, Wayne Brown, and Connie Sadaka as well as Sandra Smith of the Psychology Department for their assistance throughout the years. The staff at the UMDNJ Advanced Imaging Center: Pat Singh, Bobby Singh, Evelyn Oliviera, and Melissa Ortiz were most helpful during my time of data collection and I am very appreciative of their assistance. I am also grateful to all of the students who volunteered their time participating in my studies. I am so lucky to have the support of my close friends Molly Reed, Lidza Kalifa, Megan Sweeny, Avinash Reddy, Usman Khan, and my entire extended family, who have supported me every step of the way. I am also so lucky to have the best parents and siblings Mom, Dad, Nate, Lilly, and Kate I truly could not have done this without you, your love has given me so much encouragement when I needed it most. Lastly I vi

8 would like to thank my partner Alon Amir for his love and support. I am so happy that we had this journey together; it would not have been the same without you. vii

9 Table of Contents Abstract of the Dissertation Dedication Acknowledgments Table of Contents List of Figures List of Tables List of Appendices ii iv v viii xii xiv xvi Chapter 1: Introduction General Introduction The Neuroanatomy of Multiple Memory Systems Medial temporal lobes and declarative memory Basal ganglia and nondeclarative memory Anatomical circuitry of the MTL and BG Neuroanatomical connections between memory systems Interactions between Multiple Memory Systems Functional Significance Evidence supporting competitive interactions Evidence supporting cooperative interactions Evidence supporting parallel engagement Description of Dissertation Experiments Significance 29 viii

10 Chapter 2: General Materials, Methods, & Data Analyses General Materials & Methods Experimental designs Participants fmri acquisition General Data Analyses General linear model Granger causality analyses Prediction error analyses Cluster lever statistical threshold estimator Sequential Bonferroni correction Note regarding test phase neuroimaging results 42 Chapter 3: Experiment Introduction: Background & Rationale Materials & Methods Experimental design Data analysis Results Behavioral results Neuroimaging results Discussion 62 ix

11 Chapter 4: Experiment Introduction: Background & Rationale Materials & Methods Experimental design Data analysis Results Behavioral results Neuroimaging results Exploratory analyses Discussion 116 Chapter 5: Experiment Introduction: Background & Rationale Materials & Methods Experimental design Data analysis Results Behavioral results Neuroimaging results Discussion 166 x

12 Chapter 6: General Discussion Purpose & Summary of Experiments Limitations Conclusions & Future Directions 187 References 189 Figures 208 Tables 223 Appendices 236 Vita 255 xi

13 List of Figures Figure 3.1 Schematic of the Task used in Experiment Figure 3.2 Experiment 1 Behavioral Results 209 Figure 3.3 Experiment 1 Neuroimaging Results: Main Effect of Cue Difficulty 210 in the Striatum and Hippocampus Figure 3.4 Experiment 1 Neuroimaging Results: Granger Causality Analysis 211 Functional Connectivity in the Caudate Nucleus using the Hippocampus as the Seed Region Figure 3.5 Experiment 1 Neuroimaging Results: Prediction Error Analysis in 212 the Hippocampus and Striatum Figure 4.1 Schematic of the Task used in Experiment Figure 4.2 Experiment 2 Behavioral Results 214 Figure 4.3 Experiment 2 Neuroimaging Results: Learning Phase Main Effect 215 of Cue Difficulty in the Striatum and MTL Feedback Group Figure 4.4 Experiment 2 Neuroimaging Results: Learning Phase Main Effect 216 of Cue Difficulty in the Striatum and MTL Observation Group Figure 4.5 Experiment 2 Neuroimaging Results: Update Phase Main Effect 217 of Cue Difficulty in the Striatum and MTL Feedback and Observation Groups Figure 4.6 Experiment 2 Neuroimaging Results: Granger Causality Analysis 218 Effective Connectivity using the Midbrain as the Seed Region Figure 4.7 Experiment 2 Neuroimaging Results: Prediction Error Analysis 219 Figure 5.1 Schematic of the Task used in Experiment xii

14 Figure 5.2 Experiment 3 Behavioral Results 221 Figure 5.3 Experiment 3 Neuroimaging Results: ROI Analysis during the 222 Stimulus Presentation Period xiii

15 List of Tables Table 1.1 Evidence Supporting Interactions between Multiple 25 Memory Systems Table 3.1 Experiment 1 Learning Phase ANOVA: Brain Regions Showing 223 a Main Effect of Cue Difficulty Table 3.2 Experiment 1 Learning Phase ANOVA: Brain Regions Showing 224 a Main Effect of Learning Type Table 3.3 Experiment 1 Test Phase Contrast of Previously Studied 225 vs. Novel Table 4.1 Experiment 2 Learning Phase ANOVA: Brain Regions Showing 226 a Main Effect of Cue Difficulty Feedback Group Table 4.2 Experiment 2 Learning Phase ANOVA: Brain Regions Showing 227 a Main Effect of Cue Difficulty Observation Group Table 4.3 Experiment 2 Test Phase 1: Contrast of Previously Studied vs. 228 Novel Feedback Group Table 4.4 Experiment 2 Test Phase 1: Contrast of Previously Studied vs. 229 Novel Observation Group Table 4.5 Experiment 2 Update Phase ANOVA: Brain Regions Showing 230 a Main Effect of Cue Difficulty Feedback & Observation Groups Combined Table 4.6 Experiment 2 Update Phase ANOVA: Brain Regions Showing 231 a Main Effect of Learning Type Feedback & Observation Groups Combined xiv

16 Table 4.7 Experiment 2 Update Phase ANOVA: Brain Regions Showing 232 a Main Effect of Group Table 4.8 Experiment 2 Test Phase 2: Contrast of Previously Studied vs. 233 Novel Feedback Group Table 4.9 Experiment 2 Test Phase 2: Contrast of Previously Studied vs. 234 Novel Observation Group Table 5.1 Experiment 3 Test Phase: Contrast of Previously Studied 235 vs. Novel xv

17 List of Appendices Appendix 1 Neuroimaging Test Phase Results for Experiments Appendix 2 Experiment 3 Correlations Observed across the 238 Independent Region of Interest Analysis Appendix 3 Behavioral Questionnaires Administered in Experiments xvi

18 1 Chapter One: Introduction 1.1 General Introduction Accumulating evidence from both animal and human-based research supports the existence of multiple memory systems that rely on distinct neural substrates in the human brain (Sherry and Schacter, 1987; Squire, 1992b; Squire and Zola, 1996). An important division is between a declarative memory system, which relies on the integrity of the medial temporal lobes (MTL), and a nondeclarative memory system, which engages many brain regions including the basal ganglia system (BG; Sherry and Schacter, 1987; Squire, 1992b). How these two regions interact during learning, however, remains unclear and is a topic of debate in the literature. While some evidence points towards competitive interactions, such that when one system is engaged it suppresses or blocks the other (Poldrack et al., 2001; Poldrack and Packard, 2003; Foerde et al., 2006; Lee et al., 2008), other research supports cooperative interactions (Dagher et al., 2001; Voermans et al., 2004; Cincotta and Seger, 2007; Sadeh et al., 2011). Moreover, a third theory posits that the MTL and BG operate in parallel and may interact in either a competitive or a cooperative manner depending on the learning scenario (White and McDonald, 2002; Albouy et al., 2008; Atallah et al., 2008). Considering the conflicting evidence regarding the nature of the interactions between these two major learning and memory systems, understanding how they interact remains a question of great interest. The goal of the experiments presented in this dissertation was to investigate the interactions between multiple memory systems, specifically the basal ganglia and the medial

19 2 temporal lobe, during probabilistic learning using functional magnetic resonance imaging (fmri) of healthy human participants. One way of approaching this debate is to consider the evolutionary theory supporting multiple memory systems as well as neuroanatomical and functional connectivity between the MTL and BG. It has been suggested that multiple learning and memory systems evolved to play distinct roles in the brain (Sherry and Schacter, 1987). Sherry and Schacter propose that natural selection may have driven the development of multiple memory systems in the brain, when functional incompatibility, or the inability of one memory system to solve a problem in a particular learning environment, lead to the development of multiple systems with distinct functions. Furthermore, it has been theorized that these memory systems may operate in parallel, encoding unique information (White and McDonald, 2002). Anatomical considerations state that these structures communicate directly via a unidirectional anatomical projection from the hippocampus, part of the MTL, to the nucleus accumbens (Kelley and Domesick, 1982) as well as the medial, ventral, rostral, and caudal aspects of the caudate nucleus/putamen, all subcomponents of the BG (Krayniak et al., 1981; Sorensen and Witter, 1983; Swanson and Kohler, 1986; Groenewegen et al., 1987). The MTL and BG also communicate indirectly via interconnectivity with dopaminergic centers in the midbrain (Swanson, 1982; Bolam et al., 2000; Poldrack and Rodriguez, 2004). Dopamine is a neurotransmitter that has been implicated in playing a fundamental role in learning and memory processes (Wise, 2004). Dopaminergic neural activation is observed in the initial acquisition of reward

20 3 contingencies (Schultz, 1997, 2002) and is associated with changes in blood oxygen level dependent (BOLD) signals in the human striatum during processes such as receipt of rewards and punishments, monitoring goal outcomes, and prediction error encoding (for review see O'Doherty et al., 2004; Balleine et al., 2007; Delgado, 2007). Moreover, dopamine neuronal activity is linked with longterm memory formation (Frey et al., 1990; Frey et al., 1991; Li et al., 2003), known to recruit the hippocampus (for review see Bliss and Collingridge, 1993) and recently midbrain reward-related activation has been linked to the enhancement of long-term memory in the hippocampus (Wittmann et al., 2005). Furthermore, functional coupling of the midbrain and hippocampus enhances memory formation (Adcock et al., 2006; Shohamy et al., 2008). Based on evidence in the literature, it is hypothesized that this functional neuroconnectivity with dopamine plays an integral role in facilitating interactions between multiple memory systems. Therefore, drawing from the evolutionary theory of multiple memory systems and functional neuroconnectivity with dopamine, it is theorized that the MTL and BG operate in parallel during probabilistic learning with the ability to complement each other, which may manifest itself as both cooperation and competition. For the purpose of this dissertation, the following definitions of parallel, cooperative, competitive interactions are employed, modified for interpreting neuroimaging data based on White and McDonald s theory (2002) as well as interpretations of current neuroimaging studies in the field (Poldrack et al., 2001; Voermans et al., 2004; Foerde et al., 2006; Doeller et al., 2008; Sadeh

21 4 et al., 2011). Parallel engagement is defined as the engagement of multiple memory regions simultaneously during learning, while displaying unique patterns of BOLD activation, suggesting that they may be encoding distinct information in parallel. Cooperative interactions are defined as the exhibition of similar patterns of BOLD signal in both the MTL and BG during learning as well as positive correlations across distinct areas. Lastly, competitive interactions are defined as the exhibition of opposing patterns of BOLD signal in both the MTL and BG during learning and negative correlations across distinct regions. 1.2 The Neuroanatomy of Multiple Memory Systems Over the past two decades the predominant theory regarding human memory suggests that memory is not a unitary process, but rather consists of multiple systems that rely on distinct neural substrates (Sherry and Schacter, 1987; Squire, 1992b; Squire and Zola, 1996). One system is referred to as declarative memory, which encompasses episodic (memory for distinct episodes) and semantic memory (facts and events), both of which consist of flexible associations that may be rapidly acquired (Squire et al., 2004). Recalling one s college graduation day (episode) and knowing that George Washington was the first President of the United States (fact) are examples of declarative memory. A second system is referred to as nondeclarative memory, and is a broader collection of many types of memory, which are more inflexible in nature and take longer to acquire (Reber et al., 1996; Squire et al., 2004). Habit and procedural memory, emotional memory, as well as priming and perceptual memory are all

22 5 types of nondeclarative memory (Squire et al., 2004). Remembering how to ride a bicycle is an example of a nondeclarative memory Medial temporal lobes and declarative memory: Declarative and nondeclarative memory are thought to differ not only in the types of memory they encompass and their respective properties, but also in the neural substrates they engage. Numerous studies, spanning rodent, primate, and human-based research, document that declarative memory relies on the integrity of the medial temporal lobes (see Squire, 1992a for review). A significant portion of rodentbased research focuses on spatial maze learning tasks (Packard et al., 1989; Packard and White, 1989; Packard and White, 1991; Packard and McGaugh, 1992, 1994). In these maze learning tasks, a rodent is trained to navigate a maze to retrieve a food reward or escape water. One example is the radial maze task, which has two versions termed the win-shift and the win-stay radial maze (Packard et al., 1989). The win-shift radial maze task requires animals to visit each arm of the maze only once to retrieve a food reward. It is theorized to represent declarative learning because the animal must store rapidly acquired information based on a single experience, e.g., it must remember which arms of the maze it has and has not visited in order to obtain all of the food rewards while not making any errors (re-entering an arm where the food has already been obtained; White and McDonald, 2002). Lesions to the fimbria-fornix region, the fiber bundle which arises from the subiculum and projects to the hypothalamus, impair performance on the win-shift version of the radial maze task (Packard et al., 1989) and a water maze paradigm (Packard and McGaugh, 1992). Because

23 6 lesions to the fimbria-fornix impair performance on the win-shift radial maze task, it is thought that the MTL plays an important role in episodic-like learning in rodents. Additionally, lesions to the CA1 and CA3 subfields of the hippocampus produce spatial and associative memory deficits in rodents (Stubley-Weatherly et al., 1996). In primate-based research, evidence suggesting that the MTLs are important for declarative learning comes from a host of discrimination related tasks (such as pattern discrimination, delayed-nonmatching-to-sample, delayed retention of object discrimination, and concurrent discrimination). These studies have shown that lesions to the medial temporal lobes impair discrimination tasks (Zola-Morgan and Squire, 1984, 1985; Zola-Morgan and Squire, 1986), but leave motor and cognitive skill learning intact (Zola-Morgan and Squire, 1984). Furthermore, studies using human MTL amnesic patients also support the theory that the medial temporal lobes are necessary for declarative memory, as patients who are missing part or all of their MTLs suffer profound declarative memory deficits (Scoville and Milner, 1957; Penfield and Milner, 1958; Corkin et al., 1984; Rempel-Clower et al., 1996). Lastly, fmri and positron emission tomography (PET) studies corroborate the theory that the MTLs are involved in encoding and recalling declarative memories (for review see Schacter and Wagner, 1999; Eldridge et al., 2000; Davachi and Wagner, 2002; Zeineh et al., 2003; Law et al., 2005). Specifically, in learning paradigms the hippocampus is believed to play a role in encoding episodes (see Gluck et al., 2003 for review), stimulus-stimulus

24 7 representations (Gluck and Myers, 1993; Bunsey and Eichenbaum, 1995; Eichenbaum and Bunsey, 1995), as well as flexibly adapting information to use in novel contexts (Myers et al., 2003; Shohamy and Wagner, 2008). In category learning, it has been suggested by Gluck and Myers (Gluck and Myers, 1993; Gluck et al., 2003) that the hippocampus functions to compress overlapping stimulus information and differentiate between distinct stimulus information. These compressed and differentiated representations in the hippocampus are formed over time through exposure to many trials during training, and are subsequently communicated to cortical regions. It has also been suggested that the connections of the MTL with the cortex are designed for rapid learning of individual events (O'Reilly and Munakata, 2000). This type of learning is particularly important for learning arbitrary category associations, for example in unstructured categorization tasks which do not have an exemplar, do not involve information integration, do not use prototype distortion, nor require learning of a specific rule (Seger and Miller, 2010). Such unstructured categorization tasks have been previously shown to engage the MTL (Poldrack et al., 1999; Poldrack et al., 2001; Seger and Cincotta, 2005; Foerde et al., 2006). In non-human primates, neurons in the hippocampus and temporal cortex have been shown to respond to category-specific information (Hampson et al., 2004), suggesting that category-specific information may be encoded within this region. In accord with this theory, the hippocampus is thought to be involved in encoding episodes, which may occur in as little as one learning instance. It is believed that the hippocampus encodes a conjunction of many features which comprise the

25 8 episode, rather than encoding the individual features distinctly (Gluck et al., 2003) Basal ganglia and nondeclarative memory: In contrast, nondeclarative memory is thought to recruit a host of distinct brain regions including the basal ganglia (procedural or habit memory), the amygdala (emotional memory), and the neocortex (priming and perceptual memory) (Squire et al., 2004). For the purposes of this dissertation, the focus was on the role of the basal ganglia in human probabilistic learning. Research using both animal models and human studies demonstrates that the basal ganglia are involved in several aspects of nondeclarative learning and memory. In animal models, rodent maze studies reveal that lesions of the caudate nucleus, part of the striatum, impair performance on a win-stay version of the radial maze task. This version of the task requires animals to approach only certain cued arms of the maze to retrieve a food reward, which is thought to be akin to stimulus-response or habit learning (Packard et al., 1989; Packard and McGaugh, 1992). Accumulating evidence suggests that the basal ganglia are also involved in many aspects of cognitive learning (see Middleton and Strick, 2000; and Packard and Knowlton, 2002 for review). In human based studies, recent fmri research has implicated this area in several other learning related functions including: habit learning (see Yin and Knowlton, 2006 for review), skill learning (Atallah et al., 2007), instrumental conditioning (O'Doherty et al., 2004), as well as reward learning and decision-making (Tricomi et al., 2004; Balleine et al., 2007; Delgado, 2007). It has become well-documented in the literature that the basal

26 9 ganglia are not solely a structure important for motor learning, but that they are also critical for many types of nondeclarative memory and cognitive functioning. The basal ganglia system has also been implicated in playing a fundamental role in category learning in humans (Knowlton et al., 1994; Knowlton et al., 1996; Poldrack et al., 1999; Poldrack et al., 2001; Shohamy et al., 2004b; Shohamy et al., 2004a; Seger and Cincotta, 2005; Cincotta and Seger, 2007). It has been suggested that the basal ganglia may be involved in encoding goal outcomes (Tricomi and Fiez, 2008; Carter et al., 2009), error correction learning (O'Doherty et al., 2004; O'Doherty, 2004; Abler et al., 2006; Pessiglione et al., 2006), stimulus-response associations (Rolls, 1994; Packard and Knowlton, 2002; Ashby and Spiering, 2004), as well as mediating action selection (Seger, 2008). Work by Seger has suggested the involvement of multiple corticostriatal loops, the visual (caudate nucleus body/tail), motor (putamen), executive (head of caudate nucleus), and motivational (ventral striatum) loop, which play distinct roles in category learning (2008). The author has proposed that the visual corticostriatal loop may be involved in action selection, while the motor loop may be involved in action selection as well as automatizing sequences. The executive and motivational corticostriatal loops are engaged in feedback processing, with the executive loop encompassing the head of the caudate nucleus which may be involved in updating behavior via error correction. Consistent with the actor-critic model, it has been theorized that the dorsal striatum (head of caudate nucleus) acts as the actor and may be involved in deciding which action to take, while the ventral striatum acts as the critic tracking

27 10 whether or not an excepted reward has been received (Joel et al., 2002; O'Doherty et al., 2004) Anatomical circuitry of the MTL and BG: It is important to consider the anatomical circuitry of the MTL and BG individually before considering how they may interact. Both the MTL and BG are large structures, consisting of multiple subregions. The MTL is comprised of the parahippocampal cortex (PHC), perirhinal cortex (PRC), entorhinal cortex (EC), and the hippocampal complex (HC) (Squire et al., 2004). The parahippocampal and perirhinal cortices receive information from unimodal and polymodal association areas and send this information to the EC. The entorhinal cortex has multiple sublayers and serves as the input unit to the hippocampus. The hippocampus itself also consists of multiple layers, including the dentate gyrus, and the Cornu Ammonis (CA) fields, referred to as CA1, CA2, CA3 and CA4 (Andersen, 1971; Amaral, 1993). The predominant loop arises as input from layer two of the EC, which sends projections to the mossy fibers of the dentate gyrus and layers CA2 and CA3, referred to as the perforant path (there is also a projection from the EC layer 3 to CA1 and the subiculum). The mossy fibers of the dentate gyrus then project to the pyramidal cells of the CA3 field, which project via the Schaffer collaterals to the pyramidal cells in CA1 in a unilateral circuit. CA1 then projects to the subiculum and deep layers of the EC. This flow of information from the perforant path dentate gyrus CA3 CA1 is referred to as the trisynaptic circuit. The CA1 field is a sparse network, which functions well in pattern separation, while CA3 has many interconnections, and functions well in pattern

28 11 completion. The output unit of the hippocampal unit is the subiculum and fimbriafornix fiber bundle. Turning to basal ganglia circuitry, the basal ganglia system is comprised of the striatum, the globus pallidus, the subthalamic nucleus, and the substantia nigra (Packard and Knowlton, 2002; Yin and Knowlton, 2006). The striatum consists of the caudate nucleus, putamen, and nucleus accumbens. The caudate nucleus and putamen are often referred to collectively as the dorsal striatum, and are implicated in learning tasks; while the nucleus accumbens is referred to as the ventral striatum, and is more heavily involved in reward and conditioning tasks (O'Doherty et al., 2004; Balleine et al., 2007; Delgado, 2007). The BG also contains the globus pallidus, which is divided into an internal and external component, as well as the subthalamic nucleus. In addition, the substantia nigra (SN), a midbrain structure, is involved in basal ganglia circuitry. The SN is divided into two main components, the SN reticulata (SNr) and the SN compacta (SNc), the latter of which contains dopamine neurons that project to the striatum. As a note, dopaminergic neurons in the ventral tegmental area (VTA) also project to the striatum (Haber, 2003). The BG is generally categorized into containing two main circuits, referred to as the direct pathway and the indirect pathway. The direct pathway is as follows: Cortex (glutamatergic) striatum (GABAergic) SNr-GPi (GABAergic) thalamus (glutamatergic) cortex, forming a functional loop. Strong excitatory input from the cortex results in strong inhibitory input from the striatum to the SNr-GPi complex, which then weakly inhibits the thalamus, resulting in a stronger

29 12 excitatory projection to the cortex. The end result of this loop therefore is excitation of the cortex. The indirect circuit is as such: Cortex (glutamatergic) striatum (GABAergic) GPe (GABAergic) subthalamic nucleus (glutamatergic) SNr-GPi (GABAergic) thalamus (glutamatergic) cortex. Excitatory input from the cortex results in strong inhibition of the GPe from the striatum. The GPe then weakly inhibits the STN, which then strongly excites the SNr-GPi complex, inhibiting the thalamus, which can then only weakly excite the cortex. The end result of this loop therefore is weak excitation of the cortex. Consequently, the direct and indirect pathways have antagonistic functions in the brain Neuroanatomical connections between memory systems: In order to better understand how two major learning and memory systems operate in the human brain, much recent research has focused on examining the interactions between the medial temporal lobes and the basal ganglia. Anatomically, these regions are connected by a unilateral projection from the hippocampus to the nucleus accumbens (Kelley and Domesick, 1982) and the medial, ventral, rostral, and caudal aspects of the caudate nucleus/putamen (Krayniak et al., 1981; Sorensen and Witter, 1983; Swanson and Kohler, 1986; Groenewegen et al., 1987). Lisman and Grace (2005) formulated a theory of how the striatum and hippocampus may interact via a loop with the ventral tegmental area. This model is as follows:

30 13 Layer II of the EC dentate gyrus and CA3 (glutamatergic) CA1 (which also receives input from EC layer III; glutamatergic) subiculum (glutamatergic) nucleus accumbens (GABAergic) ventral pallidum (GABAergic) VTA (dopaminergic) CA1, layer III of the EC, and the nucleus accumbens. Lisman and Grace propose that this model is involved in novelty detection and encoding of information into long term memory. Newly detected information in the hippocampus is sent as a novelty signal via the subiculum, nucleus accumbens and ventral pallidum to the VTA in the midbrain. Subsequent DA release from the VTA projects back up to the hippocampus, where it is involved in facilitating long-term potentiation. Of critical importance for the purposes of this dissertation, is the fact that the VTA also projects to the nucleus accumbens, thereby effectively communicating with both the hippocampus and the ventral striatum. In addition to the Lisman and Grace model, Devan and White (1999) have presented four anatomical links between the hippocampal complex and the dorsomedial striatum: 1) CA1 and subiculum via the fimbria/fornix ventral striatum and caudate nucleus/putamen (Groenewegen et al., 1987; Brog et al., 1993) 2) Perirhinal cortex dorsal CA1, EC, and NAcc/medial caudate nucleus/putamen (Krayniak et al., 1981; Sorensen and Witter, 1983; Swanson and Kohler, 1986; McGeorge and Faull, 1989; Burwell et al., 1995; Liu and Bilkey, 1996)

31 14 3) Hippocampal output posterior cingulate cortex medial caudate nucleus/putamen (Domesick, 1969; McGeorge and Faull, 1989) 4) Hippocampal output medial prefrontal cortex ventral and dorsomedial striatum (Swanson, 1981; Gerfen, 1984; Swanson and Kohler, 1986; Jay et al., 1989) Therefore it is possible that the hippocampus and striatum may also communicate via a third region, such as the prefrontal cortex (Poldrack and Rodriguez, 2004) or even the cingulate cortex (Devan and White, 1999). Evidence from animal literature indicates that lesions to the cingulate cortex (Sutherland et al., 1988) and medial prefrontal cortex (Sutherland et al., 1982; Kolb et al., 1994) impair performance on water maze tasks in rodents, suggesting that the connections between these regions is functionally important. Recent research in particular has emphasized that connections with the prefrontal cortex may be facilitating interactions between the MTL and the BG. In one study using dynamic causal modeling, the dorsolateral prefrontal cortex (DLPFC) was found to be the sole structure involved in receiving reward-related information directly, which subsequently influenced activity in the midbrain (VTA) and ventral striatum (NAcc) (Ballard et al., 2011). The authors suggest that the DLPFC integrates information about reward and then sends this information to the ventral striatum and midbrain dopaminergic regions, resulting in the initiation of motivated behavior. As the midbrain projects to both the MTL and ventral striatum, it is possible that these areas may interact via the PFC or the midbrain. The work of

32 15 Ballard and colleagues is consistent with a previous model posited by Poldrack and Rodriguez (2004) which implicated the prefrontal cortex in mediating interactions between the BG and MTL. In this theory, Poldrack and Rodriguez posit that the MTL and BG most likely do not interact in a direct manner, but may interact via the prefrontal cortex or via the action of neuromodulators. Both the BG and MTL form anatomical loops with frontal cortical areas (Middleton and Strick, 2000; Thierry et al., 2000; Seger, 2006). In terms of potential neuromodulators, the authors nominate a couple of candidates, one of which is dopamine. In addition to playing a role in reward learning and memory functioning, dopamine also has an important influence on successful classification learning (Beninger et al., 2003). Beninger and colleagues conducted an experiment where schizophrenic patients were administered either typical or atypical antipsychotic medications and asked to perform a probabilistic classification learning task. Those on typical antipsychotic medication, which is believed to block D2 receptors in the dorsal striatum, were impaired at classification, whereas those on atypical medication, which does not block dopamine receptors in the dorsal striatum, and healthy control participants, performed normally. This study highlights the role dopamine plays in normal acquisition of category learning information. In accord with this general theory, a distinct experiment examining the neural correlates of instructed knowledge observed that regions of the ventromedial prefrontal cortex (vmpfc), ventral striatum, and MTL were involved in reward-related learning and were functionally negatively correlated with a

33 16 region of the DLPFC during the instructed learning session when positive outcomes were received (Li et al., 2011). The authors concluded that DLPFC may be modulating action-outcome responses in reward-related structures dependent on the presence and usefulness of explicit knowledge (instruction) compared with only action-outcome information (no instruction). Lastly, computational models exploring interactions among distinct neural substrates mediating learning have suggested that the PFC may work together with the hippocampus during rule-governance. In particular, Doll and colleagues (2009) posit two hypotheses regarding how the MTL and BG interaction. The first hypothesis suggests that the PFC and hippocampus (HC) may bias what the striatum learns (bias model), while a second hypothesis posits that the BG learns contingencies independently from the PFC/HC, but later the PFC/HC overrides these learned contingencies (override model). Both of these models have been shown to be valid (Doll et al., 2009); therefore further research is required to explore how these systems interact. Thus, converging evidence suggests that one manner by which the MTL and BG may interact is via the PFC. 1.3 Interactions between Multiple Memory Systems Functional Significance Summarizing the recent work of many researchers suggests three main conflicting hypotheses regarding the nature of interactions between the basal ganglia and the medial temporal lobe: 1) memory systems compete during learning, 2) memory systems cooperate during learning, or 3) memory systems

34 17 operate in parallel during learning. Evidence in favor of each of these hypotheses is described below: Evidence supporting competitive interactions: Initial evidence suggested that these memory systems interact in a competitive manner, such that when one region is engaged, it suppresses or decreases activity in the other region via either direct or indirect connections (Poldrack et al., 2001; Packard and Knowlton, 2002; Foerde et al., 2006; Lee et al., 2008). Such studies have ranged across species, from rodent-based lesion studies to human-based fmri experiments involving probabilistic category learning. Evidence of competitive interactions in animal-based work has been observed during lesion experiments when knocking out or pharmacologically modulating one region, for example the caudate nucleus of the striatum results in an enhancement in performance on a MTL dependent task and vice versa (Mitchell and Hall, 1988; Packard et al., 1989; McDonald and White, 1993; Packard, 1999; Lee et al., 2008). Most recently, Lee and colleagues showed a double dissociation between hippocampal and striatal-based learning paradigms in mice. Disrupting striatal functioning, via either lesions or inhibiting a transcription factor impaired cuedlearning (dependent on the striatum) and enhanced spatial-learning (hippocampus-dependent). Likewise, lesions to the hippocampus impaired spatial-learning while enhancing cued-learning. Based on these findings, the authors concluded that the memory systems interact in a competitive manner. In human fmri studies, competition has been shown using probabilistic category learning tasks (Poldrack et al., 2001; Foerde et al., 2006; Seger and

35 18 Cincotta, 2006). For example, Poldrack and colleagues (2001) employed a between-subjects, blocked-fmri design and a popular category learning task, the weather prediction task. In the weather prediction task, participants are required to learn to predict the outcome of an event (rainy or sunny weather) based on the presentation of 1-4 cue cards (modified from the original medical symptom classification task of Gluck and Bower, 1988). Each card is associated with a certain probability of predicting either rain or sun (e.g., 20, 40, 60 or 80% rain). This version of the task contained two different variations. The first was a feedback task, designed to be analogous to nondeclarative learning, as the probabilistic nature of the cards and trial and error approach to learning would presumably make it difficult for participants to outwardly memorize the associations (Knowlton et al., 1996). The second version was an observational task, similar to declarative learning, as the cue card and outcome were presented simultaneously, presumably making it easier for participants to directly memorize the associations. One group of participants completed the feedback task and a separate group completed the observational task. The authors found the feedback version produced activation in the basal ganglia and deactivation in the MTL. The MTL exhibited greater BOLD responses to the observational compared with the feedback version of the task. Furthermore, when the authors conducted a functional connectivity analysis, they found a negative correlation between the MTL and a region in the right caudate nucleus across participants. Additionally, in a separate event-related fmri experiment using only the feedback version of the task, the authors reported that the MTL was initially activated and

36 19 the caudate nucleus deactivated during the task. Over time, the MTL became deactivated and the caudate nucleus became activated, suggesting a temporal element to the engagement of the regions. Based on this evidence, the authors concluded that the memory systems interact in a competitive manner during learning. A recent study replicated Poldrack and colleagues 2001 result by reporting negative correlations between the hippocampus and specifically the head of the caudate nucleus during a rule-learning task (Seger and Cincotta, 2006). The authors concluded that the antagonistic interaction between these two regions is probably not direct, but rather is mediated by frontal cortical areas Evidence supporting cooperative interactions: A second hypothesis regarding how these regions interact during learning suggests they cooperate, such that they work together to facilitate learning (Dagher et al., 2001; Voermans et al., 2004; Cincotta and Seger, 2007; Sadeh et al., 2011). Human-based fmri evidence supporting this theory was observed in a relatively recent experiment which examined specifically how responses in the striatum are modified when learning via feedback versus learning via observation (Cincotta and Seger, 2007). In this block-design study, the authors used an information integration task and reported that BOLD responses in the caudate nucleus and the hippocampus were analogous in nature. Contrary to Poldrack and colleagues 2001 results where MTL and BG activation negatively correlated and the regions were active at alternating time points, this observation suggests that the two regions are at times active simultaneously and show similar patterns of activity

37 20 during learning. Similar patterns of activity within distinct memory regions may be interpreted as cooperative engagement of these areas during learning. Additionally, a recent study demonstrated cooperation between the hippocampus and putamen during a declarative memory encoding task (Sadeh et al., 2011). Activation within the hippocampus and putamen was associated with successful memory encoding and critically, activity within these two regions was correlated in participants for later remembered, but not forgotten words. Lastly, the strength of the correlation between the putamen and hippocampus predicted successful memory, such that the stronger the correlation across regions, the more words participants subsequently remembered. This observation is further supported by fmri studies which posit that the MTL and BG may cooperate in order to maintain normal functioning in compromised brain states (Dagher et al., 2001; Voermans et al., 2004). Dagher and colleagues (2001) investigated planning in Parkinson s disease patients and healthy control patients using the Tower of London task in a PET study. Results indicated that Parkinson s patients performed as well as healthy control participants, but engaged a different network of brain regions during the task. Healthy control participants primarily engaged areas of the PFC and the caudate nucleus, while PD patients engaged similar prefrontal areas, but no caudate nucleus activation was observed. Critically, activation within the hippocampus was observed in PD patients, but was diminished in healthy control participants. These results suggest that within PD patients, where striatal functioning is compromised, hippocampal activation increased in order to maintain overall

38 21 successful planning in an executive function task. A similar result was observed in a different disease population using a separate learning paradigm. In a study by Voermans and colleagues (2004), Huntington s disease (HD) patients were required to perform a route navigation task. Results indicated that increasing MTL BOLD responses helped compensate for decreased striatal functioning in order to maintain successful route recognition in HD patients. In healthy control participants, no such increase in MTL activity was observed. In fact, healthy control participants engaged greater caudate nucleus activation, while HD patients elicited greater hippocampus activation during route recognition. Furthermore, healthy control participants exhibited greater functional connectivity between the hippocampus and caudate nucleus than HD patients. The authors suggest that in HD patients, this connectivity may change such that the MTL may adapt to have a more independent and compensatory role due to compromised striatal function. Such cooperation between learning systems suggests they may work together to facilitate a normally functioning brain state. Converging evidence across multiple disease states (PD, HD) employing distinct experimental tasks (Tower of London, route navigation) provides strong evidence that the medial temporal lobe and striatum do cooperate during some learning scenarios, specifically when such cooperation maintains overall successful learning and performance levels Evidence supporting parallel engagement: Lastly, a third hypothesis describing how the MTL and BG interact posits that the memory systems operate in parallel, complementing each other s function in the brain, and therefore have

39 22 the ability to either cooperate or compete during learning (McDonald and White, 1994; White and McDonald, 2002; Albouy et al., 2008; Doeller et al., 2008; Tricomi and Fiez, 2008). White and McDonald (2002) outline their theory of three parallel memory systems in the rodent brain: the hippocampus, the dorsal striatum, and the amygdala. They suggest that in a normal brain, these regions process and store information simultaneously and in parallel. The authors also state that the memory systems at times cooperate and at times compete during learning depending on the specific learning situation. Evidence supporting this theory in animal studies is seen in radial maze experiments which demonstrate that similar behaviors, such as certain types of place learning, may be acquired via either the dorsal striatum, particularly the medial dorsal section, or the hippocampus (McDonald and White, 1995; Devan and White, 1999). Furthermore, recent fmri studies involving various, distinct paradigms support parallel engagement of the medial temporal lobe and striatum. Doeller and colleagues (2008) employed an environmental learning paradigm, adapted from traditional animal maze tasks, to investigate interactions between the hippocampus and striatum. Participants were required to learn the location of several objects in a virtual environment, using a distinct environmental landmark or a boundary for orientation purposes. Results indicated that the posterior hippocampus was engaged in learning and remembering locations via environmental boundaries, whereas the caudate nucleus was engaged when learning and remembering locations via single, intramaze landmarks. These results support the recruitment of the dorsal striatum during trial and error,

40 23 associative learning via distinct landmarks, and the engagement of the hippocampus during boundary-based, incidental learning. Furthermore, the authors employed a dynamic causal modeling analysis to investigate interactions between these regions during learning. The results from this connectivity analysis supported independent and parallel engagement of the hippocampus and striatum, rather than direct interactions between the regions of interest. The hippocampus and striatum therefore supported distinct types of learning and were engaged independently and in parallel during learning of the environment in this paradigm. A second, unrelated study examined engagement of the striatum and hippocampus during a declarative word pair learning task (Tricomi and Fiez, 2008). Participants were required to learn arbitrary word pairs via feedback, with the same pairs of words being presented over three rounds. Importantly, feedback was administered in each round, but on the first round, as participants were randomly guessing the answer, feedback simply indicated a correct guess and not a correct memory, as occurred in rounds 2 and 3. The authors reported that both the caudate nucleus and the hippocampus were engaged during learning, but exhibited distinct patterns of activity. The caudate nucleus displayed an interaction of round by time, showing greater activation in the last 2 rounds, during the feedback presentation period. Furthermore, the caudate nucleus differentiated between correct and incorrect feedback in the later rounds, when feedback was meaningful. The hippocampus was also engaged, exhibiting a main effect of time, initially decreasing after trial onset and subsequently

41 24 increasing after feedback presentation. These results are inconsistent with purely competitive interactions between the BG and MTL as the two regions were simultaneously active and did not negatively correlate. Furthermore the results support parallel engagement of the MTL and BG as these areas were engaged during the learning process, playing distinct roles during the acquisition of declarative information. The authors concluded that the caudate nucleus may be representing goal achievement during declarative learning. Lastly, parallel engagement between the BG and MTL has been observed in a fmri motor sequence learning and memory task (Albouy et al., 2008). Both the hippocampus and regions of the striatum, specifically the caudate nucleus and putamen, were involved in initial motor sequence learning. Interestingly, hippocampal activation during training predicted behavioral performance improvement observed the next day. Additionally, a functional connectivity analysis indicated competitive interactions between the hippocampus and the putamen during learning. However, this interaction became cooperative 24 hours following sequence training in participants who were fast learners. This result elegantly illustrates the parallel nature of the interactions between the MTL and BG, showing that the two regions may both compete and cooperate during learning and memory processes. To summarize therefore, there are three conflicting hypotheses regarding the nature of interactions between multiple memory systems. Evidence supporting each theory is summarized in the table below:

42 25 Table 1.1 Evidence Supporting Interactions between Multiple Memory Systems Competitive Interactions Cooperative Interactions Parallel Engagement Mitchell and Hall, 1988 Dagher et al., 2001 McDonald and White, 1994 Packard et al., 1989 Voermans et al., 2004 Devan and White, 1999 Packard 1999 Cincotta and Seger, 2007 Atallah et al., 2008 McDonald and White, 1993 Sadeh et al., 2011 Albouy et al., 2008 Poldrack et al., 2001 Tricomi and Fiez, 2008 Foerde et al., 2006 Doeller et al., 2008 Seger and Cincotta, 2006 Lee et al., 2008 For review see Poldrack and Packard Description of Dissertation Experiments Thus it is clear there is an ongoing debate regarding the nature of interactions between the medial temporal lobes and basal ganglia. The purpose of this dissertation therefore was to investigate how these two memory systems interact during learning. Based on the evolutionary theory of multiple memory systems as well as anatomical and functional connectivity with dopaminergic midbrain centers (SN/VTA), it was hypothesized that these distinct neural regions will act in parallel during probabilistic learning, exhibiting signs of parallel engagement, as well as competitive and cooperative interactions. Furthermore, based on the compilation of evidence documenting dopamine s involvement in learning and memory, it was hypothesized that dopamine may play a role in facilitating interactions between the MTL and BG during reward-related learning. The goal of the first experiment was to investigate the relative engagement and interactions between the MTL and the BG during probabilistic learning using a within-subjects, event-related fmri design and a novel, simple probabilistic learning task. The purpose of this experiment was to explore in a

43 26 basic manner how each system, the BG and MTL, is engaged individually and how these distinct systems interact during multiple types of probabilistic learning (feedback and observation). It was hypothesized that the MTL and BG will operate in parallel during learning, exhibiting parallel engagement as well as potentially both cooperative and competitive interactions, which were measured by relative activation within each region as well as functional and effective connectivity between these regions during learning. It was further hypothesized that such parallel engagement may be mediated by midbrain dopaminergic connectivity, as indirectly assessed by the application of a reinforcement learning model postulated to reflect dopaminergic activity during trial and error learning (Schultz, 1997; O'Doherty et al., 2002). The results of this experiment provided a fundamental understanding of the involvement of the MTL and BG during different types of probabilistic learning. The goal of the second experiment was to examine interactions between the MTL and BG during re-learning or updating of probabilistic information caused by a reversal in learning contingencies. The purpose of this experiment was to examine how the MTL and BG interact during learning, using a novel manner of addressing this question, via the employment of multiple types of learning and a reversal in probabilistic cue contingencies. It was hypothesized that immediately following reversal (cue contingencies switch) both the MTL and BG will act in parallel to process new contingencies. Specifically, it was predicted that initial inconsistencies between the old and new information will diminish as the old contingencies are updated, regardless of whether updating

44 27 occurs within the same learning type (e.g., feedback-feedback or observationobservation) or across learning types (e.g., feedback-observation or observationfeedback). It was theorized that both competitive and cooperate interactions may be observed during the updating process in accord with the parallel theory of engagement among multiple memory systems. These interactions were measured by the relative engagement of each system during initial learning and reversal as well as functional and effective connectivity across the brain during reversal. Lastly, in order to examine putative dopaminergic involvement in mediating these interactions, a reinforcement learning model was applied. The results of this experiment provided evidence of how multiple memory systems interact when learning types evolve and learning contingencies reverse. The goal of the third experiment was to probe a specific learning condition in which the MTL and BG may exhibit cooperative interactions. The purpose of this experiment was to test a direct hypothesis that was developed by observing the engagement of the MTL and BG in experiments 1 and 2. Specifically, a declarative memory recognition task was employed in order to engage the medial temporal lobe (akin to a behavioral knockout of one region of interest, the MTL). Concurrent with this interfering memory task, participants performed a probabilistic feedback learning task, which contained both an easy/consistent as well as a hard/inconsistent learning condition. Additionally, control conditions were employed which did not contain the interfering declarative memory task, and thereby did not aim to engage the MTL. The effect of the interfering memory task was observed on behavior as well as brain regions engaged during

45 28 probabilistic feedback learning. It was hypothesized that multiple memory systems will cooperate when information is inconsistent and therefore challenging, specifically when both systems are available to be involved in feedback probabilistic learning (control conditions), while memory systems will operate more independently when information is more consistent and less challenging, in support of a parallel processing model of learning. The relative engagement of distinct memory regions was examined across learning sessions (control sessions and declarative memory interference session) and interactions across regions were examined by the application of simple correlation analyses. The results of this experiment provided evidence of the engagement and interactions between multiple memory systems in normal (single-task) versus dual-tasking learning scenarios. The experiments presented in this document were event-related and mostly within-subjects fmri designs which employed a novel probabilistic learning task adapted from Delgado et al., Event-related fmri designs have certain advantages over block designs previously used (e.g., Poldrack et al., 2001; Cincotta and Seger, 2007) in that they allow the experimenter to track the BOLD signal changes to specific events. Additionally, when it is feasible, within-subject designs are also preferable as they eliminate possible sources of variance (across-subject comparisons). However, sometimes between-subjects comparisons are necessary (experiment 2 for example) based on the theoretical question and experimental design. While much important work has been conducted using the popular weather prediction task (WPT) a novel probabilistic

46 29 categorization task was chosen for these experiments for the following two reasons. First, the tasks designed in this thesis are simpler than the weather prediction task, in that they contained only 1 learnable cue per stimulus, as opposed to multiple cue cards. Second, the task is based on a card guessing game, which has been previously shown to robustly engage the striatum in a trial and error learning variant one of the key regions of interest for these experiments (Delgado et al., 2005). Other than these two distinctions, the tasks employed in this thesis and the WPT are similar and therefore results from the experiments presented here may be somewhat comparable to studies that employ the weather prediction task. 1.5 Significance The results from the experiments outlined in this dissertation aimed to provide a richer understanding of how the MTL and BG interact during human probabilistic learning. By better understanding how these systems are engaged during learning and memory, and how they interact, a more complete picture of how healthy humans solve probabilistic learning tasks was obtained. Understanding interactions among human memory systems is critical, as these neural areas are implicated in many neurodegenerative and neuropsychiatric diseases. Therefore, these results may eventually aide research efforts directed towards understanding and improving treatment for Parkinson s disease, Huntington s disease, Alzheimer s disease, MTL amnesia, and schizophrenia. Additionally results from these experiments may shed some light on individual

47 30 differences in learning, such as individuals who learn better via observation via those who learn better via trial and error experiences. Chapter Two: General Materials, Methods, & Data Analyses 2.1 General Materials & Methods This section provides a brief overview of the common elements in experiments 1-3. Details containing information specific to individual experiments are presented in the materials and methods section of the appropriate chapter Experimental designs: Each task was programmed using E-Prime Version 2.0 (PST, Pittsburgh). Participants inside the MRI machine saw the task via use of a back projection system using a mirror affixed to the head coil. A MRI compatible button box was used to record behavioral responses during the experiments. For all experiments, participants were given detailed instructions and completed a short practice version outside of the MRI prior to beginning the actual task (different stimuli were used during practice sessions). In experiment 2, which was a between-subjects design, additional instructions were administered while the participants remained inside the MRI machine after the first test session was completed in order to train them on the type of learning they did not experience during the learning phase (e.g. trained on feedback learning if the participant was in the observation learning group). Monetary compensation consisted of a minimum of $25 per hour upon arrival at the imaging center for completion of the study, with additional incentives based on participants

48 31 performance during the task for experiments 1 and 2. Specifically, participants were awarded additional money based on a paper test administered after completion of the MRI portion of the experiment (see Appendix 3 for the questionnaires administered in experiments 1-3). In experiment 1, total compensation ranged between $50 and $65; in experiment 2, total compensation ranged between $50 and $56. In experiment 3, participants were told they would be paid additional money based on how well they learned the cues and how well they discriminated between old and new natural scenes. However, all participants who completed the study were paid $60 and were told they performed well in the task, irrespective of how well they actually performed in the experiment. Participants were paid the same amount in experiment 3 solely to simplify the payment process Participants: All participants were right handed adults over the age of 18 years. Every participant was screened for head injury and a history of psychiatric or neurological impairments as well as contraindications to MRI (metal implants, claustrophobia). All females were administered a pregnancy test, which was verified to be negative, prior to beginning the MRI session. Informed consent was obtained from each participant before beginning the experiment. All studies were approved by the Institutional Review Boards of Rutgers University and the University of Medicine and Dentistry of New Jersey (UMDNJ). Experiment 1 contained seventeen participants (nine female, mean (M) age 24 years, standard deviation (SD) 4.1); experiment 2 had fifty-two participants (26 female, M age 22 years, SD 4.7); and experiment 3 had twenty-

49 32 eight participants (15 female, M age 22 years, SD 5.0). Final analysis in experiment 1 included 16 participants (9 female), as one person was excluded due to a scanner malfunction in the middle of the MRI session. In experiment 2, final analysis consisted of forty-one participants (20 female), as 6 people missed an excessive amount of trials (greater than 2 SDs from the M; some due to falling asleep), 3 people were excluded due to excessive motion (movement beyond 5 millimeters or systematic spikes), 1 person was excluded due to a scanner malfunction, and 1 person was excluded due to failure to learn the task (chance accuracy, 50%, in the learning phase). Experiment 2 was a between-subjects design consisting of a feedback group and an observation group: final data analysis included 21 participants in the feedback group and 20 in the observation group. The complete data set for experiment 3 consisted of twenty-four participants (12 female), as two participants were excluded due to malfunctions with the MRI overheating, one person did not return for the second day (fmri session), and one person was excluded due to failure to attend during the day 1 encoding session fmri acquisition: All MRI experiments were conducted at the UMDNJ Advanced Imaging Center. The MRI machine at this facility is a 3-Tesla Siemens Allegra, which was used to collect the structural (T1-weighted MPRAGE: 256 x 256 matrix; FOV = 256 mm; mm sagittal slices) and functional images (single-shot echo EPI sequence; TR = 2000 ms, TE = 25ms; FOV = 192 cm; flip angle = 80 ; matrix = 64 x 64; slice thickness = 3 mm). Forty contiguous oblique-axial slices (3 x 3 x 3 mm voxels) were acquired parallel to

50 33 the anterior commissure posterior commissure line. Functional data preprocessing and analysis was completed using Brain Voyager QX software (Versions 1.10; 2.0, and 2.2 Brain Innovation, Maastricht, The Netherlands). Preprocessing consisted of three dimensional motion correction (six-parameters), spatial smoothing (4 mm, FWHM), voxel-wise linear detrending, high-pass filtering of frequencies (3 cycles per time course), and normalizing the data to Talairach sterotaxic space (Talairach and Tournoux, 1988). In addition, the mean intensity of the BOLD signal was examined in experiments 2 and 3, but was not included in the preprocessing of the data. A canonical two gamma hemodynamic response function was used in all experiments to convolve the events of interest. 2.2 General Data Analyses This section provides descriptions and explanations of the common analyses conducted in experiments 1-3. Study specific information for each experiment is provided in the data analysis section of the appropriate chapter. Following fmri preprocessing, functional data analysis was completed using a whole brain analysis technique for all experiments, as well as region of interest (ROI) analyses for experiment General linear model: In all phases of each experiment, randomeffects general linear models (GLM) were performed. Predictors of interest (which varied based on the experiment and are described in the relevant chapters) were included in the model as well six motion parameters which were

51 34 included as regressors of no interest in order to account for potential motion related issues. In experiments 1 and 2, data analysis was conducted across the 4 second time period where the stimulus, response, and feedback occurred. The analysis was performed in this time period in order to most appropriately compare the feedback and observation trials. Relevant information for the feedback trials occurred in the last 2 seconds, where feedback was provided, whereas relevant information for the observation trials was present in the first 2 seconds, where participants observed the cue and association (informative arrow indicating the cue value). Therefore, to most accurately compare these distinct trial types, the stimulus/response and feedback periods were combined for the analyses performed in experiments 1 and 2. In experiment 3, analyses were conducted over several time points. The first analysis examined BOLD activation during the stimulus presentation period, where participants viewed the natural scenes/probabilistic cues and made a choice regarding the probabilistic cue value. The second analysis examined the BOLD signal during the feedback period, where participants received feedback contingent on their response in the cue period, which indicated the outcome of their probabilistic cue value choice (correct, incorrect, missed trial). The third and final analysis was a region of interest analysis examining BOLD signals during the stimulus presentation period in 7 a priori ROIs: the bilateral midbrain (centered on the ventral tegmental area), bilateral ventral striatum, bilateral hippocampus, and the left caudate nucleus. Details about the ROIs used are provided in Chapter 5.

52 Granger causality analyses: In addition to the basic analyses performed, a granger causality analysis (GCA) was conducted in order to examine functional and effective connectivity in the brain during various phases of the experiments (details provided in the methods sections of experiments 1 and 2). This analysis is especially informative as the experiments conducted in this dissertation were designed to examine interactions between the MTL and BG, which are difficult to probe using traditional neuroimaging analyses alone. Granger causality is advantageous as it examines functional and effective connectivity among multiple brain regions during a designated time period. GCA operates by looking for relationships between time course data from a seed region (x) with the rest of the brain (y). Using vector autoregression, Geweke (1982) proposed a measure of linear dependence, F x,y between two hypothetical times series of data, x[n] and y[n]. F x,y is the sum of three components: F x,y = F x y + F y x + F x y F x y measures the directed influence from x to y; probing if past values of x improve the current predicted value of y. F y x measures the directed influence from y to x; probing if past values of y improve the current predicted value of x. F x y measures the undirected instantaneous influence of x and y. This measure adds in the current value of x or y to the model already containing the past values of x and y. Therefore, GCA examines both undirected instantaneous influence (F x y ) and directed (F x y and/or F y x ) influences between a designated seed region (x) and

53 36 the rest of the brain (see Goebel et al., 2003; Roebroeck et al., 2005 for more details). For the analyses performed in the experiments in this dissertation, connectivity was examined across the entire run (first to last volume/tr) and an autoregressive model order of 1 TR (2 seconds) was used, consistent with the BrainVoyager plugin and with previous studies (Goebel et al., 2003; Roebroeck et al., 2005). Therefore Granger causality between distinct brain regions was conducted by examining 2 seconds into past time course data. Of special interest in this set of experiments was examining connectivity between the MTL and BG, with potential involvement of the midbrain Prediction error analyses: Reinforcement learning models were applied to learning trials in experiments 1 and 2. The purpose of applying the reinforcement learning models was to gain a better understanding of not only what regions were engaged during learning, but also how mechanistically participants were solving the tasks. Specifically, it was hypothesized that dopaminergic influences may play a role in facilitating interactions between the MTL and BG. In addition to examining BOLD activation in the dopaminergic midbrain, which is an anatomically small region whose exact location is difficult to determine using standard neuroimaging techniques, it was theorized that the application of a reinforcement learning model would aid in examining putative dopaminergic influences. Dopamine neurons are known to be involved in feedback-based learning and prediction error encoding and project to both the striatum and hippocampus (Scatton et al., 1980; Lynd-Balta and Haber, 1994; Schultz et al., 1997; Pagnoni et al., 2002; McClure et al., 2003; Abler et al.,

54 ). Therefore, one way of examining putative dopaminergic influences during learning is to apply a prediction error learning model to the behavioral accuracy data, with the aim of examining which brain areas track prediction error signals during learning. If the BG and/or MTL vary parametrically with a PE signal, it is possible that these regions are receiving similar information via midbrain dopaminergic neurons. Regressors for the prediction error models used in experiments 1 and 2 were calculated based on a Q-learning model (Watkins, 1989). The Q learning model was chosen to model behavioral choices as the literature supports this type of learning as an effective means of predicting instrumental choice in probabilistic reward learning tasks (Samejima et al., 2005; Pessiglione et al., 2006; Burke et al., 2010). A Q learning model operates by estimating the expected value of choosing action A and action B based on the individual sequence of choices and outcomes obtained by participants. Q models use prediction error signals to update action values according to the Rescorla- Wagner learning rule. In the context of experiments 1 and 2, this choice is making a button press to indicate whether a probabilistic cue has either a higher or lower than 5 value (e.g., choosing a high value). The expected value of making a cue-action choice is termed a Q value and represents the expected reward obtained by taking a certain action, in this case receiving feedback when making an individual choice. In order to model behavior, the following equations were used:

55 38 1. Q j, i+1 Q j, i + λδ 2. δ = r Q j, i The model contains the following parameters: 1) initial Q values (Q j ), 2) learning rate, λ, and 3) the softmax function temperature, m (akin to slope). A maximum likelihood estimation algorithm was used to estimate optimal values for these three parameters. The cue-action values (Qs) were then updated according the Rescorla-Wagner learning rule. Specifically, following each trial, the value of choosing higher than 5 and lower than 5 was updated according to Equation 1: Q j, i+1 Q j, i + λδ. Equation 2, which calculates the prediction error, δ = r Q j, i is simply the difference between the expected (Q j,i ) and the actual outcome (r) of a trial. The probability of choosing each action was estimated by using the softmax rule: for example if choosing high, the softmax function is: P(Q, i ) = exp(q i /m)/(exp(q i /m)+exp(q i /m). The variable m is a temperature parameter which measures variability, such that for low temperatures, the probability of choosing the action which has the highest expected reward approaches 1 (steep slope, exploitative behavior), where as for high temperatures, all actions have almost the same probability (moderate slope, explorative behavior). In the model, prediction error is represented by δ; r equals the reward amount, which in the feedback trials was 1 for correct trials or 0 for incorrect trials; i = trial number and j represents the type of action: participants choice of high ( ) or low ( ). A single set of free parameters was used for all

56 39 participants when generating the prediction error regressors which were later used in the neuroimaging analyses in experiments 1 and 2. For experiment 2, the prediction error model described above was modified for the observation trials, based on the model presented by Burke and colleagues (2010). Specifically, an action prediction error signal was generated for the observation trials. In order to do so, it was conceived that in the observation learning condition, participants could simply view the cue/value associations passively (although they were instructed not to), and therefore theoretically think of each cue s value as being assigned or chosen by the computer. Therefore, for the participant, the observation trials could be thought of as similar to observing another agent, in this case the computer, play to learn the cue/value associations, while the participant observed and subsequently learned the cue/value associations as well. An action prediction error is defined as δ action = action(h,l) - p(h, L) where action (H, L; H: higher than five cue value; L: lower than five cue value). The probability values, p(h, L) of observing each cue-action pair were calculated by the Q( ) and Q( ) values used in the softmax function. For example, for trial i, if the model predicted that the probability of choosing H (higher than 5 cue value) is 0.7 and the participant observed that choosing H is the correct choice, according the model: δ action = action i (H,L) - p(h, L), δ action = = 0.3. Hence a low surprise or saliency signal is generated. If the participant observed that in fact L (lower than 5 cue value) was the correct choice, then δ action = = 0.7, therefore a greater surprise/salience signal. Due to the way δ action is constructed, an action

57 40 prediction error value is always positive (as action always sums to 1), and can be used as an indicator for how salient the computer s choice was to the participant for that particular trial. As a note, a prediction error analysis was not conducted in experiment 3. The predominant reason for this was that the third experiment was much more complex in design than experiments 1 and 2 and as a result would be difficult to model. In experiment 3, during the stimulus presentation period, there was also a natural scene present in all learning sessions and presumably participants were trying to 1) ignore this information, 2) make a perceptual decision, or 3) remember the scene, based on the session type. Therefore, this was not a pure feedback learning environment as there were multiple cognitive demands present. Even though prediction errors were most likely being generated in response to the cue outcomes, since this was a much more complex learning environment, it was decided not to conduct a prediction error analysis in this experiment Cluster level statistical threshold estimator: The cluster level statistical threshold estimator is a plugin included in the BrainVoyager analysis package which provides a correction for multiple comparisons (Forman et al., 1995; Goebel et al., 2006). This plugin operates by employing Monte Carlo simulations in order to determine the likelihood of observing clusters of various sizes in a given map at a set threshold. In order to use the plugin, the statistical parametric map of interest is first thresholded at the desired level (e.g., p<0.005, uncorrected). Proceeding the initial thresholding, the plugin estimates the spatial

58 41 smoothness of the map and performs a whole brain correction using Monte Carlo simulations which estimate the rate of false positives in the present map at the cluster level. In all analyses reported, 1,000 iterations (the recommended number of the plugin) were chosen. After the simulations, the selected map automatically applies the minimum cluster size threshold which produces the desired cluster-level false-positive alpha rate (5% was chosen in all analyses). All remaining active clusters in the corrected map are used in creating a table that summarizes the number of clusters above the chosen threshold (5%) for each cluster size. When the analysis is complete, each cluster size is assigned an alpha value determined by the frequency of its occurrence in the map. This analysis therefore corrects for multiple cluster tests in the map Sequential Bonferroni correction: In all experiments, post-hoc analyses conducted on the behavior and neuroimaging data consisting of more than two t-tests within a family of comparisons were corrected for multiple comparisons with the sequential Bonferroni technique (Holm, 1979; Rice, 1989). This analysis was developed in 1979 by Holm as a less strict correction for multiple comparisons, compared to the traditional Bonferroni correction. In this analysis, the desired alpha level (e.g., 0.05) is divided by the number of t-tests to be performed, k (e.g., 3). The table-wide alpha level then becomes that value (e.g., 0.05/3 = ). The p values of all three t-tests are then ranked in order from most to least significant (P 1 to P k, or 0.004, 0.02, 0.04 for example). If the most significant p value (0.004) is less than alpha/k (e.g., ), then it can be considered significant at the table-wide alpha level and the experimenter may

59 42 proceed in examining the next test. If the most significant p value is greater than alpha/k, then all resulting p values are termed insignificant at the table-wide level. In the given example, is less than , so the experimenter may examine the next test. The alpha level is then divided by k-1, in this example (3-1=2). The new alpha level is now 0.05/2 or and the next most significant p value is 0.02, so it is also considered significant at the table-wide level. The experimenter may proceed on to the last test, using an alpha level of 0.05/(3-2, or 1). Therefore in this example, all tests are significant at the table-wide level Note regarding test phase neuroimaging results: All of the test phase neuroimaging results are presented in Appendix 1. The test phases were not primary phases of interest in experiments 1-3 but rather were performed as secondary analyses in order to ensure that participants successfully acquired cue contingencies in all learning sessions. These phases were also performed to examine the engagement of memory regions when recalling previously learned cue values in an exploratory fashion. Because the experimental sessions of interest were long (learning and update phases), the test sessions were short; especially the tests administered in experiments 1 and 2, which contained a limited number of trials (30 trials in the experiment 1 test 3 presentations of each cue; 32 trials in the experiment 2 tests 4 presentations of each cue). For these reasons, the results were not of primary interest and are presented in Appendix 1.

60 43 Chapter Three: Experiment Introduction: Background & Rationale It is theorized that memory is not a unitary process, but rather consists of multiple systems which rely on distinct neural substrates (Sherry and Schacter, 1987; Squire, 1992; Squire and Zola, 1996). A significant body of research, spanning across species, supports the existence of multiple memory systems in the mammalian brain. A major division is between the declarative memory system, dependent on the medial temporal lobes, and a nondeclarative memory system, which engages several brain structures, one of which is the basal ganglia system. While decades of research have investigated the individual roles of the aforementioned areas during multiple types of learning, a key remaining question of interest is examining how the medial temporal lobes and basal ganglia interact during learning and memory formation. This question has aroused great debate in the literature, with some evidence supporting competitive interactions (Poldrack et al., 2001; Poldrack and Packard, 2003; Foerde et al., 2006; Lee et al., 2008), other research supporting cooperative interactions (Dagher et al., 2001; Voermans et al., 2004; Cincotta and Seger, 2007; Sadeh et al., 2011) and a third body of evidence supporting parallel engagement of the medial temporal lobes and basal ganglia system (White and McDonald, 2002; Albouy et al., 2008; Atallah et al., 2008). For the purposes of this thesis, McDonald and White s definition of parallel memory systems is employed (2002), which suggests that these distinct systems may be online simultaneously and involved in distinct aspects of learning. The authors posit

61 44 that information passes independently through each system, such that each system receives the same information, but specializes in representing different aspects of the information. Furthermore, it is theorized that these systems may either interact directly in a cooperative or a competitive manner or may simultaneously influence behavior in parallel. While examining the nature of the interactions between the medial temporal lobes and the basal ganglia, it is important to consider how these distinct regions are functionally and anatomically connected. Research suggests that there are direct anatomical projections from the hippocampal formation to the nucleus accumbens and the caudate nucleus/putamen in the rodent (Jung et al., 2003). Furthermore, some researchers have proposed models describing how these regions may interact during learning. One key model of interest, developed by Lisman and Grace (2005) proposes that the MTL and BG may interact via anatomical connectivity with the dopaminergic midbrain. In the model, the hippocampus functions to detect the entrance of novel information and sends this novelty signal to the ventral tegmental area in the midbrain via projections through the subiculum, nucleus accumbens (ventral striatum), and ventral pallidum. The dopaminergic neurons within the VTA fire in response to this novelty signal, subsequently releasing dopamine into the hippocampus where it enhances long term potentiation. Importantly, the VTA also sends a projection back to the nucleus accumbens. Therefore, this model suggests a functional loop between the hippocampus, nucleus accumbens, and midbrain, which specializes in novelty detection and the entrance of information into long-

62 45 term memory. Dopamine is a logical candidate for facilitating interactions between these regions as it is believed to play a critical role in learning and memory processes (Wise, 2004; Shohamy and Adcock, 2010). Dopamine is implicated in reward-related learning, which is typically associated with striatal function (for review see Schultz, 2002) as well as facilitating long-term memory formation in humans, which is more traditionally associated with medial temporal lobe activation (Wittmann et al., 2005; Adcock et al., 2006; Shohamy and Wagner, 2008; Shohamy and Adcock, 2010). Therefore, anatomical as well as functional evidence supports a role for dopamine in facilitating interactions between the basal ganglia and medial temporal lobe. Given the debate regarding the nature of interactions between multiple memory systems, further research is required to examine multiple memory systems and how they interact during learning and memory formation. The purpose of the first experiment of this dissertation was to investigate in a broad manner, the relative engagement and interactions between the MTL and the BG during a simple learning paradigm using a within-subjects fmri design. Probabilistic learning was chosen because a major portion of the literature has employed probabilistic learning tasks to examine both the BG s and the MTL s involvement in learning. It has been suggested to be an effective manner of engaging both nondeclarative/implicit and declarative/explicit learning mechanisms (Knowlton et al., 1996; Shohamy et al., 2004b), although this theory is debated (Lagnado et al., 2006). A novel paradigm was developed for this set of experiments based on a card guessing task (Delgado et al., 2005), which was

63 46 previously shown to engage the striatum. It was decided to modify this task in particular due to the fact that it strongly engaged one of the key systems of interest (the basal ganglia). Modifications were performed to the task design in order to try to engage the MTL as well; specifically an observation version of the task was developed, modified from the task design used by Poldrack and colleagues (2001). In the experiment, participants were required to learn the value of easy and hard probabilistic cues either through feedback (trial and error) or via observation (paired-association). Based on the synthesis of current research in the field, three main points were considered while formulating a hypothesis regarding the nature of interactions between the MTL and BG: 1) both cooperative and competitive interactions between these regions have been reported in the literature, 2) evolutionary considerations suggest these two systems play fundamentally different roles in the brain, and 3) White and McDonald s theory of parallel engagement among memory systems in the brain posits that the MTL and BG may operate in parallel or interact directly in a cooperative or competitive manner. Based on the above evidence from the literature, it was hypothesized that the MTL and BG will operate in a parallel manner, potentially exhibiting both cooperative and competitive interactions. This hypothesis was tested by examining the relative activation within both the MTL and the BG during feedback and observation learning as well as functional and effective connectivity between these regions during learning. It was further hypothesized that such parallel engagement may be mediated by midbrain dopaminergic connectivity, based on dopamine s involvement in reward-related

64 47 learning and long-term memory formation as well as the known neuroanatomical projections from the midbrain to both the striatum and hippocampus (Shohamy and Adcock, 2010). Dopamine s influence in mediating interactions between the medial temporal lobe and the basal ganglia was indirectly assessed by the application of a reinforcement learning model postulated to reflect dopaminergic activity during feedback learning. 3.2 Materials & Methods Experimental design The probabilistic learning task used in this study was a modification of a previously used feedback-based card learning paradigm (Delgado et al., 2005). In this adapted version of the task, participants were asked to perform a categorization task that involved determining the numerical value of several probabilistic visual cues (e.g., circle is higher than the number 5; Figure 3.1). There were two distinct phases of the task, a learning phase, where participants acquired the values of the visual cues, followed by a test phase, where participants were asked to remember the cue associations they learned in the previous phase. Participants were instructed that each visual cue had been assigned a numerical value on the scale from 1-9 and could be thought of as having a value of either higher (6-9) or lower (1-4) than the number 5. They were instructed that they did not need to learn the exact value of each cue (e.g. circle is 7), but rather group the shapes into those with a value of higher than five and those with a value of lower than five.

65 48 Two independent variables were manipulated in the task. The first was the type of learning, or how the participants acquired the cue associations. In the learning phase, there were two distinct ways in which participants learned about the values of 8 cues. One manner was via feedback (4 cues), where participants guessed and received feedback regarding the cue s value, akin to trial-and-error or nondeclarative learning (Figure 3.1A). The second manner was via observation (4 cues) where participants observed the cue paired with its value, more similar to declarative learning. During feedback trials, participants saw a visual cue and made a button press indicating their belief regarding the value of the cue (button 1 = higher than 5; button 2 = lower than 5). Visual cue presentation and response (2 seconds total) were followed by feedback contingent on the participants response (check mark for correct trials, X mark for incorrect trials, and # symbol for missed trials; 2 seconds). Feedback was followed by a jittered inter-trial-interval (ITI; 2-14 seconds) to allow for the hemodynamic response to return to baseline. During observation trials, participants observed the cue paired with an informative arrow indicating the value of the cue (upward-facing arrow denotes a higher than 5 value; downwardfacing arrow denotes a lower than 5 value) and made a response indicating their belief regarding the value of the cue (button 1 = higher than 5; button 2 = lower than 5; Figure 3.1B). The visual cue and response period (2 seconds) was followed by a message informing the participants if their response had or had not been recorded (2 seconds). As long as the participants made a button press within the 2 second visual cue/response period, they received a message that

66 49 their answer was recorded, if they failed to respond within 2 seconds for any reason, they received a message that their answer was not recorded (missed trial). This message was followed by a jittered ITI (2-14 seconds) before presentation of the next visual cue. The button response in the observation trials served as a motor control for the button response made during the feedback trials, equated the duration of each trial type (4 seconds total), and was an attempt to equate the difficulty of the trial types. During the learning phase, participants completed 4 blocks, alternating between feedback and observation blocks (2 of each type). Each block contained 40 trials, for a total of 160 trials in the learning phase. Feedback and observation blocks differed in terms of the cues presented as well as cue color (e.g., feedback cues were pink, observation cues were blue). Block order and cue color were counterbalanced, and within each block, cue trial order was randomized. The second independent variable manipulated in the paradigm was the probability of the cues (predictive outcome of the numerical value for each cue), referred to as cue difficulty. Four cues (2 feedback and 2 observation) were 85% predictive of the outcome (higher or lower than 5) and are referred to as easy cues and the remaining four cues (2 feedback and 2 observation) were 65% predictive of the outcome and are referred to as hard cues. Cue difficulty was included as a variable as both the BG and MTL are known to be modulated by the difficulty of material to-be-learned. During the learning phase, participants were specifically told to optimize their responses to the cues. They were told that as the value of the cues was probabilistic, for the observation trials, the direction

67 50 of the arrows would change. For example, if a cue is 85% higher than five it is thereby 15% lower than five. Participants were instructed to pay careful attention to the direction of the arrows and to try to determine what the value of each cue was most of the time. On all trials, they were instructed to press the button corresponding to the value that cue was associated with for the majority of the trials in the task. Therefore, on the inconsistent trials, (cue is lower than five) participants could choose to push the button indicating higher, if that is what they believed the value of the cue to be most of the time. As a result of these instructions, participants were actively making a choice regarding each cue s value and were not passively pushing the button that always matched the direction of the arrow. Because participants were instructed to optimize their responding, behavioral accuracy was scored according to participants actual choices. That is if they always pushed the button indicating a higher than five response, for a cue which was 85% or 65% higher than 5, they would receive a score of 100% correct; if they followed the arrows exactly they would probability match and be 85% or 65% correct respectively. As a point of clarification, participants were not told the exact probabilities (e.g., 85%) but were simply introduced to the concept of probabilistic information. The test phase was presented immediately following completion of the learning phase, while the participants remained inside the scanner (Figure 3.1C). The test phase contained all eight visual cues presented in the learning phase, in addition to two novel cues presented for the first time in the test phase. The novel cues were presented to provide a control for comparing previously learned information with

68 51 novel information. There were 30 trials total in the test phase, three presentations of each of the ten cues. Trials consisted of a self-timed cue presentation/response period followed by a jittered ITI (6-14 seconds) before onset of the next cue. Critically, feedback (from the feedback trials) and informative arrows (from the observation trials) were not presented in this phase. Participants saw only the cue by itself (e.g., circle) and made a button press to indicate its value, based on what they learned in the learning phase. A second test phase was administered a couple of days following completion of the MRI study in a behavioral-only test session completed in the laboratory (M = 2.63 days, SD = 1.63 days) Data analysis GLM: A random-effects GLM analysis was conducted in the learning phase to examine BOLD activity during the 4-second cue presentation + feedback period, using the following five predictors: learning type (feedback and observation), cue difficulty (easy and hard), and missed trials. From this GLM statistical parametric maps (SPM) were generated and thresholded at p<0.005 with a cluster threshold correction of 5 contiguous voxels (135mm 3 in 1x1x1mm units). A repeated-measures ANOVA was then conducted within BrainVoyager using learning type (feedback and observation) and cue difficulty (easy and hard) as within-subjects factors. The primary SPM of interest investigated a main effect of cue difficulty. This analysis was of interest as it provided a non-biased examination of the relative engagement and pattern of activity within the BG and the MTL during learning. Functional ROIs were defined based on the resulting

69 52 map and the BOLD signal (characterized by beta weights or mean parameter estimates) was extracted in order to examine potential similarities and differences between feedback and observation learning in post-hoc analyses. In addition, potential effects of learning changes over time were examined. In order to model time, the learning phase was divided according to run/block. The BOLD signal was examined in the functional regions of interest (e.g., caudate nucleus and hippocampus) for the first learning run/block of each learning type (e.g., feedback trials 1-40; observation trials 1-40) and similarly for the second learning run/block of each learning type (e.g., feedback trials 41-80; observation trials 41-80). Subsequently, the mean BOLD signal from the first or early learning run was then compared with the mean BOLD signal from the second or late learning run for both learning types in both regions. The second SPM of interest directly investigated a main effect of learning type. The third SPM of interest examined potential interactions of cue difficulty and learning type. During the test phase, BOLD activity was examined during the stimulus onset and participant response period. A random effects GLM was conducted with observation (easy and hard), feedback (easy and hard), and novel cues as predictors, in addition to 6 motion parameters included as regressors of no interest. As the GLM is unbalanced, a 2x2 (learning session x cue difficulty) within-subjects ANOVA could not be performed in this phase. Therefore, a contrast of previously studied (observation and feedback cues collapsed across difficulty) vs. non studied (novel) cues was performed, thresholded at p < The cluster level statistical threshold estimator plugin was used on the resulting

70 53 SPM. Parameter estimates were extracted from the resulting region and examined with post-hoc analyses. Any reported regions not withstanding correction are labeled clearly and should be interpreted with caution (Poldrack et al., 2008). GCA: A Granger causality analysis was performed specifically to probe potential interactions between the hippocampus and striatum during learning. The functionally defined hippocampus ROI from the main effect of cue difficulty ANOVA in the learning phase was used as the principal seed region for this analysis. This ROI was chosen as the primary seed region given the direct anatomical projections from the hippocampus to the basal ganglia (Kelley and Domesick, 1982). Functional and effective connectivity maps were generated separately for the feedback and observation runs. At the individual subject level, the two feedback blocks were combined to form one feedback map and the two observation blocks were combined to form one observation map. Time course values entered into the analysis contained the entire run, from the first to the last time point (240 volumes per run). At the multi-subject level, each participant s feedback map was combined to make one group feedback map and each participant s observation map was combined to make one group observation map. Maps were given a threshold of p< with a cluster threshold correction of 7 continuous voxels for the observation maps and 8 continuous voxels for the feedback maps (cluster-level false-positive rate of 5%). Standard second level statistics were performed on the group maps; specifically a t-test was calculated. Reported results do not contain effective connectivity (F y x

71 54 and/or F x y ), as no effective connectivity was observed between the regions of interest (MTL and BG). Therefore the results reported represent functional connectivity (instantaneous influence with no directionality information; F x y ). Lastly, a second Granger causality analysis was performed in the same manner as a control analysis, using the caudate nucleus region obtained from the learning phase ANOVA main effect of cue difficulty analysis. The resulting maps were thresholded at p< with a cluster threshold correction of 12 continuous voxels (cluster-level false-positive rate of 5%) unless otherwise noted. PE: The prediction error values generated from the Q learning model presented in the general methods chapter were then used as a regressor in the neuroimaging GLM analysis. Eight additional predictors were included in the GLM: trial event and missed responses, as well as six motion parameters. Within the GLM, the PE values were coded during the two second feedback presentation time period for only the feedback learning trials. A SPM which probed regions of the brain that varied parametrically with the PE values during feedback learning was generated, thresholded at p<0.005 with a voxel contiguity of 5 continuous voxels (cluster-level false-positive rate of 5%). 3.3 Results Behavioral results Learning phase: Accuracy The primary analysis probed accuracy differences between the learning types and levels of cue difficulty. A 2 (learning type: feedback vs. observation) x

72 55 2 (cue difficulty: easy vs. hard) repeated measures ANOVA was performed. No main effect of learning type (F (1,15) = 1.47; p > 0.05), a main effect of cue difficulty, (F (1,15) = 16.04; p< 0.01), and no significant interaction (F (1,15) = 0.85; p > 0.05) were observed (Figure 3.2A). In order to examine the main effect of cue difficulty within and across conditions, post-hoc t-tests were performed. Participants accuracy was significantly better on easy compared to hard cues for both the feedback (t (15) = 2.45; p < 0.05) and observation (t (15) = 2.54; p < 0.025) trials. There was marginally better performance for easy cues during observation compared with feedback learning (t (15) = 2.24; p = 0.04; trend after sequential Bonferroni correction), with no performance differences for hard cues between learning types (t (15) = 0.23; p > 0.05). A secondary analysis examined changes in learning accuracy over time in feedback and observation trials by comparing performance early (first block) versus late (second block) during the learning phase. A 2 (learning type: feedback vs. observation) x 2 (cue difficulty: easy vs. hard) x 2 (time: early vs. late) repeated measures ANOVA indicated a no main effect of learning type (F (1,15) = 1.38; p > 0.05), a main effect of cue difficulty (F (1,15) = 16.09; p < 0.01), a main effect of time (F (1,15) = 35.07; p < 0.01), and no significant interactions. The significant main effect of time was driven by better performance for the hard but not the easy trials over time. Test phase: Accuracy The first analyses examined differences between learning sessions and levels of cue difficulty in each individual test session (immediate and follow-up).

73 56 During the individual test phases, no differences between feedback and observation learning or between easy and hard cues were observed. In the immediate test phase, a 2 (learning type: observation vs. feedback) x 2 (cue difficulty: easy vs. hard) repeated measures ANOVA (excluding the novel information and examining only the previously studied material) revealed no significant main effect of learning type (F (1,15) = 0.06; p > 0.05), no significant main effect of cue difficulty (F (1,15) = 0.23; p > 0.05), and no significant interaction (F (1,15) = 1.92; p > 0.05). These results suggest that participants successfully learned contingencies independent of learning type and/or level of cue difficulty. The same 2 x 2 repeated measures ANOVA was performed for the follow up test phase. No significant main effect of learning type (F (1,15) = 0.51; p > 0.05), no significant main effect of cue difficulty (F (1,15) = 0.37; p > 0.05), and no significant interaction (F (1,15) = 0.00, p > 0.05) were observed. The second analysis probed for differences in accuracy between learning material over time. In order to test this, one analysis examined differences across the two test sessions. A 3 (learning material: observation, feedback, novel) x 2 (time: immediate vs. follow up test session) repeated measures ANOVA was performed. A main effect of learning material (F (1.90,28.43) = 14.04; p < 0.01), no main effect of time (F (1,15) = 0.11; p > 0.05), and a marginally significant interaction (F (1.36,20.36) = 3.33; p = 0.07) were observed (all factors were Greenhouse-Geisser corrected; Figure 3.2B). The main effect of learning material indicated greater accuracy for studied (feedback and observation) compared with non-studied (novel) material. Post-hoc t-tests revealed that

74 57 participants performance in the feedback version of the task marginally declined in the follow-up test session (t (15) = 2.13; p = 0.05; trend after sequential Bonferroni correction), but no differences were observed for observation accuracy over time (t (15) = 1.04; p > 0.05). As anticipated, participants accuracy in the novel condition did not change across time (t (15) = 1.01; p > 0.05) Neuroimaging results Learning phase Main Effect of Cue Difficulty: The primary analysis of interest in the learning phase was to examine a main effect of cue difficulty, as that was the significant result in the behavioral data and would provide for a non-biased examination of the MTL and BG s activation during learning. A 2 (learning type: feedback vs. observation) x 2 (cue difficulty: easy vs. hard) within-subjects ANOVA was performed within BrainVoyager and a main effect of cue difficulty was examined (Table 3.1). A region of the left caudate nucleus was identified as processing a main effect of cue difficulty (Figure 3.3), along with an area in the left hippocampus that did not survive cluster threshold correction. Mean parameter estimates from these two functionally defined ROIs were then extracted for further analyses. In the left caudate nucleus (x, y, z = -15, 20, 7; Figure 3.3A and Figure 3.3B), the pattern of BOLD responses was similar for both feedback and observation learning, with no differences observed between type of learning when collapsed across cue difficulty (t (15) = 1.00; p > 0.05). Posthoc t-tests indicated a greater BOLD response for easy compared with hard cues in both types of learning [feedback: (t (15) = 5.02; p < 0.025); observation: (t (15) =

75 ; p < 0.05]. In the left hippocampus (x, y, z = -36, -28, -8; Figure 3.3C and Figure 3.3D) a marginally significant effect was observed when comparing mean parameter estimates from the feedback and observation sessions (t (15) = 2.01; p = 0.06), with a marginally greater BOLD response for the observation trials. This difference was driven mostly by performance for the hard trials (t (15) = 2.40; p = 0.03; trend after sequential Bonferroni correction). Post-hoc t-tests indicated a greater BOLD response for easy compared with hard cues for the feedback learning (t (15) = 3.73; p < 0.025) and a marginally greater response for easy than hard cues in the observation trials: (t (15) = 1.79; p = 0.09). As a secondary analysis, changes in the BOLD response during feedback and observation sessions over time were examined by comparing the BOLD signal early (first block) versus late (second block) during the learning phase. A 2 (learning type: feedback vs. observation) x 2 (cue difficulty: easy vs. hard) x 2 (time: early vs. late) repeated measures ANOVA conducted on the mean BOLD responses extracted from the caudate nucleus revealed no main effect of learning type (F (1,15) = 0.82; p > 0.05), a main effect of cue difficulty (F (1,15) = 95.47; p < 0.01), no main effect of time (F (1,15) = 1.97; p > 0.05), and no significant interactions. A nearly significant increase in mean parameter estimates over time was observed for the observation hard cues in the caudate nucleus (t (15) = 2.25; p = 0.04; trend after sequential Bonferroni correction). The same 2 x 2 x 2 repeated measures ANOVA was conducted on the BOLD responses extracted from the hippocampus ROI. A nearly significant main effect of learning type (F (1,15) = 3.36; p = 0.09), a main effect of cue difficulty (F (1,15) =

76 ; p < 0.01), a main effect of time (F (1,15) = 5.35; p < 0.05), and a nearly significant interaction between learning type and time (F (1,15) = 3.76; p < 0.07) were observed. Post-hoc comparisons indicated a nearly significant increase in the hippocampus for the feedback hard cues as learning progressed (t (15) = 2.38; p = 0.03; trend after sequential Bonferroni correction). Main Effect of Learning Type: The second analysis examined a potential main effect of learning type, generated from the same learning type x cue difficulty ANOVA described previously (Table 3.2). Several regions of the basal ganglia were engaged including the right ventral portion of the head of the caudate nucleus (x, y, z = 6, 3, 4) and the left ventral caudate nucleus extending into the globus pallidus (x, y, z = -12, 2, 4). No voxels within the medial temporal lobe were observed at a threshold of p < These results indicate that some regions of the BG are involved in primarily processing feedback information more so than observation information, supporting a parallel model of engagement. Interaction of Cue Difficulty and Learning Type: The last main analysis from the learning phase ANOVA examined a potential interaction between learning type and cue difficulty. Results revealed activation in an area of the left medial prefrontal cortex. Post-hoc t-tests conducted on the mean parameter estimates extracted from this region indicated greater activity for observation easy than hard trials (t (15) = 4.63; p < 0.025); but no difficulty differences for the feedback trials (t (15) = 1.26; p > 0.05). The interaction was driven by a greater BOLD response to observation easy compared with feedback easy cues (t (15) = 2.72; p < 0.025) and a marginally greater response to feedback than observation

77 60 hard cues (t (15) = 1.94; p = 0.07). These results indicate that this region of the prefrontal cortex is involved in processing feedback and observation information, modified by cue difficulty. Learning phase: Correlations between striatum and medial temporal lobe As a first pass at exploring the relationship between the caudate nucleus and hippocampus s BOLD responses during learning, a series of Pearson s correlations were performed. A significant positive correlation during the observation learning session was observed between mean parameter estimates extracted from the hippocampus and the caudate nucleus identified in the main effect of cue difficulty analysis (r = 0.498, p = 0.05). There was no relationship between the two regions BOLD responses overall in the feedback session. However, exploratory analyses revealed a marginally significant positive correlation between the caudate nucleus and hippocampus during the last block of feedback learning, specifically for easy cue trials when participants expectations were violated by the receipt of incorrect feedback (r = 0.530, p = 0.08). Granger Causality Analysis To more thoroughly assess the level of connectivity and potential interactions between the BG and MTL during learning, a Granger causality analysis was performed. The left hippocampus from the main effect of cue difficulty learning phase analysis was used as the primary seed region. The resulting Granger causality maps highlight instantaneous correlations between the hippocampus (seed region) and regions of the striatum during both

78 61 observation and feedback probabilistic learning sessions. In particular, instantaneous influence between the hippocampus and two regions of the right caudate nucleus were observed during feedback learning (x, y, z = 14, 18, 13 and x, y, z = 14, 12, 19; not shown), as well as nearly the identical ROIs in the caudate nucleus (x, y, z = 14, 18, 13 and x, y, z = 14, 10, 18; not shown; Figure 3.4A) and one region of the right ventral putamen in observation learning (x, y, z = 22, 3, -4; not shown) (Figure 3.4B). A second Granger causality analysis using the caudate nucleus as the seed region was performed as a control analysis because the hippocampus seed was taken from an uncorrected region in the learning session. The results yielded instantaneous influence between the caudate nucleus and bilateral hippocampal regions in observation learning sessions (x, y, z = -31, -38, -3 and x, y, z = 32, -29, -12; not shown) corrected to a cluster level false positive rate of 5%, as well as an area near the right hippocampus in feedback learning sessions (x, y, z = 29, -8, -15; uncorrected for multiple comparisons). Prediction Error Analysis The last analysis conducted examined a putative influence of midbrain dopaminergic neurons in learning, which are known to project to both the striatum and the hippocampus (Scatton et al., 1980; Lynd-Balta and Haber, 1994) and are involved in learning and memory processes (Wise, 2004; Lisman and Grace, 2005; Shohamy and Adcock, 2010). The prediction error learning signal is believed to be a correlate of physiological firing of dopamine neurons during reward-related learning (Schultz, 1997; Schultz et al., 1997; Schultz,

79 ). Based on the observation of activation of both the striatum and hippocampus during the learning phase - processing differences in cue difficulty, a reinforcement learning algorithm was applied to the behavioral data. The results yielded an area of the left putamen (x, y, z = -30, 2, 4; Figure 3.5A) and the right hippocampus (x, y, z = 27, -28, -14; Figure 3.5B) whose activation parametrically varied with the PE signals generated from the model. Thus, the use of a reinforcement learning model during feedback learning engaged the striatum, a typically reported region in prediction error encoding (Pagnoni et al., 2002; McClure et al., 2003; Abler et al., 2006) as well as the hippocampus, a novel finding in an area which has more often been associated with mismatch signals (Ploghaus et al., 2000; Chen et al., 2011). 3.4 Discussion The purpose of this experiment was to investigate the relative engagement and interactions of multiple memory systems during probabilistic learning. The nature of how multiple memory systems interact is debated, with several studies suggesting that the BG and MTL compete during learning (Poldrack et al., 2001; Foerde et al., 2006; Lee et al., 2008). The data collected in this experiment, however, are inconsistent with the hypothesis of purely competitive interactions, and support an alternative hypothesis that these memory systems operate in parallel during learning (White and McDonald, 2002; Albouy et al., 2008; Atallah et al., 2008). Using an event-related probabilistic task and within-subjects measures of learning allowed for a simple and controlled

80 63 investigation of the engagement and interactions between multiple memory systems. Behavioral results indicated that as learning progressed, accuracy was modulated by cue difficulty (easy and hard) across learning types (feedback or observation). Regions within the BG and MTL, specifically the caudate nucleus of the striatum and the hippocampus, tracked these behavioral changes as they were modulated by cue difficulty, displaying similar patterns of activity in response to both feedback and observation information during learning. Furthermore, BOLD signals within these regions were positively correlated during learning, as assessed by simple correlations and a more sophisticated functional connectivity analysis using both the hippocampus and the caudate nucleus as reference regions. The functional connectivity observed between these areas may potentially be explained by modulation of midbrain dopaminergic neurons during reward-related learning (Lisman and Grace, 2005; Shohamy et al., 2008; Shohamy and Adcock, 2010) as loci within both the striatum and MTL were found to vary parametrically with a prediction error signal. The results of the prediction error analysis further corroborate the hypothesis that these distinct memory systems may operate in parallel while processing probabilistic information as both the striatum and hippocampus were actively engaged in learning scenarios where violations of expectations or mismatches occurred, which may function as useful updating signals during the learning process. The results from the present experiment complement the visuomotor and simple association learning literature, which supports the theory that the basal ganglia and medial temporal lobes may be engaged simultaneously in learning

81 64 arbitrary visuomotor associations (Toni et al., 2001; Amso et al., 2005; Law et al., 2005; Haruno and Kawato, 2006). The current results are consistent in particular with a recent study by Mattfeld and Stark (2010) in which the interaction of the BG and MTL was investigated using an arbitrary visuomotor association task. Several regions of the BG and MTL exhibited increases in BOLD signal as the strength of memory increased during a trial and error task, suggesting that regions of the BG and MTL are engaged when learning arbitrary associations. The observation in the present experiment of the caudate nucleus and hippocampus demonstrating larger BOLD responses to easy compared to hard cues complements Mattfeld and Stark s result. The authors also employed a functional connectivity analysis, results of which indicated functional connectivity between both the ventral (nucleus accumbens) and dorsal (caudate nucleus) striatum with the hippocampus during learning, corroborating the interactive nature of the hippocampus and striatum during learning and memory processes. One key distinction between the task employed in the current experiment and more traditional visuomotor tasks is the inclusion and examination of different learning types (observation in addition to feedback). The current experiment s results enhance recent neuroimaging findings demonstrating cooperative interactions between the BG and MTL during category learning (Voermans et al., 2004; Cincotta and Seger, 2007). In the present experiment, an event-related design allowed for the decoupling of factors such as cue difficulty and examining changes in learning over time to lend further support to the hypothesis that parallel processing in the BG and MTL contributes

82 65 to overall learning during probabilistic paradigms. A significant positive correlation between the BG (caudate nucleus) and MTL (hippocampus) during the observation learning session and a marginally significant positive correlation during the feedback learning session were observed in the present experiment. Whereas negative correlations have been interpreted as competition between memory systems (Poldrack et al., 2001), positive correlations may suggest cooperative, perhaps synergistic interactions, leading to the interpretation that the systems may communicate with each other in specific contexts to facilitate probabilistic learning. This interactive nature between the MTL and BG is illustrated by Granger causality results in the present study which indicated that loci within the striatum and the hippocampus were correlated at simultaneous time points during the learning phase when using both the caudate nucleus and hippocampus as reference regions. Based on the Lisman and Grace model (2005), it may have been predicted that ventromedial regions of the striatum would be engaged during the present paradigm, correlating with hippocampus activation, as opposed to more dorsal regions. Consequently, it may be surprising that dorsal regions of the striatum were observed during learning. However, the literature supports a role for the dorsal striatum during instrumental tasks (O'Doherty et al., 2004), particularly when learning is action contingent (Tricomi et al., 2004). Moreover, the dorsal striatum, especially the caudate nucleus, has been shown to be involved in cognitive tasks involving feedback (Poldrack et al., 2001; Seger and Cincotta, 2005; Seger, 2008; Tricomi and Fiez, 2008). Considering this

83 66 evidence, it is logical that the strongest loci of activation within the striatum were observed in the dorsal rather than ventral striatum in the current cognitive learning paradigm. Importantly, there is also a direct projection from the hippocampus to the caudate nucleus in the rodent, albeit a ventral region of the caudate nucleus (Jung et al., 2003). Lastly, it has been posited that the ventral striatum may communicate with the dorsal striatum via multiple spiral loops connecting the striatum with dopaminergic midbrain centers (Heimer et al., 1997; Haber, 2003). Therefore, it is possible that interactions between the hippocampus, midbrain DA areas, and more dorsal regions of the striatum exist, via a ventromedial to dorsolateral movement of information through the aforementioned spiral loops. In addition to the similarities in the activation observed in the hippocampus and regions of the basal ganglia during the task, two differences emerged. First, regions within the basal ganglia were modulated by a main effect of learning type, while no voxels were identified in the MTL showing such differentiation. Several neuroimaging papers have shown that reward processing and feedback recruit regions of the ventromedial striatum (for review see Delgado, 2007); as such, it is not surprising that this region was recruited more strongly during the feedback learning trials. It may have been expected that the MTL would be selectively modulated by observation trials given previous findings (Poldrack et al., 2001); however, this result was not observed in the current experiment. While a null result in neuroimaging studies does not indicate any particular finding, and the context and experimental design of the present paradigm differ

84 67 from previous probabilistic learning studies, it is possible that the MTL is recruited during both feedback and observation learning in the present task as suggested by the main effect of difficulty analysis. A second difference that was observed during learning between the hippocampus and caudate nucleus was that the hippocampus showed a main effect of time (early x late learning) during the learning session, whereas responses within the caudate nucleus were not significantly modulated across time. This effect was driven by changes in activity during feedback learning (primarily for the hard cues), which increased during later compared to early stages of learning in the hippocampus. This result may suggest that the hippocampus is recruited more heavily for difficult feedback trials later in the learning process. The observation of both the hippocampus and striatum processing differences in cue difficulty, but only the ventral striatum exhibiting stronger engagement during feedback learning is indicative of parallel engagement of these regions during probabilistic learning. These results suggest that the striatum plays a dynamic role during probabilistic learning, processing both learning type as well as cue difficulty. Furthermore, the results suggest that the striatum may be processing similar or distinct information as the hippocampus suggesting they may operate independently and in parallel. Neither purely cooperative interactions (similar pattern of activity in the BG and MTL in all learning scenarios) nor purely competitive interactions (opposite patterns of activity during learning or negative correlations) were observed in the present paradigm.

85 68 As a caveat for the present study, it should be noted that the hippocampus locus obtained from the main effect of cue difficulty analysis was reported at a threshold of p<0.005, uncorrected for multiple comparisons. Additionally, two areas in the test phase analysis, a region adjacent to the hippocampus and the caudate nucleus, were also reported at p<0.005 uncorrected. Although this information may be useful for studies in the future, regions reported at an uncorrected threshold should be regarded with caution due to the increased likelihood of observing a Type I error (Poldrack et al., 2008). There are several open questions remaining based on the results of experiment 1. The primary question of interest is addressing in a more direct manner how the BG and MTL interact during probabilistic learning. One manner of addressing this question is by employing multiple types of learning as well as by examining how systems interact during the reversal of probabilistic contingencies. The present study examined the relative engagement of distinct regions during different types of learning and probed interactions between these regions using correlations, functional connectivity analyses, and reinforcement learning models. However, because all participants completed both learning types in the same learning session, it is possible that the current results are biased by the presentation of both learning types (feedback and observation) within the same learning phase. Therefore, in order to address this concern, it is necessary to examine the engagement of these discrete regions in observation and feedback learning in a distinct manner. Experiment 2 attempts to address these unresolved issues by 1) employing a between-subjects design such that

86 69 participants initially learn via only one type of learning and 2) by reversing contingencies in the middle of the learning session to probe in a novel manner how the memory systems interact during probabilistic learning. Chapter Four: Experiment Introduction: Background & Rationale The nature of interactions between multiple memory systems is currently debated in the literature. Evidence supporting cooperative (Dagher et al., 2001; Voermans et al., 2004; Cincotta and Seger, 2007; Sadeh et al., 2011), competitive (Poldrack et al., 2001; Foerde et al., 2006; Seger and Cincotta, 2006; Lee et al., 2008), and parallel (McDonald and White, 1994; White and McDonald, 2002; Albouy et al., 2008; Doeller et al., 2008; Tricomi and Fiez, 2008) interactions has been documented across species using several different types of learning paradigms. In experiments specifically involving human participants probabilistic category learning (Knowlton et al., 1994; Knowlton et al., 1996; Poldrack et al., 1999; Poldrack et al., 2001; Hopkins et al., 2004; Shohamy et al., 2004b; Shohamy et al., 2004a; Shohamy et al., 2008), information integration category learning (Ashby et al., 2002; Cincotta and Seger, 2007), visuomotor learning (Seger and Cincotta, 2005; Mattfeld and Stark, 2010), spatial navigation (Voermans et al., 2004), and declarative memory tasks (Tricomi and Fiez, 2008; Sadeh et al., 2011) have all been employed when investigating this question. Some studies have examined interactions between memory systems by employing distinct types of learning, in particular feedback and observation

87 70 learning (Poldrack et al., 2001; Ashby et al., 2002; Shohamy et al., 2004b; Cincotta and Seger, 2007; Schmitt-Eliassen et al., 2007; Djonlagic et al., 2009; Li et al., 2011). Feedback learning is traditionally associated with engagement of the basal ganglia, in particular the striatum (see Delgado, 2007 for review), whereas human observation learning (or instructed learning) has been associated with medial temporal lobe (Poldrack et al., 2001; Li et al., 2011), striatal (Cincotta and Seger, 2007; Li et al., 2011), as well as dorsolateral and ventromedial prefrontal cortex activation (Burke et al., 2010; Li et al., 2011). Results from fmri experiments that specifically employed feedback and observation learning have supported both competitive (Poldrack et al., 2001) and cooperative interactions (Cincotta and Seger, 2007) between the medial temporal lobes and basal ganglia. Given that the nature of these interactions is debated, further research is required to probe how multiple memory systems interact during probabilistic learning. The goal of this experiment was to address this question by employing multiple types of learning and examining interactions across two stages of the learning process: initial acquisition and later updating following a reversal in cue contingencies (e.g., cue value reversal). Several studies have demonstrated a role for the basal ganglia during reversal learning (Annett et al., 1989; Schoenbaum and Setlow, 2003; Setlow et al., 2003; Frank and Claus, 2006). One study of interest examined the neural correlates of probabilistic reversal learning and demonstrated activation of frontostriatal regions during the key time point of behavioral reversal (Cools et al., 2002). In this probabilistic reversal learning task, participants initially acquired

88 71 cue contingencies via trial and error. After a predetermined number of correct responses, cue contingencies reversed, unbeknownst to the participants. The participants were then required to update the new cue associations and change their responding in order to receive correct feedback. Cools and colleagues demonstrated that a frontostriatal network was engaged when processing the critical, final error preceding participants reversing their behavioral choices in order to receive correct feedback. This study illustrates the involvement of the frontostriatal network during the updating of probabilistic cue contingencies. This result is relevant as the task employed in the present experiment contained one reversal of probabilistic cue contingencies during learning. Furthermore, the striatum is one of the key areas of interest in the studies presented in this dissertation, and it is therefore pertinent to know that it may be mediating the reversal of probabilistic cue contingencies. In addition, the medial temporal lobes have also been implicated in playing a fundamental role in reversal learning. Both research using non-human animals and humans has documented the involvement of the hippocampus during reversal learning. Lesions to the hippocampus in animals impair reversal learning (Zola and Mahut, 1973; Berger, 1983; Fagan and Olton, 1986; Marston et al., 1993). In human studies, patients with bilateral damage to the hippocampus are impaired at reversing previously acquired stimulus-outcome associations (Myers, 2000; Carrillo et al., 2001; Myers et al., 2006) and tend to perseverate by choosing previously a correct association. A study recently confirmed a role for the basal ganglia and the medial temporal lobes during

89 72 probabilistic reversal learning using a feedback-based task with Parkinson s patients and MTL amnesic patients (Shohamy et al., 2009). Shohamy and colleagues modified the weather prediction task, making a simpler task in which each cue predicted the correct outcome 80% of the time and each cue pattern (every trial consisted of a pattern of 3 cue cards on the screen) predicted the outcome 100% of the time. Results indicated that while both PD and MTL amnesic patients were able to initially acquire cue contingencies, both patient groups demonstrated impairments with reversal. PD patients did successfully reverse cue contingencies, but did so by choosing to respond to a new cue among the pattern, thereby learning a new cue-outcome association, rather than updating the old cue-outcome association and thus opting out of the reversal. MTL amnesic patients failed to successfully reverse cue contingencies completely and demonstrated perseverative behavior to previously correct associations. The authors concluded that the basal ganglia and medial temporal lobes therefore both contribute to probabilistic reversal learning, but in unique ways. The basal ganglia may be required for updating old cue-outcome associations, whereas the medial temporal lobes may play a broader role in flexibly modifying cue-outcome associations. These data are in accord with the parallel model of engagement between multiple memory systems. Of additional interest, Swainson and colleagues (2000) demonstrated that probabilistic reversal learning was impaired in mild Parkinson s patients (PD), who were receiving dopaminergic medication as a part of disease treatment. The authors concluded that their results were in agreement with the dopamine

90 73 overdosing hypothesis (Gotham et al., 1988; Frank et al., 2004; Frank, 2005; Cools, 2006). The overdosing hypothesis suggests that dopaminergic medicine, while augmenting depleted dopamine levels in the dorsal striatum in the early stages of PD, overdoses the still intact ventral striatum s dopamine levels, one of the consequences of which is washing out critical prediction error signals thus preventing them from influencing behavior. This elegant study therefore demonstrated a critical role for dopamine during the acquisition of probabilistic information. This result is relevant to the present experiment, and to this dissertation as a whole, as it is hypothesized that dopamine may be facilitating interactions among multiple memory systems during probabilistic learning. The task in experiment 2 combines the use of multiple types of learning (feedback and observation) with probabilistic reversal learning in order to probe how multiple memory systems interact during a dynamic learning environment. It may be informative to examine the engagement of memory systems when learning types evolve and when learning contingencies change, as this may provide a novel way of examining how distinct neural substrates interact during the learning process. The purpose of experiment 2 therefore was to examine in a novel manner, how multiple memory systems interact, by probing these interactions when learning types evolve. In particular, the experimental task investigated changes in the engagement and interactions between multiple memory systems when probabilistic cue contingencies were initially acquired via one learning type and subsequently updated via the same as well as a distinct learning type. Specifically, the experiment was a between-subjects, event-

91 74 related fmri design involving two groups of participants. While undergoing functional neuroimaging, one group initially acquired all probabilistic cue contingencies via feedback learning, and subsequently updated the cue contingencies via the same learning type, feedback, as well as a new, different learning type, observation (group 1/feedback group). A second group of participants initially acquired cue contingencies via observation learning and subsequently updated contingencies via both the same, observation, and a new, different learning type, feedback (group 2/observation group). This design therefore allowed for probing a key question of interest, that being the investigation of the relative engagement and interactions between memory systems when updating probabilistic cue information via the same learning type as initial cue acquisition (feedback-feedback/group1 and observationobservation/group2) and a different learning type as initial cue acquisition (feedback to observation/group1 and observation to feedback/group2). Addressing the question in this manner allowed for the examination of neural substrates in initial feedback learning versus observation learning, as well as the investigation of what systems were engaged and potential interactions between these systems when the learning type changed (that being either feedbackobservation or observation-feedback) versus when the learning type remained the same (feedback-feedback or observation-observation). Additionally, by employing this task design, an examination of pure feedback learning (group 1) as well as pure observation learning (group 2) was initially examined. This was necessary as the results from experiment 1

92 75 indicated involvement of both the striatum and hippocampus during feedback and observation learning. It is possible that this result was observed, however, not due to the fact that the striatum and hippocampus were both involved in both learning types, but rather that each region was engaged in one learning type (e.g., striatum during feedback and hippocampus during observation) and activation was simply contaminated in the areas due to the presence of both learning types in the same learning session. Therefore, a between-subjects design was employed in experiment 2 in order to address this confounding issue in experiment 1. While between-subjects designs do contain increased variability, as comparisons are made across subjects, at times such designs are deemed necessary. Following the initial acquisition period, a test was administered in both groups to ensure participants had fully acquired the cue contingencies. After the test, the update phase was administered, in which participants in both groups 1 and 2 learned about cue contingencies via both learning types (feedback and observation). Critically, cue contingencies were reversed in this session, unbeknownst to the participants, requiring them to update previously correct answers in order to receive correct feedback and perform well in the task. Lastly, a second test was administered in order to test whether participants had correctly updated the new cue contingencies. The following behavioral hypotheses were formulated: in the learning phase, it was hypothesized that participants would initially acquire cue contingencies well, irrespective of how contingencies were initially acquired (feedback/group1 vs. observation/group2 learning). In the update phase, it was

93 76 hypothesized that participants accuracy would initially decrease following probabilistic cue reversal for both learning types (feedback and observation trials) in both groups (groups 1 and 2), and subsequently improve over time as participants updated old cue contingencies. Lastly, two additional hypotheses were made regarding behavioral accuracy in the update phase: 1) participants would be more efficient at updating cue associations within the same learning type (observation-observation or feedback-feedback) or 2) participants would be more efficient at updating cue associations by employing a new learning type (observation-feedback or feedback-observation). The following neuroimaging hypotheses were formulated: based on the results of experiment 1, it was hypothesized that the basal ganglia and medial temporal lobes would be engaged during the initial acquisition (learning phase) of both feedback (group 1) and observation (group 2) information. During the update (reversal) phase, it was hypothesized that both memory systems would be engaged and involved in updating cue contingencies in parallel, potentially exhibiting both cooperative as well as competitive interactions during the updating process. Interactions between distinct memory systems during updating were measured by the relative engagement of each region during the update phase, the application of simple correlation analyses, as well as the application of more sophisticated functional and effective connectivity analyses. Lastly, reinforcement learning models were applied to the learning and update phases in order to examine the role of putative dopaminergic influences during the initial acquisition of feedback (group 1) and observation (group 2) information,

94 77 as well as during the updating of cue contingencies via both feedback and observation learning in both groups. 4.2 Materials & Methods Experimental design This experiment was a modification of the task used in experiment 1 (Figure 4.1). In this between-subjects version of the probabilistic learning task, participants were asked to learn the numerical value of six different visual cues, which ranged on a numeric scale from 1-9 and could be categorized as either higher or lower than 5. There were five phases of the task: learning phase (Figure 4.1A), test phase 1 (Figure 4.1B), update phase (Figure 4.1C), test phase 2 (Figure 4.1D), and a follow-up test (not shown) administered on a subsequent day in the laboratory. The same two independent variables were manipulated as those in experiment 1, with some modifications: learning type (feedback and observation) and cue difficulty (easy and hard, with the addition of a new type which was deterministic). In the learning phase, participants acquired the value of six cues in only one manner, via either feedback or observation. Those who completed the feedback learning session are referred to as group1/feedback group and those who completed the observation learning session are referred to as group2/observation group. In the update phase, all participants learned the new cue values via both feedback and observation trials (half of the cues via feedback; half of the cues via observation). In terms of cue difficulty, there were three types of cues: 100% predictive of the outcome (2 cues referred to as

95 78 deterministic), 87.5% predictive of the outcome (2 cues referred to as easy), and 62.5% predictive of the outcome (2 cues referred to as hard). The learning phase consisted of 2 blocks of 48 trials each, all of which were either feedback or observation (feedback shown in Figure 4.1A). Feedback and observation trial structure and timing in this version of the task were identical to that used in experiment 1 (2 second stimulus/response phase, 2 second feedback period, 2-14 second jittered ITI). All cues in this task were blue in color. As in experiment 1, participants were instructed to optimize their responding, in both the learning and update phases. Following completion of the learning phase, participants advanced to the first test session, which contained the 6 cues learned in the learning phase, as well as two novel cues (Figure 4.1B). There were 32 trials total in the test phase, four presentations of each of the eight cues, presented in random order. The cues from the learning phase were blue, and the novel cues were white in color. Trial structure and timing in this test phase were identical to that used in the test phase from experiment 1 (self-timed stimulus/response phase, no feedback or observation information provided, 6-14 second jittered ITI). Following completion of the first test phase, participants were instructed on the type of learning they had not experienced in the learning phase (feedback learning for the observation group; observation learning for the feedback group). After completion of a short practice session while inside the MRI, they began the update phase. Participants were informed that they were going to continue learning about the value of the cues, now via both types of learning. Participants

96 79 were also instructed that some things may have changed since the first part of the experiment (the learning phase), but they were not given any specific information other than the vague indication that something may be different. The structure of the update phase was very similar to that of the learning phase and consisted of 2 blocks of 48 trials each (Figure 4.1C). However, there were two critical manipulations in the update phase. The first was that half of the trials were feedback (3 cues: 1 deterministic, 1 easy, and 1 hard) and the remaining half were observation (3 cues: 1 deterministic, 1 easy, and 1 hard). The second key manipulation was that all of the cue contingencies switched, (e.g., circle was higher than 5 in the learning phase, but lower than 5 in the update phase); however the probabilities of the cues remained the same (e.g., circle was 100% higher than five in the learning phase and 100% lower than five in the update phase). Trial structure and timing were identical to that used in the learning phase (2 second stimulus/response phase, 2 second feedback period, 2-14 second jittered ITI). Trial type (feedback or observation) and trial order were randomized; therefore feedback and observation trials were intermixed during this phase. Participants were not given any indication of the nature of the upcoming trial type; they simply determined the trial type based on the cue presentation/response period. If it was a feedback trial, they saw only the cue, if it was an observation trial, they saw a cue paired with an arrow. Participants were instructed that the arrow was an informative clue indicating the value of the cue (higher or lower than 5). Following completion of the update phase, a second test phase was administered that was identical to the first test phase (six

97 80 blue cues shown in the update phase and the same 2 white novel cues that were shown in the first test phase; Figure 4.1D). All four phases were completed while the participants remained in the MRI. Lastly, a follow-up behavioral-only test session was administered a couple of days following completion of the MRI portion of the experiment in the laboratory (M = 2.24 days, SD = 1.82 days) Data analysis Behavioral Data Analysis: In order to examine the behavioral accuracy data, a series of ANOVAs were performed. The primary question of interest was to examine accuracy in each phase of the experimental task (learning phase, test 1, update phase, test 2, follow-up test) as it varied according to learning type (feedback and observation for the learning and update phases, with the addition of novel for the test phases) and cue difficulty (deterministic vs. easy vs. hard). It was also a question of interest to examine behavioral performance across groups, to determine if accuracy was significantly greater for one group than the other. In order to do so, group was used as between-subjects factor in all of the analyses presented in the behavioral results section and learning type and cue difficulty were used as within-subjects factors. Details regarding specific ANOVAs performed in each session are presented in the appropriate behavioral results sections. As a note, one subject s data from the first test session was lost; therefore the feedback group test 1 data set consists of 20 subjects. Neuroimaging Data Analysis:

98 81 GLM: In the learning phase participants who completed the feedback and observation learning sessions were analyzed separately in order to examine the engagement of the basal ganglia and medial temporal lobes during feedback learning (group 1) distinct from observation learning (group 2). However, the same analyses were performed on each group s data; specifically two randomeffects GLMs were conducted to examine BOLD activity during the 4-second cue presentation + feedback period using the following predictors: learning type (feedback for the feedback group and observation for the observation group) and cue difficulty (deterministic, easy, and hard). Missed trials, six motion parameters, and the mean intensity BOLD signal were included as regressors of no interest. Mean intensity was included in this experiment only as a regressor of no interest as an examination of the mean intensity of all participants indicated fairly substantial deviations from the mean (M = 5 SDs away from the average intensity signal). Two one-way ANOVAs using cue difficulty as a within-subjects factor were conducted within BrainVoyager. Resulting statistical parametric maps were set at a threshold of p<0.005 and a cluster threshold level of 6 contiguous voxels or 136mm 3 (corrected to a cluster-level false-positive rate of 5%). Functionally defined regions were identified from the resulting maps (group 1/feedback and group 2/observation) and the BOLD signal was extracted for further post-hoc analyses. In the update phase, the comparison of greatest interest was to examine which regions were engaged when updating information via the same (feedbackfeedback for group 1 and observation-observation for group 2) or a different

99 82 (feedback-observation for group 1 and observation-feedback for group 2) type of learning. In order to examine this, all 41 participants were combined into one analysis. One random-effects GLM was performed to examine BOLD activity during the 4-second cue presentation + feedback period using the following predictors: learning type (same and different) and cue difficulty (deterministic, easy, and hard). In addition, missed trials, six motion parameters, and the mean intensity signal were included as regressors of no interest. A 2 (learning type: same vs. different) x 3 (cue difficulty: deterministic vs. easy vs. hard cues) x 2 (group: feedback vs. observation) repeated-measures ANOVA was performed within BrainVoyager, using learning type and cue difficulty as within-subjects factors and group as a between-subjects factor. Based on this ANOVA a main effect of learning type, main effect cue difficulty, main effect of group, as well as interactions were examined within BrainVoyager. SPMs were generated and corrected to a cluster-level false-positive rate of 5% which was calculated individually for each main effect and interaction: 1) main effect of learning type: p<0.005, cluster threshold = 6 or 137mm 3 ; 2) main effect of cue difficulty: p<0.005, cluster threshold = 5 or 129mm 3 ; 3) main effect of group: p<0.005, cluster threshold = 6 or 136mm 3 ; 4) interaction of learning type x group: p< , and an assigned cluster threshold of 10 (the cluster threshold plugin was unable to run as the p value was so significant; therefore a cluster threshold of 10 was chosen by the experimenter). Functional ROIs were identified and the mean parameter estimates were extracted for further post-hoc comparisons.

100 83 For the test phases, BOLD activity was examined across the entire trial (stimulus onset and participant response). Four random-effects GLMs were performed, one per test session per group. All GLMs included learning type (which varied based on the analysis and is described below) and cue difficulty (deterministic, easy, and hard) as factors: 1) feedback group test 1 - learning material (feedback and novel) and cue difficulty; 2) observation group test 1 - learning material (observation and novel) and cue difficulty; 3) feedback group test 2 - learning material (feedback, observation, and novel) and cue difficulty; 4) observation group test 2 - learning material (feedback, observation, and novel) and cue difficulty. SPMs were generated and in every test session a contrast of studied versus non-studied (novel) cues was performed. Mean parameter estimates were extracted from resulting functional ROIs and used for further post-hoc analyses. GCA: A granger causality analysis was performed in the update phase using the midbrain as the seed region. The midbrain was chosen as the principal seed region in order to examine potential functional and effective connectivity between dopaminergic regions and the BG and MTL (Shohamy and Adcock, 2010). Previously published midbrain regions (Adcock et al., 2006), encompassing the bilateral ventral tegmental area were used as the seed regions (x, y, z = -4, -15, -9 and x, y, z = 5, -14, -8). Functional and effective connectivity maps were generated at the individual subject level for both runs of the update phase. Time course values over the entire run, from the first to last time point (290 volumes per run) were entered into the analysis. At the multi-

101 84 subject level, each participant s map was combined to make one group map (which contained both feedback and observation trial information, as the trial types were intermixed in this phase). For the feedback group, a technical error prevented two subjects data from being included; therefore, the feedback group consisted of 19 subjects in the GCA analysis. For this group, functional connectivity maps were given a threshold of p< with a cluster threshold correction of 3 voxels (cluster-level false-positive rate of 5%) and observation group maps were thresholded at p< and a cluster threshold correction of 3 voxels (cluster-level false-positive rate of 5%). Effective connectivity maps were thresholded at p<0.05 for both groups and corrected to a cluster-level falsepositive rate of 5% (FB group: right VTA cluster threshold = 37 or 998mm 3 and left VTA cluster threshold = 38 or 1005mm 3 ; for the OB group: right VTA cluster threshold = 39 or 998mm 3 and left VTA cluster threshold = 37 or 965mm 3 ). Standard second level statistics were performed on the group maps; specifically a t-test was calculated. PE: Four GLMs were created for the prediction error analyses: 1) feedback group learning phase; 2) observation group learning phase; 3) feedback group update phase; 4) observation group update phase. The prediction error values generated from the Q learning model presented in the general methods chapter were used as a regressor for the feedback trials in GLM 1 (feedback learning phase). The action prediction error values generated for the observation trials were used as a regressor for the observation trials in GLM 2 (observation learning phase). In GLM 3 and 4, feedback prediction errors and

102 85 action prediction errors were generated for the feedback update phase trials and the observation update phase trials respectively. In GLMs 3 and 4, one regressor contained the feedback prediction error signal values (feedback trials) and a separate regressor in the same model contained the action prediction error signal values (observation trials). GLM 3 contained both of these regressors for group1/feedback group, while GLM 4 contained both of these regressors for group2/observation group. In addition, in all four of the GLMs, eight additional predictors were included: trial event and missed responses, as well as six motion parameters. Within GLMs 1, 3, and 4, for the feedback trials, the feedback prediction error values were coded during the two second feedback presentation period (indicating a correct, incorrect, or missed trial). In GLMs, 2, 3, and 4, for the observation trials, the action prediction error values were coded during the two second cue presentation period (where the cue and informative arrow, indicating the cue s value, were presented simultaneously). Four SPMs which probed regions of the brain that varied parametrically with either feedback prediction error signals or action prediction error signals (depending on the GLM) were generated, set to a threshold of p<0.005 with an appropriate voxel contiguity that corrected to a cluster-level false-positive rate of 5% (FB group: learning phase cluster threshold = 6 or 138 mm 3 ; update phase cluster threshold = 5 or 136 mm 3 for both the feedback and observation trial models; OB group: learning phase cluster threshold = 5 or 135 mm 3 ; update phase cluster threshold = 5 or 131 mm 3 for both the feedback and observation trial models).

103 Results Behavioral results Learning phase: Accuracy The analysis of interest in the learning phase was to investigate how participants accuracy varied according to cue difficulty (deterministic vs. easy vs. hard) across groups (group1/feedback vs. group2/observation). In order to address this question a 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: feedback vs. observation) repeated measures ANOVA was conducted using cue difficulty as a within-subjects measure and group as a betweensubjects measure. A main effect of cue difficulty (F (1.730,67.49) = 77.13; p < ), a main effect of group (F (1,39) = 18.29; p < ), and a marginally significant interaction of cue difficulty and group (F (1.730,67.49) = 77.13; p = 0.085) were observed. Post-hoc t-tests were conducted to probe the main effect of cue difficulty and main effect of group results. As expected, behavioral accuracy scaled according to difficulty: deterministic > easy > hard (paired-samples t-tests, collapsed across groups: deterministic > easy t (40) = 4.26; p < 0.001; easy > hard t (40) = 7.59; p < 0.001). An independent samples, post-hoc t-test was performed to examine the main effect of group and indicated that participants in the observation group (mean accuracy = 85%) performed significantly better than participants in the feedback group (mean accuracy = 75%; t (27.12) = 4.36; p < 0.001). In order to examine the marginally significant interaction of cue difficulty by group, independent samples t-tests were performed and indicated that participants in the observation group performed significantly better than

104 87 participants in the feedback group for both deterministic and easy cues (t (28.61) = 4.89; p < and t (26.87) = 4.20; p < 0.001, respectively), with no performance difference exhibited for the hard cues (t (39) = 0.94; p > 0.05; Figure 4.2A and B). This result suggests that the two learning types are perhaps best equated in the hardest cue condition (62.5% probability). Test 1: Accuracy In order to confirm that participants acquired all cue contingencies successfully in the learning phase, a test phase was administered which contained the probabilistic cues that were acquired in the learning phase, now presented without any cue value information (e.g., no feedback in the trials for group1/feedback group; no observational cue information in the trials for group2/observation group). Additionally, novel information was presented in the test phase as a control in order to compare accuracy for previously learned with novel information presented for the first time in the test session. Within each group, a paired-samples t-test was performed to compare accuracy for previously learned information with novel information (collapsed across levels of cue difficulty). In the feedback group a t-test of feedback versus novel information was performed and for the observation group, a t-test of observation versus novel information was conducted. Participants in both groups performed significantly better on previously learned information compared to novel material (FB group: feedback > novel t (19) = 3.94; p < 0.001; OB group: observation > novel t (19) = 4.95; p < 0.001).

105 88 In addition to examining differences within each group comparing previously studied with novel information, potential differences in the test phase among cue difficulty were also investigated. This question was examined across groups to determine if participants in one group were more accurate than the other group in the test phase. A 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: feedback vs. observation) repeated measures ANOVA was conducted using cue difficulty as a within-subjects measure and group as a between-subjects measure. A main effect of cue difficulty (F (1.609,61.15) = 14.60; p < 0.001), no main effect of group (F (1,38) = 1.65; p > 0.05), and no significant interaction between cue difficulty and group (F (1.609,61.15) = 1.40; p > 0.05) were observed. Post-hoc paired-samples t-tests were performed to explore the main effect of cue difficulty result, collapsed across group. The results indicated that accuracy nearly scaled according to difficulty with marginally better performance for deterministic compared with easy cues (t (39) = 1.80; p = 0.08), and greater accuracy for both deterministic and easy cues compared with hard cues (t (39) = 5.16; p < and t (39) = 3.20; p < respectively). To summarize, in the first test phase, participants accuracy nearly scaled according to cue difficulty, with no accuracy differences observed across the feedback and observation groups, indicating that both groups acquired initial cue contingencies well. Update phase: Accuracy Two analyses were performed examining participants behavioral accuracy during the update (reversal) phase. The first probed potential differences across different types of learning (feedback vs. observation trials) as

106 89 well as the various levels of cue difficulty (deterministic vs. easy vs. hard) across groups (group1/feedback vs. group2/observation). Therefore, a 2 (learning type: feedback vs. observation) x 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: feedback vs. observation) repeated-measures ANOVA, using learning type and cue difficulty as within-subjects factors and group as a betweensubjects factor was conducted. A main effect of learning type (F (1,39) = 5.19; p < 0.05), a main effect of cue difficulty (F (2,78) = 31.81; p < 0.001), no main effect of group (F (1,39) = 0.226; p > 0.05), and no significant interactions (all p values > 0.05) were observed. A post-hoc paired-samples t-test was performed to examine the main effect of learning type (collapsed across group), with results indicating greater accuracy for observation compared with feedback cues (t (40) = 2.31; p < 0.05). The main effect of cue difficulty was caused by significantly greater accuracy for deterministic and easy compared with hard cues (pairedsamples t-tests: t (40) = 6.93; p < and t (40) = 5.73; p < respectively), with no significant difference observed between deterministic and easy cue performance (t (40) = 1.35; p > 0.05). To summarize, participants were more accurate for observation than feedback cues in the update phase, as occurred in the learning phase, with accuracy modulated by difficulty such that participants performed better on the easier cues (deterministic and easy cues) compared with the hard cues. Because there was a reversal in cue contingencies in the beginning of the update phase, a second analysis examined changes in accuracy at multiple time points throughout this phase. It was expected that participants performance

107 90 would reflect this reversal by exhibiting initial poor accuracy, followed by increases in performance as participants updated previously learned contingencies over time. In order to do this, trials were binned into 12-trial increments and participants accuracy for those 12 trials was averaged. Mean accuracy was examined at trials: 1-12 (time point 1), (time point 2), (time point 3), (time point 4) for block 1 and the same time points: 1-12, 13-24, 25-36, for block 2 (which can also be thought of as trials 49-60, 61-72, 73-84, and of the task). To examine behavioral accuracy changes across the aforementioned time points, a 2 (learning type: feedback vs. observation trials) x 4 [time points: 1 (trials 1-12) vs. 2 (13-24) vs. 3 (25-36) vs. 4 (37-48)] x 2 (group: group1/feedback vs. group2/observation) x 2 (block: early vs. late) repeated-measures ANOVA was performed using learning type, time point, and block as within-subjects factors and group as a between-subjects factor. Results revealed a significant main effect of learning type (F (1,39) = 8.74; p < 0.01), a main effect of time point (F (3,117) = 27.70; p < 0.001), a main effect of block (F (1,39) = 33.70; p < 0.001), and no main effect of group (F (1,39) = 0.35; p > 0.05; Figure 4.2C and D). Significant interactions included an interaction of learning type by time point (F (3,117) = 7.66; p < 0.001), learning type by block (F (1,39) = 13.21; p < 0.005), time point by block (F (3,117) = 8.91; p < 0.001), and a three way interaction of learning type by time point by block (F (3,117) = 6.49; p < 0.001), with no other significant interactions observed (all p values > 0.05). Post-hoc t-tests were completed to examine the significant main effects and interactions. The main effect of learning type

108 91 indicated greater accuracy for observation compared with feedback trials (t (40) = 2.31; p < 0.05) as previously reported in the first ANOVA performed in the update phase. The main effect of time point revealed that as expected, participants accuracy increased across trial time points, time points 4 (81%) > 3 (76%) > 2 (72%) > 1 (67%) [time point 4 > time point 3 (t (40) = 2.59; p < 0.05); time point 3 > time point 2 (t (40) = 2.67; p < 0.05); time point 2 > time point 1 (t (40) = 2.67; p < 0.05)]. As expected, the main effect of block was driven by greater accuracy in the second (80%) compared with the first block (68%; t (40) = 5.96; p < 0.001). The interaction of learning type by time point was driven by an increase in accuracy across time points for the feedback trials [time points 4 (81%) > 3 (73%) (t (40) = 3.04; p < 0.005); time point 3 (73%) marginally > 2 (68%) (t (40) = 1.80; p = 0.08); time point 2 (68%) > 1 (59%) (t (40) = 3.28; p < 0.005)], but not for the observation trials (all p values > 0.05). The significant interaction of learning type by block was caused by significantly greater accuracy for observation (74%) than feedback (62%) cues in the first block (t (40) = 3.37; p < 0.005) with no difference between learning types observed in the second block (FB = 79%; OB = 81%; t (40) = 0.57; p > 0.05). This was most likely due to the fact that participants significantly increased their accuracy for the feedback trials throughout the course of the first block in the update phase (as seen in the learning type x time point x block interaction). The significant interaction of time point by block was driven by increases across time points in block 1 [time point 4 (79%) marginally greater than 3 (73%) (t (40) = 1.78; p = 0.08); time point 3 (73%) > 2 (66%) (t (40) = 2.80; p < 0.01); time point 2 (66%) > 1 (55%) (t (40) = 3.26; p < 0.005)], but no

109 92 significant increases in accuracy across time points in block 2 [only marginally greater accuracy for time point 4 (84%) > (79%) (t (40) = 1.85; p = 0.07), all other p values > 0.05], suggesting that participants accuracy was stabilized in the second block of the update phase, demonstrating that participants updated cue contingencies successfully by the end of the first block. Lastly, the three way interaction of learning type by time point by block was driven by a significant increase in only feedback trial accuracy in the first block only [time point 4 (79%) > 3 (69%) (t (40) = 2.38; p < 0.05); time point 3 (69%) > 2 (59%) (t (40) = 2.63; p < 0.05); time point 2 (59%) > 1 (38%) (t (40) = 4.43; p < 0.001)], with no significant changes in the second block for feedback trials (all p values > 0.05), and no significant changes in either block for the observation trials (all p values > 0.05; Figure 4.2 C and D). To summarize, these results suggest that participants in both groups were initially worse at specifically feedback cues following probabilistic cue value reversal, but rapidly updated their responses such that by the end of the first update block, there was no significant difference between feedback and observation cue performance. Interestingly, neither a significant main effect of group nor any significant interactions with group were observed, suggesting that both groups updated cue contingencies in a similar fashion. Test 2: Accuracy The second test phase was administered to determine how well participants learned the new cue contingencies, after the reversal in the update phase. Critically, as in all test sessions, no informative information about the cue was provided (no feedback for the feedback trials; no observational information

110 93 for the observation trials). In this test, feedback, observation, and novel trials were administered for both group 1 and group 2. The first question of interest examined if participants in both groups performed better on information learned in the update phase compared with novel information and compared accuracy across the two groups to determine if one group performed better than the other in this test session. To address this question, a 3 (learning material: feedback vs. observation vs. novel) x 2 (group: feedback vs. observation) repeatedmeasures ANOVA was conducted using learning material as a within-subjects factor and group as a between-subjects factor. A main effect of learning material (F (1.67,65.10) = 26.28; p < 0.001), no main effect of group (F (1,39) = 0.003; p > 0.05), and no significant interaction between learning material and group (F (1.67,65.10) = 0.42; p > 0.05) were observed. Post-hoc paired-samples t-tests were performed to examine the main effect of learning material, collapsed across group. As expected, results indicated no accuracy differences between previously learned material in the update phase (feedback vs. observation trials; t (40) = 1.36; p > 0.05), and greater accuracy for both feedback and observation cues compared to novel cues (t (40) = 6.28; p < and t (40) = 5.13; p < 0.001, respectively). The second analysis probed for potential differences between levels of cue difficulty and learning types, across groups, excluding novel information. In order to do so, a 2 (learning type: feedback vs. observation) x 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: group1/feedback vs. group2/observation) repeated-measures ANOVA was conducted using learning type and cue difficulty as within-subjects factors and group as a between-

111 94 subjects factor. No significant results were observed (no main effect of learning type (F (1,39) = 1.80; p > 0.05); no main effect of cue difficulty (F (1.55,60.57) = 1.31; p > 0.05); no main effect of group (F (1,39) = 0.007; p > 0.05); and no significant interactions - all p values greater than 0.05). These results suggest that irrespective of group or learning type, participants updated cue contingencies well. Follow-up Test: Accuracy A final test was conducted behaviorally in the laboratory to examine changes in accuracy approximately 2 days after participation in the fmri portion of the experiment. The same two analyses were performed to examine potential performance differences in this phase as were performed in the second test phase administered inside the MRI. First, to examine potential differences within learning material across groups, a 3 (learning material: feedback vs. observation vs. novel) x 2 (group: group1/feedback vs. group2/observation) repeatedmeasures ANOVA was conducted using learning material as a within-subjects factor and group as a between-subjects factor. A main effect of learning material (F (1.67,64.97) = 15.33; p < 0.001), no main effect of group (F (1,39) = 0.219; p > 0.05), and no significant interaction between learning material and group (F (1.67,64.97) = 1.16; p > 0.05) were observed. Post-hoc paired-samples t-tests probing the main effect of learning material (collapsed across group) indicated no difference between previously learned material (feedback vs. observation cues; t (40) = 1.00; p > 0.05) and significantly greater accuracy for both feedback and observation cues compared to novel cues (t (40) = 4.96; p < and t (40) = 6.51; p < 0.001,

112 95 respectively) as was expected based on the results from the previous test phases. Second, to probe for potential cue difficulty differences across groups in this follow-up test phase, a 2 (learning type: feedback vs. observation) x 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: group1/feedback vs. group2/observation) repeated-measures ANOVA was conducted using learning type and cue difficulty as within-subjects factors and group as a betweensubjects factor. As anticipated, based on the results from the second test administered on day 1, there were no significant findings. Specifically, no main effect of learning type (F (1,39) = 1.34; p > 0.05); no main effect of cue difficulty (F (2,78) = 0.376; p > 0.05); no main effect of group (F (1,39) = 1.05; p > 0.05); and no significant interactions (all p values greater than 0.05) were observed. These results therefore confirm the results of the second test administered on day 1, suggesting that participants updated cue contingencies well, irrespective of how they initially acquired the cue associations (the group they were assigned to: group1/feedback vs. group2/observation), the trial learning type, or the difficulty of the probabilistic cues Neuroimaging results Learning phase Main Effect of Cue Difficulty: One unresolved question from the results of experiment 1 was to determine if the MTL and BG are engaged in pure feedback learning as well as pure observation learning (nonbiased by both learning types presented in the same learning session). In order to examine the engagement of

113 96 these areas during distinct types of learning, the feedback groups and observation groups neuroimaging data were analyzed separately. Therefore, two independent within-subjects, repeated-measures ANOVAs investigating a main effect of cue difficulty were performed within BrainVoyager: one for the feedback learning participants (group 1) and a separate ANOVA for the observation learning participants (group 2). Results indicated that a network of similar regions was engaged in both the feedback learning group (Table 4.1) and observation learning group (Table 4.2), including loci within the striatum, medial temporal lobe, insula, cingulate cortex, and prefrontal areas. Feedback learning group: A region of the right caudate nucleus (x, y, z = 5, 1, 12; Figure 4.3A) and left medial temporal lobe, encompassing the amygdala and hippocampus (x, y, z = -22, -8, -15; Figure 4.3C) were engaged during feedback learning. Mean parameter estimates were extracted from these regions of interest for further analysis. Activity within the caudate nucleus was greatest in response to hard cues and decreased according to cue difficulty (hard > easy > deterministic) (hard cues marginally greater than easy cues, t (20) = 2.90; p = 0.07; easy > deterministic t (20) = 4.30; p < ; Figure 4.3B), while the left MTL showed the opposite pattern (deterministic > easy > hard) (deterministic marginally greater than easy cues, t (20) = 1.96; p = 0.06; easy > hard t (20) = 3.10; p < 0.01; Figure 4.3D). Observation learning group: Within the observation group, a region of the right caudate nucleus (x, y, z = 5, -2, 15), the left caudate nucleus extending into the left putamen (x, y, z = -13, 1, 9) (Figure 4.4 A), and left hippocampus (x, y, z

114 97 = -25, -20, -12; Figure 4.4C) were engaged during learning. Mean parameter estimates were extracted from these regions for further examination. A similar pattern of activity appeared within the right caudate nucleus, left caudate nucleus extending into the putamen, and the hippocampus as was observed in the feedback learning group. Specifically in the right caudate nucleus, hard cues elicited stronger activation compared with easy and deterministic cues (t (19) = 4.77; p < and t (19) = 4.26; p < , respectively; Figure 4.4B), with a similar pattern observed in the left caudate/putamen region (hard > easy and deterministic; t (19) = 4.11; p < and t (19) = 4.55; p < , respectively; not depicted in the figure). The left hippocampus showed the opposite pattern with deterministic and easy cues eliciting stronger activation than hard cues (t (19) = 3.62; p < and t (19) = 4.28; p < , respectively; Figure 4.4D). These results therefore are a partial replication of the results observed in experiment 1. Loci within both the caudate nucleus and hippocampus were involved in feedback learning, as well as observation learning and were modulated by cue difficulty. The hippocampus exhibited the same pattern of results as reported in experiment 1, however the caudate nucleus, was engaged differentially in the learning task employed in experiment 1 versus the task employed in experiment 2; an interpretation of this differentiation is provided in the discussion section. Learning phase: Correlations between striatum and medial temporal lobe As a rudimentary manner of examining interactions between the striatum and medial temporal lobe during learning, a series of Pearson s correlations were

115 98 performed on mean parameter estimates extracted from the striatal and medial temporal lobe regions in the feedback and observation learning groups. Specifically, the areas functionally defined by the main effect of cue difficulty analyses were used in the analyses. Feedback learning group: In the feedback group, no significant positive or negative correlations were observed between the mean BOLD signal extracted from the right caudate nucleus (x, y, z = 5, 1, 12) and left medial temporal lobe (x, y, z = -22, -8, -15). Observation learning group: In the observation group, a significant positive correlation was observed between the right caudate nucleus (x, y, z = 5, -2, 15) and the left MTL (x, y, z = -25, -20, -12) for observation deterministic cues (r = 0.467, p = 0.04) with a nearly significant positive correlation observed between these same regions for easy cues (r = 0.434, p = 0.056). No other significant positive or negative correlations were observed. Update phase The key question of interest in the update phases was to examine what brain regions were engaged during the updating of probabilistic learning, when learning types remained the same (feedback-feedback and observationobservation) versus when learning types evolved (feedback-observation and observation-feedback). It was hypothesized that this novel manner of examining how multiple memory systems interact during a dynamic learning environment may provide new insight to the debate regarding the nature of interactions among memory regions. In order to address this question of interest, one analysis was

116 99 performed which included all forty-one participants data combining groups 1 (feedback) and 2 (observation). A 2 (learning type: same vs. different) x 3 (cue difficulty: deterministic vs. easy vs. hard) x 2 (group: group1/feedback vs. group2/observation) repeated-measures ANOVA was performed within BrainVoyager using learning type and cue difficulty as within-subjects measures and group as a between-subjects measure. The results presented below highlight the most relevant findings. Main Effect of Cue Difficulty: A similar network of regions engaged in processing differences in cue difficulty was observed during the update phase as was engaged during the learning phases, including loci within the striatum, MTL, insula, cingulate cortex, as well as several frontal and parietal regions (Table 4.5). Mean parameter estimates were extracted from key areas of interest, specifically two regions bordering the right putamen/insula (x, y, z = 26, 13, 3 and x, y, z = 29, -14, 12; Figure 4.5A and C) and bilateral parahippocampus (x, y, z = 20, -20, -18, not shown in the figure, and x, y, z = -22, -14, -21; Figure 4.5E). Interestingly, the mean BOLD signal in the more anterior region of the lateral putamen/insula (x, y, z = 26, 13, 3) scaled according to difficulty with the greatest response elicited for hard cues (irrespective of learning type): hard > easy > deterministic (hard > easy t (40) = 3.12; p < 0.005; easy > deterministic t (40) = 2.13; p < 0.05; Figure 4.5B). However, the more lateral and posterior region of the putamen/insula identified (x, y, z = 29, -14, 12) showed the opposing pattern of activity, with the greatest response elicited to deterministic cues (irrespective of learning type): deterministic > easy > hard (deterministic > easy t (40) = 2.28; p <

117 ; easy > hard t (40) = 2.39; p < 0.05; Figure 4.5D). Furthermore, within the parahippocampus, the BOLD signal also scaled according to difficulty with both regions showing the greatest response to deterministic cues: deterministic > easy > hard (irrespective of learning type; right parahippocampus: deterministic > easy: t (40) = 2.30; p < 0.05; easy > hard: t (40) = 2.72; p < 0.01; left parahippocampus: deterministic marginally > easy: t (40) = 1.85; p = 0.07; easy > hard: t (40) = 2.67; p < 0.05; Figure 4.5F). This result of both similar and opposing patterns of BOLD responses within regions of the putamen and parahippocampus when updating cue contingencies following a reversal in cue values supports parallel engagement of these regions during the updating process. Main Effect of Learning Type: Brain regions which processed differences in learning type, that being variations in BOLD responses to the same versus a different learning type as was employed in the learning phase, were also investigated. A group of areas involved in processing differences in learning type included a locus within the cingulate cortex (x, y, z = -10, -5, 36), a locus within the insula (x, y, z = 41, 7, 6), as well as several parietal and temporal regions (Table 4.6). Interestingly, no areas within the medial temporal lobes or basal ganglia exhibited a main effect of learning type, indicating that these regions did not differentiate between whether the learning type remained the same or changed compared with initial cue acquisition. Main Effect of Group: Areas processing differential activity for participants in the feedback group versus those in the observation group were also

118 101 examined. The main effect of group analysis revealed activation within two areas which survived correction, the precentral gyrus and the thalamus (Table 4.7). Interestingly, no engagement of the basal ganglia, MTL, or midbrain was observed in processing a main effect of group. While null findings in neuroimaging studies are difficulty to interpret, and must be done with caution, one possible interpretation is that this null result may be indicative of a parallel mode of operations between multiple memory regions, as no key region of interest was engaged in processing greater activation to the feedback versus the observation group, or vice versa. Interaction of Learning Type and Group: Several regions exhibited an interaction of learning type and group, including loci within the striatum and midbrain, as well as the insula, and cingulate cortex. Specifically the bilateral caudate nucleus (right: x, y, z = 8, 3, 11; and left x, y, z = -8, 3, 8) as well as the bilateral midbrain, centered on the VTA (right: x, y, z = 7, -15, -9; and left: x, y, z = -4, -15, -9) exhibited an interaction of learning type by group. Mean parameter estimates were extracted from these regions of interest for further examination. In all regions examined, the interaction was driven by a larger response to same than different learning trials for the feedback group (e.g., feedback trials), and a larger response to different than same learning trials for the observation group (e.g., feedback trials) [right caudate: FB group: same > different t (20) = 6.85; p < ; OB group: different > same: t (19) = 8.34; p < ; left caudate: FB group: same > different t (20) = 8.77; p < ; OB group: different > same: t (19) = 5.14; p < ; right midbrain: FB group: same > different t (20) = 8.28; p <

119 ; OB group: different > same: t (19) = 4.95; p < ; left midbrain: FB group: same > different t (20) = 7.90; p < ; OB group: different > same: t (19) = 5.92; p < ]. Therefore, within both the feedback and observation groups, a greater response to feedback trials was observed in loci within the caudate nucleus and midbrain. This result is consistent with the literature suggesting a role for the striatum and midbrain in processing feedback information (Packard and Knowlton, 2002; Wise, 2004; Delgado, 2007). Update phase: Correlations between striatum and medial temporal lobe Correlations were performed on mean parameter estimates extracted from striatal and medial temporal lobe regions in the update phase, functionally defined by the main effect of cue difficulty analysis. Only correlations across regions, which were matched according to cue difficulty (deterministicdeterministic, easy-easy, hard-hard) were examined. The only significant acrossregion correlation observed was between the right parahippocampus (x, y, z = 20, -20, -18) and right putamen/insula (x, y, z = 29, -14, 12) for deterministic cues in the same learning condition (r = 0.308, p = 0.050). Granger Causality Analysis Granger causality analyses were conducted for the feedback and observation groups separately during the update phase. The purpose of this analysis was to examine functional and effective connectivity during the dynamic portion of the task when participants were updating previous cue contingencies via both feedback and observation trials. The midbrain (centered on the VTA) was chosen as the seen region given its anatomical projections to both the

120 103 striatum and the hippocampus and its critical role in reward-related learning and memory processes (Figure 4.6A). 1) Functional connectivity: Feedback group: Within the feedback group, using the right VTA as the seed region resulted in functional connectivity with the bilateral caudate nucleus extending into the putamen (x, y, z = 6, 4, 6 and x, y, z = -17, 1, 5) as well as the right hippocampus (x, y, z = 24, -20, -6). Functional connectivity maps using the left VTA seed revealed very similar instantaneous correlations with the bilateral caudate nucleus extending into the putamen/globus pallidus (x, y, z = 8, 1, 12 and x, y, z = -12, -2, 4) and the bilateral hippocampus (x, y, z = 24, -22, -8 and x, y, z = -32, -15, -13). Observation group: Functional connectivity in the observation group using the right and left VTA seeds occurred with similar regions of the basal ganglia, namely loci within the bilateral globus pallidus. Two loci of the globus pallidus, (x, y, z = 8, 1, -1 and x, y, z = -15, 1, -2) correlated with the right VTA, while two similar areas of the globus pallidus (x, y, z = -8, 1, -3 and x, y, z = 18, -2, -6) correlated with the left VTA. In addition, the right VTA exhibited functional connectivity with the bilateral parahippocampus (x, y, z = 16, -23, -12 and x, y, z = -19, -28, -9) while the left VTA exhibited connectivity with the bilateral posterior entorhinal cortex (x, y, z = 17, -23, -9 and x, y, z = -20, -25, -9). To summarize the functional connectivity results, instantaneous correlations were observed between the bilateral VTA, the caudate nucleus extending into the globus pallidus, and the hippocampus in the feedback group

121 104 while somewhat overlapping regions in the globus pallidus and medial temporal lobes were functionally connected with the bilateral VTA in the observation group. Therefore, similar functional connectivity in known reward-related and memory regions (midbrain, basal ganglia, and medial temporal lobes) was observed across groups, suggesting interactions between these areas during the updating of previously learned cue contingencies. 2) Effective connectivity: Feedback group: Maps using the left VTA as a seed region revealed directed connectivity originating from the bilateral putamen (x, y, z = 15, 2, 5 and x, y, z = -19, 0, 5) to the left VTA (Figure 4.6B). Using the right VTA seed revealed directed influences from the right VTA to the right hippocampus (x, y, z = 26, -20, -7; Figure 4.6C) and right parahippocampus (x, y, z = 16, -31, -12; not shown). Observation group: Effective connectivity using the left VTA as a reference region indicated no regions of interest surviving cluster threshold correction. Using the right VTA as a reference region revealed directed influences from the right VTA going to the left hippocampus (x, y, z = -30, -21, - 9), the left parahippocampal gyrus (x, y, z = -25, -29, -9) (Figure 4.6D), as well as the posterior cingulate and thalamus. In sum, a unilateral flow of information was observed in which directed influences from the putamen were sent to the left VTA, and directed influences from the right VTA were sent to the hippocampus and parahippocampus in the feedback group. The observation group exhibited a segment of this arc, that

122 105 from the right VTA to the left hippocampus and parahippocampus [a region of the left putamen (x, y, z = -24, -3, 6) did send directed influences to the left VTA, but did not survive cluster threshold correction]. Prediction Error Analysis Learning phase: Feedback learning group: A traditional Q learning model was used to estimate prediction error values for the feedback trials in group 1 (feedback group). Regions that positively correlated with this prediction error signal included three loci within the basal ganglia, as well as an area within the anterior cingulate (x, y, z = -1, 34, -6) and the inferior frontal gyrus (x, y, z = -49, 25, 15). Within the basal ganglia, the ventral striatum (nucleus accumbens: x, y, z = 5, 4, -3), the internal segment of the globus pallidus (x, y, z = -13, 1, -3) (Figure 4.7A), and the head of the caudate nucleus (x, y, z = 14, 22, 9; not shown) all positively correlated with the prediction error signal. Observation learning group: An action prediction error, which is a nonsigned measure of participants surprise at viewing a given cue-value association (saliency signal), was calculated for the observation learning trials for group 2 (observation group). Areas of interest which positively correlated with this action prediction error signal included the medial frontal gyrus (x, y, z = 5, -2, 48), the midbrain (x, y, z = 2, -26, -6), and the bilateral insula (x, y, z = 29, 16, 0 and x, y, z = -28, 16, 6) (not depicted in the figure). To summarize the learning phase prediction error results, areas of the basal ganglia were involved in encoding feedback prediction errors, including the

123 106 ventral striatum, globus pallidus, and caudate nucleus, consistent with previously reported results in the literature (Ashby et al., 2002; O'Doherty et al., 2004; Tanaka et al., 2004; Pessiglione et al., 2006). For the observation trials, the action prediction error signal engaged distinct regions including the medial frontal gyrus, the midbrain, and the insula, which did not replicate previous findings in the literature (Burke et al., 2010). Update phase: Both feedback prediction errors (for the feedback trials) and action prediction errors (for the observation trials) were examined in the update phase, for both group 1 (feedback) and group 2 (observation). Feedback group: Areas of interest which positively correlated with the prediction error signal for the feedback trials included the right ventral caudate nucleus (x, y, z = 2, 10, 0) and the external component of the left globus pallidus (x, y, z = -19, -2, -6), regions fairly similar to those engaged during the learning phase while encoding the prediction error signal for this group (Figure 4.7B). For the observation trials, an area of interest that positively correlated with the action prediction error signal included the superior frontal gyrus (x, y, z = -22, 13, 48) a region distinct from the medial frontal gyrus which encoded this signal in the observation group leaning phase (not depicted in the figure). Observation group: Loci within the ventral striatum (bilateral nucleus accumbens: x, y, z = 5, 4, -3 and x, y, z = -10, 4, -3; Figure 4.7C), the medial temporal lobe (encompassing the hippocampus/amygdala: x, y, z = -25, -5, -21; Figure 4.7D), and the medial frontal gyrus (x, y, z = -1, 22, -15; not shown) positively correlated with the prediction error signal for the feedback trials. For

124 107 the observation trials, the cingulate gyrus (x, y, z = 11, -29, 36) was the only area that positively correlated with the action prediction error signal (not shown in the figure). To summarize, in the update phase, regions of the basal ganglia correlated with feedback prediction errors in both groups, with the observation group additionally showing correlations with the medial temporal lobe and medial frontal gyrus. The action prediction error signal correlated with distinct regions in the feedback (superior frontal gyrus) and observation groups (cingulate gyrus) Exploratory analyses Two additional analyses were performed for exploratory purposes in order to examine potential effects of cue accuracy and cue consistency on the BOLD responses in the BG and MTL, as well as to probe for potential differences in these regions for participants who learned to optimize their responses versus those who did not. The first analysis investigated potential differences in the BG and MTL when accounting for cue accuracy (correct versus incorrect) in the feedback trials and cue consistency (consistent versus inconsistent) in the observation trials. This analysis was performed as the BG and MTL are known to be modulated by both accuracy and consistency during learning (Delgado et al., 2000; Delgado, 2007; Chen et al., 2011; Li et al., 2011). For each region described below, BOLD responses were extracted from the previously defined functional regions of interest processing a main effect of cue difficulty (FB group: right caudate nucleus and left MTL). A within-subjects ANOVA was performed examining potential differences in cue difficulty and cue accuracy for the

125 108 feedback trials in each ROI. To summarize the results briefly, no significant main effects of cue accuracy or significant interactions were observed (right caudate nucleus: main effect of cue difficulty F (2,38) = 11.69; p < 0.001; marginally significant main effect of cue accuracy F (1,19) = 3.19; p = 0.09; no significant interaction F (2,38) = 0.29; p > 0.05; left MTL: main effect of cue difficulty F (2,38) = 9.10; p < 0.005; no main effect of cue accuracy F (1,19) = 0.55; p > 0.05; no significant interaction F (2,38) = 0.43; p > 0.05). However, for exploratory purposes, Student s t-tests were performed in order to examine potential differences across levels of difficulty according to accuracy (corrected for multiple comparisons) as it was desired to examine if the responses in the BG and MTL were driven by responses to correct rather than incorrect trials (e.g., easy correct > hard correct, but no difference for incorrect trials). Note one person was excluded from this analysis as he/she experienced no incorrect trials in the deterministic condition. In the learning phase, for the feedback group, the right caudate nucleus and the left MTL exhibited differences in cue difficulty for both correct and incorrect trials. In the right caudate nucleus, BOLD response for hard correct trials was greater than deterministic correct trials (t (19) = 2.89; p < 0.01) while a marginally greater response for hard than easy correct trials (t (19) = 1.88; p = 0.08) was observed. The same pattern was observed for incorrect trials (hard > deterministic t (19) = 3.06; p < 0.01 and marginally greater for hard than easy t (19) = 2.26; p = 0.04; trend after sequential Bonferroni correction), suggesting that the pattern of activity in the caudate was not driven by one type of feedback alone. The left MTL however, exhibited greater responses for deterministic than hard

126 109 correct trials (t (19) = 3.17; p < 0.005) and marginally greater responses for easy than hard correct trials (t (19) = 1.63; p = 0.12). Marginally greater BOLD signals were observed for deterministic than hard incorrect trials (t (19) = 2.26; p = 0.04; trend after sequential Bonferroni correction) and significantly greater for easy than hard incorrect trials (t (19) = 2.65; p < 0.05). This pattern of results suggests that both the caudate nucleus and MTL were not primarily modulated by correct versus incorrect trials. Furthermore, accounting for accuracy did not change the pattern of results observed in these areas; that is, in the striatum the greatest BOLD response was elicited in response to hard cues, while in the MTL the largest BOLD response occurred for deterministic cues. A similar analysis was performed in the update phase, including both the feedback and observation group as these groups were combined in this phase. Note five people were excluded from this analysis as they experienced no incorrect trials for at least one condition. Four repeated-measures ANOVAs were performed using cue difficulty and feedback cue accuracy as within-subjects measures and group as a between-subjects measure (feedback group and observation group; one ANOVA per ROI). To summarize the results briefly, no significant main effect of cue accuracy was observed (anterior putamen/insula: F (1,34) = 0.58; p > 0.05; posterior putamen/insula F (1,34) = 1.79; p > 0.05; right parahippocampus: F (1,34) = 0.89; p > 0.05; left parahippocampus: F (1,34) = 2.47; p > 0.05). The anterior putamen/insula exhibited a significant three way interaction of cue difficulty, cue accuracy, and group (F (2,68) = 3.60; p < 0.05) and the posterior putamen/insula exhibited a significant interaction of cue difficulty by cue

127 110 accuracy (F (1.66,56.40) = 3.59; p < 0.05). For exploratory purposes, Student s t- tests were performed in order to examine potential differences across levels of difficulty according to accuracy (corrected for multiple comparisons). Interestingly, differences across levels of cue difficulty in these ROIs occurred in correct, but not incorrect trials. The anterior putamen exhibited greater responses for hard than deterministic (t (35) = 3.39; p < ) and marginally greater responses for hard than easy correct trials (t (35) = 1.89; p = 0.07) and easy than deterministic correct trials (t (35) = 2.02; p = 0.05). The more posterior region of the putamen exhibited the opposing pattern of activity, with greater responses for deterministic compared with easy (t (35) = 3.04; p < 0.005) and hard (t (35) = 4.29; p < 0.001) correct trials. In the right parahippocampus marginally greater responses were observed for deterministic compared with hard correct trials (t (35) = 1.82; p = 0.07). In the left parahippocampus, greater BOLD responses were elicited for both deterministic and easy compared hard correct trials (t (35) = 3.41; p < and t (35) = 2.89; p < 0.01 respectively). No differences were observed across levels of cue difficulty for the incorrect trials in any ROI (all p values > 0.05). Therefore, in the update phase, the putamen and parahippocampus exhibited differences in BOLD responses across levels of difficulty in correct but not incorrect trials. However, accounting for cue accuracy did not change the pattern of results observed in these regions. Lastly, the pattern of the putamen and parahippocampal ROIs in the update phase observation trials was examined, when accounting for cue consistency. Note one person was excluded from this analysis as he/she

128 111 experienced no easy inconsistent trials (due to missed trials). Four repeatedmeasures ANOVAs were performed using cue difficulty and cue consistency as within-subjects measures and group as a between-subjects measure (one ANOVA per ROI). To summarize briefly, the putamen ROIs exhibited no effects of cue consistency (anterior putamen/insula: F (1,38) = 1.17; p > 0.05; posterior putamen/insula F (1,38) = 0.29; p > 0.05), while the parahippocampal ROIs were modulated by cue consistency (right parahippocampus: F (1,38) = 11.32; p < and a marginally significant interaction of cue difficulty and cue consistency F (1,38) = 3.96; p = 0.05; left parahippocampus: F (1,38) = 5.13; p < 0.05). The right parahippocampus indicated a main effect of cue consistency, with greater BOLD responses observed for inconsistent (e.g., circle is lower than 5) than consistent trials (e.g., circle is higher than 5 value of circle is 87.5% higher than 5; t (39) = 2.75; p < 0.01) while the left parahippocampus exhibited a marginally greater response for inconsistent trials (t (39) = 1.80; p = 0.08). In both regions, this effect was driven by differences in easy, but not hard, cue consistency (right parahippocampus: easy inconsistent > consistent; t (39) = 3.27; p < 0.005; left parahippocampus: easy inconsistent > consistent; t (39) = 2.80; p < 0.01). The second exploratory analysis probed for potential differences between participants who learned to optimize their responses, termed optimizers, versus those who did not optimize their responses, termed non-optimizers, during the learning and update phases. It was theorized that the responses in the MTL and BG may be modulated by the ability to optimize during the task (e.g., optimizers may exhibit greater MTL activation but reduced BG recruitment compared with

129 112 non-optimizers). In the learning phase, behavioral accuracy was examined in the last 24 trials of the second block of learning (trials 72-96). Participants who were 100% correct for either all the easy cues or all the easy and hard cues presented in the last 24 trials were termed optimizers, whereas participants who were less than 100% correct were categorized as non-optimizers. In the feedback group, 9 participants were categorized as optimizers while the remaining 12 were categorized as non-optimizers. In the observation group, 8 participants were categorized as optimizers while the remaining 12 were categorized as nonoptimizers. Two repeated-measures ANOVAs using cue difficulty (deterministic, easy, and hard) as a within-subjects factor and optimizer group (optimizer and non-optimizer) as a between-subjects factor were performed (one for the FB and one for the OB group) in order to examine potential differences in accuracy throughout learning for participants who optimized their responding at the end of the learning phase versus those who did not. In the feedback group, results indicated a main effect of cue difficulty (F (2,38) = 25.95; p < 0.001), a main effect of optimizer group (F (1,19) = 15.38; p < 0.005), and a significant interaction (F (2,38) = 4.06; p < 0.05). Optimizers accuracy (82%) in the entire learning phase was significantly better than nonoptimizers (70%; t (19) = 3.92; p < 0.005). This was driven by significantly greater accuracy for easy cues (t (16.99) = 7.14; p < 0.001) and marginally greater accuracy for deterministic cues (t (19) = 1.83; p = 0.08) for optimizers compared with nonoptimizers. For the observation group, results indicated a main effect of cue

130 113 difficulty (F (1.39,25.05) = 66.07; p < 0.001), no main effect of optimizer group (F (1,18) = 0.38; p > 0.05), and no significant interaction (F (1.39,25.05) = 0.67; p > 0.05). The next analyses examined whether the BOLD responses in the functionally defined ROIs in the learning phase were modulated by optimizer group. Specifically, the regions of the caudate nucleus and MTL which exhibited a main effect of cue difficulty were examined. A repeated-measures ANOVA was performed on the mean BOLD responses extracted from each ROI using cue difficulty as within-subjects factor and optimizer group as a between-subjects factor. In the FB group, the right caudate nucleus exhibited a main effect of cue difficulty (F (2,38) = 14.71; p < 0.001), no main effect of optimizer group (F (1,19) = 1.86; p > 0.05), and a significant interaction of cue difficulty by optimizer group (F (2,38) = 7.09; p < 0.005). The interaction was driven by marginally greater BOLD responses for easy cues in the non-optimizers than the optimizers (t (16.19) = 2.37; p = 0.03; trend after sequential Bonferroni correction). This result is consistent with the critical role of the striatum in acquiring cue contingencies and then decreasing activation once contingencies are acquired (Tricomi et al., 2004; Delgado et al., 2005). The left MTL however exhibited a main effect of cue difficulty (F (2,38) = 11.98; p < 0.001), but no main effect of optimizer group (F (1,19) = 0.97; p > 0.05), and no significant interaction (F (2,38) = 0.22; p > 0.05). The same ANOVA was performed on the regions engaged in learning in the observation group. The right and left caudate nucleus and left MTL exhibited main effects of cue difficulty (right caudate nucleus: F (2,36) = 13.38; p < 0.001; left caudate nucleus: F (2,36) = 12.88; p < 0.001; left MTL: F (2,36) = 8.83; p < 0.005), but

131 114 not optimizer group (right caudate nucleus: F (1,18) = 0.11; p > 0.05; left caudate nucleus: F (1,18) = 1.35; p > 0.05; left MTL: F (1,18) = 0.49; p > 0.05), and no significant interaction (right caudate nucleus: F (2,36) = 2.46; p > 0.05; left caudate nucleus: F (2,36) = 0.00; p > 0.05; left MTL: F (2,36) = 3.20; p = 0.05). To summarize therefore, only the right caudate nucleus in the feedback group was modulated by optimizer group, no other region s BOLD responses were modulated by whether or not participants optimized their responding at the end of the learning phase. In the update phase, in order to classify optimizer versus non-optimizers different criteria were used than the criterion used in the learning phase. In this phase, behavioral accuracy was examined in the second block of the update phase as a whole, rather than examining only the last 24 trials. This was done because when examining the last 24 trials of the second block, nearly every participant optimized their responses and since participants were relearning cue contingencies, they had more experience with the task. Therefore, any participant who was 100% correct for the easy cues was categorized as an easy cue optimizer, any participant who was 100% correct for the easy and hard cues was categorized as an easy/hard cue optimizer, and those who did not optimize for either easy or easy and hard cues were termed non-optimizers. Using this criterion, 15 participants were easy cue optimizers, 13 participants were easy/hard cue optimizers, and the remaining 13 participants were nonoptimizers. A repeated-measures ANOVA was performed on the behavioral data using learning type (same or different) and cue difficulty (deterministic, easy, and

132 115 hard) as within-subjects measures and learning phase group (feedback or observation) and optimization group (easy cue optimizers, easy/hard cue optimizers, and non-optimizers) as between-subjects factors. Results indicated no main effect of learning type (F (1,35) = 0.08; p > 0.05), a main effect of cue difficulty (F (2,70) = 33.73; p < 0.001), no main effect of learning phase group (F (1,35) = 0.08; p > 0.05), and a main effect of optimization group (F (2,35) = 7.05; p < 0.005). Significant interactions included learning type by learning phase group (F (1,35) = 6.05; p < 0.05), and a three way interaction of cue difficulty, learning phase group, and optimization group (F (4,70) = 3.07; p < 0.05). Of interest was the observation that accuracy overall was modulated by optimization group with the easy/hard cue optimizers (83%) significantly more accurate than easy cue optimizers (73%;t (21.04) = 3.09; p < 0.01) and non-optimizers (67%; t (24) = 4.10; p < 0.001). Lastly, the ROIs functionally defined by the main effect of cue difficulty analysis were examined to probe for potential differences between optimizers and non-optimizers in the update phase. The same ANOVA performed on the behavioral data was conducted on the neuroimaging data, one ANOVA per ROI. To summarize the findings briefly, no region exhibited a main effect of optimizer group [anterior putamen: (F (2,35) = 0.35; p > 0.05); posterior putamen: (F (2,35) = 1.96; p > 0.05); right parahippocampus: (F (2,35) = 1.16; p > 0.05); left parahippocampus: (F (2,35) = 0.41; p > 0.05)], nor any significant interactions with optimizer group (all p values > 0.05). Therefore, the BOLD responses during the update phase in the putamen and parahippocampus were not significantly

133 116 different for participants who optimized their responses in this phase versus those who did not. 4.4 Discussion The purpose of this experiment was to examine in a novel manner how the medial temporal lobe and basal ganglia interact during probabilistic learning, by employing multiple types of learning (feedback and observation trials) and a reversal of probabilistic cue contingencies. Behavioral results indicated that participants in the observation group initially acquired cue contingencies better than participants in the feedback group. In the update phase, interestingly, all participants, irrespective of group, were initially more efficient at updating old cue contingencies with new cue values via observation learning. Over the course of the update phase, however, updating via feedback improved for both groups, such that by the end of the update phase and in the test phases, no learning type differences were observed. Neuroimaging results revealed the engagement of the MTL and BG during both feedback (group 1) and observation (group 2) learning, supporting the hypothesis that these regions are engaged simultaneously and may operate in parallel during probabilistic learning. In further support of this hypothesis, while both of these regions were engaged during learning, they exhibited differential patterns of activity, with the basal ganglia exhibiting greatest BOLD responses to hard cues and the medial temporal lobe exhibiting greatest BOLD responses to deterministic cues. Results from the update phase corroborate this finding as loci within the basal ganglia

134 117 and medial temporal lobe were engaged in the update phase, processing differences in cue difficulty. In the update phase, interestingly, distinct loci within the basal ganglia exhibited both differential (BOLD signal greatest for hard cues) as well as similar (BOLD responses greatest for deterministic cues) patterns of activity as those observed in the medial temporal lobe (BOLD signal greatest for deterministic cues). The application of functional and effective connectivity allowed for a more sophisticated examination of interactions among these regions during the reversal of probabilistic cue contingencies. Results revealed both functional (instantaneous correlations) and effective (directed influences) connectivity among key reward-related and memory structures, specifically the midbrain, loci within the basal ganglia (caudate nucleus, putamen, and globus pallidus), and loci within the medial temporal lobe (hippocampus and parahippocampus). Lastly, the application of reinforcement learning models allowed for the examination of putative dopaminergic influences during both initial acquisition as well as updating of cue contingencies. Feedback trial prediction error signals were encoded within regions of the basal ganglia (ventral striatum, caudate nucleus, and globus pallidus) during both learning and updating of probabilistic cues, replicating findings previously reported in the literature (O'Doherty et al., 2004; Tanaka et al., 2004; Pessiglione et al., 2006). An action prediction error, representing an unsigned surprise signal generated during participants viewing cue/association pairs, exhibited recruitment of several distinct regions, including the superior frontal gyrus, medial frontal gyrus, midbrain, as well as the cingulate gyrus. Taken as a whole, these results

135 118 implicate a critical role for the basal ganglia and medial temporal lobes during initial acquisition of feedback and observation information (Poldrack et al., 1999; Poldrack et al., 2001; Foerde et al., 2006; Cincotta and Seger, 2007) as well as reversal learning in accord with the literature (Annett et al., 1989; Myers, 2000; Carrillo et al., 2001; Cools et al., 2002; Schoenbaum and Setlow, 2003; Setlow et al., 2003; Frank and Claus, 2006; Myers et al., 2006; Shohamy et al., 2009) and in support of the parallel model of engagement of these distinct regions. It was hypothesized that participants would initially acquire cue contingencies well, irrespective of how cue values were initially acquired (feedback vs. observation learning). However, behavioral results indicated that participants in the observation group acquired cue contingencies in the learning phase better than feedback group participants, in accord with a recent study which demonstrated greater behavioral accuracy for instructed compared with feedback learning (Li et al., 2011). However, this result was not hypothesized, based on the results of experiment 1, and suggests that observation learning may have be easier for participants than learning via feedback in this specific learning environment, which included deterministic as well as probabilistic cues. The presence of deterministic cues may have helped improve observation learning, while not facilitating feedback learning to the same extent. In support of this theory, accuracy differences were greatest across learning types for deterministic cues while accuracy was most equated across learning types for the hard probabilistic cues. It was further hypothesized that participants accuracy would initially decrease in the update phase following probabilistic cue

136 119 reversal for both learning types in both groups, and subsequently improve over time as participants updated old cue contingencies. This hypothesis was partially supported, with a significant improvement in feedback trial accuracy within the first block of the update phase, but no significant increase in observation trial performance over time (as participants were already performing quite well, mean = 74% in block 1). Lastly, two hypotheses were made regarding behavioral accuracy in the update phase: 1) participants may have been more efficient at updating cue values within the same learning type (observation-observation or feedback-feedback) as they would be accustomed to this manner of learning and perhaps could readily employ neural structures already involved in initial learning to update contingencies. 2) Alternatively, participants may have exhibited more efficient updating of cue values when employing a new learning type (observation-feedback or feedback-observation) as perhaps engaging a new learning mechanism would facilitate faster updating (particularly for feedback learning, which can be habit-driven and at times impervious to changes in stimulus-outcome associations for review see (Packard and Knowlton, 2002; Yin and Knowlton, 2006). Neither of these two hypotheses were supported, however, as interestingly, all participants, irrespective of group, initially were more efficient at updating cue contingencies via observation learning. This result suggests that employing observation learning may be a faster and more effective manner of updating information early in learning. This observation learning type advantage was not permanent however, as feedback trial accuracy steadily improved throughout the first update block, resulting in no learning type

137 120 differences in the second update block. Therefore, while updating cue contingencies via observation information was initially beneficial, both learning types were equally effective in the long term, consistent with the behavioral results of experiment 1 and reports in the literature of equivalent accuracy for information acquired via feedback and observation (Poldrack et al., 2001; Shohamy et al., 2004b). The pattern of activity reported within the striatal regions in the learning phases of the current experiment, showing greater responses to hard compared with easy and deterministic cues, was not expected based on the results of experiment 1. However, such a finding is not uncommon in the literature. Delgado and colleagues (2005) suggest that the striatum is involved in the initial learning of cue contingencies and decreases activation once these contingencies have been acquired. In a probabilistic card guessing task, from which the experiments in this dissertation were modified, participants were required to learn the value of several cues: two cues were deterministic (100%), two cues were probabilistic (67%), and one cue was random (50%). The authors discovered that the striatum was initially engaged for all three cue types relatively equally, early in the task when participants were initially learning the cue outcomes and therefore feedback was meaningful (e.g., feedback indicated a correct choice for the 100% and 67% cues). However, as time progressed and participants acquired the cue contingencies, the striatum was differentially engaged depending on the cue probability, exhibiting decreased activation in the feedback period of the trial for the 100% cues and greater BOLD signal responses during

138 121 the feedback period for the unlearnable cue (50%). The authors theorized that the striatum plays a pivotal role in learning cue associations, and becomes less engaged once these associations have been successfully acquired. The cue probabilities employed in the current task were 100%, 87.5%, and 62.5%. It is possible that the addition of the deterministic cue changed the learning environment, effectively making the hard cues seem harder. Therefore, the striatum was most engaged in learning the value of the 62.5% cue. An environment where everything is probabilistic (task employed in experiment 1) compared with an environment where only some things are probabilistic [tasks employed in experiment 2 and (Delgado et al., 2005)] is most likely a very different experience for the participants. In the task employed in experiment 1, there were only two probabilities, 85% and 65% which most likely made for a simpler learning environment. Greater striatal BOLD activation in response to the 85% cue in experiment 1 was most likely driven by more correct feedback for the easy compared with hard cues. The learning phase results support the hypothesis that the basal ganglia and medial temporal lobes operate in parallel during learning, by providing evidence of parallel engagement, as well as competitive and cooperative interactions between these regions. The fact that these two distinct regions were engaged simultaneously, suggests that they may be online and involved in learning at the same time. This ability to be engaged during the same time period indicates parallel processing of these areas during learning. Evidence of cooperative interactions was observed from positive correlations exhibited

139 122 between the hippocampus and caudate nucleus in the observation group for the deterministic and easy cues. It is thought provoking that despite the fact that these two regions displayed opposing patterns of activity the BOLD signal was still positively correlated between the areas for only the observation group. Perhaps acquiring the information in an observational manner, possibly by employing a declarative-like strategy, modified the interactions of these regions despite their opposing pattern of BOLD signals. This result may also have been driven by individual variations in mean BOLD responses in the caudate nucleus and hippocampus. This is an interesting result which merits further exploration in order to fully understand its significance. Lastly, the finding that the pattern of activity within the hippocampus and caudate nucleus/putamen was opposing, suggests that the striatum and hippocampus may encode distinct information in the learning phase of the current task. Opposing patterns of activity and differential activation within the hippocampus and striatum may be interpreted as competitive interactions, as has been previously done in the literature (Poldrack et al., 1999; Poldrack et al., 2001; Foerde et al., 2006; Seger and Cincotta, 2006). Furthermore, this finding partially replicates Poldrack and colleagues 2001 study, which reported activation of the striatum and deactivation of the MTL during feedback learning. However, the authors also observed greater activation in the MTL during an observation version of their task (WPT) compared with the feedback version; whereas in the present study BOLD responses below baseline occurred for both feedback and observation trials. Mean BOLD signal below baseline in the MTL has been reported in several probabilistic learning tasks

140 123 (Poldrack et al., 1999; Poldrack et al., 2001; Aron et al., 2004; Foerde et al., 2006). It has been suggested that rest in neuroimaging experiments may engage the MTL, and therefore BOLD responses below baseline during task periods may not be indicative of active suppression of this region during learning (Foerde et al., 2006). Additionally, a seminal study by Stark and Squire demonstrated that BOLD responses within the MTL may appear as either activation or deactivation in the same task, dependent solely on the rest/baseline condition with which MTL BOLD responses are compared (Stark and Squire, 2001). Results from the update phase also support parallel engagement of the basal ganglia and medial temporal lobes. Compared with the learning phase, distinct regions within the basal ganglia and MTL were engaged in the update phase, processing differences in cue difficulty, specifically loci within the left putamen and bilateral parahippocampal gyri. The right and left parahippocampus displayed the same pattern of activation as the hippocampus region in the learning phase, while only one putamen area exhibited greater responses to hard than easy and deterministic cues. The more posterior putamen area displayed the same pattern of activity as seen within the MTL; that is greater responses to deterministic and easy compared with hard cues. This result also supports parallel engagement between the MTL and basal ganglia, as different striatal areas processed probabilistic information in opposing manners, making obvious the fact that the dorsal striatum plays several roles during the learning process. There are multiple loops within the striatum (Haber, 2003),

141 124 which connect to distinct sections of both the midbrain and cortex. It is possible that discrete regions of the putamen, one being more posterior in the putamen, and one located more anterior and medially, may be engaged in distinct functional loops and therefore convey different information to both the midbrain and the cortex. The observation of activation bordering the putamen/insula makes interpreting these results slightly more challenging however, as this result could be driven more by insula than putamen BOLD responses. This caveat should be taken into consideration as it is possible that more medial regions of the striatum may display a different pattern of results; however give the learning phase results this seems unlikely. The null findings within the main effect of learning type and main effect of group analyses in the update phase may possibly be interpreted as evidence of parallel engagement of the MTL and basal ganglia. Neither the striatum nor any regions within the MTL were more strongly engaged in updating information that was the same (feedback-feedback or observation-observation) or different (feedback-observation or observation-feedback) as the manner in which cuevalue information was initially acquired when both groups were combined into one analysis. However, regions of the cingulate and insular cortex were engaged and exhibited greater BOLD responses to the same than a different learning type. Neither were loci within either the MTL or basal ganglia more strongly engaged for either the feedback or the observation group in the update phase. It could have been theorized, that if the striatum was selectively engaged in feedback learning, it may have been recruited in initial feedback acquisition

142 125 (group 1), and as a result of this strong early activation, subsequently online during the reversal phase to a greater degree in group 1 compared with group 2 participants. Likewise, it could have been theorized that if the MTL was strongly recruited during initial observation learning (group 2) it may have been online and more strongly engaged throughout the reversal phase for group 2 compared with group 1. This result was not observed; however, null findings in neuroimaging studies are difficult to interpret, and not necessarily indicative of any finding per se. This null result therefore should be interpreted with caution and may be an area of interest for further pursuit in future studies. The interaction of learning type by group observed in the update phase revealed that regions of the striatum and midbrain were more strongly involved in processing feedback compared with observation trials. It is not surprising that these areas were more strongly engaged during feedback trials, which contained outcome relevant information, as compared with observational trials, which did not carry outcome relevant information. This result is consistent with a large body of research implicating both the striatum and the midbrain in feedback processing and outcome monitoring (Delgado et al., 2000; Knutson et al., 2001; Delgado et al., 2003; Aron et al., 2004; Tricomi et al., 2004; Marco-Pallares et al., 2007; Carter et al., 2009). The application of reinforcement learning models to the behavioral data provided the opportunity to probe how neural substrates are engaged during initial acquisition and updating of probabilistic information. Furthermore, it allowed for the investigation of putative dopaminergic influences during learning

143 126 and reversal of cue contingencies. The results implicated the traditional network of regions commonly reported to encode reward prediction errors, namely the nucleus accumbens, the globus pallidus, and the caudate nucleus (O'Doherty et al., 2004; Tanaka et al., 2004; Pessiglione et al., 2006). Interestingly, for the observation learners, the medial temporal lobe (hippocampus/amygdala) and medial frontal gyrus were also engaged in processing feedback trial prediction errors in the update phase. One possible explanation for this result is that participants in the observation group employed a more declarative-like learning strategy, and thus engaged the medial temporal lobe, which then tracked outcomes that violated participants expectations when updating old cue contingencies. This finding is similar to the observation of the hippocampus tracking feedback trial prediction error signals in experiment 1. Furthermore, a recent study (Chen et al., 2011) observed mismatch signals in the CA1 region of the hippocampus and the perirhinal cortex during associative retrieval when previously viewed face-house pairs where mismatched. Additionally, the amygdala has previously been reported in processing prediction error signals (Bray and O'Doherty, 2007) and integrating prediction error signals with other learning signals (Belova et al., 2007); therefore this result is not completely unexpected. However, it is interesting that the MTL was engaged in encoding prediction errors for the feedback trials, only within the observation group. Results from the action prediction error analyses, which measured the level of surprise or saliency when participants viewed cue/value associations during the observation trials, revealed the engagement of two prefrontal areas

144 127 including the medial frontal gyrus and superior frontal gyrus as well as the midbrain, insula, and cingulate gyrus. This network of regions, in particular the midbrain, insula, and cingulate gyrus, was commonly activated in both the learning and update phases of the task. Therefore it is reasonable that this network also positively correlated with a saliency signal during observation trials. It may have been expected that the dorsal lateral prefrontal cortex (DLPFC) would be engaged during encoding of the action prediction error signal in the learning and update phases, based on the findings of Burke and colleagues (2010). The authors developed an observational-based action prediction error model in order to examine the neural correlates of observational learning. Burke and colleagues reported that a signal correlating with this model was encoded in the DLPFC. The task employed in this experiment, however, is dramatically different from the experiment utilized by Burke et al., which may explain the discrepancy in the results. In the current experiment, participants, although observing the cue and outcome on the screen, were instructed to be active participants in the task. They were informed to optimize their responding and told that the amount of money they would be paid depended on their performance. In contrast, Burke and colleagues employed both an active and a passive (observer) condition for the participants. In the observer condition, participants were told they were viewing the actual choice and outcome of another human player s choice. On the observer trials, the participants were not required to make choices, but rather simply observe the other player s choices and outcomes. In this learning environment, perhaps a more realistic

145 128 observational learning situation, action prediction errors were encoded in the DLPFC. Burke et al. s task therefore is fundamentally different from the task employed in the current experiment, and as such the result of distinct areas of activation involved in encoding the action prediction error signal employed in this experiment may not be so surprising. Granger causality analyses using the ventral tegmental area as a reference region support the theory that the MTL and BG operate in parallel during probabilistic learning. Functional connectivity results revealed a relatively similar network of regions was engaged when updating cue contingencies, including the caudate nucleus, putamen, globus pallidus, hippocampus, and parahippocampus, across the observation and feedback groups. Some distinctions arose however, with the feedback group correlating primarily with the caudate nucleus, putamen and hippocampus, while the observation group exhibited instantaneous correlations with the globus pallidus and parahippocampus. The functional connectivity results implicate that key reward learning and memory structures are correlated during probabilistic reversal learning. This finding is in accord with a broad literature linking the involvement of the BG and MTL during reversal learning and enhances previous results by suggesting that these regions may not act independently during the reversal process, but rather are functionally connected (Annett et al., 1989; Myers, 2000; Carrillo et al., 2001; Cools et al., 2002; Frank and Claus, 2006; Myers et al., 2006; Shohamy et al., 2009). Furthermore, directed connectivity results revealed that the putamen influenced the VTA, and the VTA influenced the hippocampus

146 129 and parahippocampus during probabilistic reversal. This result supports the theory that the MTL and BG may not interact directly, but rather may communicate via midbrain dopaminergic centers (Poldrack and Rodriguez, 2004; Shohamy and Adcock, 2010). Regions of the putamen do exhibit reciprocal anatomical projections with the VTA/SN (Haber and Knutson, 2010), and the VTA does send dopaminergic projections to the hippocampus (Haber and Knutson, 2010). Therefore the effective connectivity results observed during the updating of probabilistic cue contingencies in the present data set are anatomically possible. Subsequent studies exploring the potential flow of information from the putamen to VTA and from the VTA to the hippocampus during probabilistic reversal learning may be an area of future research. Lastly, the results from the learning phase provided an important clarification regarding the results from experiment 1. In the learning phase of the task employed in experiment 1, both a region of the caudate nucleus and the hippocampus were engaged in processing differences in cue difficulty. However, in that experiment, both feedback and observation learning trials were completed in the same phase and therefore the results may have been skewed by the presence of both trial types within the same phase. As a consequence, it was important to examine the engagement of the hippocampus and striatum in feedback and observation learning independently. The learning phase of experiment 2 provided such a distinct examination as one group of participants acquired cue contingencies via feedback and a separate group acquired cue contingencies via observation. Importantly, in both the feedback group and the

147 130 observation group, regions of the MTL, specifically the hippocampus extending into the amygdala, and the striatum, including the caudate nucleus and the putamen, were engaged in processing cue difficulty. Perhaps equally interesting was the result that in this experiment the striatal and hippocampal regions displayed opposing patterns of activity. Both regions were involved in processing cue difficulty, but whereas the MTL (activity encompassing the hippocampal/amygdala regions) showed greatest activity to deterministic cues and decreased in activation according to difficulty, the striatal regions displayed greatest activity to hard cues and decreased with respect to difficulty. This finding therefore partially replicates the result of experiment 1, where the hippocampus and the caudate nucleus displayed stronger activation to easy compared with hard cues. It is theorized that this hippocampal pattern of activity (easy > hard) represents a learning signal, as the hippocampus is known to be involved in encoding and recalling information. Several studies have shown greater hippocampal activation is elicited for information that is subsequently remembered, compared with information that is subsequently forgotten (Brewer et al., 1998; Wagner et al., 1998; Law et al., 2005; Adcock et al., 2006; Shrager et al., 2008). While follow up tests did not support this effect in this study, it is possible that if a test had been administered one week following completion of the task, participants would have better remembered the easier cues. To summarize, results from experiment 2 suggest that regions within MTL and BG are involved in the initial acquisition and updating of both feedback and observation probabilistic information. Furthermore, prediction error and Granger

148 131 causality analyses lend support to the theory that these distinct memory systems operate in parallel during learning and may interact via midbrain dopaminergic regions. However, it is still unknown what specific learning contexts drive memory systems to operate in parallel or interact directly via competition or cooperation. Experiment 3 was designed to address this question directly. Chapter Five: Experiment Introduction: Background & Rationale One manner of investigating the nature of interactions between multiple memory systems in non-human animals is to lesion, or temporarily knockout, one brain region and observe the effect on behavior. In humans, this question has been addressed by studying multiple memory systems in patients who have abnormal functioning of one system (e.g., Parkinson s disease or MTL amnesia). Examining this question in healthy human participants, however, is challenging. One manner of doing so is to employ behavioral interference tasks, whereby one region of interest (e.g., the MTL) is recruited during learning in order to observe the subsequent effect on behavior and memory system interactions. The goal of this experiment was to engage the MTL during learning of a secondary task in order to examine interactions between multiple memory systems. It was theorized, given the rich animal lesion and human patient literature that new insights would be gained into the nature of how multiple memory systems interact during behavioral interference. A specific hypothesis, based on the results of experiments 1 and 2, and described in detail below, was formulated in which

149 132 memory systems would cooperate when information to-be-learned was challenging and operate more independently when information to-be-learned was easy, in accordance with the parallel processing model of multiple memory systems. The results of animal lesion studies in rodents have demonstrated a clear double dissociation between the function of the basal ganglia and the medial temporal lobe in learning, with the MTL supporting spatial learning and the BG supporting stimulus-response instrumental learning (Packard et al., 1989; Packard and McGaugh, 1992, 1996). One relatively recent exemplary study demonstrated bidirectional competitive interactions between these distinct regions, as knocking out one region, the dorsal striatum, actually improved hippocampal dependent learning, and likewise, knocking out the dorsal hippocampus, improved striatal dependent learning in mice (Lee et al., 2008). In addition, the application of neurotransmitters, in particular glutamate and dopamine has been shown to influence the expression of hippocampal and striatal-based learning (Packard and White, 1991; Packard, 1999). In a plus maze task, rodents were always placed in the same start location in the maze (south arm) and required to navigate to one arm which was baited with a food reward (west arm) (Packard, 1999). Rodents naturally solve this task in one of two ways, and a probe trial is administered in order to determine how each individual rodent solves the task. On the probe trial, rodents are placed in a new starting location in the maze (north arm). Rodents that navigate to the correct arm (west arm) are labeled place learners, as they remembered the correct

150 133 location of the food reward. Rodents which make the previously correct motor response (turn left) to navigate to the previously correct location (west arm), will now end up in the opposite arm (east arm), and are termed response learners because they learned the correct motor response in order to navigate to the food location. However, when rodents are over-trained on this task, all rodents typically engage in response learning, suggesting a shift in the engagement of memory systems over time. Interestingly, intrahippocampal injections of the neurotransmitter glutamate bias rodents towards place learning both early and late in the training process, indicating a blockade by the hippocampus in shifting towards response learning. Likewise, intracaudate injections of glutamate produce response learning both early and late in the task, suggesting accelerated response learning. In a related study using an eight-arm radial maze task, the affects of intracerebral injections of dopamine agonists in both the hippocampus and the caudate nucleus were examined in rodents (Packard and White, 1991). In this version of the win-shift radial maze task, animals obtained food from 4 randomly selected arms, then a delay period was administered, after which rodents were returned to the maze and the other 4 arms, which previously did not contain food, now held the food reward. In the win-stay version of this task, animals learned to approach 4 randomly lit arms, twice within a trial, in order to obtain food. Posttraining intrahippocampal injections of three types of dopamine agonists (indirect, D1, and D2) improved retention on the win-shift version of the task, compared with control rodents (received saline injections), while intracaudate injections of

151 134 these dopamine agonists had no affect on the win-shift version. Likewise, intracaudate injections of all three dopamine agonists improved acquisition of the win-stay version of the task, compared with control rodents, while intrahippocampal injections had no effect on this task version. This important animal work demonstrates two key points: one, both the hippocampus and striatum are involved in spatial learning tasks, are online simultaneously, and contribute to solving these tasks in unique ways; two, how these memory systems are engaged and the relative degree of their engagement can be manipulated by the administration of neurotransmitters to bias the engagement of one memory system over the other, thereby affecting how learning tasks place. In studies involving human participants, the question of the medial temporal lobes and basal ganglia s contribution to learning and how these multiple memory systems interact has been addressed in a multitude of manners including behavioral and functional neuroimaging experiments using healthy human participants (Poldrack et al., 1999; Poldrack et al., 2001; Seger and Cincotta, 2005; Cincotta and Seger, 2007; Shohamy and Wagner, 2008; Mattfeld and Stark, 2010), patient populations (Knowlton et al., 1994; Knowlton et al., 1996; Dagher et al., 2001; Myers et al., 2003; Hopkins et al., 2004; Voermans et al., 2004; Shohamy et al., 2005; Shohamy et al., 2006; Schmitt-Eliassen et al., 2007) as well as by employing tasks involving cognitive interference (Foerde et al., 2006; Brown and Robertson, 2007; Foerde et al., 2007). Studies examining probabilistic learning in human patient populations have provided evidence that the medial temporal lobes play a crucial role in the flexible transfer of information,

152 135 while the basal ganglia is involved in the initial acquisition of stimulus associations (Myers et al., 2003) as well as the formulation of sequences of stimulus outcome associations (Nagy et al., 2007). One manner of examining the involvement of distinct neural substrates in healthy human participants is to employ tasks of cognitive interference in order to engage one neural substrate and observe the outcome on behavior, akin to a behavioral knockout of one region of interest. One study examined the interaction of multiple memory systems using fmri of healthy human participants who completed a feedback-based probabilistic learning task while simultaneously engaging in a second cognitive demanding task (tone counting task) (Foerde et al., 2006). Specifically, participants completed a feedback-based version of the weather prediction task, where they were required to learn the associations of various cues via feedback. The task consisted of two conditions: one condition was a control condition in which tones were played in the background and participants were instructed to ignore the tones (single-task); the second condition required participants to simultaneously attend to and count the number of high tones (high and low pitch tones were played; dual-task). The authors hypothesized that the engagement of the MTL and BG could be modulated by distraction, such that the addition of a cognitively demanding task (tone counting) would occupy working memory, thereby decreasing declarative memory encoding and biasing the engagement of the basal ganglia system. Results revealed that behavioral accuracy was only moderately diminished in the dualtask compared with the single-task condition. In a subsequent probe session (no

153 136 feedback administered, no tones played, cues learned under single and dual-task conditions were intermixed), no accuracy differences were observed between cues learned under single versus dual-task conditions. Furthermore, in the probe session, BOLD signal within the medial temporal lobe correlated with behavioral accuracy for associations made under the single-task condition only, while BOLD signal within a region of the striatum, specifically the putamen, correlated with behavioral accuracy for those associations made under the dual-task condition only. Lastly, BOLD activity in the MTL during the probe session positively correlated with declarative cue knowledge for cues learned under the single-task condition only. The authors concluded that this probabilistic classification task can be successfully learned via the MTL or the BG, and that the engagement of these regions depends on the task demands such that the presence of a distracting working memory task biased the engagement of the BG during learning, while learning under normal conditions engaged the MTL. Importantly, while the MTL and the striatum were both able to support successful performance on the weather prediction task, the nature of the knowledge that was acquired differed, with only the MTL supporting declarative knowledge about learned cue associations. A second, unrelated behavioral study examined interactions between declarative and procedural memory during the consolidation process, which consisted of either a period of participants staying awake versus a period of sleep (Brown and Robertson, 2007). This study consisted of two experiments: in experiment 1 the influence of declarative learning (word list learning) was

154 137 measured on procedural consolidation (serial reaction time task/srtt); in experiment 2 the influence of procedural learning was measured on declarative consolidation (using the same tasks). In experiment 1 participants first performed the SRTT, then performed the word list learning task, and after a 12 hour period of either wakefulness or sleep, were retested on the SRTT. Interestingly, the authors observed that following a period of wakefulness, declarative learning interfered with subsequent improvements on the procedural skill (SRTT). Furthermore, a negative correlation was observed between subsequent skill performance and declarative list learning, such that the more words participants learned, the worse they performed on the procedural skill. However, following a period of sleep, declarative learning had no interference on skill performance. The authors concluded that during periods of wakefulness, declarative learning interfered with off-line processing, or the consolidation, of the procedural skill. The mirror result was observed when declarative learning was followed by procedural skill learning. Procedural skill learning disrupted offline processing of declarative information following a period of wakefulness, and a negative correlation was observed such that greater motor skill was associated with worse word recall. Additionally, no interfering effect of procedural skill learning was observed on declarative learning following a period of sleep. The authors proposed several models by which multiple memory systems may interact, 1) potentially exhibiting interconnected or overlapping neural circuits during the awake state thereby interfering with each other s consolidation process during wakefulness, while 2) exhibiting functional uncoupling or the

155 138 engagement of independent systems during the sleep state such that consolidation processes among distinct brain regions are independent during sleep. To summarize, new insights into how multiple memory systems interact during learning can be achieved by employing tasks of cognitive interference, such as the studies employed by Foerde and colleagues (2006) as well as Brown and Robertson (2007). Foerde et al. demonstrated that the relative engagement of multiple memory systems can be biased by task demands, while Brown and Robertson demonstrated that the exact nature of interactions between multiple memory systems is dynamic, still yet to be completely characterized, and may change depending on whether consolidation of declarative and procedural information occurs during an awake versus a sleep state. One remaining open question is characterizing under which specific learning scenarios memory systems may cooperate versus compete. The purpose of the present experiment was to address this question by employing a cognitive interfering task, in order to engage the medial temporal lobe and examine the effects of such a behavioral knockout of the MTL on a feedbackbased probabilistic learning task. The aim of employing such a design was to gain new insights into the relative engagement and interactions between the MTL and BG during feedback probabilistic learning by probing one learning scenario in which multiple memory systems may cooperate. The following hypothesis was formulated based on the results of experiments 1 and 2: the MTL and BG may cooperate during

156 139 challenging/inconsistent learning scenarios and may operate independently during easy/consistent learning scenarios. Results from experiment 1 suggested that the BG and MTL may cooperate when learning is inconsistent as the following results were observed: 1) positive correlations between the hippocampus and caudate nucleus s BOLD signal during probabilistic observation learning; 2) a positive correlation between the hippocampus and caudate nucleus s BOLD signal for feedback easy trials, late in learning when participants received incorrect feedback (prediction error-like situation); 3) marginally greater hippocampal BOLD signal for feedback hard cues, later in learning and marginally greater caudate nucleus BOLD signal for observation hard cues, later in learning; and 4) encoding of a feedback prediction error signal within both the striatum and hippocampus. Furthermore, results from experiment 2 corroborate the theory that these memory systems have the ability to both compete and cooperate during learning, as mutual engagement of these regions was observed during the learning and update phases, at times exhibiting similar patterns of activation (potential cooperation) and at other times exhibiting opposing patterns of activity (potential competition). Positive correlations were observed in both the learning and update phases across the striatum and MTL for deterministic cues and a feedback prediction error signal was encoded within loci in the striatum and the MTL (hippocampus/amygdala). Additionally, it has been theorized that depending on the learning scenario, memory systems may operate in parallel or may interact directly via competition or cooperation (White and McDonald, 2002). However, the exact learning scenarios when cooperation

157 140 or competition occur remain uncharacterized. Therefore, it was hypothesized, based on the aforementioned results and theoretical work that the BG and MTL operate in parallel and may interact directly via cooperation when the learning scenario is particularly challenging/inconsistent. The rationale for this hypothesis is that in inconsistent and therefore challenging learning scenarios, when cue value contingencies are low, e.g., 70% predictive of the value, it is possible that neither the MTL nor the BG system will learn the cue values well. Therefore, if both systems independently are not able to acquire the values, it is theorized that they may cooperate, by communicating and sharing information each system has individually acquired in order to learn the cue values in a synergistic manner. However, when the contingencies are consistent, and therefore less challenging to learn, e.g., 90%, one system alone may successfully acquire the cue values or both systems independently may learn the contingencies and therefore the systems do not need to cooperate. In order to test this hypothesis, a within-subjects fmri study was conducted using a feedback probabilistic learning task, which contained both easy (90%) and hard (70%) to learn information. The hypothesis was that the MTL and BG may cooperate during acquisition of difficult (70% cues), but not easy (90% cues) information. Furthermore, in order to demonstrate that both the MTL and BG are required to learn challenging/inconsistent information, multiple learning sessions were utilized, one of which was designed to engage the MTL in an unrelated task (behavioral knockout of the MTL). This was a two-day experiment, in which participants came to the laboratory one day and performed

158 141 a declarative memory scene encoding task and returned the following day to complete a feedback probabilistic learning task while undergoing functional neuroimaging. Specifically probabilistic cue learning occurred via feedback in three distinct sessions. In each session, a probabilistic cue was presented on the screen along with a natural scene (landscape image). The three sessions were: 1) a feedback control session, in which participants were required to learn only the value of probabilistic cues and ignored the natural scenes. This session provided for a baseline examination of multiple memory systems during feedback learning; 2) a perceptual decision session, in which participants were required to learn the value of the probabilistic cues as well as attend to the natural scenes and make a perceptual decision about each scene. This session was employed as a control for the cognitive demands elicited in the declarative memory interference condition (dual-tasking); 3) a declarative memory interference condition, in which participants were required to learn the value of the probabilistic cues while simultaneously engaging in a declarative memory recognition task (determining whether each natural scene was old or new). Of key importance was the declarative memory interference condition, which was designed to behaviorally sequester or knock out the medial temporal lobe by engaging it in the recognition of previously viewed scenes. This task design therefore allowed for an examination of the relative engagement of the MTL and BG and interactions between these two regions when the MTL is available to assist in learning (control conditions feedback control and perceptual decision session) and when it may not be available to assist in learning (experimental

159 142 condition declarative memory interference condition). 5.2 Materials & Methods Experimental design This study involved two days of participation. On day one, participants completed a behavioral declarative memory scene encoding task in the laboratory. Participants viewed 40 natural landscape scenes on a computer and were asked to determine whether or not each scene contained water (sixty percent of the scenes contained water, e.g., a lake, stream, waterfall; each scene was presented one time; Figure 5.1A). Every scene was a unique landscape image, borrowed from a database from the Center for Cognitive Neuroscience at Duke University. The purpose of the water detection question was to ensure that participants paid attention to the scenes during the encoding session. Participants were told that they should try their best to learn the scenes as they would be tested on them the following day and their payment on the subsequent day would be determined based on how well they correctly discriminated between old scenes (shown on day 1) and new scenes (to be shown on day 2). During the encoding session, each scene was displayed for 4 seconds during which time the participants studied the scenes and looked for water. This was followed by a 2-4 second jittered inter-stimulus-interval (ISI), followed by a 4 second period in which participants made a choice regarding the presence or absence of water in the scene they had just viewed. Participants were instructed to press 1 if there was definitely water in the scene, 2 if there was probably water

160 143 in the scene, 3 if there was probably no water in the scene, and 4 if there was definitely no water in the scene. They were instructed to press 2 or 3 if they had trouble remembering the scene or if it was ambiguous as to whether the scene contained water. A blue box highlighted participants choice and no penalty was assigned if the participant missed a trial. The water response period was followed by a 4-10 second jittered ITI before the onset of the next scene. Participants were paid $10 for completion of the scene encoding session on day 1. On the second day of the experiment, participants completed the MRI portion of the study, which consisted of a learning and memory task that was a modification of the tasks employed in experiments 1 and 2. Participants were required to learn the numerical value (range 1-9) of six different visual cues. The value of each cue was either higher or lower than the number five. There were two phases in the experiment, a learning phase and a test phase. Two independent variables were manipulated in the learning phase: the learning session type and the difficulty of the probabilistic cues. There were three distinct learning sessions, all of which contained feedback learning: the feedback control session (Figure 5.1B), the perception decision session (Figure 5.1C), and the declarative memory interference session (Figure 5.1D). The second independent variable was the probabilistic value of the cues (predictive outcome of the numerical value of each cue): 3 cues were 90% predictive of their value, termed easy cues and 3 cues were 70% predictive of their value termed hard cues. Participants were instructed on all three session types and performed a short

161 144 practice version with different probabilistic cues and different natural scenes prior to beginning the task inside the MRI. The practice session natural scenes were drawn from the Duke University scene database as well as the McGill University Calibrated Colour Image Database (Olmos and Kingdom, 2004). In the learning phase participants primary goals were to learn the value of six cues and correctly identify old versus new natural scenes. The learning phase consisted of six blocks (two presentations of each of the three session types) of 20 trials each for a total of 120 trials. In all sessions, participants were required to learn the numerical value of two cues (one 90% cue and one 70% cue per session). Each session contained 10 presentations of each of the two cues; all trials were feedback based. The trial format in every session consisted of a 4 second natural scene/cue presentation period, followed by a 2-4 second jittered ISI, proceeded by feedback presentation (contingent on participant s response for the cue value; check mark for correct, X for incorrect, and # for missed trials), followed by a response screen which varied by session type (details provided below), and lastly a jittered ITI (4-10 seconds) before onset of the next stimulus. In all learning sessions, the stimulus presentation period consisted of a natural scene displayed at the top of the screen and a probabilistic cue presented at the bottom of the screen. Depending on the session type, the natural scene was either new (presented for the first time) or old (previously shown on the day prior during the encoding session). Each natural scene presented in the experiment was shown only one time; therefore the scenes did not overlap between sessions, and participants were made explicitly aware of

162 145 this fact. Participants were instructed to make a button press in all sessions during the stimulus presentation period regarding the value of the probabilistic cue (button 1= higher than 5; button 2 = lower than 5). Similar to the feedback trials in experiments 1 and 2, participants were instructed to optimize their responses. Instructions for the natural scenes varied by session type as described below. In the feedback control session, participants were instructed to focus on learning the value of the probabilistic cues and were told to ignore the natural scenes, which were all new (no old scenes were presented from the day prior) and unimportant (Figure 5.1B). Participants were informed that the natural scenes in the feedback control session would never be shown again and that they would not be tested on these scenes in the future. Participants viewed the natural scene/cue and made a button press indicating the value of the probabilistic cue during the 4 second stimulus presentation/response period. After the ISI, participants received feedback contingent on their response. Feedback presentation was proceeded by a response screen, where participants made a button press (button 1, button 2, button 3, button 4) which they were informed was not meaningful and had no associated value. They were told they could know why the button press was necessary in this session at the end of the experiment if they desired (it served as a motor control). The purpose of this session was to provide a baseline level of feedback learning in the task. In the perception decision session, participants were instructed to pay attention to the natural scenes and to detect water in them while simultaneously

163 146 learning the value of the probabilistic cues (Figure 5.1C). They were informed that in this session all of the scenes were new. They were also told that they did not need to learn the scenes, as they would not be tested on them, but were simply required to detect water in each scene. The purpose of this session was to serve as a cognitive-demand control for the declarative memory interference session. While looking for water may not be as mentally taxing as trying to remember a scene, it does require some attentional processing and dual-tasking. In the perception decision session participants viewed the natural scene/cue and made a button press indicating the value of the probabilistic cue during the stimulus presentation period. Following the ISI and feedback presentation, the response screen appeared and participants were instructed to make a response regarding the presence or absence of water in the natural scene they had viewed in the stimulus presentation period. This response screen was identical to that presented during the day 1 encoding session. They were instructed to press 1 if the scene definitely contained water, 2 if it probably contained water, 3 if it probably contained no water, and 4 if it definitely contained no water. In the declarative memory interference session, participants were required to determine whether the natural scene presented in each trial was old or new while simultaneously learning the value of the probabilistic cues (Figure 5.1D). During the stimulus presentation period they viewed a natural scene and probabilistic cue and were instructed to try to remember if the scene was old or determine if it was new while learning the value of the probabilistic cue. As in all other sessions, they were informed to make a button press indicating the value of

164 147 the probabilistic cue in the stimulus presentation period. Following the ISI and feedback presentation period, the response screen appeared. In this session participants were told to press 1 if they thought the scene was definitely old, 2 if it was probably old, 3 if it was probably new, and 4 if it was definitely new. Additionally, participants were instructed to press 1 only if they had a strong memory of seeing the scene on the day prior (I remember) and 2 if they recognized the scene to be old, but did not have a strong memory of seeing it the day before (I know). They were explicitly informed that half of the scenes would be old and half would be new in this session and that this was the only session in which they would see natural scenes that were shown on the day prior (old scenes). They were reminded that the total amount of money they would be paid would be determined by how well they correctly identified old versus new natural scenes and how well they learned the value of the feedback cues in all three of the learning sessions. Participants completed the sessions in one of two orders: order 1) block 1: feedback control, block 2: declarative memory interference, block 3: perception decision, block 4: feedback control, block 5: declarative memory interference, block 6: perception decision; order 2) block 1: feedback control, block 2: perception decision, block 3: declarative memory interference, block 4: feedback control, block 5: perception decision, block 6: declarative memory interference. Half of the participants completed one session order and the other half completed the other session order. Trial order in each block was pseudorandomized to ensure that the trial types were balanced throughout each block

165 148 (e.g., there were an equal number of old and new scenes paired with easy and hard cues in trials 1-10 and 11-20). Different trial orders were created for the two blocks of each session type (e.g., trial order in feedback control block 1 differed from trial order in feedback control block 2); all participants completed the identical trial order. Following completion of the learning sessions, participants completed the test session (not depicted in the figure). The test phase contained the probabilistic cues learned in all three sessions, as well as two novel cues presented for the first time. The cues from the learning phase were blue, and the novel cues were white in color. Critically, in the test session, no feedback was provided and no natural scenes were presented. Participants were simply required to indicate the value of the probabilistic cues they had learned in the learning sessions. The test session contained 80 trials total, 10 presentations of each cue, presented in random order. Trial structure and timing in this test phase were identical to that used in the test phases in experiments 1 and 2 (self-timed stimulus/response phase followed by a 6-14 second jittered ITI before onset of the next cue) Data analysis Behavioral Data Analysis: D Prime: In order to examine participants ability to correctly identify scenes which contained water versus those that did not in the day 1 and day 2 sessions as well as scenes which were old versus new in the day 2 task, D prime (d ) analyses were performed (Macmillan and Creelman, 1991). D prime

166 149 analyses are informative as they account for a participant s response bias (e.g., more likely to indicate water versus no water; or more likely to choose old versus new) when calculating a participant s ability to correctly identify a stimulus (sensitivity). Neuroimaging Data Analysis: GLM: The primary analysis of interest was to examine activation within the brain during the three distinct learning sessions (feedback control, perceptual decision, and declarative memory interference). Comparing brain activation across learning sessions was most desired as the key question of interest was examining how multiple memory systems were engaged during pure feedback learning compared with feedback learning concurrent with declarative memory interference. Therefore, three random-effects GLM analyses were conducted in the learning phase; one GLM per learning session. Each GLM contained the following two predictors: session type (which varied by session: feedback control, perception decision, and declarative memory interference) and cue difficulty (easy and hard). Missed trials and six motion parameters were also included as regressors of no interest. In order to gain the most complete picture of the engagement of brain regions during learning and memory processes, the mean BOLD signal was examined during two periods: 1) the stimulus presentation period and 2) the feedback presentation period. The stimulus presentation period of the trial occurred when the probabilistic cue and natural scene were presented on the screen. This time period was examined in order to probe for differences in cue difficulty and memory processing within the brain, with the

167 150 primary focus on reward-related learning and memory structures (e.g., striatum, medial temporal lobe, and midbrain). It was during this time period that participants were making a choice regarding the value of the probabilistic cue and presumably either ignoring or attending to the natural scene depending on the session (ignore the scene in the feedback control session; look for water in the perceptual decision session; or try to remember the scene in the declarative memory interference session). During the feedback presentation period, participants received feedback regarding their choice of the probabilistic cue value (correct, incorrect, missed trial). In the GLM created for the feedback control session, a contrast of easy vs. hard cues was performed to examine activity within the brain during both the stimulus presentation as well as the feedback presentation periods. In the perception decision GLM, a 2 (water: water vs. no water) x 2 (cue difficulty: easy vs. hard) within-subjects ANOVA was performed using water (water-scene and no water-scene) and cue difficulty (easy and hard) as within-subjects factors when examining activity during both the stimulus presentation as well as the feedback presentation periods. The primary SPM of interest investigated a main effect of cue difficulty. The water condition was included as a factor because participants attended to the presence or absence of water in the scenes during the day 1 encoding session, and discussions with participants after the experiment indicated that some participants used the presence or absence of water in the natural scenes as a memory cue (way to learn and subsequently remember the scenes). In the declarative memory GLM, a 2 (scene: old vs. new)

168 151 x 2 (cue difficulty: easy vs. hard) within-subjects ANOVA was performed using memory (old scenes and new scenes) and cue difficulty (easy and hard) as within-subjects factors when examining activity during both the stimulus presentation as well as the feedback presentation periods. SPMs of interest investigated a main effect of memory, a main effect of cue difficulty, and an interaction of memory and cue difficulty. SPMs were thresholded at p<0.005 with an appropriate cluster threshold correction, correcting to a cluster-level falsepositive rate of 5% when possible. As noted in the results section, these analyses did not produce activation within any a priori regions of interest (striatum, MTL, midbrain) at cluster threshold corrected levels in the control conditions. Therefore, the hypothesis could not be tested and results from these SPMs are not reported. For the test phase, BOLD activity was examined across the entire trial (stimulus onset and participant response). A random-effects GLM was performed using the following predictors: learning material (feedback control, perception decision, declarative memory interference, and novel) and cue difficulty (easy and hard). SPMs were generated from this GLM with the contrast of interest as previously studied (feedback control, perception decision, declarative memory interference) versus non-studied (novel) cues. Resulting SPMs were set to a threshold of p<0.005 and a cluster threshold level of 5 contiguous voxels or 132mm 3. As a note, three participants data was excluded from the test phase neuroimaging analysis due to technical errors (e.g., Eprime quitting during the scanning session).

169 152 Region of Interest Analyses: Independent regions of interest were drawn in order to examine activity within a priori brain regions in an unbiased manner. Specifically, the following seven regions were examined: bilateral hippocampus (x, y, z = -20, -11, -17 and x, y, z = 21, -12, -18); bilateral ventral striatum (x, y, z = -11, 4, 0 and x, y, z = 10, 3, -6); bilateral midbrain, centered on the ventral tegmental area (x, y, z = -4, -15, -9 and x, y, z = 5, -14, -8); and the left caudate nucleus (x, y, z = -15, 20, 7). All but one of the ROIs (left caudate nucleus) were drawn from a previously published paper (Adcock et al., 2006). The bilateral hippocampus, bilateral ventral striatum, and bilateral midbrain (VTA) were chosen from this study in particular as the experiment employed in Adcock and colleagues examined the effects of reward-related motivation on memory. It was theorized that the areas reported by the authors would be relevant to examine in the current experiment, which investigated the effect of declarative memory interference on reward-related learning. The left caudate nucleus ROI was taken from experiment 1 of this dissertation. The caudate nucleus region was investigated due to the observation that loci within the caudate nucleus were engaged in both experiments 1 and 2 of this dissertation, suggesting this area plays a role in feedback probabilistic learning. Mean parameter estimates were extracted from the above regions in all three learning sessions (feedback control, perception decision, and declarative memory interference) and mean BOLD responses were examined during the stimulus presentation period.

170 Results Behavioral Results Day 1 scene encoding Results from the d analysis revealed that participants hit rate for accurately identifying scenes with water (0.97; responding water to a waterscene) was significantly greater than the false alarm rate (0.16; responding water to a no-water-scene; t (23) = 48.12; p < 1.35 x ), with a bias towards labeling scenes as containing water (d =2.95; C=-0.41; Figure 5.2A). This result suggests that participants discriminated very well between scenes that contained water versus those that did not during the day 1 encoding session (97% accurate). Day 2 scene water and memory discrimination Participants ability to correctly discriminate between new scenes that contained water versus those that did not (perception decision session) as well as old and new scenes (declarative memory interference session) was examined. In the perception decision session, participants hit rate (0.93) was significantly greater than the false alarm rate (0.11; t (23) = 32.97; p < 7.24 x ) with a bias towards labeling the scenes as containing water (d = 2.98; C=-0.14). Participants ability to accurately detect water in the natural scenes on day 1 was compared with their ability to accurately detect water in the natural scenes on day 2. Results revealed that participants had a significantly higher hit rate during the day 1 encoding task (97% > 93%; t (23) = 2.91; p < 0.01), with a significantly greater false alarm rate observed in the day 1 encoding task as well (16% > 11%;

171 154 t (23) = 2.20; p < 0.05) suggesting that on a whole, participants performed better during the day 1 encoding session. In the declarative memory interference session, participants hit rate (0.61; responding old to an old scene) was significantly greater than the false alarm rate (0.24; responding old to a new scene; t (23) = 9.58; p < 1.76 x 10-9 ) with a bias towards labeling scenes as new (d =1.11; C=0.25), suggesting that participants were able to successfully discriminate between old and new scenes. Furthermore, participants hit rate was greater than chance (61% > 50%; t (23) = 3.12; p < 0.005). In addition, participants hit and false alarm rates were compared across session types (perception decision vs. declarative memory interference). Results revealed a significantly higher hit rate in the perception decision session compared with the declarative memory interference session (93% > 61%; t (23) = 9.39; p < 2.48 x 10-9 ) and a lower false alarm rate in the perception decision session compared with the declarative memory interference session (24% > 11%; t (23) = 3.12; p < 0.005; Figure 5.2B), suggesting that, as expected, identifying scenes which contained water was an easier task than discriminating between old versus new scenes. Day 2 probabilistic cue accuracy Behavioral accuracy was examined in all three cue learning sessions: feedback control, perception decision, and declarative memory interference (Figure 5.2C). One question of interest was to examine differences within each session type as it varied according to cue difficulty. A second question of interest was to examine behavioral accuracy across sessions in order to determine if

172 155 accuracy was greater for one session compared with another (e.g., feedback control > declarative memory). Lastly, as this was a demanding task, it was hypothesized that participants accuracy would improve over time (early/first block vs. late/second block of learning). Therefore, the first block (trials 1-20) of each session was used as the early time factor, while the second block (trials 21-40) of each session was used as the late time factor. In order to examine these questions, a 3 (learning session: feedback control vs. perception decision vs. declarative memory interference) x 2 (cue difficulty: easy vs. hard) x 2 (time: early vs. late) within-subjects, repeated-measures ANOVA was performed. Results indicated a main effect of session type (F (1.65,37.89) = 16.60; p < 0.001), a main effect of cue difficulty (F (1,23) = 9.56; p < 0.01), a marginally significant effect of time (F (1,23) = 3.09; p = 0.09), a marginally significant interaction of cue difficulty by time (F (1,23) = 3.89; p = 0.06), and a significant three way interaction (session type x cue difficulty x time) (F (2,46) = 3.40; p < 0.05). Post-hoc t-tests were performed in order to examine the significant main effects and interactions. When examining the main effect of session type, results revealed better performance for cues learned in the feedback control (83%) and perception decision session (78%) compared with cues in the declarative memory interference session (63%) (t (23) = 5.76; p < and t (23) = 3.54; p < 0.005, respectively), with no accuracy differences observed between the feedback and perception decision control sessions (t (23) = 1.60; p > 0.05). A post-hoc t-test was also performed in order to examine the main effect of cue difficulty result, and as expected performance was significantly greater for easy (80%) compared with

173 156 hard (70%) cues (t (23) = 3.13; p < 0.005). In addition, a post-hoc t-test was performed to examine the marginally significant main effect of time, which as was theorized, was caused by nearly better performance in the second block (76%) compared with the first block (73%; t (23) = 1.71; p = 0.1). The marginally significant interaction of cue difficulty by time was driven by significantly greater performance for easy (80%) than hard (67%) cues in the first block (t (23) = 3.52; p < 0.005), but only marginally better accuracy for easy (79%) compared with hard (73%) cues in the second block (t (23) = 1.76; p = 0.09). This result was most likely driven by the increase in accuracy for the hard cues over time (t (23) = 2.25; p < 0.05), but not the easy cues (t (23) = 0.27; p > 0.05). Post-hoc t-tests were also performed to examine the significant three way interaction (session type x cue difficulty x time). This interaction was driven by significant changes in performance from the first to the second learning block in the feedback control learning session only (81% to 85%), with no changes observed over time in either the perception decision (77% to 79%) or the declarative memory interference sessions (61% to 65%; all p values > 0.05). In the feedback control session, performance for hard cues increased over time (t (23) = 3.31; p < 0.005) while performance for easy cues was marginally worse in the second compared with the first block (t (23) = 2.14; p = 0.04; trend after sequential Bonferroni correction). Correlations between natural scene sensitivity (d ) and probabilistic cue accuracy Two correlations were performed in order to examine potential relationships between participants ability to perform the probabilistic cue learning

174 157 task concurrently with the scene discrimination tasks (both the perception decision detection task and declarative memory interference task). Specifically, one correlation was conducted to probe a potential relationship between participants ability to differentiate between water/no-water scenes while learning the probabilistic cue values in the perception decision session. Participants hit rate, false alarm rate, and probabilistic cue accuracy for easy and hard cues were entered into a Pearson s correlation analysis. Results revealed a significant positive correlation between participants hit rate and easy cue accuracy (r=0.510, p = 0.01), but not hard cue accuracy (r=0.275, p = 0.19). A second correlation was conducted to examine a potential relationship between participants ability to discriminate between old and new scenes while learning probabilistic cue values. No significant relationships were observed between participants hit rate, false alarm rate, and probabilistic easy and hard cue accuracy (all p values > 0.05). Day 2 test phase A test phase was performed while participants remained inside the MRI to probe for behavioral performance differences on probabilistic cues (easy and hard) that were acquired across the three session types. The first analysis probed for potential differences in session type, irrespective of cue difficulty, by conducting a repeated-measures, within-subjects ANOVA using session type as a within-subjects factor (feedback control, perception decision, declarative memory interference, and novel). A significant main effect of session type was observed (F (2.11,48.62) = 13.21; p < 0.001). Post-hoc t-tests indicated significantly

175 158 greater performance for cues presented in all sessions compared with novel cues (feedback control vs. novel: t (23) = 5.18; p < ; perception decision vs. novel: t (23) = 4.54; p < ; declarative memory interference vs. novel: t (23) = 3.19; p < 0.005). Marginally significant better accuracy was observed for cues previously presented in the feedback control session (81%) compared with the declarative memory interference session (64%; t (23) = 2.19; p = 0.04; trend after sequential Bonferroni correction). No other significant differences were observed between cues previously presented in the various learning sessions (all p values >0.05). The second analysis examined differences across session types and levels of cue difficulty, excluding novel cues. In order do so, a 3 (learning session: feedback control vs. perception decision vs. declarative memory interference) x 2 (cue difficulty: easy vs. hard) repeated measures, withinsubjects ANOVA was performed. Results revealed a main effect of session type (F (1.21,27.87) = 3.88; p < 0.05), no main effect of cue difficulty (F (1,23) = 0.07; p > 0.05), and no interaction of cue difficulty and session type (F (2,46) = 0.60; p > 0.05). Post-hoc t-tests indicated marginally greater accuracy for the feedback control compared with the declarative memory cues, collapsed across difficulty (t (23) = 2.19; p = 0.04; trend after sequential Bonferroni correction) Neuroimaging results Learning phase analyses Region of Interest Analyses: In order to examine the engagement of learning and memory regions in an unbiased manner during the learning sessions, and to try to test the hypothesis that the MTL and BG correlate during

176 159 inconsistent learning conditions, independent regions of interest were drawn. It was hypothesized that regions of the MTL and BG would correlate during the control sessions (feedback and perception decision) most strongly for the inconsistent learning condition (hard cues) as the MTL may be online and available to assist in probabilistic cue learning. It was further hypothesized that these regions would not correlate during the declarative memory interference condition as the MTL would be engaged in the declarative memory task rather than the probabilistic cue learning task. Specifically activity within the bilateral hippocampus, bilateral ventral striatum, bilateral midbrain, and left caudate nucleus was examined. Two sets of analyses were performed. The first analysis examined activity within each ROI during all learning sessions independently. In the feedback control session, this analysis therefore examined differences in BOLD signal according to levels of cue difficulty; in the perception decision session, this analysis examined differences in BOLD signal across levels of cue difficulty and the presence or absence of water in the natural scenes; and in the declarative memory session, this analysis examined differences in BOLD signal across levels of cue difficulty and the declarative memory scene condition (old or new scene). The second analysis examined activity within each ROI across different session types (feedback control vs. perception decision vs. declarative memory interference) and levels of cue difficulty (easy vs. hard), collapsed across the memory scene condition (water/no-water or old/new). Results are presented below for the first analysis (activity in each learning session independently)

177 160 followed by the results from the second analysis (comparison of activity within each ROI across learning sessions). Feedback control session: In the feedback control session, pairedsamples t-tests were performed in each ROI to probe for differences in the BOLD signal response to easy and hard probabilistic cues. The right ventral striatum exhibited significantly greater responses to hard than easy cues (t (23) = 2.79; p < 0.05), while marginally greater responses for hard compared with easy cues was observed in the left ventral striatum (t (23) = 1.82; p = 0.08). No other regions of interest displayed any modulation in activation by cue difficulty. Perception decision session: In the perception decision session, pairedsamples t-tests were performed in each ROI to explore potential differences in the mean BOLD signal in response to easy versus hard cues; however no significant differences were observed in any region of interest in this session. Declarative memory interference session: In the declarative memory session, modulation of the mean BOLD signal in response to levels of cue difficulty as well as old versus new declarative memory scenes was examined. The right midbrain exhibited greater BOLD responses to hard compared with easy cues (t (23) = 2.18; p < 0.05) [a 2 (memory scene: old vs. new) x 2 (cue difficulty: easy vs. hard) within-subjects, repeated-measures ANOVA was conducted, results indicated no main effect of memory scene (F (1,23) = 0.87; p > 0.05); a main effect of cue difficulty (F (1,23) = 4.76; p < 0.05); and no interaction of memory scene by cue difficulty (F (1,23) = 0.05; p > 0.05)]. No other significant effects were observed.

178 161 Session type comparison: A comparison of the mean BOLD signal within each ROI across learning sessions (feedback control vs. perception decision vs. declarative memory interference) was also performed. In order to do so, seven separate 3 (session type: feedback control vs. perception decision vs. declarative memory interference) x 2 (cue difficulty: easy vs. hard) repeated-measures, within-subjects, ANOVAs were conducted (one per ROI) using session type and cue difficulty as within-subjects factors. Interestingly, every ROI except the left caudate nucleus exhibited a significant main effect of session type and no significant main effect of cue difficulty; only the right ventral striatum displayed an interaction of session type by cue difficulty. For each ROI, the main effect of session type was driven by greater BOLD responses in the declarative memory interference session compared with both the perception and feedback control sessions (except for the caudate nucleus, which exhibited no main effect of session type; representative results are shown in Figure 5.3 beta plots are exhibited for the left ROIs as the pattern of activity was similar across bilateral ROIs). The right hippocampus mean BOLD signal scaled according to session type with declarative memory interference > perception decision > feedback control. All other regions exhibited no differences between the mean BOLD signal elicited in response to feedback versus the perception decision control sessions. These results suggest that the hippocampus, ventral striatum, and ventral tegmental area are all more strongly engaged during the memory session than the control sessions, but interestingly

179 162 are not modulated by probabilistic cue difficulty. The results of this analysis are presented below according to ROI: 1) Right hippocampus: main effect of session type (F (2,46) = 13.09; p < 0.001); no main effect of cue difficulty (F (1,23) = 0.59; p > 0.05); no interaction of session type and cue difficulty (F (2,46) = 1.16; p > 0.05). Declarative memory interference session > perception decision (t (23) = 2.26; p < 0.05); perception decision > and feedback control session (t (23) = 3.00; p < 0.01). 2) Left hippocampus: main effect of session type (F (1.59,36.45) = 8.37; p < 0.005); no main effect of cue difficulty (F (1,23) = 0.07; p > 0.05); no interaction of session type and cue difficulty (F (2,46) = 0.73; p > 0.05). Declarative memory interference session > perception decision and feedback control session (t (23) = 3.52; p < and t (23) = 4.48; p < , respectively), with no differences observed between BOLD response to the control conditions (t (23) = 0.95; p > 0.05; Figure 5.3A). 3) Right ventral striatum: main effect of session type (F (2,46) = 3.96; p < 0.05); no main effect of cue difficulty (F (1,23) = 2.06; p > 0.05); an interaction of session type and cue difficulty (F (2,46) = 5.06; p < 0.05). Declarative memory interference session marginally > perception decision (t (23) = 2.32; p = 0.03; trend after sequential Bonferroni correction); declarative memory interference session > feedback control session (t (23) = 2.63; p < 0.05, respectively), with no differences

180 163 observed between BOLD response to the control conditions (t (23) = 0.72; p > 0.05). 4) Left ventral striatum: main effect of session type (F (2,46) = 9.01; p < 0.001); no main effect of cue difficulty (F (1,23) = 2.22; p > 0.05); no interaction of session type and cue difficulty (F (2,46) = 1.22; p > 0.05). Declarative memory interference session > perception decision and feedback control session (t (23) = 4.00; p < and t (23) = 4.03; p < 0.001, respectively), with no differences observed between BOLD response to the control conditions (t (23) = 0.24; p > 0.05; Figure 5.3B). 5) Right midbrain: main effect of session type (F (2,46) = 9.73; p < 0.001); no main effect of cue difficulty (F (1,23) = 1.10; p > 0.05); no interaction of session type and cue difficulty (F (1.57,36.01) = 1.29; p > 0.05). Declarative memory interference session > perception decision and feedback control session (t (23) = 3.12; p < and t (23) = 4.47; p < , respectively), with no differences observed between BOLD response to the control conditions (t (23) = 1.27; p > 0.05). 6) Left midbrain: main effect of session type (F (2,46) = 9.37; p < 0.001); no main effect of cue difficulty (F (1,23) = 1.96; p > 0.05); no interaction of session type and cue difficulty (F (1.55,35.74) = 0.42; p > 0.05). Declarative memory interference session > perception decision and feedback control session (t (23) = 2.67; p < 0.05 and t (23) = 4.73; p < , respectively), with no differences observed between BOLD response to the control conditions (t (23) = 1.49; p > 0.05; Figure 5.3C).

181 164 7) Left caudate nucleus: no main effect of session type (F (2,46) = 1.13; p > 0.05); no main effect of cue difficulty (F (1,23) = 0.13; p > 0.05); no interaction of session type and cue difficulty (F (2,46) = 1.00; p > 0.05; Figure 5.3D). Independent ROI Correlation Analyses: In order to test the hypothesis that the mean BOLD signal within the striatum and hippocampus would be positively correlated during the control conditions, specifically for the inconsistent learning situation (hard cues), with potential correlations observed between the midbrain, a series of Pearson s correlations were performed. Specifically three correlations, one for each learning session, were conducted. An abundance of correlations were observed across regions. Therefore, for simplicity of presentation, a summary of the correlations observed is described below and the details regarding the correlations (which areas correlated with each other and at what statistical value) are presented in Appendix 2: Feedback control session: five positive correlations were observed across anatomical regions, four occurred in the hard cue condition and one occurred in the easy cue condition. In the hard cue condition, the hippocampus correlated with the ventral striatum and caudate nucleus, while the ventral striatum additionally correlated with the midbrain. In the easy cue condition, the midbrain correlated with the ventral striatum. Perception decision session: seventeen positive correlations were observed across anatomical regions, five occurred in the hard cue condition and

182 165 twelve occurred in the easy cue condition. In the hard cue condition the midbrain correlated with the hippocampus and ventral striatum. In the easy cue condition, the hippocampus correlated with the ventral striatum, the caudate nucleus, and midbrain; the midbrain additionally correlated with both the ventral striatum and the caudate nucleus. Declarative memory interference session: fourteen positive correlations were observed across anatomical regions, seven in the hard cue condition and seven in the easy cue condition. In the hard cue condition, the hippocampus correlated with the ventral striatum, caudate nucleus, and midbrain, while the ventral striatum correlated with the midbrain. In the easy cue condition, the hippocampus correlated with the ventral striatum and midbrain, and the ventral striatum correlated with the midbrain. To summarize, the correlation results indicate that while there were significant positive correlations in the control conditions between the striatum, hippocampus, and midbrain for the hard cues, positive correlations were also observed in the declarative memory interference condition and in all conditions for the easy cues. The observation of positive correlations in all learning sessions for both easy and hard cues indicates that these regions did not correlate exclusively when the learning condition was inconsistent (hard cue condition). However, it is thought provoking that several more correlations were observed across anatomical regions in the more challenging conditions (perception decision and declarative memory interference) compared with the

183 166 feedback control condition. A possible interpretation of this result is presented in the discussion. Stimulus Presentation & Feedback Presentation Period Whole Brain Analyses: As described in the methods section, the stimulus presentation period of the trial, when the probabilistic cue and natural scene were presented on the screen, was examined in order to probe areas of the brain that were involved in processing differences in cue difficulty and memory information (old and new scenes). Additionally, the feedback period of the trial was examined in order to probe for differences in the brain while processing feedback regarding the probabilistic cues. However, no BOLD activation was observed during either of the control conditions (feedback or perception decision) in any area of interest (MTL, BG, midbrain) at a threshold of p<0.005, uncorrected during the stimulus or the feedback presentation periods. Therefore, this hypothesis could not be tested by examining functional activation in these regions during either the stimulus presentation or the feedback presentation period and as a consequence, no results are reported during these time periods. 5.4 Discussion The purpose of the current experiment was to investigate interactions between memory systems during feedback-based probabilistic learning concurrent with declarative memory interference. Behavioral sensitivity (d ) as well as accuracy measures revealed that participants successfully encoded natural scenes (day 1) and subsequently remembered them the following day

184 167 while performing a feedback-based probabilistic cue learning task. Participants accuracy during the feedback-based learning task was significantly modulated by the presence of the concurrent declarative memory task, such that performance during the control sessions was greater than the declarative memory interference session. In addition, behavioral differences in levels of cue difficulty (easy cue accuracy > hard cue accuracy) were only observed in the feedback control condition, suggesting that session type had a stronger influence in modulating behavior. Neuroimaging results revealed that within independent regions of interest, BOLD activity was also modulated by session type, perhaps tracking these behavioral measures. Likewise, only one region, namely the ventral striatum exhibited differences in BOLD responses according to cue difficulty in the feedback control condition. No ROIs were modulated by cue difficulty in the perception decision session, and in the declarative memory interference session only the right midbrain s BOLD signal was modulated by cue difficulty. The right ventral striatum and right midbrain both displayed greater BOLD responses to hard compared with easy cues, replicating a similar result observed in experiment 2. Lastly, a series of Pearson s correlations revealed widespread positive correlations across ROIs in all learning sessions, with the most correlations observed in the dual-tasking sessions. Interestingly, no negative correlations were observed across these independently defined regions. The significance of these results is postulated below. Behavioral results indicated that participants successfully attended to the presentation of natural scenes during the day 1 encoding session, as

185 168 demonstrated by the high hit rate (97%) on a perceptual detection task during encoding. Furthermore, participants performed the perception decision task (water detection) well, while simultaneously performing feedback-based probabilistic cue learning on day 2. Behavioral sensitivity (d ) was significantly worse on the water detection task on the second day, when learning the probabilistic cues; however, a hit rate of 93% is still indicative of participants ability to successfully detect water in the new natural scenes despite having to dual-task. Participants sensitivity at detecting old versus new natural scenes in the declarative memory interference session was significantly worse compared with the perception decision detection task (hit rate: 61% versus 93%), indicating that the tasks were not equated in their difficulty. This result complicates interpreting the neuroimaging results, as described below. However, participants did discriminate between old versus new scenes at above chance levels, suggesting that the task was not impossible to complete. An examination of behavioral probabilistic cue accuracy revealed that, as expected, participants were better at learning cue values in the feedback control condition (83%) compared with the declarative memory interference condition (63%). However, it was hypothesized that accuracy would scale according to session type, such that feedback control > perception decision > declarative memory interference. Contrary to this hypothesis however, no significant difference was observed between participants cue accuracy in the feedback control compared with the perception decision session (83% 78%; p = 0.12). It appears that this form of dual-tasking did not significantly affect participants

186 169 ability to learn the probabilistic cues. In fact, a correlation analysis conducted on participants hit rate and probabilistic cue accuracy revealed a significant positive relationship for probabilistic easy cues only, in the perception decision session. No such positive relationship was observed for the hard cues. This result indicates that participants ability to accurately perform these distinct tasks simultaneously was related. A similar correlation performed for the declarative memory interference session produced no significant relationships between participants hit rate, false alarm rate, or cue accuracy, for neither easy nor hard cues. This result may suggest that perhaps participants solved these distinct tasks in different ways and that their ability to perform each task: old/new scene discrimination versus probabilistic cue learning was unrelated. The main effect of session type observed in the bilateral hippocampus, ventral striatum, and midbrain ROIs is an interesting and unexpected finding. These reward-related learning and memory regions displayed a global increase in the mean BOLD signal in the declarative memory interference session compared with the control sessions. Two theories were formulated to interpret this finding. The first theory is that participants motivation to gain increased monetary payment modulated the BOLD signal in key learning and memory ROIs. It is possible that participants were motivated to perform well, especially in the declarative memory interference condition, and as a result of this motivation, ramping up of activation in these areas occurred during this session. In support of this theory, Adcock and colleagues (2006) demonstrated modulated BOLD signal in the exact regions examined in this experiment (ROIs, except for the

187 170 caudate nucleus, were drawn from this paper), in a declarative memory encoding task. BOLD signal within these areas varied depending on the reward value associated with to-be-learned-stimuli. In Adcock and colleagues study, participants performed a declarative memory encoding task (viewed natural scenes) while undergoing functional neuroimaging. Critically, the presentation of each scene was preceded by a monetary value, either $5 or 10 cents which indicated to the participant how much they would be paid if they subsequently remembered the scene correctly on a follow up test. Interestingly, only highreward scenes ($5) that were subsequently remembered on a behavioral test administered the next day elicited greater BOLD signal in the hippocampus, ventral striatum, and midbrain. High-reward scenes that were subsequently forgotten as well as low-reward scenes (that were both remembered and forgotten) did not elicit increased engagement of these learning and memory regions preceding encoding. Furthermore, greater correlation between the hippocampus and the right midbrain (VTA) for high-reward compared with lowreward scenes, and for high-reward scenes that were subsequently remembered was observed. This data provides strong evidence that reward-related motivation influences learning by specifically predicting the information which is subsequently remembered. The authors concluded that their results are in accord with the hypothesis that reward-motivated dopamine release in the hippocampus prior to learning promotes memory formation (Lisman and Grace, 2005; Shohamy and Adcock, 2010).

188 171 In accord with Adcock et al. s result, Wittman and colleagues (2005) demonstrated that pictures which predicted monetary reward versus those that did not were associated with greater BOLD activation in the substantia nigra. Furthermore, on a test administered three weeks later, participants were more accurate at remembering the pictures which predicted reward versus those that did not. Additionally midbrain and hippocampal BOLD activation was greater for subsequently recognized reward-predicting pictures versus those that were forgotten. Therefore, this study also highlights a critical relationship between reward-related motivation and learning and memory. An additional line of research examining genetic polymorphisms in dopamine clearance pathways among human individuals supports the hypothesis that dopamine is involved in human memory formation (Meyer-Lindenberg et al., 2005; Schott et al., 2006). A relatively recent study demonstrated differential degrees of engagement of reward-related memory structures depending on individual polymorphisms in dopamine clearance pathways (Schott et al., 2006). In this experiment, polymorphisms in the dopamine transporter (DAT1) gene affected levels of midbrain activation in an episodic encoding task. Additionally, catechol-o-methyl transferase (COMT) polymorphisms modulated activity in the right prefrontal cortex, such that participants with low COMT activity displayed stronger coupling between the hippocampus and PFC during the encoding task. The authors concluded that individual genetic variations in dopamine clearance pathways in the brain effect both prefrontal cortex and midbrain BOLD activation in an

189 172 episodic encoding task, implicating a strong role for dopamine in human memory formation. Therefore based on the literature, it is possible that the engagement of reward-related learning and memory structures was modulated by participants motivation to earn monetary compensation. Perhaps participants were highly motivated to perform well in especially the declarative memory interference condition as they were aware that their payment would be determined based on how well they learned the probabilistic cues in all conditions and how accurately they discriminated between old and new natural scenes. Unfortunately, no questionnaire addressed this theory directly by asking participants how motivated they were to perform in each session individually and is therefore a limitation of this study. The potential role of dopamine, in particular, in influencing this global BOLD signal increase in reward-related learning structures is only moderately supported by the observation that a region of the dorsal striatum, namely the caudate nucleus, did not display such a global increase in BOLD responses during the declarative memory interference session. Given that midbrain projections do not reach the dorsal components of the caudate nucleus (Devan and White, 1999; Haber and Knutson, 2010), it is possible that a lack of dopamine modulation in this area resulted in no differentiation of the BOLD signal by learning session. However, this theory is impossible to test in the current paradigm and therefore should be regarded with caution, especially given that null findings in neuroimaging studies are not indicative of any particular finding. This theory may represent an area of future studies, namely exploring differential

190 173 BOLD activation within regions of the brain that receive direct dopamine modulation versus those that do not in reward-related learning and memory tasks. The second possible explanation for the observed increased BOLD activation during the declarative memory interference session is that it was the most cognitively demanding session. Because it was the hardest condition, as demonstrated by behavioral sensitivity and accuracy measures, this session may have recruited the greatest neural activation, eliciting the largest metabolic demand, and causing the most substantial increase in BOLD responses in reward-related learning structures. This explanation is in accord with one theory regarding the nature of the BOLD signal (Heeger and Ress, 2002). Additionally, the literature supports the notion that some brain regions do correlate with increased task demand, in particular several prefrontal regions, including the posterior dorsolateral prefrontal cortex BA44/46 (see Buckner and Wheeler, 2001 for review). However, such an increase was not observed in the caudate nucleus, which may be an indication that this increased BOLD signal was not uniform throughout the brain in the current paradigm. Given that the perceptual decision and declarative memory interference sessions were not equated in their difficulty makes it challenging to disentangle these two interpretations of the neuroimaging results, and is a limitation of this study. If the two tasks had been equated in difficulty, it would have strengthened the argument that learning and memory regions are ramping up activation due to reward-related motivational influences.

191 174 The results from the independent ROI analyses therefore, did not support the hypothesis that the MTL and BG cooperate, as measured by simple correlations, during inconsistent learning conditions (hard probabilistic cue condition) and operate independently during more consistent learning conditions (easy probabilistic cue condition). Examining the mean BOLD signal extracted from these a priori regions of interest revealed that rather than tracking differences in cue difficulty in individual sessions, all ROIs, with the exception of the caudate nucleus, exhibited increased BOLD signal in the declarative memory interference session compared with both the control sessions (perception decision and feedback learning). Furthermore, an abundance of correlations were observed across the ROIs for both the easy and hard cues in all learning sessions. Therefore, while these data do not support the exclusivity of the hypothesis generated (correlations for only inconsistent learning condition), the data do not completely refute it either, as there were positive correlations across regions for hard cues in the control sessions. Additionally, it is possible that in the present task design, the declarative memory interference condition itself functioned as the challenging learning condition, rather than the hard probabilistic cue condition. Evidence supporting this theory comes from a post-experimental questionnaire in which participants were asked to indicate which session was the most difficult for them. The majority of participants indicated that the declarative memory interference session was the hardest (14 participants), although others indicated that the perceptual decision session was the most challenging (9 participants; 1 person stated that the feedback control session was the most

192 175 difficult). If in fact the declarative memory interference session was the most difficult, then the data do somewhat support the theory that the MTL and BG cooperate when the learning environment is challenging, as demonstrated by worse behavioral performance in the declarative memory interference session as well as an increased number of correlations in this session compared with the feedback control condition. However, the lack of behavioral interference and large number of positive correlations across regions in the perception decision session make this theory weak. To summarize, these results suggest that distinct learning and memory regions are more globally engaged when simultaneously learning the value of probabilistic cues via feedback and performing a declarative memory recognition task than when performing less cognitively demanding tasks (control conditions). Although the data in this experiment did not support the proposed hypothesis, employing a behavioral knockout of the MTL did provide new insights into the engagement and interactions of multiple memory systems (modulation of behavior and BOLD activation across multiple memory regions, as well as the number of positive correlations across regions, by learning session type) and may be an area of further research in the future. In particular it is still unknown under what conditions multiple memory systems compete versus cooperate during learning.

193 176 Chapter Six: General Discussion 6.1 Purpose & Summary of Experiments The purpose of the experiments performed in this dissertation was to examine the engagement and interactions between multiple human memory systems during probabilistic learning. Specifically the engagement and interactions between the medial temporal lobe, with focus on the hippocampus, and the basal ganglia, with emphasis on the striatum was examined during feedback and observation probabilistic learning. The tasks employed investigated this question in young healthy human participants using both within and between-subjects, event-related designs which allowed for the examination of BOLD activity within regions to discrete events across the dynamic learning process. Multiple types of learning as well as variations in cue difficulty were employed in order to probe how distinct brain substrates, namely the MTL and BG, were modulated in multiple learning environments. The key hypothesis for the experiments performed in this dissertation was that multiple memory systems, specifically the MTL and BG, operate in parallel they are online simultaneously, engaged in learning and may operate independently by encoding unique information or may compete or cooperate directly. Furthermore, it was hypothesized that interactions between these anatomical regions may be facilitated by dopaminergic midbrain activity, as investigated by the application of reinforcement learning models which are postulated to represent a dopaminergic learning signal in experiments 1 and 2.

194 177 Nature of representations in the MTL and BG: It is evident that the BG and the MTL are involved in probabilistic learning as demonstrated by the results in the current studies as well as an abundance of evidence in the literature. However, important questions to consider include what is the nature of representations in each of these regions? Is it the same or distinct? While these are difficult questions to illuminate completely in the experiments utilized in this dissertation, one may theorize about the representations encoded in the MTL and BG during the feedback and observation tasks in the present experiments. As the MTL is often implicated in encoding stimulus-stimulus associations and episodic information (Gluck and Myers, 1993; Bunsey and Eichenbaum, 1995; Eichenbaum and Bunsey, 1995) as well as recently implicated in tracking rewardrelated information (Li et al., 2011), it is theorized that for the observation trials in experiments 1 and 2, the MTL encoded the stimulus-stimulus associations of the cue (stimulus 1) and the informative arrow (stimulus 2). Over the course of learning, as these two stimuli are repeatedly paired (e.g., circle and upwardfacing arrow) the MTL may have encoded the value for the circle as higher than 5, binding these two associations together. For the feedback trials, since the trial structure is distinct, it is possible that the MTL encoded either the episodic experiences of incrementally learning the cue-outcome associations over time (participant remembers previous episodes of receiving correct feedback for a cue, which then informs the current cue-value choice) or that the hippocampus was tracking the reward value of the probabilistic cues (differentiates between correct and incorrect feedback; consistent with Li and colleagues 2011). The

195 178 basal ganglia however, may have encoded distinct representations during both feedback and observation learning trials than the MTL. For the feedback trials, one hypothesis is that the basal ganglia, in particular the striatum, was involved in feedback learning tracking goal outcomes and updating cue contingencies via error correction, consistent with the literature. During observation learning, the striatum may have been involved in action selection as well as encoding cuearrow associations, in accord with Seger s theory of the role of the BG during category learning (2008). One manner of testing the above hypotheses and examining whether or not the MTL and BG were encoding similar or distinct information may be to examine their ability to flexibly use previously acquired cue-outcome associations in a novel manner. It has been suggested that the hippocampus is involved in flexibly using knowledge in novel scenarios whereas the BG is not (Myers et al., 2003). Therefore, a probe test could have been administered for both trial types in order to determine in if a new context, where participants would need to flexibly use previously acquired cue-value information in a novel manner, the hippocampus would be engaged in the flexible usage of the knowledge, while the BG would not. An alternative idea would be to track the BOLD responses over the learning phase in a trial by trial basis. BOLD responses in the striatum may initially have been driven by responses to the feedback period, but as learning progressed it may have shifted to the cue period. Examining these phases (cue versus feedback) distinctly could have allowed for decoupling of such activation. However, the task was not designed to address this question and there was no

196 179 inter-stimulus-interval placed between the cue and feedback periods. As a consequence, examining the cue and feedback periods separately would be challenging and specifically the results obtained by examining the feedback period may have been skewed by the BOLD responses in the cue period. Some evidence supporting the theory that the MTL may have been encoding stimulus-stimulus associations was observed in the BOLD responses to consistent versus inconsistent information. In both experiments 1 and 2, the striatum (caudate nucleus and putamen) was not modulated by cue consistency, whereas the MTL (hippocampus and parahippocampus) was modulated by the consistency of the observation cues. In both studies, the MTL exhibited greater BOLD responses (sometimes manifested as the least negative BOLD responses) to inconsistent compared with consistent cue values, perhaps encoding a mismatch signal between the expected (e.g., circle paired with upward-facing arrow) and actual (e.g., circle paired with downward-facing arrow) cue-values observed during learning. It is interesting that only the MTL regions were modulated by cue consistency, and not the striatum, suggesting that perhaps these areas may have encoded distinct representations of cue values, and that the MTL was involved in binding stimulus-stimulus associations and encoding mismatch signals, consistent with the literature (Gluck and Myers, 1993; Bunsey and Eichenbaum, 1995; Eichenbaum and Bunsey, 1995; Ploghaus et al., 2000; Chen et al., 2011). The main findings from experiments 1-3 support the hypothesis that the MTL and BG are engaged simultaneously during feedback (experiments 1-3) and

197 180 observation (experiments 1 and 2) probabilistic learning and operate in parallel, by at times exhibiting parallel processing, and at times interacting directly via cooperation or competition. Evidence supporting each manner of interaction is summarized below: Evidence of parallel engagement: Parallel processing for the purpose of this dissertation is defined as the engagement of multiple memory regions in the same learning session, while displaying unique patterns of BOLD activation. This result suggests that both areas have the capability of being engaged simultaneously during the learning process, but are encoding unique information, in accord with White and McDonald s theory (2002). The examination of the brain substrates involved in feedback versus observation learning revealed areas of the ventral striatum which exhibited greater BOLD signal to feedback learning trials than observation learning trials in experiment 1 (as shown by a main effect of learning type in the learning phase). A similar effect was observed in experiment 2, in which regions of the ventral striatum and midbrain exhibited stronger engagement during feedback than observation learning trials (as shown by an interaction of learning type by group in the update phase). Interestingly, only regions within the basal ganglia were modified by a main effect of learning type, while no areas were identified in the MTL exhibiting such a differentiation. Several neuroimaging papers have shown that feedback and reward processing engages the ventral striatum (for review see Delgado, 2007), thus, it is not surprising that this area was recruited more strongly during feedback learning, which contained outcome-relevant information. It may have been expected that

198 181 the MTL would be selectively modulated during observation learning, as previously reported (Poldrack et al., 2001); however this effect was not observed. While a null result in neuroimaging is not indicative of any particular finding and the details of the paradigms employed in this dissertation differ from previous probabilistic learning studies, it is possible that the MTL BOLD signals observed in experiments 1 and 2 indicated mutual activation during both feedback and observation learning as suggested by the main effect of cue difficulty analyses. Therefore, this result suggests that at times, memory systems may operate independently and encode unique information. In further support of parallel engagement during learning, Granger causality analyses, which allowed for an investigation of both functional and effective connectivity across the brain during feedback and observation learning distinctly (experiment 1) and concurrently (experiment 2), revealed recruitment of a common network of neural substrates during learning. Results from both experiments supported the engagement of a similar network of regions, including the striatum, medial temporal lobe, and midbrain. In experiment 1, using the hippocampus as the seed region revealed functional connectivity with the caudate nucleus, while using the caudate nucleus as a seed region revealed connectivity with the hippocampus. In experiment 2, using the midbrain as a reference region elicited functional connectivity with a larger network of regions including the caudate nucleus/putamen, globus pallidus, hippocampus, and parahippocampus. Furthermore, effective connectivity results implicated a potential unilateral flow of information whereby the putamen sent directed

199 182 influences to the midbrain (VTA) and the midbrain sent directed influences to the hippocampus and parahippocampus. This observation that key learning and memory regions were functionally and effectively connected during probabilistic learning is strengthened by the fact that three reference regions employed in the analyses: the caudate nucleus, the hippocampus, and midbrain, all produced predominately converging results. The theoretical implications of these results suggest that rather than the MTL, BG, and midbrain functioning independently during learning, instead dynamic interactions may be occurring between these areas such that they influence each other during both initial acquisition and reversal of probabilistic information. Effective connectivity results suggest that this interaction may be occurring via connectivity with midbrain dopaminergic areas. Evidence of cooperative interactions: Cooperative interactions for the purpose of this dissertation are defined as the exhibition of similar patterns of BOLD signal in both the MTL and BG during learning as well as positive correlations across these distinct regions. In experiment 1, a region of the caudate nucleus and hippocampus both displayed greater BOLD signal to easy compared with hard probabilistic cues for feedback and observation trials. In experiment 2, one region of the basal ganglia, the putamen, as well as multiple areas of the MTL, including the hippocampus and parahippocampus, exhibited greater BOLD signal in response to both deterministic and easy compared with hard cues. This result is in agreement with studies in the literature which have also documented similar patterns of activity in the MTL and BG during simple

200 183 visuomotor learning tasks (Toni et al., 2001; Amso et al., 2005; Law et al., 2005; Haruno and Kawato, 2006) as well as information integration category learning tasks (Cincotta and Seger, 2007). Furthermore, multiple positive correlations across distinct regions (MTL, BG, and midbrain) were observed in all three experiments. In fact negative correlations, which have previously been used as evidence of competition (Poldrack et al., 2001; Seger and Cincotta, 2006), were never observed across these distinct learning and memory structures, even when the BOLD signal displayed opposite patterns of activity across regions (experiment 2) as well as when the BOLD signal was investigated using independent regions of interest (experiment 3). Lastly, both loci within the MTL and BG were involved in encoding a feedback-based prediction error signal in experiments 1 and 2. This result suggests three things, one that a feedback learning signal is encoded in the MTL, in addition to the basal ganglia, which has previously been shown in the literature (O'Doherty et al., 2004; Tanaka et al., 2004; Pessiglione et al., 2006); second that distinct memory regions may at times encode the same learning signal; and third, that dopamine may be facilitating interactions among discrete regions during learning in accord with the effective connectivity results observed in this dissertation as well as the theory that dopamine modulates reward-related learning (Lisman and Grace, 2005; Shohamy and Adcock, 2010). Evidence of competitive interactions: Competitive interactions in this thesis are defined as the exhibition of opposing patterns of BOLD signal in both the MTL and BG during learning and negative correlations across distinct

201 184 regions. In experiment 2, multiple regions of the basal ganglia, including the ventral striatum, caudate nucleus, and putamen, as well as multiple areas of the MTL, including the hippocampus and parahippocampus, exhibited opposing patterns of activation in response to varying levels of probabilistic cue difficulty. Specifically, the MTL displayed greater responses to easy than hard cues, whereas the loci within the basal ganglia exhibited greater activation in response to hard than easy cues. Opposing patterns of BOLD signal across distinct brain regions has been interpreted as competitive interactions in the literature (Poldrack et al., 2001; Foerde et al., 2006). 6.2 Limitations Perhaps the largest limitation of this dissertation is that it is very difficult to examine interactions between multiple memory systems using functional magnetic resonance imaging. While this question is an active area of research in the functional neuroimaging field, it is one that contains rather significant challenges. The primary concern is that using BOLD as a dependent measure, it is difficult to examine if distinct regions are actually functionally interacting. Simply because the pattern of BOLD responses observed in discrete regions is similar or distinct, does not necessarily indicate that the regions are interacting. Therefore, the definitions of cooperative and competitive interactions during learning used in this dissertation, while consistent with the literature, should be considered with some amount of caution. Specifically because of this limitation, more sophisticated analyses were employed including a Granger causality

202 185 analysis as well as a prediction error analysis in order to explore in a more in depth manner how distinct memory systems interact during learning. The major differences between the feedback and observation versions of tasks employed in this dissertation were outlined in the materials and methods sections of experiments 1 and 2. Despite their differences, however, the two learning sessions share the common goal of learning the value of probabilistic cues. As such, participants may have engaged in a variety of cognitive strategies in order to successfully complete the task. For example, as learning progressed over time in the feedback session it is possible that during the stimulus presentation period participants may have employed a more declarativebased cued-recall strategy. Participants may have also engaged in verbal rehearsal strategies, irrespective of the task version, during the learning phase. Research investigating how participants solve a distinct probabilistic learning task, the WPT, may provide insight into possible declarative and nondeclarative components of category learning in addition to exploring the declarative knowledge that participants posses during these types of learning tasks (Gluck et al., 2002; Meeter et al., 2006). Newell and colleagues (2007) recently reported that participants had comparable declarative knowledge on a feedback and observation version of the WPT. As a result, the authors suggested that the feedback version of the WPT may not be an exclusively nondeclarative task. Corroborating this theory, Meeter and colleagues (2008) have suggested that participants may solve the WPT via several distinct strategies including engagement of rule-learning, incremental learning (both of which are thought to

203 186 engage the BG), memorization techniques (MTL dependent), or some combination of these three strategies. Furthermore, Shohamy and colleagues (2004b) suggest that in order to solve probabilistic categorization tasks participants most likely recruit multiple parallel learning systems. Therefore, it is quite possible that both the observation and feedback versions of the tasks employed in experiments 1 and 2 contain some declarative and nondeclarative components. As a result, the use of the terms declarative and nondeclarative in this context are meant as a reference, and are not meant to indicate the sole manner in which participants may solve the tasks. The possibility that the tasks used in experiments 1 and 2 contain elements of declarative and nondeclarative learning may have contributed to the observation that the hippocampus and striatum were involved in both observation and feedback learning primarily modified by cue difficulty rather than learning type. The involvement of multiple cognitive operations (e.g., cued recall or rehearsal strategies) therefore, may have facilitated the mutual engagement of these regions. One limitation of the studies employed in this dissertation is that this possibility cannot be eliminated definitively. Future studies may be better able to parse out the distinct contributions that multiple cognitive processes may have on these tasks and the subsequent BOLD signals observed in the MTL and BG. In addition, some limitations of the prediction error analyses employed in experiments 1 and 2 exist. Namely, it is difficult to assess at which time point in the trial (given the task designs specifically no jittered ISI between the cue and

204 187 feedback presentation periods) the individual correlations with the PE signals occurred within the MTL and BG. Further, it is possible that the trial structure employed did not allow separation of the prediction error signal from an uncertainty signal (between cue and outcome) that has been shown to be linked with local field potentials in the anterior hippocampus (Vanni-Mercier et al., 2009). However, PE signals carry valence information (i.e., an unexpected negative outcome produces a negative PE signal), while a saliency, novelty, or uncertainty signal may be positive irrespective of the valence of the outcome (more akin to the action prediction error used for the observation trials in experiment 2). The finding of feedback trial PE signals in the MTL is not commonly reported in the literature and may point to an area of future research investigating the nature of these signals in the MTL and potential interactions with the striatum that may underlie parallel processing in these distinct regions during probabilistic learning. 6.3 Conclusions & Future Directions To conclude, the set of experiments performed in this dissertation support the model of parallel processing of the medial temporal lobe and basal ganglia during probabilistic learning. However, several questions remain unaddressed. For example, it is unknown under what circumstances memory systems cooperate versus compete. Experiment 3 of this dissertation was designed to address this question, but the results were somewhat inconclusive. A second open question is probing how exactly dopamine modulates motivated learning in

205 188 the hippocampus and other learning structures (for review see Shohamy and Adcock, 2010). Lastly, other neurotransmitters are involved in learning and memory functions, in addition to dopamine, for example acetylcholine (Hasselmo, 2006). The role that multiple neurotransmitters play during human probabilistic learning also remains to be fully characterized. These are key questions which will help to illustrate how two major learning and memory systems interact, which has implications not only for healthy humans, but may also impact our understanding and treatment of several neuropsychiatric disorders that target these structures, including Parkinson s disease, schizophrenia, and MTL amnesia.

206 189 References Abler B, Walter H, Erk S, Kammerer H, Spitzer M (2006) Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 31: Adcock RA, Thangavel A, Whitfield-Gabrieli S, Knutson B, Gabrieli JD (2006) Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 50: Albouy G, Sterpenich V, Balteau E, Vandewalle G, Desseilles M, Dang-Vu T, Darsaud A, Ruby P, Luppi PH, Degueldre C, Peigneux P, Luxen A, Maquet P (2008) Both the hippocampus and striatum are involved in consolidation of motor sequence memory. Neuron 58: Amaral DG (1993) Emerging principles of intrinsic hippocampal organization. Curr Opin Neurobiol 3: Amso D, Davidson MC, Johnson SP, Glover G, Casey BJ (2005) Contributions of the hippocampus and the striatum to simple association and frequencybased learning. Neuroimage 27: Andersen P, Bliss, T.V.P., Skrede, K.K. (1971) Lamellar organization of hippocampal excitatory pathways. Exp Brain Res 13: Annett LE, McGregor A, Robbins TW (1989) The effects of ibotenic acid lesions of the nucleus accumbens on spatial learning and extinction in the rat. Behav Brain Res 31: Aron AR, Shohamy D, Clark J, Myers C, Gluck MA, Poldrack RA (2004) Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning. J Neurophysiol 92: Ashby FG, Spiering BJ (2004) The Neurobiology of Category Learning. Behavioral and Cognitive Neuroscience Reviews.3:pp.

207 190 Ashby FG, Maddox WT, Bohil CJ (2002) Observational versus feedback training in rule-based and information-integration category learning. Mem Cognit 30: Atallah HE, Rudy JW, O'Reilly RC (2008) The role of the dorsal striatum and dorsal hippocampus in probabilistic and deterministic odor discrimination tasks. Learn Mem 15: Atallah HE, Lopez-Paniagua D, Rudy JW, O'Reilly RC (2007) Separate neural substrates for skill learning and performance in the ventral and dorsal striatum. Nat Neurosci 10: Ballard IC, Murty VP, Carter RM, MacInnes JJ, Huettel SA, Adcock RA (2011) Dorsolateral prefrontal cortex drives mesolimbic dopaminergic regions to initiate motivated behavior. J Neurosci 31: Balleine BW, Delgado MR, Hikosaka O (2007) The role of the dorsal striatum in reward and decision-making. J Neurosci 27: Belova MA, Paton JJ, Morrison SE, Salzman CD (2007) Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55: Beninger RJ, Wasserman J, Zanibbi K, Charbonneau D, Mangels J, Beninger BV (2003) Typical and atypical antipsychotic medications differentially affect two nondeclarative memory tasks in schizophrenic patients: A double dissociation. Schizophrenia Research.61:pp. Berger TW, and Orr, W.B. (1983) Hippocampectomy selectively disrupts discrimination reversal conditioning of the rabbit nictitating membrane response. Behavioral Brain Research 8: Bliss TV, Collingridge GL (1993) A synaptic model of memory: Long-term potentiation in the hippocampus. Jan Nature.361:pp. Bolam JP, Hanley JJ, Booth PA, Bevan MD (2000) Synaptic organisation of the basal ganglia. J Anat 196 ( Pt 4):

208 191 Bray S, O'Doherty J (2007) Neural coding of reward-prediction error signals during classical conditioning with attractive faces. J Neurophysiol 97: Brewer JB, Zhao Z, Desmond JE, Glover GH, Gabrieli JD (1998) Making memories: brain activity that predicts how well visual experience will be remembered. Science 281: Brog JS, Salyapongse A, Deutch AY, Zahm DS (1993) The patterns of afferent innervation of the core and shell in the "accumbens" part of the rat ventral striatum: immunohistochemical detection of retrogradely transported fluoro-gold. J Comp Neurol 338: Brown RM, Robertson EM (2007) Off-line processing: Reciprocal interactions between declarative and procedural memories. Journal of Neuroscience 27: Buckner RL, Wheeler ME (2001) The cognitive neuroscience of remembering. Nat Rev Neurosci 2: Bunsey M, Eichenbaum H (1995) Selective damage to the hippocampal region blocks long-term retention of a natural and nonspatial stimulus-stimulus association. Hippocampus 5: Burke CJ, Tobler PN, Baddeley M, Schultz W (2010) Neural mechanisms of observational learning. Proc Natl Acad Sci U S A 107: Burwell RD, Witter MP, Amaral DG (1995) Perirhinal and postrhinal cortices of the rat: a review of the neuroanatomical literature and comparison with findings from the monkey brain. Hippocampus 5: Carrillo MC, Gabrieli JD, Hopkins RO, McGlinchey-Berroth R, Fortier CB, Kesner RP, Disterhoft JF (2001) Spared discrimination and impaired reversal eyeblink conditioning in patients with temporal lobe amnesia. Behav Neurosci 115: Carter RM, Macinnes JJ, Huettel SA, Adcock RA (2009) Activation in the VTA and nucleus accumbens increases in anticipation of both gains and losses. Front Behav Neurosci 3:21.

209 192 Chen J, Olsen RK, Preston AR, Glover GH, Wagner AD (2011) Associative retrieval processes in the human medial temporal lobe: Hippocampal retrieval success and CA1 mismatch detection. Learn Mem 18: Cincotta CM, Seger CA (2007) Dissociation between striatal regions while learning to categorize via feedback and via observation. J Cogn Neurosci 19: Cools R (2006) Dopaminergic modulation of cognitive function-implications for L- DOPA treatment in Parkinson's disease. Neurosci Biobehav Rev 30:1-23. Cools R, Clark L, Owen AM, Robbins TW (2002) Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. J Neurosci 22: Corkin S, Sullivan EV, Carr FA (1984) Prognostic factors for life expectancy after penetrating head injury. Arch Neurol 41: Dagher A, Owen AM, Boecker H, Brooks DJ (2001) The role of the striatum and hippocampus in planning: a PET activation study in Parkinson's disease. Brain 124: Davachi L, Wagner AD (2002) Hippocampal contributions to episodic encoding: insights from relational and item-based learning. J Neurophysiol 88: Delgado MR (2007) Reward-related responses in the human striatum. Ann N Y Acad Sci 1104: Delgado MR, Locke HM, Stenger VA, Fiez JA (2003) Dorsal striatum responses to reward and punishment: effects of valence and magnitude manipulations. Cogn Affect Behav Neurosci 3: Delgado MR, Miller MM, Inati S, Phelps EA (2005) An fmri study of rewardrelated probability learning. Neuroimage 24: Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA (2000) Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84:

210 193 Devan BD, White NM (1999) Parallel information processing in the dorsal striatum: Relation to hippocampal function. Apr The Journal of Neuroscience.19:pp. Djonlagic I, Rosenfeld A, Shohamy D, Myers C, Gluck M, Stickgold R (2009) Sleep enhances category learning. Learn Mem 16: Doeller CF, King JA, Burgess N (2008) Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory. Proc Natl Acad Sci U S A 105: Doll BB, Jacobs WJ, Sanfey AG, Frank MJ (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 1299: Domesick VB (1969) Projections from the cingulate cortex in the rat. Brain Res 12: Eichenbaum H, Bunsey M (1995) On the binding of associations in memory: Clues from studies on the role of the hippocampal region in pairedassociate learning. Feb Current Directions in Psychological Science.4:pp. Eldridge LL, Knowlton BJ, Furmanski CS, Bookheimer SY, Engel SA (2000) Remembering episodes: a selective role for the hippocampus during retrieval. Nat Neurosci 3: Fagan AM, Olton DS (1986) Learning sets, discrimination reversal, and hippocampal function. Behav Brain Res 21: Foerde K, Knowlton BJ, Poldrack RA (2006) Modulation of competing memory systems by distraction. Proc Natl Acad Sci U S A 103: Foerde K, Poldrack RA, Knowlton BJ (2007) Secondary-task effects on classification learning. Mem Cognit 35: Forman SD, Cohen JD, Fitzgerald M, Eddy WF, Mintun MA, Noll DC (1995) Improved assessment of significant activation in functional magnetic

211 194 resonance imaging (fmri): use of a cluster-size threshold. Magn Reson Med 33: Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17: Frank MJ, Claus ED (2006) Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol Rev 113: Frank MJ, Seeberger LC, O'Reilly R C (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306: Frey U, Schroeder H, Matthies H (1990) Dopaminergic antagonists prevent longterm maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res 522: Frey U, Matthies H, Reymann KG (1991) The effect of dopaminergic D1 receptor blockade during tetanization on the expression of long-term potentiation in the rat CA1 region in vitro. Neurosci Lett 129: Gerfen CR (1984) The neostriatal mosaic: compartmentalization of corticostriatal input and striatonigral output systems. Nature 311: Geweke J (1982) Measurement of Linear Dependence and Feedback between Multiple Time Series. J Am Stat Assoc 77: Gluck MA, Bower GH (1988) From conditioning to category learning: an adaptive network model. J Exp Psychol Gen 117: Gluck MA, Myers CE (1993) Hippocampal mediation of stimulus representation: a computational theory. Hippocampus 3: Gluck MA, Shohamy D, Myers C (2002) How do people solve the "weather prediction" task?: individual variability in strategies for probabilistic category learning. Learn Mem 9:

212 195 Gluck MA, Meeter M, Myers CE (2003) Computational models of the hippocampal region: linking incremental learning and episodic memory. Trends Cogn Sci 7: Goebel R, Esposito F, Formisano E (2006) Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis. Hum Brain Mapp 27: Goebel R, Roebroeck A, Kim DS, Formisano E (2003) Investigating directed cortical interactions in time-resolved fmri data using vector autoregressive modeling and Granger causality mapping. Magn Reson Imaging 21: Gotham AM, Brown RG, Marsden CD (1988) 'Frontal' cognitive function in patients with Parkinson's disease 'on' and 'off' levodopa. Brain 111 ( Pt 2): Groenewegen HJ, Vermeulen-Van der Zee E, te Kortschot A, Witter MP (1987) Organization of the projections from the subiculum to the ventral striatum in the rat. A study using anterograde transport of Phaseolus vulgaris leucoagglutinin. Neuroscience 23: Haber SN (2003) The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 26: Haber SN, Knutson B (2010) The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35:4-26. Hampson RE, Pons TP, Stanford TR, Deadwyler SA (2004) Categorization in the monkey hippocampus: a possible mechanism for encoding information into memory. Proc Natl Acad Sci U S A 101: Haruno M, Kawato M (2006) Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J Neurophysiol 95: Hasselmo ME (2006) The role of acetylcholine in learning and memory. Curr Opin Neurobiol 16:

213 196 Heeger DJ, Ress D (2002) What does fmri tell us about neuronal activity? Nat Rev Neurosci 3: Heimer L, Alheid GF, de Olmos JS, Groenewegen HJ, Haber SN, Harlan RE, Zahm DS (1997) The accumbens: beyond the core-shell dichotomy. J Neuropsychiatry Clin Neurosci 9: Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6: Hopkins RO, Myers CE, Shohamy D, Grossman S, Gluck M (2004) Impaired probabilistic category learning in hypoxic subjects with hippocampal damage. Neuropsychologia 42: Jay TM, Glowinski J, Thierry AM (1989) Selectivity of the hippocampal projection to the prelimbic area of the prefrontal cortex in the rat. Brain Res 505: Joel D, Niv Y, Ruppin E (2002) Actor-critic models of basal ganglia: New anatomical and computational perspectives. Neural Networks.15:pp. Jung Y, Hong S, Haber SN (2003) Organization of Direct Hippocampal Projections to the Different Regions of the Ventral Striatum in primate. The Korean J Anat 36: Kelley AE, Domesick VB (1982) The distribution of the projection from the hippocampal formation to the nucleus accumbens in the rat: an anterograde- and retrograde-horseradish peroxidase study. Neuroscience 7: Knowlton BJ, Squire LR, Gluck MA (1994) Probabilistic classification learning in amnesia. Learn Mem 1: Knowlton BJ, Mangels JA, Squire LR (1996) A neostriatal habit learning system in humans. Science 273: Knutson B, Adams CM, Fong GW, Hommer D (2001) Anticipation of increasing monetary reward selectively recruits nucleus accumbens. The Journal of Neuroscience.21:pp.

214 197 Kolb B, Buhrmann K, McDonald R, Sutherland RJ (1994) Dissociation of the medial prefrontal, posterior parietal, and posterior temporal cortex for spatial navigation and recognition memory in the rat. Cereb Cortex 4: Krayniak PF, Meibach RC, Siegel A (1981) A projection from the entorhinal cortex to the nucleus accumbens in the rat. Brain Res 209: Lagnado DA, Newell BR, Kahan S, Shanks DR (2006) Insight and strategy in multiple-cue learning. J Exp Psychol Gen 135: Law JR, Flanery MA, Wirth S, Yanike M, Smith AC, Frank LM, Suzuki WA, Brown EN, Stark CE (2005) Functional magnetic resonance imaging activity during the gradual acquisition and expression of paired-associate memory. J Neurosci 25: Lee AS, Duman RS, Pittenger C (2008) A double dissociation revealing bidirectional competition between striatum and hippocampus during learning. Proc Natl Acad Sci U S A 105: Li J, Delgado MR, Phelps EA (2011) How instructed knowledge modulates the neural systems of reward learning. Proc Natl Acad Sci U S A 108: Li S, Cullen WK, Anwyl R, Rowan MJ (2003) Dopamine-dependent facilitation of LTP induction in hippocampal CA1 by exposure to spatial novelty. Nature Neuroscience.6:pp. Lisman JE, Grace AA (2005) The hippocampal-vta loop: controlling the entry of information into long-term memory. Neuron 46: Liu P, Bilkey DK (1996) Direct connection between perirhinal cortex and hippocampus is a major constituent of the lateral perforant path. Hippocampus 6: Lynd-Balta E, Haber SN (1994) The organization of midbrain projections to the ventral striatum in the primate. Neuroscience 59: Macmillan NA, Creelman CD (1991) Detection Theory: A User's Guide. New York: Cambridge University Press.

215 198 Marco-Pallares J, Muller SV, Munte TF (2007) Learning by doing: an fmri study of feedback-related brain activations. Neuroreport 18: Marston HM, Everitt BJ, Robbins TW (1993) Comparative effects of excitotoxic lesions of the hippocampus and septum/diagonal band on conditional visual discrimination and spatial learning. Neuropsychologia 31: Mattfeld AT, Stark CE (2010) Striatal and Medial Temporal Lobe Functional Interactions during Visuomotor Associative Learning. Cereb Cortex. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38: McDonald RJ, White NM (1993) A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behav Neurosci 107:3-22. McDonald RJ, White NM (1994) Parallel information processing in the water maze: evidence for independent memory systems involving dorsal striatum and hippocampus. Behav Neural Biol 61: McDonald RJ, White NM (1995) Hippocampal and nonhippocampal contributions to place learning in rats. Behavioral Neuroscience.109:pp. McGeorge AJ, Faull RL (1989) The organization of the projection from the cerebral cortex to the striatum in the rat. Neuroscience 29: Meeter M, Myers CE, Shohamy D, Hopkins RO, Gluck MA (2006) Strategies in probabilistic categorization: results from a new way of analyzing performance. Learn Mem 13: Meeter M, Radics G, Myers CE, Gluck MA, Hopkins RO (2008) Probabilistic categorization: how do normal participants and amnesic patients do it? Neurosci Biobehav Rev 32: Meyer-Lindenberg A, Kohn PD, Kolachana B, Kippenhan S, McInerney-Leo A, Nussbaum R, Weinberger DR, Berman KF (2005) Midbrain dopamine and

216 199 prefrontal function in humans: interaction and modulation by COMT genotype. Nat Neurosci 8: Middleton FA, Strick PL (2000) Basal ganglia output and cognition: evidence from anatomical, behavioral, and clinical studies. Brain Cogn 42: Mitchell JA, Hall G (1988) Learning in rats with caudate-putamen lesions: unimpaired classical conditioning and beneficial effects of redundant stimulus cues on instrumental and spatial learning deficits. Behav Neurosci 102: Myers CE, Deluca J, Hopkins RO, Gluck MA (2006) Conditional discrimination and reversal in amnesia subsequent to hypoxic brain injury or anterior communicating artery aneurysm rupture. Neuropsychologia 44: Myers CE, Shohamy D, Gluck MA, Grossman S, Kluger A, Ferris S, Golomb J, Schnirman G, Schwartz R (2003) Dissociating hippocampal versus basal ganglia contributions to learning and transfer. J Cogn Neurosci 15: Myers CE, Hopkins, R.O., Kesner, R.P, Monti, L., Gluck, M.A. (2000) Conditional spatial discrimination in humans with hypoxic brain injury. Psychobiology 28: Nagy H, Keri S, Myers CE, Benedek G, Shohamy D, Gluck MA (2007) Cognitive sequence learning in Parkinson's disease and amnestic mild cognitive impairment: Dissociation between sequential and non-sequential learning of associations. Neuropsychologia 45: Newell BR, Lagnado DA, Shanks DR (2007) Challenging the role of implicit processes in probabilistic category learning. Psychon Bull Rev 14: O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: O'Doherty JP (2004) Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr Opin Neurobiol 14:

217 200 O'Doherty JP, Deichmann R, Critchley HD, Dolan RJ (2002) Neural responses during anticipation of a primary taste reward. Neuron 33: O'Reilly R, C., Munakata Y (2000) Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain. Cambridge, MA: MIT Press. Olmos A, Kingdom FA (2004) A biologically inspired algorithm for the recovery of shading and reflectance images. Perception 33: Packard MG (1999) Glutamate infused posttraining into the hippocampus or caudate-putamen differentially strengthens place and response learning. Proc Natl Acad Sci U S A 96: Packard MG, White NM (1989) Memory facilitation produced by dopamine agonists: Role of receptor subtype and mnemonic requirements. Jul Pharmacology, Biochemistry and Behavior.33:pp. Packard MG, White NM (1991) Dissociation of hippocampus and caudate nucleus memory systems by posttraining intracerebral injection of dopamine agonists. Behav Neurosci 105: Packard MG, McGaugh JL (1992) Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: Further evidence for multiple memory systems. Behavioral Neuroscience.106: Packard MG, McGaugh JL (1994) Quinpirole and d-amphetamine administration posttraining enhances memory on spatial and cued discriminations in a water maze. Mar Psychobiology.22:pp. Packard MG, McGaugh JL (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65: Packard MG, Knowlton BJ (2002) Learning and memory functions of the Basal Ganglia. Annu Rev Neurosci 25:

218 201 Packard MG, Hirsh R, White NM (1989) Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems. J Neurosci 9: Pagnoni G, Zink CF, Montague PR, Berns GS (2002) Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci 5: Penfield W, Milner B (1958) Memory deficit produced by bilateral lesions in the hippocampal zone. AMA Arch Neurol Psychiatry 79: Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopaminedependent prediction errors underpin reward-seeking behaviour in humans. Nature 442: Ploghaus A, Tracey I, Clare S, Gati JS, Rawlins JN, Matthews PM (2000) Learning about pain: the neural substrate of the prediction error for aversive events. Proc Natl Acad Sci U S A 97: Poldrack RA, Packard MG (2003) Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia 41: Poldrack RA, Rodriguez P (2004) How do memory systems interact? Evidence from human classification learning. Neurobiol Learn Mem 82: Poldrack RA, Prabhakaran V, Seger CA, Gabrieli JD (1999) Striatal activation during acquisition of a cognitive skill. Neuropsychology 13: Poldrack RA, Fletcher PC, Henson RN, Worsley KJ, Brett M, Nichols TE (2008) Guidelines for reporting an fmri study. Neuroimage 40: Poldrack RA, Clark J, Pare-Blagoev EJ, Shohamy D, Creso Moyano J, Myers C, Gluck MA (2001) Interactive memory systems in the human brain. Nature 414: Reber PJ, Knowlton BJ, Squire LR (1996) Dissociable properties of memory systems: differences in the flexibility of declarative and nondeclarative knowledge. Behav Neurosci 110:

219 202 Rempel-Clower NL, Zola SM, Squire LR, Amaral DG (1996) Three cases of enduring memory impairment after bilateral damage limited to the hippocampal formation. J Neurosci 16: Rice WR (1989) Analyzing Tables of Statistical Tests. Evolution 43: Roebroeck A, Formisano E, Goebel R (2005) Mapping directed influence over the brain using Granger causality and fmri. Neuroimage 25: Rolls ET (1994) Neurophysiology and cognitive functions of the striatum. Aug- Sep Revue Neurologique.150:pp. Sadeh T, Shohamy D, Levy DR, Reggev N, Maril A (2011) Cooperation between the hippocampus and the striatum during episodic encoding. J Cogn Neurosci 23: Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310: Scatton B, Simon H, Le Moal M, Bischoff S (1980) Origin of dopaminergic innervation of the rat hippocampal formation. Neurosci Lett 18: Schacter DL, Wagner AD (1999) Medial temporal lobe activations in fmri and PET studies of episodic encoding and retrieval. Hippocampus 9:7-24. Schmitt-Eliassen J, Ferstl R, Wiesner C, Deuschl G, Witt K (2007) Feedbackbased versus observational classification learning in healthy aging and Parkinson's disease. Brain Res 1142: Schoenbaum G, Setlow B (2003) Lesions of nucleus accumbens disrupt learning about aversive outcomes. J Neurosci 23: Schott BH, Seidenbecher CI, Fenker DB, Lauer CJ, Bunzeck N, Bernstein HG, Tischmeyer W, Gundelfinger ED, Heinze HJ, Duzel E (2006) The dopaminergic midbrain participates in human episodic memory formation: evidence from genetic imaging. J Neurosci 26:

220 203 Schultz W (1997) Dopamine neurons and their role in reward mechanisms. Curr Opin Neurobiol 7: Schultz W (2002) Getting formal with dopamine and reward. Neuron 36: Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: Scoville WB, Milner B (1957) Loss of recent memory after bilateral hippocampal lesions. J Neurol Neurosurg Psychiatry 20: Seger CA (2006) The basal ganglia in human learning. Neuroscientist 12: Seger CA (2008) How do the basal ganglia contribute to categorization? Their roles in generalization, response selection, and learning via feedback. Neurosci Biobehav Rev 32: Seger CA, Cincotta CM (2005) The roles of the caudate nucleus in human classification learning. J Neurosci 25: Seger CA, Cincotta CM (2006) Dynamics of frontal, striatal, and hippocampal systems during rule learning. Cereb Cortex 16: Seger CA, Miller EK (2010) Category learning in the brain. Annu Rev Neurosci 33: Setlow B, Schoenbaum G, Gallagher M (2003) Neural encoding in ventral striatum during olfactory discrimination learning. Neuron 38: Sherry DF, Schacter DL (1987) The evolution of multiple memory systems. Psychological Review 94: Shohamy D, Wagner AD (2008) Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 60:

221 204 Shohamy D, Adcock RA (2010) Dopamine and adaptive memory. Trends Cogn Sci 14. Shohamy D, Myers CE, Onlaor S, Gluck MA (2004a) Role of the basal ganglia in category learning: how do patients with Parkinson's disease learn? Behav Neurosci 118: Shohamy D, Myers CE, Kalanithi J, Gluck MA (2008) Basal ganglia and dopamine contributions to probabilistic category learning. Neurosci Biobehav Rev 32: Shohamy D, Myers CE, Grossman S, Sage J, Gluck MA (2005) The role of dopamine in cognitive sequence learning: evidence from Parkinson's disease. Behav Brain Res 156: Shohamy D, Myers CE, Geghman KD, Sage J, Gluck MA (2006) L-dopa impairs learning, but spares generalization, in Parkinson's disease. Neuropsychologia 44: Shohamy D, Myers CE, Hopkins RO, Sage J, Gluck MA (2009) Distinct hippocampal and basal ganglia contributions to probabilistic learning and reversal. J Cogn Neurosci 21: Shohamy D, Myers CE, Grossman S, Sage J, Gluck MA, Poldrack RA (2004b) Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. Brain 127: Shrager Y, Kirwan CB, Squire LR (2008) Activity in both hippocampus and perirhinal cortex predicts the memory strength of subsequently remembered information. Neuron 59: Sorensen KE, Witter MP (1983) Entorhinal efferents reach the caudato-putamen. Neurosci Lett 35: Squire LR (1992a) Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychol Rev 99:

222 205 Squire LR (1992b) Declarative and nondeclarative memory: Multiple brain systems supporting learning and memory. Journal of Cognitive Neuroscience 4: Squire LR, Zola SM (1996) Structure and function of declarative and nondeclarative memory systems. Proc Natl Acad Sci U S A 93: Squire LR, Stark CE, Clark RE (2004) The medial temporal lobe. Annu Rev Neurosci 27: Stark CE, Squire LR (2001) When zero is not zero: the problem of ambiguous baseline conditions in fmri. Proc Natl Acad Sci U S A 98: Stubley-Weatherly L, Harding JW, Wright JW (1996) Effects of discrete kainic acid-induced hippocampal lesions on spatial and contextual learning and memory in rats. Brain Res 716: Sutherland RJ, Kolb B, Whishaw IQ (1982) Spatial mapping: definitive disruption by hippocampal or medial frontal cortical damage in the rat. Neurosci Lett 31: Sutherland RJ, Whishaw IQ, Kolb B (1988) Contributions of cingulate cortex to two forms of spatial learning and memory. J Neurosci 8: Swainson R, Rogers RD, Sahakian BJ, Summers BA, Polkey CE, Robbins TW (2000) Probabilistic learning and reversal deficits in patients with Parkinson's disease or frontal or temporal lobe lesions: possible adverse effects of dopaminergic medication. Neuropsychologia 38: Swanson LW (1981) A direct projection from Ammon's horn to prefrontal cortex in the rat. Brain Res 217: Swanson LW (1982) The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res Bull 9:

223 206 Swanson LW, Kohler C (1986) Anatomical evidence for direct projections from the entorhinal area to the entire cortical mantle in the rat. J Neurosci 6: Talairach J, Tournoux P (1988) Co-Planar Stereotaxic Atlas of the Human Brain. New York: Thieme Medical Publishers, Inc. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits corticobasal ganglia loops. Nat Neurosci 7: Thierry AM, Gioanni Y, Degenetais E, Glowinski J (2000) Hippocampo-prefrontal cortex pathway: anatomical and electrophysiological characteristics. Hippocampus 10: Toni I, Ramnani N, Josephs O, Ashburner J, Passingham RE (2001) Learning arbitrary visuomotor associations: temporal dynamic of brain activity. Neuroimage 14: Tricomi E, Fiez JA (2008) Feedback signals in the caudate reflect goal achievement on a declarative memory task. Neuroimage 41: Tricomi EM, Delgado MR, Fiez JA (2004) Modulation of caudate activity by action contingency. Neuron 41: Vanni-Mercier G, Mauguiere F, Isnard J, Dreher JC (2009) The hippocampus codes the uncertainty of cue-outcome associations: an intracranial electrophysiological study in humans. J Neurosci 29: Voermans NC, Petersson KM, Daudey L, Weber B, Van Spaendonck KP, Kremer HP, Fernandez G (2004) Interaction between the human hippocampus and the caudate nucleus during route recognition. Neuron 43: Wagner AD, Schacter DL, Rotte M, Koutstaal W, Maril A, Dale AM, Rosen BR, Buckner RL (1998) Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. Science 281:

224 207 Watkins CJCH (1989) Learning from delayed rewards. In. England: University of Cambridge. White NM, McDonald RJ (2002) Multiple parallel memory systems in the brain of the rat. Neurobiol Learn Mem 77: Wise RA (2004) Dopamine, learning and motivation. Nat Rev Neurosci 5: Wittmann BC, Schott BH, Guderian S, Frey JU, Heinze HJ, Duzel E (2005) Reward-related FMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 45: Yin HH, Knowlton BJ (2006) The role of the basal ganglia in habit formation. Nat Rev Neurosci 7: Zeineh MM, Engel SA, Thompson PM, Bookheimer SY (2003) Dynamics of the hippocampus during encoding and retrieval of face-name pairs. Science 299: Zola-Morgan S, Squire LR (1984) Preserved learning in monkeys with medial temporal lesions: sparing of motor and cognitive skills. J Neurosci 4: Zola-Morgan S, Squire LR (1985) Medial temporal lesions in monkeys impair memory on a variety of tasks sensitive to human amnesia. Behav Neurosci 99: Zola-Morgan S, Squire LR (1986) Memory impairment in monkeys following lesions limited to the hippocampus. Behavioral Neuroscience 100: Zola SM, Mahut H (1973) Paradoxical facilitation of object reversal learning after transection of the fornix in monkeys. Neuropsychologia 11:

225 208 Figures Figure 3.1 Schematic of the Task used in the Experiment 1

226 Figure 3.2 Experiment 1 Behavioral Results 209

227 Figure 3.3 Experiment 1 Neuroimaging Results: Main Effect of Cue Difficulty in the Striatum and Hippocampus 210

228 Figure 3.4 Experiment 1 Neuroimaging Results: Granger Causality Analysis Functional Connectivity in the Caudate Nucleus using the Hippocampus as the Seed Region 211

229 Figure 3.5 Experiment 1 Neuroimaging Results: Prediction Error Analysis in the Hippocampus and Striatum 212

230 Figure 4.1 Schematic of the Task used in Experiment 2 213

231 Figure 4.2 Experiment 2 Behavioral Results 214

232 Figure 4.3 Experiment 2 Neuroimaging Results: Learning Phase Main Effect of Cue Difficulty in the Striatum and MTL Feedback Group 215

233 Figure 4.4 Experiment 2 Neuroimaging Results: Learning Phase Main Effect of Cue Difficulty in the Striatum and MTL Observation Group 216

234 Figure 4.5 Experiment 2 Neuroimaging Results: Update Phase Main Effect of Cue Difficulty in the Striatum and MTL Feedback and Observation Groups 217

235 Figure 4.6 Neuroimaging Results: Granger Causality Analysis Effective Connectivity using the Midbrain as the Seed Region 218

236 Figure 4.7 Experiment 2 Neuroimaging Results: Prediction Error Analysis 219

237 Figure 5.1 Schematic of the Task used in Experiment 3 220

238 Figure 5.2 Experiment 3 Behavioral Results 221

239 Figure 5.3 Experiment 3 Neuroimaging Results: ROI Analysis during the Stimulus Presentation Period 222