Prefrontal Cortex: Delay-related activity. Spatial delayed-response task; Funahashi et al, 1989

Transcription

1 Prefrontal Cortex: Delay-related activity Spatial delayed-response task; Funahashi et al, 1989

2 Remember the SRN? (chap 6)

3 Remember the SRN? (chap 6) this is a gating network: context only updated at discrete timepoints

4 Simple SRN story is not flawless How is hidden copy function implemented biologically? During settling, context must be actively maintained(ongoing hidden activity has no effect on context). Assumes all context is relevant: What if distracting information presented in middleof sequence? Wanttoonly hold on torelevant context. What ifwantto holdon tomore than one piece of information inwm atatime?? Or toselectively updateone partof WM whilecontinuing to robustly maintain others? And whatif thedecision of whether ornot toupdateinformation depends on currently internal WM state?

5 Schizophrenia: Impaired PFC/BG Function SZ Patientshave deficitsin the samecognitive tasksasthose seen in patients with PFC damage.

6 Schizophrenia: Impaired PFC/BG Function SZ Patientshave deficitsin the samecognitive tasksasthose seen in patients with PFC damage. But PFC patients don t have delusions, hallucinations, psychosis. Also some imaging studies show increase in PFC activity in SZ (e.g., Manoach, 2003; Calicott et al, 2003)

7 Schizophrenia: Impaired PFC/BG Function SZ Patientshave deficitsin the samecognitive tasksasthose seen in patients with PFC damage. But PFC patients don t have delusions, hallucinations, psychosis. Also some imaging studies show increase in PFC activity in SZ (e.g., Manoach, 2003; Calicott et al, 2003) Psychosis thought to stem from increased DA in BG (Weinberger, O Donnell, Grace, etc)

8 Schizophrenia: Impaired PFC/BG Function SZ Patientshave deficitsin the samecognitive tasksasthose seen in patients with PFC damage. But PFC patients don t have delusions, hallucinations, psychosis. Also some imaging studies show increase in PFC activity in SZ (e.g., Manoach, 2003; Calicott et al, 2003) Psychosis thought to stem from increased DA in BG (Weinberger, O Donnell, Grace, etc) Other disorders with BG/DA dysfunction are associated with frontal-like cognitive deficits (PD, addiction, ADHD, etc); these deficits correlate w/da in BG (Muller et al,2000; Remyet al,2000)

9 Widely Accepted Role of Prefrontal Cortex (PFC) PFC helps keep us on task, promotes cognitive flexibility via ability to rapidly switch state(e.g., task switching): Robust maintenance of neural activity (working memory)

10 Widely Accepted Role of Prefrontal Cortex (PFC) PFC helps keep us on task, promotes cognitive flexibility via ability to rapidly switch state(e.g., task switching): Robust maintenance of neural activity (working memory) Flexibility from adaptive gating(via Basal Ganglia, BG) that switches between maintenance and rapid updating

11 Widely Accepted Role of Prefrontal Cortex (PFC) PFC helps keep us on task, promotes cognitive flexibility via ability to rapidly switch state(e.g., task switching): Robust maintenance of neural activity (working memory) Flexibility from adaptive gating(via Basal Ganglia, BG) that switches between maintenance and rapid updating Extensive interconnectivity, allowing top-down biasing of task-relevant info in other cortical / subcortical areas. In short, PFC is the central executive!

12 Working Memory Demands: Updating & Maintenance Hochreiter & Schmidhuber, 1997; Braver & Cohen, 1999; Frank et al, 2001 a) Update Working Memory Gating open Sensory Input My phone # is Working memory: robust maintenance of information, but must also have ability to be rapidly updated requires gating. You ve got toknowwhen to hold em, know whento fold em.

13 Working Memory Demands: Updating & Maintenance Hochreiter & Schmidhuber, 1997; Braver & Cohen, 1999; Frank et al, 2001 a) Update b) Maintain Working Memory Gating open closed Sensory Input My phone # is Should I Stay or Should I Go..? Working memory: robust maintenance of information, but must also have ability to be rapidly updated requires gating. You ve got toknowwhen to hold em, know whento fold em.

14 Working Memory Demands: Updating & Maintenance Hochreiter & Schmidhuber, 1997; Braver & Cohen, 1999; Frank et al, 2001 a) Update b) Maintain Working Memory Gating open closed open Sensory Input Should I Stay or Should I Go..? Jenny, I got your # c) Update: OOPS! My phone # is Working memory: robust maintenance of information, but must also have ability to be rapidly updated requires gating. You ve got toknowwhen to hold em, know whento fold em.

15 But who controls the controller?

16 BG damage deficits in motor, learning, motivation, working memory, cognitive control

17 BG Model Extension to Working Memory and Attention In the motor domain, the BG selectively facilitates one command while suppressing others (Mink, 1996)

18 BG Model Extension to Working Memory and Attention In the motor domain, the BG selectively facilitates one command while suppressing others (Mink, 1996) In parallel circuits, the BG may reinforce the updating of PFC working memory representations (Alexander et al, 1986; Frank, Loughry & O Reilly, 2001).

19 BG Model Extension to Working Memory and Attention In the motor domain, the BG selectively facilitates one command while suppressing others (Mink, 1996) In parallel circuits, the BG may reinforce the updating of PFC working memory representations (Alexander et al, 1986; Frank, Loughry & O Reilly, 2001). Dopamine in PFC supports robust maintenance over time (Lewis& O Donnell, 2000; Durstewitz & Seemans, 2002)

20 BG Model Extension to Working Memory and Attention In the motor domain, the BG selectively facilitates one command while suppressing others (Mink, 1996) In parallel circuits, the BG may reinforce the updating of PFC working memory representations (Alexander et al, 1986; Frank, Loughry & O Reilly, 2001). Dopamine in PFC supports robust maintenance over time (Lewis& O Donnell, 2000; Durstewitz & Seemans, 2002) Phasic DA bursts thought to occur for task-relevant ( positive ) information, reinforcing BG updating signals (O Reilly& Frank, 2006; Frank & O Reilly, 2006).

21 BG Model Extension to Working Memory and Attention In the motor domain, the BG selectively facilitates one command while suppressing others (Mink, 1996) In parallel circuits, the BG may reinforce the updating of PFC working memory representations (Alexander et al, 1986; Frank, Loughry & O Reilly, 2001). Dopamine in PFC supports robust maintenance over time (Lewis& O Donnell, 2000; Durstewitz & Seemans, 2002) Phasic DA bursts thought to occur for task-relevant ( positive ) information, reinforcing BG updating signals (O Reilly& Frank, 2006; Frank & O Reilly, 2006). Time course of DA activity: maintenance in PFC, updating thru BG.

22 BG Gating of PFC Working Memory / Attention

23 BG Disinhibition of PFC = Gating of Working Memory / Attention excitatory inhibitory modulatory prefrontal cortex D1 Go NoGo D2 striatum GPe SNc Direct Indirect thalamus GPi Base state: Thalamus inhibited = Gate is closed PFC robustly maintains prior states.

24 BG Disinhibition of PFC = Gating of Working Memory / Attention excitatory inhibitory modulatory prefrontal cortex Go NoGo D1 D2 striatum GPe SNc Direct Indirect thalamus GPi Base state: Thalamus inhibited = Gate is closed PFC robustly maintains prior states. Striatum fires: Thalamus disinhibited = Gate opened PFC rapidly updated to maintain new information.

25 Parallel Stripes = Selective Gating Posterior Cortex excitatory inhibitory { PFC thalamus GP (tonic act) Striatum (matrix) } dis inhib PFC/BG loops form independent stripes = selective gating.

26 BG gates flow: superficial Deep PFC Maintenance via Thalamocortical loops, BG disinhibits Superficial reflects inputs and maintenance Separate Maintenance vs. Output PFC/ BG stripes

27 Multiple stripes, competition Competition in striatum/gp between Maint vs Output and diff stripes

28 time Example Task: Specific Working Memory Demands Target (R): 1 = A-X 2 = B-Y L 1 L A R X L B L X L 2 L B R Y Must maintain outer loop (1,2) while updating inner loop (A,X..). inner loops outer loop

29 time Example Task: Specific Working Memory Demands Target (R): 1 = A-X 2 = B-Y L 1 L A R X L B L X L 2 L B R Y Must maintain outer loop (1,2) while updating inner loop (A,X..). Robust maintenance(over intervening stimuli; outer loop). inner loops outer loop

30 time Example Task: Specific Working Memory Demands Target (R): 1 = A-X 2 = B-Y L 1 L A R X L B L X L 2 L B R Y Must maintain outer loop (1,2) while updating inner loop (A,X..). Robust maintenance(over intervening stimuli; outer loop). Rapid and selective updating(keep 1,2 while updating A,X..). inner loops outer loop

31 time Example Task: Specific Working Memory Demands Target (R): 1 = A-X 2 = B-Y L 1 L A R X L B L X L 2 L B R Y Must maintain outer loop (1,2) while updating inner loop (A,X..). Robust maintenance(over intervening stimuli; outer loop). Rapid and selective updating(keep 1,2 while updating A,X..). Many real-world examples: language, planning... inner loops outer loop

32 a) PFC/BG Model of 1-2-AX (Frank, Loughry & O Reilly, 2001) 1 1 {

33 FC/BG Model of 1-2-AX (Frank, Loughry & O Reilly, 2001) a) b) 1 1 C C 1 { {

34 FC/BG Model of 1-2-AX (Frank, Loughry & O Reilly, 2001) a) b) 1 1 C C 1 { { c) A A 1 {

35 FC/BG Model of 1-2-AX (Frank, Loughry & O Reilly, 2001) a) b) 1 1 C C 1 { { c) d) A A 1 X R A 1 { {

36 But How do BG Know What to Update/Ignore?

37 But How do BG Know What to Update/Ignore? Correct: Incorrect:

38 DA effects onbg Updating OfPFC DA Burst: Increased Go DA Dip: Increased NoGo Posterior Cortex Frontal Cortex Posterior Cortex Frontal Cortex dorsal striatum dorsal striatum excitatory inhibitory modulatory D1 + Go NoGo thalamus VA,VL,MD D2 GPe excitatory inhibitory modulatory D1 + Go NoGo thalamus VA,VL,MD D2 GPe SNc SNr SNc SNr DA bursts/dips modulate Go vs. NoGo firing and learning. Burst =Excitatory D1 on Go Dip = Release ofd2 inhibition on NoGo Same mechanism as in basic motor circuitry! (Frank, 05; Gerfen, 00)

39 PFC/BG Model, Learns 12-AX and other WM tasks (O Reilly & Frank, 2006) Input X Y Z A B C Output L R X Y Z A B C X Y Z A B C X Y Z A B C X Y Z A B C PFC (Context) Hidden (Posterior Cortex) Go NoGo Go NoGo Go NoGo Go NoGo NoGo NoGo NoGo NoGo Go Go Go Go LVe LVi PVe PVi SNrThal DA SNc Striatum (Gating, Actor) PVLV (Critic)

40 Phonological Loop Task Store Phonological Loop Task pfc bg a a out go b a b c go go input s 1 a s 2 b s 3 c Recall a b c a b c a b c a b c r 1 r 2 r 3

41 Phonological Loop Task (O Reilly & Frank, 2006) Epochs to Criterion Phono Loop Training Time PBWM LSTM RBP SRN Algorithm Generalization Error %100 Phono Loop Generalization PBWM LSTM RBP SRN Algorithm PBWM = Prefrontal Basal-ganglia Working Memory; LSTM = Long Short Term Memory (Schmidhuber et al.); RBP = Recurrent BackPropagation; SRN = Simple Recurrent Network.

42 % Networks Solving Task Accuracy Training Needed to Learn 50 0 Epochs to Criterion AX Working Memory Performance 12AX Working Memory Performance Intact SZ (+ BG DA) Intact SZ (+ BG DA) Simulating BG DA increases in SZ: Effects on 12AX Working Memory Performance

43 Evidence for BG gating of PFC Instruction cue signals whether subsequent yellow information is distracting(and should not be stored in working memory). McNab & Klingberg (2008), Nature Neuroscience

44 Evidence for BG gating of PFC: Lesion patients Baier et al (2010), Journal of Neuroscience

45 Evidence for BG gating of PFC: Neuroimaging Distractor versus no-distractor contrast: NoGo! McNab & Klingberg (2008), Nature Neuroscience

46 BothBG and PFCact correlate withwm capacity Better WM function correlates with PFC(middle frontal gyrus) and BG activity in Dist vs NoDist contrast. Globus pallidus = output ofbg; stronger morenogo McNab & Klingberg (2008), Nature Neuroscience

47 Evidence for BG gating of PFC: Neuroimaging More GP activity Less unnecessary storage of distracting info in parietal cortex (spatial storage; this parietal region is sensitive to memory load).

48 Evidence for BG gating of PFC: Neuroimaging More GP activity Less unnecessary storage of distracting info in parietal cortex (spatial storage; this parietal region is sensitive to memory load). No such correlation in PFC, suggesting that BG = filter (gate). McNab & Klingberg (2008), Nature Neuroscience

49 time Testing the Model: AX-CPT Task Cohen etal(1997);barch etal (2001);Frank&O Reilly, 2006 A 5 X R B X L A 7 L 4 Y targetsequence is A -X; press Rbutton. non-targetsequences A -Y,B-X, B -Y;press L button. ignore all distractors (5,7)

50 time Testing the Model: AX-CPT Task Cohen etal(1997);barch etal (2001);Frank&O Reilly, 2006 A 5 X R B X L A 7 L 4 Y targetsequence is A -X; press Rbutton. non-targetsequences A -Y,B-X, B -Y;press L button. ignore all distractors (5,7) Tests gating and maintenance components of working memory/ attention.

51 B

52 X

53 A

54 X

55 A

56 5

57 X

58 A

59 7

60 4

61 Y

62 A

63 7

64 5

65 X

66 A

67 5

68 7

69 Y

70 A

71 X

72 B

73 4

74 5

75 4

76 X

77 Working Memory Demands: AX-CPT Task A-X target sequence on 70% of trials. B-Y is control condition, not dependent on working memory. Key conditions: B-X, A-Y

78 B-X: Working memory, performance a) Update B Working Memory B Gating open Sensory Input B

79 B-X: Working memory, performance a) Update B b) Maintain B Working Memory B B Gating open closed Sensory Input B 5

80 B-X: Working memory, performance a) Update B b) Maintain B c) Prepare NonTarg Resp. Working Memory B B B Motor Gating open closed Response Sensory Input B 5

81 B-X: Working memory, performance a) Update B b) Maintain B c) Prepare NonTarg Resp. Working Memory B B B OK! Motor Gating open closed Response Sensory Input B 5 X

82 B-X: Working memory, performance a) Update B Working Memory B Gating open Sensory Input B

83 B-X: Working memory, performance a) Update B b) Update 1 Working Memory B 5 Gating open open Sensory Input B 5

84 B-X: Working memory, performance a) Update B b) Update 1 c) No Relevant Context. Working Memory B 5 5 Motor Gating open open Response Sensory Input B 5

85 B-X: Working memory, performance a) Update B b) Update 1 c) No Relevant Context. Working Memory B 5 5 Motor Gating open open Response Sensory Input B 5 X OOPS!

86 A-Y: Working memory, performance a) Update A Working Memory A X Gating open Sensory Input A

87 A-Y: Working memory, performance a) Update A b) Maintain A Working Memory A X A X Gating open closed Sensory Input A 5

88 A-Y: Working memory, performance a) Update A b) Maintain A c) Prepare Targ Resp Working Memory A X A X X A Gating open closed Motor Response Sensory Input A 5

89 A-Y: Working memory, performance Working Memory a) Update A A X b) Maintain A c) Prepare Targ Resp A X X A Gating open closed Motor Response Sensory Input A 5 Y OOPS!

90 A-Y: Working memory, performance a) Update A Working Memory A X Gating open Sensory Input A

91 A-Y: Working memory, performance a) Update A b) Update 1 Working Memory A X 5 Gating open open Sensory Input A 5

92 A-Y: Working memory, performance a) Update A b) Update 1 c) No Relevant Context: Working Memory A X 5 5 Gating open open Motor Response Sensory Input A 5

93 A-Y: Working memory, performance a) Update A b) Update 1 c) No Relevant Context: Working Memory A X 5 5 Gating open open Motor Response Sensory Input A 5 Y OK!

94 Working Memory Demands: AX-CPT Task More active maintenance of task-relevant info more A-Y false alarms, less B-X false alarms

95 WM context index =BX -AYaccuracy Short Long Delay WM Context Index AX-CPT Working Memory Controls Schiz Barch etal(2001) SZ: Impaired Working Memory

96 Psychopharmacological Studies Frank& O Reilly, 2006 Double blind, within-subjects design(n=28). Cabergoline and Haloperidol: D2 agonist and D2 antagonist D2 agents: preferential action in BG (Camps et al, 1987; Moghaddam& Bunney, 1990; Arnsten et al., 1994; Seemans & Yang, 2004)

97 time time Working memory gating task A L Attentional Shift L 1 1 A L L X R B 3 R 2 L L X L L L A L L L 3 1 L 3 L 2 L Y X B 4 L

98 BG Gating of PFC Working Memory Frank etal2001, O Reilly & Frank 2006, Hazy etal

99 DA drug effects on working memory gating see Moustafaetal08 and Franketal 07forsimilar drug effects inparkinson s and ADHD

100 Analogous drug effects on learning and working memory

101 How do multiple BG-FC circuits interact in motivated behavior?

102

103

104 Only some maintained PFC reps relevant for processing during intermediate stages of processing

105 Reduces numberof S-R mappingsneeded to be learned by motorcircuit

106 Contextual dependencies for output gating (e.g., arithmetic); see also LSTM (Hochreiter & Schmidhuber 97)

107

108 Hierarchical interactions among BG-FC circuits BG gates frontal actions (motor, working memory, context) Anterior PFC influence posterior circuits via BG output gating BG-FC circuits learn reward probabilities via DA prediction error signals

109 Hierarchical interactions in BG-FC circuits: PFC influences on BG learning Collins & Frank 2013, Psych Rev;Frank & Badre 2012

110 Broader speculations: Why does motor control develop so slowly in humans?? Standard story: infants born early due to large head, small birth canal Fourth trimester

111 Broader speculations: Why does motor control develop so slowly in humans?? Standard story: infants born early due to large head, small birth canal Fourth trimester But 3 month old infants still pretty incompetent (from babycenter.com):

112 Broader speculations: Why does motor control develop so slowly in humans?? Standard story: infants born early due to large head, small birth canal Fourth trimester But 3 month old infants still pretty incompetent (from babycenter.com): You no longer need to support his head. When he s on his stomach, he can lift his head and chest. He can open and close his hands..

113 Broader speculations: Why does motor control develop so slowly in humans?? Standard story: infants born early due to large head, small birth canal Fourth trimester But 3 month old infants still pretty incompetent (from babycenter.com): You no longer need to support his head. When he s on his stomach, he can lift his head and chest. He can open and close his hands.. Hypothesis: human brain is wired to discover generalizable structure... which is initially inefficient.

114

115

116

117

118

119

120

121

122 seealso Gershmanetal 2010

123 Model mimicry: C-TS and hierarchical BG-PFC network Sparseness of context-pfc connectivity matrix is linked to α clustering Both models are approximations of the same process: building TS structure fmri evidence for hierarchical PFC-BG mechanisms Badre et al 2010; Badre & Frank 2012 Collins & Frank2013 Psych Rev

124

125 Hierarchical learning and clustering: First, in adults

126

127

128

129 Clustering affords faster learning within existing rule sets...

130 But initial(phase 1) clustering is inefficient and slow!

131

132

133 Now, in Infants! Werchan etal, 2015; in prep

134 Badre Hierarchical Learning Task Badre, Kayser& D Esposito, 2010 Flat(conjunctive) Hierarchical

135 Learning Curves

136 Rostro-caudal axis and hierarchical control prepmd: 2nd level of hierarchical control (from Badre, Koechlin) activity predicts learning in 2nd level hier condition, declines in flat condition

137 Striatum implicated in hierarchical rule learning How do BG-PFC circuits interact to discover hierarchical structure?

138 Hierarchical interactions among BG-FC circuits

139 Application to Badre task Anterior PFC affects posterior circuits via BG output gating Reduces # of stimulus-response mappings needed to be learned in motor circuit.

140 Application to Badre task

141 This is due torewardprediction error signals punishing gatingofprepmd whenno hierarchical structure exists (stay tuned) As in fmri data, modelprepmdactivitydecreases inflatrelativeto Hier DA-based RL ingatingnetworks needed for rapidlearning in Hiercond Epoch Trial Percent Error hier_bgout hier_nodamod hier_bgmnt nohier Normalized Max Activation0.8 Hier prepmd Flat prepmd 0.7 Model prepmd activity by condition Hierarchical Task Modeling BG-PFC hierarchical learning

142 Modeling individual learners: abstract account Neural model makes plausible links to biology, but unsuitable for quantitative fits to individual subject behavior We developed abstract version of hierarchical rule learning using Bayesian mixture of experts (MoE), motivated by neural model

143 Modeling individual learners: abstract account Separate experts learn P(Rew Resp, Shape), P(Rew Resp, Orient), etc..

144 Modeling individual learners: abstract account Separate experts learn P(Rew Resp, Shape), P(Rew Resp, Orient), etc.. Hierarchical experts separately learn statistics about each dimension contingent on a candidate higher order feature (cf. prepmd context for output gating).

145 Modeling individual learners: abstract account Separate experts learn P(Rew Resp, Shape), P(Rew Resp, Orient), etc.. Hierarchical experts separately learn statistics about each dimension contingent on a candidate higher order feature (cf. prepmd context for output gating). Credit assignment mechanism learns probability that each expert contributes to observed rewards Attentional mechanism selects among experts based on learned probability of success P(Rew e) e.

146 P(θ r,o δ1...δn) P(δ1...δn θ r,o )P(θ r,o )

147

148 weight each expert in proportion to learned probability of expert success P(Rew e) e.

149

150

151

152

153 Goal: infer latent hypotheses (attentional wts) by observing sequence of responses & rewards, maximizing likelihood of choices with few free params for each subject

154 Params: priors to attend to each dimension, to attend to conjunctions, to consider hierarchy. 3 softmax slopes (motor, within experts, between flat/hier).

155 MoEfits tohumans

156 Attentional Wts Trial Trial Attentional Wts Subject118 Hierarchical Subject118 Flat Example attentional weights: MoE fit to human

157

158 Attentional Wts Trial Trial w H MoE fit to BG PFC Net All Nets: Attention to Hierarchy MoE fitto BG-PFCmodel

159 Trial w H All subjects: Attention to Hierarchy Individual differences in attention to Hierarchy

160 Median split on AttenH vs AttenF more prepmd in Hier condition Trial w H All subjects: Attention to Hierarchy Individual differences in attention to Hierarchy

161 Reward prediction errors tracked by entire striatum (cf. O Doherty et al, 2004) Trial PE Model-based fmri: reward prediction errors

162 Prediction errors and attention to Hier vs Flat experts

163 Decline in prepmd should be predicted by sensitivity of BG to PE s associated with hiearchical rule Trial Normalized Max Activation Hier prepmd Flat prepmd Model prepmd activity by condition PE s predict PPMD decline in Flat condition

164 left caudate right caudate Decline in prepmd should be predicted by sensitivity of BG to PE s associated with hiearchical rule Trial Normalized Max Activation Hier prepmd Flat prepmd Model prepmd activity by condition PE s predict PPMD decline in Flat condition

165 S N N M spikes per second Problem with recurrent: strongest activations at any point determine what is maintained not necc. true: best stimulus worst stimulus Intrinsic bistability: selectively activated ion channels. Recurrent excitation: Activations can be maintained in two different ways: Computational/Biological Details

166 Computational/Biological Details Our proposal: Thalamic disinhibition activates layer 4 FC. Convergence of layer 4 and cortico-cortical 2/3 projections on either 2/3 or 5/6 neurons triggers maintenance ion channels. Other proposals: Thalmocortical loops themselves drive maintenance (enough thalamic neurons to specify what is maintained?) Just recurrent excitation within FC (unstable, but useful complement to ion channels).