A reinforcement learning model of song acquisition in the bird

Transcription

1 A reinforcement learning model of song acquisition in the bird Michale Fee McGovern Institute Department of Brain and Cognitive Sciences Massachusetts Institute of Technology 9.54 November 12, 2014

2 Frequency Structure of zebra finch song 10 khz Motif Motif 0 khz Syllable (~100ms) Note (~10ms) 1s

3 Decreased Variability Songbirds learn to sing by imitating their parents Subsong Plastic Song Crystallized Increased Similarity to Tutor Tutor Song

4 Overview The songbird as a model system for understanding how the brain generates and learns complex sequential behaviors Review some current understanding of the mechanisms of song production Describe progress in elucidating the role of cortical and basal ganglia circuits in song learning. Some speculations on how insights from the songbird may inform our understanding of mammalian BG function

5 A circuit for vocal production RA HVC Motor Pathway Cortex nxii Uva Thalamus Nottebohm et al, 1976, 1982

6 Antidromic activation allows identification of RA-projecting neurons in HVC Extracellular recording electrode Stimulation electrode HVC RA X Hahnloser, Kozhevnikov and Fee, 2002

7 l HVC neurons burst throughout the song Bird A t HVC neurons form a continuous tor activity []. l out the song, and that HVC activity HVC coding st the GTE hypothesis by looking for these signatures in a large dataset C projection neurons termine the probability of the data under each hypothesis by generating ate datasets under the uniform null hypothesis and under the GTE esis. VC burst activity covers the song burst around GTEs otif, supporting the clock model. # IBIs Amador et al. further concluded that HVC projection neurons burst only at Bird GTEs, A: and 66 bursts, nowhere 40 else neurons [7]. Bird B: 56 bursts, 44 neurons Strong GTE hypothesis: HVC projection neurons burst around Bird B GTEs, and nowhere else. The weak GTE hypothesis is not inconsistent with the clock model. song production The as strong a low GTE hypothesis is incompatible with the clock model, because it ameters. predicts gaps in HVC activity. on neurons, 1 Amador et al. observed 1 to extrema and discontinuities of urst # Burst # trajectory extrema (GTE). jection neurons 100ms 100ms preferentially Here we test the GTE hypotheses with a large GTE P =.71 dataset of HVC neurons Null projection neurons burst only at [6] Leonardo and Fee (J Neurosci., 2004) 0.02 [7] Amador et al (Nature, 2013) Bird B: 56 bursts, 44 neurons Bird C: 91 bursts, 64 neurons ojection neurons burst around urst # 1 potheses with 100ms a large 56 Many short IBIs C neurons. rosci., 2004) Long IBIs Amador et al. (Nature, 2013) Cross correlation With a relatively small dataset of projection neurons, Amador et al. observed 0.02 GTE Null 0.2 Interburst interval distributions Burst # Null % 100ms HVC burst activity covers the song Bird A offset (ms) 20 motif, supporting the clock model. 10 P (#IBIs) P (#IBIs) Bird C offset (ms) inconsistent motif, with supporting 20 the GTE model the clock model The GTE model of HVC coding We 30 test 40 the GTE hypothesis by looking for these signatures in a large dataset 10 Bird A: 66 bursts, 40 neurons Interburst interval (IBI) [ms] of HVC projection neurons 0 We examined various measures of the IBI distribution under the GTE and Average cross correlation We determine between the probability of the HVC burst and GTE times for each model uniform data under null hypotheses each hypothesis by generating 0.8 surrogate datasets under the uniform The observed null hypothesis IBI distributions and under are the highly GTE inconsistent with the GTE hypothesis hypothesis GTE 0.6 and are consistent with the null hypothesis Amador et al. modeled the biophysics of song production as a low dimensional system with two control parameters. that burst times appeared to be aligned to extrema and discontinuities of these control parameters, called gesture trajectory Burst # extrema (GTE). stent with the clock model. ble with the clock model, because it Weak GTE hypothesis: HVC projection neurons preferentially Burst # ms 100ms Distribution of the # of IBIs < 5ms GTE Null P =.27 Count Bird A: 66 bursts, 40 neurons P = Count Distribution of the # of IBIs < 5ms P = # IBIs Count IBI (ms) P(Longest IBI) P(Longest IBI) Distribution of longest IBIs 9 x Burst # Uniform 5-95% Bird B IBI (ms) P =.09 P = 9x Longest IBI [ms] Distribution of longest IBIs 100ms 91 P = IBI (ms) P = Longest IBI [ms] Skew [ms 3 ] GTE 5-95% Skew [ms 3 ] Variance [ms 2 ] 0.02 GTE Lynch, Okubo and Fee, in Null preparation 0.01 Interburst interval distributions Uniform 5-95% Bird C IBI (ms) GTE 5-95% GTE 5-95% Bird B Distribution of variance vs. skew Distribution of the # of IBIs < 5ms GTE 400 P = Variance Null [ms 2 ] P = Interburst interval distributions Bird C: 91 bursts, 64 neurons Burst # P (#IBIs) P (#IBIs) Distribution of variance vs. skew 60 P (#IBIs) Bird C # IBIs # IBIs Distribution of the # of IBIs < 5ms 0 Null P =.27 Bird B: 56 bursts, 44 neurons 100ms Count Count Count P = Distribution of the # of IBIs < 5ms P =.81 P = # IBIs IBI (m P(Longest IBI) P(Longest IBI) P(Longest IBI) 0 9 x IBI (m Int U 5- Bir Interb Unif 5-95 Bird C IBI (m

8 Extracellular recording electrode RA HVC Activity of RA neurons during singing Motif Leonardo and Fee, 2005 Yu and Margoliash, 1996

9 Simple sequence generation circuit Sparse representation of time Output Leonardo and Fee, 2005

10 Simple sequence generation circuit Sparse representation of time Output Leonardo and Fee, 2005

11 HVC is the clock of the song motor pathway Brain cooling to localize dynamics Bilateral cooling of HVC causes uniform slowing of the song p n 0.25A HVC RA 0.0A -0.25A -0.5A nxii -0.75A... 5 mm Long and Fee, Nature 2008

12 A simple reinforcement model of song learning Auditory feedback Song evaluation Auditory Memory - Error/Reinforcement signal Song Song motor system Exploratory variability Doya and Sejnowski, 1989

13 A separate circuit for song learning Instructive signal RA HVC Cortex LMAN Motor Pathway Anterior Forebrain Pathway (AFP) nxii Thalamus DLM Area X Basal Ganglia The learning pathway is not necessary for adult song production, but is required for learning (Bottjer, 1984, Scharff and Nottebohm, 1991) Bottjer proposed that the AFP transmits an instructive signal that guides plasticity in the motor pathway

14 Separate premotor pathways for stereotyped song and variability Sequence generator HVC RA Variability generator LMAN nxii Kao et al, 2005 Ölveczky et al, 2005 Aronov et al, 2008 Stepanek and Doupe, 2010

15 Separate premotor pathways for stereotyped song and variability Sequence generator TTX or Muscimol HVC RA Variability generator LMAN nxii Kao et al, 2005 Ölveczky et al, 2005 Aronov et al, 2008 Stepanek and Doupe, 2010

16 Transient inactivation of the learning pathway RA HVC LMAN nxii 55 day old bird Olveczky, Andalman, and Fee, 2005

17 Residual Pitch (Hz) LMAN drives exploratory variability in song LMAN intact 20 LMAN inactivated Time (ms)

18 LMAN also drives early song babbling LMAN intact LMAN inactivated 30 db 250 ms Goldberg and Fee, 2011

19 HVC lesions abolish all stereotyped song structure RA HVC Motor Pathway Learning Pathway (AFP) LMAN nxii

20 HVC lesions abolish all stereotyped song structure Pre HVC lesion Post HVC lesion Subsong bird Plastic song bird Adult bird Aronov, Andalman and Fee, Science 2008, Transient pharmacological inactivation of HVC produces the same effect

21 The basal ganglia are not necessary for subsong or vocal variability in juvenile birds HVC Pre-lesion Subsong RA LMAN nxiits DLM X X 30 db Post-lesion Lesions of the BG have little or no acute effect on juvenile song variability. Local cooling in LMAN slow timescales of babbling exploratory vocal variability is generated by local circuit dynamics within LMAN. 250 ms Goldberg and Fee, 2011

22 Separate premotor pathways for stereotyped song and variability Sequence generator HVC RA Variability generator LMAN nxii Kao et al, 2005 Ölveczky et al, 2005 Aronov et al, 2008 Stepanek and Doupe, 2010

23 Separate premotor pathways for stereotyped song and variability Sequence generator RA HVC Instructive signal Variability generator LMAN nxii Kao et al, 2005 Ölveczky et al, 2005 Aronov et al, 2008 Stepanek and Doupe, 2010

24 Separate premotor pathways for stereotyped song and variability Sequence generator RA HVC Instructive signal Variability generator LMAN nxii DLM Area X

25 Song learning is slow Days of Training Pupil Tutor Pitch (Hz) Hz Harmonic Stack Tchernichovski, Mitra, Lints, Nottebohm, 2001

26 Heard Pitch (khz) Speaker Vocalized Experimental control of song learning DSP Brain Mic Targeted region of syllable 0.6 Pitch threshold 0.5 Cranial airsac Feedback Noise Andalman and Fee 2009; Tumer and Brainard 2007

27 Pitch (Hz) Pitch (Hz) Conditional auditory feedback drives pitch learning h 2 h 4 h Tumer and Brainard ms Andalman and Fee 2009

28 Pitch (Hz) Observations Pitch (Hz) Observations Many days of sequential learning Up Days 10 Down Days ΔPitch, Day (Hz) Days Post Hatch Up Days Down Days Days Post Hatch Days Post Hatch ΔPitch, Overnight (Hz)

29 Where does this learning occur in the song control circuit? AFP-driven variability Motor pathway Motor parameter space

30 Where does this learning occur in the song control circuit? AFP-driven bias AFP-driven variability Motor pathway Motor parameter space Error gradient (reduced error)

31 Where does this learning occur in the song control circuit? AFP-driven bias AFP-driven variability Motor pathway Motor parameter space Plasticity in motor pathway Error gradient (reduced error)

32 Where does this learning occur in the song control circuit? Motor Pathway Anterior Forebrain Pathway (AFP) AFP-driven bias HVC RA LMAN nxiits DLM X HVC Plasticity in motor pathway RA LMAN nxiits DLM X

33 Pitch (Hz) Observations Pitch (Hz) Observations Pitch (Hz) 600? 470 PBS TTX PBS TTX PBS TTX PBS TTX cap drug reservoir inflow tube outlet tube dental acrylic skull dialysis membrane LMAN 25 Hz TTX Δ 8 TTX h Pitch (Hz) Up Days Down Days Andalman and Fee, PNAS 2009 Vehicle Vehicle Up Days Down Days Pitch (Hz)

34 Does AFP-driven variability become biased to reduce vocal errors? Yes!! AFP-driven error-reducing bias AFP-driven variability Motor pathway Motor parameter space Plasticity in motor pathway Error gradient (reduced error)

35 Pitch (Hz) Is all song learning mediated by AFP bias? 700 Many days of sequential learning Days Post Hatch

36 Pitch (Hz) Pitch (Hz) Is all song learning mediated by AFP bias? 600 LMAN(+) LMAN(-) baseline Days post-hatch 600 LMAN(+) LMAN(-) baseline Days post-hatch

37 Correlation Coefficient (r 2 ) Pitch Night Δm (Hz) (down days inv.) AFP bias is highly predictive of motor pathway plasticity within the next 24 hours 100 Days Lag = -2 d Lag = -1 d Lag = 0 d Day 1 Day 2 Day Estimated AFP bias (Hz, down days inverted) β Δm Andalman and Fee, 2009 Warren et al, Lag (days)

38 Motor pathway plasticity appears to integrate AFP bias Day 1 Day 2 Day 3 AFP-driven variability Motor pathway AFP-driven bias motor pathway plasticity motor pathway plasticity Motor parameter space Error gradient (reduced error)

39 How is AFP bias generated? HVC RA LMAN nxii X VTA Area X receives an efference copy of variability signals sent to RA. Schultz, 2000 If Area X also receives an evaluation signal, then X could figure out which variations lead to better song performance. Dopaminergic midbrain (VTA) has been shown to signal reward prediction error Do X-projecting VTA neurons carry error-related signals?

40 A descending pathway from higher-order auditory areas to VTA/SNc Ventral Intermediate Arcopallium (AIV) CM AIV Aud RA X VTA AIV Keller and Hahnloser, 2008 Gale, Perkel 2008 Mandelblat-Cerf et al, 2014 Retrograde label from VTA

41 Is AIV necessary for song learning? HVC LMAN nxii VTA X NeuN Las, Denisenko, Mandelblat-Cerf, elife, 2014

42 Is AIV necessary for song learning? Bird tutored in home cage Bird isolated AIV lesion Check imitation Age (days post hatch)

43 AIV lesion produces profound song learning deficits Tutor AIV lesioned pupil #2 Adult song Example 1 Example 2

44 AIV lesions produce profound song learning deficits Tutor AIV lesioned adult song Lesioned control Unlesioned control AIV lesion Similarity of unrelated birds

45 Voltage (mv) Do AIV neurons transmit an error signal to VTA during singing? HVC LMAN 0.0 nxii VTA X Time from stim (ms)

46 Do AIV neurons transmit an error signal to VTA during singing? Noise burst 200 ms

47 AIV neurons show error-related signals Mandelblat-Cerf, Las, Denisenko, under review

48 A descending pathway from higher-order auditory areas to VTA/SNc Ventral Intermediate Arcopallium (AIV) CM AIV Aud RA X VTA AIV Keller and Hahnloser, 2008 Gale, Perkel 2008 Mandelblat-Cerf et al, 2014 Retrograde label from VTA

49 How it all works: a hypothesis HVC RA LMAN nxii

50 How it all works: a hypothesis CM Aud X VTA

51 How it all works: a hypothesis HVC LMAN X VTA

52 How it all works: a hypothesis HVC LMAN DLM X

53 How it all works: a hypothesis HVC RA LMAN

54 4 khz A model of basal ganglia function with functionally distinct inputs for context, motor efference copy, and reward HVC (Time Sequence) HVC (X) firing patterns To RA VTA 2 3 LMAN MSN Area X 100 ms Pallidal Thalamus The AFP forms a classic cortical-bg-thalamo-cortical loop

55 A model of basal ganglia function with functionally distinct inputs for context, motor efference copy, and reward HVC (Time Sequence) To RA VTA Learning rule: Strengthen HVC synapse after coincidence of LMAN, HVC and DA inputs LMAN MSN Area X Pallidal Thalamus

56 A model of basal ganglia function with functionally distinct inputs for context, motor efference copy, and reward HVC (Time Sequence) 1 To RA HVC 2 VTA 3 LMAN MSN MSN Pallidal Area X Thalamus LMAN Time-dependent bias of one LMAN neuron Goldberg and Fee 2010

57 A model of basal ganglia function with functionally distinct inputs for context, motor efference copy, and reward HVC (Time Sequence) HVC synapses To RA LMAN MSN VTA Timing Drive MSNs Plastic Selective for single synapses Area X LMAN synapses Pallidal Thalamus Action Do NOT drive MSNs Not plastic Global signal

58 A learning rule with an eligibility trace allows delayed reward E HVC-X = LH DW HVC-X = be HVC-X R HVC HVC LMAN VTA HVC LMAN LMAN MSN E HVC-X Eligibility trace MSN VTA ΔW HVC-X

59 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 HVC on spines MSN LMAN on dendritic shafts LMAN

60 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 MSN t = 1 LMAN

61 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 MSN t = 2 LMAN

62 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 MSN t = 2 LMAN

63 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 MSN t = 2 LMAN

64 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 Dopamine MSN t = 2 LMAN

65 Hypothesis for HVC-LMAN synaptic interaction on striatal MSNs HVC 1 HVC 2 Synapse Strengthened MSN LMAN

66 Serial Block Face Scanning EM Collaboration with Winfried Denk and Jörgen Kornfeld

67 Distinct morphology of HVC and LMAN axons LMAN HVC 200 µm 200 µm Axonal arbor of LMAN neuron in Area X Axonal arbor of HVC neuron in Area X 200 µm 200 µm Michael Stetner

68 Inputs onto MSN spines originate primarily from HVC Putative LMAN axons Putative HVC axons MSN HVC-like LMAN-like ~94% of synapses onto spines are from HVC-like axons

69 The role of the basal ganglia in songbird vocal learning LMAN directly drives exploratory variability in the song motor pathway. LMAN-driven variability becomes biased during learning, in the direction of improved song performance. We have found evidence that a dopaminergic pathway to the songbird BG may carry performance error-related information. We hypothesize that the basal ganglia determine which song variations lead to better performance and bias the variability in the direction of improved performance. We have proposed a testable model of basal ganglia function that explicitly incorporates an efference copy of cortically-generated motor actions.

70 The Fee Lab Current Lab Members Anusha Narayan Natalia Denissenko Tatsuo Okubo Michael Stetner Emily Mackevicius Galen Lynch web.mit.edu/feelab Funding: National Institutes of Health - NIMH, NIDCD Former Lab Members Richard Hahnloser Alexay Kozhevnikov Anthony Leonardo Michael Long Bence Ölveczky Dmitriy Aronov Aaron Andalman Lena Veit Jakob Förster Liora Las Jesse Goldberg Yael Mandelblat

71 Separate premotor pathways for stereotyped song and variability Sequence Stereotypy Precision Uva DLM Randomness Variability Exploration Adult song HVC XLMAN Subsong RA Motor Output

72 LMAN drives subsong Stimulating electrode HVC RA LMAN nxii DLM Area X

73 Instantaneous firing rate (Hz) LMAN(RA) neurons exhibit premotor correlation with subsong syllables 700 Sound amplitude 0 20 db 250 ms

74 Instantaneous firing rate (Hz) LMAN(RA) neurons exhibit premotor correlation with subsong syllables db Sound amplitude ms

75 Mean firing rate (Hz) Instantaneous firing rate (Hz) LMAN(RA) neurons exhibit premotor correlation with subsong acoustic structure Neuron 14 * 700 * 0 20 db Sound amplitude Sound amplitude 250 ms Time relative to offset (ms)

76 Summary The AFP can generate a direct premotor bias that reduces vocal errors. The learning accumulated across many days of training is encoded primarily in plasticity in the motor pathway. The contribution of the AFP is limited to the learning that occurred most recently (during the same day). AFP bias is predictive of subsequent plasticity in the motor pathway within the next 24 hours