Synaptic theory of gradient learning with empiric inputs. Ila Fiete Kavli Institute for Theoretical Physics. Types of learning

Size: px

Start display at page:

Download "Synaptic theory of gradient learning with empiric inputs. Ila Fiete Kavli Institute for Theoretical Physics. Types of learning"

Darlene Nicholson
5 years ago
Views:

1 Synaptic theory of gradient learning with empiric inputs Ila Fiete Kavli Institute for Theoretical Physics Types of learning Associations among events (and reinforcements). e.g., classical conditioning How one s actions affect events (and reinforcements), how to shape actions. e.g., instrumental conditioning Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 1

2 behavioral classical conditioning neural Hebbian learning instrumental conditioning? Outline Learning with reinforcements as optimization Synaptic learning rule Scale-up problem Example: birdsong Implications with Sebastian Seung Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 2

behavior in way that tends to improve reinforcement. Dr.

3 Instrumental conditioning Learning by reinforcement Thorndike, 1898 Try multiple strategies; modify behavior in way that tends to improve reinforcement. Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 3

The law of effect Of several responses made to the same situation those which are accompanied or closely followed by satisfaction to the animal will become more likely to recur; those which are

4 The law of effect Of several responses made to the same situation those which are accompanied or closely followed by satisfaction to the animal will become more likely to recur; those which are accompanied or closely followed by discomfort to the animal will, other things being equal, have their connections to the situation weakened, so that, when it recurs, they will be less likely to occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond. Thorndike, 1910 evaluation, not instruction Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 4

instrumental conditioning neural Hebbian learning? Dr.

5 Trial, reward, and optimization reward = performance on task behavioral classical conditioning instrumental conditioning neural Hebbian learning? Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 5

6 Octopamine system Insect olfactory learning VUMmx1 Dopamine system Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 6

7 Neural basis of learning Global signal (Reward, motor error, etc.) Local signals (Voltage, calcium, etc.) Synaptic plasticity The interaction between global and local signals is largely uncharacterized. Why difficult?? R Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 7

8 Learning with empiric inputs evaluator performer experimeter outputs Fiete and Seung Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 8

9 regular synapse s empric i ξ i synapse R syn i g = W s + ξ i learning rule W T = ηre ( ξ ξ ) e = dt s i i 0 gradient following W R = W example: spiking network dvi syn ionic Cm = gi ( Vi VE ) gαi ( Vi Vα ) dt ds = fs ( V, s,{ use,...}) dt g syn i α = W s + ξ i R = R[ V ] local eligibility W? = Rs g Sensitivity lemma R syn i W = = b i Rs W s + bi + ξi Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 9

Stochastic gradient ascent -Reward optimization, but only on average. -Model-free. -Local to synapses aside from evaluation. -Deals with delayed evaluation and dynamic synapses. -Slow?

10 Stochastic gradient ascent -Reward optimization, but only on average. -Model-free. -Local to synapses aside from evaluation. -Deals with delayed evaluation and dynamic synapses. -Slow? Learning time - Individual correlations small long averaging? - Apply to biological example. Birdsong learning with Sebastian Seung & Michale Fee Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 10

Behavior days 0 sensory 30 60 90 120 sensorimotor

11 Behavior days 0 sensory sensorimotor Social feedback and tutor song not needed. Auditory feedback of own song crucial. Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 11

12 Song circuit motor HVC anterior forebrain X DLM RA LMAN resp syrinx Lesion: no song premature crystallization Hahnloser et al. Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 12

13 RA activity Song representation HVC generates a sparse sequence later stages learn/perform the motor map Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 13

14 LMAN activity is variable Hessler & Doupe Song system schematic auditory evaluation motor network motor commands vocal apparatus LMAN song Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 14

1 2 HVC m V 0-25 -50 1 2 LMAN 3 RA m V 0-25 -50 3 S ( t) 4 P( ω) motor m p a 4 50 100 150

15 1 2 HVC m V LMAN 3 RA m V S ( t) 4 P( ω) motor m p a time (ms) t ω E 10 ms Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 15

16 a begining middle c d end tutor 2000 epochs 100 ms Scaling to large networks Correlation of single neuron with reward = 1/N. Time to learn ~ N? factor of 4000 more neurons in bird: model bird HVC RA outputs Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 16

a 10 3 error 10 1 10-1 20,200,2000 2,5,10 10-3 b 10-5 0 5 10 15

1500 2000 iterations Redundancy and rank Correlation of single

Model: Bird: 2 outputs 8 outputs Neural or synaptic noise.

17 a 10 3 error ,200,2000 2,5, b iterations/(no+2) 40 error 30 tutor iterations Redundancy and rank Correlation of single output with reward = 1/No. Time to learn ~ No. Model: Bird: 2 outputs 8 outputs Neural or synaptic noise. Time: <8000 epochs. Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 17

18 Simple tasks can be learned Learning time is independent of network size in redundant networks. Zebra finches practice 80,000 times. Motor control is redundant and learning is often slow. Experimental probe x evaluator x i ξ i experimenter W > 0 W < 0 Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 18

19 Why is the brain so noisy? Why is spiking irregular? Why are synapses unreliable? Does the brain use noise for learning? Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 19

20 Similarity to Hebbian learning? x x i ξ i ( ξ ξ ) W = R s i i strong noise input and weak regular inputs R ( δ ξ ) s i i need segregated noise inputs so noise baseline correct Supervised, reinforced, or unsupervised? noise as exploration noise as instruction post noise spike no noise spike reward increase active weights decrease active weights however, instruction is stochastic, uncorrelated with error Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 20

21 Evidence of heterosynaptic plasticity entorhinal cortex CA3 pyramidal MF dentate gyrus Tsukamoto et al., 2003 also cerebellar CF, PF convergence at Purkine cells Practice vs. performance undirected directed Hessler and Doupe, 1999 Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 21

22 Song circuit HVC RA Delayed reward s ξ i e 45 ms R W > 0 W < 0 t Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 22

23 IF dynamics dv Cm = gl( V VL) ge ( V VE ) dt ds τ s dt g = s ( t) = V W s ( t) Ei = V thres V : = Vreset s : = s + Gradient following: R R = W W R R W = R = > 0 W W 2 REINFORCE: R W = = W W P( s) R( s) ln P( s) = P( s) R( s) s W = ln P( s) R( s) W ln P( s) W = R( s) Re W s Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 23

24 Dr. Ila Fiete, KITP (KITP, Understanding the Brain Program 9/21/04) 24

Reinforcement Learning. With help from

Reinforcement Learning. With help from Reinforcement Learning With help from A Taxonomoy of Learning L. of representations, models, behaviors, facts, Unsupervised L. Self-supervised L. Reinforcement L. Imitation L. Instruction-based L. Supervised