Computational Approaches in Cognitive Neuroscience

Transcription

1 Computational Approaches in Cognitive Neuroscience Jeff Krichmar Department of Cognitive Sciences Department of Computer Science University of California, Irvine, USA Slides for this course can be found at: Full course and slides can be found at: 1

2 Outline Computational Cognitive Neuroscience modeling Neuronal coding (rate vs. temporal) Biophysical model of an action potential Dynamical systems model of a neuron Models of plasticity Reinforcement learning Case Studies Neuromodulation as a robot controller Large-Scale Spiking Simulation of Visual Cortical Processing What is Computational Neuroscience? Computational Neuroscience is the theoretical study of the brain to uncover the principles and mechanisms that guide the development, organization, information processing and mental abilities of the nervous system. Computational Neuroscience is an interdisciplinary science that links the diverse fields of neuroscience, electrical engineering, computer science, and applied mathematics. It serves as the primary theoretical method for investigating the function and mechanism of the nervous system. 2

3 Computational/theoretical tools in context Topics in Computational Neuroscience Single Neuron Modeling Development, Axonal Patterning and Guidance Sensory processing Memory and synaptic plasticity Behaviors of Networks Cognition, Discrimination and Learning Neural Correlates of Consciousness 3

4 Rate Code vs. Temporal Code The Neural Code and Firing Rate Hypothesis Rate code Number of spikes increase when the strength of the stimulus increases Easily detectable Dominates the neurophysiological search for stimuli that drive neurons Tuning curves capture many neuron s responses Gain functions for Rate Models 4

5 Network input into a Firing-Rate Model Feedforward inputs to a single neuron ( ) = F v i = F w u N j =1 w ij u i Population Dynamics: Modeling the Average Behavior of Neurons Rate models cannot incorporate all aspects of networks of spiking neurons. However, many principles of brain information processing can be illuminated by population models. 5

6 Firing Rates and Population Averages Temporal averaging Population averaging v(t) = # spikesδt ΔT A(t)dt = 1 N N i=1 ( ) δ t' t f Rate vs. Temporal Code Correlation Codes and Coincidence Detectors Response of MT neurons to visual stimulus Buracus et al., Neuron 20:959-69,

7 7/8/11 Rate vs. Temporal Code Correlation Codes and Coincidence Detectors decharms, R.C., and Merzenich, M.M. (1996). Primary cortical representation of sounds by the coordination of action-potential timing. Nature 381, Derivation of Spiking Models Generation of Nerve Action Potential Depolarization membrane potential moves in a positive direction. Hyperpolarization membrane potential moves in a negative direction. 7

8 State of Na and K Channels During an Action Potential The Hodgkin-Huxley Model Alan Hodgkin & Andrew Huxley Landmark Model of Neural Excitability Series of articles in 1952 Nobel Prize in 1963 for Physiology and Medicine 8

9 Voltage Clamp Space clamp maintains a uniform spatial distribution of Vm over the region being measured. Voltage clamp maintains Vm at a desired level. Voltage Clamp of Squid Axon Current vs. time in ms Voltage clamped at -9mV from -65mV 9

10 Voltage Clamp Experiments Characterizing the K conductance HH Mathematical Model E K = -75 mv, E Na = +60 mv, E L ~ -60 mv 10

11 Output of the Hodgkin Huxley model 120 I=10nA 120 I=30nA Time Time 11

12 Phase Plane of the HH Model Izhikevich Simple Spiking Model From Izhikevich, E.M. (2004). IEEE Trans Neural Netw 15, v = 0.04v 2 + 5v u + I u = a(bv u) If v v peak (30mV) Where v = c u = u + d v is the membrane potential u is the recovery variable a is the recovery time constant b < 0 amplifying; b > 0 resonating c is the spike reset d describes the outward minus inward currents activated by the spike affecting after spike behavior 12

13 Networks of Spiking Neurons The current I can be synaptic input Pre 1 w 1 Pre 2 w 2 Post Pre 3 w 3 I Post = w 1 *spike(pre 1 ) + w 2 *spike(pre 2 ) + w 3 *spike(pre 3 ) Where spike = 1 if vpeak 30mV; 0 otherwise 13

14 Which Model to Use for Cortical Spiking Neurons? Neurons are dynamical systems Plasticity and Learning Unsupervised learning Network responds to a series of inputs based only on the intrinsic connections and dynamics Network self-organizes depending on the plasticity rule Supervised learning A desired set of input-output relationships is imposed on the network by a teaching signal Reinforcement learning Evaluative feedback about network performance is provided in the form of reward and punishment Non-Hebbian Plasticity Synaptic strength is modified solely on the basis of pre- or postsynaptic activity 14

15 Plasticity and Learning Hebb rule When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased. Neurons that fire together, wire together. Donald O. Hebb Long-Term Potentiation (LTP) and Long-Term Depression (LTD) LTP and LTD at the Schaffer collateral inputs to the CA1 region of a rat hippocampal slice. 15

16 General Hebb Rule Δw = εr j r i Presynaptic node: j Postsynaptic node: i Presynaptic firing rate: r j Postsynaptic firing rate: r i Learning rate: ε Synaptic Plasticity Rules Bienenstock, Cooper, Munro (BCM) Rule dw τ w dt = vu v θ dθ ( v); τ v θ dt Bienenstock, Cooper, Munro (1982), J Neurosci, 2: LTP and LTD self-stabilizing with a sliding threshold threshold for change decreases when average activity is low threshold for change increases when average activity is high Experimental evidence Hippocampus Visual system monocular deprivation receptive field formation = v 2 θ v 16

17 Empirical Evidence for BCM Rittenhouse, C.D., Shouval, H.Z., Paradiso, M.A., and Bear, M.F. (1999). Monocular deprivation induces homosynaptic long-term depression in visual cortex. Nature 397, BCM Plasticity Rule 17

18 Spiking Specific Plasticity Rule Spike Time-Dependent Plasticity (STDP) Bi, G.. & Poo, M. J. Neurosci. 18, (1998). Markram, H., Lubke, J., Frotscher, M. & Sakmann, B. Science 275, (1997). LTP if pre-synaptic spike precedes post-synaptic spike LTD if post-synaptic spike precedes pre-synaptic spike Model of STDP 18

19 Synaptic Weighting and Weight Distributions Song, S., Miller, K.D., & Abbott, L.F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nat Neurosci 3, Spike timing dependent plasticity (STDP) A. 19

20 7/8/11 Dopamine and Prediction of Reward Schultz, W., Dayan, P., and Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275,

21 Predicting Future Reward Temporal Difference Rule t v(t) = w(τ)u(t τ) τ = 0 w(τ) w(τ) + εδ(t)u(t-τ) δ(t) = r(t) + v(t+1) - v(t) Predicting Future Reward: Temporal Difference Learning 21

22 Predicting Future Reward Temporal Difference Learning Variations known as Actor-Critic model Machine learning Q-Learning Reinforcement learning The Power of Reinforcement Learning 22

23 Temporal Difference and Action Choice Rewards and punishments are associated with the actions an animal takes Animals develop polices, or plans of actions to increase rewards In static action choice, the reward or punishment immediately follows the action In sequential action choice, rewards may be delayed until several actions are completed Static Action Choice Two Arm Bandit Foraging Task Bees forage for blue and yellow flowers The model bee has a stochastic policy for choosing Softmax distribution [ ] = exp(βm a ) P a N actions i=1 exp(βm i ) β is a parameter for the explorationexploitation tradeoff 23

24 Direct Actor Choose action values directly to maximize the average expected reward m b m b +ε( δ ab P[ b] ) r a r ( ); where δ ab = 1 action = b 0 action = y If blue is selected and the amount of reward the bee gets is higher than expected m b increases for blue choices If yellow is selected and the amount of reward bee gets is higher than expected m b decreases for blue choices <r b > = 1 and <r y > = 2 for the first 100 flower visits <r b > = 2 and <r y > = 1 for the next 100 flower visits A and B. β = 1 C and D. β = 50 24

25 Sequential Action Choice Several actions needed to get a reward Complicated because of the delay Policy iteration improve policy to determine actions at each point Actor maintains and improves the policy Critic uses temporal difference learning to estimate the total future reward The Actor-Critic Model Critic learns to predict the correct motor command for future actions and supervises the Actor Actor is the motor command generator Basal ganglia may be a neural correlate SP spiny neurons in striatum ST sub-thalamic nucleus PD pallidus DA dopamine neurons in substantia nigra compacta 25

26 The Maze Task Actor-Critic Model Critic learning rule v v(u) + εδ δ = r(u) + γv(u ) v(u) γ discounting factor. Rewards and punishments soon after an action are more important than those received later. 0 γ 1; usually very close to 1 26

27 Actor Critic Model Actor learning rule m a (u) m a (u) + ε(δ aa P[a ;u])δ δ aa is the Kronecker delta. δ aa is 1 if action a = a P[a ;u] is the probability of taking action a at location u δ is from the Critic Action selection with the SoftMax function P[ a;u] = exp(βm ( u)) a L;R exp(βm i ( u)) i=1 Actor-Critic Results in Maze Task 27

28 Case Study 1 IEEE Robotics & Automation Magazine 16, (2009) Value Systems and Neuromodulation Adapt behavior when an important environmental event occurs. Neuromodulatory systems: Regulate fundamental behavior. Set the organism s internal states. Critical for an organism s survival. 28

29 7/8/11 Neuromodulatory System Architecture PFC/ACC Neocortex Raphe Nucleus Basal Forebrain Striatum Hippocampus Substantia Locus Nigra Coeruleus Cost prediction Attention Effort Novelty and saliency Reward prediction and wanting Ventral Tegmental Area Amygdala Nucleus Accumbens 5-HT ACh NE DA Krichmar (2008) Adaptive Behavior, 16,

30 Phasic vs. Tonic Neuromodulation Tonic Mode 1-6 Hz activity. Behavior is distracted and exploratory. Phasic Mode Transient burst of activity. Behavior is attentive, decisive, and exploitive. A possible phasic neuromodulatory mechanism: Amplify thalamo-cortical and inhibitory currents Dampen cortico-cortical currents Phasic Neuromodulatory Responses Noradrenergic Dopaminergic Aston-Jones and Cohen, J Comp Neurol, 2005 Redgrave & Gurney, Nature Rev Neurosci,

31 Phasic Neuromodulatory Responses Sharpen Tuning, Increase Signal to Noise Ratio Noradrenergic Serotonergic Hurley and Pollack, J Neurosci, 1999 Kobayashi, Eur J Neurosci, 2000 Possible Mechanism of Phasic Neuromodulation ( ) = Ext1w ext + E 2 ( t) w int E 2 ( t) w inh+ rnd( 1,+1) ( ) = Ext2 w ext + E 1 ( t) w int E 1 ( t) w inh+ rnd( 1,+1) E1 t+1 E2 t+1 A. Tonic Mode B. Phasic Mode Extrinsic Stimulus 1 Extrinsic Stimulus 2 Extrinsic Stimulus 1 Extrinsic Stimulus 2 E1 E2 E1 E2 I1 I2 I1 I2 NM NM 31

32 Role of Neuromodulation Exploit Environmental Events and Explore New Behaviors 5- HT DA NA ACh cost reward surprise effort phasic neuromodula2on Exploi2ve Decisive Exploratory Curious tonic neuromodula2on Krichmar (2008) Adaptive Behavior, 16, Network Model of Neuromodulation and its Effect on Attention 15 neural areas, 6,700 neuronal units, and 1.3 million synaptic connections. 32

33 Neural Dynamics and Synaptic Plasticity Mean firing rate model Computationally Efficient Roughly equivalent to recording from ~100neurons over ~100ms Neural Dynamics and Synaptic Plasticity Biologically plausible learning rule Bienenstock, Cooper & Munro learning rule with neuromodulation 66 33

34 CARL s Behavior and Neural Activity Cox & Krichmar, IEEE Robotics & Automation Magazine, September Find Response 34

35 Flee Response Effect of Neuromodulation on Behavior and Neuronal Response Find Responses Flee Responses 1.4 Signal to Noise Color Response Control BF Raphe VTA 1.2 BF+Raphe BF+VTA Ctrl BF RapheVTA BF+ BF+ Raphe VTA 0 Ctrl BF Raphe VTA BF+ BF+ Raphe VTA Dopamine was necessary for value-laden wanting responses. Serotonin was needed to respond appropriately to threatening stimuli. Acetylcholine enhanced attention to salient objects. Phasic Neuromodulation needed for appropriate action selection. amplifies salient objects and suppresses distracters Find Flee 35

36 Case Study 2 Under review Code will be made public if/when accepted Spiking Neuron Simulations 36

37 Simulation Functionality Simulation Performance of Random Networks Efficient implementation and easy-to-use interface x GHz NVIDIA Tesla GPU cards 37

38 Network Architecture for Cortical Visual Processing 32x32 Resolution 138,240 neurons; ~30 million synapses. Runs in real-time on GPU!! 64x64 Resolution 552,960; ~120 million synapses. Random-Dot Kinematogram Test 38

39 Comparing Simulated RDK responses to Human Response Model compared with human psychophysics data Resulaj et al., Nature 2009 Decision criteria Compare MT left neuron with MT right neuron Race condition: first neuron to spike 10 times V4 Orientation Responses V4 spiking neuron response to oriented gratings. 39

40 Response of V4 Neurons to Hues V4 Response 64x64 pixels 40

41 Slides for this course can be found at: Full course and slides can be found at: 41