Mechanisms of Learning a Perceptual Decision. Joshua I. Gold University of Pennsylvania

Transcription

1 Mechanisms of Learning a Perceptual Decision Joshua I. Gold University of Pennsylvania Cognitive aspects of decision making Research Workshop Crystal City, VA September 21, 2008

2 It is a mistake to consider perception and separately because what one learns is strongly constrained by what one perceives, and what one perceives depends on what one has experienced. - Horace Barlow, 1990

3 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

4 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus Banburismus and the Brain: Decoding the Relationship between Sensory Stimuli, Decisions, and Reward This article relates a theoretical framework developed by British codebreakers in World War II to the neural computations thought to be responsible for forming categorical decisions about sensory stimuli. - Gold & Shadlen, 2002 Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

5 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus d s d s-1 There is an impressive amount of converging evidence that experimental training leads to changes to very early stages of information processing. - Goldstone, 1998 Fine & Jacobs, 2002 Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

6 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus Vnew = Vold + η(outcome prediction) Arguably, some of the most profound developments in psychology and neuroscience in the last two decades have stemmed from the use of normative ideas from reinforcement in thinking about and studying behavior and the brain....[however], wholly different categories of, such as perceptual, stimulus stimulus and episodic, do not use TD prediction errors. - Niv & Schoenbaum, 2008 Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

7 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus Readout How does the brain select and weigh the neural signals used to form decisions? Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

8 Perceptual decision-making Categorical judgments about the presence or identity of a sensory stimulus Working hypothesis: Reward-prediction errors can be used to adjust how population activity in sensory cortex is read out to form perceptual decisions, thereby improving perceptual sensitivity. Perceptual Reinforcement Relatively long-lasting changes to an organism s perceptual system that improve its ability to respond to its environment Learning how to map situations to actions to maximize reward

9

10

11 Felleman and Van Essen, 1991

12 Perceptual decisions: Banburismus Message 01: KCOW WKLE BNVE UGBF POIJ ASJD... Message 02: UIHG NCDJ WQPO LNVD BNJV HFNE... Message 03: MNVE WUIE NJLC EJKU BGER EIUF... Message 04: BEFB FCEL ISEB LOAN HAXZ POLQ... Message 05: WPOC BYUE GLWF NEWB FNEU QWED... Message 06: HEBD WFLN UASV JCBL KJNO CVEV... Message 07: NEDJ SJLS JDNK LENJ LSJD BFFV... Message 08: TYRI EWBC KEJW OZNF KCPQ WOIQ... Message 09: JFOE IUEW NBVC ZMXE EDNB EVFW... Message 10: HPIO UVEW EJFN LEWI UHRN ULIE... Message 11: NMZX BCVU HDEE LIJV MSAN LUBV... Message 12: NMCD EHNF ECSN IEFL VENS OIHN... Message 13: OAIS JCEU ICWI UNGV BTIO UERG Problem: How to decide between two alternatives, given readily available but uncertain evidence? Compute weight of evidence Accumulate Stop at pre-defined threshold

13 Perceptual decisions: Banburismus Message 01: KCOW WKLE BNVE UGBF POIJ ASJD... Message 02: UIHG NCDJ WQPO LNVD BNJV HFNE... Message 03: MNVE WUIE NJLC EJKU BGER EIUF... Message 04: BEFB FCEL ISEB LOAN HAXZ POLQ... Message 05: WPOC BYUE GLWF NEWB FNEU QWED... Message 06: HEBD WFLN UASV JCBL KJNO CVEV... Message 07: NEDJ SJLS JDNK LENJ LSJD BFFV... Message 08: TYRI EWBC KEJW OZNF KCPQ WOIQ... Message 09: JFOE IUEW NBVC ZMXE EDNB EVFW... Message 10: HPIO UVEW EJFN LEWI UHRN ULIE... Message 11: NMZX BCVU HDEE LIJV MSAN LUBV... Message 12: NMCD EHNF ECSN IEFL VENS OIHN... Message 13: OAIS JCEU ICWI UNGV BTIO UERG A deciban or half-deciban is about the smallest change in weight of evidence that is directly perceptible to human intuition. I feel that it is an important aid to human reasoning and will eventually improve the judgments of doctors, lawyers and other citizens. IJ Goode Compute weight of evidence Accumulate Stop at pre-defined threshold

14 Perceptual decisions: Banburismus Turing: Banburismus Wald, Bernard: SPRT Optimal in the sense that for a fixed error probability (threshold), this procedure guarantees the smallest number of samples; i.e., RT Compute weight of evidence Accumulate Stop at pre-defined threshold

15 Perceptual decisions: direction discrimination Motion is leftward look leftward Motion is rightward look rightward Compute weight of evidence Accumulate Stop at pre-defined threshold

16 Gold and Shadlen, 2007

17 Perceptual decisions: direction discrimination 70 end accumulation? Sensory evidence MT decision variable LIP SC Firing rate (spikes/sec) begin accumulation? high coherence medium coherence low coherence FEF (oculomotor areas) aligned to motion onset Time (msec) aligned to saccade Compute weight of evidence Accumulate Stop at pre-defined threshold Gold and Shadlen, 2002

18 Perceptual decisions: direction discrimination p i = 1 t = tanh(b µ ) 1 + e -2B µ i i i 1 Probability correct Response time (msec) Motion strength (% coherence) Motion strength (% coherence) Compute weight of evidence Accumulate Stop at pre-defined threshold Gold and Shadlen, 2002

19 Perceptual : physiology evidence in area MT decision in area LIP Sensory neurons tuned for the direction of motion Neuronal sensitivity matches behavioral sensitivity Electrical microstimulation biases performance Lesions impair performance MT LIP Sensory-motor neurons tuned for the location of saccade targets Activity during motion viewing reflects accumulation of motion information into categorial choice Electrical microstimulation biases performance dt

20 Perceptual : physiology 60 Sessions 1 50 N=33 40 Sessions 1 50 N= early training Sessions N= Sessions N= middle training Sessions N= Sessions N=29 MT Activity (sp/s) 30 late training LIP Activity (sp/s) Time (s) Time (s) Law & Gold, 2008

21 Reinforcement : a working model In the world In the brain Stimulus MT population responses, x i high high Pooling weights, w i Decision noise # right Decision rule Outcome Sensitivity low left right Direction tuning Sensitivity low left w n+1 $ w n + %w Learning rule right Direction tuning!w i x i " Probability of reward for choosing right Choice left 0 Pooled response Reward prediction 0 Pooled response!!

22 Reinforcement : a working model 80 Data Model 1.0 Threshold,! (% coh) Lapse rate Session (d) Session (x1000 simulated trials)

23 Reinforcement : assessment of optimality high Coarse discrimination Fine discrimination Reinforcement model simulations Neurometric sensitivity low high Analytic solutions Neurometric sensitivity low! Direction tuning! Direction tuning