The neurodynamics of simple decisions: drift-diffusion models for single brains, and group behaviors

Transcription

1 The neurodynamics of simple decisions: drift-diffusion models for single brains, and group behaviors Philip Holmes, Princeton University Program in Applied & Computational Mathematics, Neuroscience Institute, MAE Dept. Fuat Balci, Eric Brown (U Wash), Rafal Bogacz (Bristol U), Jonathan Cohen, Philip Eckhoff (Intellectual Ventures), Stephanie Goldfarb, Jeff Moehlis (UCSB), Andrea Nedic, Deborah Prentice, Patrick Simen, Damon Tomlin, Marieke van Vugt, KongFatt Wong-Lin (U Ulster), Miriam Zacksenhouse (Technion). Thanks to: NIMH and AFOSR. SIAM Life Sciences, Pittsburgh, PA, July 2010.

2 Contents : a continuing saga 1: Perceptual decisions. A normative theory; why simple driftdiffusion models matter. 2: The speed-accuracy trade-off: optimal and suboptimal performances in a free-response perceptual-choice task. 3: A framework for modeling decisions in social groups Conclusions: Stochastic dynamical systems can help us understand brain function (ODEs, analysis, probability theory). There s much more! MSs 7,14,16,21,25, 30,34,43,52,61.

3 1: A simple perceptual decision On each trial you will be shown one of two stimuli, drawn at random. You must identify the direction (L or R) in which the majority of dots are moving. The experimenter can vary the coherence of movement (% moving L or R) and the delay between response and next stimulus. Correct decisions are rewarded. Your goal is to maximize rewards over many trials in a fixed period. You must be fast, and correct! 30% coherence 5% coherence Courtesy: W.T. Newsome Behavioral measures: reaction time distributions, accuracy; speed-accuracy tradeoff.

4 An optimal decision procedure for noisy data: the Sequential Probability Ratio Test Mathematical idealization: During the trial, we draw noisy samples from one of two fixed distributions p L (x) or p R (x) (left or right-going dots). p L (x) p R (x) The SPRT works like this: set up two thresholds running tally of the ratio of likelihood ratios: and keep a When R n first exceeds B or falls below 1/B, declare victory for R or L. Theorem: (Wald, Barnard) Among all fixed sample or sequential tests, SPRT minimizes expected number of observations n for given accuracy. For fixed n & B =1 SPRT maximizes accuracy (Neyman-Pearson lemma).

5 The continuum limit is a DD process Take logarithms: multiplication in becomes addition. Take a continuum limit: addition becomes integration. The SPRT becomes a drift-diffusion (DD) process (a cornerstone of 20th century physics, and perhaps the simplest stochastic ODE): drift rate noise strength Here is the accumulated evidence (the log likelihood ratio). When reaches either threshold,, choose R or L respectively. But do humans (or monkeys, or rats) drift and diffuse? Evidence comes from three sources: behavior, neural recordings, and mathematical models.

6 Behavioral evidence: RT distributions Human reaction time data in free response mode can be fitted to the first passage threshold crossing times of a DD process. thresh. +Z drift A thresh. -Z Prior or bias toward one alternative can be implemented by setting starting point. Extended DD: variable drift rates & starting points. Simple expressions for mean RT, accuracy. R. Ratcliff et al., Psych Rev. 1978, 1999, P. Simen et al., J. Exp. Psych.:HPP 35(6), 2009.

7 Neural evidence 1: cortical firing rates Spike rates of neurons in oculomotor areas rise during stimulus presentation, monkeys signal their choice after a threshold is crossed. thresholds J. Schall, V. Stuphorn, J. Brown, Neuron, Frontal eye field (FEF) recordings. J.I Gold, M.N. Shadlen, Neuron, Lateral interparietal area (LIP) recordings.

8 Neural evidence 2: EEG measurements Biased stimuli: the lateralized readiness potential (LRP), originating in motor cortex prior to left/right button presses, correlates with the probability of left/right motion: blue 60%, green 75%, red 90% prob (below left). Smaller LRP increase for more certain cases. Responselocked LRPs A competing leaky accumulator model (below left), based on reduction of a spiking network, reproduces this behavior (above right). inhibn. leak y 1 y 2 inhibn. leak noise s 2 s 1 noise M.K. van Vugt, P. Simen et al., FENS Poster, 2010, and Int. Conf on Cog. Modeling, 2010.

9 Model evidence: Reduction of spiking neurons Working hypothesis: motion sensitive cells in visual cortex MT pass noisy signals on to LIP, FEF, (a network?), where integration occurs. Electrophysiological data: K.H. Britten, M.N. Shadlen, W.T. Newsome, J.D. Schall, A. Movshon & many others, in various papers, threshold LIP MT Advertisement: Come and hear more on this in MS14: Understanding the link between neuronal dynamics and Neuronal computation. 4:00-6:00 pm, Room 409. Simulation & analysis of spiking neuron models: X.-J. Wang, Neuron, 2002; KF. Wong & X.-J. Wang, J. NSci., Model of NE modulation of LIP: P. Eckhoff, K-F. Wong, H. J. Neurosci, 29 (13), 2009; Mean field reduction to 4- and 2-population models: P. Eckhoff, K-F. Wong, H. SIADS (submitted, in review), 2010.

10 2: DD predicts a speed-accuracy tradeoff The task: maximize rewards for a succession of trials in a fixed period: RT D RT Maximize the Reward Rate: (% correct /average time for resp.) response-to-stimulus interval Threshold too low X $ X X RT D D RT D RT D RT D RT D $ Too high $ $ RT D RT D RT Just right $ $ $ X RT D RT D RT D RT

11 Explicit optimum thresholds How fast to be? How careful? The DDM delivers an explicit solution to the speed-accuracy tradeoff in terms of only 3 parameters: threshold-todrift and signal-to-noise ratio and D Taking the threshold, we can express RT in terms of ER and calculate a unique, parameter-free Optimal Performance Curve: RT/D = F(ER) R. Bogacz et al. Psych. Review 113, 2006.

12 The Optimal Performance Curve (OPC) High SNRs: Middle SNRs: Low SNRs: inaccurate, accurate & fast. It pays to think. but better to go fast! For each fixed SNR and delay (RSI) D, the OPC specifies a unique threshold-todrift ratio that maximizes average rewards. In this sense, every point on the OPC is optimal. Of course, it s better to have a higher SNR, which might come with practice, but there are limits to how well we can discriminate between noisy signals. A normative theory for 2-alternative decision tasks.

13 80 subjects, 8 different D, SNR conditions. Do they adopt the optimal strategy? Some do; some don t. Are they optimizing a different function, e.g. weighting accuracy more? Are they allowing for uncertainties? Are they trying, but limited By physiological constraints? Do they need more practice? A behavioral test The model delivers quantitative predictions. Its successes and failures generate precise questions, suggest new experiments. OPC R. Bogacz et al. Quart. J Exp. Psych. (in press), 2009.

14 Suboptimality: trying to avoid errors? Modified reward rate functions with penalties for errors give families of OPCs parameterized by the accuracy weight q, e.g.: accuracy weight increasing data fit OPC W. Maddox & C. Bohil, J. Exp. Psych. 24, They provide better fits for the whole dataset, but this not the only potential explanation. Short version: Holmes et al., IEICE Trans. E88A, Long versions: R. Bogacz et al. Psych. Rev. 113, 2006, and Quart. J Exp. Psych. 63(5), 2010.

15 Suboptimality: taking necessary care? Q: Suboptimal behavior could be reckless (threshold too low) or conservative (threshold too high)? Why do most people tend to be conservative? Could it be a rational choice? Which type of behavior leads to smaller losses? A: Examine the RR function; the slope on high threshold side is smaller than slope on low threshold side, so for equal magnitudes, erring on the side of accuracy costs less. 0.5 threshold too high threshold too low optimum Appeal to Information-gap theory: choose the threshold that maximizes the worst RR over a given uncertainty range.? threshold

16 Poor timing may account for suboptimality Split the subjects into 3 groups, based on total scores over all conditions. Fit various performance curves, each with one uncertainty parameter (RSI, SNR, ). OPC All fit the top 30% well, but minimax info-gap theory based on timing (RSI) uncertainty fits best (MLE criterion). M. Zacksenhouse, H & R. Bogacz. J. Math. Psych.54, For information-gap theory, see: Y. Ben-Haim. Information Gap Decision Theory: Decisions Under Severe Uncertainty. Academic Press, New York, 2006.

17 Back to the lab: Test performance and timing Allow more training: 17 subjects, sessions. Practice makes better, if not perfect. Early sessions are better fitted by RR m, but most subjects approach OPC by session 10/13. This is done partly by Improving SNR and partly by speeding up, without significant loss in accuracy. Can fit by RA,RRm or Info-gaps OPC However, lowest SNRs are treated differently, & constant threshold fits almost as well (another story). F. Balci, P. Simen, R. Niyogi, A. Saxe, H, J.D. Cohen. Attn, Psychophys (in review), 2010.

18 Poor timers tend to overestimate optimal RTs Although major source of approach to OPC seems due to reduction of accuracy weights q, significant correlations exist between CV (SD/mean) of subjects interval time estimates and distance from OPC. Timing ability was estimated by coefficient of variation in peak interval task. 1 sec CV q for RR m Combined R 2 = (including session 1) R 2 = (excluding session 1) Multiple regressions Green: significant, p < 0.05 Blue: trend, 0.05 < p < 0.1 Incl. session 1 Excl. session 1 (MR analyses passed multicollinearity diagnostics (variance inflation factor); predictors thrown in simultaneously rather than hierarchically.) F. Balci, P. Simen, R. Niyogi, A. Saxe, H, J.D. Cohen. Attn, Psychophys (in review), 2010.

19 3. Gambling in groups: modeling social feedback A two-armed bandit game, motivated by Herrnstein, via Egelman & Montague, with limited social feedback. Feedback on last choice(s) None (alone), Choices, Rewards, Both. 115 participants, playing in groups of 5: Latin square design, all subjects played every game, all experienced every feedback condition at least once. Free choice of A or B on each trial in 2 sec window followed by delay and feedback; subjects performed 150 synchronized trials (choices) in each session. Experimental design and data collection: Damon Tomlin: Behavioral and fmri imaging data (hyperscanning x5). Here, we focus on the behavioral data, choices and rewards obtained.

20 History-dependent reward schedules Subjects play four games, choosing freely and obtaining rewards based on their 20 previous choices, with nonlinear schedules from simple converging gaussians (A), tricky diverging gaussians (B), to complicated rising optima cases (C,D), inspired by Egelman, Person & Montague. Rewards for A choice for B choice

21 The original E-P-M rising optimum task Egelman et al. found that individual subjects playing a rising optimum task divided into two types: conservatives (exploiters) remained near the matching point; risk-takers (explorers) crossed the low rewards region and approached closer to the optimal (all A) strategy. Data f B f A Allocation distributions Model exploiters explorers D. Egelman et al., J. Cog. Neurosci. 10, 1998; P. Montague & G. Berns, Neuron 36, 2002; R. Bogacz et al. Brain Research 1153, (Fixed model network to capture explorers.)

22 A model for choice informed by experience Dopamine system Soft-max choice function p A = 1 1+ e (µ(w B "w A )) Identical to correct choice probability predicted by DD process! Difference between expected rewards replaces drift due to stimulus; we update expected rewards by temporal difference reinforcement learning: A chosen on nth trial B chosen on the nth trial w A (n +1) = (1" #)w A (n) + #r, w B (n +1) = (1" #)w B (n) + #r, w B (n +1) = w B (n); w A (n +1) = w A (n); 2 parameters: specifies slope of sigmoid: => purely random choices; => deterministic limit (hard-max). Learning rate λ specifies fading memory of previous choices: λ=0 => no learning; λ=1 => memory of previous choice erased. D. Egelman et al., J. Cog. Neurosci. 10, 1998; P. Montague & G. Berns, Neuron 36, 2002.

23 Couple choice models with feedback rules Choice feedback: I know what they are doing, but not how well they re doing. Follow the majority (herding)? Go it alone (contrarian)? Add a bias to the difference in expected rewards: Reward feedback: I know they re doing better or worse that I, but what are they doing? Promote exploration: bias in favor of either choice at random: Many other rules are possible, and we have tried several. We test them by having the model predict each subject s choice sequences, given feedback from her group members. A. Nedic et al., Proc 47th IEEE-CDC, 2008, and in preparation, 2010.

24 Compare models by fitting to data Assess via max. likelihood of model, using probability predicted for each new choice given subject s previous choices and feedback from group. Inclusion of feedback yields significant improvement in model predictions, notably for choice feedback in CRO, SRO. Choice feedback 26% improvement Figures show ML values for fits to all subjects playing each game under appropriate feedback condition, ML = 1: perfect prediction; ML = 0.5: chance level; ML = 0: perfect anti-prediction. Vertical bars divide games: Rewards feedback 21% improvement Bot h feedback 19% improvement CG DG CRO SRO. CG DG CRO SRO fit significance A. Nedic et al., Proc 47th IEEE-CDC, 2008, and in preparation, 2010.

25 New findings 1: The perils of feedback Data Data Color codes show rewards, choice, and no feedback. Variances of allocation distributions Reward magnitudes Model Model Reward feedback produces great er variance t han choice, or no feedback. Performance suffers as time goes on! In the converging Gaussian game, with a stable matching point at max reward, information on other group members rewards causes greater exploration, causing a progressive decline in one s own rewards with successive choices. Choice feedback does not produce a decline. The model predicts this quite well. A. Nedic et al., Proc 47th IEEE-CDC, 2008, and in preparation, 2010.

26 New findings 2: The pleasures of feedback In contrast, on rising optimum games with multiple maxima, rewards and both feedback allows more subjects to explore and discover global maxima: blue bars = fraction who found max; red bars = fraction who failed to. The model captures this and as well as more subtle subgroup size effects, that depend on feedback mode. Graphs show distributions of subgroup sizes (1 to 5) within all groups in which at least one subject reached global optimum in SRO. more find max No fdbk Choices Rewards Both Data Model A. Nedic et al., Proc 47th IEEE-CDC, 2008, and in preparation, 2010.

27 Can the model assist fmri data analysis? We have already found that activity in certain brain regions, especially the insula, correlates well with choice alignment relative to, and reward rank in, the group. Averaged over all subjects, insula activity significantly improves prediction of next choice (below, left). Bimodality in parameter estimates suggests splitting data into two groups (long or short memory; more or less susceptible to feedback) and re-analyzing. D. Tomlin et al., Neuron (submitted, in review) Stay tuned!

28 Some of the people who did the work, and some who did other stuff.

29 Conclusions Neural activity in simple decisions resembles a DD process: DD models fit behavioral data, are consistent with electrophysiology, EEG, fmri. The DD model predicts an optimal speed-accuracy tradeoff. Experiments show some subjects are optimal, others not. A bias toward accuracy? Uncertain estimates of durations? More theory and experiments: Accuracy bias and timing ability both play roles, but subjects can learn to abandon their dislike of errors. A coupled set of DD models can capture behavior in a social gambling task: Quantifies susceptibility to influence of others behavior and performance; offers entry to brain scan (fmri) analyses. Thank you for your attention.