The Computing Brain: Decision-Making as a Case Study

Transcription

1 The Computing Brain: Decision-Making as a Case Study Prof. Angela Yu Department of Cognitive Science ajyu@ucsd.edu August 9, 2012

2 Understanding the Brain/Mind Behavior Neurobiology? Cognitive Neuroscience A powerful analogy: the computing brain

3 Understanding the Brain/Mind X A powerful analogy: the computing brain

4 Brain << Supercomputer

5 Brain >> Supercomputer

6 What is the Brain Good for? We have a brain for one reason and one reason only and that s to produce adaptable and complex movements. (Daniel Wolpert TED takj) The sea squirt has a cerebral ganglion used for movement... It directs the animal to an appropriate rock to settle down on. When a rock is found, the young squirt attaches to it, and proceeds to digest the cerebral ganglion, spinal cord, and the tail for energy. The brain is important for behavior Adaptive behavior requires making intelligent decisions

7 How do We Make Decisions? Example 1: Ordering food Deliberation Delay... optimizes choice... eating alone Speed-Accuracy Trade-off

8 How do We Make Decisions? Example 2: Stop or go at traffic light Sensory uncertainty how far away is the intersection? is that a yellow light or a yellow street lamp? Action uncertainty can the car stop in time? (rental car, rain) Prior information duration of yellow light, P(cop), $ ticket Relative costs cops/tickets versus temporal delay

9 Fundamental Trade-off Speed vs. Accuracy Slow response greater accuracy but higher time cost What is the optimal tradeoff? Are humans/animals optimal? How does the brain implement the computations?

10 Marr s Three Levels of Analysis David Marr (1969): Brain = Information Processor computational goals of computation why things work the way they do algorithmic representation of input/output how one is transformed into the other V := sup (,µ) E 1 m { +T0 < } j=1 r j1 {µ=j,m=j} E [( + T 0 ) ] implementational how is the system physically realized neural representation and dynamics

11 Marr s Three Levels of Analysis behavior our approach computational algorithmic implementational biology

12 Decision Making = Bayesian Inference + Decision Theory Decision-making Sensory processing (statistical inference) Action selection (value comparison) inferring state of the world prediction what happens next learning statistical contingencies decision policy: mapping between perceived world states and actions action selection should be contextdependent/goal directed

13 Time Matters The observer decides when to act, as well as how

14 Random Dot Coherent Motion Paradigm Left or Right? Easy Difficult 30% coherence 5% coherence vs.

15 Monkey Decision-Making Random dot coherent motion paradigm

16 Behavioral Data Accuracy vs. Coherence more accurate <RT> vs. Coherence (Roitman & Shadlen, 2002) faster Easier

17 Neural Data Saccade generation system 100% (easy) MT: sustained response Response MT tuning function coherence direction 26% (hard) Response Time Preferred Time Anti-preferred Direction ( ) (Britten & Newsome, 1998) Time Time (Britten, Shadlen, Newsome, & Movshon, 1993)

18 Saccade generation system Neural Data LIP: ramping response (Roitman & Shadlen, 2002)

19 Neural Data Saccade generation system (Roitman & Shalden, 2002) LIP response reflects sensory information ramping slope depends on coherence (difficulty) LIP response reflects perceptual decision 0% stimulus different response depending on choice neural activation reaches identical peak before saccade

20 Marr s Three Levels of Analysis behavior our approach computational algorithmic implementational biology

21 Decision-Making: Computations Random dot coherent motion paradigm Repeated decision: Left or right? Stop now or collect more data?

22 Decision-Making: Computations (50 ms) (100 ms) (150 ms) +: more accurate wait wait -: more time L R L R L R

23 Decision-Making: Computations Luckily, Mathematicians Solved the Problem Wald & Wolfowitz (1948): hypothesis 1 (left) vs. hypothesis 2 (right) Optimal policy cost function: Pr(error) + c RT accumulate evidence over time: Pr(left) versus Pr(right) stop if evidence/confidence exceeds left or right boundary

24 Decision-Making: Algorithm Wald & Wolfowitz (1948): hypothesis 1 (left) vs. hypothesis 2 (right) Optimal policy cost function: Pr(error) + c RT accumulate evidence over time: Pr(left) versus Pr(right) stop if total evidence exceeds left or right boundary Evidence trial 1 trial 2 left boundary right boundary

25 Decision-Making: Implementation MT: motion sensory input LIP: evidence accumulation coherence direction Direction ( ) (Britten & Newsome, 1998) (Roitman & Shalden, 2002)

26 Model Behavior Model: harder slower more errors easy hard boundary Accuracy vs. Coherence <RT> vs. Coherence

27 Model Neurobiology Model LIP Neural Response

28 Putting it All Together behavior Evidence biology

29 Marr s Three Levels of Analysis David Marr (1969): Brain = Information Processor computational goals of computation why things work the way they do L( )=ce( )+P (d = s) algorithmic representation of input/output how one is transformed into the other Q t b 0 a implementational how is the system physically realized neural representation and dynamics

30 Extensions Multiple choices (Yu & Dayan, 2005) Temporal uncertainty about stimulus onset (Yu, 2007) Decision deadline (Frazier & Yu, 2008) Learning about Pr(right) vs. Pr (left) over trials (Yu, Dayan, & Cohen, 2009) Minimizing Pr(error)+c*(mean RT) versus maximizing reward rate = Pr(correct)/(RT) (Dayanik & Yu, in press)

31 Caveat: Model Fit Imperfect (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) Accuracy Mean RT Errors slower than correct responses

32 Standard DDM Predicts Identical RT B 1-B P(correct) = B for each RT Correct & error RT distributions should be identical

33 Fix 1: Variable Drift Rate (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) (idea from Ratcliff & Rouder, 1998) Accuracy Mean RT

34 Fix 2: Increasing Drift Rate (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) Accuracy Mean RT

35 A Principled Approach Trials aborted w/o response or reward if monkey breaks fixation (Data from Roitman & Shadlen, 2002) (Analysis from Ditterich, 2007) more urgency over time as risk of aborting trial increases fixation breaking acts as stochastic deadline

36 Imposing a Stochastic Deadline (Frazier & Yu, 2007) Loss function depends on error, delay, and whether deadline exceed Optimal policy is a pair of decaying/collapsing bounds 1 Probability 0 Time Generalized SPRT Classic SPRT Intuition for decay With limited time, there is limited opportunity to observe enough data to bias the evidence in the opposite direction

37 1Probability Decaying Thresholds Slower Error RT Generalized SPRT 0 Time Classic SPRT p(τ) correct error τ

38 Imposing a Stochastic Deadline (Frazier & Yu, 2007) Earlier deadline translation of thresholds Probability Time

39 Timing Uncertainty (Frazier & Yu, 2007) Timing uncertainty faster decaying thresholds Probability Time result applies to many deadline distributions: delta, gamma, normal also to more general super-linear time costs beyond deadline

40 Beyond Linear Loss Function? Classic loss/risk is linear in error and delay: What if it s non-linear, e.g. maximize reward rate: (Gold & Shadlen, 2002; Bogacz et al, 2006; Simen et al, 2009) R( )= L( )=ce( )+P (d = s) P (d = s) E[ ], L( )= Wald & Wolfowitz also proved a dual statement: E[ ] 1 P (d = s) Amongst all decision policies satisfying the criterion SPRT (with some threshold) minimizes the expected sample size Ε[τ] Thus, SPRT is optimal for all loss functions that increase with P(error) and (linearly in) delay (HW: proof by contradiction), including max reward rate.

41 Min Bayes Risk & Max Reward Rate (Dayanik & Yu, under review) No such dual statement exists for deadlined decision-making R( )= P (d=s)p( <D) E[ ] It cannot be solved numerically directly (dynamic programming) but is equivalent to a special case of the linear loss function L( )=c E[ ]+P (d=s)p ( <D)+P ( =D), c=max we can show L * (π;c * )=1 L * (π;c) monotonically increases in c bijection algorithm: guess c, use DP to compute L * (π;c) if L * (π;c) 1, stop elseif L * (π;c)<1, increase c elseif L * (π;c)>1, decrease c repeat P (d=s)p( <D) E[ ]

42 Additional References Mathematical Foundation Sheldon Ross, Introduction to Stochastic Dynamic Programming, Academic Press Inc., New York, Y. S. Chow, Herbert Robbins, and David Siegmund, Great expectations: the theory of optimal stopping, Houghton Mifflin Co., Boston, Mass., Applications to cognitive science/neuroscience Yu, A J (2007). Optimal change-detection and spiking neurons. Advances in Neural Information Processing Systems 19: MIT Press, Cambridge, MA. Frazier, P, Yu, A J (2008). Sequential hypothesis testing under stochastic deadlines. Advances in Neural Information Processing Systems, 20: MIT Press, Cambridge, MA. Shenoy, P & Yu, A J (2011). Rational decision-making in inhibitory control. Frontiers in Human Neuroscience, 5:48. doi: /fnhum

43

44 Bayesian (Sequential) Statistical Inference Generative Model: statistical assumptions about the world/task prior p(s; ) hidden variable (world state, e.g. stimulus identity) s likelihood p(x s; ) x t 1 x t x t+1 independent/iid noise p(x t s; )= t i=1 p(x i s; ) x t := (x 1,...,x t )

45 Bayesian Probability Theory Recognition Model: infer causes of observations s hidden variable: L or R? observed data x t 1 x t x t+1 Bayes Rule p(s x t ) p(x t s)p(s x t 1 ) posterior(t) likelihood(t) posterior(t-1)=prior(t)

46 Bayesian Decision Theory Decision policy A policy π: x d is a mapping from input x into decision d, where x is a sample from the distribution p(x s). Loss (cost/risk) function: The loss function l(d(x); s) depends on the decision d(x) and the hidden variable s. Expected loss (Bayes risk, versus e.g. minimax): The expected loss associated with a policy π is averaged over x and s, and d if the policy is stochastic: L( )=E[l(d(x); s)] x,s,d The optimal decision policy π* minimizes this expected loss.

47 Examples of Basic Decision Theory Ex. 1: loss function is mean square error l(ŝ; s) =(ŝ s) 2 For each observation x, we need to minimize: E[l(ŝ; s)] = (ŝ s) 2 p(s, x) ds dx ŝ Differentiate wrt and set derivative = 0, we find: ŝ(x)p(s x)ds p(x)dx = sp(s x)ds p(x)dx ŝ(x)p(x)dx = E[s x]p(x)dx Optimal policy: report the mean of posterior p(s x) ŝ

48 Other Basic Examples Ex. 2: loss function is absolute error: Optimal policy: median of posterior (homework!) ŝ Ex. 3: loss function is 0-1 error: 0 if correct, 1 all other choices Optimal policy: mode of posterior; MLE/MAP (homework!) ŝ

49 Special Distributions with Trivial Solutions Ex 1: Binary distribution 1-q p(s) q Trivial optimal policy: choose better option Ex 2: Gaussian distribution Trivial optimal policy: mean = median = mode Unfortunately, life is usually more complicated

50 Understanding the Brain/Mind? Behavior Neurobiology? Cognitive Neuroscience A powerful analogy: the computing brain

51 Sequential Probability Ratio Test Log posterior ratio undergoes a random walk: r t is monotonically related to q t, so we have (a,b ), a <0, b >0. b r t 0 a In continuous-time, a drift-diffusion process w/ absorbing boundaries.

52 Generalize Loss Function We assumed loss is linear in error and delay: What if it s non-linear, e.g. maximize reward rate: Wald also proved a dual statement: (Bogacz et al, 2006) Amongst all decision policies satisfying the criterion SPRT (with some thresholds) minimizes the expected sample size <τ>. This implies that the SPRT is optimal for all loss functions that increase with inaccuracy and (linearly in) delay (proof by contradiction).

53 Saccade generation system Neural Implementation LIP - neural SPRT integrator? (Roitman & Shalden, 2002; Gold & Shadlen, 2004) LIP Response & Behavioral LIP RTResponse & Coherence Time

54 Caveat: Model Fit Imperfect (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) Accuracy Mean RT RT Distributions

55 Fix 1: Variable Drift Rate (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) (idea from Ratcliff & Rouder, 1998) Accuracy Mean RT RT Distributions

56 Fix 2: Increasing Drift Rate (Data from Roitman & Shadlen, 2002; analysis from Ditterich, 2007) Accuracy Mean RT RT Distributions

57 A Principled Approach Trials aborted w/o response or reward if monkey breaks fixation (Data from Roitman & Shadlen, 2002) (Analysis from Ditterich, 2007) More urgency over time as risk of aborting trial increases Increasing gain equivalent to decreasing threshold

58 Imposing a Stochastic Deadline (Frazier & Yu, 2007) Loss function depends on error, delay, and whether deadline exceed Optimal Policy is a sequence of monotonically decaying thresholds Probability Time

59 Timing Uncertainty Timing uncertainty lower thresholds (Frazier & Yu, 2007) Probability Time Theory applies to a large class of deadline distributions: delta, gamma, exponential, normal

60 Some Intuitions for the Proof (Frazier & Yu, 2007) Continuation Region Shrinking!

61 Summary Review of Bayesian DT: optimality depends on loss function Decision time complicates things: infinite repeated decisions Binary hypothesis testing: SPRT with fixed thresholds is optimal Behavioral & neural data suggestive, but imperfect fit. Variable/increasing drift rate both ad hoc Deadline (+ timing uncertainty) decaying thresholds Current/future work: generalize theory, test with experiments

62 Aside: Saccadic Eye Movement

63