Neurobiological Foundations of Reward and Risk

Transcription

1 Neurobiological Foundations of Reward and Risk... and corresponding risk prediction errors Peter Bossaerts 1 Contents 1. Reward Encoding And The Dopaminergic System 2. Reward Prediction Errors And TD (Temporal Difference) Models 3. Alternative Approaches: 3.1. Pharmacological 3.2. Direct Dopamine Measurement 4. Risk: Variance 5. Risk Prediction Errors 6. The Norepinephrine Story 7. Skewness 8. Correlation 9. Processing Time 2

2 1. Reward Encoding And The Dopaminergic System Go back to Class 2: Single-Unit Recording Of Dopamine Neurons fmri Analysis Of Reward (And Risk) Remark: fmri focuses on projection areas of Dopamine Neurons 3 2. Reward Prediction Errors And TD (Temporal Difference) Models Dopamine neurons do NOT signal expected rewards but reward prediction errors! (Expected rewards/values are encoded in ventro-medial prefrontal cortex, among others) 4

3 Prediction Error Learning Simple Rescorla Wagner Learning Rule! Prediction! F t +1 = F t + αη t. Notice relation between math and emotions! +:!Elation! DOPAMINE!,:!Disappointment! Bossaerts!@!Claremont!Athenaeum! 7! 5 TD Models: Learning (To Do) The Right Thing Through Reinforcement (Prediction Errors) Can learn to assign value (of discounted future rewards) to complex signals Derives from dynamic programming... 6

4 Dynamic Programming Value function V States S (transiting to S ) Actions to be taken... while learning value function (Converges IF RIGHT EXPLORATION; see Watkins-Dayan 1992) 7 Dorsal vs Ventral Striatum Top: Ventral Striatum (A: Pavlovian; B: Instrumental (Conditioning) Bottom: Caudate (From O Doherty) 8

5 3.1. Pharmacological Evidence L-Dopa (Green): Dopamine agonist Haloperidol (Red): Dopamine Antagonist Placebo = Grey... in a two-armed bandit task (From Pessiglione) 9 (Curious Difference In Loss Learning) INSULA MANIPULATION AFFECTS LOSS LEARNING Same task, but now with insula lesion patients (Unpublished, Pessiglione) 27 OPPONENT PROCESS THEORY INSULA ACTIVATION IN LOSS AVOIDANCE TASK PREDICTS SUCCESS IN (LOSS) BANDIT PROBLEM 1 MONTH LATER For best control, let two opponent forces balance each other (thumb + index) (a) Reward prediction errors in striatum, GAIN and (2 nd row) LOSS conditions (b) Punishment prediction errors in insula, LOSS trials only Notice: opponent process in loss trials (Not unlike pain avoidance/relief: Seymour ea 2005) 26 a b Insula Peak Voxel z = 4.71, p < , p rep =.999 Avoidance Learning (% Correct) Younger Adults Older Adults 100 r =.45, p <.05, p rep = Percentage Signal Change 0.2 Fig. 1. Task structure for a representative trial. Samanez- Larkin ea, Psych Reviews 28 10

6 Important Remark The idea that gains and losses are to be valued separately (as in Prospect Theory or Disappointment Aversion) squares well with the neurobiological foundation Of course, it is not clear (yet) where the brain sets the reference point! (What is a loss?) Direct Dopamine Measurement Day ea, Nat Neuro 2007: dopamine release in Nucleus Accumbens of rats is correlated with reward predictive cues Notice learning effect! 12

7 4. Risk: Variance See Class 2 Slides. (Can one use signals to predict choice? Yes!) Using brain signals to predict choice! VSt (ventral striatum): correlates with expected reward! ACC (anterior cingulate cortex): correlates with (objective) risk! IFG (inferior frontal gyrus): correlates with (subjective) risk (Christopoulos ea, J Neuroscience 2009) (ACC and IFG have opposite effects: opponent theory in biology) fmri brain signals predicts risk taking Getting At Causality... Disrupting process using transcranial magnetic stimulation! Disrupting inferior frontal gyrus leads to reduced risk aversion (Knoch ea, J Neuroscience 2006) 14 14

8 5. Risk Prediction Errors Risk Prediction Error = SIZE of reward prediction error minus EXPECTED SIZE (variance) (Driving term of a GARCH process) The Norepinephrine Story See pupil dilation slide, Class

9 (Sophisticated - and only 1 step ahead) EXAMPLE: BET= SECOND CARD LOWER CARD 1=3 CARD 2=2 t 0 : Prediction of Forecast = 0 t 1 : Forecast based on card 1 = -5/9 Risk Prediction Errors t 2 : Outcome = +1 (t 0,t 1 ) (Forecast) Risk ~ 0.6 (t 1,t 2 ) (Outcome-Forecast) Risk ~ Relation With Tracking Changes In The Environment? AND NOREPINEPHRINE Enhancing NE levels induces rats to abandon old hypotheses and find the newly optimal paths in a navigation task (Or are they more sensitive to signals of UNEXPECTED UNCERTAINTY, i.e., that something changed? Yu-Dayan, Neuron 2005) 32 18

10 7. Higher Moments Skewness! Skewness = one-sided outliers! Positive & negative skewness: anterior insula (again)! Positive skewness also ventral striatum (see expected reward)! Wu ea, PLoS ONE (2011) Variance-Skewness See MEG results, Class 2 20

11 8. Correlation The Task 21 Results... (Wunderlich, Symmonds, ea Neuron 2011) 22

12 8. Processing Time Computing value takes time because all components have to be evaluated (reward, risk,...) and then put together... What happens if we constrain subjects to make a decision within 1s, 3s (and 5s)? 23 Results 1s Prob(gamble) Increases in Expected Reward Decreases in Variance Decreases in Skew (Risk Loving for Losses) Increases in PRICE 5s Prob(gamble) Increases in Expected Reward Insensitive to Variance Decreases in Skew (more!) Insensitive to PRICE 24