Choosing the Greater of Two Goods: Neural Currencies for Valuation and Decision Making

Transcription

1 Choosing the Greater of Two Goods: Neural Currencies for Valuation and Decision Making Leo P. Surgre, Gres S. Corrado and William T. Newsome Presenter: He Crane Huang 04/20/2010

2 Outline Studies on neural correlates of simple perceptual decisions Interactions between decision making and reward Studies of value-based decisions on free-choice tasks

3 Neural correlates of perceptual decisions Sensory input Motor output Sensation Action LIP FEF SEF V1 V4 Area 46 IT Figure 1 Visual and oculomotor systems of the primate Visual and oculomotor systems of the primate brain

4 Action Conceptual framework for (2AFC) decision making 2-Alternative forced-choice discrimination task a Conceptual framework for simple perceptual decisions Sensory Sensory input b Conceptual framework f simple value-based dec Actor Sensory input + physiological need Sensory representation Common reward currency Decision E Action implementation Probability of choice Choice Decision representation Probability of choice P

5 REVIEWS (Perceptual) Decision-making tasks a Perceptual discrimination task % left choices Left? Motion b Linking behaviour to perception Right? d Linking behaviour to valuation and recordings from monkey caudate neurons during simple associative conditioning tasks show activity that is consistent with the creation of such stimulus response bonds However, the direct yoking of stimuli to actions and outcomes implied by the current generation of these models fails to capture the facility with which higher organisms construct complex representations of value and flexibly link them to action selection (BOX 2). Responding to these limitations, more recent theoretical proposals have expanded the role of the dopamine signal to include the shaping of more abstract models of valuation Consistent with this approach, FIG. 2b portrays the dopamine system as a critic whose influence extends beyond the generation of simple associative predictions to the construction and modification of complex value s. In this scheme, the striatum is considered to have the crucial role of liaison between actor and critic. If correct, this proposal indicates that dopamine neurons have access to the value representation depicted in FIG. 2b. Consistent with this idea, Nakahara and colleagues 50 recently showed that dopamine responses were strongly modulated by contextual information that pertained to the evolution Roitman of and reward Shadlen, probability 2002 across successive trials in a task, even when this information was not accompanied by any explicit sensory LIP activity during the RT-direction discrimination task 9482 J. Neurosci., November 1, 2002, 22(21): Roitman and Shadlen Response Stimulus strength (left) Figure 7. Time course of LIP activity in the RT-direction-discrimination task. A, Average response from 54 LIP neuro Figure (Task 3 difficulty Decision-making decrease/ tasks. Coherence a General Increases) motion strength and choice as indicated by color and line type. The responses are aligned to two events in the trial. On structure to the of a onset perceptual of stimulus discrimination motion. Response task, in averages in this portion of the graph are drawn to the median RT for each m which a monkey reports its judgement of the direction activity of motion within in a random 100 msec dot ofstimulus eye movement with initiation. On the right, responses are aligned to initiation of the eye movemen

6 LIP activity in RT-motion discrimination task Implements the decision. Convert a sensory representation of visual motion into a decision variable Predicts not only the decision, but when the decision has been reached. [box1]: Encode a mixture of sensory, motor and cognitive signals that might guide decisions about upcoming behavioral responses. LIP reflects a general decision variable that is monotonically related to the log likelihood ratio that the animal will select on of the two alternatives. -- Thursday a paper

7 Decision making and reward Adaptive decisions -take reward into account a Conceptual framework for b Conceptual framework for simple perceptual decisions simple value-based decisions a Conceptual framework for b Conceptual Actor framework Sensory forinput simple perceptual decisions simple value-based + physiological decisions needs Sensory input Actor Sensory input + physiological needs Sensory Striatum Common reward currency Striatum Sensory Common reward representation Decision currency Error Outcomes signal Decision Action implementation Probability of choice Action implementation Choice Probability of choice Decision representation Decision Action implementation Probability of choice Error signal representation Probability Predictions of choice Dopamine (VTA) Predictions Critic Outcomes Dopamine (VTA) Critic Action Choice implementation Figure 2 Conceptual frameworks for decision making. A conceptual framework that illustrates proposed processing stages for Choice the formation of simple perceptual and value-based expect Psychol the infl making notably reward most ph ing hol activity transfor tors beg order to adaptive A co value-b clusive n from w internal sentatio hand si propose

8 Decision making and reward A common neural currency for reward Reward: anything that an animal will work to acquire, consists motivational and affective dimensions. Brain stimulation reward (BSR): there Decisionis a dedicated neural network devoted to reward processing. a Conceptual framework for simple perceptual decisions Probability of choice Shizgal and colleagues: BSR contributes Action as a implementation reward signal that is responsible for valuation. b Conceptual framework for simple value-based decisions Actor Sensory input + physiological needs Decision Action implementation Common reward currency representation Probability of choice Choice Striatum Error signal Predictions Outcomes Dopamine (VTA) Critic Figure 2 Conceptual frameworks for decision making. A conceptual framework that illustrates proposed processing stages for the formation of simple perceptual and value-based expectati Psycholog the influe making in notably abs reward is most physi ing hold r activity th transform tors begun order to e adaptive b A conc value-base clusive no from whic internal re sentation hand side proposed this fram comprises

9 Decision making and reward REVIEWS Incentives and errors framework for eptual decisions bility of b Conceptual framework for simple value-based decisions Actor Sensory input + physiological needs Decision Action implementation Common reward currency representation Probability of choice Choice Striatum Error signal Predictions Outcomes Dopamine (VTA) Critic eptual frameworks for decision making. A conceptual framework that sed processing stages for the formation of simple perceptual and value-based expectation of the likely abundance of fish. Psychologists and economists have long appreciated the influence of reward and valuation on decision making in higher mammals 24, but these factors were notably absent from our preceding discussion. Although reward is an implicit variable in every operant task, most physiological studies of perceptual decision mak- Dopamine system: ing hold reward contingencies constant to isolate activity that is specifically related to sensorimotor s (FIG. 2a). Only recently have investigators begun to manipulate reward independently in order to explore the neural basis of valuation and adaptive behaviour. a central role in processing the motivational aspect of reward do not signal the occurrence of rewards, but A conceptual can be framework considered within to which code to consider reward prediction value-based error choice is proposed in FIG. 2b. Neither conclusive nor complete, it is intended as a starting point from which to discuss the basic steps in building an internal representation of value and using that representation to guide behaviour. Focus first on the lefthand side of this diagram (labelled actor ). Like the proposed framework for perceptual decisions (FIG 2a), this framework for value-based decision making comprises three key processing stages. At the first stage, a value takes the input

10 -based decision making The cortex as the stage for valuation Anatomically, several regions within the prefrontal and parietal association cortices are positioned to link reward to behavioral responses (motor planning). V1 V4 LIP IT FEF a Conceptual framework for simple perceptual decisions Decision SEF Action implementation Area 46 Probability of choice Figure 1 Visual and oculomotor systems of the primate b Conceptual framework for simple value-based decisions Actor Sensory input + physiological needs Common reward currency Striatum Error signal Outcomes Critic identified sensory representations as well as decisionrelated signals in areas of the parietal representation Dopamine and frontal (VTA) cortices. At the neural level, differentiating sensory signals from decision-related Decision signals is relatively straightforward. First, sensory signals require the presence of the Predictions sensory stimulus, Probability and of extinguish with stimulus offset. Second, and more choiceimportantly, in discrimination tasks in which behavioural Action decisions and neural activity are implementation measured across a range of stimulus strengths, animals make both correct Choice and incorrect judgements in response to the presentation of identical stimuli. On these trials, sensory neurons encode the visual stimulus itself, Figure 2 Conceptual frameworks for decision making. A conceptual framework that illustrates proposed processing stages for the formation of simple perceptual and value-based expectat Psycholog the influe making in notably ab reward is most phys ing hold activity th transform tors begu order to e adaptive b A conc value-base clusive no from whic internal re sentation hand side proposed this fram comprise stage, a v

11 -based decision making free choice task design c Free-choice task Improve Left? Right? Limitation of the the imperative tasks: -The value is all or none -The decision is simple one-to-one mapping

12 -based decision making Demonstrating behavioral control Two different approaches: Nash equilibrium from the theory of competitive games Provide a means of assessing behavioral control. The matching law from a general principle of animal foraging behavior.

13 -based decision making Understanding local strategy Nash Equations Matching law average behavior at equilibrium Local strategies that produce these average behavioral phenomena? (-based behavior control: behavior is under stimulus control ) Quantitative modeling of local choice strategy: the variables link reward history to behavior Neurophysiological exploration of model variables: how to understand the model at the neural level.

14 -based decision making Three free-choice studies signals in frontal cortex The matching pennies game, Barraclough and colleagues signals in parietal cortex behavioral dynamics behavioral equilibria Matching behavior and value representation, Leo P. Sugre, et al. The inspection game, Dorris and Glimcher

15 -based decision making free-choice study 1: signals in frontal cortex Frequency histograms of P(right) a Matching pennies Computer chooses Left Right 1 0 Monkey chooses Left Right Payoff for monkey Payoff for computer Figure 4 Payoff matrices for competitive g Barraclough and colleagues: matching pennies game Foreperiod delay

16 -based decision making free-choice study 1: signals in frontal cortex Reinforcement learning algorithms provide a general framework for finding optimal strategies in a dynamic environment. Prefrontal cortex might have a key role in optimizing decision-making strategies.

17 -based decision making free-choice study 2: signals in parietal cortex (area LIP)-behavioral dynamics Matching behavior in monkeys Leo P. Sugre, et al * Rewards are assigned to the two colors at rates that are independent and stochastic. * Once assigned, a reward remains available until the associated color is chosen. * This persistence of assigned rewards means that the likelihood of being rewarded increases with the time since a color was last chosen... and ensures that matching approximates the optimal probabilistic strategy in this task.

18 -based decision making free-choice study 2: signals in parietal cortex (area LIP)- behavioral dynamics Dynamic matching behavior a A local model of matching behavior Leaky integrators Local income Local fractional income Probability of choice τ I red FI red PC red Reward histories τ I green b c d Probability of choice (red) Model Monkeys Local fractional income (red) 1 Cumulative red choices Model Monkeys Cumulative green choices Neural response (spikes s 1 ) Into the RF Out of the RF 0 1 Local fractional income (RF target) Figure 6 A local model of matching behaviour. a A linear nonlinear probabilistic model uses leaky integration over recent

19 -based decision making free-choice study 2: signals in parietal cortex (area LIP)-behavioral dynamics Local fractional income is the valuation variable that modulates LIP firing rates. LIP neuron activity predicts the monkey s eye movement responses, contribute to plan shifts in gaze or visual attention.

20 -based decision making free-choice study 3: signals in parietal cortex (area LIP)-behavioral equilibria Human vs. Human in 3 blocks Nash prediction Risky (subject) Inspect (opponent) Dorris and Glimcher, 2004

21 -based decision making free-choice study 3: signals in parietal cortex (area LIP)-behavioral equilibria c Probability of choice (%) Trial number Neural response (spikes s 1 ) A player s overall choice distribution should equalize the average payoff resulting from alternative action.(global Nash equilibrium) LIP encodes each alternative s average payoff (which is a constant), rather then the probability of choosing that alternative (which varies). LIP encodes an abstract representation of the stimuli apart from specific motor plan.

22 Conclusion New efforts to understand value-based decision making might bring together two areas of neuroscience that have traditionally existed in separate spheres the study of perception and cognition, and the study of reward and motivation. The study of value-based choice might be uniquely positioned to lay the foundations for this unified neurobiology of choice behavior.