Decision Making in Robots and Autonomous Agents

Transcription

1 Decision Making in Robots and Autonomous Agents A Brief Survey of Models from Neuroeconomics Subramanian Ramamoorthy School of Informa>cs 31 March, 2015

2 What is Neuroeconomics? Studies that take the process of choice seriously Studies that make use of data on the processes by which choices are made Content of the science of Neuroeconomics: Loca=ng correlates of economic (decision theore=c) concepts in the brain Causal studies A structural model of simple choice 31/3/2015 2

3 Method: Loca=ng Economic Concepts in the Brain Run a behavioral experiment to get people to exhibit some behaviour e.g. Punishment Scan (e.g., using fmri) brain while they are doing so Find areas of brain whose ac=vity correlates with behavior Conclude that this is where the related preference lives Preference for equality (or punishment) 31/3/2015 3

4 Recall: Ul=matum Game (Sanfey et al., Science Vol. 300, 2003) Player A (Proposer) has $10 Makes an offer to player B (Responder) of the form I will take x and you take $10- x Player B can either accept offer, or reject offer in which case and both get $0 31/3/2015 4

5 Experiment with Ul=matum Game Responder s brain ac=va=ons measured by fmri in a $10 UG. A responder faces each of three condi=ons ten =mes. Offers from a (supposed) human partner Random offers from a computer partner Research Ques=ons: Which brain areas are more ac=vated when subjects face... fair offers (3-5) rela=ve to unfair offers (1-2). the offer of human proposer rela=ve to random computer offer Method (very simplified): Regression of ac=vity in every voxel (i.e, 3D Pixel) in the brain on the treatment dummy (i.e., unfair offer dummy, human proposer dummy) 31/3/2015 5

6 Ul=matum Game Experiment 31/3/2015 6

7 Ul=matum Game Experiment 31/3/2015 7

8 Ul=matum Game Experiment: Differences in brain ac=vity between unfair and fair offers from a human proposer 31/3/2015 8

9 Ul=matum Game Experiment 31/3/2015 9

10 Ul=matum Game: Some Results Regions showing stronger ac=va=ons if subjects face unfair human offers rela=ve to fair human offers (the same regions also show more ac=va=on if the unfair human offer is compared to unfair random offers). Bilateral anterior Insula, anterior cingulate Cortex Emo=on- related region Insula also has been associated with nega=ve emo=ons such as disgust and anger. Dorsolateral prefrontal Cortex (DLPFC) Cogni=on- related region Associated with control of execu=on of ac=ons Associated with achievement of goals. Unfair offers are more likely to be rejected if insula ac=va=on is stronger. 31/3/

11 Causal Studies 31/3/

12 Oxytocin, Trust and Trustworthiness (Kosfeld et al., Nature Vol. 435, 2005) Step 1: Based on evidence from human and animal studies authors made an informed guess about how certain hormones may affect specific social behaviors in humans. Oxytocin is a hormone, which induces labor in human and nonhuman mammals, during lacta=on of young animals and during ma=ng. Step 2: Conduct a placebo- controlled hormone study that isolates the specific impact. This provides causal informa=on about the impact of the hormone Oxytocin is conjectured to play a key role in different social behaviors 31/3/

13 Investor Trustee Game 31/3/

14 Investor Trustee Game Both subjects receive an ini=al endowment of 12 monetary units (MU). The investor can send 0, 4, 8 or 12 MU to the trustee. The experimenter triples each MU the investor transfers. Ajer the investor's decision is made, the trustee is informed about the investor's transfer. Then the trustee has the op=on of sending any amount between zero and his total amount available back to the investor. 31/3/

15 Investor Trustee Game E.g., if investor has sent 12 MU, the trustee possesses 48 MU (12 MU own endowment + 36 MU tripled transfer) and can, therefore choose any back transfer from 0 to 48 MUs. The experimenter does not triple the back transfer. Investor's final payoff = ini=al endowment transfer to trustee + back transfer from the trustee. Trustee's final payoff = ini=al endowment + tripled transfer of the investor back transfer to the investor. Each subject made 4 decisions in the same player role while paired with 4 different, randomly selected interac=on partners. 31/3/

16 Does Oxytocin Affect Reciprocity/ Inequality aversion? Answer: NO 31/3/

17 Does Oxytocin Affect Investor Behaviour? Answer: YES 31/3/

18 Is Risk Aversion the Cause? Risk experiment: Replace trustee by a random choice. NO. Oxytocin seems to increase trust of first mover in the game. 31/3/

19 Structural Model of Simple Choice 31/3/

20 Evidence of Single Neural Currency (Paddoa- Schioppa et al., Nature Vol. 441, 2006) A monkey is offered the choice between different amounts and types of juice For example 3 ml of water or 1 ml of Kool Aid Record choices from different pairs Record ac=vity from neurons in orbitofrontal cortex (OFC) 31/3/

21 Calculate Behavioural Tradeoff Between Different Types of Juice 31/3/

22 Some Results 31/3/

23 Some Results OFC neurons record value of chosen op=on on a single scale Regression of ac=vity on u=lity of chosen object gives r- squared of 0.86 Berer fit than just amount of juice. Other studies show this area (striatum) values other items Choosing between gambles [Tom et al 2007] Current vs delayed monetary rewards [Kable and Glimcher, 2007] Food items [Plassman et al. 2007; Hare et al. 2009] Ac=vity in this area can (weakly) predict choice between consumer goods (Levy et al. 2011) 31/3/

24 Methods: What do we Learn about Human/Animal Choice? 31/3/

25 More Methods Brain s=mula=on (e.g., electrical s=mula=on of the amygdala elicits violence and aggressivity; at special loci, electrical brain s=mula=on is highly reinforcing) Single neuron measurement (e.g., track high frequency dopamine release in animal models, Schultz et al, Glimcher) Transcranial magne=c s=mula=on (TMS) EEG, PET, fmri (taking pictures of the ac=ve brain; e.g., McCabe, Houser, Ryan, Smith, and Trouard 2001). 31/3/

26 Interes=ng Findings Brain activity in the frontoparietal system and mesolimbic dopamine reward system predict behavior 31/3/

27 Variety of Neuroimaging Methods Electroencephalography (EEG) Magnetoencephalography (MEG) Transcranial Magne=c S=mula=on (TMS) Positron Emission Tomography (PET) Structural Magne=c Resonance Imaging (MRI) Func=onal Magne=c Resonance Imaging (fmri) 31/3/

28 31/3/

29 Cri=que: Why Neuro? Inspira=on Lirle new theory, except maybe op=mal coding Breaking up the problem e.g., rela=on between oxytocin and trust game Ruling out all mechanisms that could generate a mapping Hard, few examples include eye tracking and backwards induc=on Robustness/Out of Sample Predic=ons Issues such as selec=on of priors makes this hard too Crea=ng a Brain Map We are s=ll far from being able to establish any of this conclusively 31/3/

30 Cri=que, More General Thagard & Milgram (1995) An eminent philosopher of science once encountered a noted decision theorist in a hallway at their university. The decision theorist was pacing up and down, murering, What shall I do? What shall I do? What s the marer, Howard? asked the philosopher. Replied the decision theorist, It's horrible, Ernest I ve got an offer from Harvard and I don t know whether to accept it. Why, Howard, reacted the philosopher, you re one of the world s great experts on decision- making. Why don t you just work out the decision tree, calculate the probabili=es and expected outcomes, and determine which choice maximizes your expected u=lity? With annoyance, Howard replied, Come on, Ernest. This is serious. 31/3/

31 Acknowledgements Many of these slides are adapted from those by Mark Dean at Princeton 31/3/