Bayesian(Updating( Goals( 8/3/12& Relation(With(Reinforcement(Learning( To(highlight(core(characteristics(of(Bayesian(updating:(

Transcription

1 Bayesian(Updating( Peter(Bossaerts,(Caltech( Goals( Relation(With(Reinforcement(Learning( To(highlight(core(characteristics(of(Bayesian(updating:( 1. Optimal(Integration(of(Prior(belief(and(Evidence((via( Likelihood)( 2. Optimality:(Martingales( 3. ModelLBased(Learning(Approach( 4. Integration(of(Hypotheses(( Marginalization )( 5. Polyvalent(Uncertainty( Humans(Are(Not(Bayesians?( Monty(Hall( 2" 1&

2 Reinforcement(Learning( Most(of(the(examples(in(psychology/neuroscience(are(about(formation(of( beliefs(about(events/stimuli(that(have(a((fixed)(affective(value((reward/loss).( In(such(a(context,(psychologists/neuroscientists(usually(talk(about( reinforcement(learning.( One(distinguishes(two(types((Daw,(Niv,(Dayan(2005)( ModelLfree:( Pure(Pavlovian:(TD(learning((see(before)( Instrumental:(Q(learning((WatkinsLDayan(1992)( ModelLbased( Example:(Bayesian(learning( I(will(casually(talk(about(modelLfree(learning(as( reinforcement(learning ((RL)( while(identifying(modellbased(learning(as( Bayesian. (( 3" 1.(Integration(of(Prior(belief(and(Evidence( (via(likelihood)( Posterior(=(Prior(*(Likelihood( (Compare(to(Prediction(Error(based(learning:(New(Belief(=( Old(Belief(+(Learning(Rate*Prediction(Error)( 4" 2&

3 /3/12& Sensorimotor(Learning(Example( (Körding/Wolpert,(Nature(2004)( Prior(unobserved(lateral(shift( Noisy(observation( 5" Results( letters to nature Figure 2 Results for a gaussian distribution. Colour codes as in Fig. 1. a, The lateral deviation of the cursor at the end of the trial as a function of the imposed lateral shift for a typical subject. Error bars denote s.e.m. The horizontal dotted lines indicate the prediction from the full compensation model and the dashed line is the fit for a model that ignores sensory feedback on the current trial and corrects only for the mean over all trials. The solid line is the bayesian model with the level of uncertainty fitted to the data. b, The slopes for the linear fits are shown for the full population of subjects. On the basis of the visual estimate of the lateral shift. In this model, increasing the uncertainty of the feedback for a particular lateral shift (by increasing the blur) would affect the variability of the pointing but not the average location. Crucially, this model does not require subjects to 6" hypothesis that the slope should increase with increasing visual uncertainty, we performed a repeated-measures analysis of variance on the slope, with visual uncertainty as a factor (main effect of visual uncertainty F 3,27 ¼ 82.7; p, 0.001). Planned comparisons of the slopes between adjacent uncertainty levels were all significant (asterisk, p, 0.05; three asterisks, p, 0.001). c, The bias against gain for the linear fits for each subjects and condition. The solid line shows the bayesian solutions. d, The inferred priors and the true prior (red) for each subject and condition. By examining the influence of the visual feedback on the final deviation from the target we can distinguish between these three models (Fig. 1e). If subjects compensate fully for the visual feedback (model 1), the average lateral deviation of the cursor from the target 3&

4 Results((c d)( (Note:(Posterior(MEAN(only;(recent(evidence:(tradeLoff(posterior(meanLvariance)( 7" Drawing(from(an(Urn:(Conservatism( b Bet(whether(right(urn(was(selected ( a b Observed Observed Bayesian Robust Bayesian Bayesian Update Observed Bayesian Robsut Bayesian See Orange Ball See Green Ball (D Acremont(ea,(under(review)( 8" 4&

5 2.(Bayesian(Beliefs(Form(A(Martingale( What(is(a(martingale?(E[X(t+1)( (Past(Data](=(X(t).( One(cannot(predict(direction+magnitude(of(changes(in(X. ( (Still(possible:(predict(E[(X(t+1)LX(t))^2( (Past(Data]!)( Fundamental(concept(in(stochastic(process(theory((and( mathematical(finance)( 9" Doob s(lemma( Bayesian(beliefs(form(a(martingale.( That(is:(E[Posterior(outcome)( (Past(Data](=(Prior(outcome).( Intuition:(If(this(were(violated,(one(could(predict(changes(in(one s(own( beliefs,(which(means(that(one s(own(beliefs(have(not(been(updated( enough. ( This(is(the(essence(of( rational(learning. ( Remarks:( Martingale(Convergence(Theorem:(Bayesian(beliefs(are(expected(to( converge.( When(beliefs(are(a(martingale,(updates( maximize(surprise, (and(hence( beliefs(incorporate(as(much(information(as(possible( (information(theory.( 10" 5&

6 Why(are(Bayesian(beliefs(a(martingale?( Because(Bayesians(update(based(on(the(likelihood((ratio):( likelihood(of(observed(data(( stimulus/signal )(given(one( hypothesis(compared(to(likelihood(of(observed(data(given( alternatives.( (Contrast(this(with(standard(predictionLerror(based(learning( schemes(like(rescorlalwagner,(which(are(based(on:( PE(=(Outcome(L(Prediction( 11" Still,(predictionLerror(learning(models(can( be(made(to( emulate (Bayesian(learning( Nicest(example((I(think):(Sutton(1992.(( He(sets(the(learning(rate(( gain )(such(that(one(expects(to(minimize(the(size(of( the(subsequent(prediction(errors.( Sutton(proves(that(this(is(the(same(as(to(minimize(the(correlation((over(time)(of( the(prediction(error.( If(prediction(errors(are(positively(correlated,(one s(learning(rate(is(too(low;( If(negatively(correlated,(the(learning(rate(is(TOO(HIGH.( If(predictions(form(a(martingale,(changes(in(predictions(are(uncorrelated( So,(Sutton(attempts(to(generate(a(martingale ( (Sutton s(algorithm(works(very(well!!)( 12" 6&

7 Back(to(Urn(Betting ( Martingale(test(accepted ( a b Update Sample Size Covariance Sample Size 13" (despite(conservatism( (because(participants(used(a( robust(prior,(not(the( true ((announced)(prior,( unlike(in(kördinglwolpert.( (Robust(prior:(mixturesLofL binomials)( Density High range Low range Expected prior More conservatism Less conservatism Probability 14" 7&

8 Remarks( Truth(is(more(complicated:(Bayesian(beliefs(are(a(martingale( only(from(the(perspective(of(the(learner.( Specifically,(they(may(not(be(a(martingale(from(the( perspective(of(an(observer(who(knows(more((e.g.,(which(urn( is(more(likely(to(be(correct?)( Doob s(result(can(be(extended((bossaerts,(restud02004)...( 15" Neurobiological(basis?( YangLShadlen((Nature,( 2007):(recordings(in(monkey( parietal(cortex(shows( updating(based(on(likelihood( ratio( a 1,500 ms: 4th shape on 1,000 ms: 3rd shape on 2,400 2,600 ms: Fixation off, saccade 2,000 ms: shapes off (In(their(task,(information(is( not(i.i.d.(conditional(on( correct(target(location.)( 500 ms: 2nd shape on 0 ms: Target on 1st shape on Fixation Time Favouring red Shapes Assigned weights 0.9 Favouring green 16" 8&

9 Results ( b 80 Epoch 1 Epoch 2 Epoch 3 Epoch 4 a Response (sp s 1 ) Targets and 1st shape on 2nd shape on 3rd shape on 4th shape on All shape off T in T out Response (sp s 1 ) c Time (ms) + loglr for T in 0 0 1,000 2,000 3,000 Time (ms) Response (sp s 1 ) ± ± ± ± loglr (ban) 17" 3.(Bayesian(Learning(Is(ModelLBased( Bayesian(learning(is(about( inverting( beliefs ((Laplace)(to(assess(the(veracity(of( underlying( causes (( This(requires(a(model0(of0the0hidden0causes);0S(t)( (medication)(and(y(t)((symptoms)(are(not(just( correlated,(but(s(t)(causes(x(t)((infection)(which( causes(y(t).( This(contrasts(with(Reinforcement0Learning(which( only(involves(observables(((certain(s(t)( (medication)(help(y(t)((symptoms),(but(the(rl( agent(does(not(care(to(probe(why?0 St Xt Yt (But(modelLbased(learning(does(not(need( Bayesian(updating )( 18" 9&

10 Neurobiological(Foundation?( Reversal(Task:(Does(the((human)(brain(record(that(when(one( option(goes(bad,(the(other(must0be0better?0(hampton(ea,(jn0 2006;(threeLoption(case:(Beierholm(ea,(NeuroImage02011)0 19" More(Challenging (see(correlation(study( in(class(3( Underlying(correlation(changes( Do(humans(learn(by(trial(and(error((reinforcement)(or(by(explicitly(tracking( correlation((bayesian)?( (Wunderlich(ea,(Neuron02011)( 20" 10&

11 Choices ( Subject Model Complete Info 21" ( Brain(Activation ( ( A R Correlation( z = 7 Correlation(Prediction( Error( 22" 11&

12 4.(Bayesians(Follow(Evidence(For(ALL( Hypotheses( (as(opposed(to( attention(gating ((hypothesis(testing):(pick( one(hypothesis(and(accept(it(until(evidence(gathers(against(it.( Bayesians( marginalize (across(hypotheses.( 23" The(Task.( Two(modalities(( dimensions )(may( cause (reward;(choose( Top(or(Bottom((Wunderlich(ea,(J0Neurophys02011)( 24" 12&

13 Analysis:(Weight(on(each(dimension( Subject(could(choose(based(on(motion(even(if(she(is(more(confident(that( color(is(right(because(confidence(in(choice(condition(on(motion(is(higher ( A COLOR green red DIMENSION color motion B MOTION CERTAINTY DIM EXEMPLAR right left trial " trial C P (actual choices) Model predicted value for top stimulus choice of top stimulus Activation ( choice of bottom stimulus To(be(able(to(weigh(appropriately(the(evidence(for(the(two( dimensions(in(final(choice,(you(need(a(signal(of(confidence( (left)(or(uncertainty((right)(for(the(two(dimensions((summed( here)( A B x = 2 x = 0 z = 35 z = 10 26" 13&

14 5.(Polyvalent(Uncertainty( In(Reinforcement(Learning,( there(is(only(uncertainty( about(the(relation(between( S(t)(and(Y(t).( For(Bayesians,(there(is( uncertainty(about(x(t),(about( Y(t)(given(X(t),(and(even( about(whether(the(relation( (S(t),(X(t))(changes (( St Yt St Xt Yt 27" Uncertainty (( Irreducible"uncertainty"or"risk:(Decision(Maker((DM)(knows(that( the(chance(of(heads(on(a(fair(coin(is(0.5;(dm(doesn t(know(whether( the(next(toss(will(be(heads(or(tails.((concerns0the0relation0between0 X(t)0and0Y(t))( Estimation"uncertainty"or"ambiguity:(DM(is(given(a(new(coin(and( doesn t(know(whether(it(is(fair;(dm(needs(to(learn(the(probability(of( heads.((concerns0how0sure0one0is0of0x(t))( Unexpected"uncertainty"or"jump"risk"(or" volatility ):(Unknown( to(dm,(the(coin(is(replaced(with(another((possibly(unfair)(coin.( (Concerns0whether0X(t)0has0changed)( Model"or" Knightean "uncertainty:(is(the(coin(being(replaced( regularly(or(are(coin(tosses(correlated?((concerns0the0nature0of0x(t))" 28" 14&

15 Remarks( By(suitably(changing(the(learning(rate,(even(the(RL(agent(can( behave(as0if(she(cares(about(the(separate(underlying(sources(of( uncertainty( E.g.:( When(the(environment(becomes(inherently(less(predictable,(then( learning(rate(should(be(lower( When(the(environment(becomes(more(unstable(( volatile ),(then( the(learning(rate(should(be(higher( The(distinguishing(features(really(are:( Is(the(agent(behaviorally(sensitive(to(separate(sources(of( uncertainty((e.g.,(ambiguity(averse)?( Does(the(brain(form(explicit(representations(of(the(separate(source( of(uncertainty?( 29" Take(unexpected(uncertainty(or( volatility ( Jumps, (e.g.,(binary(gamble:(reward(probability(reverts(with( probability(v( V(could(be(called( volatility ((don t(be(confused( (it(means( something(else(in(finance)( (Behrens(ea.(2007( (Intuitive:( As(v(increases,(INCREASE(learning(rate((older(data(become( obsolete)( As(v(decreases,(DECREASE(learning(rate( Learning(rate:(effect(of(last(prediction(error(on(new( prediction( 30( 15&

16 Reversal(Learning(Task(( With(Changing(Reversal(Rate( Estimated(volatility(tracks( reversal(rate( ( Learning(rates(track( volatility((optimally)( (Behrens(ea(2007)( 31( Brain(Activation( Volatility(correlates(with(ACC( activation(in( monitoring (period( (after(outcome(is(revealed(and( before(subsequent(decision( period)( (Could(also(be(learning(rate,( (consistent(with(animal(studies( where(lesions(to(acc(lead(to( impairment(in(adjusting( memory (of(learning)( ACC(activates(also(as(a(function( of(total(uncertainty(( variance( of(reward)(which(combines( volatility (and( irreducible( uncertainty ((which(was( DIFFERENT(across(stable(and( volatile(periods)( 32( 16&

17 Role(of(Norepinephrine((NE)(and(Acetylcholine( (ACh)(In(Expected/Unexpected(Uncertainty( Uncertainty( about(cue( validity(is( irreducible0 Uncertainty( about(the( right(cue(can( be(reduced( over(time( ( estimation( uncertainty( (YuLDayan:( unexpected( uncertainty)( 33( Evolution(Over(Time( ( In(YuLDayan(algorithm,(estimation(uncertainty(stays(high(for(10(trials(after( perceived(context(switch(rather(than(gradually(decreasing;( gamma=prob(cue(is(correct),(so(irreducible(uncertainty=gamma*(1l gamma)( 34( 17&

18 Relation(with(pupil(dilation( (which(is(thought(to(correlate(with(ne(fluctuations( See(Preuschoff(ea(pupil(dilation(study(in(earlier(class( (Unexpected(uncertainty(=(risk(prediction(error)( See(also(Nassar(ea,(Nature(Neuroscience(June(2012.( 35( Final(Remark:(Humans(Are(Not( Bayesian!?( Monty(Hall ( Most(people( cannot(solve(this( problem(correctly( 36" 18&