How to Combine Expert (or Novice) Advice when Actions Impact the Environment?

Transcription

1 How to Combin Exprt (or Novic) Advic whn Actions Impact th Environmnt? Danila Pucci d Farias Dpartmnt of Mchanical Enginring Massachustts Institut of Tchnology Cambridg, MA pucci@mit.du Nimrod Mgiddo IBM Almadn Rsarch Cntr 650 Harry Road, K53-B2 San Jos, CA mgiddo@almadn.ibm.com Abstract Th so-calld xprts algorithms constitut a mthodology for choosing actions rpatdly, whn th rwards dpnd both on th choic of action and on th unknown currnt stat of th nvironmnt. An xprts algorithm has accss to a st of stratgis ( xprts ), ach of which may rcommnd which action to choos. Th algorithm larns how to combin th rcommndations of individual xprts so that, in th long run, for any fixd squnc of stats of th nvironmnt, it dos as wll as th bst xprt would hav don rlativ to th sam squnc. This mthodology may not b suitabl for situations whr th volution of stats of th nvironmnt dpnds on past chosn actions, as is usually th cas, for xampl, in a rpatd non-zro-sum gam. A nw xprts algorithm is prsntd and analyzd in th contxt of rpatd gams. It is shown that asymptotically, undr crtain conditions, it prforms as wll as th bst availabl xprt. This algorithm is quit diffrnt from prviously proposd xprts algorithms. It rprsnts a shift from th paradigms of rgrt minimization and myopic optimization to considration of th long-trm ffct of a playr s actions on th opponnt s actions or th nvironmnt. Th importanc of this shift is dmonstratd by th fact that this algorithm is capabl of inducing coopration in th rpatd Prisonr s Dilmma gam, whras prvious xprts algorithms convrg to th suboptimal non-cooprativ play. 1 Introduction Exprts algorithms. A wll-known class of mthods in machin larning ar th socalld xprts algorithms. Th goal of ths mthods is to larn from xprinc how to combin advic from multipl xprts in ordr to mak squntial dcisions in an onlin nvironmnt. Th gnral ida can b dscribd as follows. An agnt has to choos rpatdly from a givn st of actions. Th rward in ach stag is a function of th chosn action and th choics of Natur or th nvironmnt (also rfrrd to as th advrsary or th opponnt ). A st of stratgis {1,..., r} is availabl for th agnt to choos from. W rfr Work don whil at IBM Almadn Rsarch Cntr, San Jos, California.

2 to ach such stratgy as an xprt, vn though som of thm might b simpl nough to b calld a novic. Each xprt suggsts a choic of an action basd on th history of th procss and th xprt s own choic algorithm. Aftr ach stag, th agnt obsrvs his own rward. An xprts algorithm dircts th agnt with rgard to which xprt to follow in th nxt stag, basd on th past history of actions and rwards. Minimum Rgrt. A popular critrion in dcision procsss is calld Minimum Rgrt (MR). Rgrt is dfind as th diffrnc btwn th rward that could hav bn achivd, givn th choics of Natur, and what was actually achivd. An xprt slction rul is said to minimiz rgrt if it yilds an avrag rward as larg as that of any singl xprt, against any fixd squnc of actions chosn by th opponnt. Indd, crtain xprts algorithms, which at ach stag choos an xprt from a probability distribution that is rlatd to th rward accumulatd by th xprt prior to that stag, hav bn shown to minimiz rgrt [1, 2]. It is crucial to not though that, sinc th xprts ar compard on a squnc-bysqunc basis, th MR critrion ignors th possibility that diffrnt xprts may induc diffrnt squncs of choics by th opponnt. Thus, MR maks sns only undr th assumption that Natur s choics ar indpndnt of th dcision makr s choics. Rpatd gams. W considr a multi-agnt intraction in th form of a rpatd gam. In rpatd gams, th assumption that th opponnt s choics ar indpndnt of th agnt s choics is not justifid, bcaus th opponnt is likly to bas his choics of actions on th past history of th gam. This is vidnt in nonzro-sum gams, whr playrs ar facd with issus such as how to coordinat actions, stablish trust or induc coopration. Ths goals rquir that thy tak ach othr s past actions into account whn making dcisions. But vn in th cas of zro-sum gams, th possibility that an opponnt has boundd rationality may lad a playr to look for pattrns to b xploitd in th opponnt s past actions. W illustrat som of aformntiond issus with an xampl involving th Prisonr s Dilmma gam. Th Prisonr s Dilmma. In th singl-stag Prisonr s Dilmma (PD) gam, ach playr can ithr cooprat (C) or dfct (D). Dfcting is bttr than cooprating rgardlss of what th opponnt dos, but it is bttr for both playrs if both cooprat than if both dfct. Considr th rpatd PD. Suppos th row playr consults with a st of xprts, including th dfcting xprt, who rcommnds dfction all th tim. Lt th stratgy of th column playr in th rpatd gam b fixd. In particular, th column playr may b vry patint and cooprativ, willing to wait for th row playr to bcom cooprativ, but vntually bcoming non-cooprativ if th row playr dos not sm to cooprat. Sinc dfction is a dominant stratgy in th stag gam, th dfcting xprt achivs in ach stp a rward as high as any othr xprt against any squnc of choics of th column playr, so th row playr larns with th xprts algorithm to dfct all th tim. Obviously, in rtrospct, this sms to minimiz rgrt, sinc for any fixd squnc of actions by th column playr, constant dfction is th bst rspons. Obviously, constant dfction is not th bst rspons in th rpatd gam against many possibl stratgis of th column playr. For instanc, th row playr would rgrt vry much using th xprts algorithm if h wr told latr that th column playr had bn playing a stratgy such as Tit-for-Tat. 1 In this papr, w propos and analyz a nw xprts algorithm, which follows xprts judiciously, attmpting to maximiz th long-trm avrag rward. Our algorithm diffrs from prvious approachs in at last two ways. First, ach tim an xprt is slctd, it is followd for multipl stags of th gam rathr than a singl on. Scond, our algorithm taks 1 Th Tit-for-Tat stratgy is to play C in th first stag, and latr play in vry stag whatvr th opponnt playd in th prcding stag.

3 into account only th rwards that wr actually achivd by an xprt in th stags it was followd, rathr than th rward that could hav bn obtaind in any stag. Our algorithm njoys th appaling simplicity of th prvious algorithms, yt it lads to a qualitativly diffrnt bhavior and improvd avrag rward. W prsnt two rsults: 1. A worst-cas guarant that, in any play of th gam, our algorithm achivs an avrag rward that is asymptotically as larg as that of th xprt that did bst in th rounds of th gam whn it was playd. Th worst-cas guarant holds without any assumptions on th opponnt s or xprts stratgis. 2. Undr crtain conditions, our algorithm achivs an avrag rward that is asymptotically as larg as th avrag rward that could hav bn achivd by th bst xprt, had it bn followd xclusivly. Th conditions ar rquird in ordr to facilitat larning and for th notion of a bst xprt to b wll-dfind. Th ffctivnss of th algorithm is dmonstratd by its prformanc in th rpatd PD gam, namly, it is capabl of idntifying th opponnt s willingnss to cooprat and it inducs cooprativ bhavior. Th papr is organizd as follows. Th algorithm is dscribd in sction 2. A bound basd on actual xprt prformanc is prsntd in sction 3. In sction 4, w introduc and discuss an assumption about th opponnt. This assumption givs ris to asymptotic optimality, which is prsntd in sction 5. 2 Th algorithm W considr an xprts stratgy for th row playr in a rpatd two-prson gam in normal form. At ach stag of th gam, th row and column playr choos actions i I and j J, rspctivly. Th row playr has a rward matrix R, with ntris 0 R ij u. Th row playr may consult at ach stag with a st of xprts {1,..., r}, bfor choosing an action for th nxt stag. W dnot by σ th stratgy proposd by xprt, i.., σ = σ (h s ) is th proposd probability distribution ovr actions in stag s, givn th history h s. W rfr to th row playr as th agnt and to th column playr as th opponnt. Usually, th form of xprts algorithms found in th litratur is as follows. Dnot by M (s 1) th avrag rward achivd by xprt prior to stag s of th gam 2. Thn, a rasonabl rul is to follow xprt in stag s with a probability that is proportional to som monoton function of M (s 1). In particular, whn this probability is proportional to xp{η s M (s 1)}, for a crtain choic of η s, this algorithm is known to minimiz rgrt [1, 2]. Spcifically, by ltting j s (s = 1, 2,...) dnot th obsrvd actions of th opponnt up to stag s, and ltting σ X dnot th stratgy inducd by th xprts algorithm, w hav s s =1 E[R(i, j s ) : i σ X (h s )] sup 1 s s E[R(i, j s ) : i σ (h s )] o(s). (1) Th main dficincy of th rgrt minimization approach is that it fails to considr th influnc of chosn actions of a playr on th futur choics of th opponnt th inquality (1) holds for any fixd squnc (j s ) of th opponnt s movs, but dos not account for th fact that diffrnt choics of actions by th agnt may induc diffrnt squncs of th opponnt. This subtlty is also missing in th xprts algorithm w dscribd abov. At ach 2 In diffrnt variants of th algorithm and dpnding on what information is availabl to th row playr, M (s 1) could b ithr an stimat of th avrag rward basd on rward achivd by xprt in th stags it was playd, or th rward it could hav obtaind, had it bn playd in all stags against th sam history of play of th opponnt. s =1

4 stag of th gam, th slction of xprt is basd solly on how wll various xprts hav, or could hav, don so far. Thr is no notion of larning how an xprt s actions affct th opponnt s movs. For instanc, in th rpatd PD gam dscribd in th introduction, assuming that th opponnt is playing Tit-for-Tat, th algorithm is unabl to stablish th rlationship btwn th opponnt s cooprativ movs and his own. Basd on th prvious obsrvations, w propos a nw xprts algorithm, which taks into account how th opponnt racts to ach of th xprts. Th ida is simpl: instad of choosing a (potntially diffrnt) xprt at ach stag of th gam, th numbr of stags an xprt is followd, ach tim it is slctd, incrass gradually. W rfr to ach such st of stags as an itration of th algorithm. Following is th statmnt of th Stratgic Exprts Algorithm (SEA). Th itration numbr is dnotd by i. Th numbr of itrations during which xprt has bn followd is dnotd by N. Th avrag payoff from itrations in which xprt has bn followd is dnotd by M. Stratgic Exprts Algorithm (SEA): 1. For = 1,..., r, st M = N = 0. St i = With probability 1/i prform an xploration itration, namly, choos an xprt from th uniform distribution ovr {1,..., r}; othrwis, prform an xploitation itration, namly, choos an xprt from th uniform distribution ovr th st of xprts with maximum M. 3. St N = N + 1. Follow xprt s instructions for th nxt N stags. Dnot by R th avrag payoff accumulatd during th currnt itration (i.., ths N stags), and st M = M + 2 N +1 ( R M ). 4. St i = i + 1 and go to stp 2. Throughout th papr, s will dnot a stag numbr, and i will dnot an itration numbr. W dnot by M 1 (i),..., M r (i) th valus of th rgistrs M 1,..., M r, rspctivly, at th nd of itration i. Similarly, w dnot by N 1 (i),..., N r (i) th valus of th rgistrs N 1,..., N r, rspctivly, at th nd of itration i. Thus, M (i) and N (i) ar, rspctivly, th avrag payoff accumulatd by xprt and th total numbr of itrations this xprt was followd on or bfor itration i. W will also lt M(s) and M(i) dnot, without confusion, th avrag payoff accumulatd by th algorithm in th first s stags or first i itrations of th gam. 3 A bound basd on actual xprt prformanc Whn th SEA is mployd, th avrag rward M (i) that was actually achivd by ach availabl xprt is bing trackd. It is thrfor intrsting to compar th avrag rward M(s) achivd by th SEA, with th avrags achivd by th various xprts. Th following thorm stats that, in th long run, th SEA obtains almost surly at last as much as th actual avrag rward obtaind by any availabl xprt during th sam play. Thorm 3.1. ( Pr lim inf s M(s) max ) lim inf (i) i = 1. (2) Although th claim of Thorm 3.1 sms vry clos to rgrt minimization, thr is an ssntial diffrnc in that w compar th avrag rward of our algorithm with th avrag rward actually achivd by ach xprt in th stags whn it was playd, as opposd to th stimatd avrag rward basd on th whol history of play of th opponnt.

5 Not that th bound (2) is mrly a statmnt about th avrag rward of th SEA in comparison to th avrag rward achivd by ach xprt, but nothing is claimd about th limits thmslvs. Thorm 5.1 proposs an application of this bound in a cas whn an additional assumption about th xprts and opponnt s stratgis allows us to analyz convrgnc of th avrag rward for ach xprt. Anothr intrsting cas occurs whn on of th xprts plays a maximin stratgy; in this cas, bound (2) nsurs that th SEA achivs at last th maximin valu of th gam. Th sam holds if on of th xprts is a rgrt-minimizing xprts algorithm, which is known to achiv at last th maximin valu of th gam. Th rmaindr of this sction consists of a sktch of th proof of Thorm 3.1. Sktch of proof: Dnot by V b th random variabl max lim inf i M (i), and dnot by Ē th xprt that achivs that maximum (if thr is mor than on, lt Ē b th on with th last indx). For any logical proposition L, lt δ(l) = 1 if L is tru; othrwis δ(l) = 0. Th proof of Thorm 3.1 rlis on stablishing that, for all ɛ > 0 and any xprt, ( ) N (i) δ(m (i) V ɛ) Pr lim = 0 = 1. (3) i i Thr ar thr possibl situations for any xprt : (a) Whn lim inf i M (i) > V ɛ, th inquality is satisfid trivially. (b) Whn lim sup i M (i) < V, thr is an itration I such that for all i I, M (i) < M Ē (i), so that xprt is playd only on xploration itrations, and a larg dviations argumnt stablishs that (3) holds. (c) Th most involvd situation occurs whn lim inf i M (i) V ɛ and lim sup i M (i) V. To show that (3) holds in this cas, w ar going to focus on th trajctory of M (i) ach tim it gos from abov V ɛ/2 to blow V ɛ + δ/2, for som 0 < δ < ɛ. W offr th two following obsrvations: 1. Lt I k b th k th itration such that M (i) V ɛ + δ/2, and lt Ik 0 b th first itration bfor I k such that M (i) V ɛ/2. Thn, btwn itrations Ik 0 and I k, xprt is slctd at last N (Ik 0 )(ɛ δ)/(6u) tims. Dnoting by I j k, j = 1,..., P k, th itrations whn xprt is slctd, btwn Ik 0 and I k, w hav M (I j k ) M (I j 1 k )(N (I 0 k ) + j 1)(N (I 0 k ) + j) (N (I 0 k ) + j)(n (I 0 k ) + j + 1). A simpl induction argumnt shows that, in ordr to hav M (I k ) V ɛ 2 M (I 0 k) ɛ δ 2 xprt must b slctd a numbr of tims P k N (Ik 0 )(ɛ δ)/(6u). 2. For all larg nough k, th itrations I j k whn xprt is slctd ar xclusivly xploration itrations. This follows trivially from th fact that, aftr a crtain itration I, w hav M Ē (i) V ɛ/2, for all i I, whras M (i) < V ɛ/2 for all i btwn Ik 0 and I k., From th first obsrvation, w hav N (I k ) I k N (I 0 k ) + P k I k I 0 k (1 + 6u)P k (ɛ δ)i k Ik 0,

6 Sinc xprt is slctd only during xploration itrations btwn Ik 0 and I k, a larg dviations argumnt allows us to conclud that th ratio of th numbr of tims P k xprt is slctd, to th total numbr of itrations I k Ik 0, convrgs to zro with probability on. W conclud that (3) holds. W now obsrv that M(i) = N (i)(n (i) + 1)M (i) N. (4) (i)(n (i) + 1) By a simpl optimization argumnt, w can show that N (i)(n (i) + 1) i(i/r + 1). (5) Using (3) and (5) to bound (4), w conclud that (2) holds for th subsqunc of stags s corrsponding to th nd of ach itration of th SEA. It is asy to show that th avrag rward M(s) in stags s in th middl of itration i bcoms arbitrarily clos to th avrag rward at th nd of that itration M(i), as i gos to infinity, and th thorm follows. 4 Th flxibl opponnt In gnral, it is impossibl for an xprts algorithm to guarant, against an unknown opponnt, a rward clos to that of th bst availabl xprt. It is asy to construct xampls which prov this impossibility. Exampl: Rpatd Matching Pnnis. In th Matching Pnnis (MP) gam, ach of th playr and th advrsary has to choos ithr H ( Hads ) or T ( Tails ). If th choics match, th playr loss 1; othrwis, h wins 1. A possibl stratgy for th advrsary in th rpatd MP gam is: Advrsary: Fix a positiv intgr s and a string σ s {H, T } s. In ach of th first s stags, play th 50:50 mixd stratgy. In ach of th stags s + 1, s + 2,..., if th squnc of choics of th playr during th first s stags coincidd with th string σ s, thn play T ; othrwis, play th 50:50 mixd stratgy. Suppos ach availabl xprt corrsponds to a stratgy of th form: Exprt: Fix a string σ {H, T } s. During th first s stags play according to σ. In ach of th stags s + 1, s + 2,..., play H. Suppos an xprt with σ = σ s is availabl. Thn, in ordr for an xprts algorithm to achiv at last th rward of, it nds to follow th string σ s prcisly during th first s stags. Of cours, without knowing what σ s is, th algorithm cannot play it with probability on, nor can it larn anything about it during th play. In viw of th rpatd MP xampl, som assumption about th opponnt must b mad in ordr for th playr to b abl to larn how to play to against that opponnt. Th ssnc of th difficulty with th abov stratgy of th opponnt is that it is not flxibl th playr has only on chanc to guss who th bst xprt is and thus cannot rcovr from a mistak. Hr, w introduc th assumption of flxibility as a possibl rmdy to that problm. Undr th assumption of flxibility, th SEA achivs an avrag rward that is asymptotically as high as what th bst xprt could b xpctd to achiv. Dfinition 4.1 (Flxibility). (i) An opponnt playing stratgy π(s) is said to b flxibl with rspct to xprt ( = 1,..., r) if thr xist constants µ, τ > 0.25 and c such that for vry stag s 0, vry possibl history h s0 at stag s 0 and any numbr of stags s, E [ 1 s s0+s s=s 0+1 R(a (s), b(s)) µ : a (s) σ (h s ), b(s) π(h s ) ] c s τ

7 (ii) Flxibility with rspct to a st of xprts is dfind as flxibility with rspct to vry mmbr of th st. In words, th xpctd avrag rward during th s stags btwn stag s 0 and stag s 0 +s convrgs (as s tnds to infinity) to a limit that dos not dpnd on th history of th play prior to stag s 0. Exampl 4.1 : Finit Automata. In th litratur on boundd rationality, playrs ar oftn modlld as finit automata. A probabilistic automaton stratgy (PAS) is spcifid by a tupl A = M, O, A, σ, P, whr M = {1,..., m} is th finit st of intrnal stats of th automaton, A is th st of possibl actions, O is th st of possibl outcoms, σ i (a) is th probability of choosing action a whil in stat i (i = 1,..., m) and P o = (Pij o ) (1 i, j m) is th matrix of stat transition probabilitis, givn an outcom o O. Thus, at any stag of th gam, th automaton picks an action from a probability distribution associatd with its currnt stat and transitions into a nw stat, according to a probability distribution which dpnds on th outcom of th stag gam. If both th opponnt and an xprt play PASs, thn a Markov chain is inducd ovr th st of pairs of th rspctiv intrnal stats. If this Markov chain has a singl class of rcurrnt stats, thn th flxibility assumption holds. Not that w do not limit th siz of th automata; a largr st of intrnal stats implis a slowr convrgnc of th avrag rwards, but dos not affct th asymptotic rsults for th SEA. Exampl 4.2 : Boundd dpndnc on th history. Th numbr of possibl historis at stag s grows xponntially with s. Thus, it is rasonabl to assum that th choic of action would b basd not on th xact dtail of th history but rathr on th mpirical distribution of past actions or pattrns of actions. If th opponnt is blivd not to b stationary, thn discounting prvious obsrvations by rcncy may b snsibl. For instanc, if th frquncy of play of action j by th opponnt is rlvant, th playr might condition his choic at stag s + 1 on th quantitis τ j = s s =1 δ βs s jjs whr β < 1 and δ is th Kronckr dlta. In this cas, only actions j s at stags s that ar rlativly rcnt hav a significant impact on τ j. Thrfor stratgis basd on τ j should xhibit bhavior similar to that of boundd rcall, and lad to flxibility in th sam circumstancs as th lattr. 5 A bound basd on xpctd xprt prformanc In this sction w show that if th opponnt is flxibl with rspct to th availabl xprts, thn th SEA achivs almost surly an avrag payoff that is asymptotically as larg as what th bst xprt could achiv against th sam opponnt. W first prov a lmma that shows that, undr th flxibility assumption, th avrag rward achivd by ach xprt is asymptotically almost surly th sam as th rward that would hav bn achivd by th sam xprt, had h bn th only availabl xprt. Lmma 5.1. If th opponnt is flxibl with rspct to xprt, thn with probability on, lim i M (i) = µ. Sktch of proof: Lt b any xprt. By th Borl-Cantlli lmma, xploration occurs infinitly many tims, hnc is followd during infinitly many itrations. Lt I j = I j (), (j = 1, 2,...) b th itration numbrs in which is followd. By Markov s inquality, for vry ɛ > 0, Pr( M (I j ) µ > ɛ) ɛ 4 E[(M (I j ) µ ) 4 ].

8 If w could show that E[(M (I j ) µ ) 4 ] <, (6) j=1 thn w could conclud, by th Borl-Cantlli lmma, that with probability on, th inquality M (I j ) µ > ɛ holds only for finitly many valus of j. This implis that, with probability on, lim i M (i) = µ. It follows that if th opponnt is flxibl with rspct to xprt, thn for som ν > 0, as j tnds to infinity, E[(M (I j ) µ ) 4 ] = O(j 1 ν ), which suffics for (6). Thorm 5.1. If an opponnt π is flxibl with rspct to th xprts 1,..., r, thn th avrag payoff up to stag s, M(s), satisfis ( ) Pr lim inf M(s) max µ = 1. s Thorm 5.1 follows from Lmma 5.1 and Thorm 3.1. According to th lattr, th SEA achivs, with probability on, an avrag rward as larg as th avrag rward achivd by th bst xprt during th sam play. Flxibility coms into play as a way of nsuring that th valu of following any givn xprt is wll-dfind, and can vntually b stimatd as long as th SEA follows that xprt a sufficintly many tims. In othr words, flxibility nsurs that thr is a bst xprt to b larnd, and that larning can ffctivly occur bcaus actions takn by othr xprts, which could affct th bhavior of th opponnt, ar vntually forgottn by th lattr. Exampl 5.1 : Rpatd Prisonr s Dilmma rvisitd. Considr playing th rpatd PD gam against an opponnt who plays Tit-for-Tat, and suppos thr ar only two xprts: Always dfct (AD) and Always cooprat (AC). Thus, AC inducs coopration in vry stag and yilds a payoff highr than AD, which inducs dfction in vry stag of th gam xcpt th first on. It is asy to vrify that Tit-for-Tat is flxibl with rspct to th xprts AC and AD. Thrfor, Thorm 5.1 holds and th SEA achivs an avrag payoff at last as much as that of AC. By contrast, as mntiond in th introduction, in ordr to minimiz rgrt, th standard xprts algorithm must play D in almost vry stag of th gam, and thrfor achivs a lowr payoff. Rfrncs [1] Aur, P., Csa-Bianchi, N., Frund, Y. & Schapir, R.E. (1995) Gambling in a riggd casino: Th advrsarial multi-armd bandit problm. In Proc. 36th Annual IEEE Symp. on Foundations of Computr Scinc, pp , Los Alamitos, CA: IEEE Computr Socity Prss. [2] Frund, Y. & Schapir, R.E. (1999) Adaptiv gam playing using multiplicativ wights. Gams and Economic Bhavior 29: [3] Fostr, D. & Vohra, R. (1999) Rgrt and th on-lin dcision problm. Gams and Economic Bhavior 29:7 35. [4] Fudnbrg, D. & Lvin, D.K. (1997) Th Thory of Larning in Gams. Cambridg, MA: Th MIT Prss. [5] Littlston, N. & Warmuth, M.K. (1994) Th wightd majority algorithm. Information and Computation 108 (2):