Realizing Undelayed n-step TD Prediction with Neural Networks

Transcription

1 ealizing Undelayed n-step TD Pediction with eual etwok Jani Zute Faculty of Computing, Univeity of Latvia aina bulvai 9-44, LV-586 iga, Latvia Abtact Thee exit vaiou technique to extend einfocement leaning algoithm, e.g., eligibility tace and planning. In thi papen appoach i popoed, which combine eveal extenion technique, uch a uing eligibilitylike tace, uing appoximato a value function and exploiting the model of the envionment. The obtained method, undelayed n-tep TD pediction (TD-P, ha poduced competitive eult when put in condition of not fully obevable envionment. I. EXTESIO TECHIQUES OF EIFOCEMET LEAIG A. Eligibility Tace Eligibility tace ae one of the baic mechanim of einfocement leaning to handle delayed ewad. The idea behind all eligibility tace i vey imple. Each time a tate i viited it initiate a hot-tem memoy poce tace, which then decay gadually ove time. Thi tace mak the tate a eligible fo leaning.[] Algoithm aa_lambda (n Q tate-action value π(q policy deived fom Q λ decay ate α leaning ate maximum eligibility tace length Initialize Q Tace Empty andom nonteminal tate a chooe action fom tate accoding policy π(q While i not final tate Append <> to Tace, ' Take action a, obeve ewad and next tate a' chooe action fom tate ' accoding policy π(q δ γq(',a' γq(, λ * Fo tep < * * > in Tace fom the end backwad Q( *,a * Q( *,a * αδλ * λ * γλλ * ' Fig.. Saa(λ (with limited tace length Thee ae two way to view eligibility tace. The moe theoetical view of eligibility tace i called the fowad view, and the moe mechanitic view i called the backwad view. The fowad view i mot ueful fo undetanding what i computed by method uing eligibility tace, wheea the backwad view i moe appopiate fo developing intuition about the algoithm themelve.[] Fig. depict taditional Saa algoithm amed with eligibility tace. Fo implicity and conitency, it i aumed that the envionment i deteminitic and the algoithm ue tate-action value (not jut tate value. To view the ame poce accoding the fowad view, we hould look fowad fom the cuent tate (to the ubequent tate to compute it value; and it can be caied out though making a delay in modification of tate-action value. B. einfocement Leaning and eual etwok Standad einfocement leaning algoithm aume that tate-action value ae toed in a table one value pe tateaction pai. Algoithm aa_neual Ψ upevied neual netwok to toe tate-action value Initialize Ψ, «'» Take action a, obeve ewad and next tate Tain Ψ with, v Fig.. Claic Saa algoithm complemented by a neual netwok to toe tate-action value Unfotunately thi i not alway poiblend big total amount of tate and action i only one of poible eaon. Moeove, thee can be too complicated to accuately fill uch a table of value, o even to identify all the poible tate and action, becaue tate-action pace may include continuou vaiable o complex enation. In thi cae the iue i that of genealization; and a upevied neual netwok (e.g., multi-laye pecepton i a typical choice to ealize that. The implet way of uing a neual netwok Ψ i to apply it intead of value table Q (ee Fig. fo the claic Saa algoithm with a neual netwok a a value function. To

2 bette decibe the algoithm in uch condition, it i woth to add a notion of et of featue, e.g., denote a et of featue of ome tate. In thi pape, it i aumed that poible action ae countablend «a» mean jut the code of action a obtained in ode to be paed to a neual netwok a an input. Eligibility tace ae alo ued combined with function appoximato (ee []. Accodingly alo neual netwok can be incopoated into a einfocement leaning algoithm with eligibility tace (Fig.. Algoithm aa_neual_lambda ( Ψ upevied neual netwok to toe action value λ decay ate η leaning ate fo Ψ maximum eligibility tace length Initialize Ψ Tace Empty Append <> to Tace, «'» Take action a fom, obeve eult η * η Fo tep < * * > in Tace fom the end backwad Tain Ψ with <, v> uing leaning ate η * η * η * γλ Fig.. Saa(λ (with limited tace length adjuted fo uing neual netwok a a value function C. Planning in einfocement Leaning By planning we mean looking ahead in a model of an envionment in ode to enhance the einfocement leaning poce. The model i ued to imulate the envionment and get ome imulated expeience. Fig. 4. elationhip among leaning, planningnd acting[][] Extenion of einfocement leaning with planning i caied out by integating it with leaning and acting pocee (See Fig. 4. Thee exit vaiou method to integate planning into einfocement leaning. A well known one i that of Dyna[4]. In Dyna, to extend leaning fom eal expeience, ome leaning fom imulated expeience i alo added. The idea of Dyna algoithm i the following: in each tep of the epiodedditional taining i pefomed fo andomly choen tate-action paiccoding the value obtained by the help of the envionment model. Fig. 5 how the Dyna algoithm [4] adapted fo Saa leaning and neual netwok a a value function. Algoithm dyna_aa_neual Ψ upevied neual netwok to toe tate-action value upevied neual netwok to model envionment Initialize Ψ, cuent (nonteminal tate, «'» Take action a, obeve ewad and next tate Tain Ψ with, v Tain with <, <, «'»>> epeat time «*» andom peviouly obeved tate a * andom action peviouly taken in *, «**» («*,a *» % Take action in model a ** chooe action fom «**» accoding policy π(ψ v * γψ(«**,a **» Tain Ψ with «* *», v Fig. 5. Dyna-Saa algoithm II. UDELAYED -STEP TD PEDICTIO Thi chapte i to decibe the Undelayed n-tep TD pediction (TD-P, the autho popoed appoach to oganize fowad-looking mechanim though exploing the envionment model, ealized by a neual netwok. In ome technical detail, thi appoach link both with eligibility tace mechanim (fo uing a tace to viit othe tate and the Dyna algoithm (fo uing a model to pedict the behaviou of the envionment. A. Inpiation The main idea of the popoed appoach i jut looking fowad fo additional infomation (intead of looking back, ued in mot algoithm becaue of being le complicated in tem of computation to enhance the algoithm. Theoetically, looking fowad can be ealized via eveal method, e.g., n-tep TD pediction, whee tate-action value ae updated in the diection of n-tep taget (See Eq. (, adapted fom []:

3 ( t... γ γ n γ Q( γ. Hee, looking fowad i fomalctually ealized though delay mechanim (i.e., it i jut a fowad view, not eal looking fowad. A tue looking fowad i poible though involving pediction of futue tate. B. The Algoithm -tep pediction can be ealized diectly though looking fowad in the model of envionment (intead of the envionment itelf, like in the Dyna algoithm (Fig. 5 thu modelling futue tep of the epiode (So Eq. ( tun into Eq. (: ( t... γ γ Q( γ ( γ ( ( whee i actual ewad; i modelled futue ewad; ( i modelled futue tate. Pocedue aa_p ( Ψ upevied neual netwok to toe action value upevied neual netwok to model envionment maximum fowad tace length n(, fowad tace length function Initialize Ψ, T 0 T T, «'» Take action a fom, obeve eult v γ * γ «*» «'» n * n(t While «*» doe not epeent teminal tate and n * 0 a * chooe action fom «*» accoding policy π(ψ *, «**» («*,a *» % Take action in model n * n * If «**» epeent teminal tate o τ * 0 v v γ * * Ele v v Ψ(«*,a *» γ * γγ * «*» «**» Tain Ψ with <, v> Tain with <, <, «'»>> Fig. 6. Saa-P: Saa with TD-P Unlike the Dyna algoithm (Fig. 5 and alo eligibilitytace-baed algoithm (Fig., in thi appoachdditional activitie within a tep of an epiode concen pedicted futue tate, intead of peviouly obeved one. Thee one moe complication within the context of thi appoach: in the beginning, while the available infomation on the envionment i inufficient, it in t eaonable to look fa fowad. Theefoe, to avoid unacceptable looking fowad in the beginning of the tial, the maximum count of tep (to look fowad hould be eplaced by ome function n(, (See Eq. (, deigned in a way that fo the fit epiode no looking fowad i pefomed. γ (n(, T T, t n(, T T, n(, T T, γ γ... ( n(, T T, Q( T, n(, T T, n(, T whee T numbe of the epiode, maximum fowad tace length. The peudocode of Fig. 6 depict Saa-P algoithm: the appoach of TD-P applied to the Saa algoithm. C. Benefit of the Appoach Of coue, thee no idea to apply neual-netwok-baed algoithm fo imple poblem, e.g., fo thoe with fully obevable envionment and eay identifiable tate. Any pactical benefit fo uch algoithm can be expected only in had condition, o hee we aume that the poblem ae had enough to be woth uing neual netwok to model the value function. Advantage of appoache with eligibility tace ove taditional TD algoithm ae well known uch algoithm ae fate. In tem of ideology, the main advantage of the popoed appoach of TD-P ove eligibility tace i diminihed amount of taining opeation needed fo the neual netwok, ealizing the tate-action value function: In algoithm with eligibility tace, the value i computed fo the cuent tatend then popagated back ove the tace by pefoming taining many time fo each tate-action in the tace; In thi TD-P appoach, tavelling ove the tace i done jut to compute the valuend then taining i pefomed only once fo the cuent tate. Although thi advantage tend to be a bit heuitic (becaue it can be ened only in pecial condition; it can be conideable, if the taining poce i heavy to pefom. III. EXPEIMETAL WOK A. The Poblem to Solve The expeimentation wa done on elative imple poblem jut handled a if the envionment wa not fully obevable. The goal of the expeimentation wa to how uch an appoach woking well a be able to outpefom othe method in cetain had condition.

4 The idea of the expeiment wa vey imple: the algoithm wee teted on two deteminitic gid-wold poblem: 5 6 and 7 8 (See Fig 7 and 8 and the output wee compaed againt the uboptimal policie (The et of uboptimal policie wa obtained a tatitic though taditional TD algoithm a a et of poible action fo each tate encounteed at leat in 5% of cae. Each tial wa un until a uboptimal policy wa obtained (good eult o 00,000 epiode wee eached (bad eult. The dicount ate γ=0.9 wa ued. The envionment of the poblem wee et to behave a if they wee not fully obevable. White cell denote nomal (nonteminal tate; black cell tand fo the wall, while haded cell epeent teminal tate. A numbe in a cell denote the ewad of a move to thi cell. Thee ae 4 action: up, down, left, ight, which caue the agent to move one cell in the coeponding diection. The tate emain unchanged if the agent tie to go off the gid o againt the wall Fig. 7. Poblem 5 6 and it uboptimal policie Saa (Fig. ; Saa(λ (Fig. with two diffeent maximum tace length and 5; Saa-P (Fig. 6 with two diffeent maximum tace length and 5. Fo neual netwok, the following coding wa ued: State two numbe fom the inteval [0, ] (coeponding to the hoizontal and vetical poition of the tate in the gid; Action one numbe fom the inteval [0, ]. eual netwok (multi-laye pecepton achitectue ued (one hidden laye: Value function: -40- Tanition ewad (Saa-P only: -40- Tanition tate (Saa-P only: -40- Othe paamete of the expeiment: Maximum count of epiode in a tial 00,000. Leaning ate fo neual netwok 0.. Decay ate λ=0.9 (Saa(λ only Policie wee computed fom the coeponding value function in ε-geedy manne with ε=0%. The fowad tace length function n(, wa ealied by the following function (Eq. (4: 0; n(, T = n(, T g( n(, T ; T = 0 T > 0 whee T numbe of the epiode, maximum fowad tace length, g change ate (fo the cuent expeimentation g wa et to ( Fig. 8. Poblem 7 8 and it uboptimal policie B. Configuation of the Teting Envionment The poblem wee tied to olve via thee following algoithmll of which ued a neual netwok a a value function: Fowad tace length, n (,T Epiode numbe in a tial, T Fig. 7. Illutation of the fowad tace length function n(, (Eq. (4 ued in Saa-P algoithm (Fig. 6 to avoid looking fowad too fa in the fit epiode C. Expeimental eult The total of 64 expeimental tial wa pefomed to fom the final eult (Fig. 8. Two of thee poblem wee olved via five diffeent vaiant of the algoithm ( 5=0 cae. Amount of tial which conveged to a uboptimal policy (i.e., a atifactoy eult wa obtained. Ca. 58% of tial of the eaiet poblem 5 6 conveged uccefully. Bet eult wee obtained by Saa(λ. = =5

5 Only about 8% of tial conveged in the cae of the hade poblem 7 8nd hee the bet eult wa eached by the Saa-P algoithm. Summay of the eult fo diffeent kind of the Saa algoithm featued by neual netwok a value function: Bet eult fo the poblem 5 6 wee eached by Saa(λ; Saa(λ wa geneally bette than taditional Saa. Thi would be jut a little complement to cedibility of expeimentation with uch configuation; Bet eult fo the poblem 7 8 wee eached by Saa-P; Taditional Saa wa abolute unable to olve the poblem 7 8; Saa-P wa upiingly weak on olving the eay poblem 5 6. Convegence ate 0% 00% 80% 60% 40% 0% 0% 9% 9% 8% % 78% 75% 0% 7% 00% 4% Saa-P =5 Saa-P = Saa Saa(λ = Saa(λ =5 Fig. 8. Expeimental eult pecentage of tial convegedccoding the poblem olved and algoithm applied The mot diappointing one wee the eult of Saa-P on olving the eaiet 5 6 poblem (9% and 8% depending on the maximum length of the tace. It would pobably ignal fo intability of the algoithm o it cuent ealization. IV. COCLUSIO The popoed appoach combine eveal method o technique to extend einfocement leaning: Uing tace (like eligibility tace to viit othe tate; Uing appoximato fo value function; Exploiting the model of the envionment to obtain additional infomation. The appoach ha been deigned fo ue in pecific had condition of poblem olving not fully obevable envionment with potentially unidentifiable tate. The main goal of the expeimentation wa to how uch an appoach wokingnd thi ha been eached the coeponding algoithm ha hown competitive eult againt othe method. Although Saa-P wa weak in olving one of the poblem: 5 6; it clealy outpefomed othe algoithm on olving the moe complicated poblem 7 8nd thi keep the popoed appoach of TD-P pomiing fo futhe development. EFEECES [] K. Famling. eplacing eligibility tace fo action-value leaning with function appoximation, Euopean Sympoium on Atificial eual etwok, Buge, Belgium: Ap. 007 [] S. P. Singh and. S. Sutton, einfocement Leaning with eplacing Eligibility Tace, Machine Leaning, :-58, 996. []. S. Sutton and A. G. Bato, einfocement Leaning. An intoduction. Cambidge,MA: MIT Pe/A Badfod Book, 998. [4]. S. Sutton, Dynan integated achitectue fo leaning, planning, and eacting, SIGAT Bulletin, :60 6, 99.