COMPARISON OF ENTROPY AND MEAN SQUARE ERROR CRITERIA IN ADAPTIVE SYSTEM TRAINING USING HIGHER ORDER STATISTICS

Transcription

1 COMPARIO OF ETROPY AD MEA QUARE ERROR CRITERIA I ADAPTIVE YTEM TRAIIG UIG IGER ORDER TATITIC Dnz Erdogmus and Jos C. Prncp Computatonal uro-engnrng Laboratory Dpartmnt of Elctrcal and Computr Engnrng Unvrsty of Florda, Gansvll, FL 36 E-mal: dnz@grov.ufl.du and prncp@cnl.ufl.du ABTRACT Th rror-ntropy-mnmzaton approach n adaptv systm tranng s addrssd n ths papr. Th ffct of Parzn wndowng on th locaton of th global mnmum of ntropy has bn nvstgatd. An analytcal proof that shows th global mnmum of th ntropy s a local mnmum, possbly th global mnmum, of th nonparamtrcally stmatd ntropy usng Parzn wndowng wth Gaussan krnls. Th prformancs of rror-ntropy-mnmzaton and th man-squar-rrormnmzaton crtra ar compard n short-trm prdcton of a chaotc tm srs. tatstcal bhavor of th stmaton rrors and th hghr ordr cntral momnts of th tm srs data and ts prdctons ar utlzd as th comparson crtra.. ITRODUCTIO tartng wth th arly work of Wnr [] on adaptv fltrs, th man squar rror (ME) has bn almost xclusvly mployd n th tranng of all adaptv systms ncludng artfcal nural ntworks. Thr ar manly two rasons lyng bhnd ths choc: Analytcal smplcty, and th assumpton that most ral-lf random phnomna may b xprssd accuratly by th Gaussan dstrbuton. Th probablty dnsty functon (pdf) of th Gaussan s charactrzd by only ts frst and scond ordr statstcs. nc, undr ths lnarty and Gaussanty assumptons, ME, whch concntrats on scond ordr statstcs, would b abl to xtract all possbl nformaton from a data st whos statstcs ar solly dfnd by ts man and varanc. owvr, most ral-lf problms ar govrnd by nonlnar quatons and most random phnomna ar far from bng normally dstrbutd. Thrfor, for th tranng of adaptv systms, a crtron that not only consdrs th scond ordr statstcs but that also taks nto account th hghr-ordr statstcal bhavor of th systms s rqurd. Th ntropy of a gvn probablty dstrbuton functon, ntroducd by hannon [], s a scalar quantty that provds a masur of th avrag nformaton contand n th dstrbuton. By dfnton, nformaton s a functon of th pdf tslf hnc th ntropy s rlatd to th pdf rathr than any partcular statstcs of t. Applcaton of th ntropy crtron to th systm dntfcaton problm s concptually qut straghtforward. Gvn a tm srs producd by an unknown systm to b usd as th tranng data, th ntropy of th stmaton rror ovr th tranng data st must b mnmzd [3]. Th ntrprtaton of ths s as follows. Whn ntropy of th rror s mnmzd, th xpctd nformaton contand n th stmaton rror s mnmzd; hnc th adaptv systm s trand optmally n th sns that th mutual nformaton btwn th tm srs and th modl output s maxmzd. In practc, an analytcal xprsson for th pdf of a random varabl, whch s ncssary for th computaton of th ntropy, s not avalabl n most cass. Thrfor, t has to b stmatd non-paramtrcally from th sampls of th random varabl. On way to approxmat th pdf of a gvn sampl dstrbuton s to utlz Parzn wndowng [3]. In Parzn wndowng, th pdf s approxmatd by a sum of vn, symmtrc krnls whos cntrs ar translatd to th sampl ponts. A sutabl and commonly usd krnl functon s th Gaussan. Th Gaussan functon s prfrabl bcaus t s contnuously dffrntabl, and thrfor th sum of Gaussans s contnuously dffrntabl on th spac of ral vctors of any dmnson. Th organzaton of th papr s as follows. Frst, th backpropogaton tranng algorthms for both hannon s and Rny s ntropy wth paramtr ar gvn for th

2 on-dmnsonal cas. cond, an analytcal proof showng us that th global mnmum of th ntropy s stll a mnmum of th Parzn wndow stmatd ntropy whn Gaussan krnls ar mployd. Thn, th rsults of a cas study whr stmaton-rror-ntropymnmzaton crtron s appld to th short-trm prdcton of a chaotc tm srs wth a tm-dlay nural ntwork (TD) ar prsntd. Th prformancs of ME trand and ntropy trand TDs ar compard n trms of rror powr and hghr ordr statstcs.. BACKPROPOGATIO FOR TD TRAIIG: ETROPY CRITERIO A typcal systm dntfcaton schm wth a TD s shown n Fg.. Th tranng crtron s what charactrzs th larnng procdur and dtrmns th ovrall prformanc of th fnal modl. Unknown ystm x k TD Crtron Fgur : ystm Idntfcaton chm Th purpos of ths schm s to fnd th TD wghts that optmz th crtron. If th adaptaton crtron s chosn to b th mnmzaton of th man squar rror, and th optmzaton procdur s fxd to b stpst dscnt, thn th orgnatng tranng algorthm s th wll-known backpropogaton algorthm [4]. owvr, du to th rasons statd bfor, f th adaptaton crtron s pckd to b th mnmzaton of hannon s ntropy of th rror wth stpst dscnt, th tranng algorthm bcoms ˆ ( ) w( n + ) w( n) η () r, hannon s ntropy [] s gvn by (), whr th rror pdf s approxmatd by Parzn wndowng wth Gaussan krnls of zro man and varanc σ from ts sampls as shown n (3). + _ xˆk k ) f ( ξ )log f ( ξ ) ( dξ () fˆ ( ξ ) κ ( ξ, σ ) (3) Th gradnt to b usd n th tranng algorthm s thrfor ˆ ( ) xˆ ( ξ ) κ ( ξ )log fˆ ( ξ ) dξ r, xˆ / can b computd as n standard backpropogaton. ot that ths algorthm rqurs th numrcal valuaton of a complcatd ntgral ovr th ral ln. Thrfor, ths algorthm s xtrmly slow and computatonally nffcnt. Ths problm can b solvd by mployng Rny s ntropy wth [6]. Rny s ntropy wth paramtr s gvn by [7] ) log f ξ d ξ ( R ) (4) ( (5) ot that, for, ths xprsson bcoms as n (6) and mnmzaton of ths rducs th maxmzaton of th nformaton potntal [6] R ( ) log f ( ) ξ dξ (6) V ( ) f ( ξ ) dξ In practc, th pdf of rror s agan stmatd usng Parzn wndowng as shown n (3), and th gradnt vctor to b usd n th stpst ascnt algorthm for th maxmzaton of th nformaton potntal s found as Vˆ( ) σ κ ( ( xˆ,σ ) ) xˆ (7) (8)

3 otc that th sum of ntgrals n th hannon s ntropy cas s rplacd by a doubl summaton n ths algorthm wth a doublng n th standard dvaton of th krnl to b usd. Th proprty that mad ths possbl s that th ntgral of two Gaussans wth qual standard dvatons s a Gaussan wth twc th standard dvaton. Ths algorthm s much fastr compard to th prvous on du to ths lmnaton of th ntgral. Ths drvaton may b asly xtndd to th mult-varabl cas. On mportant pont to b mntond n tranng wth ntropy s that, th ntropy dos not chang wth th man of th rror ovr th data st. Ths can b asly shown by a smpl chang of varabls of th ntgraton,.. ζ ξ µ. Du to ths proprty of th ntropy, th algorthm wll convrg, wth a probablty of on, to a st of optmal wghts, whch do not yld zro rrorman. owvr, ths can b corrctd by modfyng th bas wght of th output nuron proprly to yld zro man rror ovr th tranng data st ust aftr tranng nds. It must also b notd that th ntropy s a cost functon wth many local mnma. Th nonlnar natur of th optmzaton was xprmntally vrfd. 3. PROOF OF MIIMA EQUIVALECE ow w procd wth provng that th global mnmum of th ntropy s stll a mnmum of th nonparamtrcally stmatd ntropy for both hannon s ntropy dfnton and Rny s ntropy, whn Parzn wndowng wth Gaussan krnls s utlzd. hannon s Entropy: hannon s ntropy s gvn by (). Clarly, th global mnmum of hannon s ntropy s achvd whn th pdf of rror s th Drac-dlta functon. f ( ξ )log f ( ξ ) dξ f f log f + f dξ f f log f dξ ( ξ ) κ ( ξ ) f dξ log f dξ (9) nc ntropy s ndpndnt of th man of th rror, w can concntrat on th cas whr man of s zro wthout loss of gnralty. Th gradnt of th ntropy for Gaussan krnls s gvn n (9). Evaluatng ths gradnt at [ ] K, ξκ( ξ)logκ ( ξ) dξ () snc w ar ntgratng an odd functon. nc s a statonary pont of (). ow w contnu wth th computaton of th ssan to s f t s furthrmor a mnmum. Usng th sam approach as abov, th dagonal and off-dagonal ntrs of th ssan ar found to b k k σ σ () () Th gnvalus of th ssan matrx thn can b computd to b λ wth multplcty and λ /( ) wth multplcty (-), hnc th ssan s postv sm dfnt. Th valu of th ntropy may dcras n th drcton of th gnvctor that corrsponds to th gnvalu, whch s found as r (3) [ ] T K Ths mans, f th rror changs along ths drcton t wll stll hav constant ntrs. owvr, w hav shown abov that, th man of dos not chang th valu of th ntropy. Thrfor, whn changs along th r, th valu of th ntropy rmans drcton of constant. o w conclud that hannon s ntropy approxmatd by Parzn wndowng wth Gaussan krnls has a mnmum at th pont whr rror s compltly constant ovr th whol data st. Rny s Entropy: Rny s ntropy s dfnd by (5), and s known to approach hannon s ntropy as gos to [7]. It s also ndpndnt of th man of.

4 Th gradnt of Rny s ntropy n th cas of Gaussan krnls valuatd at s found to b R ( ) ξκ dξ κ dξ (4) nc s a statonary pont of Rny s ntropy approxmatd by Parzn wndowng wth Gaussan krnls. Aftr som stps smlar to thos followd n th hannon s ntropy cas, th dagonal and off-dagonal lmnts of th ssan matrx valuatd at ar found as R R k σ σ (5) (6) ot that th ssan matrx for th Rny s ntropy s ndpndnt of, and t s dntcal to th hannon s ntropy. nc, t has th sam gnvalus as th ssan matrx for hannon s ntropy. mlarly, th gnvctor corrspondng to th zro gnvalu s qual as wll and thrfor all th argumnts rlatd ar also vald for Rny s ntropy. Thus w conclud that Rny s ntropy approxmatd by Parzn wndowng wth Gaussan krnls has a mnmum at th pont whr th rror s compltly constant ovr th whol data st. As a cas study, th short-trm prdcton of th Macky- Glass chaotc tm srs [8] wth paramtr τ 3 usng both ME trand and (Rny s) ntropy trand TDs s prsntd hr. Th tm-dlay TD has a 3-tap nput, 5 nurons n th hddn layr and a sngl lnar output nuron. Th mbddng dmnson s chosn to b 3 hr. Ths s lss than th mbddng dmnson suggstd by Takn s Embddng Thorm, namly 5, for th Macky-Glass srs [9]. For ths sz of th rconstructon spac th dffculty lvl of th prdcton problm ncrass. Incrasd dffculty s dsrd snc w know that vn th ME crtron prforms qut wll for ths tm srs whn a 5-tap nput s mployd. Th TDs ar trand ovr a tranng data st of lngth startng from randomly chosn ntal wghts, so that hopfully th global optmal soluton s on of th solutons suggstd by ths Mont Carlo typ tranng approach. In ths squnc, th wghts of th ME trand TD s tratd tms for ach ntal st of wghts whras thos of th ntropy trand TD ar tratd for 3 tms accordng to thr corrspondng backpropogaton algorthms usng th conugat gradnt algorthm []. At th nd of th mntond Mont Carlo tranng approach, th bst st of wghts obtand by ach of th crtra ar takn and chckd for furthr mprovmnt by mployng a small constant stp sz. For ths spcfc cas, ths xtra tratons dd not mprov th cost functon for thr of th crtra. Fnally, th bas wght of th output nuron of both artfcal nural ntworks ar adustd to gv zro man rror ovr th tranng data st aftr tranng had fnshd r ME.4 4. ORT TERM PREDICTIO OF CAOTIC TIME ERIE W hav sn n th prvous sctons that th ntropy crtron dos not dscrmnat btwn pdfs that hav dstnct mans but xactly th sam hghr ordr cntral momnts. Thrfor, whn a TD s trand wth th ntropy crtron, th xpctd valu of th rror ovr th tranng data st wll not convrg to zro. W hav mntond that, ths problm can asly b solvd by adustng th bas wght of th last layr to mak th man of stmaton rror zro ovr th tranng st aftr th larnng fnshs. If th output nuron n th TD s chosn to b a lnar on, ths modfcaton s a smpl addton of th man of th currnt rror to th bas wght of th output nuron. Probablty Dnsty Error Valu Fgur : Th probablty dstrbuton functons for th stmaton rrors of ME and ntropy trand TDs. Th trand ntworks ar tstd on an ndpndntly cratd tst data st of lngth. W do not prsnt th rror plots of th two TDs ovr th tst data st hr bcaus thy do not bar any nformaton whn prsntd n that form. Rathr, w wll concntrat on th statstcal proprts of th rror sgnals. Th pdfs of

5 ths two rror dstrbutons approxmatd by a hstogram of qually spacd bns ar shown n Fg.. As obsrvd from ths plot, th rror dstrbuton of th ntropy trand TD s mor concntratd around zro whras th rror dstrbuton of th ME trand TD s mor unformly dstrbutd ovr ts support. 4 x x -4 3 gnal and Estmatons Data ME r Autocorrlaton of ME Error - Autocorrlaton of r Error ampls Fgur 6: Th short-trm prdcton of Macky-Glass tm srs by ME and ntropy trand TDs Dlay Amount x Dlay Amount x 4 Fgur 3: Th autocorrlaton functons of th stmaton rrors; ME trand and ntropy trand TDs rspctvly. PD of Error (db) r ME ormalzd Frquncy Fgur 5: Th PD stmats of th stmaton rrors. It s obsrvd from Fg. 3 that, th autocorrlaton functon of ntropy trand TD s rror contans hghr frquncy componnts compard to that of th ME trand TD. Ths s also vdnt from th PD stmats of th two random procsss. Th PD of th rror of ntropy trand TD dcays slowr than that of th ME trand on n th lowr frquncs. For compltnss, w prsnt hr th data wth ts stmatons suprmposd. Th data wndow s chosn to nclud th pont whr th ntropy trand TD maks ts maxmum rror, namly th sampl W nvstgat th cntral momnts of th two rror dstrbutons for a mor quanttatv comparson. Th obvous dal soluton should hav a Drac-dlta dstrbutd rror. All cntral momnts of ths dsrd rror dstrbuton ar zro. In othr words, w would lk th cntral momnts of th rror dstrbuton as clos to zro as possbl. Th followng tabl lsts th frst 5 cntral momnts of th rror dstrbutons for th two tranng crtra. ME ME n Ent Ent n n E [( µ ) ] E [( µ ) ].377x -3.85x x x x -6.8x x x -8 Tabl : Cntral Momnts of th rror dstrbutons. Anothr way of lookng at th sam problm s to compar th cntral momnts of th data dstrbuton and th prdctd tm srs dstrbuton. What w ar ssntally tryng to do s to match th probablty dstrbuton of th stmatd tm srs to that of th data sampls. In othr words, w would lk th cntral momnts of th prdctons to b as clos to thos of th tst data st as possbl. Th cntral momnts of th tst data st and th prdctons by th ME and ntropy trand TDs ar gvn nxt. n ME ME n Ent Ent n n E [( x µ ) ] E [( xˆ µ ˆ ) ] E [( xˆ µ ˆ ) ] x.585x -.48x -.6x x x x x x x x x -5.97x -5 Tabl : Cntral Momnts of th dsrd data sampls and th prdcton sampls. x x

6 ot that th vn momnts of th ntropy-trand prdctor ar vry wll matchd to th orgnal tm srs, whl th odd momnts ar stll n rror, but abov th tru valus. Th momnts of th ME trand prdctor ar always smallr than th orgnal, tndng to a unform rror dstrbuton. Ths rsults pont out clarly that th TD prdctons approxmat th statstcal bhavor of th Macky-Glass chaotc attractor bttr whn t s trand wth th ntropy crtron compard to th ME crtron. Th pdf stmatons of th tst data and th TD prdctons for ths data ar prsntd n th followng fgur wth a hstogram of qually spacd bns. Data Entropy ME stogram gnal Valu Fgur 7: Th hstogram of th MG3 and ts prdctons by ntropy-trand and ME-trand TDs rspctvly. 5. COCLUIO In ths papr, an nformaton thortcal larnng crtron for adaptv systms, namly stmaton-rror-ntropymnmzaton, has bn nvstgatd. An analytcal proof, whch shows that Parzn wndowng approxmaton wth Gaussan krnls of th probablty dstrbuton functon to b usd n ntropy computaton prsrvs th orgnal global mnmum of th ntropy as on of th mnma, possbly th global mnmum, of th stmatd ntropy functon. Ths proof nabls us to us Parzn wndowng wth Gaussan krnls n ntropy stmaton safly from th ntropy-mnmzaton pont of vw. A tm-dlay nural ntwork has bn trand for th short-trm prdcton of th Macky-Glass chaotc tm srs usng both th ntropy and man-squar-rror crtra. A Mont Carlo approach has bn takn n trms of th ntal wghts of th tm-dlay nural ntwork n ordr to avod th local mnmum solutons. owvr, t bcam vdnt that vn wth ths approach, th global mnmum of th ME has not bn achvd, snc th ntropy soluton had smallr rror powr. Th bst solutons obtand by th man-squar-rror and th ntropy crtra wr compard n trms of th statstcal bhavor of th stmaton rrors and th prdcton valus thmslvs. Th comparson of cntral momnts of th rror dstrbutons rvald th fact that th rror of th ntropy-trand tm-dlay nural ntwork s closr to th dal soluton. Th comparson of th cntral momnts of tst data sampls and th prdcton sampls lad to th sam concluson. Th prdctons of th tmdlay nural ntwork trand wth th ntropy crtron approxmat th statstcal bhavor of th actual output of th unknown systm bttr than th on trand wth th mn-squar-rror crtron. Acknowldgmnt: Ths work was partally supportd by F grant EC REFERECE [] aykn,., Introducton to Adaptv Fltrs, MacMllan, Y, 984. [] Papouls, A., Probablty, Random Varabls, and tochastc Procsss, 3 rd d., McGraw-ll, Inc., UA, 99. [3] Fshr, J.W., onlnar Extnsons to th Mnmum Avrag Corrlaton Enrgy Fltr, Ph.D. Dssrtaton, Unvrsty of Florda, 997. [4] Parzn, E., On Estmaton of a Probablty Dnsty Functon and Mod, n Tm rs Analyss Paprs, oldn-day, Inc., CA, 967. [5] Rumlhart, D.E., Chauvn, Y., Backpropogaton: Thory, Archtcturs, and Applcatons, Lawrnc Erlbaum Assocats, J, 995. [6] Rny, A., A Dary on Informaton Thory, Wly, Y, 987. [7] Prncp, J., Xu, D., Fshr, J., Informaton Thortc Larnng, n Unsuprvsd Adaptv Fltrng, mon aykn Edtor, 65-39, Wly,. [8] Kaplan, D., Glass, L., Undrstandng onlnar Dynamcs, prngr-vrlag, Y, 995. [9] Kuo, J.M., onlnar Dynamc Modlng Wth Artfcal ural tworks, Ph.D. Dssrtaton, Unvrsty of Florda, 993. [] Lunbrgr, D.G., Lnar and onlnar Programmng, Addson-Wsly Pub. Co., MA, 973.