Detection of Language (Model) Errors

Transcription

1 Dttion of Languag (Modl) Errors K.Y. Hung, R.W.P. Luk, D. Yung, K.F.L. Chung and W. Shu Dpartmnt of Computing Hong Kong Polythni Univrsity Hong Kong {skyhung, srluk, sdanil, skhung, Abstrat Th bigram languag modls ar popular, in muh languag prossing appliations, in both Indo-Europan and Asian languags. Howvr, whn th languag modl for Chins is applid in a novl domain, th auray is rdud signifiantly, from 96% to 78% in our valuation. W apply pattrn rognition thniqus (i.. Baysian, dision tr and nural ntwork lassifirs) to disovr languag modl rrors. W hav xamind 2 gnral typs of faturs: modlbasd and languag-spifi faturs. In our valuation, Baysian lassifirs produ th bst rall prforman of 80% but th prision is low (60%). Nural ntwork produd good rall (75%) and prision (80%) but both Baysian and Nural ntwork hav low skip ratio (65%). Th dision tr lassifir produd th bst prision (81%) and skip ratio (76%) but its rall is th lowst (73%). Introdution Languag modls ar important post-prossing moduls to improv rognition auray of a wid varity of input, namly sph rognition (Balh t al., 1983), handwrittn rognition (Elliman and Lanastr, 1990) and printd haratr rognition (Sun, 1991), for many human languags. Thy an also b usd for txt orrtion (Ron t al., 1994) and part-of-sph tagging. For Indo-Europan languags, th word-bigram languag modl is usd in sph rognition (Jlink, 1989) and handwriting rognition (Nathan t al., 1995). Various ways to improv languag modls wr rportd. First, th modl has bn xtndd with longr dpndnis (.g. trigram) (Jlink, 1991) and using non-ontiguous dpndnis, lik triggr pairs (Rosnfld, 1994) or long distan n-gram languag modls (Huang t al., 1993). For bttr probability stimation, th modl was xtndd to work with (hiddn) word lasss (Brown t al., 1992, Ward and Issar, 1996). A mor rror-drivn approah is th us of hybrid languag modls, in whih som dttion mhanism (.g. prplxity masurs [Kn and O Kan, 1996] or topi dttion [Mahajan t al., 1999]) slts or ombins with a mor appropriat languag modl. For Asian languags (.g. Chins, Japans and Koran) rprsntd by idographi haratrs, languag modls ar widly usd in omputr ntry baus ths Asian languags hav a larg st of haratrs (in thousands) that th onvntional kyboard is not dsignd for. Apart from using sph and handwriting rognition for omputr ntry, languag modls for Asian languags an b usd for sntn-basd kyboard input (.g. Lohovsky and Chung, 1997), as wll as dtting impropr writing (.g. dialtspifi words or xprssions). Unlik Indo-Europan languags, words in ths Asian languags ar not dlimitd by spa and onvntional approximat string mathing thniqus (Wagnr and Fishr, 1974; Oommn and Zhang, 1974) in handwriting rognition ar sldom usd in Asian languag modls. Instad, a widly usd and rportd Asian languag modl is th haratr-bigram languag modl (Jin t al., 1995; Xia t al., 1996) baus it (1) ahivd high rognition auray (around 90-96%) (2) is asy to stimat modl paramtrs (3) an b prossd quikly and (4) is rlativly asy to implmnt. Improvmnt of ths languag modls for Indo- Europan languags an b applid for th Asian languags but words nd to b idntifid. For Asian languags, th modl was intgratd with syntati ruls (Chin, Chn and L, 1993). Class basd languag modl (L and Tung, 1995) was also xamind but th lasss ar basd on smantially rlatd words. A nw approah (Yang t al., 1998) is rportd using sgmnts

2 xprssd by prfix and suffix trs but th omparison is basd on prplxity masurs, whih may not orrlat wll with rognition improvmnt (Iyr t al., 1997). Whil attmpts to improv th (bigram) languag modls wr (quit) sussful, th high rognition auray (about 96%) is not adquat for profssional data ntry srvis, whih typially rquir an rror rat lowr than 1 in 1,000. As part of th quality ontrol xriss, ths srvis stimat thir rror rat by sampling, and thy idntify and orrt th rrors manually to ahiv th rquird quality. Fad with a larg volum of txt, th ability to automatially idntify whr th rrors ar is prhaps mor important than automatially orrting rrors, in post-diting baus (1) manual orrtion is mor rliabl than automati orrtion, (2) manual rror sampling an b arrid out and (3) mor manual fforts ar rquird in rror idntifiation than orrtion du to th larg volum of txt. For xampl, if th idntifiation of rrors is 97% and thr ar no rrors in rror orrtion, thn th auray of th languag modl is improvd from 96% to 99.9% aftr rror orrtion. In typial appliations, th auray of th bigram languag modl may not b as high as thos rportd in th litratur baus th data may b in a diffrnt gnr than that of th training data. For valuation, w tstd a bigram languag modl with txt from a novl domain and its auray droppd signifiantly from 96% to 78%, whih is similar to English (Mahajan t al., 1999). Improvmnt in th robustnss of th bigram languag modl aross diffrnt gnr is nssary and svral approahs ar availabl, basd on dtting rrors of th languag modl. On (adaptiv) approah is to automatially idntify th rrors and manually orrting thm. Th information about th orrtion of rrors is usd to improv th bigram languag modl. For xampl, th bigram probabilitis of th languag modl may b stimatd and updatd with th orrtd data. In this way, futur ourrns of ths rrors ar rdud. Anothr (hybrid) approah uss anothr languag modl to orrt th idntifid rrors. This languag modl an b omputationally mor xpnsiv than th bigram languag modl baus it is applid only to th idntifid rrors. Also, topi dttion (Mahajan t al., 1999) and languag modl sltion (Kn and O Kan, 1996) an b applid to thos ara to find a mor appropriat languag modl baus usually topi-dpndnt words ar thos ausing rrors. Anothr (intgrativ) approah improvs th languag modl auray using mor sophistiatd rognizrs, instad of a omplmntary languag modl. Th mor sophistiatd rognizr may giv a st of diffrnt rsults that th bigram languag modl an r-apply on or this rognizr simply givs th rognizd haratr. This intgrats wll with th oars-fin rognition arhittur proposd by Nagy (1988) bak in th 1960s. Coars rognition provids th andidats for th languag modl to slt. Fin, xpnsiv rognition is arrid out only whr th languag modls faild. Finally, it is possibl to ombin all th diffrnt approahs (i.. adaptiv, hybrid and intgrativ). Givn th signifian in dtting rrors of languag modls, thr is littl work in this ara. Prhaps, it was onsidrd that ths rrors wr random and thrfor hard to dtt. Howvr, usrs an dtt rrors quikly. W suspt that som of ths rrors may b systmati du to th proprtis of th languag modl usd or du to languag spifi proprtis. W adopt a pattrn rognition approah to dtting rrors of th bigram languag modl for th Chins languag. Eah output is assignd to ithr th lass of orrt output or th lass of rrors. Th assignmnt of a lass to an output is basd on a st of faturs. W xplor a numbr of faturs to dtt rrors, whih ar lassifid into modl-basd faturs and languag-spifi faturs. Th proposd approah an work with Indo- Europan languags at th word-bigram lvl. Howvr, languag-spifi faturs hav to b disovrd for th partiular languag. In addition, this approah an b adoptd for n-gram languag modls. In prinipal, th modl-basd faturs an b found or valuatd similar to th bigram languag modl. For xampl, if th trigram probability (instad of bigram probability) is low, thn th liklihood of a languag modl rror is high. This papr is organizd as follows. Stion 1 disusss various faturs and som prliminary valuation of thir suitability for rror

3 idntifiation. Stion 2 dsribs 3 typs of lassifirs usd. In stion 3, our valuation is rportd. Finally, w onlud. 1. Faturs W valuat individual faturs for rror dttion baus thy ar important to th suss of dttion. Artils from Yazhou Zhoukan (YZZK) magazin (4+ Mbyts)/PH orpus (Guo and Liu, 1992) (7+ Mbyts) ar usd for valuation. W us th rall and prision masurmnts for valuation. Th rall is th numbr of rrors idntifid by a partiular fatur dividd by th total numbr of rrors. Th prision is th numbr of rrors idntifid by a partiular fatur dividd by th total numbr of tims th fatur indiat that thr ar rrors. In th first substion, w dsrib som modl-basd faturs. Nxt, w dsrib th languag-basd faturs. In th last substion, w disuss th ombind us of both typs of faturs. 1.1 Modl-basd faturs Th bigram languag modl slts th most likly path P max out of a st S. Th probability of a path s in S is simply th produt of th onditional probabilitis of on haratr i aftr th othr i-1 whr s = s, aftr making th Markov assumption. Formally, P max = arg max s S { p( s) } = arg max p( 0) s S i p( i i 1 ) s = s Th st s is gnratd by th st of andidat haratrs for ah rognition output. Th rognizr may supply th st of andidat haratrs. Altrnativly, a oars rognisr may simply idntify th bst-mathd group or lass of haratrs. Thn, mmbrs of this lass ar th andidat haratrs. Formally, w us a funtion h(.), that maps th rognition position to a st of andidat haratrs, i.. h(i) = { i,j }. W an also dfin th st of sntns in trms of h(.), i.. S = {s s = 0 1 n, i, i h(i)} Faturs basd on zro probabilitis (F 1,1 ) On fatur to dtt rrors is to ount th numbr of onditional probabilitis p( i i-1 ) that ar zro, btwn 2 onsutiv positions. Zro onditional probabilitis may b du to insuffiint training data or may b baus thy rprsnt th languag proprtis. Figur 1 shows th liklihood of an rror ourring against th prntag of th onditional probabilitis that ar zro. 60% 50% 40% 30% 20% 10% prision 0% 50% 60% 70% 80% 90% 100% 110% Figur 1: Th languag modl output rrors against prntags of zro onditional probabilitis Faturs basd on low probability (F 1,2 ) Whn thr ar insuffiint data, th onditional probabilitis that ar small ar not rliabl. If P max hav sltd som onditional probabilitis that ar low, thn probably thr ar no othr hois from th andidat sts. Hn, th insuffiint data problm may our in that partiular P max. In Figur 2, w plot th liklihood of rrors idntifid against th diffrnt logarithmi onditional probability valus. Whn th rall inrass, unfortunatly, th prision drops. Figur 2: Th prision, rall and auray (i.. rall prision) of dtting languag modl rrors by xamining th logarithm onditional probabilitis on th maximum liklihood path. 1.2 Languag-spifi Faturs Th languag-spifi faturs ar basd on applying th word sgmntation algorithm (Kit t al., 1989) to th maximum liklihood path. Th ROCLING (Chn and Huang, 1993) word list usd for sgmntation has 78,000+ ntris.

4 1.2.1 Faturs basd on word lngth (F 2,1 ) If th mathd word in th maximum liklihood path is long, thn w xpt th liklihood of an rror is low baus long words ar spifi. Figur 3 shows th prision of dtting th mathd word is orrt and th rall of rrors in multi-haratr words. In gnral, th longr th mathd words, th mor likly that thy ar orrt and th liklihood of missing undttd long words is small. 120% 100% 80% 60% 40% 20% 0% prision Figur 3: Th prision of orrt mathd words against word lngths Faturs basd on singl-haratr squns (F 2,2 ) In word sgmntation, whn thr ar no ntris in th ditionary that an math, th input is sgmntd into singl haratrs. Thus, Lin t al (1993) notd that singl-haratr squns aftr word sgmntation might indiat sgmntation problms. Hr, w apply th sam thniqu for th dttion of rrors. If w ount on th pr haratr basis, th rall of rror is 80% and th prision in rror idntifiation is 35%. If w ount multi-haratr words and a squn of singl-haratrs as bloks, thn th rall of rrors is 79% and th prision in finding on or mor rrors in th blok is inrasd to 51%. 120% 100% 80% 60% 40% 20% 0% prision rall Figur 4: Th prision and rall of singl-haratr squns of diffrnt lngths. Similar to mathd words in th maximum liklihood path, th rror dttion prforman of singl-haratr squns may dpnd on thir lngth. Thrfor, w plottd th rall and prision of dtting rrors against th lngth of th singl-haratr squns. Aording to Figur 4, as th lngth of th singl-haratr squn is larg, th liklihood of an rror is largr. Th rall of rrors is partiularly low for singl-haratr squns that hav 2 haratrs. Th othr singl-haratr squns (i.. its lngth is not quals to 2) hav almost 100% rall. On possibl rason why 2 singl-haratr squns ahivd low prision is that thr ar many spurious bigrams and thrfor fals math. 1.3 Combind us of Faturs W arrid out a prliminary study using th faturs mntiond in substion 1.1 and 1.2. Our Baysian lassifir (Stion 2.1) ahivd 83% rall but 35% prision, whih an b ahivd using languag spifi faturs only (F 2,2 ). Thrfor, w try to ombin th us of ths faturs in a mor arful mannr. W dividd th dttion into 3 snarios: (1) singl haratr (fatur F 2,2 ); (2) singl-haratr squn of lngth 2 (fatur F 2,2 ) and (3) 2 haratr words (fatur F 2,1 ). Eah as is assignd a lassifir to dtt rrors. Singl-haratr squns longr than 2 ar onsidrd as having rrors (Figur 4). Words of lngth longr than 2 ar onsidrd orrt (Figur 3) Singl haratrs Aftr word sgmntation, singl haratrs ar thos ass whn thr ar no multi-haratr words in th ditionary that an math with it and its following substring. Th singl haratrs hav diffrnt part-of-sph tags. Figur 5: Th singl haratrs and thir orrsponding languag modl output auray for diffrnt part-ofsph tags. Figur 5 shows that th auray of th languag modl for ths singl haratrs with part-of-

5 sph tags rlatd to xlamations ar low. For rror dttion, a fatur is assignd to ah partof-sph tag. Th languag modl auray for singl haratrs may dpnd on th availability of th lft and right ontxt to form high probability bigrams. Thrfor, w xpt that languag modl auray of singl haratrs at th bginning (70%) and nd (70%) of a sntn is lowr than thos in th middl (85%) of th sntn. Th worst as ours whn th sntn has only a singl haratr, whr th masurd auray is only 8.75% (i.. no bigram ontxt) Two-singl-haratrs squn Figur 6 shows that languag modl output auray inrass as th bigram probability of singl-haratr squns of lngth 2 inrass. Hn, th bigram probabilitis an b usd as a fatur for dttion. Figur 6: Th bigram (logarithm) probability of th singl-haratr squn of lngth 2. Similar to singl haratrs, th languag modl auray for 2-singl-haratrs squns at th start, middl and nd of a sntn ar 48%, 47% and 30%, rsptivly. Th auray is 33% if th sntn is th 2-singl-haratrs squn. Anothr fatur for 2-singl-haratrs squns is to xamin whthr th haratrs in th two andidat sts an form words that math with th ditionary. Ths mathd words ar alld hiddn words. Figur 7 shows that if thr ar hiddn words, th languag modl auray droppd from 60% to 25%. Sin thr ar not many ass with 6-8 hiddn words, th auray for ths ass ar not rliabl Two-haratr words For 2 haratr words, th bigram probability (Figur 8) an b usd as a fatur similar to th singl-haratr squns. Th position of ths 2 haratr words in th sntn dos not rlat to th languag modl auray. Our masurd auray is 91%, 89% and 91% for th bginning, th nd and th middl of th sntn, rsptivly. Evn sntns with a singl 2- haratr word ahivd 90% auray. Hn, thr is no nd to assign faturs for th position of th 2 haratr words in a sntn. Similar to 2-singl-haratrs squns, th languag modl auray (Figur 9) drass as th numbr of hiddn words inras in th orrsponding 2 sts of andidat haratrs. Figur 9: Th languag modl auray against diffrnt numbr of hiddn words. Figur 7: Languag modl auray against diffrnt numbr of hiddn words (s txt). Figur 8: Th languag modl auray of 2 haratr words against th bigram probability.

6 2 Classifirs On of th problms with using individual faturs is that th rall and prision ar not vry high, xpt th languag-spifi faturs. It is also diffiult to st th thrshold for dttion baus of th prision-rall trad-off. In addition, thr may b som improvmnt in dttion prforman if faturs ar ombind for dttion. Thrfor, w adopt a pattrn rognition approah to dtt rrors. Svral lassifirs ar usd to did for rror idntifiation baus w do not know whthr partiular faturs work wll with partiular lassifirs, whih mak diffrnt assumptions about lassifiation. Thr typs of lassifrs will b xamind: Baysian, dision tr and nural ntwork. 2.1 Baysian Classifir Th Baysian lassifir is simpl to implmnt and is ompatibl with th modl-basd faturs. Givn th fatur vtor x, th Baysian dttion shm assigns th orrt lass w and th rror lass w, using th following rul: g ( x) > g ( x) Othrwis whr g (.) and g (.) ar: assign w assign w T 1 g ( x) = ( x µ ) ( x µ ) log + 2 log p( w ) T 1 g ( x) = ( x µ ) ( x µ ) log + 2 log p( w ) µ and µ ar th man vtors of th lass w and w, rsptivly, Σ and Σ ar th ovarian matris of th lass w and w, rsptivly, and. is th dtrminant. 2.2 Dision Tr Originally, w trid to us th support vtor mahin (SVM) (Vanpik, 1995) but it ould not onvrg. Instad, w usd th dision tr algorithm C4.5 by Quinlan (1993). Dision trs ar known to produ good lassifiation if lustrs an b boundd by som hypr-rtilinar rgions. W traind C4.5 with a st of fatur vtors, dsribd in Stion Nural Ntwork W us th multi-layr prptron (MLP) baus it an prform non-linar lassifiation. Th MLP has 3 layrs of nods: input, hiddn and output. Nods in th input layr ar fully onntd with thos in th hiddn layr. Likwis nods in th hiddn layr ar fully onntd to th output layr. For our appliation, on input nod orrsponds to a fatur in stion 1.3. Th valu of th fatur is th input valu of th nod. Two output nods indiat whthr th urrnt haratr is orrt or rronous. Th numbr of hiddn nods is 2-4, alulatd aording to (Fujita, 1998). Th output of ah nod in th MLP is th wightd sum of its input, whih is transformd by a sigmoidal funtion. Initially, th wights ar assignd with small random numbrs, whih ar adjustd by th gradint dsnd mthod with larning rat 0.05 and momntum Evaluation In th valuation, th training data is th PH orpus and th tst data is th YZZK magazin artils (4+ Mbyts), downloadd from th Intrnt. In handwrittn haratr rognition, th optimal siz of th numbr of andidats is 6 (Wong and Chan, 1995). For robustnss, ah rognizd haratr in our valuation is sltd from 10 andidats. W masurd th prforman in trms of rall, prision and th manual ffort rdution in sanning th txt for rrors. Th rall is th numbr of idntifid rrors ovr th total numbr of rrors. Th prision is th numbr of idntifid rrors ovr th total numbr of ass lassifid as rrors. Th amount of saving in manual sanning for rrors is alld th skip ratio, whih is th numbr of bloks lassifid as orrt ovr th total numbr of bloks. Th rall and th skip ratio ar mor important than th prision baus post rror orrtion (manual or automati) an improv th rognition auray. It is possibl to ombin th rall and prision into on, using th F masurs (Van Rijsbrgn, 1979) but th valu for rating th rlativ importan is subjtiv. Tabl 1 shows th lassifiation prforman of th Baysian lassifir. Th rall of rrors by th Baysian lassifir has rdud slightly from 83% using a singl lassifir to 79% using 3 lassifirs but th prision improvd from 51% to 60%. Also, th skip ratio is 65%, whih is muh highr than th skip ratio of 0.1% if w did not us th lassifir. Although th MLP has a highr prision (80%), its rall is slightly lowr than

7 th Baysian lassifir. Th skip ratio of th both Baysian and MLP lassifirs ar about th sam. Cass Masur Bays C4.5 MLP Singl haratr 2 singl haratrs 2-haratr words Rall 71% 56% 28% Prision 40% 75% 71% Rall 60% 84% 83% Prision 88% 82% 80% Rall 60% 17% 9% Prision 29% 60% 62% Aknowldgmnt This work is supportd by th (Hong Kong) Univrsity Grants Counil undr projt PolyU 5109/97E and th PolyU Cntral Rsarh Grant G-S603. W ar gratful to Guo and Liu for providing th PH orpus and ROCLING for providing thir word list. Rfrns Bahl, L.R., F. Jlink and R.L. Mrr (1983) "A maximum liklihood approah to ontinuous sph rognition", IEEE Trans. PAMI, 5:2, pp Ovrall Rall 79% 73% 75% Prision 60% 81% 80% Skip Ratio 65% 76% 66% Brown, P.F., V.J. Dlla Pitra, P.V. dsouza and R.L. Mrr (1992) "Class-basd n-gram modls of natural languag", Computational Linguistis, 4, pp Tabl 1: Th prformans of th 3 typs of lassifirs in dtting languag modl rrors. 4 Summary and Futur Work W hav valuatd both modl-basd and languag-spifi faturs for dtting languag modl rrors. Individual modl-basd faturs did not yild good dttion auray, suffring from th prision-rall trad-off. Th languagspifi faturs dtt rrors bttr. In partiular, mathd multi-haratr words ar usually orrt. If th modl-basd and languag-spifi faturs ar aggrgatd as a singl fatur vtor, th rall and prision of rrors ar 83% and 35%, rsptivly, whih ar th sam if w just us languag-spifi faturs. Hn, instad of a singl lassifir, w sparatd 3 situations idntifid by th languag-spifi faturs and 3 lassifirs ar usd to dtt ths rrors individually. Th Baysian lassifir (simplist) ahivd an ovrall 79% rall, 60% prision and 65% skip ratio and th MLP ahivd an ovrall 75% rall, 80% prision and a 66% skip ratio. Similar rall and prision prformans ar ahivd using dision trs, whih ar prfrrd sin thir skip ratio is highr (i.. 76%). Although th prision (so far) is not high (60% - 80%), it is not th most important rsult baus (1) this only rprsnts a minor wast of hking ffort, ompard with sanning th ntir txt, and (2) th idntifid rrors will b hkd furthr or orrtd ithr manually or automatially. Chn, K.-C. and C.R. Huang (ds.) (1993) "Chins word lass analysis", Thnial Rport 93-05, Chins Knowldg Information Prossing Group, Institut of Information Sin, Aadmia Sinia, Taiwan. Chin, L.F., Chn, K.J., L, L.S. (1993) "A bstfirst languag prossing modl intgrating th unifiation grammar and Markov languag modl for sph rognition appliations", IEEE Trans. Sph and Audio Prossing, 1:2, Pag(s): Elliman, D.G. and I.T. Lanastr (1990) "A rviw of sgmntation and ontxtual analysis thniqus for txt rognition", Pattrn Rognition, 23:3/4, pp Fujita, O. (1998) "Statistial stimation of th numbr of hiddn units for fdforward nural ntworks", Nural Ntworks, 11, Guo, J. and H.C. Liu, "PH a Chins orpus for pinyin-hanzi transription", ISS Thnial Rport, TR , Institut of Systms Sin, National Univrsity of Singapor, Huang, X., F. Allva, H. Hon, M. Hwang,, K. L and R. Rosnfld (1993) "Th SPHINX-II sph rognition systm : an ovrviw", Computr Sph and Lanaguag, 2, Iyr, R., M. Ostndorf and M. Mtr (1997) "Analyzing and prditing languag modl

8 prforman", Pro. IEEE Workshop Automati Sph Rognition and Undrstanding, pp Jlink, F. (1989) "Slf-organizd languag modling for sph rognition", in Radings in Sph Rognition, Morgan Kayfmann. Jlink, F. (1991) "Up from trigrams", Pro. Eurosph 91, pp Jin, Y., Y. Xia and X. Chang (1995) Using ontxtual information to guid Chins txt rognition, Pro. ICCPOL 95, pp Knn, P.E. and M. O Kan (1996) "Hybrid languag modls and spontanous lgal disours", Pro. ICSLP, Vol. 2, pp Kit, C., Y. Liu and N. Liang (1989) "On mthods of Chins automati word sgmntation", Journal of Chins Information Prossing, 3:1, Law, H.H-C. and C. Chan (1996) N-th ordr rgodi multigram HMM for modling of languags without markd word boundaris, Pro. COLING 96, pp L, H-J. and C-H Tang (1995) "A languag modl basd on smantially lustrd words in a Chins haratr rognition systm", Pro. 3 rd Int Conf. on Doumnt Analysis and Rognition, Vol. 1., pp Lin, M-Y., T-H. Chiang and K-Y. Su (1993) A prliminary study on unknown word problm in Chins word sgmntation, Pro. ROCLING VI, pp Lohovsky, A.F. and K-H. Chung (1997) "Homonym rsolution for Chins phonti input", Communiations of COLIPS, 7:1, Mahajan, M., D. Bfrman and X.D. Huang (1999) "Improving topi-dpndnt modling using information rtrival thniqus", Pro. IEEE ICASSP 99, Vol. 1, pp Nagy, G. (1988), "Chins haratr rognition: twnty-fiv-yar rtrosptiv", in Pro. 9th Int. Conf. on Pattrn Rognition, Vol. I, pp Nathan, K.S., H.S.M. Bigi, J. Subrahmonia, G.J. Clary and H. Maruyama (1995) "Ral-tim onlin unonstraind handwriting rognition using statistial mthods", Oommn, B.J. and K. Zhang (1996) "Th normalizd string diting problm rvisitd", IEEE Trans. on PAMI, 18:6, pp Quinlan, J.R. (1993) "C4.5 programs for mahin larning", Morgan Kaufmann, CA. Ron, D., Y. Singr and N. Tishby (1994) "Th powr of Amnsia: larning probabilisti automata with variabl mmory lngth", to appar in Mahin Larning Rosnfld, R. (1994) "Adaptiv statistial languag modling" a maximum ntropy approah", Ph.D. Thsis, Shool of Computr Sin, Carngi Mllon Univrsity, Pittsburgh. Sun, S.W. (1991), "A Contxtual Postprossing for Optial Chins Charatr Rognition", in Pro.Int. Sym. on Ciruits and Systms, pp Vapnik, V. (1995) Th Natur of Statistial Larning Thory, Springr-Vrlag, Nw York. Van Rijsbrgn (1979) Information Rtrival, Buttrworths, London. Wagnr, R.A. and M.J. Fishr (1974) "Th string to string orrtion problm", J. ACM, 21:1, pp Ward, W. and S. Issar (1996) "A lass basd languag modl for sph rognition", Pro. IEEE ICASSP 96, Vol. 1, pp Wong, P-K. and C. Chan (1999) "Postprossing statistial languag modls for handwrittn Chins haratr rognizr", IEEE Trans. SMC, Part B, 29:2, Xia, Y., S. Ma, M. Sun, X. Zhu, Y. Jin and X. Chang (1996) "Automati post-prossing of offlin handwrittn Chins txt rognition", Pro. ICCC, pp Yang, K-C., T-H. Ho, L-F. Chin and L-S. L (1998) "Statistis-basd sgmnt pattrn lxion - a nw dirtion for Chins languag modling", Pro. IEEE ICASSP 98, Vol. 1., pp