A Machine-Vision Technique for Automated American Sign-Language Alphabets Recognition

Transcription

1 A Machine-Vision Technique for Auomaed American Sign-Language Alphabes Recogniion Aaron R. Rababaah Mah & Compuer Science Universiy of Maryland Easern Shore Princess Anne, 21853, USA Absrac- Wih he echnological rend in man-machine inerfaces and he machine inelligence, exploiing hese powers has become a challenge in many fields. In paricular, i was observed ha he body gesure-based ineracions of human o human and human o machine are rapidly increasing, especially in he area of sign language inerpreaion. Saisics in he Unied Saes of America srongly sugges ha he populaion of deaf and mue people is on he rise and here is a need o rain more people he American Sign Language (ASL) o bridge he gap. Furhermore, he elecronic devices such as TVs, PCs, PDAs Robos, cameras, ec. are advanced and buil o read users gesures and respond o heir commands. Therefore, i is of a grea ineres o ry o conduc research in his area and propose efficien and effecive soluions for man-machine gesure-based ineracion. In his sudy, a sysem for auomaic American Sign Language alphabes recogniion was proposed and developed. The adoped approach is differen from previous work in he sense i provides a muli-color based encoding scheme o esablish he differen signaures or paerns of he differen hand-signs. A series of image and vecor processing operaions were used in order o ransform a visual hand gesure ino a spoken leer and displayed ex. The domain and scope of his sudy is he sandard American Sign Language alphabes. The experimenal resuls of he developed sysem indicae ha he sysem is effecive wih an accuracy greaer han 93%. Keywords- Machine Vision; Machine Inelligence;American Sign Language;Auo Recogniion;Image Processing; Colorbased Segmenaion I. INTRODUCTION AND RELATED WORK This era is marked by he informaion revoluion. Humans are rapidly moving oward auomaion in many aspecs of life especially in informaion echnology. We are highly moivaed by auomaion due o is various benefis including: produciviy, accuracy, efficiency, qualiy, consisency, safey, ec. In he area of sign language inerpreaion, some of hese moivaions apply, for example produciviy (speed of inerpreaion), accuracy (correcness of he inerpreaion), and consisency (objeciviy of he sysem vs. subjeciviy of he human inerpreer). There are wo main approaches o auomae his applicaion, daa gloves-based and vision-based [7]. The daa glove approach relies on a glove equipped wih sensors o deec he posiions of he fingers and esablish a daa vecor ha uniquely represen a gesure. On he oher hand, he vision-based approach image processing is used o segmen and exrac he hand gesure off of he scene and pos-process i o exrac some feaures o uniquely represen a gesure paern. In boh approaches, a machine-learning echnique is used for raining and classificaion, such as neural nework, selforganizing maps, fuzzy logic, clusering echniques, ec. Since daa gloves are expensive and no readily accessible [7], a visionbased echnique will be presened in his paper for auomaing Sign Language Recogniion (SLR). In his secion, several aemps in previous relaed work and heir experiences in heir sudies are presened. Liang and Ming [1] proposed a real ime recogniion sysem for Taiwanese Sign Language. Their sysem is a muli-phase recogniion process relying on four differen feaures: posure, posiion, orienaion and moion. These feaures are hen mached wih an esablished gesure model for evaluaing he candidae posures and recognizing he sign. This sudy focuses on he echnology used in recognizing he hand posure, which was done using he DaaGlove sysem equipped wih acceleromeers in each finger o sense and conver he relaive finger posiions ino daa ha can be used for furher analysis. A 95% accuracy was repored for he posure feaure. Pansare e al. [2] proposed a vision-based hand gesure classificaion. Their sysem is based on human skin color feaures o exrac he hand region and conver i ino a binary image. The binary image is hen subjeced o morphological operaors and edge deecion o exrac he premier of he hand and finally, he curren hand is compared o a se of reference models for classificaion. They repored a classificaion accuracy of 90%. In anoher aricle, Sakshi Goyal [3] proposed a mehod for gesure recogniion. The proposed algorihm consised of four major seps, which are Image Acquisiion, Feaure Exracion, Orienaion Deecion and Gesure Recogniion. The feaure exracion phase is based on he well-known algorihm of SIFT [4]. Using he highes key poins maching beween he curren gesure and he regisered gesures, hey were able o

2 recognize he differen signs wih an accuracy of 95%. Vaishali e al. proposed a sysem for auomaic saic hand gesure recogniion based on hree main feaure exracion echniques of hisogram, Hough ransform, and edge deecion. The feaure space is hen used o rain and es a neural nework model for classificaion. They repored a classificaion accuracy of 92.33%. Wen Gao e al. [6] inroduced a complee sysem for Chinese language recogniion sysem, bu he ineres of his sudy is only in he sign feaure exracion and gesure recogniion. They used he un-supervised neural nework model of self-organizing maps o classify he differen hand gesures. In heir sudy, a 48-dimensional vecor is formed of hand shape (36 elemens), posiion (6 elemens) and orienaion vecor (6 elemens) for he wo hands. The daa from differen signers are calibraed by some fixed movemens performed by each signer. They repored a classificaion accuracy of 82.9%. The purpose of his sudy is o devise a new echnique for SLR ha is effecive, efficien and inexpensive so ha i should be very accessible o he average person. Also, as i was repored in [7], where a comparaive sudy was performed among he available echniques in ASR said The cusom made color glove mehod requires he color segmenaion o be performed. Bu he accuracy is less, he aim of his sudy is o improve his shorcoming of he colored gloves and show ha i can be effecive, accurae and economical. The res of he paper is organized as follows: secion II discusses he heory and echnical approach, secion III presens he sysem esing and experimenal work and finally, secion IV draws conclusions and discusses fuure work. II. heory and echnical approach of he proposed echnique The proposed sysem is called American Sign Language Alphabes Recogniion Sysem (ASLARS). The projec is a design of American Sign Language sysems ha recognizes and ranslaes physical hand gesures ino readable ex displayed on a compuer monior. The sysem enables users o use American Sign Language wih universally recognized characers while generaing prin on a compuer monior and speaking he generaed prin. Today we have many languages used across naions, especially American Sign Language, which is on he rise since If his coninues, one would believe ha schools will have o aler heir curriculum by aemping o fix he increase. This ASLARS applicaion will be used o help reduce he gap beween he verbally impaired (mues) and hose who do no know American Sign Language. Applicaion developmen was carried ou using an inegraed environmen in Malab called Inelligence Sysem Inegraed Developmen Environmen (ISIDE), which was developed by Dr. Aaron Rababaah a Mah/Compuer Science, Universiy of Maryland Easern Shore [8]. Fig. 1: Block Diagram of he Proposed Sysem In he secion, he echnical approach and heoreical background in he developmen of he proposed echnique will be discussed. The differen sages of he sysems, as shown in Fig. 1, are presened

3 A. Building The Reference Templaes Wih he use of a black glove wih finger ips colored Red, Green, Blue, Yellow, Cyan for he humb o pinkie respecively, he sofware will recognize hrough he developed algorihm he posiion of each finger, herefore he corresponden leer. A Reference Templae ha conained he 26 leers in he Alphabe was developed. Each image of each leer conained he ASL sign for he leer wih he finger-colored ips. Then each image was run hrough he algorihm o ge he x and y coordinaes of each finger-colored ip. Afer having all he coordinaes for he image, hey were saved in a vecor, which would sore he 26 leer coordinaes. The sample images of he differen signs and heir corresponding vecors are shown in Fig. 2. B. Inpu Image Fig. 2: Sample Reference Signs and heir Corresponding Paern Vecors In he image acquisiion sage, he acquired video was convered o image frames by he video-o-frame converer sofware componen in Malab [12]. The video was discreized ino single frames and sored in a direcory o be processed ino reference emplaes in he raining mode or o be aken as he curren image o be recognized in he esing mode. C. Color Segmenaion The hird sep was a For loop ha compues he vecor for each one of he color segmens of each finger. Firs, i would localize each color segmen in he image by he given RGB values for he Red, Green, Blue, Yellow, Cyan, using an RGB filer ha reurns he pixels ha are wihin a 15% olerance range. Table 1 shows an example of compuing he absolue disances beween a reference vecor and an example pixel vecor. This work is comparable o he mehods surveyed in [7]: skin-based [9], coloredgloved [11] and finger deecion [12]. TABLE 1: EXAMPLE OF ABSOLUTE DISTANCE COMPUTATION BETWEEN TEMPLATE VECTOR AND AN EXAMPLE VECTOR In his sage, wo differen segmenaion mehods were esed, RGB color space-based and HSV color space-based filers. Fig. 3 shows a comparison beween he wo mehods, where i was shown hrough experimenal work ha he RGB-based mehod performed significanly beer. As i can be observed, he qualiy and compleeness of he resuling segmened objec is significanly beer in he RGB-based mehod

4 D. Objec Enhancemen Fig. 3: Sample Comparison beween RGB filer (Righ) vs. HSV filer (Lef) Alhough filering is very useful in noise eliminaion in digial image processing, i does no always produce he required resuls. There exiss anoher noise reducion echnique known as morphology. Morphology is a broad se of image processing operaions ha process images based on heir shapes. Morphological operaions apply a srucuring elemen o an inpu image, creaing an oupu image of he same size. In a morphological operaion, he value of each pixel in he oupu image is based on a comparison of he corresponding pixel in he inpu image wih is neighbors. By choosing he size and shape of he neighborhood, a morphological operaion can be consruced sensiive o specific shapes in he inpu image. The mos basic morphological operaions are dilaion and erosion. Dilaion adds pixels o he boundaries of objecs in an image, ha is, i expands he componen of an image, while erosion removes pixels on objec boundaries. The number of pixels added or removed from he objecs in an image depends on he size and shape of he srucuring elemen used o process he image. Fig. 4: Example of Noise segmenaion resul In he morphological dilaion and erosion operaions, he sae of any given pixel in he oupu image is deermined by applying a rule o he corresponding pixel and is neighbors in he inpu image. The rule used o process he pixels defines he operaion as dilaion or erosion [12]. Afer he segmenaion sage, he resuling objecs migh have some problems, including noise represened by scaered small regions, voids in he body of he segmened regions, and disconneced pars of he same objec. These are ypical

5 problems ha can be effecively recified by morphological operaions. For small noise regions, he area filer, where an area hreshold was used o eliminae any area below ha hreshold, was used. For he voids in he objec body, he close and fill morphological operaions were used. For he disconneced objecs, he dilae and bridge morphological operaions were used. An example is given in Fig. 4. E. Conneced Componen Analysis (CCA) In labeling, he general goal is o assign a label o each poenial arge pixel such ha he pixels ha belong o he same physical arge will share he same label. Labeling is also known as conneced componen analysis; in is simple form i involves he assignmen of he same label o adjacen pixels. I is an algorihmic applicaion of graph heory, where subses of conneced componens are uniquely labeled based on a given heurisic. The algorihm of conneced componen analysis is illusraed in he pseudo-code algorihm below [12]. The erm blob is used someimes o refer o hose labelled pixels. A sample oupu of he CCA is given in Fig. 5. ALGORITHM CCA( I F ( x, ) // 1 s pass For each pixel in I F ( x, If I F ( x, 0 N k = geneighbors(x,; If ALL N k = 0 Else I F ( x, = newlabel; I F ( x, = minlabel( N k ); } } } Sore equivalenlabels in N k ; // 2nd pass For each pixel in I F ( x, If I F ( x, 0 } I F ( x, = minlabel(equivalenlabels); F. Paern Vecor Generaion Fig. 5: Sample resul of he CCA algorihm Afer he CCA was applied o all regions, he cenroid of each segmened and idenified region was compued as in he following formula and demonsraed in Fig

6 , (1) Fig. 6: Cenroid Compuaion Example Afer he cenroids were compued, a series of ransformaions were carried ou o ensure normalizaion of scale and ranslaion. This sage is illusraed in Figs. 7a and 7b. If a color is no found, boh coordinaes, x and y are se o zero (0,0). Now comes one of he mos imporan seps in he algorihm, normalizaion. The sign vecor needs o be normalized o make each one of he images locaion and scale independen. This means ha if he glove in he image is closer or furher away from he camera, or shifed lef, righ, up or down i will no have any effecs on he sign vecor. To achieve his normalizaion, some basic arihmeic operaions were applied o he values in he vecor. Firs, he mean of he x and y values was found for he Red, Green, Blue, Yellow and Cyan colors and saved in a variable. To ge he localizaion normalizaion, he mean was exraced from each one of he original values, and hen saved in a new vecor. This would show how far away each one of he values are from he cener of he Caresian plane creaed by he finger-colored ips, making he locaion independen. As he absolue value of each one of hem is obained and he maximum values for x and y found separaely (he maximum value will help us wih he scale normalizaion), hey were saved in emporary variables. The values used for he scale normalizaion are he ones obained afer subracing he mean from he original values. Each one of hese values was divided by he maximum value obained in he previous sep. This division would provide values beween -1 (negaive one) and 1 (one) only. This made he values scale independen because no maer wha he original scales were, afer he processing hey will all be wihin his range. Fig. 7a: Concep of Paern Vecor Compuaion and Normalizaion

7 Raw Daa R G B Y C Mean X Y Locaion Normalizaion (subrac mean) R G B Y C Scale Normalizaion (divide by max(abs)) R G B Y C Absolue Value R G B Y C Max G. Sign Paern Recogniion Fig. 7b: Paern Vecor Normalizaion of Scale and Translaion This normalized vecor was he finalized signed vecor from he previous sep. To recognize he vecor, Templae Maching Technique was used. The curren vecor was compared wih he Reference Templaes o find he closes mach. Using Euclidean disances, each x and y coordinae from he signed vecor was compared wih he coordinaes in he Reference Templae vecor o find he closes mach and fech he leer. The index of he closes mach was passed o a ex o speech funcion and a wrie funcion ha will prin he leer on he screen. An example of he complee cycle is demonsraed in Fig. 8. The algorihm is given below, adoped from [13]. ALGORITHM: Templae_Maching 1. Train maching engine by esablishing a daa se of reference emplaes xref, xi x1, x2,..., xn}, where x ref : he se of reference emplaes x i : he i h reference emplae in he se x ref n : number of reference emplaes in he se x ref 2. Le yi be he ih signal inpu o he emplae maching engine. 3. For each xi in xref, 3.1 Compue he signal correlaion facor as in (2) [16]. 3.2 Form a se (V) of voed emplaes wih he pair ( i, c i ), where, c i is he class label of he reference emplae x i. 3.3 For each class in he x ref, compue he sum of all i s as he cumulaive weighed voes for ha paricular class. 3.4 Compare all of he cumulaive voes (V i s) for each class and selec he highes figure (V*) o be he mos likely class label o be assigned o he inpu signal y i

8 3.5 If V* >= confidence level hreshold, assign he inpu signal y i he label of he winner class. 3.6 Repea sep (2) while here are inpu signals Finish Where, = The correlaion coefficien beween he wo variables X 1, X 2 ; Cov = The covariance of he wo variables X 1, X 2 Cov( X 1 X 2) E( X1X 2) E(x) = The mahemaical expecaion of a variable E ( x) u( x) f ( x) dx x S = The sandard deviaion. ; 1 2 ; Cov( X1, X1) (2) 1 2 Fig. 8: Complee Cycle of One Sign Recogniion III. SYSTEM TESTING A. Sysem Calibraion The mos imporan calibraion is he RGB-filer in he segmenaion sage since i is crucial for he qualiy of he finger ips objecs, which in urn will significanly impac he accuracy of he paern vecor generaion and ulimaely he recogniion accuracy. To do he RGB-filer calibraion, he following seps were followed: 1. Sample image frames of individual signs were colleced; 2. For each of he frames in (1), each fingerip was sampled muliple imes and a sample space was buil; 3. For each se of samples represening a color/fingerip, a saisical model of mean and sigma olerance was compued; 4. The model in (3) was esed on unseen samples and he calibraion accuracy compued;

9 5. If calibraion accuracy is accepable (>90), hen he model was acceped, else more samples were added and he algorihm repeaed from 1 hrough 5. The calibraion ool is shown in Fig. 9. Inpu Image Frame Color-Space Transformaion Calibraion Tool Inerface Exraced Targe Fingerip B. Experimens Fig. 9: Calibraion ool for RGB-filer Seveny-six samples were colleced for all of he alphabes muliple samples per each leer. All of hese samples were processed wih he model in Fig. 1 and he following are he observaions: Eigheen samples were eliminaed for very bad resoluion and/or exreme iling. If all samples were considered uncaegorized per leer, ASLARS was able o recognize 49 ou of 58 leers accuracy of 49/58 = 84.48%. If he caegory of leers was considered only, ASLARS was able o recognize 24 ou of 26 leers accuracy of 24/26 = 92.31%. If he reference emplae is improved and exended, i is more han likely ha he accuracy will be above 95%. The sampling synchronizaion affeced he qualiy of he recorded images. IV. CONCLUSIONS AND FUTURE WORK The design and developmen of a new echnique o auomae he sign language recogniion were presened. This new echnique is called American Sign Language Alphabes Recogniion Sysem (ASLARS). The main moivaion for his echnique was o provide an inexpensive accessible mehod for sign language auomaic recogniion ha can be considered for fuure developmen o be complee sign language inerpreer and rainer. The echnique depends on a cusom-colored glove o faciliae he deecion of he fingerips and generae a sign paer vecor using he cenroid of he exraced fingerips regions. Gaussian RGB-filer was used o exrac he fingerips off of he gesure. The conneced componen analysis is a very imporan sage in he echnique o idenify individual fingers and generae he unique paern vecor accordingly. Throughou he experimen, i was observed ha he echnique s recogniion accuracy is comparable or beer someimes o/han oher repored echniques. Our fuure work includes he following asks: Creae a hreshold o rejec unrecognizable leers; Exend reference emplae o include numbers 0 o 9; Improve glove qualiy; Cusomizable signs for differen applicaions; Improve he efficiency o creae a real-ime sysem; Use he ASLARS sofware as an educaional and raining ool for people who wan o learn sign language; Use he ASLARS sofware o es ineresed professional in he area of sign language inerpreaion

10 REFERENCES [1] Rung-Huei Liang and Ming Ouhyoung, A Real-ime Coninuous Gesure Recogniion Sysem for Sign Language, IEEE, pp , Japan [2] Jayashree R Pansare e al., Real-Time Saic Hand Gesure Recogniion for AmericanSign Language (ASL) in Complex Background, Journal of Signal and Informaion Processing 2012, 3, [3] Sakshi Goyal e al. Sign Language Recogniion Sysem For Deaf And Dumb People, Inernaional Journal of Engineering Research & Technology (IJERT), ISSN: , Vol. 2 Issue 4, April [4] David G. Low, Disincive Image Feaures from Scale-Invarian Keypoins, Inernaional Journal of Compuer Vision, [5] Vaishali S. Kulkarni and S.D.Lokhande, Appearance Based Recogniion of American Sign Language Using Gesure Segmenaion, (IJCSE) Inernaional Journal on Compuer Science and Engineering, Vol. 02, No. 03, 2010, [6] Wen Gao e al., A Chinese sign language recogniion sysem based on SOFM/SRN/HMM, Eslevier, Paern Recogniion 37 (2004) [7] Rashmi D. Kyaanavar and P. R. Fuane, Comparaive Sudy of Sign Language Recogniion Sysems, Inernaional Journal of Scienific and Research Publicaions, Volume 2, Issue 6, June ISSN [8] Aaron R. Rababaah, Inelligen Sysems Inegraed Developmen Environmen (ISIDE), a MaLab-based sofware plaform, Mah & CS Deparmen, Universiy of Maryland Easern Shore, [9] Paulraj M P, Sazali Yaacob, Hazry Desa, Hema C.R., Exracion of Head & Hand Gesure Feaure for Recogniion of sign language, InernaionalConference on Elecronic Design, Penang, Malaysia, December 1-3, 2008 [10] Rini Akmeliawai, Melanie Po-Leen Ooi and Ye Chow Kuang, Real-Time Malaysian Sign Language Translaion using Colour Segmenaion and Neural Nework, IMTC Insrumenaion and MeasuremenTechnology Conference Warsaw, Poland, 1-3, May [11] Ravikiran J, Kavi Mahesh, Suhas Mahishi, Dheeraj R, Sudheender S, NiinV Pujari, Finger Deecion for Sign Language Recogniion, Proceedingsof he Inernaional MuliConference of Engineers and Compuer Scieniss2009 Vol I IMECS 2009, Hong Kong, March 18-20, [12] Aaron Rababaah, Adapive Decision Suppor Technique for Machine Vision-Based Securiy Sysems o Deec Anomalous Evens, Journal of Precision Insrumens and Mechanology (PIM), [13] Aaron R. Rababaah, Even Deecion, Classificaion And Fusion For Non-Saionary Vehicular Acousic Signals, Inernaional Journal of Science & Informaics, Vol. 1, No. 1, Fall, 2011, pp [14] Aaron Rababaah, Efficien Visual Tagging of Human Subjecs in Inelligen Video Surveillance Sysems, Journal of Modern Technology and Managemen Insiue, [15] Aaron R. Rababaah, Alaa M. Rababaah, Compac Visual Ideniy Characerizaion (CVIC) To Enhance Informaion Verificaion In Video Securiy Sysems, he Global Digial Business Annual Conference, , [16] Aaron R. Rababaah, Even Deecion, Classificaion And Fusion For Non-Saionary Vehicular Acousic Signals, Inernaional Journal of Science & Informaics, Vol. 1, No. 1, Fall, 2011, pp