Recognition of ASL for Human-robot Interaction

Transcription

1 66 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July 2017 Recognton of ASL for Human-robot Interacton Md. Al-Amn Bhuyan College of Computer Scences & Informaton Technology, Kng Fasal Unversty, Al Ahsa, Saud Araba. Summary Ths artcle addresses on the development of a framework of Amercan Sgn Language for human-robot nteracton. The approach s based on the computer vson strateges and ntellgent task schedulng for an entertanment robot. The system s organzed wth the dentfcaton of sgns n Amercan Sgn Language (ASL) usng adaptve resonance theory neural network. Expermental results ndcate that the method s capable of dentfyng the ASL sgns wth an accuracy of more than 90% under dfferent llumnaton condtons. Key words: Amercan Sgn Language, Hstogram equalzaton, Human-robot nteracton, Abo, ART Neural network, Skn color segmentaton. 1. Introducton ASL s a vsual gesture language that works as the prncpal way of expressng thoughts of deaf communtes. Over the last few decded, ASL has occuped a poston of overwhelmng domnance n research n the felds of mage processng, computer vson, pattern recognton and so on. Snce t offers a natural and effectve way of explorng expressons, ASL s, therefore, provdng remarkable nterest n the advancement of human-robot nteracton. ASL nvolves about 6000 gestures of common words wth fnger spellng used to communcate ndstnct words or proper nouns. Fnger spellng uses one hand and 26 gestures to communcate the 26 letters of the alphabet. The 26 alphabets of ASL are shown n Fg. 1. Numerous research papers have been publshed on ASL [1-5]. Vogler and Metaxas traned context-dependent Hdden Marcov Models and modelng transent movements between sgns nspred by the characterstcs of ASL phonology. They employed computer vson technques for 3D object shape and moton parameter extracton to acheve accurate 3D movement parameters of ASL sentences [6]. Fok et al [7] developed a framework of a real-tme mult-sensor recognton system for ASL. They collected data from leap moton sensors and fused them for recognton employng hdden Markov models. Charayaphan and Marble [8] proposed vson technques to recognze ASL. Fels and Hnton [9] proposed a neural network nterface between a VPL data-glove and a speech syntheszer. Ther method used backpropagaton neural network for mappng hand movements. Starner and Pentland [10] employed a Hdden Marcov model to extract the features for recognzng solated sgns of ASL. Grobel and Assan [11] employed Hdden Marcov model and was able to dentfy ASL sgns wth 91.3% accuracy. They pulled out the features from vdeo recordngs of sgners wearng colored gloves. Bowden and Sarhad [12] used a nonlnear pont dstrbuton model, allowng a much rcher descrpton of hand shape, and augment ths wth a Markov model for Amercan Sgn Language. Ther method s based on one-state transtons of the Englsh Language, whch are projected nto shape space for trackng. Munb et al [13] employed Hough transform and neural networks to recognze ASL sgns. Zhang et al. [14] propoed an ASL recognton method dependng on acton recognton strateges to dentfy the ASL sgns. They desgned a corpus for analyzng the red, green and blue components of color model and aggregated multple sgnal modaltes for depth vdeos of dfferent sentences of ASL. Ths system s organzed to dentfy all the sgns of the ASL. Users do not requre to wear any gloves to nteract wth the system. The sgners vary ther hand shapes only. The system s based on Affne transformaton [15],.e., scalng, translaton and rotaton on presentng the gesture. A B C D E F G H I J (dynamc) K L M N O P Q R S T U V W X Y Z (dynamc) Fg. 1 Amercan Sgn Language of alphabets. Manuscrpt receved May 17, 2017 Manuscrpt revsed June 29, 2017

2 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July The remanng part of the artcle s organzed as follows. Secton 2 descrbes the System desgn and archtecture. The necessary mage processng and mage segmentaton process s explaned n detals n dfferent subsectons. Secton 3 descrbes the expermental results and performance of the system. Fnally, Secton 4 draws the overall conclusons of ths research and proposes the future drectons. 2. System Desgn and Archtecture The ASL recognton system comprses wth two segments: () the feature extracton and () dentfcaton of sgns, as llustrated n Fg. 2. The feature extracton process s ntated wth an mage processng procedure, whch nvolves an algorthm to detect and segment varous desred segments of the sgn. For ths, each color mage s reszed and converted from RGB to HSV color space. After applyng the skn color segmentaton process a bnary mage s generated from whch the largest connected component s beng analyzed, converted nto a feature vector that s compared to the feature vectors of a tranng set of sgns. The feature extracton process conssts of mage standardzaton, mage enhancement, flterng, and skn color segmentaton. 2.1 Image Standardzaton Images of sgns were reszed to 30 24, by default uses nearest neghbor nterpolaton to determne the values of pxels n the output mage. Ths research employs a lowpass flter before nterpolaton to reduce alasng. 2.2 Fuzzy Hstogram Equalzaton FHISEQ() Ths algorthm ncrease contrast of mage usng classcal hstogram equalzaton technque. Let the maxmum gray level s L, s k th gray level over [0,L-1] and s the r k transformed gray level of r k. Step1. Construct the fuzzy mage of the gven mage by usng the followng membershp functon: r k k L Step2. Calculate the fuzzy hstogram of the constructed fuzzy mage n step 1. Step3. Calculate the fuzzy equalzed hstogram. Step4. Transform the gray level of each pxel as follows: s k (2) sk L * fhe( rk ) (3) Start Resze the Image Convert RGB to HSV Skn color segmentaton Connected component analyss Feature extracton stage The hand mages may be of poor contrast because of the lmtatons of the llumnaton condtons. Therefore, fuzzy hstogram equalzaton s used to remburse for the llumnaton condtons and recover the contrast of the mage, as shown n Fg. 3. No ART network Is movement? Fuzzy hstogram equalzaton technque s based on the fuzzy normalzed hstogram of the mage. The equalzed fuzzy hstogram s defned: th k k j gray level fherk fp r ( r j ), j0 j0 gray level (1) where k 0,1,2,... L 1. The algorthm to mprove the contrast of the mage usng fuzzy hstogram equalzaton technque s as follows: Yes ASL command generaton Robot acton Fg. 2 System Archtecture. Classfcaton stage

3 68 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July 2017 The fuzzy expectaton of a dscrete random varable x havng values x 1 x, x,... xn wth respectve fuzzy, 2 3 probabltes P x ), P( x ), P( x ),... P( x ) s defned by ( n n n FE( x) x P( x ), where P( x ) 1. (4) Flterng The ASL mages are sometmes corrupted by numerous sources of nose. Therefore, Prewtt flterng s used to suppress the nose. 2.4 Skn color segmentaton Skn color segmentaton s organzed wth vsual nformaton of the hand skn colors extracted from dfferent mages. Ths research uses HSV color space for skn color segmentaton. In the HSV color space a color s descrbed by three attrbutes: hue, saturaton and value. Hue s the vsual attrbute of color sensaton lnked wth the domnant colors, saturaton mples the relatve purty of the color content and value measures the brghtness of a color. The transformaton from RGB to HSV s gven by the equatons [16-20]: 1 (R G) (R B) 1 H 2 1 cos (5) 2 (R G) (R B)(G B) rangng [0,2], where H 360 H1 ; H H 1 f B G; otherwse max(r, G, B) mn(r, G, B) S, max(r, G, B) (6) max(r, G, B) V, 255 where R,G,B are the red, green, and blue component values whch exst n the range [0,255]. Snce the human skn colors are clustered n color space and dffer from person to person and of races, so n order to detect the hand parts n an mage, the skn pxels are thresholded emprcally [21-23]. In ths research, the hue values are chosen h= [0, 40]. The detecton of hand regon boundares by such an HSV segmentaton process s llustrated n Fg. 4. The exact locaton of the hand s then determned from the mage wth largest connected regon of skn-colored pxels. 2.5 Classfcaton phase The classfcaton phase nvolves neural network tranng for the recognton of bnary mage patterns of the hand. In the classfcaton stage, an Adaptve Resonance Theory (ART) neural network s employed. The ART contans neurons n the nput layer, 604 (70% of nput) neurons n the hdden layer, and 26 neurons n the output layer. The algorthm for the ART network s gven bellow. (a) Hand mage (b) Hstogram of mage (a) (c) Fuzzy equalzed mage (d) Hstogram of mage (c) Fg. 3 Fuzzy Hstogram equalzaton. (a) Orgnal Color mage (b) Color segmentaton (c) Connected component analyss Fg. 4 Skn color segmentaton.

4 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July ART1 Algorthm Step 0 : Intalze parameters and weghts [16]: L L 1,0 1, 0 bj(0), tj(0) 1. L 1 n Step 1: Whle stoppng condton s false do Steps 2-13 Step 2: For each tranng nput, do steps 3-12 Step 3: Set actvatons of all F2 unts to zero. Set actvatons of F1(a) unts to nput vectors. Step 4: Compute the norm of s: s s Step 5: Send nput sgnal from F1(a) to the F1(b) layer:. x s Step 6: For each F2 node that s not nhbted: If then y j b x. j Step 7: Whle reset s true, do Steps Step 8: Fnd J such that for all nodes j. If y j 1 yj y j then all nodes are nhbted and ths pattern cannot be clustered. Step 9: Recompute actvaton x of F1(b): x s t Step 10: Compute the norm of vector x: Step 11: Test for reset: f node J): f x s x s J x x y J then y 1 (nhbt then proceed to Step 12. Step 12: Update the weght for node J: Lx bj( new), tj( new) x L 1 x Step 13: Test for stoppng condton. 3. Expermental Results and Performance The effectveness of the system has been justfed usng dfferent hand movements and ssung commands to an entertanment robot named Abo. Experments were carred out wth an IntelI Core 5-CPU@2.70 GHz wth 4 GB RAM. The algorthm has been mplemented usng Vsual C++. The data set employed for tranng and testng the ASL sgn recognton system conssts of color mages. Fve samples for each ASL sgn were taken from fve dfferent users. For each sgn, three out of fve samples were employed for tranng purpose, whle the remanng two sgns were used for testng. The samples were captured from dfferent dstances by dgtal camera, and wth dfferent llumnaton condtons and algnments. J The performance of the system has been evaluated wth two parameters lke precson and recall. Precson s the fracton of the correct retreved ASL mages among the retreved nstances, whle recall s the fracton of relevant nstances that have been retreved over total relevant nstances n the mage. Both precson and recall are therefore based on an understandng and measure of relevance. The precson and recall are expressed by the followng equatons: TP Precson, (7) TP FP TP Recall, (8) TP FN where TP s the true postve, FP the false postve, FN s the false negatve, respectvely. The precson and recall results for dfferent ASL sgns are shown n Fg. 5. Obvously, the ASL for the characters C, E, M, N, O, P, Q and S performed poor results due to ther confusng recognton characterstcs. 100% 80% 60% 40% 20% 0% A C E G I K M O Q S U W Y Precson Recall Fg. 5 Precson and recall results for dfferent ASL sgns. The error versus teraton graph for the ART tranng process s shown n Fg. 6. Obvously, as the weghts are updated the error s reducng towards zero. Fnally, an entertanment robot Abo s beng controlled by means of commands drected by the ASL sgns. The snapshot for the nterface between ASL and AIBO robot s shown n Fg. 7. The popular command correspondng to ASL y, that s move backward and left s llustrated n ths fgure. The robot has several movements, such as: Move Forward, Move Backward, Turn left, Turn Rght, Lght on, Lght off and so on dependng on the ASL sgn languages F, B, L, R, W, E, respectvely. Some of the ASL sgns employed for

5 70 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July 2017 controllng the robot are lsted n Table 1. The entertanment robot, AIBO could follow the nstructons successfully. Error ASL Sgn F B L R O D V Y S M C W E Fg. 6 Error versus teraton for tranng the ART. Fg. 7 Snapshot for ASL controlled Robot movement. Table 1: Commands to control AIBO Robot Acton Move Forward Move Backward Turn Left Turn Rght Forward + rght Forward + left Backward + rght Backward + left Stop Load or Hde Mssle Chamber Fre Mssle Turn on Lght Turn off Lght 4. Conclusons Ths artcle presents mplementaton of ASL sgn recognton to control an entertanment robot. The work has been accomplshed by tranng a number of hand mages wth dfferent expressons for ASL. The system s capable of performng the dentfcaton of pattern wthout the need of any hand gloves. Images were captured at dfferent llumnaton condtons and the proposed fuzzy hstogram equalzaton algorthm was capable of equalzng the mages properly. The developed system proved to be robust aganst changes n gestures, poston, sze and drecton. Ths s because the extracted features method usng Affne transformaton to make the system translaton, scalng and rotaton nvarant. The proposed system was able to reach a recognton rate of more than 92.0% for tranng data and about 85% for testng data. Although ths research was nvolved wth statc sgns of ASL, but nevertheless, the dynamc sgns can also be detected for consecutve mage sequences through proper mappng of the feature vectors. Our future plan s to nclude mages wth non-skn color background and make the system sutable for real lfe applcatons, such as eye trackng, gaze drecton, facal expresson, and gesture recognton. References [1] 1. C. Chuan, E. Regna, and C. Guardno, " Amercan Sgn Language Recognton Usng Leap Moton Sensor, 13th Internatonal Conference on Machne Learnng and Applcatons (ICMLA), Detrot, MI, USA, pp , [2] C. Savur and F. Sahn, "Real-Tme Amercan Sgn Language Recognton System by Usng Surface EMG Sgnal", 14th IEEE Internatonal Conference on Machne Learnng and Applcatons, pp , [3] P. Pandey and V. Jan, "Hand Gesture Recognton for Sgn Language Recognton: A Revew", Internatonal Journal of Scence, Engneerng and Technology Research (IJSETR), Vol. 4, No. 3, pp , [4] N. Sarawate, M. Chan, and C. OZ, "A real-tme Amercan Sgn Language word recognton system based on neural networks and a probablstc model", Turksh Journal of Electrcal Engneerng & Computer Scences, Vol. 23, pp , [5] Md. Mohmnul Islam ; Sarah Sddqua ; Jawata Afnan, "Real tme Hand Gesture Recognton usng dfferent algorthms based on Amercan Sgn Language", IEEE Internatonal Conference on Imagng, Vson & Pattern Recognton (ICIVPR), pp. 1-6, [6] C. Vogler and D. Metaxas, Parallel hdden Markov models for Amercan sgn language recognton, Proceedngs of the Seventh IEEE Internatonal Conference on Computer Vson, pp , [7] K. Fok, N. Ganganath. C. Cheng and C. Tse, A Real-

6 IJCSNS Internatonal Journal of Computer Scence and Network Securty, VOL.17 No.7, July Tme ASL Recognton System Usng Leap Moton Sensors, Internatonal Conference on Cyber-Enabled Dstrbuted Computng and Knowledge Dscovery, pp , [8] C. Charayaphan and A. Marble, Image processng system for nterpretng moton n Amercan sgn language, Journal of Bomedcal Engneerng, Vol. 14, pp , [9] S. Fels and G. Hnton, GloveTalk: a neural network nterface between a DataGlove and a speech syntheszer, IEEE Transactons on Neural Networks, Vol. 4, pp. 2 8, [10] T. Starner and A. Pentland, Vsual recognton of Amercan sgn language usng hdden Markov models, Internatonal Workshop on Automatc Face and Gesture Recognton, Zurch, Swtzerland, pp , [11] K. Grobel and M. Assan, Isolated sgn language recognton usng hdden Markov models. In Proceedngs of the nternatonal conference of system, man and cybernetcs, pp , [12] R. Bowden and M. Sarhad, A non-lnear model of shape and moton for trackng fnger spelt Amercan sgn language, Image and Vson Computng, Vol. 9 10, pp , [13] Q. Munb,, M. Habeeb, B. Takrur, H.A. Malk, Amercan sgn language (ASL) recognton based on Hough transform and neural networks, Expert Systems wth Applcatons, Volume 32, Issue 1, pp , [14] C. Zhang, Y. Tan and M. Huenerfauth, "Multmodalty Amercan Sgn Language recognton, IEEE Internatonal Conference on Image Processng (ICIP), Phoenx, AZ, USA, pp , [15] M.A. Bhuyan and H. Hama, Identfcaton of Actors Drawn n Ukyoe Pctures, Pattern Recognton, Vol. 35, No. 1, pp , [16] R. C. Gonzalez and R. E. Woods, Dgtal Image Processng, Pearson Educaton Inc., 2 nd Edton, Delh, [17] M.A. Bhuyan, Toward Face Recognton usng Egneface, Internatonal Journal of Advanced Computer Scence & Applcatons, Vol. 7, No. 1, pp , [18] M.A. Bhuyan, V. Ampornaramveth, S. Muto, and H. Ueno, On Trackng of Eye for Human-Robot Interface, Internatonal Journal of Robotcs and Automaton, Vol. 19, No. 1, pp , [19] A. Khatun and A. Bhuyan, Neural Network based Face Recognton wth Gabor Flters, Internatonal Journal of Computer Scence and Network Securty, Vol. 11, No. 1, pp , [20] M.A. Bhuyan and C.H. Lu, Intellgent Vson system for Human-Robot Interface, Proc. Of World Academy of Scence, Engneerng and Technology, pp , [21] M.A. Bhuyan, On Gesture Recognton for Human- Robot Symboss, The 15 th IEEE Internatonal Symposum on Robot and Human Interactve Communcaton, ROMAN 2006, pp , [22] M.E. Islam, N. Begam, and M.A. Bhuyan, Vson system for human-robot nterface, 11 th Internatonal Conference on Computer and Informaton Technology (ICCIT), pp. 1-6, [23] S. Sharma and M. Varshney, An effcent approach for Web-log mnng usng ART, Internatonal Conference on Educaton, Management and Technology (ICEMT), 2010, pp , Dr. Md. Al-Amn Bhuyan receved both hs Bachelor and Master degree from Unversty of Dhaka, Bangladesh, n 1987 and 1988, respectvely. He got hs PhD degree from Osaka Cty Unversty, Japan n Currently he s a an Assocate Professor at the Department of Computer Engneerng, Kng Fasal Unversty, Saud Araba (under len leave from Jahangrnagar Unversty, Bangladesh where s employed as a Professor). Pror to ths, Dr. Bhuyan lent hs teachng and research experences at several Unverstes n Japan, Bangladesh and UK. Hs research nterests nclude mage processng, computer graphcs, pattern recognton, artfcal ntellgence, neural networks, robotc vson, and so on. He has publshed numerous artcles n Internatonal refereed journals and conference proceedngs.