3. MANUAL ALPHABET RECOGNITION STSTM

Transcription

1 Proceedings of the IIEEJ Image Electronics and Visual Computing Workshop 2012 Kuching, Malaysia, November 21-24, 2012 JAPANESE MANUAL ALPHABET RECOGNITION FROM STILL IMAGES USING A NEURAL NETWORK MODEL Makoto J. Hirayama and Masahiro Funakawa Kanazawa Institute of Technology ABSTRACT Japanese manual alphabet is used mainly by hearing impaired persons as complements of sign language. Using neural network model, a Japanese manual alphabet recognition system is implemented. Still images of finger shapes from 2 directions are used as inputs of a multi-layer perceptron and the classification of the alphabet is the output. Overall performance of recognition is around 88% in average, although they varied depending on evaluation conditions. Keywords: Japanese manual alphabet, sign language recognition, image processing, neural network model 1. INTRODUCTION To consider reasonable accommodation to persons with disability in aural speech communication, preparing accessible environment is duty on any kind of places. For hearing impairment, communication aids to assist communication between hearing impaired and hearing persons should be supplied. To show information contents, accessible presentation should be provided. Convention of the rights of persons with disabilities [1] was carried by United Nations in 2006, although Japan has not ratified so far, yet. It prohibits discrimination by disabilities and supports accessibility and participation of persons with disabilities. It is necessary to supply communication assists. To enable communication assists, technologies for communication aids must be studied an developed. In Japan, about 300 thousand hearing impaired persons in terms of counts by the ministry of welfare [2]. About 50 thousand persons are said to use sign language [3] as their usual communication method. Although supplying information and communication in sign language should be guaranteed for these people, it is difficult for most of Japanese speakers to use sign language. Therefore, we are considering making a sign language interpreter device based on sign language recognition in future. As the initial step of Japanese sign language interpreting technology, we research into Japanese manual alphabet recognition. A method for recognizing Japanese manual alphabet from images captured by two web cameras is studied to be applied to developing wellbeing system for assisting communication between hearing impaired persons. Some pattern recognition algorithms of Japanese manual alphabet have been proposed so far. For example, Watanabe et. al. [4] proposed a recognition method using color glove and single camera capturing. Mitome et. al. [5] proposed a fast speed recognition method for ten numeral finger shapes using single camera image. Some other methods use special devices such as 3D scanners [6], glove devices [7] [8], thermal sensors, etc. Also, algorithm for recognition is discussed [9] [10] [11] [12] [13] [14]. However, they have not generally completed yet. In this paper, hand areas are extracted from the original images from web cameras using color values of human skin [15]. Silhouette images of the hand representing by black and white binary format are inputs to three layer perceptron type neural network, then the Japanese manual alphabet syllable is the output. For some alphabet characters expressed by manual motions, sequential multiple still images are used as an additional information for classification. 2. JAPANESE MANUAL ALPHABET DATA 2.1. Japanese Manual Alphabet Japanese manual alphabet, which is also called as finger alphabet or finger spelling, is usually used with sign language, for expressing spoken or written Japanese language representation such as names of human, places, etc. which cannot be expressed in words in sign language. The current Japanese finger alphabet was defined by G. Osone, the principal of Osaka city deaf school, in 1931 [16], by modifying and extending American manual alphabet. Each finger alphabet character is corresponding to Japanese Kana which is syllabic unit of Japanese. The number of Japanese basic syllabic units is 46 (it is so called 50-On pronounced as Gojyu-On ), but some additional symbols are used with the basic units to express additional syllabic pronunciation symbols ( Dakuten, Han-Dakuten, Yo-On, Cho-On in Japanese). Many of Japanese finger alphabet characters are expressed by shapes of one hand without motion, but some characters and all pronunciation symbols are expressed with motion. Fig. 1 shows examples of these shapes. Only 20 finger alphabet characters, "a, i, u, e, o, ka, ki, ku, ke, ko, sa,

2 shi, su, se, so, ta, chi, tu, te, to" are shown out of 46 basic characters. The input image data to the recognition system reported in this paper are similar to them in Fig MANUAL ALPHABET RECOGNITION STSTM 3.1. System Overview Fig. 3 shows a schematic diagram of Japanese finger alphabet recognition system that was developed. The input of the system is images of motion video or still pictures from web cameras. The output of the system is a recognition result, that is, a Japanese Kana character text. The output may be connected to speech synthesis system to hear the textual sequence. Hand images from web camera Image preprocessing (Hand region extraction, Noise reduction, Binary silhouetted imaging) Fig. 1. An example of Japanese manual alphabet, "a, i, u, e, o, ka, ki, ku, ke, ko, sa, si, su, se, so." 2.2. Recognition Objects The recognition objects are 41 Japanese manual alphabet characters without motion, 4 characters with motion (/NO/, /MO/, /RI/, and /NN/), and 34 arm motions for voiced syllables, explosive syllables, palatal syllables, and long sounding marks ("Daku-on," "Handaku-on," "Yo-on," and "Cho-on" in Japanese). For expressing voiced, explosive, palatal syllables or long sounding marks, hand positions are moved by arm, from center to right, from low to up, from front to back, or up to low, all without hand shape changes during the movements. The manual alphabet images are captured by using two web cameras. The best two positions founded by previously reported camera placement study [17] [18] are used as the positions of two cameras. Fig. 2 shows example views from two web cameras. Long sleeve black or dark blue shirt was worn by subjects for the experiment for easy hand area extraction by using chroma-keying. Also, single colored back paper is used for background for easier image extractions. (a) Front view (b) Side view Fig. 2. Views from two web cameras. Characteristic parameters caculation Learned recognition engine (Neural network model) Shrinken binary image data Recognition results (Manual alphabet classification) Fig. 3. Schematic diagram of Japanese manual alphabet recognition system At first, as shown in Fig. 3, image preprocessing is done to the input image from web cameras. The most important processing at this stage is hand region extraction. After the processing, binary silhouetted image of hand is made. Then, this preprocessed data are further changed by 2 types of processing. One is characteristic parameters calculation and another is binary silhouetted image data adjustment. They will be inputs to the recognition engine.

3 As the recognition engine, a neural network model of three layer perceptron is used. This type of neural network model is often used for image recognition. Finally, the recognition engine outputs recognition results as Japanese Kana text characters. In the current system, the recognition is based on still images. Information obtained from video image in time domain is not used. Manual alphabet with motion and pronunciation symbols are recognized as sequences of still images, for example, 3 images of start, mid, and end points are recognized independently, as parts of one character. More detailed algorithms of the system are explained in the following sections Image Preprocessing Using human skin color detection in the RGB color space [15], a hand area is extracted in each picture. Simply, values are in relation of R > G > B, the pixel is a candidate of the hand area. After this extraction, using R values, the color image is converted to a gray scale image, then, using threshold value, it is converted to a black and white binary image. Then, using the Feret's diameter, a hand region is extracted Characteristic Parameters Calculation and Shrunken Binary Image Data Preparation Extracted hand region binary image data described above as image preprocessing is the base of characteristic parameters calculation. The left figure in Fig. 4 is an example. The data are further changed by 2 types of processing. One is characteristic parameters calculation and another is binary silhouetted image data adjustment. They will be inputs to the recognition engine. (2) Height and width ratio H A = H + W, where H denotes height and W denotes width. (3) Ratio of areas of silhouette to Feret's diameter D S = H W Then, the image is shrunken to a 25 x 25 pixel square for normalization of data used for recognition. Right figure in Fig. 4 is a shrunken image. Parameters (4) through (5) are calculated from this shrunken image. And (6) is 25 x 25 = 625 binary data themselves. (4) Ratio of areas of silhouette and square of left, leftcenter, center, right-center, and right regions (5 parameters per image). Fig. 5 shows these areas. A B C D Fig. 5. Black and white ratios within vertically splitted regions (5) Ratio of areas of silhouette and square of up, upmiddle, middle, middle-low, low (5 parameters per image). Fig. 6 shows these areas. E H L S shrink 25 pixel 25 pixel W Fig. 4. Shrinking black and white binary image into 25 x 25 pixel image. At first, parameters (1) through (3) are calculated using the original image before shriking. (1) Roundness 4π S C = 2 L, where S denotes area square of the silhouette and L denotes perimeter. Fig. 6. Black and white ratios within horizontally splitted regions (6) Binary pixel values (25 x 25 = 625 values per image) as shown in Fig. 7. Fig. 7. Binary pixel values (25 x 25 = 625)

4 3.4. Neural Network Model Hand shape recognition from still image is done using a three layer perceptron neural network model. For the inputs, parameters (6), that is, 625 binary pixel values, are mainly used. In addition, parameters (1) through (5) are added as input, as better performances were obtained in the experiment. The output of the network is the result of characters from /A/, /I/, /U/, /E/, /O/,, and /NN/ (41 units). The output unit of highest value is taken as the recognition result. 300 hidden layer units were used for the final evaluation, since it had the best performance in many trials of changing numbers of hidden layer units. For the simulation, MATLAB software with neural network toolbox is used Pixel Values Hidden Output Fig. 8. Three layer perceptron type neural netowrk 3.5. Hand Motion Tracking For the hand motion recognition, time series of center of mass of the silhouette in the binary images are tracked. By using information of the movements with the recognition results of still hand shape image, manual alphabet recognition including pronunciation symbols (i. e., Dakuten, Han-Dakuten, Yo-On, Cho-On in Japanese). So far, segmentation of character boundaries has not been done automatically yet and the time series of one character is supplied by human operation. Center of mass of the silhouette is calculated both for front and side views. The position representation is based on whole area of the web camera frame. From the sequence of the center of mass, movement is detected either as (1) from center to left in web camera frame for voiced syllables, (2) from up to down for long sounding marks, (3) from down to up for plosive syllables, or (4) from front to back for palatalized syllables. Fig. 9 shows the center of mass of silhouettes and movement direction axises. By adding information for the movement, the still image recognition results are separated. For example, still image of Japanese "KA" manual character with center to left movement expressing "Dakuten" is recognized as "GA". A I NN Fig. 9. Center of masses of the hand shape silhouettes and movement directions 4. SIMULATION RESULTS AND DISCUSSION The overall performance of the recognition from still images of hand pictures from 10 subjects is 88% correct in average. As the movement tracking itself is 100% correct by using simple method of subtraction of starting and ending positions, although segmentation of character boundary is not automatically but manually. The overall performance of the recognition for voiced syllables, plosive syllables, palatalized syllables, and long sounding marks, including both shape and motion recognitions are 88%, 95%, 90%, and 95% correct respectively. Using the proposed method, many Japanese manual alphabet characters can be recognized. However, higher recognition rate is needed for applying real situations. Some Japanese manual alphabet characters are very similar. For example, silhouettes of "U" and "RA" are almost same except for the index finger angle. Some additional algorithm may be required to be added for classification of similar shapes. Neural network model looked perform good job for the recognition of the current data set, however, trying other recognition algorithms or fine tuning of the current recognition engine is one thing to be done. Silhouettes of "I" and "CHI" are similar. In the initial study using one digital camera, recognition of "I" and "CHI" were around 50% so that proposed two web cameras method was good as recognition was drastically improved. However, from the viewpoint of real application, two cameras are hard to be installed into consumer personal computers or smart phone like devices. We still want to try one camera method with several enhancements, too. For motion detection, automatic tracking of objects and automatic segmentation of character boundaries, not included in this paper, should be considered in near future. The current data is lab experimentation setting data, which use single color background, stable lighting, stable camera placing, and clear behavior of Japanese manual alphabet. For the real application, robust methods for real situations and real time execution are needed, so that, serious system development with several improved algorithms and methods will be done.

5 5. CONCLUSION Algorithm and implementation of Japanese manual alphabet recognition, so far, was explained. To improve recognition results, we are trying some improvements to the recognition algorithm. When reasonable recognition rates were obtained after the improvements, applications for communication support between hearing and hearing impaired peoples will be planned to be developed. REFERENCES [15] D. Chai, K. N. Ngan: Locating facial region of a headand-shoulders color image, Proc. 3rd Int. Conf. on Automatic Face and Gesture Recognition, pp , [16] National Center of Sign Language Education: New Sign Language Classroom Introductory, Japanese Federation of Deaf, (In Japanese) [17] F. Kimoto, M. Funakawa, M. J. Hirayama, A studio of recognition method and camera position for finger spelling using two silhouette images, 71th IPSJ Annual Conference, 4T-8, pp. (2) , (In Japanese) [18] M. J. Hirayama, M. Funakawa, "Finger alphabet recognition from still images using neural network," 72th IPSJ Annual Conferece, 1D-1, pp. (2) 13-14, (In Japanese) [1] United Nations, Convention on the rights of persons with disabilities, [2] Ministry of Health, Labor and Welfare: Health, Labor and Welfare Whitepaper, Ministry of Health, Labor and Welfare, (In Japanese) [3] Japanese Federation of Deaf: New Sign Language Handbook, Sanseido:Tokyo, (In Japanese) [4] K. Watanabe, Y. Iwai, Y. Yagi, M. Yachida: Gesture Recognition Using Color Gloves, J. IEICE D-II, Vol. J80-D-II, No. 10, pp , (In Japanese) [5] A. Mitome, K. Ichige, R. Ishii: Hand Shape Analysis and Recognition by Masking and Standardization of Extracted Hand Images, J. IEICE D, Vol. J89-D, No. 6, pp , (In Japanese) [6] Y. Wang, S. Itai, S. Ono, S. Nakayama: Human recognition with ear image by principal component analysis, J. Japan Society of Information and Knowledge, 16(1), pp , (In Japanese) [7] K. Tabata, T. Kuroda: Improvement for finger alphabet recognition using distinctive features, Proceeding of the 34th Japan Sign Language Society Convention, pp. 5-6, (In Japanese) [8] M. Osato, M. Suzuki, A. Ito, S. Makino: Feature value combination for finger character recognition using a color glove, IEICE Technical Report, PRMU , pp , (In Japanese) [9] Y. Hamada, N. Shimada, Y. Shirai: Hand shape estimation using sequence of multiple viewpoint images based on transition network, J. IEICE D-II, Vol. J-85-D-II, No. 8, pp , (In Japanese) [10] A. Imai, N. Shimada, Y. Shirai,: 3-D hand posture recognition by learning contour variation, J. IEICE D-I, Vol. J88-D-II, No. 8, pp , (In Japanese) [11] T. Morozumi: Recognition of manual kana using characteristics in Images, Takushoku University Technical Report, Vol. 9, No. 2, (In Japanese) [12] S. Iwasaki, T. Asakura, K. Hirose: Recognition of finger spelling using neural network, Japan Society of Mechanical Engineers Hokuriku-Shinetsu branch 34th convention, pp , (In Japanese) [13] M. Osato, M. Suzuki, A. Ito, S. Makino: An interpolation method of the feature vector for finger character recognition, 2006 IEICE Convention, A-19-15, p. 333, (In Japanese) [14] D. Hara, Y. Nagashima, A. Ichikawa, K. Kanda, M. Terauchi, K. Morimoto, Y. Shirai, Y. Horiuchi, K. Nakazono: The development of sindex V.3: Collaboration between engineering and linguistics in abstracting distinctive features of Japanese sign, Human Interface Symposium 2007, pp , (In Japanese)