A Real-Time Large Vocabulary Recognition System For Chinese Sign Language

Transcription

1 A Real-Time Large Vocabulary Recognition System For Chinese Sign Language Wang Chunli (1) GAO Wen (2) Ma Jiyong (2) (1) (Department of Computer, Dalian University of Technology, Dalian ) (2) (Institute of Computing Technology, Chinese Academy of Science, Beijing ) {chlwang, wgao, Abstract The major challenge that faces Sign Language recognition now is to develop methods that will scale well with increasing vocabulary size. In this paper, a real-time system designed for recognizing Chinese Sign Language (CSL) signs with a 5100 sign vocabulary is presented. The raw data are collected from two CyberGlove and a 3-D tracker. An algorithm based on geometrical analysis for purpose of extracting invariant feature to signer position is proposed. Then the worked data are presented as input to Hidden Markov Models (HMMs) for recognition. To improve recognition performance, some useful new ideas are proposed in design and implementation, including modifying the transferring probability, clustering the Gaussians and fast matching algorithm. Experiments show that techniques proposed in this paper are efficient on either recognition speed or recognition performance. 1 Introduction The hand gesture recognition that can contribute to a natural man-machine interface is still a challenging problem. Closely related to the field of gesture recognition is that of sign language recognition. Sign language, as a kind of structured gesture, is one of the most natural means of exchanging information for most deaf people. The aim of sign language recognition is to provide an efficient and accurate mechanism to transcribe sign language into text or speech. Attempts to automatically recognize sign language began to appear in the 90 s. Charaphayan and Marble [1] investigated a way using image processing to understand American Sign Language (ASL). This system can recognize correctly 27 of the 31 ASL symbols. Starner[2] reported a correct rate for 40 signs achieved 91.3% based on the image. The signers wear color gloves. By imposing a strict grammar on this system, the accuracy rates in excess of 99% were possible with real-time performance. Fels and Hinton[3][4] developed a system using a VPL DataGlove Mark II with a Polhemus tracker attached for position and orientation tracking as input devices. The neural network was employed for classifying hand gestures. Takahashi and Kishino[5] investigated a system for understanding the Japanese kana manual alphabets corresponding to 46 signs using a VPL dataglove. Their system

2 could correctly recognize 30 of the 46 signs. Y. Nam and K. Y. Wohn[6] used three dimensional data as input to HMMs for continuous recognition of a very small set of gestures. They introduced the concept of movement primes, which make up sequences of more complex movements. R. H. Liang and M. Ouhyoung[7] used HMM for continuous recognition of Tainwan Sign language with a vocabulary between 71 and 250 signs based Dataglove as input devices. Kisti Grobel and Marcell Assan [8] used HMMs to recognize isolated signs with 91.3% accuracy out of a 262- sign vocabulary. They extracted the features from video recordings of signers wearing colored gloves. C. Vogler and D. Metaxas[9] used HMMs for continuous ASL recognition with a vocabulary of 53 signs and a completely unconstrained sentence structure. C. Vogler and D. Metaxas[10][11] described an approach to continuous, whole-sentence ASL recognition that uses phonemes instead of whole signs as the basic units. They experimented with 22 words and achieved similar recognition rates with phoneme-based and word-based approaches. Wen Gao[12] proposed a Chinese Sign language recognition system with a vocabulary of 1064 signs. The recognition accuracy is about 93.2%. So far, the challenge that sign language recognition faces is to develop methods that will resolve well large vocabulary recognition. Chinese Sign Language (CSL) consists of about 5500 elementary vocabularies including postures and gestures. Therefore the task of CSL recognition becomes very challenging. We proposed a Chinese Sign language recognition system with a vocabulary of 1064 signs in [12]. An ANN-DP combined approach was employed for segmenting subwords automatically from the data stream of sign signals. To tackle the epenthesis movement problem, a DP-based method has been used to obtain the context-dependent models. The recognition accuracy is about 93.2%. This paper is an extension of the work done in [12]. Our goal is to realize the 5500-sign vocabulary CSL recognition. Two CyberGlove and a Pohelmus 3-D tracker with three receivers positioned on the wrist of CyberGlove and the waist are used as input device in this system. The raw gesture data include hand postures, positions and orientations. An algorithm based on geometrical analysis for purpose of extracting invariant feature to signer position is proposed in this paper. In large vocabulary continuous speech recognition, phonemes are used as the basic units; the number of them is much smaller compared with those in CSL. There are 50~60 phonemes in speech and about 2500~3000 basic units in sign language. Because there is not the description about all the basic units in CSL at hand and the possible number of them is quite large, we do not build HMMs for the basic units. A HMM is built for each sign in this system, and there are 5100 HMMs. The numbers of basic gestures included in different signs are variable, so these HMMs include different number of states. It is difficult to realize real-time recognition because of too much computational cost if CHMMs (Continuous HMMs) were employed in large vocabulary recognition. State Binding is proposed to reduce the computational cost, so the vectors on the states need to be classified. But the number of possible combinations of the six data streams in sign language could be approximately an order of If they were dealt with together, the code words will be too many. In this system, six codebooks are set for these six parts. Another key factor for speeding up the recognition procedure and reducing the memory resources is how to pruning unlikely hypothesis as soon as possible.

3 Selecting the candidate signs are used during the decoding process in this system. The experiments show that the signs can be recognized in real-time. The organization of this paper is as follows: Section 2 describes the outline of the designed system. Section 3 presents a feature extraction method of hand gesture. Section 4 discusses recognition approaches of sign language recognition. Section 5 demonstrates the performance evaluation of the proposed approaches. The summary and discussion are given in the last section. 2 System Architecture The architecture of this designed system is shown in Fig.1. The sign data collected by the gesture-input devices is fed into the feature extraction module, the output of feature vectors from the module is then input into the training module, in which a model is built for each sign. The number of the states of a model is set according to the transformation of the sign s signal. In order to decrease the computational cost, the Gaussians on the states are classified, then the index of the code word that is the nearest to the Gaussian on each state is recorded. The details are discussed in Sec.3. The search algorithm we used will be described in Sec.4. When the sign is output from the decoder, the sign drives the speech synthesis module to produce the voice of speech. Signal DataGlove Gesture Input Feature Extraction Dictionary Sign Decoder Speech Training CHMMs Clustering the Gaussians CodeBook CHMMs Fig.1. Sign Language Recognition. 3 Feature Extraction To calculate all gesture data from the left hand, right hand and body parts in a welldefined space, we need to consider the modeling of the relative 3D motion of three

4 receivers working with a transmitter. The timely motion of the transmitter will be also considered. 3D motion of receivers can be viewed as the rigid motion. It is well known that 3D displacement of a rigid object in the Cartesian coordinates can be modeled by an affine transformation as the following, X = R( X S) (1) where R is a 3 3 rotation matrix, cos β 0 sin β cosγ sinγ 0 R = 0 cosα sinα sinγ cosγ 0 0 sinα cosα sin β 0 cos β (2) cos β cosγ cos β sinγ sin β = sinα sin β cosγ cosα sinγ sinα sin β sinγ + cosα cosγ sinα cos β cosα sin β cosγ + sinα sin γ cosα sin β sinγ sinα cosγ cosα cos β = ( x, x x ) t and = ( x, x x ) t X 1 2, 3 X 1 2, 3 denote the coordinates of the transmitter and receiver respectively, S is the position vector of the receiver with respect to Cartesian coordinate systems of the transmitter. The receiver outputs the data of Eulerian angles, namely, α, β,γ, the angles of rotation about X, 1 X and X 2 3 axes. Normally these data cannot be used directly as the features because inconsistent reference might exist since the position of the transmitter might be changed between the processing of training and that of testing. Therefore, it is necessary to define a reference point so that the features are invariant wherever the positions of transmitter and receivers are changed. The idea we propose to fix this problem is as follows. There is a receiver on each hand, and the third receiver is mounted at a fixed position on the body, such as the waist or the back. Suppose that S, r S, and S are the position vectors of the receivers at right hand, left l t t hand and on the body. It is clear that the product RR, r RR, l R( S r S) and R( S l S) are invariant to the positions of the transmitter and the signer, where R, l R, and R r t are the rotation matrix of the receivers at right hand, left hand and on the body. R is r the transpose matrix of R that is the rotation matrix of the receiver at the right hand r respect to Cartesian coordinate systems of the transmitter. The raw gesture data, which in our system are obtained from 36 sensors on two datagloves, and three receivers mounted on the datagloves and the waist, are formed as 48-dimensional vector. A dynamic range concept is employed in our system for satisfying the requirement of using a tiny scale of data. The dynamic range of each element is different, and each element value is normalized to ensure its dynamic range Sign Language Recognition Hidden Markov Models (HMMs)[13] have been used successfully in speech recognition, handwriting recognition, etc. A HMM is a doubly stochastic state machine that has a Markov distribution associated with the transitions across various states, and a probability density function that models the output for every state. A key

5 assumption in stochastic gesture processing is that the signal is stationary over a short time interval. CHMMs is used in the system proposed in [12], which can recognize 1064 signs. But the system is not applicable to 5500-sign vocabulary because the computational cost is too much. In order to improve the performance of the system, some useful techniques are used CHMMs with Different Number of States In our system, a HMM is built for each sign. But the numbers of basic gestures included in different words are variable. Some signs are simple, such as chair (Fig.2(a)) and Room (Fig.2(b)). There is only one basic gesture in each one of these signs. It is enough to set 3 states in their HMMs. But some signs are complicated, such as Chinese idioms zuo-jing-guan-tian (Fig.2(c)) and hu-tou-shewei (Fig.2(d)). There are at least four basic gestures in these signs. (a) (b) (c) (d) Fig.2. (a) Chair ;(b) Room ; (c) zuo-jing-guan-tian (d) hu-tou-she-wei If the number of the states in HMMs is set to 3, each vector on the states in the complicated signs does not correspond to a gesture. The accuracy will be affected. On the other hand, if the number of the states in HMMs is set to the maximum number of the basic gestures included in one sign, the computational cost will be too much. The number of states in a HMM should be consistent with the number of basic gestures included in the sign. In our system, an approach based on dynamic programming is used to estimate the number of the states. There is a problem of matching of long signs and short ones if the number of states is different. The minimum number of states is 3 and the maximum number is 5, and a is the probability of transferring from the state i to j. For the long signs, ij a ij < 1 when i, j 3. For the short signs, a 33 = 1. a < 1 ij a 33 = 1 Fig. 3. The HMMs of long signs and short ones Given the observation sequence O = ( o1o2... ot ), the best state sequence is q = ( q1q2... qt ),where q1 is the initial state. The probability of O = ( o1o2... ot ) is obtained over the best state sequence q can be written as p O λ ) = π b ( o ) a b ( o )... a b ( o ) (3) ( q1 q1 1 q1q2 q2 2 qt 1 qt qt T

6 The interpretation of the computation in the above equation is the following. Initially (at time t = 1) we are in state q 1 with probability π q 1, and generate the symbol o 1 with probability b q ( o 1 ). The clock changes from time t to t + 1 and we 1 make a transition to state q 2 from state q 1 with probability a q 1 q 2, and generate the symbol o 2 with probability b q ( o 2 ). This process continues in this manner until the 2 last transition (at time T) from state qt 1 to state q T with probability aq T 1 q T. Because a ij < 1(i,j>3) for long sign, the decrease of the probability of long signs is larger than that of short signs. A long sign is easily recognized as the short sign that is similar to the later half of the long one. To match the long signs and short ones, the transferring probabilities a ij ( i, j 3 ) in long signs are set to 1, namely the transfers between the later states are set to null Clustering the Gaussians In systems that use a time-synchronous beam search, only a few percent of the states may be active at a given time, but even so, this implies a lot of Gaussian likelihoods need to be evaluated, which is still substantial. A common technique for a fast match for efficient Gaussian calculation is to pre-cluster all the Gaussians into a relatively small number of clusters. During recognition, the likelihoods of the clusters are first evaluated and ranked, and these clusters whose scores are higher than the threshold are set active. Only those Gaussians corresponding to the active clusters are evaluated. The amount of computation that can be saved is good-sized. There are many variations for computing the clusters, for example, VQ-based methods, and hyperplane-based methods. In our system, the vector is consisted of the left hand configuration, position, orientation, right hand configuration, position and orientation. Because the number of possible combinations of these six parts can be approximately 10 8, in this system, hand configuration, position and orientation of the mean vectors in Gaussians are classified respectively to reduce the number of codewords. There are six streams totally. As the probability in log domain can be computed by the summation of all the streams probabilities in log domain as follows: 6 log p ( x) = log p l ( x l ) (4) l= 1 where p (x) is the Gaussian probability, and p l ( x l ) is the observation probability of the l th data stream. The clustering algorithm used here is k-means[13]. For each data stream, the stream state probabilities are clustered. The probability of a class needs to be computed only once. If p l ( x l ) is belong to the i th class, the probability of p l ( x l ) is equal to that of the i th class. As the number of distinguishable patterns in each data stream is relatively small, for a given observation vector, after these six observation probabilities have been computed, the log likelihood of each signs can be easily gotten by a lookup table and by 5 times addition operations. For the case of continuous sign recognition, because the computation time for the state observation probabilities is relatively small, the probably active model candidates can be quickly determined. The likelihood computation can be reduced by a factor of 10 or more with minimal loss in accuracy.

7 4.3 Fast Matching Algorithms Given the observation sequence O = ( o1o2... ot ) and all models λ = { λ1, λ2, m, λv }(V is the number of the models), compute the probability of each model λv ouput the observation sequence. * P( O ) P( O, Q ) λ = λ (5) v * where Q is the best state sequence. Then find the sign according to the following expression: v * = arg max P( O λ ) 1 v V In order to find v *, the probabilities of all the models need to be calculated. Viterbi can be used. Viterbi[13] search and its variant forms belong to a class of breadth-first search techniques. There are more than 5000 models, so the computational cost is still large even if clustering the Gaussian is used. In order to conserve the computing and memory resources, the fast matching method is proposed in time-synchronous search. The idea is as followed. For the first several frames, the probabilities of all the codeword of each stream data are calculated. For each stream, a threshold according to the maximum one is set to select the active codeword. If the probability of one codeword is larger than the threshold, the codeword is set active, otherwise inactive. For each HMM, if there is a state on which the six codewords are active and the probability that the sign begins with this state is bigger than a threshold, this model is set active. Only the probabilities P( O λ v ),in which λ v is an active model, need to be calculated. From these probabilities, the model that has the maximum probability is selected, and the sign corresponding to this model is the recognition result. v v (6) 5 Experiment The hardware environment is Pentium III 700MHz, with two CyberGloves and three receivers of 3D tracker; each Cyberglove is with 18 sensors. The baud rate for both CyberGlove and 3D tracker is set to signs in Chinese sign language are used as evaluation vocabularies. Each sign was performed five times, four times are used for training and one for testing. The minimum number of states in HMM of each sign is 3. The maximum number of states in HMM of each sign is 3~7. The recognition rates with different maximum number of states are shown in Table.1. According to Table.1, when the maximum number of states is 5, the best recognition accuracy is 95.0% and the average number of states is only This result is very encouraging.

8 Table.1. The recognition rates with different maximum number of states The maximum number of states Average number of states Rate % % % % % The above result is with modifying the transferring probability. The comparison is given in Table 2. The recognition rate with modifying the transferring probability is 95% and the recognition rate without modifying the transferring probability is 93.8%. The results show that modifying the transferring probability has effect on the recognition rate. Table.2. The recognition rates of large vocabulary signs (5100 signs) Without modifying the transferring probability 93.8% With modifying the transferring probability 95.0% The recognition rates of isolated signs with different numbers of the codewords are shown in Table3. The test data are collected when the signs were recognized online. Table.3. The recognition rates of signs for different numbers of codewords The number of codewords Right position Right orientation Left position Left orientation Left hand shape Right hand shape The recognition rates % % % % % % According to Table 3, considering both the accuracy and the speed, the numbers of codewords of right position, right orientation, left position, left orientation, left handshape and right handshape are set to 128, 128, 128, 128, 350, 350. It spends no more than 1 second to recognition a sign online and the recognition rate is more than 90%. Real-time recognition has been realized.

9 6 Conclusion In this paper, a real-time CSL recognition system based on large vocabulary is presented using HMM based technology. Our contributions within this system are three aspects: modified the transition probability, state binding and search algorithm. Experimental results have shown that the proposed techniques are capable of improving both the recognition performance and speed. This system is for signer dependent isolated signs recognition task. The system will be extended for the task of continuous and signer independent recognition in the future. 7 Acknowledgment This research is sponsored partly by Natural Science Foundation of China (No ), National Hi-Tech Program of China (No ZD ), and 100 Talents Foundation of Chinese Academy of Sciences. References 1. C. Charayaphan, A. Marble: Image processing system for interpreting motion in American Sign Language. Journal of Biomedical Engineering, 14(1992) T. Starner : Visual recognition of American Sign Language using hidden Markov models. Master's thesis, MIT Media Laboratory, July S. S. Fels, G. Hinton: GloveTalk:A neural network interface between a DataDlove and a speech synthesizer. IEEE Transactions on Neural Networks 4(1993) S. Sidney Fels: Glove TalkII: Mapping hand gestures to speech using neural networks- An approach to building adaptive interfaces. PhD thesis, Computer Science Department, University of Torono, Tomoichi Takahashi, Fumio Kishino: Gesture coding based in experiments with a hand gesture interface device. SIGCHI Bulletin (1991) 23(2) Yanghee Nam, K. Y. Wohn: Recognition of space-time hand-gestures using hidden Markov model. To appear in ACM Symposium on Virtual Reality Software and Technology (1996). 7. R. H. Liang, M. Ouhyoung: A real-time continuous gesture recognition system for sign language. In Proceeding of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan (1998) Kirsti Grobel, Marcell Assan: Isolated sign language recognition using hidden Markov models. In Proceedings of the International Conference of System,Man and Cybernetics (1996) Christian Vogler, Dimitris Metaxas: Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods. In Proceedings of the IEEE International Confference on Systems, Man and Cybernetics, Orlando, FL (1997) Christian Vogler, Dimitris Metaxas: ASL recognition based on a coupling between HMMs and 3D motion analysis. In Proceedings of the IEEE International Conference on Computer Vision, Mumbai, India (1998)

10 11. Christian Vogler, Dimitris Metaxas: Toward scalability in ASL Recognition: Breaking Down Signs into Phonemes. In Proceedings of Gesture Workshop, Gif-sur-Yvette, France (1999) Wen Gao, Jiyong Ma, Jiangqin Wu, Chunli Wang: Large Vocabulary Sign Language Recognition Based on HMM/ANN/DP. International Journal of Pattern Recognition and Artificial Intelligence, Vol. 14, No. 5 (2000) L Rabiner, B. Juang: Fundamentals of Speech Recognition. Publishing Company of TsingHua University.