Auslan Sign Recognition Using Computers and Gloves

Transcription

1 Auslan Sign Recognition Using Computers and Gloves Mohammed Waleed Kadous School of Computer Science and Engineering University of New South Wales Abstract Wouldn t it be great if computers could understand sign language (and Auslan in particular)? This would open the door for some interesting applications for both Deaf and non-deaf people. We are seeing the development of interesting technologies for speech recognition, but no real commercial products for sign recognition. There are a number of commercial reasons for this (like the size of the market); but there are also significant technical difficulties, mainly in two areas: getting the data of body movements into the computer and trying to learn to recognise signs once they are made. However, researchers have been attacking the problem for some time, and the results are showing some promise, though it is still early days. This paper presents a general overview of research in this area, before going into greater depth about the author s current and ongoing research with Auslan recognition using instrumented gloves. In particular, the work focuses on developing techniques for helping computers tell different signs apart. Keywords: gesture recognition, sign language recognition, Auslan, machine learning. 1. Introduction In the last two decades, computers have been accepted as part of people s lives. Recent times have seen the increasing popularity of new modes of interaction with computers, such as sound, video, graphics and recently speech. However, one mode of interaction that remains largely unexplored is gesture, and consequently sign language. Interaction implies both input that is, watching a human sign and deciding what it is they are signing and output that is, getting the computer to sign back to the user. In this work, we are interested in the input stage. However, there are two fundamental problems with getting computers to accept input from signers: Collecting the data as to their movements and gestures. Processing the input and trying to extract what gestures they have made. In the remainder of this paper, we discuss why this area is so interesting, followed by a discussion of some of the practical issues and difficulties in this area. Other research in the area is also explored. The research of the author is presented, both past and current. Finally, some ideas about where the field is going and potential future improvements are considered. 1.1 Motivation A very important question is whether it is worthwhile to have such systems that understand sign what could we do with such a system? Some potential applications include: Educational purposes: Signs could be performed then matched against a dictionary. Right now, finding a sign in an Auslan dictionary can be difficult. But if you could make a sign and then have the computer look it up for you. It could also be used as an educational tool; so, for example the computer could "test" you by asking you to make a particular sign, then seeing if you made the right sign. It would also allow recording of signs and even monologues of sign. Improved user interface for the Deaf: Speech recognition is currently being built into new operating systems and will become an important part of the user experience; making it easier for speakers and computers to interact. Where does this leave the Deaf? Imagine if you could sign at the computer, and it could also sign messages back at you. This would allow the Deaf to have a more intuitive interaction with the computer. Portable translation: If the system could be made to be portable (which is looking possible now with the proliferation of hand-held computers); then a system could be built that empowered signers to interact more naturally with non-signers. Improved inter-signer telecommunications: It would be possible to capture a person's movements, transmit

2 them through the phone system or over the Internet to another person, who could then sign back. This may provide a more intuitive and appealing interface to people than the current TTYs. These are just some of the applications possible. Some (such as the the first) can be easily achieved today; others (such as the last) may be decades away. 2. Practical Issues However, before we can produce applications like those discussed in section 1, there are some practical problems and difficulties we must overcome. So we must consider what the possibilities are, what has been accomplished and what remains to be done. 2.1 Levels of development It is possible to partition the development of technologies for sign language recognition into five stages: (i) Recording, storing or transmitting the movements someone makes when signing, without doing any sort of recognition of the data. (ii) Doing some basic interpretation of a person s movements, such as recognising fingerspelling. (iii) Doing more complicated motion recognition to be able to recognise individual signs when performed separately, but without any higher level representation, such as sentences, grammar, etc. (iv) Being able to recognise fully signed Auslan, including facial gestures as well as hand movements, in sentence units and beyond. (v) Being able to translate from somebody s Auslan signing into another language (such as English), in real time. At this point in time, research is basically at level (iii). Commercial systems exist for (i) and (ii), though in some cases they are expensive. As mentioned in the introduction, right now there are two key technical difficulties. 2.2 Capturing the data Currently, there are two main ways of getting data about the signer s movements one is from video sources (eg. cameras) and the other is by using instrumented gloves or other direct interface devices. Each has its own advantages and disadvantages. Direct interface devices are attached directly to the signer s hand and body. The most common example for sign recognition is an instrumented glove combined with a position tracker. An instrumented glove contains sensors that measure finger bending and other movements of the fingers and palm. This information is then sent back to the computer at regular intervals. A position tracker is a magnetic or ultrasonic device that is attached to the hands which reports the position and orientation of each hand. Again, the information is sent back to the computer at regular intervals. The advantages of the device-based approach are that the data is relatively accurate and does not require much computing power to handle. The main disadvantages are that it encumbers the user, and it can sometimes take a few minutes to set up the gloves and that it doesn t allow for facial gesture recognition. Vision-based approaches use cameras to take images of the user at regular intervals. From this image, they extract the user s motion. The main advantage of vision-based analysis is that it allows the user to remain unencumbered. The main disadvantages are that existing vision systems require very complex computations to be performed on the images to extract usable information about a person s gestures. Furthermore, they are extremely sensitive to lighting effects, how far away the person is sitting and so on. It is also difficult from camera images to extract information about what the fingers are doing, especially if one hand is in the vicinity of the other. Furthermore, extracting any information about movement towards or away from the camera is also difficult. In addition, vision systems are very costly, not just because of the cameras, but because of the power of the computer required to handle the images. For these reasons, the current research uses a devicebased approach. 2.3 Recognising the data In some simple cases, for example, fingerspelling, it is not too difficult to manually build a system that can recognise finger positions. However, as soon as you try to go a little bit further and recognise signs that involve complex movement, it becomes far more difficult. Instances of the same sign can vary significantly, between signers, and even for a given signer depending on the time of day, their mood, how emphatic they are about what they are signing and so on. This makes it very difficult to write programs that recognise individual signs. The lexicon of possible signs is also far too large to make this approach feasible. Furthermore, any system that is preprogrammed to recognise signs would not be able to add new signs to its vocabulary, since to add a new sign a new way of recognising it would have to be written. Since signers use different sets of signs, this is probably not practical. For this reason, many researchers believe that the only way to get around this limitation is to enable sign recognition systems to learn from examples presented

3 to them. Getting machines to learn from examples, however, is extremely difficult, especially when, as in sign language, the data varies over time. 3. Previous Work This area as a whole is very new. Clearly, space in this document is insufficient for an in-depth analysis; the interested reader is encouraged to read [Kad95]. The earliest attempts at sign recognition began with fingerspelling. Dr G Grimes at AT & T Bell Labs developed the Digital Data Entry Glove [Gri83] which was designed for alphanumeric input. While it did not directly recognise the standard fingerspelling alphabet, it did recognise a dialect that was easier to recognise with the sensors on the glove. Recognition was accomplished by using preset thresholds for the various hand positions. Charayaphan and Marble [CM92] developed a visual system to extract movements of 31 Auslan signs manually, then proceeded to try to work out positions with very simple algorithms. They collected one example of each of the 31 signs and tried to hand-code ways of recognising each sign. They could recognise 27 out of the 31 using relatively simple techniques, based on their approximation on how the signs would vary. Davis and Shah [DS92] learnt a simple set of seven signs using cameras focused on a hand that had markings on the tip of each finger. However, it required explicit stop and start signals, and could only understand seven simple signs which it was programmed to recognise. Perhaps the most interesting work in this area was Thad Starner s Masters Thesis [Sta95] in recognising American Sign Language. He used a camera looking at a signer who wore two coloured gloves: an orange glove on the left hand and a yellow glove on the right hand. Approximately 40 signs were chosen and data of over 500 sentences were captured, each sentence containing approximately 4 signs. The camera was then connected to a machine with special hardware for processing video that would extract the position information about each hand. They used hidden Markov models to learn to recognise the different possible signs. Hidden Markov models are also popular in the speech recognition community and have proved very effective if you have sufficient data to train them effectively. The system worked quite well and produced a raw classification rate of approximately 95 per cent. Later on, the work continued, this time without using coloured gloves. The performance suffered, falling to an accuracy of approximately 90 per cent correct. However, this accuracy rate could be improved by using grammars for the sentences. If these grammars were sufficiently strict, much higher accuracy rates were obtained, in excess of 99 per cent. The grammar was of the form pronoun, verb, noun, adjective, pronoun, with pronouns and adjectives possibly being empty. The system would probably be considered the state of the art. However, there are questions with regards to whether it scales up to vocabularies of larger than 40 signs, for a number of reasons. The words were selected to make it simple to construct a sizable family of sentence from. The words were also selected so that they could be discretely classified as nouns, verbs or adjectives; whereas with many signs, they are not so easily classified. Also, the system did not make use of any information about the fingers; it simply treated each hand as a blob. This may have worked for the set of signs that were selected, but won t work in general, where the difference between two signs can be very subtle (eg. hearing and deaf and dumb in Auslan, which differ only in the position of the middle finger). Dorner and Hagen [DH94] began work on a sign recognition system. Dorner used special gloves with rings of colour around each joint to simplify the visual recognition task. Images of the glove were taken by a camera. Each joint had a different set of rings about it. By using a camera to capture this information, it could then be analysed to find out hand position information. Hagen also developed a deductive database of American Sign Language (ASL) that had a lot of information about structure and grammar of ASL. However, the intermediate work, between the glove and the deductive database was never completed. Fels [Fel94] took a completely different approach. Rather than trying to recognise signs, he instead created an adaptive interface that mirrored the human vocal system. By wearing a glove that the user moves in certain ways, users would learn to generate vocal sounds. At the same time, the computer would also modify its behaviour to improve the quality of the sounds made over time, using a neural network. Takahashi and Kishino [TK91] used the VPL dataglove to recognise the Kana alphabet of Japanese Sign Language, which consists of 46 signs. They succeeded in recognising 30 out of the 46 signs reliably. The other 16 could not be recognised reliably because they involved movement, and also because detecting fingertouching proved to be quite difficult. Murakami and Taguchi [MT91] also did some work on recognising signs using another technique called recurrent neural networks. Again, they used a VPL Dataglove. They succeeded in recognising the manual alphabet with approximately 98 per cent accuracy. They also tried to recognise moving signs, by selecting a set of 10 signs, which they learnt with approximately 96 per cent accuracy.

4 James Kramer [Kra90] developed the Talking Glove project. This system recognises ASL fingerspelling with high levels of accuracy. To accomplish this, James Kramer developed his own glove, called the CyberGlove. Although costly (about USD6000), it is the best glove available. Kramer went on to form Virtual Technologies, which sells a number of commercial products for fingerspelling recognition. Unfortunately, such systems are relatively costly. The fingerspelling system, called GesturePlus, costs approximately USD3500, not including the CyberGlove and the computer required to run it. Peter Vamplew [Vam96] used a single CyberGlove with position tracking. He successfully recognised fingerspelling and also attempted to recognise sign language. With a vocabulary of 52 signs, an accuracy of 94 per cent is achieved on seen signers and an accuracy of 85 per cent on unseen signers. 4. Authors Work 4.1 Honours Thesis work In 1995, the author did research on recognising Auslan using Instrumented Gloves [Kad95]. The research used a very primitive glove, called the Nintendo PowerGlove. This glove was originally designed for playing computer games. It came only in a right hand model. Position is sensed using ultrasound beeps emitted from the glove and picked up by three nearby receivers and measuring how long it took for the sound to get to the receivers. Using ultrasound introduces many errors, especially when the hand is pointing away from the receivers. Finger bend was measured using bend sensors, but only on the first four fingers and only with limited accuracy. However, the gloves are relatively cheap, at approximately AUD80 each. Examples of 95 signs that covered the gamut of handshapes, single, double and two-handed signs were collected from five signers one generous volunteer, one researcher, two professional interpreters and the author. A total of 6,500 signs were collected using the glove. These signs were collected so that each sign was separate, not part of a sentence. From the data, a set of features was extracted. These features covered things like: Duration. The bounding box. Average position over the first quarter of the sign Histograms of the different types of movements that happened. A concept learner is then used to learn descriptions of the different signs. Concept learning works as follows: you are given examples (instances) of different types of things (classes), and you have to come up with a way to tell them apart (a classifier). Sometimes, you are given hints as well about how to tell them apart (this is called background knowledge). For example, I might give a concept learner a description of different types of flowers in terms of petal colour, stem length, number of petals and so on, along with the species of flower. The concept learner would produce a classifier, such that if I gave it the description of the flower's features, it would be able to guess the species. In this case, for every example sign, we give it the features we extracted above, together with the name of the sign. It produces a classifier which, when give the features, tells us the name of the sign. We used a very simple concept learner called "nearest neighbour". It works by remembering all the examples it's seen, together with their classes. If you try to classify a new example, it finds the nearest example it already knows about and returns the class of the nearest example. Using such a system, we were able to achieve accuracies of between 75 and 80 per cent when the concept learner was trained with examples from one signer. Obviously, the system was very limited. The main limitations of the system were: The poor quality equipment for collecting the data. The way that features was extracted was "unnatural" in the sense that it had to be forced into existing concept learner. This meant throwing away a lot of the information. People, for example, use more advanced descriptions of the motion than feature extraction can provide. The current technique was sensitive to issues like emphasis, slowing down and speeding up. It performed very poorly on signers that it had not seen before. When tested on unseen signers, the accuracy was only 35 to 40 per cent. This was in part because of the above problem - that we were throwing away so much of the information. It could only recognise discrete signs - not flowing sign. It only had a very small vocabulary of about 95 signs. 4.2 Current research The current research being undertaken focuses on two of the problems: the poor quality of the equipment used and the development of algorithms that make use of the much deeper information available for signals that vary over time.

5 Improved Equipment Better equipment has been procured as a result of a grant. This equipment uses two 5th Gloves (one left, one right) for measuring finger bend. These gloves are far more accurate than the PowerGlove. For position and orientation sensing, two Ascension Technologies Birds are used. These use a magnetic field to evaluate position and orientation and are accurate to within a few centimetres. They also give useful orientation information about the direction that the hands are pointing. Furthermore, this information is updated more regularly (about 50 times a second) than the PowerGlove (about 20 times a second). While the equipment is not cheap (about AUD6500 for the whole system), prices for these devices are coming down because of the increased popularity of virtual reality applications. Learning Techniques The problem of recognising signals of the types emitted by the glove, however, have proved to be very challenging indeed. Off-the-shelf concept learners exist for domains which are static - that is, the features we are interested are assumed to not vary over time. Concept learning for dynamic features has only recently become an area of active research. It has proved to be far more difficult than static classification, for the following reasons: There is much more information for each example than in static case. Consider if an average sign takes two seconds and we are receiving 50 updates per second. Also assume that the hands are described by 22 features (things like finger bend for each feature). Then a single sign will generate 2200 pieces of information. Concept learners are usually not designed to cope with this much data. Furthermore, they do not usually cope with the total amount of information varying, which can happen in our case. For example, one sign might be 3 seconds long; while others might be only 1.3 seconds long. Time needs to be treated specially. Most concept learning algorithms can cope with variations in values caused by noise; but time needs to be treated in a special way. Even if it s the same sign performed by the same signer, there can be significant differences in not only how long the sign takes, but also in the gaps between important parts of the sign. There are a few existing systems for doing these. Two of the most popular are hidden Markov models and recurrent neural networks. Hidden Markov models, as used by Starner (see section 3) are very popular in the speech recognition community and have proved to be effective there. They have also proved to be effective for handwriting recognition. They can produce highly accurate results when given enough data. However, the do have some drawbacks. They can take a lot of examples before they learn productively. This is not a problem with the speech community, since speech is considered a very large market. For example, there are corpora for speech recognition that have millions of examples of different words. Also, they require a great deal of fine-tuning to get good results. Recurrent neural networks as used by Vamplew (see section 3) have also proved to be effective for learning signs. However, there are a number of significant problems with recurrent networks. The most serious is the problem that there may be issues with scalability. The complexity of the recurrent neural net grows very quickly as more signs are added. In addition, recurrent neural networks typically need to be retrained from scratch when new data is received. This can be quite time-consuming. Recognising signals that vary over time has become the focus of the current research. A temporal concept learning algorithm would not only be useful for recognising sign language, but also many other problems. For example, doctors to examine electrocardiographs (ECG) could use it as an aid for recognising heart conditions. A new temporal concept learning algorithm has been developed. This algorithm works in the following way: Convert the raw data describing the movements into a higher level representation that is more independent of time. By doing this, we can not only reduce the total amount of information we have to deal with, but also make the algorithm more robust to variations in time. For example, we can approximate the position information as a sequence of straight line movements. This cuts down the total information that we have to deal with. Look at all the examples of different signs and try to find common recurring patterns in the highlevel description. Find groups of similar highlevel patterns. Then characterise the features of these groups. Look at the original data and see if they have the same features as any of the groups we found above. Try to build rules for classifying each different sign we are trying to learn so that the rules cover all the examples of the sign we want to classify and no other ones A nice side effect of this is that the definition provides a simple description of what appear to it to be the distinctive parts of the sign.

6 Example An example of its application of this learning algorithm to sign language may be useful. Consider trying to learn what the sign is for the Auslan sign come. We get the following data from the glove: x position (left to right movement) y position (up and down position) z position (towards and away from the body) wrist roll (turning of the wrist along the axis of the forearm) thumb bend forefinger bend middle finger bend ring finger bend The high level representations we seek are gradual increases and decreases in these values. Thus data from a single come example would be converted automatically (and in a form more easily understood by the computer) to a description like: The x position decreases (ie. hand is moved closer to the body) gradually at the beginning of the sign. It stops increasing about one tenth of the way into the sign. It stays steady until the end. The y position increases (ie. hand is moved upwards) in the beginning of the sign, stabilises from ¼ of the way through the sign to ¾ of the way through, then decreases at the end. The z position increases sharply (ie. moves away from the body) for about the first third of the sign. Then decreases sharply (ie. moves towards the body) for the second third of the sign. It then stays stable close to the body until the end of the sign. Wrist roll starts out with palm down. About ¼ of the way through the sign, the wrist rolls anticlockwise, so that it ends palm up. It remains palm up until ¾ of the way through the sign, then rolls clockwise to the palm down position. Forefinger starts out fully bent. It remains fully bent until about ¼ of the way through the sign, when it becomes partially bent. It remains partially bent until about ¾ of the way through the sign, when it becomes fully bent again. There would be similar descriptions for the other fingers. Ten examples of the come sign might then be provided to our temporal concept learner, together with other examples of other signs. Note that it may have to deal with irrelevant data - for example, in the sign come, it doesn t make much difference whether the thumb is fully bent or partially bent. The temporal concept learner would then produce a description of the come sign, that once translated into human form, reads like this: IF There is a sharp decrease in the z position about one third of the way through the sign AND There is a gradual increase in y position about ¼ of the way through the sign AND The forefinger closes from partially bent to fully bent about two thirds of the way through the sign THEN It is a come sign. The above is an actual interpretation of the results produced by the current implementation of the temporal concept learner. Results Work is still in progress. Currently, the algorithm is being applied to the older data from the PowerGlove. However, from preliminary tests, it looks like the results produced by the current work, even on the old PowerGlove data, are significantly better than that of the previous research. In particular, performance on signers not seen before appears to be greatly improved; though it is still too early to give numerical accuracy results. In the near future, data will also be collected from professional signers using the new equipment. This data should also improve accuracy and performance a great deal. 5. Future Work The main focus in the short term will be on improving the accuracy of the algorithms for recognition of isolated Auslan signs. Obviously one avenue of expansion is to improve the system to allow for recognition of continuous Auslan signs. There are also potential improvements in making the sign recognition less signer-dependent. Other tasks that need to be expanded include: Expansion of the lexicon beyond 95 signs. Using facial gesture recognition to make the interface more intuitive and making the system capable of understanding higher levels of sign. Developing natural language processing systems for handling Auslan. Developing applications that integrate Auslan recognition in them. 6. Conclusions Research in the area of sign language recognition is progressing slowly but surely. There are even a few researchers who are specifically working on Auslan. However, there is still a long way to go and work is

7 still primitive and still far from being suitable for public consumption. But this will improve over the next decade or so, as the equipment and technologies for capturing gestures (both visual and direct device techniques) and the technologies for processing, recognising and interpreting gestures improves. 7. Acknowledgments Thanks to The Creator for giving me the ability to undertake this work. Secondarily, I would like to thank Adam Schembri for his useful comments on the linguistic aspects of sign and Auslan and for volunteering his time to provide me with sample signs. Thanks also to Andrew Young for also providing me with sample signs. Peter Vamplew was also kind enough to share some of the data he collected with me and also as a source of advice and suggestions. 8. Further Information More detailed and technical information is available on the World Wide Web. The following URLs may be useful: Machine Gesture and Sign Language Recognition Research has links and contact details of researchers and conferences in the area. My previous research (including an extensive literature survey) can be found at: My current research can be fount at: Also, if you have any further questions, feel free to e- mail me at: waleed@cse.unsw.edu.au. 9. References CM92 C. Charayaphan and A. Marble. Image Processing system for interpreting motion in American Sign Language. Journal of Biomedical Engineering, 14: , September DH94 Brigitte Dorner and Eli Hagen. Towards an American Sign Language Interface. Artificial Intelligence Review, 8(2--3): , Dor94 Brigitte Dorner. Chasing the Colour Glove: Visual hand tracking. Master s thesis, Simon Fraser University, Available at: ftp://fas.sfu.ca/pub/thesis/1994/brigittedornermsc.ps. DS93 James Davis and Mubarak Shah. Gesture recognition. Technical Report CS-TR-93-11, University of Central Florida, Fel94 S. Sidney Fels. Glove-TalkII: Mapping Hand Gestures to Speech Using Neural Networks -- An Approach to Building Adaptive Interfaces. PhD thesis, Computer Science Department, University of Toronto, FH93 S. S. Fels and G. Hinton. GloveTalk: A neural network inteface between a DataGlove and a speech synthesiser. IEEE Transactions on Neural Networks, 4:2--8, Gri83 G. Grimes. Digital Data Entry Glove interface device. Patent 4,414,537, AT & T Bell Labs, November HSM94 Chris Hand, Ian Sexton, and Michael Mullan. A linguistic approach to the recognition of hand gestures. In Designing Future Interaction Conference. Ergonomics Society/IEE, April Also available at FI94/gestures.html. Kad95 Mohammed Waleed Kadous. GRASP: Recognition of Australian Sign Language Using Instrumented Gloves. Honours Thesis, School of Computer Science and Engineering, University of New South Wales, Available from KL89 Jim Kramer and Larry Leifer. The Talking Glove: A speaking aid for non-vocal deaf and deaf-blind individuals. In Proceedings of RESNA 12th Annual Conference, pages , KL90 James Kramer and Larry J. Leifer. A Talking Glove for nonverbal deaf individuals. Technical Report CDR TR , Centre For Design Research -- Stanford University, Kra91 Jim Kramer. Communication system for deaf, deafblind and non-vocal individuals using instrumented gloves. Patent 5,047,952, Virtual Technologies, 1991.

8 MT91 Kouichi Murakami and Hitomi Taguchi. Gesture recognition using recurrent neural networks. In CHI 91 Conference Proceedings, pages Human Interface Laboratory, Fujitsu Laboratories, ACM, OTK91 T. Onishi, H. Takemura, and E. Kishino. A study of human gesture recognition for an interactive environment. In 7th Symposium on Human Interaction, pages , SP95 Thad Starner and Alex Pentland. Visual recognition of American Sign Language using Hidden Markov Models. Technical Report TR-306, Media Lab, MIT, Available at: ftp://whitechapel.media.mit.edu/pub/tech-reports/tr- 306.ps.Z. Sta95 Thad Starner. Visual recognition of American Sign Language using Hidden Markov Models. Master s thesis, MIT Media Lab, July Available at: ftp://whitechapel.media.mit.edu/pub/tech-reports/tr- 316.ps.Z. TK91 Tomoichi Takahashi and Fumio Kishino. Gesture coding based in experiments with a hand gesture interface device. SIGCHI Bulletin, 23(2):67--73, April Vam96 Vamplew, P., "Recognition of Sign Language Using Neural Networks", PhD Thesis, Department of Computer Science, University of Tasmania, 1996