1. INTRODUCTION. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 1

Transcription

1 1. INTRODUCTION Sign language interpretation is one of the HCI applications where hand gesture plays important role for communication. This chapter discusses sign language interpretation system with present status of sign language in India. It also covers motivation for Indian sign language (ISL) interpretation, proposed objectives, and research contributions. 1.1 SIGN LANGUAGE AND HUMAN COMPUTER INTERACTION Sign Language (SL) is a primary mode of communication for many deaf people in many countries. Standard sign languages are known as deaf and dumb languages. SL exhibits inherent properties since it is used by people who have communicated symbolically encoded message without the use of the speech channel. Since SLs are highly structural they are very suitable for vision algorithms in the field of HCI for to get natural communication feel. At the same time, it can also be a good way to help the disabled to interact with computers. Human Computer Interaction (HCI) is an active focus of research now days. At present, considering the processing speed of computer machines, it becomes an important component for application development. Providing the Artificial Intelligence, machine tries to emulate human brain. But it is observed that it is not possible to replace human interventions. There is a need to develop human interfaces to the machine as per the need of the application. Evolution of HCI begins from text-based interfaces through 2-D graphical-based interfaces, multimedia-supported interfaces to full-fledged multimodal based 3D virtual environment (VE) systems. It is progressing towards touch less interface with the help of human gesture interface. Different means for human communication are speech, gesture, facial and bodily expressions. Symbolic gestures such as pantomimes that signify action are found in all human cultures. (e.g. finger to lips Vision based Multi-feature HGR Algorithms for HCI using ISL Page 1

2 indicating be quiet, putting both hands together for praying and so on). Symbolic gesture and spoken language are processed by a common neural system [1]. Gestures are powerful means of communication among humans. It may be based on hand, body, lip movements (speech expression), eye movements, facial expression (mood/emotions) of the human. Among all, hand gesture is the easiest and most natural way of communication between people. With the latest advances in the field of computer vision, image processing and pattern recognition, real-time vision-based hand gesture classification is belonging more and more feasible for human-computer interaction in virtual environments. Hand gestures are an intuitive yet powerful communication modality which has not been fully explored for HCI. Sign language study shows that among various gesture communication modalities, hand gesture plays significant role. This research aim towards working on hand gesture recognition for sign language interpretation as a HCI application. 1.2 SIGN LANGUAGE IN INDIA Indian sign language (ISL) is the main mode of communication of Indian deaf society. In literature, it has been found that the number of hearing impaired people in India is more as compared to that in other countries. Not all of them use ISL but, more than one million deaf adults and around half a million deaf children use ISL as a mode of communication [2]. Till 2003, the number of deaf signer was 2,680,000.[3]. In India, with vast diversity and cultural differences, ISL varies from region to region as communication language. However, in all large towns and cities across the Indian subcontinent, deaf people use sign languages which are not universal sign languages. Extensive work has been done in this regard by creating awareness amongst the sign language teachers in implementing the standardized ISL [2]. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 2

3 In 1970, linguistic work on ISL began with contribution by a team of researchers from America, and Vasishta et al. [4]. It was found that ISL is a language in its own right and is indigenous to the Indian subcontinent. The work resulted in four dictionaries between 1977 and It was found that 75% signs are same across the region. In 1998, another researcher from Germany (Dr. Ulrike Zeshan) compared signs from many different regions across the Indian subcontinent, including regions such as Orissa, Kerala, Jammu and Kashmir, Bhopal, Chennai, Bangalore and Darjeeling. She also found that on an average about 75% of the signs are similar across different regions [4]. Further work was carried out by Zeshan and Vasishta [4] on developing ISL grammar, ISL teaching courses, ISL teacher training program and teaching material which were approved by the Rehabilitation Council of India in 2002 [4-5]. After survey, it was found that there were around 405 deaf and dumb schools in India. Most of the schools used their own native sign language as a teaching and learning aid. Many ISL cells and NGOs are working in India to help ISL teachers for incorporation of standardized ISL in teaching courses. Ali Yavar Jung National Institute for Hearing Handicapped, Mumbai released Basic course in INDIAN SIGN LANGUAGE [1]. Major research work is going on for creation of ISL dictionary tool [2]. Ramakrishna mission vidyalaya, Coimbatore has also developed Instructional ISL video for sign language teacher. This project was sponsored by International human resource development centre (IHRDC) for the disabled. Even now, deaf people are isolated from the society because of communication gap. There are two ways to bring deaf people into the mainstream of society, i) Human interpreter and ii) Computer interpreter. As compared to number of deaf people, human interpreters are much less in number and moreover they may not be available every time. Computer interface can play useful role in connecting deaf people with normal people in their own sign language. Since 2010, Indian researchers have started working on ISL interpretation. Currently few researchers are working on native ISL interpretation. Though the research on ISL interpretation started late [6], it will definitely help Indian deaf people in near future as a communication mode through computer interpreter. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 3

4 1.3 SIGN LANGUAGE INTERPRETATION SYSTEM Sign language is not a universal language. Sign language recognition is a multidisciplinary research area involving pattern recognition, computer vision, natural language processing and psychology. Figure 1.1 shows the typical architecture for sign language interpretation system. It broadly divides into two modules. First module is for converting normal English sentences in to SL (to be understood by deaf people) and another module is for converting SL into English text (to be understood by normal people). For literate hearing impaired people, those who can read English, first module is not required. But for illiterate deaf and dumb people, both modules are essential. In both the modules, language processing engine is required which is based on particular language rules. Conversion of sign into text includes the area of computer vision, image processing, pattern recognition and language processing with linguistic study. Figure 1.1: A typical sign language interpretation system There are some common misconceptions about sign language which are reported in literature [5]: Sign language is same all over the world Sign language is not a complete language. It is just a sort of pantomime or gesturing, and it has no grammar Sign language is dependent on spoken language. It is a representation of Vision based Multi-feature HGR Algorithms for HCI using ISL Page 4

5 the spoken language of the hands Sign language is the language of the hands only Sign language has been invented by other people to help deaf people Signed Hindi or signed English is better than Indian sign language Therefore, to clear misconceptions, there is a need for developing ISL interpretation system to aid Indian hearing impaired people with the help of HCI and to make them self-dependent Major Impact of ISL Interpretation: Social Development of assistive system for deaf Indian people which will help them to communicate with normal people in their own sign language. Serving the mankind through the use of technology The human aspect in dealing with the physically impaired people can be reinforced by involving them in our day to day life Blind people can also use the same system by extending it for voice interface Educational Incorporation of such a system in education, opportunities for jobs for deaf people in Industry, IT sector, and public sector can be created Education and training will be easier for deaf people Use and awareness of computer interface through ISL interpretation Industrial In addition, commercial application could be based on the result of sign language recognition such as in TV channels, film industry, gaming industry, various HCI applications as well as on android based mobile applications. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 5

6 1.4 MOTIVATION FOR THE RESEARCH After the thorough review of the literature and after considering the needs of deaf and dumb community, its size, the misconceptions about sign languages, and its impact on social, educational and industrial circles, this researcher is motivated to develop the Indian sign language interpretation system in order to help Indian deaf community. In Indian Sign languages human hand gestures, facial expressions, and lip movements are various modes of communication. Considering frequency of their use, this research is focused only on hand gestures which form a basis for developing a gestural human machine interface. Many assistive systems can be used not only for deaf but also for elderly people. 1.5 OBJECTIVES Hand gesture plays a very important role in various applications of humancomputer interaction. Analyzing and finding new method for vision based hand gesture recognition (HGR) is the practical need for complex and challenging problem such as ISL interpretation or for any HCI application in general. The objectives of the research are: To study existing sign language interpretation systems To understand and learn Indian sign language To carry out survey of techniques proposed and used by various researchers for sign language interpretation To develop a model for Indian sign language standard gesture set To design the new vision based multi-featured hand gesture recognition algorithm for HCI using Indian sign language To implement the proposed algorithm and test the same on different users gesture Set. Scope of this research is limited to conversion of hand gesture based ISL vocabulary set to text. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 6

7 1.6 RESEARCH CONTIBUTIONS: Major contributions of our research are summarized below: i) Created ISL dataset for ISL manual alphabets and numbers and made them publically available for further research. ii) Designed three-tier architecture for ISL interpretation system which is divided work into three levels, based on divide and conquer strategy. ii) There is an always demand for any HCI application with natural interface. Due to the challenges of vision system most of the researcher developed user dependant hand segmentation algorithm. In this research, various hand segmentation algorithms for vision based system are developed and simple solution is given with a new algorithm for signer independent hand segmentation which can help in any HCI application. iii) Development of new multi-feature hand gesture recognition algorithm by providing perspective that high information preservation can been done by fusing contour and region based features as well as statistical and transform based feature. The multi-feature algorithm outperformed by true positive rate of 99.61% compared to individual descriptor for alphabets and numbers. iv) Various classifiers, such as k- Nearest Neighborhood (k-nn), Nearest mean classifier (NMC), and Naïve Bays are used to analyze and compare the result with various descriptors. This research identified k-nn based simple and easy method which is better than other complex classifiers such as Neural network, SVM, Kohonen self organizing map found in literature. k-nn classifier showed that it gave high recognition accuracy compared to other classifiers as well as it performs well in real time on multi- dimensional feature vector. vi) In literature, various researchers have attempted isolated word recognition [7] for different SL or continuous word recognition using data glove based approach [8]. Developing vision based continuous sign word recognition is the practical need for any SL interpretation. Here, two dynamic hand gesture recognition algorithms for continuous word recognition were explored and tested on ISL manual dataset for word recognition. Result revealed that, dynamic time warping (DTW) based approach for continuous ISL word recognition gave accuracy of 97.8% better Vision based Multi-feature HGR Algorithms for HCI using ISL Page 7

8 than the one given by Fuzzy logic based isolated sign recognition approach [9]. vi) Linguistic study, psychology as well as culture play a very important role in the formation of sentence. It is very difficult to consider all these aspects with vision and pattern recognition system. Novel ISL sentence creation algorithm gave intelligent solution rather than construction of grammar for ISL. Obtained results are encouraging for incorporation of this novel idea for sentence formation in any SL language for robust system. vii) Grammar model has been developed for ISL gesture dataset which simplifies the task of creation of motion descriptor. viii) One of the HCI applications such as handling desktop/laptop operation using real time hand gesture recognition was developed. This application used hand gesture modality to open window applications such as notepad, paint, media player, internet explorer and to log off. New algorithms for finger count were explored ix) It will be a great service to the Indian deaf people through working on Indian sign language interpretation tool, so that they are enabled to become selfrespecting citizens and despite their deafness and muteness can play a useful role in the society. Vision based hand gesture recognition is a long-standing technical challenge in the development of an ISL interpretation system, making it a thrust area for further research and development. This research aims to contribute to the advancement of HGR by exploring and developing various computer vision and classification algorithms and techniques. 1.7 ORGANIZATION OF THE THESIS The thesis is comprised of five chapters. Chapter 1 Introduction covers the background of sign language with existing scenario of ISL in India with motivation and proposed objectives. Chapter 2 Theoretical Foundation and Literature Survey covers theoretical Vision based Multi-feature HGR Algorithms for HCI using ISL Page 8

9 foundation and review on vision based hand gesture recognition. It also covers survey on various sign languages and Indian sign languages. Challenges for sign language interpretation and existing commercial applications based on sign language are presented.. Chapter 3 Vision based Hand Gesture Recognition using ISL covers methodology adapted for GesturePreter system with three-tier architecture and dataset for ISL. It covers experiments carried out for static hand gesture recognition for ISL alphabets and numbers, dynamic hand gesture recognition for ISL words and ISL sentence construction part in detail. Chapter 4 Result and Analysis presents detail discussion on result for hand tracking and segmentation, static hand gesture recognition, dynamic hand gesture recognition and ISL sentence construction with analysis. Comparison analysis has been carried out with existing method found in literature. Chapter 5 Conclusion and Future Scope covers conclusion with future scope for ISL interpretation. Vision based Multi-feature HGR Algorithms for HCI using ISL Page 9