ADVANCES in NATURAL and APPLIED SCIENCES

Size: px

Start display at page:

Download "ADVANCES in NATURAL and APPLIED SCIENCES"

Magnus Dean
5 years ago
Views:

1 ADVANCES in NATURAL and APPLIED SCIENCES ISSN: Published BYAENSI Publication EISSN: May 11(7): pages Open Access Journal Assistive Android Application For Hearing Impaired People Using Sign Language 1 Joshi A, 2 Homeminitha MD, 3 Pavithra P, 4 Srilakshmi B 1,2,3&4 Information Technology, Panimalar Institute Of Technology Poonamallee, Chennai , India. Received 28 Feb 2017; Accepted 14 May 2017; Available online 19 May 2017 Address For Correspondence: Joshi A, Information Technology, Panimalar Institute Of Technology Poonamallee, Chennai , India joshiseeni@gmail.com Copyright 2017 by authors and American-Eurasian Network for ScientificInformation (AENSI Publication). This work is licensed under the Creative Commons Attribution International License (CC BY). ABSTRACT Hearing impaired people rarely used mobile phones before the introduction of SMS/MMS. Now texting allows both the deaf and hearing people to communicate with each other. Mobile video chat may one day replace texting but are not suitable for hearing impaired callers. Sign Language is the primary means of communication in the deaf community. The problem arises when the deaf people try to express themselves to other people with the help of these sign language grammar. Our proposed system provides a learning as well as an interactive application.it enables both normal and deaf people to learn the sign language and also provides communication between them via text messages and sign language videos. KEYWORDS: SQLite database, Speech Recognition, Natural Language Processing (NLP),Indian sign Language(ISL) INTRODUCTION Deaf people normally use sign language in order to communicate with each other. In this communication system, deaf people are not able to represent their ideas or messages which they wanted to say to other people. In today s world, technology has been developed so fast and presents each action in digital form then into images, videos or audio format. In order to make their lives more advanced, an application need to be developed which provides an opportunity to learn new things and a chance to get introduced to new technologies. Deaf people are usually deprived of normal communication with other people in the society. It has been observed that they find it really difficult at times to interact with normal people with their gestures, as only a very few of those are recognized by most people. Sign Language is the primary means of communication in the deaf and dumb community. For the Deaf community, we try to improve their quality of life by developing systems that can help them communicate better with the rest of the world and amongst themselves. Our System provides a learning support to know more about sign language.it also provides communication in which input is given in the form of speech or text. The Speech Recognition technology converts audible language into normal texts on smart phones. The messages from normal persons are received as a sign language video to the hearing impaired people. In this way it makes the communication much easier. 2.Literature Survey: Sangeetha, K. and Barathi Krishna. L [1] proposed an application which detect the Indian Sign Language(ISL) via mobile camera and converts that into corresponding text or voice. G.V.S.Subhaashini,S.Divya,S. DivyaSuganya, Tharani Vimal [6] presented a method to recognise the message sent by a hearing person and display it as a video for the deaf people. Sanja Primorac and Mladen Russo [8] proposed a system where speech recognition is done using HMM model which converts speech to text and deliver it to the user. ToCite ThisArticle: Joshi A, Homeminitha MD, Pavithra P, Srilakshmi B., Assistive Android Application For Hearing Impaired People Using Sign Language. Advances in Natural and Applied Sciences. 11(7); Pages:

2 167 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: Ayu Indah Shekar Melati, Kanisius Karyono[7] proposed an Android based application using Adaboostm1 classifier with reptree weaklearner which detects sound around the hearing impaired people. Tapas Kumar Patra, Biplab Patra, Puspanjali Mohapatra[5] presented a method to design a Text to Speech conversion module using Matlab by simple matrix operations. Other supporting literature survey papers are the system which processes speech at run time and produces corresponding text in English language using Hidden Markov Model proposed by Raghavendhar Reddy.B, Mahender.E [4]. Shanmukha Swamy, Chethan M P, Mahantesh Gatwadi [2] proposed optimized approaches of implementing the famous Viola Jones algorithm with LBP (Local Binary Pattern) features for hand gesture recognition. Sinora Ghosalkar,Saurabh Pandey,Shailesh Padhra,Tanvi Apte[3] in which speech technology provides an interactive system. 3.Proposed System: In this section, we discuss the modules in the proposed system, its block diagram and the related functions. 3.1 Overall architecture: Fig.1 shows the overall architecture of the proposed system. It consists of five modules.as we aimed at providing both learning and messaging as a feature in our application, the first module comprises of a tutorial to learn various basic signs in Indian sign language(isl) which is used to initiate communication between the hearing impaired and the normal people. In the second module, the user has to register and login using their user name and password. In the third module, the input given as speech is converted into text and sent to the user. The output received by the hearing impaired people is in the form of sign language videos corresponding to the input text given as a reply by the normal person and is displayed by accessing the local server. The way of interpreting every possible text varies from person to person and our system is said to deal with every possible way that exists with the help of Natural language Processing (NLP) in the fourth module. The final module comprises of sign language recognition which displays the corresponding sign language video for the text interpreted. In order to reduce the buffering of videos while retrieval we make use of a local server offline which reduces the time consumption to display the videos. This system is an Augmentative and Alternative Communication System and is an aided system that makes use of Signs and pictures to make the hearing impaired person understand the message. Fig. 1: Architecture for Proposed System. 4.Module Description: 4.1 Learning: This module helps the user to learn the basic signs in the Indian Sign Language(ISL).It includes corresponding ISL signs for the Alphabets, Numbers, Greetings, Month calendar, Days in a week and few other basic conversational videos. 4.2 Registration and Login: This module allows the user to sign in if they already have an account by entering their user name and password. If he/she is a new user, then they have to register before logging in. The Registration module includes fields such as Username, Password, Re-enter password and Contact number. The user registration gets successful and their details are stored in the SQLite database. After Registration the user gets logged in and taken to the next page where sign language keyboard is displayed.

168 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: 166-171 4.

corresponding recipients. When speech is detected, the application opens its connection with Google s server and starts to communicate with it by sending blocks of speech signal.

3 168 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: Speech to text conversion: In this module, input is directly given as a text using the sign language keyboard or Speech given as input by the user which is converted to text and then sent to the corresponding recipients. When speech is detected, the application opens its connection with Google s server and starts to communicate with it by sending blocks of speech signal. The speech is converted to text using Google s voice Recognition. First Google tries to understand the consonants and the vowels which is the foundational layer. Next, it uses those to make intelligent guesses about the words and then higher. Fig. 2: Speech to Text conversion 4.4 Access to Local server: The message sent by the normal person is received as the corresponding sign language video for the hearing impaired people.the corresponding video for the input text is searched by accessing the local server. The way of interpreting every possible text varies from person to person. Our system is said to deal with every possible way that exists with the help of a larger database using Natural language Processing (NLP) 4.5 Sign Language Recognition: The final module of the system comprises of the main Sign Language Video. This video is displayed on the deaf party side which is obtained from the local server. Fig. 3: Block diagram for sign language recognition 5.Algorithm: A.Speech Recognition Algorithm: 1.Hidden-Markov models (HMMs): Hidden-Markov models (HMMs) are popular statistical models used to implement speech recognition technologies. The time variances in the spoken language are modeled as Markov processes with discrete state spaces. Each state produces speech observations according to the probability distribution characteristics of that state. The speech observations can take on a discrete or a continuous value. In either case, the speech observations represent a fixed time duration (i.e., a frame). The states are not directly observable, which is why the model is called the hidden-markov model. The original minimal HMM (min_hmm) algorithm was implemented on a floating-point C language program platform running under the UNIX operating system. IN HMM model, the Viterbi algorithm is used to identify a sequence of observed events and finds the mostly likely sequence from it which becomes the word uttered by the user. 6.Results: We have implemented proposed system with five individual sections. For the implemented sections and subjects the application is running perfectly and flawlessly. There by our proposed application is suitable for use in real-time with high performance.

4 169 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: Fig. 4: Learning Module Fig. 5: Learning : Alphabets and Numbers Fig. 6: Learning: Greetings and Months Fig. 7: Register and Login Module

170 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: 166-171 Fig. 8: Messaging Module 7.

5 170 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: Fig. 8: Messaging Module 7.Assistive Hardware Device for Hearing Impaired People: In addition to the application, the android device is connected to a hardware device consisting of sensors which will detect any abnormal noise or sound more than the average sound pitch of a human and will produce vibrations to the user as well as gives notifications in the android device to keep the user well informed about the things happening around them for them to react accordingly to any emergency situation immediately. It will be connected to the android device via Bluetooth to get properly notified then and there. Conclusion: By using this application deaf person can easily interact with normal person anywhere, and they can also use this application for learning basic signs used in Indian Sign Language(ISL). Future Enhancements: The proposed system has several advantages but does not act as a translator and hence in the future works, the application can be developed as one which consists of both the existing advantages as well as acts as a translator and does translation of upto 80 languages. REFERENCES 1. Sangeetha, K. and L. Barathi Krishna, Gesture Detection for Deaf and Dumb People International Journal of Development Research, 4(3): Shanmukha Swamy, Chethan M P, Mahantesh Gatwadi, Indian Sign Language Interpreter with Android Implementation, International Journal of Computer Applications ( ) 97(13): Sinora Ghosalkar, Saurabh Pandey, Shailesh Padhra, Tanvi Apte, Android Application on Examination Using Speech Technology for Blind People, International Journal of Research in Computer and Communication Technology, 3: Raghavendhar Reddy, B., E. Mahender, Speech to Text Conversion using Android Platform, International Journal of Engineering Research and Applications (IJERA) 3(1): Tapas Kumar Patra, Biplab Patra, Puspanjali Mohapatra, Text to Speech Conversion with Phonematic Concatenation, International Journal of Electronics Communication and Computer Technology (IJECCT) 2(5): Subhaashini, G.V.S., S. Divya, S. DivyaSuganya, Tharani Vimal, Ear Hear Android Application for Specially Abled Deaf People, International Journal of Computer Science and Engineering Communications, 3(3): ISSN: Ayu Indah Shekar Melati, Kanisius Karyono, Android based sound detection application for hearing-impaired using Adaboostm1 classifier with reptree weaklearner, Asia-Pacific Conference on Computer Aided System Engineering (APCASE)

6 171 Joshi A et al., 2017/Advances in Natural and Applied Sciences. 11(7) May 2017, Pages: Sanja Primorac and Mladen Russo, Android application for sending SMS messages with speech recognition interface, MIPRO, Opatija, Croatia. 9. Sonal Bhatt, Natural Language Processing with Text-toSpeech on Android Division of Computing Studies,Arizona state university.

Glove for Gesture Recognition using Flex Sensor

Glove for Gesture Recognition using Flex Sensor Mandar Tawde 1, Hariom Singh 2, Shoeb Shaikh 3 1,2,3 Computer Engineering, Universal College of Engineering, Kaman Survey Number 146, Chinchoti Anjur Phata