Developing a gesture-based game for deaf/mute people Using microsoft kinect

Transcription

1 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems Developing a gesture-based game for deaf/mute people Using microsoft kinect Fakhteh Soltani, Fatemeh Eskandari, Shadan Golestan Department of Computer Engineering Faculty of Engineering Arak University Arak, Iran f-soltani@araku.ac.ir Abstract It is essential to get disabled people involved and connected to each other and to the rest of the society. Games can be used for this purpose as well as encouraging them to be active physically. However, many of the current interactive games interact with users through voice commands which could be a problem for the deaf/mute people. Microsoft Kinect opens a new aspect for the gaming industry. This hardware can interact with players through a 3D vision and sound detector. This means players can use their body movements as well as their voice commands to control the game environment. The aim of this paper is to develop a Kinect gesture-based game suitable for deaf/mute people. The Microsoft Kinect SDK for Windows is used to develop a game which recognize the gesture command and convert the sign to the text commands in the game (in this instance Microsoft Shape game). Therefore, the deaf/mute player can enjoy taking part in this interactive game. Conclusions are drawn on how researchers can adapt and develop the new game environment which is understandable and compatible with deaf/mute peoples abilities. Keywords: Interactive game; Microsoft Kinect SDK; gesture recognition; deaf-mute people I. INTRODUCTION People with hearing/speaking difficulties experience limitations in their personal and social life. These shortages can dramatically limit their ability to perform daily tasks, such as shopping, travelling, taking part in social events etc., independently [1]. Identifying motivating and effective methods of encouraging people with hearing disabilities to perform group activities could help them to overcome the limitations they experience and maintain or enhance their independence [2; 3]. This paper is a part of an ongoing research which aims to use gesture recognition technology in designing a system to help motivate people with hearing/speaking difficulties to increase their interaction with other people and improve their communication skills and quality of life. This paper specifically concentrates on utilizing Microsoft Kinect which is a webcam-style add-on peripheral intended for the Xbox 360 game console in order to replace speech recognition with gesture recognition technique in windows environment. Kinect allows users to control and interact with the game console through a natural user interface using gestures, without the need to touch a game controller. Kinect captures how the player body moves. It comes with an RGB camera and a depth sensor, which provide full-body 3D motion capture capabilities and gesture recognition. However, the usage of Kinect has gone beyond the system s intended purpose of playing games [4]. Our proposed system uses image processing technology of Kinect to detect the players movements and uses the data to map the movements to the text commands (for example, moving hands towards each other means the smaller size). This paper begins with an introduction about the deaf/mute people difficulties and the importance of using games in peoples life. It then gives a brief description of the Microsoft Kinect and explains the research aims and methodology. It follows by conclusions and future works in the last section. II. STATE OF THE ART A. A Number of General Problems of Deaf/Mute People Research shows that deaf/mute people could face a number of disadvantages, particularly if their issues left unrecognized, untreated or under treated which will affect all aspects of their life. They might have issues in the following areas [1;5-8]: 1) Sense of independence: Relying on others in communications on one hand, and frustration and anger because they cannot perform as well as hearing people on the other, causes the lack of self-esteem and confidence. 2) Education performance: In Some cases parents of deaf children insist on registering their children in mainstream schools. Therefore, communication and interaction with others come into play which may cause difficulty in maintaining attention. In addition, they may get lower grades than their hearing peers, and this causes many to drop out early and fewer job opportunities in the future /12 $ IEEE DOI /CISIS

2 3) Social interaction and communication: Deaf-mute people often have poor communication with their environment, and this will cause lack of skills to interact with the society. It is shown that deaf people relations are limited to their own community and people who know sign language. The following section reviews a number of technologies which can be used to help and support the deaf/mute people. B. Technologies for Deaf/Mute People Technological improvements have impacted on visual communication methods to assist deaf/mute people. A number of these technologies can be categorized as follows [9;10]: 1) Visual Communication Technologies: The technologies that make use of text, video, or flashing signal for communication. Such as Text Telephones which are devices for live conversations in text form over a telephone line. 2) Telecommunications Relay Service (TRS): It is an operator service that simplifies distance communication between deaf-mute people and those who are hearing; such as fax, the internet and IP Relay Quick Connect. 3) Mobile Communications: The generations of mobile communications are text, voice and video communication. The Requirements for these devices are pager, wireless telephone, or personal digital assistant. 4) Video Communications: It uses two main networks for sign language communication, which are integrated services digital network (ISDN) and high-speed internet connection. 5) Automatic Speech Recognition (ASR): ASR refers to a technology that translates speech to text form. There are some hearing technologies available such as hearing aids with latest features like noise reduction, loudness and control feedback reduction. The majority of the available technologies mainly assist deaf/mute people in their personal life rather than their social life. Although some of them ease the interaction between deaf/mute people and their environment, but their constraints (e.g. requiring internet connection and knowing sign language) could make them difficult to access. In addition, most of them are used by deaf people in order to understand what other people say. As many hearing people don t know sign language, therefore, deaf people might find it difficult to communicate in some situations and explain what they mean. As mentioned earlier, this research studies a new way of interaction between deaf/mute people and hearing people. This project use Kinect s gesture and speech recognition technology as an intermediate interface to translate sign language into speaking language and vice versa. This could help deaf/mute people not only have more convenient personal life, but also they could have more interaction with the others in public places. C. Gaming and its Positive Impact on Personal Life Research [11-13] shows that the number of entertainment technology users grows rapidly. Computer games are a part of this technology which could help adults and children (normal, impaired and disabled) to be prepared for life and work in the future. A well designed computer game can have a positive influence on the players learning and their communication with other players. The player learns how to control what happens on the screen, and multiplayer and network games teach players how to deal with the other players [12]. Therefore, much attention has been applied to using video and computer games in recent years [11]. Recent human-computer interaction technologies like Microsoft Kinect enable players to do real life physical activities [14]. It provides the combination of multiplayer gaming and physical interaction to establish more real world playing style to the players. In addition to this, gesture recognition of Microsoft Kinect could be used to enable deaf/mute players to participate in the games. Kinect will help deaf/mute players to learn how to control the game environment by special gestures and alter it to their own interest. In brief, The only experience they need would be life experience [15]. III. MICROSOFT KINECT Majority of the current computer games in the market are controlled by mouse, keyboard or joystick. The player should press keys or buttons to interact with the game. Recent new developments promising a more natural way of interacting with the game environment. First Nintendo introduced a remote controller for the wii gaming console which player could move it around and by these movements interact and play the games. This idea modified and enhanced by Sony and became the move controller for their PS3 game console. Later Microsoft developed their so called Kinect system which uses a 3D video and audio capturing system [15]. The Kinect sensor is the motion and speech recognition system originally made for Xbox360 gaming console but later allowed to be utilized for personal computers. The Kinect sensor consists of infrared light sensors for gathering distance information of the objects, a camera for capturing colour image as an input and a stereo microphone array for the speech recognition [4;14;16]. IV. SOFTWARE DEVELOPMENT KIT (SDK) Microsoft recently released a non-commercial Kinect software development kit (SDK) for Windows. This opens access to the capabilities offered by the Microsoft Kinect device to the academic and enthusiast communities for their research [15]. The SDK unwraps the potential to repurpose the Kinect sensor for use in windows-based PC applications without any changes in physical design and capabilities of it. Some of the features of the Microsoft SDK are: skeletal tracking, 492

3 depth data capturing to indicate the distance of the objects in its field of view, audio processing and speech recognition with acoustic noise and echo cancellation [15]. Considering the release date of the SDK which is less than 6 month, the variety of applications already developed or is under progress is noticeable. Microsoft Kinect has been used to control the application by using distance and location information of body joints [14], to detect objects and even facial recognition [16;17]. Other applications of Kinect are: rehabilitating young adults with motor disabilities [18], Kinect as a tool for cardiopulmonary resuscitation (CPR) quality improvement [19] and designing a BIM-based serious game for fire safety evacuation [20]. V. AIMS AND OBJECTIVES This paper is a part of ongoing research which aims to introduce and develop an intermediate system by using advanced but affordable Kinect sensor and Microsoft SDK. The proposed application will be used for interaction between deaf/mute people and hearing people in group events and also in public places such as department stores, courthouse, terminals, etc. This paper reports the first stage of the project and introduces a gesture recognition concept using Microsoft SDK to adjust the Shape Game. The Shape Game displays the tracked skeletons of one or two players together with shapes falling from the sky (Fig. 1). The players can control the shapes by moving and speaking commands. This research modifies the game in a way that deaf-mute player could control the attributes of falling shapes by gesturing at the proper shape or attribute instead of saying it. VI. METHODS In this research, the Microsoft Kinect SDK and Visual Studio C# 2010 Express are used. The SDK provides application programming interfaces (APIs) that are supported by the Microsoft Kinect drivers. The APIs control audio and video data streams and skeletal tracking of players who are moving within the Kinect sensor s field of view in the Windows environment [15]. The Microsoft Kinect SDK has been employed to create specific body gestures in a custom specified range based on the Kinect s motion sensors. This will enable us to define appropriate gestures that a user has to perform in order to trigger the shape game commands. In addition, if a user forgets how a gesture of a command is, the system will show the correct location of the body joints against the current body joints position. Skeletal tracking feature of the Kinect SDK has been used to track one or two players. As mentioned before, the objective is to modify the Microsoft shape game which is controlled by voice command and change it to a gesture controlled instead. The following pseudo code (Figure 2) recognizes the player(s) gesture and by using a switch-case block will assign the suitable action to the game-play according to recognized gesture. We propose a Finite-State Automaton (FSA) algorithm for gesture recognition. We create a database of a number of defined gesture FSAs For this game, gesture could be a simple geometrical shape such as circle, square, etc, or a colour, or size such as tiny, small, big and giant. For example, rectangle FSA has four states. The system will accept the FSA states if they are within the defined range of tolerance around the exact coordinates of each shape. The recognizer class used above has a list of the gesture FSAs. Each of these FSAs is a class that saves the coordinate of states of the gesture in form of a list containing X and Y axis (Figure 3) void Recognizer_SaidSomething(object sender, Recognizer e) Switch(e) case e.gesturecategory == Shapes Set fallingthings.shape to Recognized_Shape case e.gesturecategory == Actions SetAction to Recognized_Action Break; case e.gesturecategory == Colors Set fallingthings.color to Reconized_Color case e.gesturecategory == Sizes Set fallingthings.size to Recognized_Size Figure 2. Pseudo code for Gesture Recognition Figure 1. Shape game screenshot 493

4 Class FSA Public List <int x, int y> FSA = new List <int x, int y>; /* Other components to be added*/ Class GestureRecognizer List <FSA> Gestures = new List<FSA>; String GestureCategory; /* Other components to be added*/ Figure 3. Pseudo code for the FSA and GestureRecognizer classe Figure 4 shows a number of the gesture FSAs of the gestures list. The gesture recognizing process starts with finding every gesture FSA in the gestures list which their first state mapped to the current gesture of the player. When the player continues to complete the gesture, the further states in the candidate FSAs from the previous step(s), will be checked. The filtering of the gesture FSAs continues until a unique gesture FSA in the gestures list is recognized. In the following, an example of recognizing the rectangle gesture is provided. Figure 5.a shows the four states of the rectangle FSA. The black dots in each state indicate the current location of the player s body joints, and the grey squares indicate the correct location of the left and right hand joints. In this method, we consider the spine joint of the defined skeleton as zero coordinates. Therefore, all the states coordinates are relative to the spine joint. In this case, different people with different heights do not have to reach the specific coordinates for each state, which otherwise make gesturing hard for children with short heights for instance. The time independency should be mention as a benefit of this method. The user can accomplish the states of a gesture rapidly or slowly. However, the system will be reset after a defined period of time. Figure 5.a shows X and Y axis and zero coordinates mapped to the spine joint of the tracked skeleton. Figure 5.b shows the gesture FSA graph for this example. In this figure, each node represents the state and the edges represent the transition between the states. If the transition continues to state D, then the rectangle gesture is recognized.... Rectangle FSA(X 1,Y 1),(X 2,Y 2),(X 3,Y 3),(X 4,Y 4) Triangle FSA(X 1,Y 1 ),(X 2,Y 2 ),(X 3,Y 3 ) Small FSA(X 1,Y 1),(X 2,Y 2)... Figure 4. Gesture list with three gesture FSAs. Putting hands together Figure 5.b. A D Figure 5.a. The Rectangle FSA Moving hands outward in a horizontal line Moving hands downward in vertical lines Moving hands inward in a horizontal line FSA Graph for Rectangle s Gesture States VII. CONCLUSION Deaf and mute people enjoy playing video games as much as anyone else. On the other hand, participating in group games could help them to improve their engagement with the others. However, it can be hard for deaf/mute people to take part in many games due to the need to use voice commands during the game. Skeletal recognition of the human body by Microsoft Kinect SDK could lead the PC applications to a new level. It could enable deaf/mute people to control the game environment by performing gestures. Future research is in progress to develop an intermediate system based on the current code to extend its application in order to improve interaction between deaf/mute people and B C 494

5 other people in public places such as shopping centres, transport terminals and in their day to day life. This goal will be achieved by recognizing the single word gestures, as well as a complete sentence or an expression. The next stage of the research is to use the Hidden Markov Model (HMM) concept for anticipating the sign language sentences. REFERENCES [1] S. Kochkin, W. Luxford, J. L. Northern, P. Mason, and A. M. Harpe, "Are 1 Million Dependents with Hearing Loss in America Being Left Behind?" Mwc/allied healthcare group publication, Hearing Review, [2] F. Carrino, J. Tscherrig, E. Mugellini, O. Abou Khaled, and R. Ingold, "Head-Computer Interface: A Multimodal Approach to Navigate through Real and Virtual Worlds" in Human-Computer Interaction. Interaction Techniques and Environments, J. Jacko, Ed. Berlin /Heidelberg: Springer, 2011, pp [3] L. A. Siple, "Cultural patterns of deaf people," International Journal of Intercultural Relations, vol. 18, 1994, pp [4] Microsoft, Kinect sensor manual, Microsoft Corporation, [5] F. Bleckly, "Issues faced by deaf children," Minerva WebWorks LLC, [6] T. Nunes, U. Pretzlik, and J. Olsson, "Deaf children's social relationships in mainstream schools." Department of Psychology, Oxford Brookes University, [7] M. Ross, "Personal and Social Identity of Hard of Hearing People." International federation of hard of hearing people, [8] S. Siavoshi, "Deaf education in Iran, problems and strategies," The Iranian deaf studies foundation, [9] J. E. Harkins and M. Bakke, "Technologies for Communication, Status and Trends," in Oxford Handbook of Deaf Studies, Language, and Education. M. Marschark and P. E. Spencer, Eds. Oxford University Press, [10] G. Khaleghi Toroghi, "Deaf People and Sign Language, Facilities and Constraints," University of Tehran, [11] M. Jo. Dondlinger, "Educational Video Game Design: A Review of the Literature," Journal of Applied Educational Technology, vol. 4, [12] M. Prensky, "What Kids Learn That's Positive From Playing Video Games," [13] M. Rauterberg, "Positive Effects of Entertainment Thechnology on Human Behaviour," in Building the Information Society. R. Jacquart, Ed. Kluwer Academic Press, 2004, pp [14] J. Kang, D. Seo, and D. Jung, "A Study on the control Method of 3-Dimensional Space Application using KINECT System," IJCSNS International Journal of Computer Science and Network Security, vol. 11, [15] Microsoft, "Kinect for Windows SDK beta, Programming Guide," Microsoft Research, July [16] A. Wu, J. Jog, S. Mendenhall, and A. Mazalek, "A Framework Interweaving Tangible Objects, Surfaces and Spaces", in Human-Computer Interaction. Interaction Techniques and Environments, J. Jacko, Ed. Berlin / Heidelberg: Springer, [17] S. Chu and J. Tanaka, "Hand Gesture for Taking Self Portrait", in Human-Computer Interaction. Interaction Techniques and Environments, J. Jacko, Ed. Berlin / Heidelberg: Springer, [18] Y. J. Chang, S. F. Chen, and J. D. Huang, "A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities," Research in Developmental Disabilities, vol. 32, Nov. 2011, pp [19] F. Semeraro, L. Marchetti, A. Frisoli, E. L.Cerchiari, and G. D. Perkins, Motion detection technology as a tool for cardiopulmonary resuscitation quality improvement, Resuscitation, vol. 83, [20] U. Ruppel, and K. Schatz, Designing a BIM-based serious game for fire safety evacuation simulations. Advanced Engineering Informatics, vol. 25, 2011, pp