LIBRE-LIBRAS: A Tool with a Free-Hands Approach to Assist LIBRAS Translation and Learning

Transcription

1 CHAPTER 12 LIBRE-LIBRAS: A Tool with a Free-Hands Approach to Assist LIBRAS Translation and Learning Evandro Terra Gonçalves *, Renata Monteiro de Oliveira, Rubens Jesse Catharino Deknes Gonçalves, Sidney Roberto de Sousa and Michel Estadulho * Instituto Federal de Educação, evandro18.goncalves@gmail.com

2 142 Chapter 12. LIBRE-LIBRAS: A... Instituto Federal de Educação, Instituto Federal de Educação, Instituto Federal de Educação, Instituto Federal de Educação, Abstract: The Brazilian Signals Language (LIBRAS) is officially recognized as first language (L1) to the Brazilian deaf community. Since its officialization, its teaching is granted by law at public and private institutions. Just like is done in any other discipline, LIBRAS teaching needs specific courseware, as well as tools to assist teachers and students at classroom. We believe that the use of tools for aiding the correct LIBRAS signaling, considering correct hands config-

3 143 uration in each signal, is intended to potentialize the good LIBRAS learning at classes. Thus, this work presents LIBRE-LIBRAS, a web application software to aid simultaneous translation and verification of LIBRAS phrases, using Leap Motion sensor to capture signals from gestures provided by the student with his free hands. The software has a module for capturing and storing signals provided by a professional interpreter of LIBRAS. This module was developed using the MEAN stack, a set of technologies (MongoDB, Express, AngularJS, and Node.js) based on JavaScript language that, together, help to leverage web development process. A signals database was created using MongoDB, where such signals are acquired from the interpreter with help of Leap Motion sensor and a web page developed with AngularJS. To send the acquired signals to the signals database, were developed RESTful web services, using Express and Node.js. It was also developed a module to query signals from the student into the signals database. This module uses the Euclidean distances between each hand palm and the fingertips, in addition to the Euclidean distances between the fingertips, to provide a signal descriptor that can be compared to signals stored into the database. We have already developed modules for storing and querying LIBRAS signals. Now, we are working in modules to provide signaling lessons and translation assistance of words and phrases in LIBRAS. To provide a better experience for the student while using LIBRE-LIBRAS, it was developed a 3D hands model that mirrors the student s hands movements, so the stu-

4 144 Chapter 12. LIBRE-LIBRAS: A... dent can see his signals or imitate the movements proposed by LIBRE-LIBRAS lessons. LIBRE-LIBRAS development is still in course. Tests on the signals queries confidence are being performed to refine our querying module. Furthermore, we intend to perform tests with LIBRE-LIBRAS at the LIBRAS classes from our institution, in order to get a good LIBRAS teaching tool that helps and encourages students in this discipline. Keywords: LIBRAS, Computer Aided Translation, Leap Motion Sensor, Languages Teaching, Courseware Tools Background For a very long time, parents and teachers of Brazilian deaf kids were concerned about making them speak. Only in the end of 1950s, LIBRAS was faced as an adequate way of expression by deaf people 1. LIBRAS is a space visual language, expressed through gestures and body expressions. Indeed, it is a language so complex as any other 2. Thus, teaching LIBRAS becomes as complex as teaching Portuguese to initial grades, or even teaching English to Brazilian to infant or teenagers. So, at the same way that teachers of these two last languages need specific courseware and tools to assist them at their classes, LIBRAS teachers and

5 12.2. Purpose/Hypothesis 145 interpreters also need such resources. However, maybe more complex than teaching LIBRAS to a deaf individual is to train a future LIBRAS teacher that, in its turn, will have to teach this language to other people including deaf people Purpose/Hypothesis We believe that the use of tools for aiding correct LIBRAS signalling, considering correct hands configuration at each signal, is intended to potentialize the good LIBRAS learning at classes. Thus, this work presents LIBRE-LIBRAS, a web application software to aid simultaneous translation and verification of LIBRAS terms, using Leap Motion sensor to capture signals from gestures provided by the student with his free hands Design/Method LIBRE-LIBRAS can be divided in two modules. The first module is responsible for capturing terms signalized by LIBRAS professional interpreters to populate the signals database. For this, the module uses Leap Motion sensor to capture the terms gestures from the interpreter s hands. A web page was built using AngularJS to provide an interface for capturing the signals from the interpreter and transform such signals in a data model

6 146 Chapter 12. LIBRE-LIBRAS: A... composed by two vectors. The first vector contains the distances from each one of the 5 fingertips to the center of the hand s palm and the distances between each fingertip, not considering thumb. This totalizes 8 distances for one hand signals and 16 distances for two hands signals. Finally, the second vector contains the values of pitch, roll, and yaw angles from the hand, thus totalizing 3 angles for one hand signals and 6 angles for two hand signals. After the term signal is captured, its data model is sent to the database server, which is served by a RESTful API implemented with Express.js, running over Node.js. Finally, the signal data is stored at a MongoDB database. We decided to use MEAN stack (MongoDB, Express.js, AngularJS, and Node.js) due to its easiness of learning, expressiveness of code, and the fact that all the four technologies use a single programming language JavaScript. The second and last module is responsible to compare a gesture from the LIBRAS student to signals stored at the signals database. We borrowed Naidu and Ghotkar approach 3, originally used at Indian Sign Language (ISL), that used Euclidean distance and cosine similarity functions to compare the user s gestures to signals. The signal s data model used in this work is similar to the one used by Naidu and Ghotkar, but in this last work the author did not consider the hands angles.

7 12.4. Results Results We have performed tests over a database with 40 static LIBRAS terms signals, where a static term is a term which signal does not have a hand translation movement during its signalization. In our signaling tests with professional LIBRAS interpreters, 79.88% of the terms stored are correctly recognized. Figure 12.1 shows LIBRE-LIBRAS translation page, which presents a threedimensional model of the user s hands, in addition to the term corresponding to the user s signalling, when correct. Figure 12.1 LIBRE-LIBRAS translation page.

8 148 Chapter 12. LIBRE-LIBRAS: A Conclusions Now, we intend to refine the signals comparison method to increase its accuracy. Furthermore, we intend to develop an approach to store and compare dynamic terms signals and, so, publish a more robust version of LIBRE-LIBRAS for online use on the web for free. Finally, we aim to create a branch of LIBRE- LIBRAS project, which will substitute use of Leap Motion sensor to capture the user s gestures by the use of webcam associated to computer vision techniques. Bibliography 1. SOUSA, M.; PORROROZI. R. Ensino de LIBRAS para os profissionais da saúde: uma necessidade premente. Revista Práxis, Volta Redonda, n. 2, p , SACKS, O. Vendo vozes: uma viagem ao mundo dos surdos. São Paulo: Companhia das Letras, NAIDU, C.; GHOTKAR, A. Hand Gesture Recognition Using Leap Motion Controlle, International Journal of Science and Research, Chhattisgarh, v. 5, n. 10, p , oct