Finger Braille teaching system-teaching of prosody of finger Braille

Transcription

1 Feb. 2009, Volume 6, No.2 (Serial No.51) Journal of Communication and Computer, ISSN , USA Finger Braille teaching system-teaching of prosody of finger Braille Yasuhiro MATSUDA, Tsuneshi ISOMURA (Faculty of Creative Engineering, Kanagawa Institute of Technology, Kanagawa , Japan) Abstract: Finger Braille is one of tactual communication media of deaf-blind people. Deaf-blind people who are skilled in Finger Braille can catch up with speech conversation, because of prosody of Finger Braille. Features of prosody are: (1) characters at the end of clauses are dotted long; (2) characters of voiced sounds and double consonants are dotted shortly. In this paper, we designed three teaching patterns which taught prosody of Finger Braille (dot conditions about duration of dotting). Teaching pattern 1 indicated a dot pattern with colored solid background. Teaching pattern 2 indicated a dot pattern with colored empty background. Teaching pattern 3 indicated a dot pattern with colored arrow (long or short). Every teaching pattern includes 6 colors (red, orange, yellow, green, blue and purple). In the questionnaire, the yellow pattern of teaching pattern 3 was selected as the most visible pattern. In the evaluation experiment, the yellow pattern of teaching pattern 3, the red and purple patterns of teaching pattern 1 (the least visible pattern) and the existing patterns without teaching of dot conditions are compared. As a result, every subject could dot long at the end of clauses and could dot shortly at the voiced sounds and double consonants. The answers of questionnaire indicated that the yellow pattern of teaching pattern 3 was also the most visible and easiest teaching pattern. Key words: finger Braille; deaf-blind; communication aid; teaching; dot condition; prosody 1. Introduction According to the surveys by the deaf-blind Association of Japan, there are 16,354 deaf-blind people of estimate in Japan. Deaf-blind people use many different communication media by the age of onset and what resources are available to them. Acknowledgment: This study was supported by the Ministry of Education, Culture, Sports, Science and Technology of Japan under a Grant-in Aid for Scientific Research (No ). Yasuhiro MATSUDA, Ph.D., associate professor; research fields: assistive technology, human interface. Tsuneshi ISOMURA, Ph.D., professor; research field: assistive technology. Yubi-Tenji or finger Braille is one of tactual communication media of deaf-blind people (see Fig. 1). In two-handed finger Braille, index finger, middle finger and ring finger of both hands are likened to keys of Braille typewriter. A sender dots Braille code on fingers of a receiver like whether he/she does the type of Braille typewriter. Then the receiver recognizes the Braille code. In one-handed finger Braille, the sender dots the left column of Braille code on the DIP joints of three fingers of the receiver, and then the sender dots the right column of Braille code on the PIP joints of them. A rule of Finger Braille is that the sender keeps on touching fingers of the receiver without dotting, because the deaf-blind receiver is uneasy without touching or any cues. Deaf-blind people who are skilled in Finger Braille can catch up with speech conversation, because of prosody of finger Braille [1-3]. Prosody is a kind of intonation of finger Braille and helps deaf-blind receiver recognize the clause structure of dotted sentence. Features of prosody of finger Braille are: (1) characters at the end of clauses are dotted long; (2) characters at the end of sentences are dotted long and strongly; (3) characters of voiced sounds (daku-on) and double consonants (soku-on) are dotted shortly; (4) the sender pauses between clauses; (5) the sender must not pause during a clause [1-3]. Because there are small non-disabled people who are skilled in finger Braille, deaf-blind people communicate only with the interpreter and their participation is greatly restricted. Recently, some Braille input devices were developed [4-5]. These devices let deaf-blind people wear gloves or a keyboard to input his/her finger Braille and actuators to output non-disabled people s 55

2 speech converted to finger Braille. In such supporting devices, deaf-blind people are burdened with wearing the sensors and the actuators, and must master new communication system with the supporting devices. Finger Braille helps deaf-blind receiver recognize the clause structure of dotted sentence, the teaching system intends to teach prosody of finger Braille. Thus deaf-blind people don t have to aware of the supporting device and may communicate by usual finger Braille without interpreters. In this paper, we designed teaching patterns which taught prosody of finger Braille (dot conditions about duration of dotting). And two questionnaires and an evaluation experiment were carried out. PIP joints Fig. 1 DIP joints Two-handed finger Braille and one-handed finger Braille Objective of this study is development of a finger Braille supporting device which respected skin contact communication, because skin contact is only a non-verbal communication for deaf-blind people. Fig. 2 shows the concept of the finger Braille supporting device. The largest features of this study are that deaf-blind people and non-disabled people who are non-skilled in finger Braille are communicate by usual finger Braille and all sensors are worn by non-disabled people. This supporting device includes two assistive systems; one is a teaching system, the other is a recognition system. The teaching system recognizes non-disabled people s speech and displays the dot pattern of Finger Braille. Non-disabled people dot Finger Braille by observing the dot pattern [6-7]. The recognition system senses deaf-blind people s finger Braille and converts to speech [8]. Because prosody of Fig. 2 Finger Braille supporting device 2. Summary of finger Braille teaching system 2.1 Configuration of the teaching system Fig. 3 shows configuration of the finger Braille teaching system [6-7]. First, speech recognition engine recognized sender s speech. Second, by using results of speech recognition, the teaching system converted Kana characters to the Braille code. Third, by parsing Braille code, the teaching system retrieved clause information and segmented the Braille code into clauses. Finally, the teaching system displayed dot pattern of the Braille code. Fig. 4 shows one-handed finger Braille communication supported by the teaching system. The teaching system was developed in a tablet PC (TC1100, HP, CPU Pentium M 1.1GHz, RAM 1024MB, LCD 10.4 inches XGA). OS was Microsoft Windows XP. The main program of the teaching system was programmed in Microsoft Visual Basic 6. Speech recognition engine was Microsoft Speech SDK (SAPI5.1). Braille code parser for clause 56

3 segmentation was programmed in Win-Prolog (LPA). Fig. 3 Configuration of finger Braille teaching system The non-disabled The deaf-blind Teaching system right to lower right). A clause not more than five characters was displayed on one column and a clause more than five characters was displayed on two or three columns. After a clause was displayed on the left column, the next clause was displayed on center column or the next page. Thus the dot pattern of clauses was displayed explicitly on columns. Red pattern indicated DIP joints and blue pattern indicated PIP joints. We designed two kinds of presentation method for the sender s experience. Presentation method A only displayed the dot pattern and presentation method B displayed the dot pattern on illustration of fingers. Presentation B was more symbolic and easy to recognize dotting fingers for beginners (level 1). Presentation method A was the most simplified sign and suitable for the experienced (level 2). According to our previous studies, non-disabled beginners could achieve level 2 after they dotted about 200 to 300 characters [7,9]. Fig. 4 One-handed finger Braille communication supported by the teaching system 2.2 Design of teaching interface To develop the teaching system, we adopted on-handed finger Braille as communication medium, because one-handed finger Braille is easy to dot for non-disabled people. The teaching system intends to teach prosody of finger Braille. We designed a teaching interface which promoted pauses between clauses and prevented pauses during a clause (see Fig. 5) [6-7]. The Braille code was displayed in upper text box. The dot pattern was displayed in fifteen picture boxes (three columns and five rows). The first clause was displayed on the left column (from upper left to lower left) and the second clause was displayed on the center column (from upper center to lower center). The third clause was displayed on the right column (from upper Presentation method A 57

4 Presentation method B Fig. 5 Two kinds of presentation methods of teaching interface 2.3 Evaluation experiment An evaluation experiment between a blind person who was skilled in finger Braille and two non-disabled people who were non-skilled in finger Braille was carried out [6-7]. Receiver was a blind person who was skilled in finger Braille (sending experiment: 20 years, receiving experiment: 8 years). Senders were two college students who were non-skilled in finger Braille (M1/F1). All senders reported normal hearing and vision abilities and were native Japanese speakers. All subjects gave informed consent after hearing a description of the study. Dialogues were four daily conversations in Japanese textbook for foreign beginners [10]. (Total 51 sentences, 143 clauses, 288 words, 686 characters). Senders were instructed to operate the teaching system, and trained SAPI5.1 by speech trainings Introduction and Introduction of speech technology. Receiver wore earplugs and headphones and heard white noise to insulate with sound during experiment. In experiment, one sender pushed key of speech recognition and spoke one sentence of the dialogues. If result of speech recognition was correct, the sender dotted finger Braille on fingers of the receiver by observing the teaching interface. If the result of speech recognition was not correct, the sender spoke the same sentence again or edited the Braille code and pushed key of restatement. Then the sender dotted Finger Braille on fingers of the receiver. The receiver spoke recognized sentence. If the receiver misrecognized, the sender dotted the same sentence again. Thus the sender repeated about the all dialogues. Senders were instructed to dot long at the end of clauses. As the results of fundamental functions, accuracy of speech recognition was 94.4%. Accuracy of conversion to Braille code was 99.4%. Accuracy of clause segmentation was 94.4%. As the results of dotting, accuracy of dotting by sender was 99.0%. Accuracy of recognition by receiver was 100% after re-dotting. Dotting speed was 34.6 characters/min (S.D.=12.3). Communication speed was 21.8 characters/min (S.D.=8.0). One of the senders dotted long at the end of clause and sentence. The senders paused long between clauses and did not pause long during a clause. These results showed that: the fundamental functions were practicable; the non-disabled senders could dot finger Braille accurately and communicate with the blind receiver directly; dotting finger Braille using the Teaching System was more accurate than inputting Braille code [4-5] ; the teaching interface could promote pauses between clauses and prevent pauses during a clause. 3. Design of teaching pattern of prosody 3.1 Design of teaching pattern The teaching interface can promote pauses with prosody of Finger Braille, but does not teach dot conditions about duration of dotting (dotting long or shortly). Because non-disabled beginners (level 1) are difficult to change duration of dotting [2,9], we intend to 58

5 teach dot conditions about duration of dotting for the experienced (level 2). Thus we designed teaching patterns of prosody, emphasizing the existing dot patterns of presentation method A. We designed three teaching patterns which taught dot conditions. Teaching pattern 1 indicated a dot pattern with colored solid background, emphasizing pattern with dot condition. Teaching pattern 2 indicated a dot pattern with colored empty background, emphasizing pattern but keeping visibility. Teaching pattern 3 indicated a dot pattern with colored arrow (long or short), symbolic pattern about dot condition. Every teaching pattern includes 6 colors (red, orange, yellow, green, blue and purple). Fig. 6-Fig. 8 show examples of the teaching pattern 1-3. Purple Red Fig. 6 Teaching pattern 1 (dot pattern with colored solid background) To evaluate the visible colors of each teaching pattern, a questionnaire of visibility of teaching pattern was carried out. Subjects were 12 college students who were not skilled in finger Braille. All subjects reported normal vision abilities and gave informed consent after hearing a description of the study. First, testor showed six patterns of teaching pattern 1 on the upper picture boxes of the teaching interface and six patterns of teaching pattern 2 on the lower picture boxes (see Fig. 9 left). The subjects placed the six patterns of each teaching pattern in order of visiblility. Second, the testor showed six patterns of teaching pattern 3 with long arrow on the upper picture boxes of the teaching interface and six patterns of teaching pattern 3 with short arrow on the lower picture boxes (see Fig. 9 right). The subjects placed the six patterns of each teaching pattern in order of visiblility. Yellow Green Fig. 7 Teaching pattern 2 (dot pattern with colored empty background) Long-yellow Short-yellow Fig. 8 Teaching pattern 3 (dot pattern with colored long/short arrow) Teaching pattern 1 and Questionnaire of visibility of teaching pattern 59

6 purple patterns are the lowest, especially in teaching pattern 1. Fig. 10 Visibility score for each color of teaching pattern Teaching pattern 3 (L/S) Fig. 9 Questionnaire of visibility of teaching pattern Next, each color of pattern was assigned visibility score. The color of pattern ranked 1st was assigned 5 points. The color of pattern ranked 2nd was assigned 4 points. The color of pattern ranked 3rd was assigned 3 points. The color of pattern ranked 4th was assigned 2 points. The color of pattern ranked 5th was assigned 1 point. The color of pattern ranked 6th was assigned 0 point. Fig. 10 illustrates mean of visibility score for each color of teaching pattern. As a result, visibility scores of the yellow patterns are the highest in each teaching pattern, and visibility scores of the 3.3 Questionnaire of selection of teaching pattern To evaluate the most visible teaching pattern and its color on the teaching interface, a questionnaire of selection of teaching pattern was carried out. Subjects were same as the questionnaire of visibility. All subjects gave informed consent after hearing a description of the study. The subjects replaced the teaching patterns with the existing pattern themselves. At the end of clause and sentence, the subjects selected the most visible teaching pattern and its color for dotting long. At the voiced sound and double consonant, the subjects also selected the most visible teaching pattern and its color for dotting shortly. Fig. 11 shows example of selection of teaching pattern. Existing pattern Fig. 11 Example of selected teaching pattern Questionnaire of selection of teaching pattern 60

7 Because the visibility of teaching pattern was depended on number of displayed characters, clauses, voiced sounds and double consonants, four sentences [10] were selected as dialogues. Dialogues were: Excuse me (Sumimasen) (5 characters, 1 clause); How do you do? (Hajimemasite) (7 characters, 1 clause, 1 voiced sound); Good morning (Ohayo gozaimasu) (11 characters, 2 clauses, 2 voiced sounds); Please get some rest (Yukkuri yasunde kudasai) (14 characters, 3 clauses, 2 voiced sounds, 1 double consonant). Total replaced patterns were 13 patterns (3 ends of clauses, 4 ends of sentences, 5 voiced sounds and 1 double consonant). Fig. 12 shows the selected ratio of teaching patterns and colors. The most selected pattern was the yellow pattern of teaching pattern 3 (30.8%). The second one was the yellow pattern of teaching pattern 1 (12.8%). Selected ratio of teaching pattern 3 of all colors was 52.6% and selected ratio of the yellow pattern of all teaching patterns was 51.3%. Therefore it was considered that the yellow pattern of teaching pattern 3 was the most visible pattern to teach dot condition about duration of dotting and the purple pattern of teaching pattern 1 was the least visible pattern. Fig. 12 Selected ratio of teaching patterns and colors 4 Evaluation experiment 4.1 Methods To evaluate that non-disabled people could dot prosody of finger Braille, an evaluation experiment was carried out. Subjects were 10 college students who were not skilled in finger Braille (level 1). All subjects reported normal vision abilities and gave informed consent after hearing a description of the study. Teaching patterns for evaluation experiment were the yellow pattern of teaching pattern 3, as the most visible pattern, the purple and red patterns of teaching pattern 1, as the least visible pattern, and the existing patten without teaching of dot conditions (see Fig. 13). Teaching pattern 3 Teaching pattern 1 Existing pattern The most visible pattern The least visible pattern Fig. 13 Teaching patterns for experiment 61

8 Dialogues were 12 sentences [10] which could be displayed in one page of teaching interface (less than or equal to 3 clauses and 15 characters). These sentences were divided into three groups as follows: (1) Excuse me (Sumimasen) (5 characters, 1 clause), How do you do? (Hajimemasite) (7 characters, 1 clause, 1 voiced sound), Good morning. (Ohayo gozaimasu) (11 characters, 2 clauses, 2 voiced sounds), Please get some rest. (Yukkuri yasunde kudasai) (14 characters, 3 clauses, 2 voiced sounds, 1 double consonant); (2) Hello (Konnichiwa) (5 characters, 1 clause), Going out? (Odekakedesuka) (9 characters, 1 clause, 2 voiced sounds), See you later. (Itte irassyai) (9 characters, 2 clauses, 1 voiced sound, 2 double consonants), What kind of pictures do you take? (Donna syasinwo torimasuka) (14 characters, 3 clauses, 2 voiced sounds); (3) No (Iie) (3 characters, 1 clause), Is it really? (Hontodesuka) (8 characters, 1 clause, 1 voiced sound), So take care of yourself. (Ja odaijini) (9 characters, 2 clauses, 3 voiced sounds), I like horses best (Tokuni umaga sukidesu) (12 characters, 3 clauses, 2 voiced sounds). Total replaced patterns were 40 patterns (9 ends of clauses, 12 ends of sentences, 16 voiced sounds and 3 double consonants). Procedures were as follows. The subjects exercised one-handed finger Braille, first. Exercise dialogues were two daily conversations in Japanese textbook for foreign beginners [10] (Total 23 sentences, 45 clauses, 241 characters). Teaching interface was presentation method B. Thus, it was considered that the subjects achieved the experienced level 2. In the experiment, tester displayed dot patterns of one sentence then the subjects dotted one-handed Finger Braille on the tester s fingers by observing the dot patterns. This was repeated 12 sentences from group 1 to group 3 per session. Experiment repeated three sessions. In the first session, sentences of group 1 were displayed by the existing pattern. Sentences of group 2 were displayed by the teaching pattern 3. Sentences of group 3 were displayed by the teaching pattern 1. In the second session, sentences of group 1 were displayed by the teaching pattern 3. Sentences of group 2 were displayed by the teaching pattern 1. Sentences of group 3 were displayed by the existing pattern. In the third session, sentences of group 1 were displayed by the teaching pattern 1. Sentences of group 2 were displayed by the existing pattern. Sentences of group 3 were displayed by the teaching pattern 3. The subjects were instructed to dot long at the end of clause and sentence and to dot shortly at the voiced sounds and double consonants in every teaching pattern. The tester put fingers on a pressure sensor sheet (Nitta Tactile Sensor System) which measured change of pressure by dotting. Sampling Frequency was 137 Hz. Measurement range was kpa. Sensibility was 527 Pa. After the experiment, the subjects answered the easiest teaching pattern and the most difficult teaching pattern to dot prosody of finger Braille. 4.2 Results First, we calculated duration of code, duration of pause and finger load for each dotting by change of pressure (see Fig. 14). Fig. 15 shows mean of duration of code for each dot condition (no condition, voiced sound and double consonant, end of clause and sentence). Fig. 16 shows mean of duration of pause for each kind of pause (inner-character, inter-character, inter-column, inter-clause). Accuracy of dotting by the subjects was 100%. Dotting speed by the subjects was 43.2 characters/min (S.D.=12.0), one-third of dotting speed by finger Braille interpreter (127.1 characters/min, S.D.=16.6) [7]. 62

9 Fig. 17 Answer of the easiest teaching pattern and the most difficult teaching pattern Fig. 14 Calculation of duration and finger load Fig. 15 Duration of code as a function of dot condition Fig. 16 Duration of pause as a function of kind of pause After the experiment, the subjects answered the easiest teaching pattern and the most difficult teaching pattern to dot prosody of finger Braille. Fig. 17 illustrates the results of answers. As the easiest teaching pattern, 60% of answers were teaching pattern 3. As the most difficult teaching pattern, none of answers were teaching pattern Discussion To analyze features of dotting by subjects, the analyses of variances (ANOVA) of duration of code and pause were carried out. The duration of code creates a 3 3 within-subjects design: dot condition (no condition, voiced sound and double consonant, end of clause and sentence), teaching pattern (existing pattern, teaching pattern 3, teaching pattern 1). A 3 3 within-subjects analysis of variances (ANOVA) revealed a significant main effect of dot condition (p<.001). Main effect of teaching pattern and interaction term of dot condition and teaching pattern were not significant. A Scheffe test on the dot condition factor revealed that: the duration of code of voiced sound and double consonant was significantly shorter than the other durations (p<.001); the duration of code of end of clause and sentence was significantly longer than the other durations (p<.001). The duration of pause creates a 4 3 within-subjects design: kind of pause (inner-character, inter-character, inter-column, inter-clause), teaching pattern (existing pattern, teaching pattern 3, teaching pattern 1). A 4 3 within-subjects analysis of variances (ANOVA) revealed a significant main effect of kind of pause (p<.001). Main effect of teaching pattern and interaction term of kind of pause and teaching pattern were not significant. A Scheffe test on the kind of pause factor revealed that: the duration of pause of inter-clause was significantly longer than the other durations (p<.001); the duration of pause of 63

10 inner-character was significantly shorter than the other durations (p<.001). These results showed that: the subjects dotted long at the end of clause and sentence; the subjects dotted shortly at the voiced sound and double consonant; the subjects paused long between clauses and did not pause long during a clause. Thus subjects could dot prosody of Finger Braille. Main effect of teaching pattern was not significant, but the 60 % of subjects answered that teaching pattern 3 was the most visible teaching pattern. Therefore, it was considered that subjects could dot prosody of Finger Braille and the yellow pattern of teaching pattern 3 was the most visible and easiest teaching pattern. 5. Conclusion In this paper, we designed three teaching patterns which taught prosody of finger Braille (dot conditions about duration of dotting). And two questionnaires and an evaluation experiment of the teaching patterns were carried out. As the results, it was considered that the yellow pattern of teaching pattern 3 was the most visible and easiest teaching pattern and subjects could dot prosody of finger Braille. As our future plans, we will evaluate that dotting Finger Braille by non-disabled people with teaching of prosody was easy to recognize for deafblind people. And we are going to improve input interface of the teaching system. References: [1] Matsuda Y., Sakuma I., Jimbo Y. et al. Analysis of emotional expression of finger Braille. IFMBE Proceedings of the 7th Asian-Pacific Conference on Medical and Biological Engineering (Vol. 19), 2008: [2] Matsuda Y., Sakuma I., Jimbo Y., et al. Study on emotional communication in skin contact emotional expression experiment by finger Braille interpreters. Transaction of Human Interface Society, November 2008, 10(4): [3] Miyagi M., Nishida M., Horiuchi Y. et al. Investigation on effect of prosody in finger Braille. Proceedings of the 10th International Conference on Computers Helping People with Special Needs, July 2006: [4] An S. S., Jeon J. W., Lee S., et al. A pair of wireless Braille-based chording gloves. Proceedings of 9th International Conference on Computers Helping People with Special Needs, July 2004: [5] Amemiaya T., Hirota K. and Hirose M. OBOE: Oboe-like Braille interface for outdoor environment. Proceedings of 9th International Conference on Computers Helping People with Special Needs, July 2004: [6] Matsuda Y., Isomura T., Sakuma I., et al. Finger Braille teaching system for people who communicate with deaf-blind people. Proceedings of the 2007 IEEE International Conference on Mechatronics and Automation, August 2007: [7] Matsuda Y., Sakuma I., Jimbo Y., et al. Development of finger Braille teaching system teaching of dotting finger and position using speech recognition. Journal of the Society of Life Support Technology, September 2007, 19(3): [8] Matsuda Y., Sakuma I., Jimbo Y., et al. Finger Braille recognition system for people who communicate with deaf-blind people. Proceedings of the 2008 IEEE International Conference on Mechatronics and Automation, August 2008: WE2-2. [9] Matsuda Y., Sakuma I., Jimbo Y., et al. Study on teaching of the way to dot of finger Braille teaching of dotting finger and position of monosyllable. Transaction of Human Interface Society, August 2005, 7(3): [10] 3A Corporation. Minna no Nihongo I Honsatsu (main textbook), (Edited by Yunflyer, Holly) 64