Proceedings of Meetings on Acoustics

Transcription

1 Proceedings of Meetings on Acoustics Volume 19, 13 ICA 13 Montreal Montreal, Canada 2-7 June 13 Psychological and Physiological Acoustics Session 2pPPb: Speech. Attention, and Impairment (Poster Session) 2pPPb3. Speech intelligibility of hearing impaired participants in long-term training of bone-conducted ultrasonic hearing aid Toshie Matsui*, Ryota Shimokura, Tadashi Nishimura, Hiroshi Hosoi and Seiji Nakagawa *Corresponding author's address: Department of Otorhinolaryngology - Head and Neck Surgery, Nara Medical University, Shijocho 8, Kashihara, , Nara, Japan, tomatsui@naramed-u.ac.jp Bone-conducted ultrasonic hearing aid (BCUHA) system is the unique device to provide the auditory sensation to profoundly hearing impaired persons without any surgical operations. To clarify effects of a long-term hearing training with this device, two deaf participants engaged the BCUHA training for more than 9 months. They were trained to use BCUHA through repetition of sentences read aloud, free conversation and singing, and then they participated in word recognition tests and monosyllable identification tests. Both participants showed that they could recognize words above chance using auditory sensation only provided by BCUHA if alternatives or context were presented to them. In addition, it was observed that monosyllable intelligibility score with both of auditory and visual cue had much increased with the days of training than the score with the auditory only cue and that with the visual only cue. The result suggests that the long-term training with BCUHA achieves efficient integration of the auditory and the visual cue of speech such as cochlear implant users showed in previous studies. Published by the Acoustical Society of America through the American Institute of Physics 13 Acoustical Society of America [DOI: / ] Received 22 Jan 13; published 2 Jun 13 Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 1

2 INTRODUCTION Cochlear implants have recently become the most common device to restore hearing sensation to the profoundly hearing impaired. A surgical operation is necessary to use the cochlear implant. It has been reported that reoperation is very difficult because the initial operation perforates the cochlear duct and to insert an electrode. The development of a hearing aid not requiring a surgical operation would greatly benefit hearing impaired people for whom surgical treatment is not an option. The bone-conducted ultrasound hearing aid (BCUHA) is currently a unique method to fill this requirement. First, Gavreau discovered that ultrasound could be perceived through bone-conduction (Gavreau, 1948). Pumphrey subsequently confirmed that bone-conducted ultrasound up to khz was perceivable (Pumphrey, 195). It was also founded that bone-conducted ultrasound modulated by speech signals was perceivable not only for normal hearing, but also for the profoundly hearing impaired (Lenhardt et al., 1991). It has been demonstrated that the boneconducted ultrasonic speech sound is processed in the auditory cortex of both normal hearing and the profoundly hearing impaired (Hosoi et al., 1998). Based on these research results, bone-conducted ultrasound hearing aids for most profoundly hearing impaired people have been developed (Nakagawa et al., 6), however, most of the investigations of speech intelligibility (Okamoto et al., 6; Kagomiya and Nakagawa, 1) and the perception mechanism of bone-conducted ultrasound (Nishimura et al., 3; Nishimura et al., 11) are currently limited to normal hearing. This study carried out long-term training for two profoundly hearing impaired persons using BCUHAs. A speech sound recognition task and a monosyllable identification task were performed as an index of the effects of training. METHODS AND MATERIALS Participants Two profoundly hearing-impaired listeners participated in the training. Participant 1 (P1) used HiSonic (Hearing innovation, Inc.), which had originally been developed as a clinical device for tinnitus suppression. Participant 2 (P2) used AIST-BCUHA-3 (National Institute of Advanced Industrial Science and Technology). The output level was adjusted by the participants themselves for the best hearing of speech in each session. The training schedules were individually decided upon depending on the participants condition. Each training session was carried out individually. P1 participated in 35 sessions that were conducted every other week from December 1 to July 12. P2 participated in 43 sessions that were conducted once or twice a week from August 11 to July 12. The training sessions consisted of word recognition tasks (multiple choice paradigm), Japanese monosyllable identification tasks, word recognition tasks (open questions), free conversation, and singing. Each session was completed within one and a half hours. The results from the word recognition tasks and Japanese monosyllable identification tasks were finally analyzed. TABLE 1. All trials of the word recognition task were carried out without any visual cue. AFC; alternative forced choice paradigm, Diff; Different. Task Difference between alternatives Paradigm Session No. Number of morae P1 P2 Consonant Vowel Word recognition 1 Different Different Diff/Same 2AFC 1- NA Word recognition 2 Different Diff/Same Same 3AFC Word recognition 3 Different Same Same 4AFC Monosyllable identification Audio only NA 5-43 Visual only Audio and visual Word recognition task Tasks were carried out face to face. A speech therapist presented all of options visually and aurally to the participant and then uttered one word from among options with their mouth covered. The therapist repeatedly uttered the word until the participant responded orally. Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 2

3 The word recognition task was separated in three types to observe the effect of the factors below. The number of morae was limited to 2-6 in all types of tasks. P1 participated in all three types of tasks. P2 participated in type 2 and 3 tasks. Tasks were assigned randomly during each session in the period indicated in Table 1. Feedback of correct choices were given after their response in each trial. 1) Number of morae of words (2AFC). This condition was set to clarify whether the number of morae was available as a cue of recognition. Therefore, the task included two condition; same number of morae condition and different number of morae condition. In the same number of morae condition, options consisted of Japanese words with same number of morae such as zou (elephant, 2 morae) and wani (crocodile, 2 morae). In different number of morae condition, words of options had different number of morae such as isu (chair, 2 morae) and tsukue (table, 3 morae). The trials were presented in a two alternative forced choice (2AFC) paradigm. 2) Vowel pattern of words (3AFC). This condition was set to clarify whether vowel pattern of a word were available as a cue of recognition. In a manner similar to the above, the task included two conditions; same vowel pattern condition and different vowel pattern condition. In the same vowel pattern condition, options consisted of words with the same vowel pattern such as narawashi (custom, 4 morae), sawagani (freshwater crab, 4 morae) and kawahagi (threadsail filefish, 4 morae). In the different vowel pattern condition, the options consisted of words which had different vowel patterns such as kusayabu (tuft of grass, 4 morae), yomibito (author of verse, 4 morae) and setoyaki (Seto pottery, 4 morae). This task was presented in a 3AFC paradigm. 3) Vowel pattern of words (4AFC). This only consisted of the same vowel pattern condition which appears in task 2. This task was, however, presented in a 4AFC paradigm. Monosyllable identification task The task was carried out face to face in a manner similar to the word recognition tasks. The task required participants to identify 5 Japanese monosyllables that consist of consonant and vowel monosyllables and vowel only monosyllables. Table 2 shows their phonemes and phonetic values. The task consisted of three conditions; audio condition: audio only cue through BCUHA was given for identification; visual condition: visual only cue (lip reading) was given; audio-visual condition: both audio and visual cues were provided. TABLE 2. Phonemes and phonetic values of 5 Japanese syllables for monosyllable identification task. Nc: no consonant. nc /k/ /s/ /t/ /n/ /h/ /m/ /y/ /r/ /w/ /g/ /z/ /d/ /b/ /a/ a ka sa ta na ha ma ja ɾa wa ga da ba /i/ i kʲi ʃi tʃi ɲi çi mi ɾʲi ʤi /u/ ɯ kɯ sɯ tsɯ ɸu mɯ jɯ ɾɯ dzɯ /e/ e ke se te ne he me de /o/ o ko so to no ho mo jo ɾo go do P1 participated in tasks with the visual and audio-visual conditions. P2 participated in tasks featuring all of three conditions. The order of monosyllable was quasi-randomized in each trial. Participants were provided with the results after all of the trials. The tasks were consecutively carried out during the period indicated in Table 1. Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 3

4 RESULTS Word recognition task The correct response rate and the mean of the number of times an utterance was repeated before a response was given by each participant were analyzed. Figure 1 illustrates the results from P1; Fig 1(a) shows the correct response rate and Fig. 1(b) shows the mean of number of repeated utterances for each condition and type of task. (136) (19) (a) Diff Same Diff Same Mora Vowel (2AFC) chance level (123) (19) (5) (3AFC) (4AFC) Mean of utterance times (b) * Diff Same Diff Same Mora Vowel (2AFC) (3AFC) (4AFC) FIGURE 1. The correct response rate and mean of utterance repetition for the word recognition tasks performed by P1. Numbers in parentheses indicate the number of trials. Error bar: standard error. The results from all of the tasks and conditions by P1 showed a correct response rate above chance (.5,.33,.25, respectively. Chi-square test, p <.1 for each task and condition). However, the number of morae of words did not affect the correct response rate significantly (Pearson s chi-square test, χ 2 (1) =.336, p =.562). The vowel pattern of words did not provide effective cues to discriminate the words either (Pearson s chi-square test, χ 2 (1) = 2.44, p =.118). Conversely, the number of repetitions before P1 s response received significant influence from the number of morae, and vowel pattern, of words. In same number of morae condition of task type 1, the number of repetitions before P1 s response were significantly less than in the different number of morae condition (two-tailed t-test, t(243) = -2.3, p =.2). In the same vowel pattern condition, mean times of utterance times was significantly less than in the different vowel pattern condition (two-tailed t-test, t(212) = 5.41, p <.1). Figure 2 illustrates the result from P2; Fig 2(a) shows the correct response rate and Fig 2(b) shows the mean of number of repetitions for each condition and type of task that P2 participated in. (a) * chance level (77) (31) (49) Diff Same Vowel (3AFC) (4AFC) Mean of utterance times (b) Diff Same Vowel (3AFC) (4AFC) FIGURE 2. The correct response rate and mean of utterance repetition for the word recognition tasks performed by P2. Numbers in parentheses indicate the number of trials. Error bar: standard error. The results from all of the tasks and conditions by P2 showed correct response rate above chance (.33,.25, respectively. Chi-square test, p <.1 for each task and each condition). In the same number of morae condition, the correct response rate was significantly higher than that in different number of morae condition (Pearson s chi-square test, χ 2 (1) = 9.51, p <.5). The number of repetitions before P2 s response did not display any significant difference between same vowel pattern condition and different one (two-tailed t-test, t(16) = -1.45, p =.15). Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 4

5 Monosyllable identification task Although not all sessions, the monosyllable identification task was routinely conducted during the training periods indicated in Table 1. Therefore, the correct response rate over all sessions and the transition of the correct response rate day by day were analyzed for each subject. Figure 3(a) illustrates the correct response rate for the visual and audio-visual conditions from all sessions of P1. The correct response rate was calculated for the consonant part, vowel part and syllable unit. The correct response rate of the audio condition and audio-visual condition are also compared. The correct response rate of consonant part and syllable unit between two conditions shows significant differences (two-tailed t-test, consonant, t(26) = -2.88, p =.79; syllable, t(26) = -3.28, p =.29). (a) P1 p <.1 (t-test) V A+V V A+V V A+V consonant vowel syllable (b) P2 * p <.5 p <.1 (Tuckey-Kramer HSD test) * A V A+V A V A+V A V A+V consonant vowel syllable FIGURE 3. The correct response rate for monosyllable identification task. The left panel illustrates the result of P1, and the right panel illustrates that of P2. Each bar corresponds to the condition. Figure 4 shows the relationship between the correct response rate and the number of training day counted forward from the first day the monosyllable identification task began. Each panel corresponds to the timelines of the bars in Figure 3(a). Solid lines, broken lines and equations represent regression lines calculated by the least mean square method. The regression coefficient of the consonant part and syllable unit in the audio-visual condition showed a marginally significant higher value than (two-tailed t-test, consonant, t(13) = 1.81, p =.931; syllable, t(13) = 2., p =.667). Consonant Vowel Syllable = *day = *day Visual Visual = *day = *day = *day Visual = *day Number of training days since the first day of monosyllable identification task FIGURE 4. The relationship between the correct response rate and training days by P1. The training days count from the first day of the monosyllable identification task. Figure 3(b) illustrates the correct response rate for each condition from all sessions performed by P2. The correct response rate was calculated for the consonant part, the vowel part and the syllable unit. Two-way ANOVA revealed a significant effect of the condition on the consonant part, the vowel part, and the syllable unit respectively (consonant, F(2,54) = 54.3, p <.1; vowel, F(2,54) = 425.6, p <.1; syllable, F(2,54) = 19.9, p <.1). Tucky- Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 5

6 Kramer HSD post hoc analysis showed significant difference in the correct response rate of the consonant part among all of the conditions (respectively, p <.1). In the vowel part, a significant difference between the correct response rate of the audio-visual condition and that of the other conditions was observed (respectively, p <.1). In the syllable unit, a significant difference among all of conditions was apparent (respectively, p <.1). A regression analysis was conducted on the correct response rate of all of the conditions by consonant, vowel, and syllable unit. The regression coefficient of the syllable unit in the visual condition showed significant deviation from zero (two-tailed t-test, t(17) = 2.18, p =.428). The consonant part and syllable unit in audio-visual condition indicated significant derivation from zero as well (two-tailed t-test, consonant, t(23) = 6.11, p <.1; syllable, t(23) = 4.67, p <.1). Furthermore, the difference between the regression coefficients of the syllable unit in the visual condition and that in the audio-visual condition was significant (two-tailed t-test, t(42) = 21.4, p <.1). Consonant = *day Visual = *day Vowel = *day Visual = *day Syllable = *day Visual = *day Audio = *day Audio = *day Audio = *day Number of training daya since the first day of monosyllable identification task FIGURE 5. The relationship between the correct response rate and training days by P2. The training days count from the first day of the monosyllable identification task. DISCUSSION The data of the word recognition task for P1 in the current research includes part of the data from a previous study by Shimokura et al. Trials of the 5-alternative forced choice task in the previous research were removed from the analysis because neither the number of morae nor the vowel pattern of the options was controlled. As for the results of P1, the correct response rate of every condition was above chance. This means that the differences between words could be distinguished when options were under four and the options were exhibited beforehand. On the other hand, the lack of a significant difference in the correct response rate between same condition and different condition implies that a difference in the number of morae and the vowel pattern did not contribute to the discrimination of the word. A larger number of cues may not make discrimination of boneconducted ultrasonic speech easier. The mean number of repeated utterances before a response rather suggests that fewer cues facilitate the discrimination of words. It is speculated that the participant perceived bone-conducted ultrasonic speech in totally different manner to original air-conducted speech, therefore many differences between options might make it difficult to memorize words and compare between them. The results of P2 showed a correct response rate as much above chance as P1. Furthermore, the correct response rate of the same condition was significantly higher than that of different condition in the vowel pattern. It is considered that an increase in the number of features to be memorized and compared made discrimination more complicated in the same way as P1. However, it should be added that P2 was able to have a conversation comfortably with BCUHA. In short, these results suggests that these two profoundly hearing impaired persons could discriminate boneconducted ultrasonic speech with auditory cue only if prior options or context were given. The results from the monosyllable identification task show transmission efficiency of each modality and their combination. The results of both participants were similar. Using audio and visual cue simultaneously, monosyllables could be identified more precisely. The results depicted in Fig. 3 shows that the auditory cue from a hearing aid was largely effective for the consonant identification while a visual cue was basically enough for the vowel identification. However, as for the correct response rate for the syllable unit in the audio-visual condition, P1 remained under % and the P2 remain around 5%. In a previous study, monosyllable identification through BCUHA by a normal Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 6

7 listener showed an approximately % correct response rate in the best SN ratio in the audio-visual condition (Yamashita, et al., 1). In the audio only condition, the mean of the correct response rate of the normal listener was around %. The correct response rate of P2 was around 1% in the same condition. Even for P2 who could hold a comfortable conversation with the BCUHA, monosyllable identification was hard because of poor redundancy for judgment. Only in the visual condition did P1 and P2 show a higher correct response rate than that of normal listeners tested by Yamashita et al. (approximately %). Although lip reading was not particularly trained in our sessions, requirement in the everyday life may have raised their precision in lip reading. Figure 4 and 5 show the change of the results of the monosyllable identification task by BCUHA training. As is the case with the word recognition task, the results of both participants were similar. It was confirmed that the presentation of multiple modalities was efficient for speech sound discrimination based on the statistical test of the difference of regression coefficients on P2 s data. It is common with reports of rehabilitation for cochlear implant users that sensation from multiple modalities (auditory and vision) can achieve effective learning of speech sound (Rouger et al., 7; Kim et al., 9). Note that improvement of integration of auditory and vision is limited as is reported by research on the cochlear implant and the rehabilitation (Massida et al., 1). It is necessary to conduct a similar investigation for hearing impaired people with different conditions to estimate the limitation of training and to create a more effective rehabilitation program. SUMMARY Two types of experiment were performed for two profoundly hearing impaired persons to confirm the effects of the bone-conducted ultrasound hearing aid and hearing training. It was verified that even profoundly hearing impaired persons could discriminate words using only a bone-conducted ultrasound hearing aid if context was provided. In addition, it was confirmed that as with, training with both auditory and visual cues was effective for monosyllable identification. ACKNOWLEDGMENTS This research was supported by Grant-in-Aid for Scientific Research (B) No and Grant-in-Aid for Young Scientists (B) No of the Japan Society for the Promotion of Science. REFERENCES Gavreau, V. (1948). Audibillité de sons de frequence élevée, Compt. Rendu. 226, Hosoi, H., Imaizumi, S., Sakaguchi, T., Tonoike, M., and Murata, K. (1998). Activation of the auditory cortex by ultrasound, Lancet, 351, Kagomiya, T. and Nakagawa, S. (1). An evaluation of bone conducted ultrasonic hearing-aid regarding transmission of japanese prosodic phonemes, Proc. ICA 1, 972, 1-5. Kim, J., Davis, C., and Groot, C. (9). Speech identification in noise: Contribution of temporal, spectral, and visual speech cues, J. Acoust. Soc. Am., 126, Lenhardt, M. L., Skellett, R., Wang, P., and Clarke, A. M. (1991). Human ultrasonic speech perception, Science, 253, Massida Z., Belin, P., James C., and Rouger, J., Fraysse, B., Barone, P., and Deguine, O. (1). Voice discrimination in cochlear-implanted deaf subjects, Hearing Research, 275, Nakagawa, S., Okamoto, Y., and Fujisaka, Y. (6). Development of a bone-conducted ultrasonic hearing aid for the profoundly sensorineural deaf, Trans. Jpn. Soc. Med. Biol. Eng. 44, Nishimura, T., Nakagawa, S., Sakaguchi, T., and Hosoi, H. (3). Ultrasonic masker clarifies ultrasonic perception in man, Hearing Research, 175, Nishimura, T., Okayasu, T., Uratani, Y., Fukuda, H., Saito, O., and Hosoi, H. (11). Peripheral perception mechanism of ultrasonic hearing, Hearing Research, 277, Okamoto, Y., Nakagawa, S., Fujimoto, K., and Tonoike, M. (5). Intelligibility of bone-conducted ultrasonic speech, Hearing Research, 8, Pumphrey, R. J. (195). Upper limit of frequency for human hearing, Nature, 166, 571. Rouger, J., Deguine, O., Barone, P., Lagleyre, S., Fraysse, B., Deguine O., and Deneve S. (7). Evidence that cochlearimplanted deaf patients are better multisensory integrators, PNAS, 14, Shimokura, R., Fukuda, F., and Hosoi, H. (12). A case study of auditory rehabilitation in a profoundly deaf participant using a bone-conducted ultrasonic hearing aid, Behavioral Science Research, 5, Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 7

8 Yamashita, A., Nishimura, T., Nagatani, Y., Sakaguchi, T., Okayasu, T., Yanai, S., and Hosoi, H. (1). The effect of visual information in speech signals by bone-conducted ultrasound, NeuroReport, 21, Proceedings of Meetings on Acoustics, Vol. 19, 588 (13) Page 8