FREQUENCY COMPRESSION AND FREQUENCY SHIFTING FOR THE HEARING IMPAIRED

FREQUENCY COMPRESSION AND FREQUENCY SHIFTING FOR THE HEARING IMPAIRED Francisco J. Fraga, Alan M. Marotta National Institute of Telecommunications, Santa Rita do Sapucaí - MG, Brazil Abstract A considerable percentage of listeners with severe hearing loss have audiograms where the losses are high for high frequencies and low for low frequencies. For these patients, lowering the speech spectrum to the frequencies where there is some residual hearing could be a good solution to be implemented for digital hearing aids. In this paper we have presented two different frequency lowering algorithms: frequency compression and frequency shifting. Results of subjective intelligibility tests have shown a slight better performance of the frequency shifting method relatively to the frequency compression method, although their performance remarkably depends on which are the specific phonemes that are being processed by these two algorithms. Key Words digital hearing aids, frequency lowering 1. Introduction There are several kinds of hearing impairment. The origin of the sensorineural hearing losses can be due to defects in the cochlea, auditory nerve or both. These problems reduce the dynamic range of hearing. The threshold of hearing is elevated, but the threshold of discomfort (at which the loudness become uncomfortable) is almost the same as for normal hearing listeners, or even may be lower. For some range of frequencies, the threshold of hearing is so high than it is equal to the threshold of discomfort, i.e., it is impossible for the listener hearing any sound at those frequencies. Hearing loss is more common for high frequency and mid frequency sounds (1 to 3 khz) than for low frequency. Frequently, there are only small losses at low frequencies (below 1 khz) but almost absolute deafness above 1.5 or 2 khz. These facts lead researchers to lower the spectrum of speech in order to match the residual low frequency hearing of listeners with high frequency impairments. Slow playback, vocoding, and zero crossing rate division are some of the methods that have been employed in the last decades. All of these methods involve signal distortion, more or less noticeable, generally depending on the amount of the frequency shifting. Many of the lowering schemes have altered perceptually important characteristics of speech, such as temporal and rhythmic patterns, pitch and durations of segmental elements. Hicks et al. [1] have done one of the most remarkable investigations about frequency lowering. Their technique involve pitch synchronous, monotonic compression of the short term spectral envelope, while at the same time avoiding some of the above-described problems observed in the other methods. Reed et al. [2] have conducted consonant discrimination experiments on normal hearing listeners. They have observed that Hick s frequency lowering scheme presented better performance for fricative and affricate sounds if compared with low pass filtering to an equivalent bandwidth. On the other hand, the performance of the low pass filtering was better for vowels, semivowels and nasal sounds. For plosive sounds, both methods have shown similar results. In general, the performance on the best frequency lowering conditions was almost the same to that obtained on low pass filtering to an equivalent bandwidth. Further, Reed et al. [3] have extended the results of Hick s et al. system to listeners with high frequency impairment. In general, the performance of the impaired subjects was inferior to that obtained by normal subjects. Few years ago, Nelson and Revoile [4] have discovered that relative to the normal hearing listeners, those with moderate to severe hearing loss required approximately double the peak to valley depth for detection of spectral peaks in bands of noise when signals have a high numbers of peaks per octave. Findings revealed that detection of spectral peaks in noise is significantly related to consonant identification abilities in listeners with moderate to severe hearing loss. All previous mentioned frequency-lowering schemes compress the speech spectrum into a narrower band of frequencies, increasing the number of peaks per octave while maintaining the peak to valley depth. According to Nelson s and Revoile s investigation, applying sharpening processing to a frequency lowered speech may allow better detection of spectral peaks and better consonant identification. Recently, Muñoz et al. [5] have combined sharpening (i.e., increasing the peak to valley depth) and frequency compression. They have demonstrated that the 417-808 352

processed speech improved the understanding of fricative and affricate sounds, while providing no significant change in identification of vowels and other sounds by listeners with severe high frequency hearing loss. Based on Nelson s and Revoile s investigation, we hypothesize that the relatively poor performance of Hick sand Muñoz s frequency lowering schemes is due to the increasing of the numbers of peaks per octave, which is inherent to the frequency compression method used in these systems. In this paper, we propose a new frequency-lowering algorithm that does not increase the number of peaks per octave because it uses frequency shifting instead of frequency compression. Furthermore, the frequency shifting is applied only for fricative and affricate sounds, leaving all others types of sounds untouched, because it is only for fricative sounds that the frequency lowering technique brings real benefits as have been demonstrated by all the previous mentioned works. We have also implemented a frequency compression algorithm based on Hick s [1] and Muñoz s [5] ideas. Preliminary results of subjective preference (considering only the qualitative aspect of the processed speech) have confirmed our hypothesis about the better performance of the frequency shifting method compared to the frequency compression method. But further subjective intelligibility tests over 20 subjects have clearly shown that their performance (now considering only the intelligible aspect of the speech) remarkably depends on which are the specific phonemes that are being processed by these two algorithms. the losses are classified as mild. Moderate losses are those which are greater than 40 db but until inferior to 70 db. From 71 to 90 db, we consider that the patient have severe hearing losses and more than 95 db of loss is classified as profound [6]. The threshold of discomfort, for normal or impaired listeners, is always below 120 db SPL. Indeed, commonly the threshold of discomfort for the impaired subjects is lower than for normal hearing subjects. Although less common, some audiograms bring both the threshold of discomfort and the threshold of hearing [7], as we can observe in Fig. 1. In this figure, the points of the audiogram corresponding to the right ear are signaled with a round mark and those corresponding to the left ear are signaled with an X mark. These marks are worldwide used in this way by audiologists [6]. The dynamic range of listening for each frequency is the threshold of discomfort minus the threshold of hearing. 2. Methodology A. Audiometric data acquisition and processing The first step of both frequency-lowering algorithms consists in audiometric data acquisition of the impaired subject. The audiometric exam is employed for measuring the degree of the hearing impairment of a given patient. In this exam, the listener is submitted to a perception test by continuously varying the sound pressure level (SPL) of a pure sinusoidal tone in a discrete frequency scale. The frequency values most frequently used are 250 Hz, 500 Hz, 1 khz, 2 khz, 4 khz, 6 khz and 8 khz. For each of these frequencies, the minimum SPL in db for which the patient is capable of perceiving the sound is registered in a graph. The audiogram is the result of the audiometric exam, which is presented by a graph with the values in db SPL for each of the discrete frequencies. This graph is done separately for each ear of the subject. Since the level of 0 db SPL is considered the minimum sound pressure level for normal hearing, the positive values in db registered on the vertical axis of the audiogram can be considered as the hearing losses of the patient s ear. If the losses are equal or inferior to 20 db, the subject is considered as having normal hearing. From 21 to 40 db, Fig. 1: Ski-slope losses case Based on the acquired audiometric data, the algorithm analyses the range of frequencies where there is still some residual hearing. The criterion used is the following: first, it is verified if the patient have a ski slope kind of losses, i.e., if the losses are increasing with frequency. Only patients with this type of impairment can be aided by any frequency lowering method. After that, the first frequency where there is a profound loss is determined. If this frequency is between 1.2 khz and 3.4 khz, a destination frequency to which the high frequency spectrum will be shifted is calculated. Otherwise, no frequency shifting is needed (residual hearing above 3.4 khz) or profitable (residual hearing below 1.2 khz). This destination frequency is considered as the geometrical mean between 900 Hz and the highest frequency where there is still some residual hearing. The geometrical mean was empirically chosen because it provides a good tradeoff between minimum spectrum distortion and maximum 353

residual hearing profit. In order to obtain more accuracy in the losses thresholds, the points of the audiogram are linearly interpolated. B. Speech data acquisition and processing The speech signal is sampled at a 16 khz rate and Hamming windowed with 25 msec windows. These windows are 50% overlapped, what means that the signal is analyzed at a frame rate that is the inverse of 12.5 msec. A 1024-point FFT is used for representing the high resolution short time speech spectrum in the frequency domain. If in the previous audiometric data analysis a ski slope kind of loss was detected and the frequency-shifting criterion was matched, a destination frequency have already been determined. Then, we have to find out (in a frame by frame basis) if the short time speech spectrum presents significant information at high frequencies that justify the frequency shifting operation. The criterion used for shifting or not the short time spectrum of each speech frame is based in a threshold. When the signal has high energy in high frequencies the algorithm shifts this high frequency information to lower frequencies. The threshold is set for suppressing the processing of all vowels, nasals and the semivowels, while activating the frequency transposition for fricatives and affricates. To decide which part of the spectrum will be shifted, the energy of 500 Hz bandwidth windows are calculated with 100 Hz spacing, from 1 khz to 8 khz. This is done with the aim of find out an origin frequency. The origin frequency is the frequency 100 Hz below the beginning of the 500 Hz bandwidth window that have maximum energy. The part of the spectrum that will be transposed corresponds to the range of all frequencies above the origin frequency. This empirical criterion guarantees that the unavoidable distortion due to the frequency lowering operation will be profitable. Because the most important part of the high energy information will be shifted to the limited range of frequencies that are above 1 khz (therefore maintaining untouched the low frequency information) but still below the highest frequency where the patient presents residual hearing. For comparison, the Hick s frequency compression scheme was already implemented, but now only when the same frequency lowering criterion (high/low frequency energy ratio) used for transposition was matched, i. e., only for fricatives and affricates. The frequency compression was done by means of an equation defined in [2]. But in practice, it is more useful to implement the inverse equation, which is f f IN S 1 1 = tan π 1 a f tan Kπ 1 + a f OUT S where f IN is the original frequency, f OUT is the corresponding compressed frequency, K is the frequency compression factor, a is the warping parameter and f S is (1) the sampling rate. For minimum distortion at low frequencies, the warping parameter must be chosen as being a = (K 1)/(K+1). The compression factor K was determined according to the degree of loss presented by the listener. Fig. 2 shows the curves of equation (1) for K = 2, 3 and 4. In this figure we can see that the low frequency information (below 1000 Hz) is barely compressed. After frequency shifting or compressing (if it occurs), the FFT spectrum of each speech frame is multiplied by the gain factor, which is calculated for each frequency in order to full compensate the hearing loss, unless the amplified sound pressure level exceed the threshold of discomfort. In this case, the gain factor is limited to the amount required for maintaining the loudness below the threshold of discomfort. The way we implemented this spectral shaping process is similar to that described in [8]. This last step was still under development in our digital hearing aids system. Fig. 2: Input vs. output frequency curves Fig. 3: Comparison of frequency lowering schemes 354

Part (a) of Fig. 3 illustrates the original FFT spectrum of a speech frame, in part (b) the same frame is shown compressed by a factor K = 4 and part (c) presents the frame after frequency shifting. It is important to observe that in the last case (frequency shifting) the shape of the spectrum is preserved, what does not occur in the case of frequency compression, where we can clearly note a great amount of shape distortion, but still preserving the low frequency information. These preliminary results indicate that the frequency shifting method was preferred by the listeners when compared with the frequency compression method. But it is important to remark that the subjective difference between the low pass filtered signal, the frequency compressed signal and the frequency-shifted signal is very slight, as perceived by normal listeners. 3. Results A. Preliminary Qualitative Tests The two frequency lowering algorithms were not already tested with hearing impaired subjects because they final spectral shaping part are not completely developed, as mentioned in the last paragraph of the previous section. But we got some preliminary results with normal listeners, considering first only the qualitative aspect of the processed speech. In this case, a simple low pass filtering process simulates the losses above the frequency where there is no more residual hearing. In this preliminary qualitative test, this cutoff frequency was fixed to 2 khz. The experiment we have carried out consists of submitting the speech signal to the two frequency lowering algorithms. After that, the resulted signals were listened by two normal hearing subjects, one man and one woman. The listeners do not know anything about the origin of the signals and are asked for ranking the signals according to their intelligibility. In this preliminary test, only two speech signals were submitted to the algorithms. The original and processed spectrograms of one of these speech signals (pronunciation of the words loose management ) are shown in Fig. 4, where we can appreciate again the visual difference between the two frequency lowering algorithms. According to the prevision, only the fricative speech sounds were frequency lowered in both algorithms. The unique exception is the phone [ l ], which is not fricative but lateral approximant. But in this case, its pronunciation had high frequency energy, as we can observe in the spectrogram of the original speech signal. The preferences of the listeners were listed in Table 1. In this table, Signal 1 is the Portuguese word pensando (which means thinking ) and Signal 2 is the English words loose management. Table 1: Listener s preferences Speech signal Man Woman Signal 1 low pass 1 st 3 rd Signal 1 compr. 3 rd 2 nd Signal 1 shifted 2 nd 1 st Signal 2 low pass 2 nd 2 nd Signal 2 compr. 3 rd 3 rd Signal 2 shifted 1 st 1 st Fig. 4: Spectrograms of loose management A. Detailed Intelligibility Tests The intelligibility test was performed over 20 listeners, 15 male and 5 female. Each of them heard 36 syllables randomly chosen from a database formed by the utterances of 6 speakers, 3 female and 3 male. The original database was formed by 21 different CV phonetic syllables, each of them is composed by one of the 7 most commonly used fricative sounds of the Portuguese language ( [ ], [ ], [ ], [ ], [ ], [ ], [ ] ) and by one of these 3 vowels: [ ], [ ] or [ ]. These syllables were pronounced once by the 6 speakers, therefore the original database was formed by 126 utterances. Each of these utterances generates 9 different processed WAVE files: original syllable, frequency compressed syllable and frequency shifted syllable, passed through 3 different low pass filters with cutoff frequencies of 1.5, 2 and 2.5 khz, forming a final speech database composed by 1134 WAVE files. After have heard 3 times a randomly chosen phonetic syllable from the final database (without any additional information than their sounds), the listener must choose one syllable from a list of 7 possibilities. The vowel is the correct one in these 7 syllables, which means that the decision will be made based only in the acoustic properties of the processed fricative sounds. The results of this test are shown in Table 2, where the column None means no processing further than low pass filtering, Compression means frequency compression and Shifting means frequency shifting. In the first column we have all the possible fricatives for each of the 3 filter cutoff 355

frequencies. In the table, the numbers signaled in boldface correspond to the greatest percentage of correct decisions made for each type of processing. Because of the random choice of the syllables that were presented to the listeners, there were some syllables that were less listened than others. But each of the 63 different processed fricatives corresponding to the cells of Table 2 was presented at least 5 times and any of them was presented more than 15 times. Table 2: Listener s correct decisions (%) Processed None Compression Shifting Syllable [ ] 1500 61,5 72,7 62,5 [ ] 2000 40,0 44,4 69,2 [ ] 2500 85,7 53,3 58,3 [ ] 1500 78,6 80,0 81,8 [ ] 2000 100,0 71,4 66,7 [ ] 2500 77,8 61,5 90,9 [ ] 1500 25,0 28,6 33,3 [ ] 2000 50,0 81,8 86,7 [ ] 2500 69,2 62,5 77,8 [ ] 1500 0,0 20,0 45,5 [ ] 2000 44,4 62,5 55,6 [ ] 2500 77,8 100,0 84,6 [ ] 1500 53,8 50,0 8,3 [ ] 2000 73,3 36,4 33,3 [ ] 2500 76,9 41,7 25,0 [ ] 1500 57,1 46,7 75,0 [ ] 2000 70,0 60,0 40,0 [ ] 2500 44,4 60,0 33,3 [ ] 1500 66,7 40,0 12,5 [ ] 2000 46,2 21,4 38,5 [ ] 2500 55,6 38,5 75,0 These results are difficult to analyze if we consider the set of syllables as a whole. But it is interesting to analyze each fricative sound in particular. For example, we can conclude from the results that for the phone [ ] the better is to do no further processing with it, but if we consider the case of the phone [ ] we conclude just the opposite: no processing leads to 0.0 % of intelligibility when the highest audible frequency is 1.5 khz! In the case of the fricative sound [ ], the better solution is to apply our frequency shifting proposed algorithm. For all other situations, the optimal solution depends on the specific phone and cutoff frequency considered. signals is large. But for the impaired subject, that never had any perception of sounds with frequencies above 2 khz, may be the difference between the processed signals was not so slight. Relatively to the results of the intelligibility test, we can conclude that if we incorporate a simple automatic phoneme classifier in the system, it is possible to choose the better frequency lowering algorithm to be applied for each specific phone, given the maximum frequency where there is some residual hearing. This is not difficult to do, considering the advances observed in the performance of automatic phoneme recognition algorithms over the last years. Finally, it is important to remark that, with all the processing being done in the frequency domain, both algorithms have demonstrated to be fast enough for enabling the usage in real time applications. References [1] B. L. Hicks, L. D. Braida, and N. I. Durlach, Pitch invariant frequency lowering with nonuniform spectral compression, in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (IEEE New York), pp. 121-124, 1981. [2] C. M. Reed, B. L. Hicks, L. D. Braida, and N. I. Durlach, Discrimination of speech processed by lowpass filtering and pitch-invariant frequency lowering, J. Acoust. Soc. Am, vol. 74, pp. 409-419, 1983. [3] C. M. Reed, K. I. Schultz, L. D. Braida, N. I. Durlach, Discrimination and identification of frequency-lowered speech in listeners with high-frequency hearing impairment, J. Acoust. Soc. Am, vol. 78, pp. 2139-2141, 1985. [4] P. Nelson, and S. Revoile, Detection of spectral peaks in noise: Effects of hearing loss and frequency regions, J. Acoust. Soc. Am, 1998. [5] C. M. Aguilera Muñoz, B. N. Peggy, J. C. Rutledge, A. Gago, Frequency lowering processing for listeners with significant hearing loss, IEEE, pp. 741-744, 1999. [6] S. Frota, Fundamentos em Fonoaudiologia, 1 st ed., vol. 1. Guanabara Koogan, 2001, pp. 40-59. [7] Y. A. Alsaka, B. McLean, Spectral Shaping for the Hearing Impaired, IEEE, pp. 103-106, 1996 [8] J. C. Tejero-Calado, B. N. Peggy, J. C. Rutledge, Combination compression and linear gain processing for digital hearing aids, IEEE, pp. 3140-3143, 1998. 4. Conclusion It is necessary to finish the spectral shaping part of the system in order to submit the processed signals to hearing impaired listeners. The slight difference in the quality observed among the processed signals may be due to the fact that the difference between the original signal (with frequencies up to 8 khz) and the low pass filtered (2 khz) 356