Speech enhancement based on harmonic estimation combined with MMSE to improve speech intelligibility for cochlear implant recipients

Transcription

1 INERSPEECH 1 August 4, 1, Stockholm, Sweden Speech enhancement based on harmonic estimation combined with MMSE to improve speech intelligibility for cochlear implant recipients Dongmei Wang, John H. L. Hansen CRSS Lab, Dept. Electrical Engineering he University of exas at Dallas Richardson, exas 58 dongmei.wang@utdallas.edu, john.hansen@utdallas.edu Abstract In this paper, a speech enhancement algorithm is proposed to improve the speech intelligibility for cochlear implant recipients. Our method is based on combination of harmonic estimation and traditional statistical method. raditional statistical based speech enhancement method is effective only for stationary noise suppression, but not non-stationary noise. o address more complex noise scenarios, we explore the harmonic structure of target speech to obtain a more accurate noise estimation. he estimated noise is then employed in the MMSE framework to obtain the gain function for recovering the target speech. Listening test experiments show a substantial speech intelligibility improvement for cochlear implant recipients in noisy environments. Index erms: speech enhancement, cochlear implant, harmonic structure, noise estimation, MMSE 1. Introduction Cochlear implant (CI) devices are able to assist the individuals with severely hearing loss to recover some level of hearing ability. Currently, CI recipients are able to achieve a relatively high level of speech intelligibility in quiet environments. However, when they present in noisy backgrounds, their speech intelligibility drops dramatically. Previous research has shown that the speech reception threshold (SR) of CI listeners is typically 15 to 5dB higher than normal hearing listeners in noisy backgrounds [1,, 3]. herefore, developing effective speech enhancement algorithms is essential to improve speech perception in noise for CI recipients. Speech is a type of over-redundant signal. Normal hearing listeners usually have no difficulties in understanding speech in noise, even at a negative signal-tonoise ratio (SNR) level, depending on noise types. On the contrary, CI recipients are only able to decode limited amounts of spectral and temporal information from the speech which are delivered by a few CI encoding channels (16 or ) [4, 5]. his leaves CI recipients unable to discriminate target speech from noise interference. o improve speech perception in noise for CI recipients, both single and multiple microphone based speech enhancement algorithms have been developed in previous research [6, ]. Multiple microphone speech enhancement algorithms are able to significantly increase speech intelligibility for CI recipients in both stationary and nonstationary noise. However, strict assumptions such as speech and noise sources must be spatially separated, have to be satisfied. In some real scenarios, such as a diffusive sound field, target speech and noise coming from the same direction, or if reverberation exists, multiple channel algorithms have very limited benefit. hus, single microphone speech enhancement to improve speech perception for CI recipients still remains as an open problem. In addition, single microphone algorithm can incorporate with multiple microphone algorithms to improve speech intelligibility in noise [8]. For single microphone based methods, existing research has been focused on i) developing noise reduction algorithm before the CI encoding [1, 9, 1, 11, 1] and ii) optimizing the CI channel selection to deliver higher SNR speech components [13, 14, 15, 16, 1]. In the former cases, noise reduction is performed as a pre-processing to improve speech representation. Such approaches include spectral subtraction method [1, 9], wiener filter [18], subspace methods [1]. In the latter cases, strategies are designed to manipulate the CI encoding channels to efficiently deliver the speech signal to the electrode array for the improved auditory neural stimulation. Both binary mask [14] and soft mask [13, 15, 16] based studies have demonstrated that SNR-dependent channel selection is more desirable than the current default n-of-m channel selection strategy [19]. Non-negative matrix factorization (NMF) method has also been used in channel selection for CI device based on speech sparsity characteristics [1]. raditional statistical based speech enhancement algorithms are desirable for CI devices because of the real time processing []. However, these methods are only effective for stationary noise. heir benefits are moderated or disappear in the fluctuating noises. In this study, we propose a single microphone speech enhancement algorithm based on harmonic estimation combined with MMSE to improve speech intelligibility for CI subjects. On one hand, MMSE method is efficient to reduce the stationary noise along time dimension. On the other hand, we explore the harmonic structures of target speech to accurately estimate the fluctuating noise along frequency Copyright 1 ISCA 186

2 Figure 1: Algorithm Overview dimension. he noise estimation from both the time and frequency dimensions are employed in the MMSE framework for speech enhancement [1]. Moreover, the harmonic structure estimation is based on the noise robust pitch estimation from our previous study [, 3].. he Proposed Speech Enhancement Algorithm.1. Algorithm Overview he overall algorithm overview is shown in Fig. 1. wo types of noise estimation are included: i) noise estimation along the frequency dimension in the voiced speech segments, ii) noise estimation along the time dimensions in both voiced and unvoiced segments. Specifically, along time dimension, the assumption of stationary noise is made during noise tracking. Along the frequency dimension, the noise estimation is based on exploring harmonic structure. he estimated noise variance from both frequency and time dimensions ( λ ˆ d and λ ˆF d ) will be inputting to the MMSE framework to obtain the gain functions ( G ˆ and G ˆ F ). Finally, the above two gain functions will be fused into one form (Ĝ) for the target speech estimation based on the MMSE [1]... Harmonic structure estimation Speech spectrum of voiced speech is comprised of a series of harmonic partials with frequencies which are integer multiple times of fundamental frequency (F). Given the estimated Fs of the target speech, the harmonic structures can be approximated by selecting the noisy spectrum peaks which are near the ideal harmonic frequencies (kf). In practice, the observed harmonic partials usually deviates from the ideal frequency due to the instability of the glottal pulse sequence/shape during speech production. herefore, we set the deviation threshold f H differently depending on the frequency band, shown as follows,, f < 5Hz Δf H = 3, 5Hz f<hz (1) 45, f Hz he F estimation for noisy speech is based on harmonic feature classification based methods [, 3].3. Noise estimation based on harmonic structure Based on harmonic model, we infer that: i) the noisy spectrum between the harmonic partials are usually dominated by noise; ii) the noisy spectrum within the harmonic partials are dominated by speech. Accordingly, we perform the noise estimation between harmonics (BH) and within harmonics (WH) separately. First, harmonic spectrum for the target speech is generated by convolving the harmonic peak vector with the spectrum of a short-term analysis hamming window, shown as follow, S H (f) =S win (f) K a k H δ(f fh), k () k=1 where a k H and f H k are the amplitude and frequency of the kth order harmonic peak, S win is the spectrum vector of a hamming window, and δ(f) is a delta function. Next, the generated harmonic spectrum amplitude S H are reduced from the noisy speech spectrum to obtain the initial estimated noise spectrum A ˆ n, shown as below, ˆ A n = max ( S n S H, ). (3) Here, we call the noise spectrum within the harmonics main lobe as WH noise, and the noise spectrum between the harmonics main lobes as BH noise. he bandwidth of the main lobe for each harmonics is set as /3 of that of the corresponding short-term hamming window spectrum. In the BH frequency range, noise is the dominant component which is seldom overlapped with speech. hus, the initial estimated noise A ˆ n in this frequency range is considered as the estimated noise spectrum, shown as, Â BH (f) =Â n (f), (4) where f [ kf+ 1 f mb, (k +1)F 1 f mb], and fmb is the bandwidth of the harmonic main lobe. However, in the WH frequency range, speech has the dominant energy. he initiated estimated noise spectrum in this frequency range is not as accurate which requires reestimation. o estimate the WH noise, we made an assumption that noise spectrum is continuously distributed along the frequency dimension. Given the noise variance in the 18

3 .8k μ = -.1 σ = k.4k.k 4k k -5dB db 5dB (a) SS, -khz 8k μ = -.6 σ = 3.9 6k.8k μ = -. σ = k.4k.k 4k k -5dB db 5dB (b) SS, -4kHz 8k μ = -.16 σ =.4 6k Amplitude Amplitude 1 noisy speech spectrum noise spectrum estimated noise spectrum Frequency/Hz (a) speech-shaped noise -5dB db 5dB (c) BB, -khz -5dB db 5dB (d) BB, -4kHz Figure : Histogram of energy ratio between neighboring frequency bands. SS: speech-shaped noise, BB: babble noise Frequency/Hz (b) babble noise Figure 3: Noise estimation based on harmonic structure near frequency bands, the noise variance in the current frequency band can be approximated based on interpolation technique. Fig. presents the statistical histogram of the logarithmic energy ratio between neighboring frequency bands. Both the bandwidth and the frequency shift are set to 1Hz. We include two different frequency ranges: -khz and k-4khz. wo types of noise are considered: speech-shaped noise and babble noise. he mean and standard variance values are shown along with the histogram. From Fig. we see that the mean values of the logarithmic energy ratio between neighboring bands are around db in all four cases. he maximum spread value for both types of noises is less than 5dB. his analysis indicates the continuity of the noise energy between adjacent frequency bands. herefore, we propose to estimate the WH noise variance using the estimated BH noise variance with linear interpolation method, shown as below, Â WH (f) =â L BH + ( â R BH â L f f BH) L BH fbh R f BH L where f [ kf 1 f mb,kf+ 1 f mb], f L BH and fbh R are the edge frequencies of left and right neighboring band near the current harmonic partial, â L BH and âr BH are the average amplitude of estimated BH noise spectrum in left and right adjacent frequency band. Fig. 3 demonstrates an example of the noise estimation based on harmonic structure. Fig. 3a is for the speech-shaped noise case, and Fig. 3b is for the babble noise case. It can be seen that both the BH and WH noise estimation is almost consistent as the original noise. (5).4. Noise tracking along time dimension he noise tracking along time dimension is performed based on a minimum statistics algorithm with optimal smoothing [4], assuming noise is stationary. In practice, noise variance is estimated in the beginning quiet section and updated during later unvoiced segments..5. Gain function estimation he noise variance estimated along frequency dimension is employed in the MMSE framework to generate the harmonic-based gain function G ˆ F [5, ]. Alternatively, the time dimension estimated noise is used to generate a gain function G ˆ. hen G ˆ F and G ˆ are combined with weights to obtain the optimal gain function for the target speech [1], shown as, Ĝ = ˆ G F ˆ λ F G F + λ ˆ + ˆ F + λ ˆ. (6) where λ ˆ F and λ ˆ are the frequency- and time-dimension based noise variances which are computed from Sec. 3.3 and Sec Experiments and Results 3.1. Experiment setting Listening test is carried out with CI subjects to evaluate the performance of the proposed algorithm. he target speech materials are comprised of sentences from the IEEE database [6]. Babble noise was used to corrupt the sentences at the SNR of db, 5dB and 1dB. Six postlingually deafened cochlear implants users, with a mean 188

4 WRR/(%) 6 4 original noisy mmse harmonic+mmse SNR/dB Figure 4: Average word recognition rate results of.5 years implant use, participated in the listening test. Subjects were paid an hourly wage for their participation. he listening task involved sentence recognition by CI subjects who were seated in a soundproof room (Acoustic System, Inc). he sentences were played to the CI subjects through a loudspeaker placed at a distance of 8cm in front of the subject. he sound pressure level of the speech sentences from the loudspeaker was set as fixed 65dB through out the test. he subjects were fitted with their daily strategy during the test. Each subject participated in a total of 9 test conditions (3 SNR levels 3 processing conditions). wo IEEE sentence lists were used per test condition, and each IEEE sentence list includes 1 sentences. he order of test conditions was randomized across subjects. Subjects were given a 5-min break every 3 min during the test sessions to avoid listening fatigue. 3.. Results he speech intelligibility performance of CI subjects is measured in terms of average word recognition rate (WRR). he results of average WRR in different conditions are presented in Fig. 4. he standard error of the mean (SEM) of WRR results is also shown along with the average value. From Fig. 4 we see that the Harmonic + MMSE approach shows benefits for CI subjects at all SNR levels in the babble noise condition. In contrast, the MMSE processing only shows minor benefit at the SNR of db. We also carry out a two-way ANOVA on the WRR results to investigate the significance of processing conditions at different SNR levels. he ANOVA results for db and 5dB are [F (, 1) = 1.6,p <.3] and [F (, 1) = 16.1, p <.] respectively indicating significant interaction between different processing conditions. However, the ANOVA result for 1dB is [F (, 1) =.6, p <.1181], indicating no significant difference between different processing conditions. he Post hoc comparisons between different processed conditions are carried out at db and 5dB which have been shown significant statistics by ANOVA analysis. Specifically, when the SNR is db, the Post hoc results show significant difference between MMSE + Harmonic processed condition and the original noisy condition. Significant difference is also shown between MMSE + Harmonic processed condition and the only MMSE (a) Clean (b) Noisy 1 3 (c) MMSE (d) Harmonic + MMSE Figure 5: Electrodogram: babble noise, SRN = db processed condition (p <. and p<.3). However, there is no significant difference between MMSE processed condition and the original unprocessed condition (p <.8). When the SNR is 5dB, the similar statistic results are found as db. Fig. 5 shows an example of electrodogram for different processing conditions. Electrodogram represents the output of the cochlear implant devices. Comparing the electrodograms among different processing conditions, we see that much more residual noise exists in MMSE processed speech than Harmonic+MMSE processed speech. Harmonic structure estimation is able to provide more accurate noise tracking, thus removing the fluctuating noise, whereas MMSE method is not as effective. 4. Conclusion and discussion A speech enhancement algorithm based on harmonic estimation combined with MMSE framework is proposed in this paper to improve the speech intelligibility for CI recipients. he proposed Harmonic + MMSE processing is able to distinguish speech harmonics from noise interference. In this way, the non-stationary noise can be estimated and removed accurately. he listening test results demonstrate the potential benefit to CI subjects by the proposed method in terms of the WRR performance. It further indicates that F is a substantial cue for speech perception in fluctuating noise for CI recipients For the future research, the investigation of tradeoff between noise reduction and speech distortion is warranted for different noise condition regarding speech intelligibility for CI recipients

5 5. References [1] Hochberg I., A. Boothroyd, and M Weiss, Effects of noise and noise suppression on speech perception by cochlear implant users, Ear and Hearing, vol. 13, no. 4, pp. 63 1, Aug [] J. Wouters and J. Vanden Berghe, Speech recognition in noise for cochlear implantees with a two-microphone monaural adaptive noise reduction system, Ear and Hearing, vol., no. 5, pp. 4 43, Oct. 1. [3] A. Spriet, L. Van Deun, K. Eftaxiadis, J. Laneau, M. Moonen, B. van Dijk, A. van Wieringen, and Wouters J., Speech understanding in background noise with the twomicrophone adaptive beamformer beam in the nucleus freedom cochlear implant system, Ear and Hearing, vol. 8, no. 1, pp. 6, Feb.. [4] J. F. Patrick, P. A. Busby, and P. J. Gibson, he development of the nucleus freedom cochlear implant system, rends Amplify, vol. 1, no. 4, pp. 15, Dec. 6. [5] P. C. Loizou, Speech processing in vocoder-centric cochlear implants, in Cochlear and Brainstem Implants, Otorhinolaryngol, A. R. Moller, Ed., vol. 64, pp Karger, Basel, 6. [6] K. Kokkinakis, B. Azimi, Y. Hu, and D. R. Friedland, Single and multiple microphone noise reduction strategies in cochlear implants, rends in Amplification, vol. 16, no., pp , Jun. 1. [] Raphael Koning, Speech enhancement in cochlear implants, Ph.D. thesis, Dept. Neurosciences, KU Leuven, 14. [8] A. A. Hersbach, K. Arora, Mauger S. J., and Dawson P. W., Combining directional microphone and singlechannel noise reduction algorithms: a clinical evaluation in difficult listening conditions with cochlear implant users, Ear and Hearing, vol. 33, no. 4, pp. e13 e19, July - Aug. 1. [9] L. P. Yang and Q. J. Fu, Spectral subtraction-based speech enhancement for cochlear implant patients in background noise, J. Acoust. Soc. Am., vol. 11, no. 3 Pt. 1, pp , Nov. 5. [1] P. C. Loizou, A. Lobo, and Y. Hu, Subspace algorithms for noise reduction in cochlear implants, J. Acoust. Soc. Am., vol. 118, no. 5, pp , Nov. 5. [11] J. Li, Q. J. Fu, H. Jiang, and M. Akagi, Psychoacoustically-motivated adaptive beta-order generalized spectral subtraction for cochlear implant patients, in Proc. ICASSP, aipei, aiwan, Ari. 9, pp [1] F. oledo, P. C. Loizou, and A. Lobo, Subspace and envelope subtraction algorithms for noise reduction in cochlear implants, in Proc.5th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, aipei, aiwan, Ari. 3, pp. 5. [13] Y. Hu, P. C. Loizou, N. Li, and K. Kasturi, Use of a sigmoid-shaped function for noise attenuation in cochlear implants, J. Acoust. Soc. Am., vol. 1, no. 4, pp. EL18 EL134, Oct.. [14] Y. Hu and P. C. Loizou, Environment-specific noise suppression for improved speech intelligibility by cochlear implant users, J. Acoust. Soc. Am., vol. 1, no. 6, pp , June 1. [15] P. W. Dawson, S. J. Mauger, and A. A. Hersbach, Clinical evaluation of signal-to-noise ratio based noise reduction in nucleus cochlear implant recipients, Ear and Hearing, vol. 3, no. 3, pp , May-June 11. [16] S. J. Mauger, K. Arora, and P. W. Dawson, Cochlear implant optimized noise reduction, J. Neural Eng., vol. 9, no. 6, pp. 1 9, Dec. 1. [1] H. Hu, M. E. Lutman, S. D. Ewert, G. Li, and S. Bleeck, Sparse nonnegative matrix factorization strategy for cochlear implants, rends in Hearing, vol. 19, pp. 1 16, Dec. 15. [18] F. Chen, Y. Hu, and M. Yuan, Evaluation of noise reduction methods for sentence recognition by mandarinspeaking cochlear implant listeners, Ear and Hearing, vol. 36, no. 1, pp. 61 1, Jan. 15. [19] Y. Hu and P. C. Loizou, A new sound coding strategy for suppressing noise in cochlear implants, J. Acoust. Soc. Am., vol. 14, no. 1, pp , July 8. [] P. C. Loizou, Speech Enhancement: heory and Practice, chapter, CRC Press, Boca Raton, USA, 1 edition,. [1] M. Krawczyk-Becker and. Gerkmann, MMSE-optimal combination of wiener filtering and harmonic model based speech enhancement in general framework, in Proc. WASPAA, New Paltz, NY, Dec. 15, pp [] D. Wang, P. C. Loizou, and J. H. L. Hansen, F estimation in noisy speech based on long-term harmonic feature analysis combined with neural network classification, in Proc. INERSPEECH, Singapore, Sep. 14, pp [3] D. Wang, C. Yu, and J. H. L. Hansen, Robust harmonic features for classification-based pitch estimation, IEEE/ACM rans. Audio Speech Lang. Process., vol. 5, no. 5, pp , May 1. [4] R. Martin, Noise power spectral density estimation based on optimal smoothing and minimum statistics, IEEE rans. Audio Speech Lang. Process., vol. 9, no. 5, pp , Jul. 1. [5] Y. Ephraim and D. Malah, Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator, IEEE rans. Acoustic Speech and Signal Processing, vol. ASSP-3, no. 6, pp , Dec [6] IEEE, IEEE recommended practice for speech quality measurements, IEEE rans. Audio Electroacoust, vol. 1, no. 3, pp. 5 46, Sep