Binaural processing of complex stimuli

Transcription

1 Binaural processing of complex stimuli

2 Outline for today Binaural detection experiments and models Speech as an important waveform Experiments on understanding speech in complex environments (Cocktail Party Problem) Understanding performance in these cases Difficulties with hearing impairments and with the design of hearing aids and cochlear implants

3 Summary so far Processing of interaural differences (ITD and ILD) is understood empirically and physiologically in terms of extracting information from input waveforms. The interpretation of this information to achieve observed performance by human listeners is still a mystery from the neural point of view. The other major topic that has been studied in binaural conditions is the detection problem; for example, the detection of a tone in a masking noise. This is now considered briefly.

4 Binaural Detection Studies Early experiments focused on the Binaural Masking Level Difference or BMLD Detection of a tone in noise for a standard reference case: Masker is identical at the two ears (like a source in front) Target is either also identical or inverted for classic BMLD case (other cases, like displaced target location, are intermediate Threshold difference in decibels is the binaural improvement Data show that performance is much better in second case.

5 Frequency dependence of BMLD (later studies showed low-frequency reductions were due to low level)

6 Theories of binaural unmasking THREE MODELS PROPOSED: Localization model Equalization-cancellation (EC) model Model based on binaural coincidences Localization model does not work EC model can be used conveniently Model based on coincidences is easier to relate to physiology

7 Symmetry is a problem for the localization model:

8 Equalization-Cancellation Theory Use interaural time and level differences to equalize the masking noise; then cancel. If target interaural differences are not the same as the noise, the target will not cancel and we will have an improvement in signal to noise ratio. This idea has worked effectively and is frequently applied to explain data.

9 EC Model Equalization parameters are constant in traditional EC Model Equalization parameters (α_o and τ_o) vary with time in the Short-Time-EC (STEC) Model

10 This EC Model fits performance in MANY EXPERIMENTS It is easy to use in terms of operations and calculations. It is frequently used for many experiments. But physiological basis of the specific operations are not clear. There are ways to base the operations on the processes given above, but this is not yet tested.

11 Now we consider complex stimuli (such as Speech) Spectral content varies with time Characterize stimuli in short time segments (such as say 50 milliseconds 20 / second) Have to consider the frequency content of the waveform as a function of time Spectrogram or short-time Fourier analysis

12 Example Spectrogram (whistling)

13 Spectrogram Example (speech)

14

15

16 Figure Waveform of the speech utterance Two plus seven is less than ten. Each line is 0.17 s in duration. Thetime-aligned phonemic transcript is indicated below the waveform. The sampling rate is 16,000 samples/s, so each line represents 2720 samples.

17

18 Figure (a) Wideband spectrogram of waveform of Figure (b) Narrowband spectrogram. Short window in spectrogram Long window in spectrogram From Oppenheim and Schafer (DSP textbook)

19 Figure (a) Wideband spectrogram of waveform of Figure (b) Narrowband spectrogram. Long window in spectrogram From Oppenheim and Schafer (DSP textbook)

20 Figure (a) Wideband spectrogram of waveform of Figure (b) Narrowband spectrogram. Short window in spectrogram From Oppenheim and Schafer (DSP textbook)

21 Now turn to multiple speakers talking simultaneously This is the problem listeners really care about This is not so clearly understood.

22

23 How Understanding can we separate speech voices in that competition are speaking with simultaneously? other speech can be challenging. Miller, G.A. (1947) "The masking of speech," Psychol. Bulletin, 44, The famous psychologist George A. Miller once wrote: "It is said that the best place to hide a leaf is in the forest, and presumably the best place to hide a voice is among other voices" (p. 118). Miller, G.A. (1947) "The masking of speech," Psychol. Bulletin, 44,

24 E.Colin Cherry s 1953 summary: How do we recognize what one person is saying when others are speaking at the same time (the cocktail party problem')? Factors: Voices from different directions Lip-reading, gestures, and the like Different speaking voices, pitches, speeds, Accents differing Transition-probabilities (subject matter, voice dynamics, syntax, )

25 Basic Masking Experiments Measure Speech Reception Threshold (SRT), increase target until level allows 50% correct. Vary level to obtain 50% correct, where correct means both <color> and <number> correct. Performance limited by both EM and IM

26 SRTs measured using Coordinate Response Measure (CRM) corpus Maskers differed in call sign, color, number eg, Ready Charlie go to green six now

27 Suzanne Carr Levy Experiment (Carr, 2010; Carr-Levy et al., 2015) Speech Target straight ahead; CRM corpus Two speech maskers in various locations: Co-located Symmetrically separated from target Anti-symmetric (both maskers on one side) Varied degrees of Room reverberation (Anechoic, Mild, Moderate) Pitch similarity (0- or 1- or 4-semitone separation) Measured SRT (Speech Reception Threshold)

28 Background: spatial distribution of sources for speech-with-two-maskers case Colocated Symmetric Anti-symmetric Masker Target Harder Easier Marrone, Mason, Kidd, 2008; Carr, 2010

29 Presented over headphones to subjects within a sound attenuating booth Speech reception thresholds (SRTs) Target sentence always masked by two other sentences from same talker, but with pitch shifts in some cases All sentences were monotonized to fundamental frequency = 100 Hz, to control the pitch cue (adopted from Carr 2010) Spatial locations were simulated with anechoic HRTFs (Gardner and Martin, 1994), with reverberation added

30 Carr (2010)

31 Strategy l See how much we can explain with a waveform based model l Plan is to add IM to the waveform model, but the IM modeling has not been pursued very far l Model chosen is the short-time equalizationcancelation model from Wan, Durlach, and Colburn (2014)

32 EC Model (Durlach, 1963), modified for short time processing (Wan, 2011), STEC (Wan et al. 2014) δ L (t) ε L (t) Band-pass Filtered Speech Band-pass Filtered Speech X L (t) X R (t) δ R (t) Time & level Jitter ε R (t) Time & level Jitter L (t) R (t) Monaural Left Equalization Binaural (α o, τ o ) Monaural Right Cancellation Y(t) DECISION choose max SNR for each frequency band Speech Intelligibility Index Predicted threshold. Jitter variances STEC Window length SII Criterion

33 Basic Implementation of Model Adjust equalization parameters (αo and τo) to minimize masker energy in time-frequency slice for each slice. Calculate resulting signal to noise ratio for each frequency band. Compute Signal-Intelligibility Index (SII) for each frequency band by weighting over frequency with the frequency importance function (ignoring very high and low SNRs). Choose criterion performance separately for each type of maskers (e.g., for noise or for one, two, or three speech waveforms).

34 Carr implementation of EC and STEC models (Carr, 2010)

35 Carr (2010)

36 IM is high for these speech tasks Results showed that when subjects were incorrect, they almost always chose an actual masker instead of a random choice Thus, there is a high confusability factor for these experiments

37 Extension to hearing-impaired listeners Carr experiment also measured performance for a set of hearing-impaired (SNHL) listeners Similar performance Modeling is not yet complete many factors Different effective levels Impaired interaural processing Impaired temporal processing Impaired frequency tuning

38 Carr (2010)

39 Complex acoustic environments Difficulty with complex environments is a frequent complaint of people with hearing problems, including those with aids or Cis. Solutions include smart processors that preprocess waveforms, such as the Eye- Directed Hearing Aid of Gerald Kidd. Work is underway to develop ways to measure abilities in natural environments as part of the fitting and clinical evaluation process.

40

41

42 Stable World Model Internal Representations Real Sources in the real Environment Physical Acoustics L(t) R(t) Peripheral Physiology ITD,f array ILD,f array freq spectra Compare Actual to Current Model of Sources and Environment Listener s Learned Transformations ITD,f array ILD,f array freq spectra Hypothesis Non-auditory inputs Fast (Add/Delete/Move Source) Slow (Pinna) Update Mechanism Colburn and Kulkarni (2005)

43 Conclusions I Speech intelligibility experiments with speech maskers are affected by informational masking and energetic masking. Complete computational model must include both factors explicitly. Doing the job with a computational model of binaural processing fits a lot of data, particularly cases with spatial separation between targets and maskers. Model is not yet a realistic algorithm for processing of waveforms.

44 LOTS to do Conclusions II