Modulation and Top-Down Processing in Audition

Transcription

1 Modulation and Top-Down Processing in Audition Malcolm Slaney 1,2 and Greg Sell 2 1 Yahoo! Research 2 Stanford CCRMA

2 Outline The Non-Linear Cochlea Correlogram Pitch Modulation and Demodulation Information Flow CASA Relational Networks

3 Cochlear Nonlinearity

4 Hair Cell Motility Cillia Pipette (changing membrane potential) By Jonathan Ashmore -

5 Auditory Nerve Patterns Frequency Time

6

7 Correlogram Architecture

8 Tone Correlograms

9 Tone Examples Distance down cochlea Center Frequency Time Interval (s) Autocorrelation Lag With help from Richard O. Duda

10 Speech Correlogram

11 Speech Examples Speech Examples Leonardo at Apple

12

13 Pitch Integrated Correlogram Narrowed Autocorrelation Pitch Peak Time Lag (ms)

14 Pitch Architecture

15 Pitch in Noise

16 Ambiguity of Pitch

17 Pitch Theories A Correlogram Temporal implementations with neural delay lines T Lag ime F requency Cochlea Sound Inpu t B Perceived Pitches Matching result Central Pattern Match Theory Templates.25 Input Spectrum CF Hz 315 Hz 175 Hz

18 Correlogram Implementation Today FFT Licklider Neural Delay Patterson Strobedtemporal integration Alternative Oscillators AM Modulator

19 Processing Chain One Dimensional (waveform) Pressure Time Two Dimensional (not a spectrogram) Cochlear Place Time Cochlear Processing Three Dimensional (neural movie) Cochlear Place Correlogram Processing Autocorrelation Lag

20 What is Modulation?

21 Applications of Demodulation Cochlear implants Chimaeric Speech Speech Enhancement Source Separation Modulation Filtering

22 Chimeara Speech

23 Demodulation Ambiguity

24 Ambiguity Example

25 Ambiguity Example

26 Goals Ability to select desired solution Flexibility to define desired solution Lower-frequency modulation (Amplitude) Higher-frequency carrier (Pitch) Handle harmonic/noisy carriers Satisfy projection property

27 Past Work Hilbert Envelope Half/Full-wave Rectifier Modulation Spectrogram S. Greenberg and B. E. D. Kingsbury. The Modulation Spectrogram: In pursuit of an invariant representation of speech. In Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing, pages , Cortical Sound Representation M. Elihilali, T. Chi, and S. A. Shamma. A spectro-temporal modulation index (STMI) for assessment of speech intelligibility. Speech Communication, 41: , Subband Carrier Demodulation S. Schimmel and L. Atlas. Coherent Envelope Detection for Modulation Filtering of Speech. In Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing, pages , 2005.

28 Hilbert Envelope

29 Half/Full Wave Rectifier

30 Modulation Spectrogram

31 Cortical Sound Representation

32 Subband Carrier Demodulation

33 Subband Decomposition? Most methods use subbands Auditory system Can t handle harmonic/noisy carriers Problems Inconsistent filter-component alignment Non-stationary components Demodulation is nonlinear Changes definition of modulation Filter Q defines modulation frequency range We want an algorithm that doesn t require it

34 Effect of Subband

35 Speech Pipeline

36 Modulation or Filtering

37 Optimal Demodulation

38 Two Solutions: Log or Linear Cepstral Domain Optimize log of signal Smooth second derivative* of m(t) Sparse carrier Restore sign Linear Domain Minimize high-frequency modulator energy Subject to m(t) > c(t)

39 Simple Examples

40 Ambiguity Example

41 Increasing Modulation Frequency

42 Modulation Sign Change

43 Harmonic Carrier

44 Stochastic Carrier

45 SNR

46 Speech

47 Block Diagrams Pressure Time Cochlear Processing Cochlear Place Time Cochlear Place Correlogram Processing Autocorrelation Lag

48 Problems Continuation Tone and Noise Parliament Cough What do you hear? Waveforms? Ideas?

49 Speech Examples Wedding Sine Natural

50 Visual Context

51 What Vowel is This? Word 1 Peter Ladefoged Word 2 Word 3

52

53

54

55 McGurk

56 Speech Object Sinewave Speech Object Wedding Speech Vowel? Environment Vision Ventroloquism Vision Dots Vision McGurk Audio Locate Audio Locate Speech

57

58 Information Flow

59 ASR Three Three Three Language model for the words: one, two, three Two Two Two One One One Word model showing phonemes for the word one /w/ / / /n/ Acoustic (phoneme) model for the phoneme / / S 1 S 2 S 3

60 Streaming of Tones

61 Auditory Scene Analysis All harmonics - Oboe Even harmonics - Soprano Odd harmonics - Clarinet

62 Old plus New Principle Slide by Dan Ellis (Columbia)

63 Conventional Scene Analysis Slide by Dan Ellis (Columbia)

64 Goto CASA with MIDI MIDI Sequence

65 Barker ASR

66 Ellis Prediction Driven

67 Saliency

68 Saliency Example Time-frequency display Saliency map shows high-interest locations

69 Saliency Maps Longer tones better Missing parts salient Modulation more salient Forward masking works

70 Sound Examples Birds Calls Cows Horse Waterfall

71 Saliency Comparison Details of saliency comparison Model predictions

72 Relational Network (Simple) Patches of neurons X Z Z Each measure one quantity M X M Y M Bidirectional relations for feedback/feedforward Thanks to Rodney Douglas m Y M

73 Relational Network (example) Relational specification Input here Relational feedback Relational Feedback

74 ASR Relational Network Bidirectional links enforce phoneme/ word constraints Cochlea Delay Phone Recognizer Phone Recognizer Word Recognizer Note: We don t know how to represent delays A patch of neurons (one of N output)

75 Desired Results Relational Feedback Without With /A/ Phoneme Patch /I/ Phoneme Patch AI Word Patch IA Word Patch Phoneme Input A A A I

76 Simulation

77 Simulation 2

78 Simulation 3

79 Grossberg ART

80 Statistical Means ICA Different distributions One Microphone GMM models of distribution

81 Conventional

82 Better?