Robustness, Separation & Pitch

Transcription

1 Robustness, Separation & Pitch or Morgan, Me & Pitch Dan Ellis Columbia / ICSI dpwe@ee.columbia.edu 1. Robustness and Separation 2. An Academic Journey 3. Future COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK Robustness, Separation, Pitch - Dan Ellis /16

2 1953: How To Separate Speech? The Cocktail Party Problem [Cherry 53] Spatial information: ATC over a single speaker Pitch differences via gender differences Auditory Scene Analysis [Bregman 90] Grouping cues Onset Harmonicity Common Fate Schema Robustness, Separation, Pitch - Dan Ellis /16

3 The Usefulness of Pitch Common pitch can link energy from a single source Brungart et al. 01 Normal mix Pitchless Robustness, Separation, Pitch - Dan Ellis /16

4 1984: Perception-Inspired Separation Model the periodicity information in the auditory nerve Lyon 1984 Weintraub 1985 Mix Female Robustness, Separation, Pitch - Dan Ellis /16

5 1996: An Academic Journey Dan in 1996: The weft "Bad dog" f/hz f/hz Wefts1, Robustness, Separation, Pitch - Dan Ellis /16

6 1999: Size Matters All you need is a BDNN Ellis & Morgan 99.. and the data (and patience) to train it WER vs. frames/weight WER for PLP12N nets vs. net size & training data GCUP GCUP GCUP 40 WER% WER% TCUP TCUP Training set / hours 5.7 TCUP TCUP Hidden layer / units frames/weight Robustness, Separation, Pitch - Dan Ellis /16

7 2001: Overlap Remains Meeting Recorder Project natural speech interactions ~10% of speech frames have overlaps Janin, Baron, Edwards, Ellis, Gelbart, Morgan, Peskin, Pfau, Shriberg, Stolcke, Wooters 03 mr backchannel floor seizure Spkr A speaker active Spkr B speaker B cedes floor Spkr C interruptions Spkr D breath noise Spkr E crosstalk Table top level/db time / secs Robustness, Separation, Pitch - Dan Ellis /16

8 2003: EARS Pushing the envelope (aside) Robustness, Separation, Pitch - Dan Ellis /16

9 2004: Pitch Based Separation Literal implementations of the process described in Bregman 1990: compute regularity cues: - common onset - gradual change - harmonic patterns - common fate Original v3n7 Brown 1992 Ellis 1996 Hu & Wang 2004 Hu & Wang 2004 Robustness, Separation, Pitch - Dan Ellis /16

10 2004: Model-Based Separation Data-driven separation Learn codebooks for individual speakers Find best combination of sources Pitch gives the grist Roweis 01 Kristjansson, Attias, Hershey 04 Robustness, Separation, Pitch - Dan Ellis /16

11 2006: Pitch for VAD Pitch is the most robust perceptual cue to speech Lee & Ellis 06 Robustness, Separation, Pitch - Dan Ellis /16

12 : The Epic Speech and Audio Signal Processing Processing and Perception of Speech and Music S E C O N D E D I T I O N Ben Gold t Nelson Morgan t Dan Ellis Robustness, Separation, Pitch - Dan Ellis /16

13 freq / khz 2012: Project Babel Noisy speech is a challenge: 4 IARPA_BABEL_OP1_204_73990_ _162632_inLine time / sec level / db How to disentangle speech and interference? Energy peaks are speech (spectral subtraction) Energy troughs are noise (Wiener, log-mmse) Speech has a known form (Factorial HMM) Voiced speech is periodic (Pitch-based) Robustness, Separation, Pitch - Dan Ellis /16

14 Classification-based Pitch Tracker Subband Autocorrelation Classification (SAcC) Pitch Tracker: Trained on noisy speech with true pitch targets Lee & Ellis 12 delay line short-time autocorrelation Neural network classifier c k Sound Cochlea filterbank frequency channels c 1 B N B 3 B 2 uv Viterbi smoother Pitch B freq lag time Correlogram slice Subband autocorrelation features PCA to reduce dimensions Robustness, Separation, Pitch - Dan Ellis /16

15 Flat-Pitch Processing Time-varying filtering is tricky if pitch variation and filter impulse response are on Solution: Flatten the pitch use local pitch estimate to resample process constant-pitch resampling is (near) invertible a similar time-scale SAcC pitch tracker pitch-flatten time map inverse time map noisy speech time-varying resampling flat-pitch-domain enhancement time-varying resampling enhanced speech Robustness, Separation, Pitch - Dan Ellis /16 freq / Hz Noisy signal Resampled to pitch = 200 Hz Filtered comb Resampled back to original pitch and mixed with original pr(vx) pitch time / s

16 Conclusions Pitch is a key feature for speech separation marking signal against other speech or noise Some ideas don t go away.. though they can change shape Impact from collaboration you can t do good work with someone without the human connection Robustness, Separation, Pitch - Dan Ellis /16