Auditory Scene Analysis in Humans and Machines

Transcription

1 Auditory Scene Analysis in Humans and Machines Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Eng., Columbia Univ., NY USA 1. The ASA Problem 2. Human ASA 3. Machine Source Separation 4. Systems & Examples 5. Concluding Remarks Auditory Scene Analysis - Dan Ellis /54

2 1. Auditory Scene Analysis Sounds rarely occurs in isolation.. but recognizing sources in mixtures is a problem.. for humans and machines chan 0 freq / khz 4 2 mr :57:43 chan 1 chan 3 freq / khz freq / khz level / db mr ce+1743.wav chan 9 freq / khz time / sec Auditory Scene Analysis - Dan Ellis /54

3 Sound Mixture Organization Goal: recover individual sources from scenes.. duplicating the perceptual effect 4000 frq/hz time/s level / db Analysis Voice (evil) Rumble Stab Voice (pleasant) Strings Choir Problems: competing sources, channel effects Dimensionality loss need additional constraints Comp. Aud. Scene Analysis - Dan Ellis /21

4 The Problem of Mixtures Imagine two narrow channels dug up from the edge of a lake, with handkerchiefs stretched across each one. Looking only at the motion of the handkerchiefs, you are to answer questions such as: How many boats are there on the lake and where are they? (after Bregman 90) Received waveform is a mixture 2 sensors, N sources - underconstrained Undoing mixtures: hearing s primary goal?.. by any means available Comp. Aud. Scene Analysis - Dan Ellis /21

5 Source Separation Scenarios Interactive voice systems human-level understanding is expected Speech prostheses crowds: #1 complaint of hearing aid users Archive analysis identifying and isolating sound events freq / khz dpwe :15: level / db time / sec Unmixing/remixing/enhancement pa wav Auditory Scene Analysis - Dan Ellis /54

6 How Can We Separate? By between-sensor differences (spatial cues) steer a null onto a compact interfering source By finding a separable representation spectral? sources are broadband but sparse periodicity? maybe for pitched sounds something more signal-specific... By inference (based on knowledge/models) acoustic sources are redundant use part to guess the remainder Auditory Scene Analysis - Dan Ellis /54

7 Outline 1. The ASA Problem 2. Human ASA scene analysis separation by location separation by source characteristics 3. Machine Source Separation 4. Systems & Examples 5. Concluding Remarks Auditory Scene Analysis - Dan Ellis /54

8 Auditory Scene Analysis Listeners organize sound mixtures into discrete perceived sources based on within-signal cues (audio +...) common onset + continuity harmonicity freq / khz spatial, modulation,... learned schema time / sec level / db Reynolds-McAdams oboe reynolds-mcadams-dpwe.wav Auditory Scene Analysis - Dan Ellis /54

9 Perceiving Sources Harmonics distinct in ear, but perceived as one source ( fused ): %req!ime depends on common onset depends on harmonics Experimental techniques ask subjects how many match attributes e.g. pitch, vowel identity brain recordings (EEG mismatch negativity ) Auditory Scene Analysis - Dan Ellis /54

10 Auditory Scene Analysis How do people analyze sound mixtures? break mixture into small elements (in time-freq) elements are grouped in to sources using cues sources have aggregate attributes Grouping rules (Darwin, Carlyon,...): cues: common onset/offset/modulation, harmonicity, spatial location,... Bregman 90 Darwin & Carlyon 95 Onset map Elements Sources Sound Frequency analysis Harmonicity map Grouping mechanism Source properties Spatial map (after Darwin 1996) Comp. Aud. Scene Analysis - Dan Ellis /21

11 Streaming Sound event sequences are organized into streams i.e. distinct perceived sources difficult to make comparisons between streams Two-tone streaming experiments: TRT: ms frequency 1 khz!f: 2 octaves asacd-36-streaming-50s ecological relevance? time Auditory Scene Analysis - Dan Ellis /54

12 Illusions & Restoration Illusion = hearing more than is there e.g. pulsation threshold example - tone is masked freq? time asacd-26-pulsation old-plus-new heuristic: existing sources continue frequency & /0z time & s Need to infer most likely real-world events observation equally good match to either case prior likelihood of continuity much higher bregnoise Auditory Scene Analysis - Dan Ellis /54

13 Human Performance: Spatial Separation Task: Coordinate Response Measure Ready Baron go to green eight now 256 variants, 16 speakers correct = color and number for Baron Accuracy as a function of spatial separation: Brungart et al. 02 crm wav A, B same speaker o Range effect Auditory Scene Analysis - Dan Ellis /54

14 Separation by Vocal Differences CRM varying the level and voice character (same spatial location) Brungart et al. 01 energetic vs. informational masking Auditory Scene Analysis - Dan Ellis /54

15 Varying the Number of Voices Two voices OK; More than two voices harder (same spatial origin) Brungart et al. 01 mix of N voices tends to speech-shaped noise... Auditory Scene Analysis - Dan Ellis /54

16 Outline 1. The ASA Problem 2. Human ASA 3. Machine Source Separation Independent Component Analysis Computational Auditory Scene Analysis Model-Based Separation 4. Systems & Examples 5. Concluding Remarks Auditory Scene Analysis - Dan Ellis /54

17 Scene Analysis Systems Scene Analysis not necessarily separation, recognition,... scene = overlapping objects, ambiguity General Framework: distinguish input and output representations distinguish engine (algorithm) and control (constraints, computational model ) Comp. Aud. Scene Analysis - Dan Ellis /21

18 Human and Machine Scene Analysis CASA (e.g. Brown 92): Input: Periodicity, continuity, onset maps Output: Waveform (or mask) Engine: Time-frequency masking Control: Grouping cues from input - or: spatial features (Roman,...) Comp. Aud. Scene Analysis - Dan Ellis /21

19 Human and Machine Scene Analysis CASA (e.g. Brown 92): ICA (Bell & Sejnowski et seq.): Input: waveform (or STFT) Output: waveform (or STFT) Engine: cancellation Control: statistical independence of outputs - or energy minimization for beamforming Comp. Aud. Scene Analysis - Dan Ellis /21

20 Human and Machine Scene Analysis CASA (e.g. Brown 92): ICA (Bell & Sejnowski et seq.): Human Listeners: Input: excitation patterns... Output: percepts... Engine:? Control: find a plausible explanation Comp. Aud. Scene Analysis - Dan Ellis /21

21 Machine Separation Problem: Features of combinations are not combinations of features voice is easy to characterize when in isolation redundancy needed for real-world communication!"l" $"%&e ()* $"%&e (%+ freq / 9Hz freq / 9Hz 4 / / 2 1 5r%6%27l (r* /0. (r* /0.=16-0>010> 0 0 0?> 1 1?> t2*e / se( (*,, -"%.e re.1234 (r* /0.-n12se (r* /0.=16-0>010>-n12se 0 0?> 1 1?> t2*e / se( crm wav -60 level / db Auditory Scene Analysis - Dan Ellis / crm noise.wav crm wav crm noise.wav

22 Separation Approaches ICA Multi-channel Fixed filtering Perfect separation maybe! CASA / Model-based Single-channel Time-varying filtering Approximate Separation target x interference n mix proc + - n^ x^ mix x+n stft proc spectro gram mask x istft x^ Very different approaches! Auditory Scene Analysis - Dan Ellis /54

23 Independent Component Analysis Central idea: Search unmixing space to maximize independence of outputs Bell & Sejnowski 95 Smaragdis a 11 a 21 a 12 a 22 s 1 s 2 x 1 x 2 s 1 w 11 a 21 a 12 a 11 w 21 w 12 w 22 x 1 s^ 1 x ^ 2 s 2 s 2 a 22!" MutInfo "w ij simple mixing a good solution (usually) exists Auditory Scene Analysis - Dan Ellis /54

24 Mixtures, Scatters, Kurtosis Mixtures of sources become more Gaussian can measure e.g. via kurtsosis (4th moment) 0.6 s 1 vs. s x 1 vs. x p(s 1 ) kurtosis = p(x 1 ) kurtosis = p(s 2 ) kurtosis = p(x 2 ) kurtosis = Auditory Scene Analysis - Dan Ellis /54

25 ICA Limitations Cancellation is very finicky hard to get more than ~ 10 db rejection mix s1 Mixture Scatter s2 kurtosis Kurtosis vs.! from Parra & Spence 00 lee_ss_in_1.wav lee_ss_para_1.wav mix 1 2 s1 s ! / " The world is not instantaneous, fixed, linear subband models for reverberation continuous adaptation Needs spatially-compact interfering sources Auditory Scene Analysis - Dan Ellis /54

26 Computational Auditory Scene Analysis Central idea: Segment time-frequency into sources based on perceptual grouping cues input mixture Front end Segment signal features (maps) Object formation discrete objects Group Grouping rules Brown & Cooke 94 Okuno et al. 99 Hu & Wang Source groups eq onset time period frq.mod... principal cue is harmonicity Auditory Scene Analysis - Dan Ellis /54

27 CASA Preprocessing Correlogram: a 3rd periodicity axis envelope of wideband channels follows pitch delay line short-time autocorrelation Slaney & Lyon 90 sound Cochlea filterbank frequency channels envelope follower freq "'&&+3'1&45 /30.+ lag time c/w Modulation Filtering [Schimmel & Atlas 05] Auditory Scene Analysis - Dan Ellis /54

28 Weft Periodic Elements Represent harmonics without grouping? Correlogram slices Smooth Spectrum Ellis 96 lag freq time commonperiod feature period freq Period Track time time hard to separate multiple pitch tracks Comp. Aud. Scene Analysis - Dan Ellis /21

29 Time-Frequency (T-F) Masking Local Dominance assumption Original Mix + Oracle Labels freq / khz Male Female level / db cooke-v3n7.wav Oraclebased Resynth freq / khz time / sec time / sec cooke-v3msk-ideal.wav oracle masks are remarkably effective! mix max(male, female) < 3dB for ~80% of cells Auditory Scene Analysis - Dan Ellis /54 cooke-n7msk-ideal.wav

30 Combining Spatial + T-F Masking T-F masks based on inter-channel properties [Roman et al. 02], [Yilmaz & Rickard 04] lee_ss_in_1.wav multiple channels make CASA-like masks better lee_ss_duet_1.wav T-F masking after ICA [Blin et al. 04] cancellation can remove energy within T-F cells Auditory Scene Analysis - Dan Ellis /54

31 CASA limitations Driven by local features problems with masking, aperiodic sources... Limitations of T-F masking need to identify single-source regions cannot undo overlaps leaves gaps from Hu & Wang 04 huwang-v3n7.wav Auditory Scene Analysis - Dan Ellis /54

32 Auditory Illusions How do we explain illusions? pulsation threshold %&$'($)*+ 3,,, 2,,, 1,,, 0,,,,,,-. / / !"#$ sinewave speech %&$'($)*+.,,, 1,,, 4,,, 0,,, /,,,,,-. / / !"#$ phonemic restoration %&$'($)*+ 1,,, 4,,, 0,,, /,,,,,,-. / /-. 0!"#$ Something is providing the missing (illusory) pieces... source models Auditory Scene Analysis - Dan Ellis /54

33 Adding Top-Down Constraints Bottom-up CASA: limited to what s there input mixture Top-down predictions allow illusions input mixture Front end Front end signal eatures signal eatures Object formation Hypothesis management prediction errors Compare & reconcile discrete objects hypotheses Noise components Periodic components predicted features Grouping rules match observations to a world-model... Source groups Predict & combine Auditory Scene Analysis - Dan Ellis /54 Ellis 96

34 Separation vs. Inference Ideal separation is rarely possible i.e. no projection can completely remove overlaps Overlaps Ambiguity scene analysis = find most reasonable explanation Ambiguity can be expressed probabilistically i.e. posteriors of sources {S i } given observations X: P({S i } X) P(X {S i }) P({S i }) combination physics source models Better source models better inference.. learn from examples? Auditory Scene Analysis - Dan Ellis /54

35 Simple Source Separation Given models for sources, find best (most likely) states for spectra: p(x i 1,i 2 ) = N (x;c i1 + c i2,σ) combination model {i 1 (t),i 2 (t)} = argmax i1,i 2 p(x(t) i 1,i 2 ) can include sequential constraints... different domains for combining c and defining E.g. stationary noise: freq / mel bin Original speech In speech-shaped noise (mel magsnr = 2.41 db) VQ inferred states (mel magsnr = 3.6 db) Roweis 01, 03 Kristjannson 04, 06 inference of source state time / s Auditory Scene Analysis - Dan Ellis /54

36 Can Models Do CASA? Source models can learn harmonicity, onset... to subsume rules/representations of CASA freq / mel band VQ800 Codebook - Linear distortion measure level / db can capture spatial info too [Pearlmutter & Zador 04] Can also capture sequential structure e.g. consonants follow vowels... like people do? But: need source-specific models... for every possible source use model adaptation? [Ozerov et al. 2005] Auditory Scene Analysis - Dan Ellis /

37 Separation or Description? Are isolated waveforms required? clearly sufficient, but may not be necessary not part of perceptual source separation! Integrate separation with application? e.g. speech recognition mix separation identify target energy t-f masking + resynthesis ASR speech models words vs. mix identify speech models find best words model words source knowledge words output = abstract description of signal Auditory Scene Analysis - Dan Ellis /54

38 Missing Data Recognition Speech models p(x M) are multidimensional... need values for all dimensions to evaluate p( ) x But: can make inferences given u p(x k,x u ) just a subset Zof dimensions x k p(x k M) = p(x k,x u M)dx u Hence, missing data recognition: dimension!?5+$+%)!3()(!2($@ time! P(x q) = P(x 1 q) P(x 2 q) P(x 3 q) P(x 4 q) P(x 5 q) P(x 6 q) hard part is finding the mask (segregation) Auditory Scene Analysis - Dan Ellis /54 y Cooke et al. 01 p(x k x u <y ) p(x k ) x k

39 The Speech Fragment Decoder Match uncorrupt spectrum to ASR models using missing data Segregation S Joint search for model M and segregation S to maximize: P" X Y$ S# Barker et al. 05 Observation Y(f ) Source X(f ) P" M$ S Y # = P" M# % P" X M#! d P" X# X! P" S Y # Isolated Source Model Segregation Model freq Auditory Scene Analysis - Dan Ellis /54

40 Using CASA cues P" X Y$ S# P" M$ S Y # = P" M# % P" X M#! d X! P" S Y # P" X# CASA can help search consider only segregations made from CASA chunks CASA can rate segregation construct P(S Y) to reward CASA qualities: Frequency Proximity Common Onset Harmonicity Auditory Scene Analysis - Dan Ellis /54

41 Outline 1. The ASA Problem 2. Human ASA 3. Machine Source Separation 4. Systems & Examples Periodicity-based Model-based Music signals 5. Concluding Remarks Auditory Scene Analysis - Dan Ellis /54

42 Current CASA State-of-the-art bottom-up separation noise robust pitch track label T-F cells by pitch extensions to unvoiced transients by onset Hu & Wang 03 1,,, 5,,, 0,,, 1,,, 5,,, 0,,, %&$'($)*+ 4,,, /,,, 3,,, %&$'($)*+ 4,,, /,,, 3,,,.,,, 2,,,,,,-.,-/,-0, /!"#$.,,, 2,,,,,,-.,-/,-0, /!"#$ Auditory Scene Analysis - Dan Ellis /54

43 f/hz City Horn1 (10/10) Horn2 (5/10) Horn3 (5/10) Horn4 (8/10) Horn5 (10/10) Crash (10/10) Prediction-Driven CASA f/hz Wefts1! f/hz Noise2,Click Weft5 Wefts6,7 Weft8 Wefts9!12 Ellis 96 Identify objects in real-world scenes using sound elements f/hz Noise Squeal (6/10) Truck (7/10)!40!50!60! db time/s Auditory Scene Analysis - Dan Ellis /54

44 Singing Voice Separation Pitch tracking + harmonic separation Avery Wang 94 freq / khz freq / khz level / db time / sec Auditory Scene Analysis - Dan Ellis /54

45 Periodic/Aperiodic Separation Harmonic structure + repetition of drums 4,,, Virtanen 03 2,,, %&$'($)*+ 0,,,.,,, 4,,,,, -. / ,!"#$ 2,,, %&$'($)*+ 0,,,.,,, 4,,,,, -. / ,!"#$ 2,,, %&$'($)*+ 0,,,.,,,,, -. / ,!"#$ Auditory Scene Analysis - Dan Ellis /54

46 Speech Separation Challenge Mixed and Noisy Speech ASR task defined by Martin Cooke and Te-Won Lee short, grammatically-constrained utterances: <command:4><color:4><preposition:4><letter:25><number:10><adverb:4> e.g. "bin white at M 5 soon" t5_bwam5s_m5_bbilzp_6p1.wav Results to be presented at Interspeech 06 See also Statistical And Perceptual Audition workshop Auditory Scene Analysis - Dan Ellis /54

47 IBM s Superhuman Separation Optimal inference on Mixed Spectra model each speaker (512 mix GMM) Kristjansson et a Interspeech 06 Noisy vector, clean vector and estimate of clean vector 35 wer / % Applied to Speech Separation Challenge: Same Gender Speakers 6 db 3 db 0 db - 3 db - 6 db - 9 db SDL Recognizer No dynamics Acoustic dyn. Grammar dyn. Human Frequency [Hz] o Infer speakers and gain o Reconstruct speech o Recognize as normal... Use grammar constraints Auditory Scene Analysis - Dan Ellis /54 Energy [db] x estimate noisy vector clean vector

48 Transcription as Separation Transcribe piano recordings by classification train SVM detectors for every piano note 88 separate detectors, independent smoothing Trained on player piano recordings Bach 847 Disklavier freq / pitch A6 A5 A4 A3 A A time / sec Sse transcription to resynthesize... level / db Music Information Extraction - Ellis p. 48/32

49 Piano Transcription Results Significant improvement from classifier: frame-level accuracy results: Algorithm Errs False Pos False Neg d SVM 43.3% 27.9% 15.4% 3.44 Klapuri&Ryynänen 66.6% 28.1% 38.5% 2.71 Marolt 84.6% 36.5% 48.1% 2.35 Breakdown by frame type: Classification error % False Negatives False Positives # notes present Music Information Extraction - Ellis p. 49/32

50 Outline 1. The ASA Problem 2. Human ASA 3. Machine Source Separation 4. Systems & Examples 5. Concluding Remarks Evaluation Auditory Scene Analysis - Dan Ellis /54

51 Evaluation How to measure separation performance? depends what you are trying to do SNR? energy (and distortions) are not created equal different nonlinear components [Vincent et al. 06] Intelligibility? rare for nonlinear processing to improve intelligibility listening tests expensive ASR performance? Transmission errors optimum separate-then-recognize too simplistic; ASR needs to accommodate separation Net effect Increased artefacts Reduced interference Agressiveness of processing Auditory Scene Analysis - Dan Ellis /54

52 Evaluating Scene Analysis Need to establish ground truth subjective sources in real sound mixtures? Auditory Scene Analysis - Dan Ellis /54

53 More Realistic Evaluation Real-world speech tasks crowded environments applications: communication, command/control, transcription Metric human intelligibility? diarization annotation (not transcription) Lags freq / khz Personal Audio - Speech + Noise Pitch Track + Speaker Active Ground Truth time / sec level / db ks-noisyspeech.wav Auditory Scene Analysis - Dan Ellis /54

54 Summary & Conclusions Listeners do well separating sound mixtures using signal cues (location, periodicity) using source-property variations Machines do less well difficult to apply enough constraints need to exploit signal detail Models capture constraints learn from the real world adapt to sources Separation feasible in certain domains describing source properties is easier Auditory Scene Analysis - Dan Ellis /54

55 Sources / See Also NSF/AFOSR Montreal Workshops 03, 04 labrosa.ee.columbia.edu/montreal2004/ as well as the resulting book... Hanse meeting: Hanse/ DeLiang Wang s ICASSP 04 tutorial Martin Cooke s NIPS 02 tutorial Auditory Scene Analysis - Dan Ellis /54

56 References 1/2 [Barker et al. 05] J. Barker, M. Cooke, D. Ellis, Decoding speech in the presence of other sources, Speech Comm. 45, 5-25, [Bell & Sejnowski 95] A. Bell & T. Sejnowski, An information maximization approach to blind separation and blind deconvolution, Neural Computation, 7: , [Blin et al. 04] A. Blin, S. Araki, S. Makino, A sparseness mixing matrix estimation (SMME) solving the underdetermined BSS for convolutive mixtures, ICASSP, IV-85-88, [Bregman 90] A. Bregman, Auditory Scene Analysis, MIT Press, [Brungart 01] D. Brungart, Informational and energetic masking effects in the perception of two simultaneous talkers, JASA 109(3), March [Brungart et al. 01] D. Brungart, B. Simpson, M. Ericson, K. Scott, Informational and energetic masking effects in the perception of multiple simultaneous talkers, JASA 110(5), Nov [Brungart et al. 02] D. Brungart & B. Simpson, The effects of spatial separation in distance on the informational and energetic masking of a nearby speech signal, JASA 112(2), Aug [Brown & Cooke 94] G. Brown & M. Cooke, Computational auditory scene analysis, Comp. Speech & Lang. 8 (4), , [Cooke et al. 01] M. Cooke, P. Green, L. Josifovski, A. Vizinho, Robust automatic speech recognition with missing and uncertain acoustic data, Speech Communication 34, , [Cooke 06] M. Cooke, A glimpsing model of speech perception in noise, submitted to JASA. [Darwin & Carlyon 95] C. Darwin & R. Carlyon, Auditory grouping Handbk of Percep. & Cogn. 6: Hearing, , Academic Press, [Ellis 96] D. Ellis, Prediction-Driven Computational Auditory Scene Analysis, Ph.D. thesis, MIT EECS, [Hu & Wang 04] G. Hu and D.L. Wang, Monaural speech segregation based on pitch tracking and amplitude modulation, IEEE Tr. Neural Networks, 15(5), Sep [Okuno et al. 99] H. Okuno, T. Nakatani, T. Kawabata, Listening to two simultaneous speeches, Speech Communication 27, , Auditory Scene Analysis - Dan Ellis /54

57 References 2/2 [Ozerov et al. 05] A. Ozerov, P. Phillippe, R. Gribonval, F. Bimbot, One microphone singing voice separation using source-adapted models, Worksh. on Apps. of Sig. Proc. to Audio & Acous., [Pearlmutter & Zador 04] B. Pearlmutter & A. Zador, Monaural Source Separation using Spectral Cues, Proc. ICA, [Parra & Spence 00] L. Parra & C. Spence, Convolutive blind source separation of non-stationary sources, IEEE Tr. Speech & Audio, , [Reyes et al. 03] M. Reyes-Gómez, B. Raj, D. Ellis, Multi-channel source separation by beamforming trained with factorial HMMs, Worksh. on Apps. of Sig. Proc. to Audio & Acous., 13 16, [Roman et al. 02] N. Roman, D.-L. Wang, G. Brown, Location-based sound segregation, ICASSP, I , [Roweis 03] S. Roweis, Factorial models and refiltering for speech separation and denoising, EuroSpeech, [Schimmel & Atlas 05] S. Schimmel & L. Atlas, Coherent Envelope Detection for Modulation Filtering of Speech, ICASSP, I , [Slaney & Lyon 90] M. Slaney & R. Lyon, A Perceptual Pitch Detector, ICASSP, , [Smaragdis 98] P. Smaragdis, Blind separation of convolved mixtures in the frequency domain, Intl. Wkshp. on Indep. & Artif.l Neural Networks, Tenerife, Feb [Seltzer et al. 02] M. Seltzer, B. Raj, R. Stern, Speech recognizer-based microphone array processing for robust hands-free speech recognition, ICASSP, I , [Varga & Moore 90] A. Varga & R. Moore, Hidden Markov Model decomposition of speech and noise, ICASSP, , [Vincent et al. 06] E. Vincent, R. Gribonval, C. Févotte, Performance measurement in Blind Audio Source Separation. IEEE Trans. Speech & Audio, in press. [Yilmaz & Rickard 04] O. Yilmaz & S. Rickard, Blind separation of speech mixtures via time-frequency masking, IEEE Tr. Sig. Proc. 52(7), , Auditory Scene Analysis - Dan Ellis /54