Lecture 8: Spatial sound. Spatial acoustics

Transcription

1 EE E682: Speech & Audio Processing & Recogniion Lecure 8: Spaial sound Spaial acousics Binaural percepion Synhesizing spaial audio Exracing spaial sounds Dan Ellis <dpwe@ee.columbia.edu> hp:// Columbia Universiy Dep. of Elecrical Engineering Spring 26 E682 SAPR - Dan Ellis L8 - Spaial sound Spaial acousics Received sound = source + channel - so far, only considered ideal source waveform Sound carries informaion on is spaial origin - e.g. ripples in he lake - evoluionary significance The basis of scene analysis? - yes and no - ry blocking an ear E682 SAPR - Dan Ellis L8 - Spaial sound

2 ! Ripples in he lake Lisener Source Wavefron c m/s) Energy " 1/r 2 Effec of relaive posiion on sound - delay =!r/c - energy decay ~ 1/r 2 - absorpion ~ G(f) r - direc energy plus reflecions Give cues for recovering source posiion Describe wavefron by is normal E682 SAPR - Dan Ellis L8 - Spaial sound Recovering spaial informaion Source direcion as wavefron normal - moving plane found from iming a 3 poins B pressure!/c =!s = AB cos# # A C wavefron ime - need o solve correspondence range r Space: need 3 parameers - e.g. 2 angles and range elevaion $ azimuh # E682 SAPR - Dan Ellis L8 - Spaial sound

3 ! The effec of he environmen Reflecion causes addiional wavefrons reflecion diffracion & shadowing freq / Hz scaering, absorpion - many pahs % many echoes Reverberan effec - causal smearing of signal energy y p y 8 freq / Hz ime / sec E682 SAPR - Dan Ellis L8 - Spaial sound ime / sec!!! Reverberaion impulse response Exponenial decay of reflecions: h room () ~e - /T freq / Hz 8 6 hlwy16-128p window ime / s Frequency-dependen - greaer absorpion a high frequencies % faser decay Size-dependen - larger rooms % longer delays % slower decay Sabine s equaion:.49v RT 6 = S& Time consan as size, absorpion E682 SAPR - Dan Ellis L8 - Spaial sound

4 Ouline Spaial acousics Binaural percepion - The sound a he wo ears - Available cues - Percepual phenomena Synhesizing spaial audio Exracing spaial sounds E682 SAPR - Dan Ellis L8 - Spaial sound Binaural percepion R L head shadow (high freq) source pah lengh difference Wha is he informaion in he 2 ear signals? - he sound of he source(s) (L+R) - he posiion of he source(s) (L-R) Example waveforms (ShATR daabase) shar78m3 waveform.1.5 Lef -.5 Righ ime / s E682 SAPR - Dan Ellis L8 - Spaial sound

5 !! freq / khz Main cues o spaial hearing Ineraural ime difference (ITD) - from differen pah lenghs around head - dominaes in low frequency (< 1.5 khz) - max ~ 75 "s % ambiguous for freqs > 6 Hz Ineraural inensiy difference (IID) - from head shadowing of far ear - negligable for LF; increases wih frequency Specral deail (from pinna relfecions) useful for elevaion & range Direc-o-reverberan useful for range Claps 33 and 34 from 627M:nf E682 SAPR - Dan Ellis L8 - Spaial sound ime / s Head-Relaed Transfer Fns (HRTFs) Capure source coupling as impulse responses & l ## $ # R $ % # r ## $ # R $ % ' Collecion: (hp://inerface.cipic.ucdavis.edu/) RIGHT HRIR_21 el HRIR_21 el 1 HRIR_21 el az 45 Azimuh / deg 1 HRIR_21 el az -45 LEFT Highly individual! -1 ime / ms ime / ms E682 SAPR - Dan Ellis L8 - Spaial sound

6 Cone of confusion azimuh # Cone of confusion (approx equal ITD) Ineraural iming cue dominaes (below 1kHz) - from differing pah lenghs o wo ears Bu: only resolves o a cone - Up/down? Fron/back? E682 SAPR - Dan Ellis L8 - Spaial sound Furher cues Pinna causes elevaion-dependen coloraion Monaural percepion - separae coloraion from source specrum? Head moion - synchronized specral changes - also for ITD (fron/back) ec. E682 SAPR - Dan Ellis L8 - Spaial sound

7 Combining muliple cues Boh ITD and ILD influence azimuh; Wha happens when hey disagree? Idenical signals o boh ears % image is cenered l() r() 1 ms l() Delaying righ channel moves image o lef r() Aenuaing lef channel reurns image o cener l() r() - around.1 ms / db E682 SAPR - Dan Ellis L8 - Spaial sound ! Binaural posiion esimaion Imperfec resuls: (Arruda, Kisler & Wighman 1992) 18 Judged Azimuh (Deg) Targe Azimuh (Deg) - lisening o wrong hrfs % errors - fron/back reversals say on cone of confusion E682 SAPR - Dan Ellis L8 - Spaial sound

8 The Precedence Effec Reflecions give misleading spaial cues R l() r() direc refleced R /c Bu: Spaial impression based on 1s wavefron hen swiches off for ~5 ms -.. even if reflecions are louder -.. leads o impression of room E682 SAPR - Dan Ellis L8 - Spaial sound Binaural Masking Release Adding noise o reveal arge Tone + noise o one ear: one is masked + Idenical noise o oher ear: one is audible + - why does his make sense? Binaural Masking Level Difference up o 12dB - greaes for noise in phase, one ani-phase E682 SAPR - Dan Ellis L8 - Spaial sound

9 Ouline Spaial acousics Binaural percepion Synhesizing spaial audio - Posiion - Environmen Exracing spaial sounds E682 SAPR - Dan Ellis L8 - Spaial sound Synhesizing spaial audio Goal: recreae realisic soundfield - hi-fi experience - synheic environmens (VR) Consrains - resources - informaion (individual HRTFs) - delivery mechanism (headphones) Source maerial ypes - live recordings (acual soundfields) - synheic (sudio mixing, virual environmens) E682 SAPR - Dan Ellis L8 - Spaial sound

10 Classic sereo L R Inensiy panning : no iming modificaions, jus vary level ±2 db - works as long as lisener is equidisan (ILD) Surround sound: exra channels in cener, sides,... - same basic effec - pan beween pairs E682 SAPR - Dan Ellis L8 - Spaial sound Simulaing reverberaion Can characerize reverb by impulse response - spaial cues are imporan - record in sereo - IRs of ~ 1 sec % very long convoluion Image model: reflecions as duplicae sources virual (image) sources refleced pah source lisener Early echos in room impulse response: direc pah early echos h room () Acual reflecion may be h reflec (), no '() E682 SAPR - Dan Ellis L8 - Spaial sound

11 Arificial reverberaion Reproduce percepually salien aspecs - early echo paern (% room size impression) - overall decay ail (% wall maerials...) - ineraural coherence (% spaciousness) Nesed allpass filers (Gardner 92) Allpass x[n] -g + z -k + y[n] z -k - g H(z) = 1 - g z -k h[n] 1-g 2 g(1-g 2 ) g 2 (1-g 2 ) g,k g -g k 2k 3k n Nesed+Cascade Allpass 5,.5 3,.7 2,.3 Synheic Reverb + AP AP 1 AP 2 g LPF + + a a 1 a 2 E682 SAPR - Dan Ellis L8 - Spaial sound Synheic binaural audio Source convolved wih {L,R} HRTFs gives precise posiioning -...for headphone presenaion - can combine muliple sources (by adding) Source Where o ge HRTFs? - measured se, bu: specific o individual, discree - inerpolae by linear crossfade, PCA basis se - or: parameric model - delay, shadow, pinna Delay Shadow Pinna z - DL (#) 1 - b L (#)z az Room echo K E z - E (!p kl (#,$) z - PkL(#,$) + z - DR (#) 1 - b R (#)z az (!p kr (#,$) z - PkR(#,$) Head moion cues? - head racking + fas updaes E682 SAPR - Dan Ellis L8 - Spaial sound (afer Brown & Duda '97)

12 Transaural sound Binaural signals wihou headphones? Can cross-cancel wrap-around signals - speakers S L,R, ears E L,R, binaural signals B L,R. 1 S L = H LL $ B L H RL S R % B L B R M 1 S R = H RR $ B R H LR S L % S L S R H LR H RL H LL H RR E L E R Narrow swee spo - head moion? E682 SAPR - Dan Ellis L8 - Spaial sound Soundfield reconsrucion Sop hinking abou ears jus reconsruc pressure + spaial derivaives )p()/ )z )p()/ )y )p()/)x p(x,y,z,) - ears in reconsruced field receive same sounds Complex reconsrucion seup (ambisonics) - able o preserve head moion cues? E682 SAPR - Dan Ellis L8 - Spaial sound

13 Ouline Spaial acousics Binaural percepion Synhesizing spaial audio Exracing spaial sounds - Microphone arrays - Modeling binaural processing E682 SAPR - Dan Ellis L8 - Spaial sound Exracing spaial sounds Given access o soundfield, can we recover separae componens? - degrees of freedom: >N signals from N sensors is hard - bu: people can do i (somewha) Informaion-heoreic approach - use only very general consrains - rely on precision measuremens Anhropic approach - examine human percepion - aemp o use same informaion E682 SAPR - Dan Ellis L8 - Spaial sound

14 Microphone arrays Signals from muliple microphones can be combined o enhance/cancel cerain sources Coinciden mics wih diff. direcional gains m 1 s 1 a a a 22 a 21 m 2 s 2 m 1 a 11 a 12 s = * 1 m 2 a 21 a 22 s 2 s + 1ˆ = A 1 * m s 2ˆ Microphone arrays (endfire), = 4D -2, = 2D -4, = D + D + D + D E682 SAPR - Dan Ellis L8 - Spaial sound Adapive Beamforming & Independen Componen Analysis (ICA) Formulae mahemaical crieria o opimize Beamforming: Drive inerference o zero - cancel energy during nonarge inervals ICA: maximize muual independence of oupus - from higher-order momens during overlap m 1 a 11 a s x 12 1 m 2 a 21 a 22 s 2 -' MuInfo 'a Limied by separaion model parameer space - only NxN? E682 SAPR - Dan Ellis L8 - Spaial sound

15 Binaural models Human liseners do beer? - cerainly given only 2 channels Exrac ITD and IID cues? Targe azimuh Cener freq / Hz Ineraural cross-correlaion lag / ms - cross-correlaion finds iming differences - consume couner-moving pulses - how o achieve IID, rading - verical cues... E682 SAPR - Dan Ellis L8 - Spaial sound Nonlinear filering How o separae sounds based on direcion? - esimae direcion locally - choose arge direcion - remove energy from oher direcions E.g. Kollmeier, Peissig & Hohman 93 frequency FFT Modulus l analysis L w L w 2 Smooh (1-a) (1-az -1 ) S LL OLA- FFT synhesis l' Crosscorrelaion L w R * w Smooh (1-a) (1-az -1 ) S LR Gain facor calc g ime FFT Modulus r analysis R w R w 2 Smooh (1-a) (1-az -1 ) S RR OLA- FFT synhesis r' X w (mh,2.k/nt) - IID from L w / R w ; ITD (IPD) from arg{l w R w * } - mach o IID/IPD emplae for desired direcion - also reverberaion? E682 SAPR - Dan Ellis L8 - Spaial sound

16 Summary Spaial sound - sampling a more han one poin gives informaion on origin direcion Binaural percepion - ime & inensiy cues used beween/wihin ears Sound rendering - convenional sereo - HRTF-based Spaial analysis - opimal linear echniques - elusive audiory models E682 SAPR - Dan Ellis L8 - Spaial sound References B.C.J. Moore, An inroducion o he psychology of hearing (4h ed.) Academic, J. Blauer, Spaial Hearing (revised ed.), MIT Press, R.O. Duda, Sound Localizaion Research, hp:// E682 SAPR - Dan Ellis L8 - Spaial sound