Acoustic Sensing With Artificial Intelligence

Transcription

1 Acoustic Sensing With Artificial Intelligence Bowon Lee Department of Electronic Engineering Inha University Incheon, South Korea NVIDIA Deep Learning Day Seoul, South Korea October 31, 2017 Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

2 Outline 1 Introduction 2 Acoustic Sensing With a Microphone Array 3 Sound Source Localization 4 Human Hearing 5 Blind Source Separation 6 Concluding Remarks Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

3 Telephone (1876) Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

4 Home Assistants (Current) Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

5 AVICAR Project at UIUC ( ) AVICAR Video Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

6 ConnectUs Project at HP Labs ( ) ConnectUs Video Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

7 Outline 1 Introduction 2 Acoustic Sensing With a Microphone Array 3 Sound Source Localization 4 Human Hearing 5 Blind Source Separation 6 Concluding Remarks Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

8 Acoustic Sensing With a Microphone Array Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

9 Acoustic Sensing With a Microphone Array Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

10 Acoustic Sensing With a Microphone Array Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

11 Acoustic Sensing With a Microphone Array Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

12 Acoustic Sensing With a Microphone Array Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

13 Audio Communication Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

14 Signal Processing for Audio Communication Output signal of the capture engine ŷ n (t) = M {f mn y m }(t) m=1 }{{} microphone array processing Topics of Acoustic Signal Processing Beamforming Blind Source Separation Sound Source Localization Multichannel Echo Cancellation + L {q ln x l }(t) l=1 } {{ } echo cancellation Dereverberation, Noise Suppression, Speaker Diarization Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

15 Signal Processing for Audio Communication Output signal of the capture engine ŷ n (t) = M {f mn y m }(t) m=1 }{{} microphone array processing Topics of Acoustic Signal Processing Beamforming Blind Source Separation Sound Source Localization Multichannel Echo Cancellation + L {q ln x l }(t) l=1 } {{ } echo cancellation Dereverberation, Noise Suppression, Speaker Diarization Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

16 Signal Processing for Audio Communication Output signal of the capture engine ŷ n (t) = M {f mn y m }(t) m=1 }{{} microphone array processing Topics of Acoustic Signal Processing Beamforming Blind Source Separation Sound Source Localization Multichannel Echo Cancellation + L {q ln x l }(t) l=1 } {{ } echo cancellation Dereverberation, Noise Suppression, Speaker Diarization Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

17 Outline 1 Introduction 2 Acoustic Sensing With a Microphone Array 3 Sound Source Localization 4 Human Hearing 5 Blind Source Separation 6 Concluding Remarks Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

18 Sound Sources in an Acoustic Scene Sound source of interest Noise source Object Microphones M-1 Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

19 Time-Delay Estimation Example: Signals at two microphones Amplitude Samples Mic 1 Mic 2 Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

20 Signal Model: Two Microphones Signals in the discrete time domain { x1 [n] = s[n]+v 1 [n] x 2 [n] = s[n ]+v 2 [n] x 1 [n], x 2 [n]: Captured signal at each microphone s[n]: Source signal = f s τ: Time difference of arrival (TDOA) in samples v 1 [n], v 2 [n]: Noise and reverberation at each microphone Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

21 Signal Model: Two Microphones Signals in the discrete time domain { x1 [n] = s[n]+v 1 [n] x 2 [n] = s[n ]+v 2 [n] x 1 [n], x 2 [n]: Captured signal at each microphone s[n]: Source signal = f s τ: Time difference of arrival (TDOA) in samples v 1 [n], v 2 [n]: Noise and reverberation at each microphone Time-Delay Estimation A problem to find given x 1 [n] and x 2 [n] Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

22 Generalized Cross Correlation Method for TDE Signals in the frequency domain { X1 (ω) = S(ω)+V 1 (ω) X 2 (ω) = S(ω)e jω + V 2 (ω) Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

23 Generalized Cross Correlation Method for TDE Signals in the frequency domain { X1 (ω) = S(ω)+V 1 (ω) X 2 (ω) = S(ω)e jω + V 2 (ω) Generalized Cross Correlation (GCC) Method ˆ = arg max ψ(ω)x 1 (ω)x2 (ω)ejω dω ω Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

24 Generalized Cross Correlation Method for TDE Signals in the frequency domain { X1 (ω) = S(ω)+V 1 (ω) X 2 (ω) = S(ω)e jω + V 2 (ω) Generalized Cross Correlation (GCC) Method ˆ = arg max ψ(ω)x 1 (ω)x2 (ω)ejω dω ω Phase Transform (PHAT) ψ PHAT (ω) = 1/ X 1 (ω)x2 (ω) X 1 (ω)x2 ˆ = arg max (ω) X 1 (ω)x2 dω (ω) ejω ω Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

25 Sound Source Localization With a Microphone Array Multi-microphone extension of the GCC method M M ˆq = arg max ψ(ω)x m (ω)xk (ω)ejω q mk dω, q Q m=1 k=1 ω Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

26 Sound Source Localization With a Microphone Array Multi-microphone extension of the GCC method M M ˆq = arg max ψ(ω)x m (ω)xk (ω)ejω q mk dω, q Q m=1 k=1 ω where q mk = q k q m and Q denotes the search space of all potential source locations. Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

27 Sound Source Localization With a Microphone Array Multi-microphone extension of the GCC method M M ˆq = arg max ψ(ω)x m (ω)xk (ω)ejω q mk dω, q Q m=1 k=1 ω where q mk = q k q m and Q denotes the search space of all potential source locations. With the PHAT weighting, it can be simplified as M X m (ω) q ˆq = arg max q Q ω X m (ω) ejω m m=1 2 dω, Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

28 Steered Response Power With Phase-Transform SRP-PHAT Power Plot Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

29 SRP-PHAT Computation on GPU SingleCPU thread Block Block Block n... Serial scan CPU!based common approach Parallel scan Our GPU!based approach Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

30 Simultaneous Speaker Detection and Localization Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

31 System Setup RGB Camera Microphone Array Camera s Field of View Eight microphones and one RGB camera Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

32 Multimodal Speaker Detection and Localization Use the HMM competition scheme Run it for each one of the detected faces The SRP-PHAT plot has many local maxima for multiple simultaneous speaker cases Mid-Fusion of the parameters before HMM competition 1 Mid-Fusion Algorithm for Speaker Detection and Localization 1 V. P.Minotto, C. R. Jung, and B. Lee, Simultaneous-Speaker Voice Activity Detection and Localization Using Mid-Fusion of SVM and HMMs, IEEE TMM 2014 Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

33 Sound Source Localization Note that: Sound source localization methods work reasonably well For long (200 ms or longer) signals, With relatively low reverberation and background noise, or With the help of multimodal sensor fusion Otherwise, they usually fail. Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

34 Sound Source Localization Note that: Sound source localization methods work reasonably well For long (200 ms or longer) signals, With relatively low reverberation and background noise, or With the help of multimodal sensor fusion Otherwise, they usually fail. It is generally believed that: Humans do much better than any algorithmic methods. Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

35 Sound Source Localization Note that: Sound source localization methods work reasonably well For long (200 ms or longer) signals, With relatively low reverberation and background noise, or With the help of multimodal sensor fusion Otherwise, they usually fail. It is generally believed that: Humans do much better than any algorithmic methods. Cocktail party effect The ultimate goal of acoustic sensing Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

36 Cocktail Party Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

37 Outline 1 Introduction 2 Acoustic Sensing With a Microphone Array 3 Sound Source Localization 4 Human Hearing 5 Blind Source Separation 6 Concluding Remarks Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

38 Human Hearing Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

39 Human Ear (source: Theory and Applications of Digital Speech Processing, Rabiner and Schafer, Pearson, 2011) Acoustic wavefront hits the eardrum (outer ear) Movement at the eardrum is transmitted via ossicles (middle ear) Stapes is attached to the oval window of the cochlea (inner ear) Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

40 Sound Representation in the Auditory System (source: Theory and Applications of Digital Speech Processing, Rabiner and Schafer, Pearson, 2011) Cochlear processing Vibration at the oval window is converted to fluid motion in the cochlea Fluid motion causes mechanical vibration on the basilar membrane Mechanical vibration causes action potential in the inner hair cell These action potentials trigger neural firings to be aggregated in the central auditory system Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

41 Auditory System: A Neural Network (source: Theory and Applications of Digital Speech Processing, Rabiner and Schafer, Pearson, 2011) Auditory System The auditory cortex serves as the central processing unit It creates multiple representations of the neural firings Then the human brain perceives the sound Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

42 Outline 1 Introduction 2 Acoustic Sensing With a Microphone Array 3 Sound Source Localization 4 Human Hearing 5 Blind Source Separation 6 Concluding Remarks Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

43 Spectrogram Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

44 Spectrogram Time-Frequency representation of audio Can be treated as a 2D image Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

45 Semantic Segmentation: Fully Convolutional Network Semantic segmentation: Pixel-wise image segmentation Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

46 Blind Source Separation: Fully Convolutaional Network Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

47 Blind Source Separation: Some Results The FCN created the binary mask from a mixture signal Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

48 Summary Acoustic Sensing Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

49 Summary Acoustic Sensing + Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

50 Summary Acoustic Sensing + Artificial Intelligence Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

51 Conclusion Past Current Future Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36

52 Questions? Bowon Lee, Inha University Acoustic Sensing With Artificial Intelligence NVIDIA Deep Learning Day / 36