SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones

Size: px

Start display at page:

Download "SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones"

Sherilyn Ray
5 years ago
Views:

1 SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury, Andrew T. Campbell Dept. of Computer Science, Dartmouth College

2 Mobile Phones are the perfect platform for large scale sensor networks

3 Sounds captured by the microphone allow us to sense people and the environment.

6 design

7 Design Issues Scalable Sound Classifica0on Privacy Resource limita0ons Phone context

8 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

9 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

10 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

11 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

12 Admission control lowers the resources consumed during continuous sensing Filter out silence and non-informative frames *.";2& Criteria: Energy & Spectral Entropy 678&!"##$#%&'$#()'& **+&,#-.)/0&,12#-&3-".-&(2-245)#& 8$2#42&,12#-&(2-24-2(& 9$34".(&:.";2& *).'".(&-)&#2<-&3-"%2&

13 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

14 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

15 Discriminative Acoustic Features Frequency Spectrum based features: Robust to distance/muffling Computa0onally very cheap aber ini0al FFT

16 Discriminative Acoustic Features Spectral Rolloff for Music and Voice Spectral Rolloff Time

17 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

18 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

19 Coarse Category Classification!"#$%&"'()*+$,)%'!"#$%$&'()*""(+,-%%$."*(!"#$%&'!%()*'+),%-./0)#'!"#$%&!"#$%&!"#$%&'()*+&,(

20 Coarse Category Classification!"#$%&"'()*+$,)%' Decision Tree Classifier!"#$%&'!%()*'+),%-./0)#'!"#$%&!"#$%&!"#$%&'()*+&,(

21 Coarse Category Classification!"#$%&"'()*+$,)%' Decision Tree Classifier Music, Music,Voice,Music, Voice, Music, Music,Voice!"#$%&'!%()*'+),%-./0)#'!"#$%&!"#$%&!"#$%&'()*+&,(

22 Coarse Category Classification!"#$%&"'()*+$,)%' Decision Tree Classifier Music, Music,Voice,Music, Voice, Music, Music,Voice S1 S2 S1 S2 S1 S2 Markov Model Recognizer S3 S3 S3!"#$%&!"#$%&!"#$%&'()*+&,(

23 Coarse Category Classification!"#$%&"'()*+$,)%' Decision Tree Classifier Music, Music,Voice,Music, Voice, Music, Music,Voice S1 S2 S1 S2 S1 S2 Markov Model Recognizer S3 S3 S3 Music, Music,Music,Music, Music, Music, Music,Music!"#$%&!"#$%&!"#$%&'()*+&,(

24 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

25 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

26 SoundSense Pipeline Sound Waveform Framing!-./%%/#0'1#0+,#2' Feature Extraction!"#$%&"'()*+$,)%' Coarse Category Classification 3)"/%/#0'4,))'12*%%/5),' 6*,7#8'6#-)2'9)"#:0/;),' Intra-Category Classification <#/")'!0*2%/%'

27 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Mystery Sound 1 Mystery Sound 2

28 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Mystery Sound 1 Mystery Sound 2

29 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Mystery Sound 1 Mystery Sound 2

30 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Car (turning signal) Mystery Sound 2

31 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Car (turning signal) Mystery Sound 2

32 Significant sounds in the life of a user Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Examples: Car (turning signal) Vacuum Cleaner

33 Our Approach Incrementally build models of encountered sounds Rank them by order of importance Human-in-the-loop labeling

34 !"##$%"&'()*&$% &$!"#$!"#$%$!"#$'$!"#$#$ #(%$

35 Sounds modeled as multivariate Gaussians!"##$%"&'()*&$%!"#$ %"#$!"($ %"($ '$ )*&$ &+#$!"&$ %"&$

36 Sounds modeled as multivariate Gaussians!"##$%"&'()*&$%!"#$ %"#$!"($ %"($ '$ )*&$ &+#$!"&$ %"&$

37 Sounds modeled as multivariate Gaussians!"##$%"&'()*&$%!"#$ %"#$!"($ %"($ '$ )*&$ &+#$!"&$ %"&$

38 Sounds modeled as multivariate Gaussians!"##$%"&'()*&$%!"#$ %"#$!"($ %"($ '$ )*&$ &+#$!"&$ %"&$

39 Sound Rank estimates the importance a sound!"##$%"&'()*&$%!"#$ %"#$ 10!"($ %"($ 30 '$ )*&$ &+#$ 50!"&$ %"&$ 3 Sound Rank based on: Dura0on of the event Frequency of the encounter

40 Sound Rank estimates the importance a sound!"##$%"&'()*&$%!"#$ %"#$ 50!"($ %"($ 30 '$ )*&$ &+#$ 10!"&$ %"&$ 3 More Important Less Important

41 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50!"($ %"($ 30 '$ )*&$ &+#$ 10!"&$ %"&$ 3

42 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50!"($ %"($ 30 '$ )*&$ &+#$ 10!"&$ %"&$ 3

43 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50!"($ %"($ 30 '$ )*&$ ignore &+#$!"&$ %"&$ 3

44 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50!"($ %"($ 30 '$ )*&$ &+#$ ignore!"&$ %"&$ 3

45 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50 hide 30!"($ %"($ '$ )*&$ &+#$ ignore!"&$ %"&$ 3

46 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$%!"#$ %"#$ 50 hide 30!"($ %"($ '$ )*&$ &+#$ ignore!"&$ %"&$ 3

47 Human-in-the-loop to identify and validate sounds!"##$%"&'()*&$% label 50!"#$ %"#$ hide 30!"($ %"($ '$ )*&$ &+#$ ignore!"&$ %"&$ 3

48 !"##$%"&'()*&$% label 50!"#$ %"#$ hide 30!"($ %"($ '$ )*&$ &+#$ ignore!"&$ %"&$ 3!""#$%&&'()*#!"#$"#%&'&(#%

49 evaluation

50 iphone implementation Jail broken iphone C, C++, Objec0ve C 300 KB applica0on footprint

51 SoundSense implementation has low levels of resource consumption on the iphone cpu usage range memory usage range GUI only < 1% 2.79 MB silence 1-5 % 3.75 MB music/speech 8% - 20% 3.80 MB ambient sound 11% - 22% 3.80 MB ~ 5 MB

52 SoundSense implementation has low levels of resource consumption on the iphone cpu usage range memory usage range GUI only < 1% 2.79 MB silence 1-5 % 3.75 MB music/speech 8% - 20% 3.80 MB ambient sound 11% - 22% 3.80 MB ~ 5 MB

53 Coarse Classification using Decision Tree only ambient sound music voice ambient sound music voice

54 Coarse Classification using Decision Tree only ambient sound music voice ambient sound music voice

55 Coarse Classification using Decision Tree only ambient sound music voice ambient sound music voice

56 Coarse Classification using Decision Tree and Markov Model smoothing ambient sound music voice ambient sound music voice

57 Unsupervised Adaptive Classifier Learning New Sound Classes Sound Rank 1st 2nd 3rd 4th 5th 6th 7th Event Phone rubbing against clothes Highway driving Walking outdoors Driving w/ acceleration In-town driving Walking indoors Taking out trash

58 Unsupervised Adaptive Classifier Learning New Sound Classes Sound Rank 1st 2nd 3rd 4th 5th 6th 7th Event Phone rubbing against clothes Highway driving Walking outdoors Driving w/ acceleration In-town driving Walking indoors Taking out trash

59 Unsupervised Adaptive Classifier Learning New Sound Classes Sound Rank 1st 2nd 3rd 4th 5th 6th 7th Event Phone rubbing against clothes Highway driving Walking outdoors Driving w/ acceleration In-town driving Walking indoors Taking out trash

60 Sound Sense Diary: Discovering Life Patterns

61 Sound Sense Diary: Discovering Life Patterns

62 Sound Sense Diary: Discovering Life Patterns

63 Music Tagger: Building Apps with SoundSense

64 Related Work recognition/audio scene analysis using resource constrained devices e.g., Clarkson et al. 98, Smith et al. 06 specific types of sound analysis: speech recognition, music genre classification, speaker identification sensing with mobile phones: UCLA, UIUC, Nokia, Microsoft, Motorola, Intel, UW, Duke

65 Summary SoundSense addresses the needs of sound classification using mobile phones: Privacy, Energy Efficiency & robustness to Phone Context Scalability via unsupervised learning of everyday sounds Future Work - Improving Ambient Sound Learning Revising Sound Rank Coping with fragmented learnt classes

66 Thanks for listening Questions?

Sound Analysis Research at LabROSA

Sound Analysis Research at LabROSA Dan Ellis Laboratory for Recognition and Organization of Speech and Audio Dept. Electrical Eng., Columbia Univ., NY USA dpwe@ee.columbia.edu http://labrosa.ee.columbia.edu/