Sound Interfaces Engineering Interaction Technologies. Prof. Stefanie Mueller HCI Engineering Group

Transcription

1 Sound Interfaces Engineering Interaction Technologies Prof. Stefanie Mueller HCI Engineering Group

2 what is sound?

3 if a tree falls in the forest and nobody is there does it make sound? <30 sec brainstorming> [George Berkley, 1710]

4 yes and no! sound:: pressure transmitted through a solid, liquid or gas mechanical wave (atoms vibrate) sound = perception of this wave through the ear + brain we can only hear specific sounds (20Hz - 20KHz)

5 example: drum drum has flexible membrane when being hit, membrane oscillates pushes atoms in the surroundings -> vibrate your ear perceives the vibrations and your brain transforms into the notion of sound

6

7 machines don t perceive sound they just perceive the vibration

8 example: contact microphone also called a piezo (piezo = pressure) piezo-electric effect = electricity from pressure

9 capture sound (use as microphone): sound in surrounding -> piezo vibrates, creates electricity create sound (use as speaker): electricity from computer -> piezo vibrates, creates sound piezoelectric material (positively charged) metal (negatively charged)

10 2015: MOGEES contact mic + gesture recognition software

11 let s look at different categories of sound interfaces

12 sound interfaces: speech output text-to-speech synthesizers input siri, google glass put that there non-speech notifications, earcons microphones as sensors

13 #1 speech output

14 Stephen Hawking used speech synthesizer cheek movement to select letters cheek movement detected with sensor

15 1993 GloveTalk gesture interface

16 1993 GloveTalk gesture interface

17

18 challenges: concatenating sounds emphasis on words sentence melody tone, pauses

19 draw-to-sing synthesis

20

21 has anyone used speech output before? <raise you hand>

22 hard to master, but easy to get started with in terminal: > say hello

23 great for debugging spatial UI when you have no text-output

24 sound interfaces: speech output text-to-speech synthesizers input siri, google glass put that there non-speech notifications, earcons microphones as sensors

25 #2 non-speech output

26 we get a lot of feedback from out computers often in the form of..

27 earcon: auditory equivalent of an icon (eye + con) can you come up with other examples? <30s brainstorming>

28 arbitrary rely on social conventions some similarities but not exactly as real-world meaning depends on real-world physics

29 Windows (symbolic) Mac trash (nomic) (be careful with mapping)

30

31

32 sound interfaces: speech output text-to-speech synthesizers input siri, google glass put that there non-speech notifications, earcons microphones as sensors

33 #3 speech input

34 1980! Put that there

35

36 benefits of speech input: voice commands are useful when hands-busy voice commands make interactions very human -like ( create yellow circle here rather than click > menu: select shape > circle > click)

37

38 easy to integrated today! has anyone used this for their projects / classes? <raise you hand>

39 live translation - holy grail speech-to-text translate text-to-speech

40 2014: Microsoft Skype Translator demo

41 live translation - holy grail speech-to-text translate text-to-speech

42 sound interfaces: speech output text-to-speech synthesizers input siri, google glass put that there non-speech notifications, earcons microphones as sensors

43 #4 non-speech input

44 command computer with voice but not speech..

45 2001: voice as sound: more immediate feedback in contrast to put-that-there

46

47

48 2007: voice draw

49

50 command computer with sounds from the environment

51 1999: pingpongplus a touch screen made from microphones

52 measure time signal needs to travel to each microphone

53 similar triangulation as seen in multitouch lecture

54

55 which of these would you use for making a touch sensor? <30s brainstorming>

56 contact microphone: vibration through surfaces (if you speak into this, nothing happens) other microphone: vibration through air none of these good for touch sensor

57 2002: before any major touch screen technology

58

59 also works with feet

60

61 microphones are still used today as part of touch sensing in mobile phones.

62

63

64 acoustic features: acoustic signature of the touch (frequency, amplitude)

65

66 implicit mode switching -> less graphical UI

67

68 2011 paper, startup 2014.

69 more acoustic gestures

70

71 not just good for touch screens, but also for markers!

72 2012 acoustic barcodes

73

74 2015: 3D printed passive sliders, dials

75 no training data needed, they use the 3D model geometry to do the prediction!

76

77 the future

78

79

80 but there s more touch input anywhere!

81 2010 Skinput: input on 3D deformable surface

82 2010 Skinput: input on 3D deformable surface

83

84 2008 Scratch Input: walls are all connected

85

86 conclusions

87 sound interfaces: speech output text-to-speech synthesizers input siri, google glass put that there non-speech notifications, earcons microphones as sensors

88 end.