11 Music and Speech Perception

Transcription

1 11 Music and Speech Perception Properties of sound Sound has three basic dimensions: Frequency (pitch) Intensity (loudness) Time (length) Properties of sound The frequency of a sound wave, measured in cycles per second or Hertz (abbreviated Hz) indicates the number of cycles each wave makes in one second. The more cycles per second, the higher the pitch we hear. Properties of sound The intensity of a sound wave is measured in decibels (abbreviated db). The higher the intensity of a sound, the louder it sounds. low intensity Time (sec) high-pitched tone low-pitched tone Time (sec) high intensity Decibel scale Rustling leaves 10 db Purring cat 30 db Bird singing gnearby 50 db Conversational speech 60 db Barking dog nearby 70 db Roaring lion 90 db Thunder 110 db Jet taking off nearby 120 db 11 Music and Speech Perception Music Speech 1

2 11 Music Music as a way to express thoughts and emotions Pythagoras: Numbers and musical intervals Some clinical psychologists practice music therapy Musical notes Sounds of music extend across frequency range: Hz 11 Frequency Range of Music Octave: The interval between two sound frequencies having ratio of 2:1 Example: Middle C (C4) has fundamental frequency of Hz; notes that are one octave from middle C are (C3) and (C5) There is more to musical pitch than just frequency! 11 Tone Height and Chroma Helix Tone height: A sound quality whereby a sound is heard to be of higher or lower pitch; monotonically related to frequency Tone chroma: A sound quality shared by tones that have the same octave interval Musical helix: visualize musical pitch 2

3 Musical instruments: Produce notes below 4 khz Listeners: Great difficulty perceiving octave relationships between tones when one or both tones are greater than 5 khz Chords: Created when three or more notes are played simultaneously Consonant or dissonant Consonant: Have simple ratios of note frequencies Dissonant: Less elegant ratios of note frequencies Cultural differences Research on music perception: Western vs. Javanese Javanese culture: Fewer notes within an octave; greater variation in note s acceptable frequencies Even young infants can learn to distinguish sounds in their native scale Melody: An arrangement of notes or chords in succession Examples: Twinkle, Twinkle Little Star, Baa Baa Black Sheep Not a sequence of specific sounds: Sensitive to change, (i.e., change in octave) Notes and chords vary in duration: Tempo; fast or slow 11 Dominant Rhythm Rhythm: Not just in music! Lots of activities have rhythm: Walking, waving, finger tapping, etc. Bolton (1894): Experiments with sequence of identical sounds, perfectly spaced in time, but no rhythm; listeners reported hearing first sound of group as accented, while the rest remained unaccented More examples: Car, train rides Syncopated auditory polyrhythms : When different rhythms are overlapped 3

4 Melody development 8-month olds: Able to learn new melodies 7-month olds: Can associate particular movements with particular melodies 11 Speech The Vocal Tract: The airway above the larynx used for production of speech. Includes the oral tract and nasal tract Humans capablility for speech sounds 5000 languages spoken today, utilizing over 850 different speech sounds flexibility of vocal tract: important in speech production Primate vocal tract The evolution of speech: a comparative review W. Tecumseh Fitch Trends in Cognitive Sciences 4(7) July 2000 Specialized vocal resonators Howler Monkey (Alouatta) Gibbon (Hylobates) air sac orangutan chimpanzee human larynx tongue body Human vocal tract Acoustics of speech Phonation Articulation 4

5 Organs of speech Lungs: apply pressure to generate air stream (power supply) Larynx: air forced through the glottis, a small opening between the vocal folds (sound source) Vocal tract: pharynx, oral and nasal cavities serve as complex resonators (filter) Source-filter theory of speech production Vibrating Lungs Vocal Vocal Output folds Tract Sound Power supply Oscillator Resonator Vocal fold oscillation Source-Filter Theory One-mass model Air flow through the glottis during the closing phase travels at the same speed dbecause of inertia, producing lowered air pressure above the glottis. Source: From Fitch, W.T. (2000). Trends in Cognitive Sciences Audio demo: the source signal Source signal for an adult male voice Source signal for an adult female voice Source signal for a 10-year child Source properties In voiced sounds the glottal source spectrum contains a series of lines called harmonics. The lowest one is called the fundamental frequency (F 0). Amplitude Spectrum Relative Amplitude (db) F Frequency (Hz) 5

6 F 0 range in speech Hz for adult males Hz for adult females Hz for young children Even-tempered Scale for the Octave Above Middle C F 0 measurement F 0 estimates 48 sentences 1 adult male (blue) 1 adult female (red) Hz Hz Intonation patterns Fundamental frequency variation Lee, Potamianos & Narayanan JASA 1999 Declination: pitch tends to fall over the course of a sentence or utterance; declination reset quency (Hz) Fundamental Freq 70 90% shift Females Males Age (years) Filter properties Demo: harmonic synthesis Additive harmonic synthesis: vowel /i/ Cumulative sum of harmonics: vowel /i/ Additive synthesis: wheel Cumulative sum of partials: The vocal tract resonances (called formants) produce peaks in the spectrum envelope. Formants are labelled F1, F2, F3,... in order of increasing frequency. Amplitude Spectrum (with superimposed LPC spectral envelope) Amplitude in db F 1 F 2 F Frequency (khz) F 3 F 4 6

7 11 The Basic Components of Speech Production (Part 1) source filter radiation = output sound / i / Amplitude / A / Frequency 11 The Basic Components of Speech Production (Part 2) Speech Production respiration (lungs) phonation (vocal cords) articulation (vocal tract) Respiration and phonation Initiating speech: diaphragm pushes air out of lungs, through trachea, up to larynx At larynx: Air must pass through two vocal folds Children: Few vocal cords, high-pitched voices Adult men: Larger mass of vocal cords, low-pitched voices Articulation Area above larynx: Vocal tract Humans have ability to change shape of vocal tract by manipulating jaw, lips, tongue, body, tongue tip, velum Manipulations: Articulation Resonance characteristics 7

8 11 Sound from Vocal Folds Peaks in speech spectrum: Formants Labeled by number, from lowest to highest (F1, F2, F3) concentrations in energy occur at different frequencies, depending on length of vocal tract For shorter vocal tracts (children, short adults): Formants are at higher frequencies than for longer vocal tracts Spectrogram Spectral analysis of speech Why perform a frequency analyses of speech? Spectral analysis of speech But: the ear is not a spectrum analyzer. Ear+brain carry out a form of frequency analysis Relevant features of speech are more readily visible in the amplitude spectrum than in the raw waveform Auditory frequency selectivity is best at low frequencies and gets progressively worse at higher frequencies. Short-term amplitude spectrum Speech spectrogram running amplitude spectra (codes amplitude changes in different frequency bands over time). Amplitude (db) F1 = 281 Hz F2 = 2196 Hz F3 = 2755 Hz Frequency (khz) 8

9 11 Sound Spectrogram Speech terminology Fundamental frequency (F0): lowest frequency component in voiced speech sounds, linked to vocal fold vibration. Formants: resonances of the vocal tract. Amplitude F0 Formant Frequency Source properties: Pitch Fundamental frequency (F 0) is determined by the rate of vocal fold vibration, and is responsible for the perceived voice pitch. Harmonicity and Periodicity Period: regularly repeating pattern in the waveform Period duration T0 = 6 ms Waveform Harmonics are integer multiples of F0 and are evenly spaced in frequency Amplitude (db) F 0 = 1000 / 6 = 166 Hz F 0 = 1 / T 0 Amplitude Spectrum F (kh ) Source properties: Pitch F 0 can be removed by filtering (as in telephone circuits) and the pitch remains the same. This is the problem of the missing fundamental, one of the oldest problems in hearing science. Pitch is determined by the frequency pattern of the harmonics (or their equivalent in the time domain, the periodicities in the waveform). Formants correspond to peaks in the spectrum envelope. Amplitude in db F 1 F 2 F 3 F Frequency (khz) 9

10 Vowel formant space: F1 x F2 Assmann & Katz JASA 2000 F2 Frequenc F2 ncy (Hz) (khz) i Males I i I U o Females E U Q o A E Q A Peterson and Barney (1952) f F2 (Hz) Frequency of Peterson and Barney (1952) i Children i ι Women ε ι i ε æ Men æ ι ε æ u u u c c Λ Λ α α Λ α c F1 F1 Frequency (Hz) F1 frequency (Hz) 11 Vowel Sounds of English Classifying speech sounds described in terms of articulation Place of articulation: (e.g., at lips, at alveolar ridge, etc.) Voicing: i Whether cords are vibrating, not vibrating English: Only small sample of sounds used by languages around the world; a lot more sounds are used! Speech perception Speech production: Very fast Experienced talkers: Coarticulation; attributes of successive speech units overlap in articulatory or acoustic patterns Example: Say the word moody a few times, observe what happens to tongue Categorical perception Research on acoustic cues used to distinguish different speech sounds Categorical perception : Sharp labeling (identification), discontinuous discrimination, predictability of discrimination 10

11 11 Categorical Perception How special is speech? Motor theory of speech perception: Special mechanisms just for perceiving speech Problems for motor theory: Speech production is just as complex, so speech perception complexity must be result of this complexity Nonhuman animals can learn to respond to speech signals in similar way to human listeners Categorical speech perception: Not limited to speech sounds; also includes musical intervals; other categorical perceptions: faces, facial expressions Coarticulation and spectral contrast Research: How speech perception is explained by general ways that hearing, and perception works Example: Perception of coarticulated speech; explained by some fundamental ways of auditory system Using multiple acoustic cues Perception depends on experience Comparison with face recognition Contrast effects: Melodies are defined by changes between adjacent notes; spectral contrast helps listeners perceive speech Learning to listen Babies learn to listen even before they are born! Prenatal experience: Newborns prefer hearing their mother s voice over other women s voices Research of babies in France Becoming a native listener Sound distinctions specific to various languages Example: r and l are not distinguished in Japanese Infants begin filtering out irrelevant acoustics long before they start to say speech sounds 11

12 Learning words How do we know where one word ends and another begins? Research (Saffran et al.): Novel language with infants; can learn to distinguish words from nonwords after two minutes Statistical learning Speech in the Brain Brain damage follows patterns of blood vessels, not brain function, so difficult to study PET and fmri studies: Help to learn about speech processing in brain Listening to speech: Left and right superior temporal lobes are activated more strongly in response to speech than to nonspeech sounds Some challenges in creating good controls for experiments Categorical perception tasks How do processes of hearing and speaking interact? Syrinx 12