Hearing the Universal Language: Music and Cochlear Implants Professor Hugh McDermott Deputy Director (Research) The Bionics Institute of Australia, Professorial Fellow The University of Melbourne Overview? MUSIC, HEARING, AND DEAFNESS MUSIC PERCEPTION WITH COCHLEAR IMPLANTS IMPROVING THE MUSIC LISTENING EXPERIENCE THE FUTURE What is music? 1
Music is the universal language of mankind Henry Wadsworth Longfellow What is music? It s hard to define, but most people have an opinion about it. Whatever the style, musical sounds are usually composed of the same basic elements Do Cochlear Implant Users Enjoy Listening to Music? Numerical ratings of enjoyment of music from 0 ( not at all ) to 10 ( very much ): 35 adult implant recipients (Mirza et al., 2003) before onset of profound hearing loss: 8.7 after receiving a cochlear implant: 2.6 16 users who listened to music with the CI: 5.6 But for children who use cochlear implants and who have no recollection of natural hearing, subjective appraisal of musical sounds is generally similar to that of their normally hearing peers There is very wide variability across individual listeners! How Well Do Cochlear Implant Users Perceive Musical Sounds? Identification of musical instrument sounds Adult CI users listened to recordings of 16 different instrument sounds, and were asked to name each instrument 2
Musical Instrument Identification Tabourine Drums Bass Drum Guitar Harp Xylophone Bells Piano Male singer Female singer Organ Trumpet Double Bass Violin Clarinet Flute 0 20 40 60 80 100 Percent correct McDermott & Looi, 2004 Subjective Quality Ratings for Musical Instrument Sounds After the identification tests, the same subjects were asked to provide a numerical quality rating for each of the sounds 0 = unrecognisable, 10 = perfectly natural (i.e. it sounds just the way I remember it ) Subjects were told what each sound was before they gave their rating Musical Instrument Sound Quality Ratings Tabourine Drums Bass Drum Guitar Harp Xylophone Bells Piano Male singer Female singer Organ Trumpet Double Bass Violin Clarinet Flute 0 1 2 3 4 5 6 7 8 9 10 Quality rating McDermott & Looi, 2004 3
Elements of Musical Sounds PERCEPTUAL ACOUSTIC Loudness rhythm Pitch melody harmony Timbre instrument identity Intensity or level rapid level variations Frequency fundamental frequency (F0) Level distribution across frequency spectrum The Piano Keyboard MIDDLE C Fundamental Frequency (f 0 ) LOW PITCH HIGH 33 Hz 262 Hz 2093 Hz Waveform of a Musical Note at Middle-C T = 1/F0: 262 Hz air pressure time 4
Spectrum of a Musical Note at Middle-C F0: 262 Hz Level (db) 0 1000 2000 3000 4000 5000 Frequency (Hertz) Pitch information is present in both the temporal and spectral features of acoustic signals The natural auditory system can make use of both types of information to determine the pitch of sounds, and therefore cochlear implants attempt to deliver both temporal and spectral information Acoustic Frequency Analysis in the Cochlea 5
Hair-cells Convert Mechanical Waves into Neural Activity Sensorineural Deafness and Cochlear Implants When the hair-cells are damaged or absent, amplification of sounds by a conventional acoustic hearing aid is often not effective but direct electrical stimulation of surviving auditory neurons can create a sensation of sound Schematic Design of a Cochlear Implant System Microphone Audio signal Sound Processor Radio signal Implant Electrical stimuli Skin 6
A Recent Cochlear Implant System Implant Electronics Package and 22-electrode Array Electrodes Compared with Hair-Cells Nucleus 22-electrode array 7
Main Functional Elements of a Modern Cochlear Implant microphone signal preprocessing spectral analysis selection of electrodes acoustic to electric level conversion digital data encoding radio transmission to implant radio reception from sound processor power recovery & digital data decoding current pulse generation electrode activation intracochlear electrodes From McDermott HJ: Cochlear Implants and Music, Chapter 11 in Hearing Loss in Musicians, Plural 2009 Sound Processing in Modern Cochlear Implants Band pass filters: LOW HIGH Frequency (Hz) 250 375 500 625 7400 SELECT BANDS WITH HIGHEST AMPLITUDES CONVERT ACOUSTIC TO ELECTRIC LEVELS APICAL Electrode array The stimuli are non overlapping current pulses BASAL Effect of Change in Vowel ar ee Level (db) 0 1000 2000 3000 4000 5000 Frequency (Hertz) 8
Spectral Envelopes of Different Vowels ELECTRODES: Vowel Differences Affect the Place of Electric Stimulation ee ar Level LOW Frequency HIGH APICAL Electrode place in cochlea BASAL Elements of Musical Sounds PERCEPTUAL ACOUSTIC Loudness rhythm Pitch melody harmony Timbre instrument identity Intensity or level rapid level variations Frequency fundamental frequency (F0) Level distribution across frequency spectrum 9
Musical Notes Middle-C (C4) and B3 C4 262 Hz B3 247 Hz Level (db) ELECTRODES: 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 0 1 2 3 4 Frequency (khz) Levels on Electrodes 22 and 21 with Changing Input Frequency tone frequency increasing E22 E21 filter filter Spectrogram of a Flute 8 - FREQUENCY (khz) 6-4 - 2 - Stimulation from Cochlear Implant 0 - Time BASAL ELECTRODES APICAL from McDermott HJ: Cochlear Implants and Music Chapter 11 in Hearing Loss in Musicians, Plural 2009 10
Amplitude Modulation of a Carrier Signal Spectrum of a Musical Note at Middle-C F0: 262 Hz Level (db) 0 1000 2000 3000 4000 5000 Frequency (Hertz) Amplitude Modulations in a Frequency Band Centered on 2 khz 1/F0 = 1/262 = 3.8 ms 11
Electric Current Pulses on One Electrode amplitude modulations of the current pulses convey temporal pitch information CURRENT 0 TIME Two Sources of Pitch Information for Cochlear Implant Listeners Temporal: amplitude modulation of current pulse trains typically represents fundamental frequency (f 0 ) may be present on many electrodes perception usually limited to below about 300 Hz Spectral: distribution of stimuli across electrodes can represent frequency content of signals across a wide range may not be perceived as changes in musical pitch pitch and timbre are often easily confused overall loudness can affect perception of timbre and pitch How Well Do Cochlear Implant Users Perceive the Elements of Musical Sounds? Rhythm Pitch Timbre 12
Rhythm Perception Several studies have shown that cochlear implant users can perceive musical rhythm almost as well as normally hearing listeners Speech understanding also relies partly on discrimination of rhythmic patterns How Well Do Cochlear Implant Users Perceive the Elements of Musical Sounds? Loudness Pitch Timbre Hierarchy of Perception INTERVAL JUDGEMENT RANKING I can hear those sounds DISCRIMINATION DETECTION 13
Hierarchy of Perception INTERVAL JUDGEMENT Those sounds are different RANKING DISCRIMINATION DETECTION Hierarchy of Perception The second sound is higher INTERVAL JUDGEMENT RANKING DISCRIMINATION DETECTION Hierarchy of Perception The sounds differ by 1 octave INTERVAL JUDGEMENT RANKING DISCRIMINATION DETECTION 14
Hierarchy of Perception INTERVAL JUDGEMENT RANKING DISCRIMINATION DETECTION Pitch Ranking of Sung Vowels How Well Do Cochlear Implant Users Perceive the Elements of Musical Sounds? Loudness Pitch Timbre 15
Timbre Perception Typically evaluated in terms of musical instrument recognition Results depend on many factors: number of different sounds in tests familiarity of listeners with instruments coincident variations across sounds (loudness, etc) In one study, the same subjects were tested before receiving a cochlear implant (with hearing aids) and then later when using the implant Average Unaided Hearing Thresholds Pre implant From: Looi V, McDermott H, McKay C, Hickson L: The effect of cochlear implantation on music perception by adults with usable pre operative acoustic hearing. International Journal of Audiology 47:257 268, 2008. Music Test Scores Pre-implant and Post-implant 100 90 Pre-Surgery (with HA) Post-Surgery (with CI) 80 70 60 % correct 50 40 30 20 10 0 Rhythm Pitch: One- Octave Pitch: Half- Octave Pitch: Quarter- Octave Instrument: Single Instrument: with Background Instrument: Ensemble Melody From: Looi V, McDermott H, McKay C, Hickson L: The effect of cochlear implantation on music perception by adults with usable pre operative acoustic hearing. International Journal of Audiology 47:257 268, 2008. 16
Do child cochlear implant users perceive music better than adult implant users? Comparisons are difficult, because simpler tests are often used, and no attempt is made to restrict the auditory cues that are available to listeners Pitch-Pattern Discrimination by Child Implant Users Short melody discrimination 100 80 Percent correct 60 40 chance score 50% 20 0 Normally hearing CI users Musical Instrument Identification by Child Implant Users Instrument ID 100 Percent correct 80 60 40 20 0 Normally hearing CI users 17
How Can Music Perception be Improved for Cochlear Implant Listeners? By using bilateral implants? most bilateral recipients generally prefer using two devices rather than one, but no evidence has been reported that it improves perception of musical pitch, etc. By using a specific type of sound-processing scheme? there is no evidence of large differences between commercial sound processors for music listening even for experimental sound-processing schemes designed specifically for music perception, most differences reported are small fine temporal structure? How Can Music Perception be Improved for Cochlear Implant Listeners? By improving the listening conditions music perception will be best when the listening situation is relatively free from background noise and excessive reverberation many implant users prefer to listen to familiar music, particularly if there are recognizable lyrics or visual cues, etc. By applying auditory training there is some evidence that structured training programs help cochlear implant users obtain more satisfaction from music listening but training is useful only when the listener can build on existing perceptual abilities How Can Music Perception be Improved for Cochlear Implant Listeners? By making best use of natural acoustic hearing whenever possible Bimodal hearing = any combination of cochlear implants and hearing aids (or unaided hearing) the number of implant recipients with usable acoustic hearing is large and increasing in general, the most useful hearing that cochlear implant recipients may have is at low frequencies low-frequency hearing complements the cochlear implant particularly for pitch perception sound quality is generally better when acoustic hearing is used in combination with a cochlear implant 18
Melody Recognition with Bimodal Hearing 100 80 CI CI+HA Percent Correct 60 40 20 0 1 2 3 4 Published Study 1. Sucher CM, McDermott HJ. Cochlear Implants Int 2009;10(Suppl 1):96 99 2. Dorman MF, et al. Audiol Neurootol 2008;13(2):105 112 3. Kong YY, et al. J Acoust Soc Am 2005;117(3 Pt 1):1351 1361 4. El Fata F, et al. Audiol Neurootol 2009;14(Suppl 1):14 21 Simultaneous Acoustic Stimulation Benefits Cochlear Implant Users! 100 N=10 CI alone HA alone CI + HA 10 80 8 Percent correct 60 40 6 4 20 2 0 Melody recognition Sound identification Sound quality rating 0 From McDermott HJ: Cochlear Implants and Music, Chapter 11 in Hearing Loss in Musicians, Plural 2009 Optimizing Bimodal Device Fittings for Music Listening Optimize the fitting of each device by itself if the listener s acoustic hearing is limited, then most benefit will be obtained from the cochlear implant When cochlear implants and acoustic hearing are used in combination, ensure the loudness and pitch are balanced between devices (and/or between ears) recommended procedures that will improve loudness balance have been published e.g. present wideband sounds at moderate levels and adjust gain, sensitivity, compression settings, etc. pitch matching is more challenging 19
Frequency Mapping Pitch Matching most-apical electrode(s) may be disabled APICAL HA CI - conventional Electrode array CI - matched BASAL LOW Frequency HIGH Summary Many cochlear implant recipients listen to music and enjoy it Some elements of musical sound particularly pitch are difficult for implant users to perceive accurately Child implant users generally report liking music more than adult recipients, but don t necessarily score better on objective tests Music perception can be enhanced by optimizing the listening conditions, and by application of focused auditory training Bimodal device usage (combining acoustic and electric hearing) generally improves music perception If implant recipients have no usable acoustic hearing, then future improvements in music perception will require further development of electrode arrays and sound processing techniques 20
Acknowledgments Many colleagues have contributed to this work, including: Valerie Looi, Colette McKay, Cathy Sucher, Brett Swanson Thanks also to: American Academy of Audiology, especially Cornelia Gallow & Kelly King Cochlear Ltd Financial support for some of the reported research was provided by the Garnett Passe and Rodney Williams Memorial Foundation. The Bionics Institute acknowledges the support it receives from the Victorian Government through its Operational Infrastructure Support Program. 21