Integration of Electrical and Acoustic Stimulation in Cochlear Implant Users with Residual Hearing: Synergy, Conflict, and Unilateral Neglect

Transcription

1 Integration of Electrical and Acoustic Stimulation in Cochlear Implant Users with Residual Hearing: Synergy, Conflict, and Unilateral Neglect Mario A. Svirsky Noel L. Cohen Professor of Hearing Science Vice-Chairman for Research Department of Otolaryngology New York University School of Medicine New York, NY, USA

2 Arlene C. Neuman Elad Sagi Mahan Azadpour Annette Zeman Jonathan Neukam J. Thomas Roland, Jr. Rob Froemke David Landsberger Matthew Fitzgerald Katelyn Glassman Maggie Miller Keena Seward Chin-Tuan Tan Kinneri Mehta Ksenia Aaron Yixin Fang Susan B. Waltzman William Shapiro

3 A tale of two cultures Once upon a time, all cochlear implant users were bilaterally, profoundly deaf. Cochlear implant audiologist, a species that will soon be extinct. Electroacoustic auditory input opens many new avenues to investigate the auditory system.

4 Parts of this talk 1. Speech perception in quiet in bimodal CI users a) Synergy, interference or neglect? b) Bimodal neglect and late onset auditory deprivation 2. Integration of conflicting cue vowels. 3. Music enjoyment by CI users with single-sided deafness

5

6 Demo of electroacoustic hearing Acoustic (LP at 750 Hz) Electrical (6-channel noise vocoder) Bimodal

7 DIFFERENCE (%) BIMODAL BENEFIT BIMODAL BENEFIT CNC WORDS IN QUIET YES, THESE ARE REAL

8 Definition of bimodal benefit for speech perception Bimodal benefit= Bimodal condition score Best unimodal score (either CI ear alone or HA ear alone) % of maximum bimodal benefit = Bimodal benefit (100 Best unimodal score)

9 CI-HA (%) -50 % Max BM Benefit -100

10 % Max BM Benefit cncw-ci vs BM Benefit CI alone (%)

11 % Max BM Benefit HA alone (%) -40

12 CONCLUSIONS (Part 1a) On average, CNC scores in quiet were higher when listening with electrical AND acoustic input than with either modality alone. HUGE individual differences. Significant synergy in some cases, interference in others. Normalized bimodal benefit is Higher when HA scores are higher Less likely when CI scores are <50% OR >90%. Less likely for listeners with CI scores much higher than their HA scores.

13 Parts of this talk 1. Speech perception in quiet in bimodal CI users a) Synergy, interference or neglect? b) Bimodal neglect and late onset auditory deprivation 2. Integration of conflicting cue vowels. 3. Music enjoyment by CI users with single-sided deafness

14 54

15

16

17

18 Mean Pre-op Thresholds and Mean Post-op Thresholds of Latest Post-op Frequency (Hz) Threshold Level (db) Frequency vs Pre-op unaided avg all Frequency vs Post-op unaided avg all (latest)

19 SPEECH PERCEPTION IN IMPLANTED EAR 100 CNC WORDS (% CORRECT) TIME AFTER INITIAL STIMULATION (YEARS)

20 SPEECH PERCEPTION - BIMODAL TIME AFTER INITIAL STIMULATION (YEARS) 100 CNC WORDS (% CORRECT)

21 SPEECH PERCEPTION IN UNIMPLANTED EAR 100 CNC WORDS (% CORRECT) TIME AFTER INITIAL STIMULATION (YEARS)

22

23

24 Binaurally aided ears: speech perception remains unchanged

25 It s quite possible that part of this effect may be due to malfunctioning or inappropriately fit HAs, despite our efforts to identify such cases.

26 GENERAL CONCLUSIONS The combined use of a cochlear implants and acoustic input generally results in modest benefits to speech perception in quiet. Some patients receive major bimodal benefit (synergy). A few rare cases may experience interference.

27

28 WHAT WE STILL DON T KNOW Which information processing mechanisms underlie bimodal benefit and bimodal interference? Which specific phonetic cues are used to understand speech in the CI, HA, and bimodal conditions? What are the mechanisms that underlie bimodal auditory neglect?

29 WHAT WE STILL DON T KNOW Is bimodal auditory neglect reversible? Can it be prevented? Should it be prevented?

30 IMMEDIATE FUTURE STEPS Prospective study of bimodal patients Multicenter consortium- retrospective study Bert Maat (University of Gröningen) Christophe Vincent (Université de Lille) Piotr H Skarzynski, Artur Lorens (Warsaw) Mathieu Marx (Toulouse) François Bergeron (Quebec) Eric Truy; Fabien Seldran (Lyon) Andy Beynon (Nijmegen) Elisabetta Genovese (Modena) Pietro Scimemi, Rosamaria Santarelli (Padua) Paul J Govaerts, Geert de Ceulaer (Antwerp) Richard Dowell, Michelle Moran (Melbourne) Valeria Goffi, Ana Teresa Magalhaes (São Paulo) Mario Svirsky, Arlene Neuman (New York)

31 Parts of this talk 1. Speech perception in quiet in bimodal CI users a) Synergy, interference or neglect? b) Bimodal neglect and late onset auditory deprivation 2. Integration of conflicting cue vowels. 3. Music enjoyment by CI users with single-sided deafness

32 PURPOSE 32 Goal: to study integration of speech information across ears in bimodal CI listeners who have a cochlear implant in one ear and acoustic hearing in the other ear. Hypotheses There are some bimodal CI users in whom the acoustic input will dominate. In other bimodal CI users the electric input will dominate. Some listeners will integrate the information from the two different types of input (acoustic and electrical).

33 INTRODUCTION 33 Acoustically stimulated ear (HA) Electrically stimulated ear (CI) /a/ /i/ Acoustic Responder /i/ /u/ or other Integration? /a/ Electric Responder If a bimodal listener is presented simultaneously with the vowel /i/ (e.g. heed ) in the HA ear and the vowel /a/ (e.g. odd ) in the CI ear, they will respond /i/ if they are an acoustic responder, /a/ if they are an electric responder, and /u/ (e.g. who ) or some other vowel depending on how they integrate the speech cues of the two inputs.

34 INTRODUCTION /i/ 2000 F2 (Hz) 1600 /ɚ/ /a/ 1200 /u/ /Ʊ/ F1 (Hz) For example, if the bimodal listener integrates the acoustic first formant (F1) of /i/ and the electric second formant (F2) of /a/, then the resulting location in the F1-F2 space (red filled circle) would be closer to /u/ than either /i/ or /a/ and would result in an /u/ response. Other possible responses, shown in gray, include /ɚ/ (e.g. heard ) and /Ʊ/ (e.g. hood ).

35 Subject CI Ear EXPERIMENT 1 CI SUBJECTS CI (implant/processor/strategy) Hearing Aid CI use (years) Age (years) CI1 left Nucleus CI512/CP810/ACE Phonak Certena BTE 1 70 CI2 left Nucleus Freedom/Freedom/ACE Widex Inteo BTE 2 58 CI3 right Nucleus Freedom (CA)/CP810/ACE Phonak Power Zoom CI4 left Nucleus CI24RE/Freedom/ACE Phonak Audeo Yes CI5 right Nucleus CI512/CP810/ACE Oticon Syncro CI6 left Nucleus CI512/CP810/ACE Phonak Micropower V CI7 left Nucleus CI24R(CA)/Freedom/ACE Phonak Savia ART CI8 right Nucleus CI24RE (CA)/CP810/ACE Phonak Naida V SP BTE

36 Subject CI Ear EXPERIMENT 1 CI (implant/processor/strategy) Hearing Aid CI use (years) Age (years) CI9 right AB HiRes-90k/Harmony/HiRes-S Oticon Sumo DM CI10 right AB HiRes-90k/Harmony/HiRes-S Phonak Naida III CI11 right Nucleus CI24RE/CP810/ACE Oticon Epoq BTE CI12 left Nucleus Freedom (CA)/CP810/ACE Naida V UP CI13 left Nucleus CI24R (CS)/CP810/ACE Phonak Exelia Art P BTE 8 76 CI14 right Nucleus CI24RE (CA)/CP810/ACE Naida III SP UP CI15 left Nucleus CI24R (CA)/Esprit 3G/ACE Oticon E28 P 5 89 CI16 right Nucleus CI512/CP810/ACE Widex Inteo M CI17 left CI SUBJECTS (cont.) Nucleus Freedom (CA)/ Freedom/ACE Unitron Element

37 EXP 1: CI SUBJECTS (cont.) 37 Unaided Thresholds (db HL) Aided Thresholds (db HL) subjects 250 Hz 500 Hz 1000 Hz 2000 Hz 250 Hz 500 Hz 1000 Hz 2000 Hz CI CI CI CI NR NR CI CI CI CI CI CI CI NR NR CI CI NR CI CI CI CI

38 EXPERIMENT 1 Stimuli were one of three synthetic vowels (/i/, /a/, or /u/) presented to each ear simultaneously. Vowels were 400 ms in duration with an F0 of 90 Hz. There were 9 possible stimuli: Acoustic 3 Non-Conflicting Cue stimuli 6 Conflicting Cue stimuli /i/ /a/ /u/ /i/ /a/ /u/ /i/ /a/ /u/ STIMULI Electric /i/ /a/ /u/ /a/ /i/ /a/ /u/ /u/ /i/ /i/ F2 = 2142 Hz F1 = 280 Hz /a/ F2 = 966 Hz F1 = 630 Hz /u/ F2 = 966 Hz F1 = 371 Hz 38

39 PROCEDURE 39 Acoustic stimuli presented over loudspeaker at 70 db SPL and comfort through HA verified. Electric stimuli presented through direct connection to the CI. CI volume was adjusted to a comfortably loud level. Personal HA and CI processors were used. Clinical parameter settings were not modified. All stimuli (i.e. non-conflicting and conflicting cue) were presented at least 10 times. Subjects were asked to select the vowel category that was the closest match to what they heard. Possible vowel categories included heed, hid, head, had, hawed, hud, hood, who d, and heard Eleven CI subjects were given the option to give different responses for left and right ears (see procedure for experiment 2; panel 20). However, they reported only hearing a fused sound and gave a single response for each stimulus all the time (N=9) or over 95% of the time (N=2).

40 EXP 1: RESULTS Example of Electric Responder (CI2) HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI i a i u i 10 a 10 a u 9 1 i a u u 4 6 HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI i i 10 a u i a 10 a u i u 9 1 a 4 6 u Output with data arranged while CI stimuli are fixed Notice how responses to different HA stimuli are nearly the same when the CI stimulus is fixed. This subject is a predominately electric responder Same output as matrix above, but with data arranged while HA stimuli are fixed 40

41 EXP 1: RESULTS Example of Acoustic Responder (CI7) HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI i a 1 8 i u 5 5 i a 10 a u 6 4 i a 1 9 u u 5 5 HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI i i a u 1 8 i a 10 a 1 9 u 5 5 i u 6 4 a 5 5 u Output with data arranged while CI stimuli are fixed Notice how responses to different CI stimuli are nearly the same when the HA stimulus is fixed. This subject is an acoustic responder Same output as matrix above, but with data arranged while HA stimuli are fixed 41

42 EXP 1: RESULTS Example of Integration (CI15) HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI i 17 1 a i u i a 16 a u i a u u HA HEED HID HEAD HAD HOD HUD HOOD WHO'D HEARD CI 17 1 i i a u i a 16 a u i u a u Output with data arranged while CI stimuli are fixed When presented with /i/ to the HA and /a/ to the CI, subject responds /u/ (i.e. hood, whod, or heard) 61% of the time. Notice how responses do not consistently follow the CI or HA. This subject is integrating the two kinds of input. Same output as matrix above, but with data arranged while HA stimuli are fixed 42

43 EXP 1: RESULTS 43 HA /i/ + CI /a/ = /u/ CNC Word Scores (% correct) Subject Type of Responses (% responses) HA CI HACI CI1 Predominantly Electric 0% 82% 76% 96% CI2 Predominantly Electric 0% 4% 70% 80% CI3 Predominantly Electric 47% 4% 64% 74% CI4 Acoustic 0% 14% 90% 84% CI5 Acoustic 0% 18% 50% 52% CI6 Acoustic 0% 18% 76% 76% CI7 Acoustic 0% 78% 78% 90% CI8 Predominantly Acoustic 20% 23% 20% 36% CI9 Integration (more electric) 0% 66% 38% 66% CI10 Integration (more electric) 0% 36% 38% 50% CI11 Integration 0% 14% 92% 88% CI12 Integration (more electric) 16% 36% 82% 84% CI13 Integration 27% 8% 82% 78% CI14 Integration 27% 28% 84% 82% CI15 Integration 61% 2% 50% 62% CI16 Integration 73% 58% 80% 86% CI17 Integration (more electric) 78% 2% 76% 70%

44 COMPUTATIONAL MODEL In the multidimensional phoneme identification (MPI) model, vowel stimuli are represented as points in a multidimensional perceptual space. The MPI model has 2 components: Internal noise model Decision model a percept is generated by adding noise to each perceptual dimension, proportional to patients JND for that dimension (internal noise model). A response is selected by finding the vowel location in the multidimensional space that is closest to the percept. Thousands of iterations of the above process are done to construct response distributions for each stimulus.

45 Perceptual dimensions are the first two formants F1 and F2 for the acoustic vowel /i/ and F1 and F2 for the electric vowel /a/. The resulting 4-dimensional percept is compared with the locations of all possible vowels, and the closest is selected as the response. Acoustic F1 MPI MODEL INTERNAL NOISE MODEL JND F1 Acoustic F1 Estimate DECISION MODEL /i/ Acoustic F2 JND F1 Acoustic F2 Estimate Response Centers Percept Compare Electric F1 JND F1 Electric F1 Estimate RESPONSE /a/ Electric F2 JND F1 Electric F2 Estimate

46 COMPUTATIONAL MODEL JND F1F2 Electric (mm) AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAA AAAAAAAAAAAA AAAAA A A A EEE E E AAAAAAAA AA AAAA AA A EEEEEEEEEEEEEEEEE EEEEEEEEEEEEEEEEEEEEEEEE EEEEEEEEEEEEEEEEEEEEEEEEEEEE EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE JND F1 Acoustic (mm) EEEEEEEEEEEE A = predominately acoustic response (i.e. HEED or HID) E = predominately electric response (i.e. HAWED or HUD) When JND for the electric input are high (i.e. poor CI performance), subjects responses are predominately acoustic. When JND for the acoustic input are high (i.e. poor HA performance), subjects responses are predominately electric. There s a sweet spot of acoustic and electric JNDs that give rise to integration of inputs; i.e. /i/ acoustic + /a/ electric = /u/.

47 COMPUTATIONAL MODEL JND F2 Acoustic = 4 mm (fixed) JND Electric: The size of the effect can also depend on how much residual hearing is available at higher frequencies (i.e. JND for F2 acoustic).

48 CONCLUSIONS 48 Approximately half of subjects were electric or acoustic responders. Approximately half of subjects showed an interesting pattern: most responses to the stimulus composed of an acoustic /i/ and an electrical /a/ were actually /u/, /Ʊ/, or /ɚ/ (blue shaded cells in matrices). This pattern is consistent with a perceptual mechanism that combines F1 information from the acoustic signal with F2 information from the electrical signal into a single perceptual space. This effect is reminiscent of the McGurk phenomenon in audiovisual integration. Future steps: Computational modeling of vowel confusion patterns. Possible application of this knowledge to the HA and CI fitting of bimodal patients.

49 Parts of this talk 1. Speech perception in quiet in bimodal CI users a) Synergy, interference or neglect? b) Bimodal neglect and late onset auditory deprivation 2. Integration of conflicting cue vowels. 3. Music enjoyment by CI users with singlesided deafness

50 Music enjoyment in SSD patients: the synergistic effect of electric and acoustic stimulation David M. Landsberger 1, Katrien Vermeire 2, Natalia Stupak 1, Annette M. Zeman 1,Paul Van de Heyning 2, Mario A. Svirsky 1 1New York University School of Medicine, New York, NY USA 2University Hospital Antwerp, Antwerp, Belgium

51 Music perception by CI users: not very good Music perception is much worse than speech perception for CI users. Many CI users describe their perception of music as meaningless noise. Previous studies have quantified many secondary aspects of music perception, but not music enjoyment. This is hard to do (rating scale of 1-100?) without an absolute reference.

52 Single-sided deafness (SSD) CI users SSD CI users, who have normal contralateral hearing present a unique opportunity to assess music enjoyment quantitatively in CI users SSD subjects allow comparing sound quality of electrical or electroacoustic stimuli relative to a normal ear. Allows comparing across subjects (as all of them have the same normal hearing fixed reference point)

53 Methods Used MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) test. Patients were asked to rate how much you enjoy listening to music in the given configuration relative to a fixed reference. Reference = Unprocessed musical piece presented only to the acoustic ear. Rating=100. Anchor = Same musical piece processed with 6 channel noise vocoder with a large mismatch between analysis filters and noise bands (6.5 mm of cochlear distance). Also presented only to the acoustic ear. Rating=0. Repeated with two songs: Ring of Fire Rhapsody in Blue

54 Methods (Stimuli): Acoustic Alone Standard Electric Alone (100) Electric and Ski-slope Acoustic Hearing Loss Bimodal Hearing Bimodal Noisy overlapping Anchor Freq (i.e. Allocation 0 ) To Acoustic Ear Acoustic Electric Description Unprocessed Unprocessed Acoustic (High Quality Anchor) Electric Unprocessed Unprocessed Acoustic + Electric Low 250Hz Low 500Hz Low 1000Hz Low Frequency Acoustic Hearing Only Low Frequency Acoustic Hearing Only Low Frequency Acoustic Hearing Only Low 250Hz Unprocessed Bimodal with varying Acoustic Hearing Low 500Hz Unprocessed Bimodal with varying Acoustic Hearing Low 1000Hz Unprocessed Bimodal with varying Acoustic Hearing Low 250Hz High 250Hz Bimodal with non-overlapping frequencies Low 500Hz High 500Hz Bimodal with non-overlapping frequencies Low 1000Hz High 1000Hz Bimodal with non-overlapping frequencies 6 Channel Noise Vocoder with 6.5 mm shift Low Quality Acoustic Anchor

55 User interface of the test

56 Subjects / Testing All subjects had normal / near-to-normal hearing in one ear and a cochlear implant in the contralateral ear. All had used their CIs for at least 1 year Acoustic sounds were presented over headphones Electric sounds were presented via direct connection. 6 MED-EL FLEXSOFT users 2 MED-EL M (24 mm) users 3 Cochlear Contour Advance Users

57 Results Trim Mean of 11 Subjects Ring of Fire Rhapsody in Blue p(a <> A+E) = p(a <> A+E) = Electric Only Acoustic < X Acoustic < X + Electric Acoustic < X + Electric > X MUSHRA Score Hz 500 Hz 1000 Hz Unprocessed 250 Hz 500 Hz 1000 Hz Unprocessed 0 Low Pass Filter Frequency

58 Results Trim Mean of 12 Subjects MUSHRA Score Ring of Fire Enjoyment of Electric Only is similar to enjoyment of acoustic low pass filtered at 250Hz Rhapsody in Blue Electric Only Acoustic Acoustic + Electric Acoustic + High Pass Filtered Electric MUSHRA Score Hz 500 Hz 1000 Hz Unprocessed 250 Hz 500 Hz 1000 Hz Unprocessed 0 Low Pass Filter Frequency

59 Results 200 Trim Mean of 12 Subjects Ring of Fire Rhapsody in Blue Combining Electric with Acoustic Sounds Better than Either Alone (Green Bars are bigger than Black Or Red) Electric Only Acoustic Acoustic + Electric Acoustic + High Pass Filtered Electric MUSHRA Score MUSHRA Score Hz 500 Hz 1000 Hz Unprocessed 250 Hz 500 Hz 1000 Hz Unprocessed 0 Low Pass Filter Frequency

60 Electroacoustic enhancement vs electric-only Ring Of Fire Rhapsody in Blue Electric + Acoustic Improvement (re: Acoustic Only) M15 M16 M11 M14 N1 M13 M12 N3 M7 M3 N Electric + Acoustic Improvement (re: Acoustic Only) M15 M14 M16 N3 M12 M3 N1 M7 N2 M11 M Electric Quality Electric Quality

61 Conclusions As we suspected, music enjoyment from electric stimulation is very poor relative to normal acoustic stimulation For listeners with unilateral acoustic hearing, providing contralateral electric stimulation significantly improves music enjoyment. Benefits of adding electrical stimulation are observed regardless of degree of (simulated) acoustic hearing loss.

62 ***Some final thoughts 1) Speech from a cochlear implant and a hearing aid can be successfully integrated. 2) There are (rare) cases of interference. 3) Some successful CI users may end up neglecting contralateral acoustic input. 4) The auditory system can successfully adapt to frequency-place mismatch- at least to some extent. 5) Music from a CI: not very enjoyable. But CI+acoustic can result in enhancement 6) What is the best way to manage patients with electrical and acoustic input?

63 ACKNOWLEDGMENTS These studies are supported by NIH/NIDCD R01-DC011329, PIs: Svirsky and Neuman R01-DC003937, PI: Svirsky R01-DC012152, PI: Landsberger Other projects in our laboratory and in our department have received support from Siemens, Sun Microsystems, Cochlear Americas, Advanced Bionics, and Med-El.

64 These look reasonable The following patients showed stable speech perception scores in the HA ear after implantation, as expected

65

66

67

68

69

70 These are a bit strange The following patients showed decreasing speech perception scores in the HA ear after implantation.

71

72

73

74