ABSTRACT. Møller A (ed): Cochlear and Brainstem Implants. Adv Otorhinolaryngol. Basel, Karger, 2006, vol 64, pp

Transcription

1 Speech processing in vocoder-centric cochlear implants Philipos C. Department of Electrical Engineering Jonsson School of Engineering and Computer Science University of Texas at Dallas P.O.Box Richardson, TX ABSTRACT The principles of most recent cochlear implant processor are similar to that of the channel vocoder, originally used for transmitting speech over telephone lines with much less bandwidth than that required for transmitting the unprocessed speech signal. An overview of the various vocoder-centric processing strategies proposed for cochlear implants since the late 1990s is provided including the strategies used in different commercially available implant processors. Special emphasis is placed on reviewing the strategies designed to enhance pitch information for potentially better music perception. The various noise suppression strategies proposed over the years based on multi-microphone and single-microphone inputs are also described.

2 1. Introduction This Chapter presents an overview of the various vocoder-centric processing strategies proposed for cochlear implants since the late 1990s (a review of earlier strategies can be found in [1]). This Chapter also offers a review of the strategies used in different commercially available implant processors HISTORICAL BACKGROUND In 1939, at the World s Fair in New York City, people watched with intense curiosity the first talking machine. The machine spoke with the help of a human operator seating in front of a console, similar to a piano keyboard, consisting of 10 keys, a pedal and a wrist bar. Inside the machine were analog circuits of bandpass filters, switches and amplifiers connected to a loudspeaker. The talking machine contained the first artificial speechsynthesis system implemented in hardware. This speech synthesis system, pioneered by Homer Dudley from Bell Laboratories, came to be known as the channel vocoder (voice coder) [2]. Dudley s vocoder idea had a profound impact not only on telephony and speech transmission applications [3,4], but also much later in the development of cochlear implant processors. The latest and most successful signal-processing strategies used in cochlear implants are based on vocoding analysis principles. All cochlear implant devices today are programmed (now digitally) with a modified version of the vocoder analysis algorithm. 2. The channel vocoder The channel vocoder [2,4] consists of a speech analyzer and a speech synthesizer (see Figures 1 and 2). In speech transmission applications, the analyzer would be utilized at the transmitter and the synthesizer at the receiver end. The incoming signal is first filtered into a number of contiguous frequency channels using a bank of band-pass filters (10 filters were used in Dudley s1939 demonstration). The envelope of the signal in each channel is estimated by full-wave rectification and low-pass filtering and is then downsampled and quantized for transmission. In addition to envelope estimation, the vocoder analyzer makes a voiced/unvoiced decision and estimates the vocal pitch (F0) of the signal. These two pieces of information are transmitted alongside the envelope information. The synthesizer modulates the received envelopes by the appropriate excitation as determined by the voiced/unvoiced (binary) signal. The excitation signal consists of random noise for unvoiced speech segments and a periodic pulse generator for voiced speech, with the period of the pulse generator being controlled by F0. The modulated signals are subsequently bandpass-filtered by the same filters and then added together to produce the synthesized speech waveform. Current cochlear implant processors (sixty years later) utilize the same blocks of the channel vocoder analyzer shown in Figure 1. At present, only the vocoder analyzer is used for transmitting envelope information to the

3 individual electrodes, but recently there has been a shift in research focus toward implementing blocks of the synthesizer as well [5,6]. Interestingly, early devices based on feature extraction strategies modulated the estimated formant amplitudes by F0 [1]. These strategies, however, were abandoned due to the inherent difficulties associated with F0 extraction in noisy environments. It is also interesting to note that the acoustic cochlear implant simulations often used to study performance of cochlear implant patients in the absence of confounding factors (e.g., duration of deafness, insertion depth) utilize the synthesizer (Fig. 2). By choosing random noise as the excitation signals for all segments of speech, we get the noise-band cochlear implant simulations [7]. Similarly, by choosing sine waves with frequencies set to the center frequencies of the bandpass filters as the excitation signals, we get the sine wave simulations [8]. 3. Vocoder-centric strategies for cochlear implants There are currently two variations of the channel vocoder (Fig. 1) that are used in all implant processors. The first implementation uses the analyzer of the channel vocoder in its original form (as per Fig. 1). The second implementation also uses the analyzer of the channel vocoder, but selects only a subset of the envelope outputs for stimulation. This section describes in detail these two variations. 3.1 Continuous Interleaved Sampling (CIS) strategy The first cochlear implant device to adopt a channel-vocoder strategy was the Ineraid device manufactured by Symbion, Inc., Utah. The signal was first compressed using an automatic gain control, and then filtered into four contiguous frequency bands, with center frequencies at 0.5, 1, 2, and 3.4 khz [9]. The filtered waveforms went through adjustable gain controls and then sent directly through a percutaneous connection to four intracochlear electrodes. The filtered waveforms were delivered simultaneously to four electrodes in analog form. A major concern associated with simultaneous stimulation is the interaction between channels caused by the summation of electrical fields from individual electrodes. Neural responses to stimuli from one electrode may be significantly distorted by stimuli from other electrodes. These interactions may distort speech spectral information and therefore degrade speech understanding. A simple solution to the channel interaction problem was proposed by researchers at the Research Triangle Institute (RTI) via the use of nonsimultaneous, interleaved pulses [10]. They proposed modulating the filtered waveforms by trains of biphasic pulses that were delivered to the electrodes in a non-overlapping (non-simultaneous) fashion, that is, in a way such that only one electrode was stimulated at a time (Figure 3). The amplitudes of the pulses were derived by extracting the envelopes of the band-passed waveforms. The resulting strategy was called the Continuous Interleaved Sampling (CIS) strategy.

4 The block diagram of the CIS strategy is shown in Figure 3. The signal is first pre-emphasized and then applied to a bank of bandpass filters. The envelopes of the outputs of these bandpass filters are then full-wave rectified and low-pass filtered (typically with 200 or 400 Hz cutoff frequency). The envelopes of the outputs of the bandpass filters are finally compressed and used to modulate biphasic pulses. A non-linear compression function (e.g., logarithmic) is used to ensure that the envelope outputs fit the patient's dynamic range of electrically evoked hearing. Trains of balanced biphasic pulses, with amplitudes proportional to the envelopes, are delivered to the electrodes at a constant rate in a non-overlapping fashion. Figure 3 shows the basic configuration for the CIS strategy. Many variations of the CIS strategy have emerged and are currently used by the three implant manufacturers. Some devices for instance use the fast Fourier transform (FFT) for spectral analysis and some use the Hilbert transform to extract the envelope instead of full-wave rectification and low-pass filtering. Although the CIS strategy is employed by all three manufacturers, it is based on different implementations CIS design parameters The CIS strategy can be configured in a number of ways by varying design parameters (e.g., filter spacing, envelope cut-off frequencies, etc) of the vocoder. These parameters include, among other things, the envelope detection method, stimulation rate (i.e., the number of pulses delivered to the electrodes per second), shape of compression function and filter spacing. A subset of these parameters may be varied to optimize speech recognition performance for each patient Stimulation rate The pulse rate (the number of pulses per sec (pps) delivered to each electrode) may be as low as 250 pulses/sec or as high as 5000 pulses/sec in some devices. It would be reasonable to expect that better recognition performance should be obtained with high pulse-rates, since high pulse-rate stimulation can better represent fine temporal modulations. This is illustrated in Figure 4, which shows the pulsatile waveforms of the syllable /t/ obtained at different rates. As shown in Fig. 4, the unvoiced stop consonant /t/ is marked by a period of silence (closure) followed by a burst and aspiration. As the pulse rate increases, the burst becomes more distinctive, and perhaps more salient perceptually. There seems to be no evidence of the burst at low rates, 200 or 400 pulses/s. This example clearly demonstrates that lower rates do not provide a good, if any at all, temporal representation of the burst in stop consonants. Despite the theoretical advantages of higher stimulation rates, the outcomes from several studies have not been consistent. While the majority of those studies [11-15] found a positive effect of high stimulation rates a few studies [16,17] found no significant effect. It is, however, consistent across these studies that some patients received large benefits with high stimulation

5 rates while other patients received, little, or no benefit. Possible reasons for the discrepancies in the outcomes between the various studies include: (a) differences in implementation of the CIS strategy, (b) differences in speech materials used, and (c) differences in electrode design between devices. Each manufacturer has its own implementation of the CIS strategy. In the Nucleus device for instance, the FFT is used for spectral analysis in lieu of the bank of bandpass filters. Limited by the FFT analysis frame rate, extremely high stimulation rates can be obtained by repeating stimulus frames. Therefore, higher stimulation rates might not necessarily introduce new information, which explains the lack of improvement with high stimulation rates [16]. The influence of speech materials when examining the effect of parametric variations of the CIS strategy was demonstrated in the study by et al. [12] which assessed speech recognition as a function of stimulation rate in six Med-El/CIS-Link cochlear implant (CI) listeners. Results showed that higher stimulation rates >2100 pulses/sec produced a significantly higher performance on word and consonant recognition than lower stimulation rates (800 pulses/sec). The effect of stimulation rate on consonant recognition was highly dependent on the vowel context. The largest benefit was noted for consonants in the /ici/ and /ucu/ contexts, while the smallest benefit was noted for consonants in the /aca/ context. This finding suggests that the /aca/ consonant test, which is widely used, is not sensitive enough to parametric variations of implant processors. The advantages of high stimulation rates are unfortunately offset by the increased channel interaction associated with extremely high stimulation rates. Since each manufacturer uses different number of electrode contacts with different electrode spacing (see Table 1), it is reasonable to assume that a wider spacing between electrodes will yield smaller amounts of channel interaction. Consequently, the electrode spacing confounds the effect of high stimulation rates on speech recognition when comparing different devices. The Nucleus device has the smallest electrode spacing (0.7mm) while the Med-El device has the widest electrode spacing (2.4mm) (The Ineraid device has in fact the widest spacing (4mm), but is not commercially available). It is therefore not surprising that most of the benefits reported with high stimulation rates were with Med-El users and not with Nucleus users. Significant benefits were reported in [15,18] with Nucleus users, but with those users fitted with a spectral-maxima strategy running at high stimulation rates. As mentioned above, some patients do receive significant benefit with the use of high stimulation rates. The "optimal" pulse rate, however, as far as speech recognition performance is concerned, varies from patient to patient. Wilson et al. [11], for instance, reported that some patients obtain a maximum performance on the 16-consonant recognition task with a pulse rate of 833 pulses/sec and pulse duration of 33 µsec/phase. Other patients obtain small but significant increases in performance as the pulse rate increases from 833 pps to 1365 pps, and from 1365 pps to 2525 pps, using 33 µsec/phase pulses. Unfortunately, there are no known methods for identifying

6 the optimal pulse rate for each patient, other than trying out different values and examining their performance. Current commercial implant processors are operating at stimulation rates ranging from 800 pulses/sec/channel to 2500 pulses/sec/channel, depending on the device. Use of very high-rates (>5000 pulses/sec) is being investigated by some as a means of restoring the stochastic independence of neural responses, which is lost with the overly synchronized electrical stimulation. In acoustic hearing, it is known that the nature of the neuron responses is stochastic in that the firing of a particular auditory-nerve fiber has no effect on the probability of a neighboring fiber firing. In electric stimulation, however, the response of single neurons is highly synchronized and also entrained with the stimulus, in that neurons fire on every stimulation cycle, up to rates of 800 Hz [19,20]. The stochastic nature (i.e., the independence) of the neural responses is lost with electrical stimulation since all the neurons in a local region fire at the same time (i.e., in synchrony). To restore the stochastic independence of neuron responses, Rubinstein et al. [21] proposed the use of high-frequency (5000 pulses/sec) desynchronizing pulse trains over the stimulus delivered by the processor. Litvak et al. [22] demonstrated that the use of desynchronizing pulse trains can improve the representation of both sinusoidal and complex stimuli (synthetic vowels) in the temporal discharge patterns of auditory nerve fibers for frequencies up to 1000 Hz. The addition of un-modulated high-rate pulse trains over the electrical stimulus can also result in significant increases in psychophysical dynamic range [23]. Another method proposed for restoring the stochastic independence of neural responses is the addition of appropriate amount of noise to the acoustic stimuli [24,25] Compression function The compression of envelope amplitudes is an essential component of the CIS processor because it transforms acoustical amplitudes into electrical amplitudes. This transformation is necessary because the range in acoustic amplitudes in conversational speech is considerably larger than the implant patient's dynamic range. Dynamic range is defined here as the range in electrical amplitudes between threshold (barely audible level) and loudness uncomfortable level (extremely loud). In conversational speech, the acoustic amplitudes may vary within a range of db [26,27]. Implant listeners, however, may have a dynamic range as small as 5 db. For that reason, the CIS processor compresses, using a non-linear compression function, the acoustic amplitudes to fit the patient's electrical dynamic range. The logarithmic function is commonly used for compression because it matches the loudness between acoustic and electrical amplitudes [28,29]. It has been shown that the loudness of an electrical stimulus in microamps is analogous to the loudness of an acoustic stimulus in db. Logarithmic compression functions of the form Y = A log(1+c ) + B are typically used, where is the acoustic amplitude (output of envelope detector), A, B and C are constants, and Y is the (compressed) electrical

7 amplitude. Other types of compression functions used are the power-law functions of the form: p y = Ax + B (1.1) where p<1. The advantage of using power-law functions is that the shape, and particularly the steepness of the compression function, can be easily controlled by simply varying the value of the exponent p. The constants A and B are chosen such that the input acoustic range is mapped to the electrical dynamic range [THR, MCL], where THR is the threshold level and MCL is the most comfortable level measured in µamps [1]. The input acoustic range, also known as input dynamic range (IDR), is adjustable in some devices and can range from db. The effect of IDR on speech recognition was examined in several studies (e.g., [27,30]). The effect of the shape of the compression function on speech recognition has been investigated in a number of studies [12,31-34]. et al. [12] modified the shape of the amplitude mapping functions ranging from strongly compressive to weakly compressive by varying the power exponent in Eq. (1.1) from p=-0.1 (too compressive) to p=0.5 (nearly linear). Results indicated that the shape of the compression function had only a minor effect on performance, with the lowest performance obtained for nearly linear mapping functions Envelope detection Two different methods can be used to extract the envelopes of filtered waveforms. The first method includes rectification (full-wave or half-wave) followed by low-pass filtering at Hz. The second method, currently used by the Med-El device, uses the Hilbert transform. No clear advantage has been demonstrated for the use of one method over the other for envelope extraction. The first method is simple to implement as it involves full-wave or half-wave rectification and low-pass filtering. The low-pass filter is a smoothing filter and also serves as an antialiasing filter, which is required prior to donwsampling (Figure 1) the filtered waveforms. The stimulation rate needs to be at least two times higher (Nyquist rate) than the cutofffrequency of the low-pass filter. Psychophysics studies [35] suggest that it should be at least four times the envelope cutoff frequency. Pitch increased with sinusoidally amplitude-modulated pulse trains up to a modulation frequency of about Hz, provided the carrier rate (stimulation rate) was at least four times the modulation frequency [35]. Similar findings were also reported in intracochlear evoked potential studies [36]. The cut-off frequency of the low-pass filter controls the modulation depth of the envelopes. The lower the cutoff frequency is, the smaller the modulation depth of the filtered waveform (see examples in Figure 7), i.e., the flatter the envelopes are. Simulation studies [7,37] demonstrated no significant effect of the envelope cutoff frequency on speech recognition by normal-hearing listeners. This was also confirmed with studies from our lab with cochlear implant patients (see Figure 5) tested on consonant and melody

8 recognition tasks [38]. No significant effect of envelope cutoff frequency on consonant and melody recognition was found. The second envelope detection method is based on the Hilbert transform [39], a mathematical tool which can represent a time waveform as a product of slowly-varying envelope and a carrier signal, containing fine structure information (see example in Fig. 6). More specifically, the filtered waveform, xi() t, in the ith band (channel) can be represented as xi() t = ai()cos t φi() t (1.2) where ai() t represents the envelope of the ith band at time t, and cos ( φ i( t) ) represents the fine-structure waveform of the ith band. Note that φ i() t is called the instantaneous phase of the signal, and the derivative of φ i() t produces the instantaneous frequency (carrier frequency) of the signal, which varies over time. The fine-structure waveform is a frequency modulated (FM) signal (Fig. 6) since the carrier frequency is not fixed but varies with time. Figure 6 shows an example of the decomposition of the time-domain waveform of the vowel /a/ into its envelope and fine-structure. It is clear from Fig. 6 that the Hilbert envelope contains periodicity information and therefore is not the same as the envelope defined by Rosen [40]. The Hilbert transform renders Rosen s [40] three-way partition of the temporal structure of speech into a two-way partition: the envelope, which also contains periodicity information, and the fine structure. This envelope/fine-structure decomposition of the signal (Fig. 6) can be done independently for each channel. Figure 7 shows examples of envelopes extracted using the above two methods: the Hilbert transform and rectification followed by low-pass filtering. Of the two methods, the Hilbert transform produces more accurate estimates of the envelope. Use of higher envelope cutoff frequencies, however, yields envelopes close to those extracted by the Hilbert transform (Fig. 7). Current implant devices transmit envelope information ( ai() t ) and discard fine-structure information ( cos ( φ i( t) )) as they implement only the analysis part of the vocoder and not the synthesis part (compare Figure 1 with Figure 3). Simulation studies [41-43] with normal-hearing listeners demonstrated the potential of including limited amounts of fine-structure information. It is not yet clear, however, how to incorporate fine-structure information in cochlear implants in a way that they can perceive it [44] Filter spacing For a given signal bandwidth (e.g., 0-8 khz), there exist several ways of allocating the filters in the frequency domain. Some devices use a logarithmic spacing while others use a linear spacing in the low-frequencies (< 1300 Hz) and logarithmic spacing thereafter (> 1300 Hz). The effect of filter spacing on speech recognition, melody recognition and pitch perception has been investigated in a number of studies [45-48].

9 Fourakis et al. [47] advocated the placement of more filters in the F1/F2 region for better representation of the first two formants. They investigated the effect of filter spacing by modifying the electrode frequency boundary assignments of Nucleus 24 patients so as to include additional filters in the F1/F2 region. Small but significant improvements were noted on vowel recognition with an experimental MAP which included one additional electrode in the F2 region. No significant improvements were found on word recognition. The fixed number of frequency tables provided by the manufacturer, limited the investigators from assigning more electrodes in the F2/F3 region. The majority of the Nucleus-24 CI users tested preferred the experimental MAP over their everyday processor. Similar findings were also found in our lab using newly implanted Clarion CII patients fitted with 16-channels of stimulation. The effect of three different filter spacings, which included log, mel [49] and critical-band [50] spacing, was investigated on recognition of 11 vowels in /hvd/ format. Results (see Figure 8) indicated that some subjects obtained a significant benefit with the critical-band spacing over the log spacing. Performance obtained with the mel frequency spacing was the lowest compared to the other two frequency spacing. This may be attributed to the number of frequency bands allotted in the F1 and F2 range. The mel-frequency spacing had the smallest number (4) of bands allocated in the 0-1 khz range, which is the F1 range for most vowels. In contrast, both the critical-band and the log spacing had 6 bands in the F1 range. In addition, the critical-band spacing had 7 bands in the 1-3 khz range (F2 range), while the log spacing had 6. The effect of filter spacing on pitch perception has been investigated in [51,52], and will be discussed later (see Section 4.1). In brief, existing data support the idea that the number of filters allocated in the F1/F2 region can have a significant effect on performance, at least on vowel recognition tasks Spectral-maxima strategy The spectral-maxima strategy implemented as the ACE (previously SPEAK) strategy on Cochlear Corporation devices [53]) and as the n-of-m strategy in other devices [54,55], has antecedents in the channel-picking vocoders of the 1950s [56] as well as Haskins Laboratories Pattern Playback speech synthesizer [57]. The principle underlying the use of this strategy is that speech can be well understood when only the peaks in the short-term spectrum are transmitted. In the case of the Pattern Playback, only 4 6 of 50 harmonics needed to be transmitted to achieve highly intelligible speech as long as the picked harmonics defined the first two or three formants in the speech signal. The spectral-maxima strategy is similar to the CIS strategy with the main difference being that the number of electrodes stimulated is smaller than the total number of analysis channels. In this strategy, the signal is processed through m bandpass filters from which only a subset n (n<m) of the envelope amplitudes are selected for stimulation. More specifically, the n maximum

10 envelope amplitudes are selected for stimulation. The spectral-maxima strategy is sometimes called the n-of-m strategy or peak-picking strategy and is available in both Med-El and Nucleus-24 devices. In the Nucleus-24 device, out of a total of 20 envelope amplitudes, maximum amplitudes are selected for stimulation in each cycle. The ACE (and SPEAK) strategy continuously estimates the outputs of the 20 filters and selects the ones with the largest amplitude. In the SPEAK strategy, the number of maxima selected varies from 5 to 10 depending on the spectral composition of the input signal, with an average number of six maxima. For broadband spectra, more maxima are selected and the stimulation rate is slowed down. For spectra with limited spectral content, fewer maxima are selected and the stimulation rate increases to provide more temporal information. Several studies compared the performance of spectral-maxima and CIS strategies [15,18,58,59]. Cochlear implant simulation studies by Dorman et al. [59] indicated high performance with the spectral-maxima strategy even when a small number of maxima were selected in each cycle. A 3-of-20 processor (i.e., a processor that selected three maximum amplitudes out of 20 amplitudes in each cycle) achieved a 90% correct level of speech understanding for all stimulus material (sentences, vowels and consonants) presented in quiet. In contrast, it required 4, 6, and 8 channels of stimulation by CIS-type processors to achieve similar levels of performance for sentences, consonants, and vowels respectively. Hence, provided that there exist a large number of output analysis filters, only a small number of maxima need to be selected, an outcome consistent with the Pattern Playback studies. In noise (0 db S/N), a minimum of 10 maxima needed to be selected for asymptotic performance on sentence recognition. A study by Skinner et al. [15] compared the performance of Nucleus- 24 implant patients fitted with the SPEAK, ACE and CIS strategies, after the patients used each strategy for a period of 4-6 weeks. Results indicated that the group mean score obtained with the ACE strategy on sentence recognition was significantly higher than the scores obtained with the SPEAK and CIS strategies. The SPEAK and ACE strategies are both spectral-maxima strategies selecting roughly the same number of envelope maxima (8-12) out of a total of 20 envelope outputs. The two strategies differ, however, in the stimulation rate. ACE s stimulation rate is significantly higher than SPEAK s and ranges from pps while SPEAK s rate is fixed at 250 pps. The higher scores obtained with ACE can therefore be attributed to its higher stimulation rate. 4. Speech coding strategies used in commercial devices There are currently three cochlear implant processors in the United States approved by the Food and Drug Administration (FDA): the Nucleus 24, the Clarion and the Med-El processor. This section provides an overview of the signal processing strategies used in commercially available implant processors.

11 4.1 Advanced Bionics Corporation (Clarion CII/Auria device) The Advanced Bionics Corporation s (ABC s) implant has undergone a number of changes in the past decade. ABC s first generation implant (Clarion S-Series) included an electrode array with 8 contacts and supported a number of stimulation strategies including a simultaneous (analog-type) stimulation strategy (see review in [60]). ABC s second generation device (termed Clarion CII) includes a 16-contact electrode array (HiFocus II) and supports simultaneous, partially simultaneous and non-simultaneous stimulation strategies. Temporal bone studies have shown that the placement of the implanted Clarion s HiFocus II electrode array is extremely close to the modiolar wall [61]. The Clarion CII device supports a high-rate CIS strategy, which can be delivered either non-simultaneously or partially simultaneously to 16 electrode contacts. Clarion s CIS strategy, called HiRes, differs from the traditional CIS strategy in the way it estimates the envelope. It uses half-wave rectification rather than full-wave rectification, and it does not use a low-pass filter. Instead, after the half-wave rectification operation, it averages the rectified amplitudes within each stimulation cycle. This averaging operation is in effect a low-pass filtering operation. The cutoff frequency of the lowpass filter depends on the number of samples to be averaged, i.e., it depends on the stimulation rate. The higher the stimulation rate is (i.e., the smaller the number of samples to average), the higher the cutoff frequency is. In the HiRes strategy, the signal is first pre-emphasized and then bandpass filtered into 16 channels. The bandpass filters span the frequency range of 250 to 8000 Hz and are logarithmically spaced. The filtered waveforms are half-wave rectified, averaged and logarithmically compressed to the patients electrical dynamic range. The compressed envelopes are transmitted via RF to the implant decoder, where are then modulated by trains of biphasic pulses for electrical stimulation. Comparisons between the conventional CIS strategy and the HiRes strategy were reported in [62,63]. The CII device utilizes a dual-action automatic gain control at the microphone input consisting of a slow-acting and fast acting stage. The slowacting control has a compression threshold of 57 db SPL with an attack time of 325 ms and a release time of 1000 ms. The second control is fast-acting and has a higher compression threshold of 65 db SPL with an attack time of <0.6 ms and a release time of 8 ms. The Clarion II device has 16 independent current sources that allow for simultaneous stimulation of two or more electrode contacts. When used in non-simultaneous mode of stimulation, HiRes operates at a stimulation rate of 2800 pps/sec using a pulse width of 11 µsces/phase. The stimulation rate can be further increased by the use of partially simultaneous stimulation whereby pairs of electrodes are stimulated simultaneously. To minimize potential channel interaction, non-adjacent pairs of electrodes are typically selected (e.g., 1-8, 2-7, etc.). For 16 electrodes configured with paired pulses and a narrow pulse width, the stimulation rate can exceed 5000 pps per channel. The combination of high rate stimulation and high cutoff frequency

12 in the envelope detectors provides a fine temporal waveform representation of the signal at each channel. Some patients are able to utilize the fine temporal modulations present in the waveform at such high stimulation rates [12,62,63]. The presence of multiple current sources allows for the implementation of virtual channel processing strategies, currently under investigation by ABC. By properly manipulating (or steering) the current delivered simultaneously to adjacent electrodes, it is possible to elicit pitches intermediate to the pitches elicited by each of the electrodes alone. These intermediate pitches may introduce intermediate virtual channels of information. Different pitches can generally be elicited by controlling the proportion of current directed to each of the two electrodes [64]. Psychophysical studies have shown that simultaneous dual-electrode stimulation can produce as few as 2 and as many as 9 discriminable pitches between the pitches of single electrodes [65]. The motivation behind the virtual channel (also known as current-steering) idea is to produce intermediate pitches between electrodes in hopes of increasing the effective number of channels of information beyond the number of electrodes. The performance of the virtual channel strategy on music appreciation is currently being investigated by ABC. Anecdotal reports by some patients (e.g., [66]) fitted with the virtual channel strategy were very encouraging. 4.2 Cochlear Corporation (Nucleus-24 ESPrit 3G/ Freedom device) The Nucleus-24 device (CI24M) is equipped with an array of 22 banded intra-cochlear electrodes and two extra-cochlear electrodes, one being a plate electrode located on the implant package and the other a ball electrode located on a lead positioned under the temporalis muscle [67]. The electrode contacts of the Nucleus 24 Contour array are oriented toward the modiolus minimizing possible current spread away from the target spiral ganglion cells. The electrodes can be stimulated in a bipolar, monopolar or common ground configuration. The extra-cochlear electrodes are activated during monopolar stimulation and can be used individually or together. Biphasic stimulus pulses are generated with electrode shorting during the inter-stimulus gap (about 8 µsecs) to remove any residual charge. The CI24M processor can be programmed with the ACE and CIS strategies[16]. Both strategies estimate the input signal spectrum using a Fast Fourier Transform (FFT) rather than a bank of band-pass filters. The filterbank is implemented using a 128 point Hanning Window and an FFT. Based on a sampling rate of 16 khz this provides an FFT channel spacing of 125 Hz and a low-pass filter cut-off frequency of 180 Hz. The FFT bins, which are linearly spaced in frequency, are used to produce n (12-22) filter bands, which are typically linearly spaced from 188 to 1312 Hz and then logarithmically spaced up to 7938 Hz. A total of n (n=20) envelopes are estimated by summing the power of adjacent FFT bins within each of the n bands. In the ACE strategy, a subset of these m envelope amplitudes is then

13 selected in each stimulation time frame. More specifically, 8-12 maximum amplitudes are selected for stimulation. In the CIS strategy, a fixed number of amplitudes are used for stimulation based on processing the signal through a smaller number of bands (10-12). The remaining electrodes are inactivated. Electrodes corresponding to the selected bands are then stimulated in a tonotopic basal to apical order. The stimulation rate can be chosen from a range of 250 to 2400 pps per channel and is limited by a maximum rate of 14,400 pps across all channels. The stimulation rate can either be constant or jittered in time by a percentage of the average rate. When the jittered rate is programmed, the inter-stimulus gap (which is equal for all stimuli within one stimulation interval) is adjusted at every stimulation interval by a random amount. The resulting stimulation rate varies between consecutive stimulation intervals but has a fixed average rate. For stimulation rates less than approximately 760 pps per channel, the filter-bank analysis rate is set to equal the stimulation rate. However, for higher stimulation rates, the analysis frequency is limited by the system to approximately 760 Hz and higher stimulation rates are obtained by repeating stimulus frames (stimuli in one stimulation interval) when necessary. For the 807 pps/channel rate, approximately one in every 17 or 18 stimulation frames is repeated. For the 1615 pps/channel rate, approximately every stimulus frame is repeated. The majority of the Nucleus users are fitted with the ACE strategy [67]. Comparisons between the performance of the ACE, SPEAK and CIS strategies on multiple speech recognition tasks can be found in [15,67]. 4.3 Med-El Corporation (Combi-40+/Tempo+/PULSARci 100 device) The Med-El cochlear implant processor is manufactured by Med-El Corporation, Austria ([68]. The Med-El cochlear implant, also referred to as COMBI-40+ (C40+), uses a very soft electrode carrier specially designed to facilitate deep electrode insertion into the cochlea [69]. Because of the capability of deep electrode insertion (approximately 31 mm), the electrodes are spaced 2.4 mm apart spanning a considerably larger distance (26.4 mm) in the cochlea than any other commercial cochlear implant. The motivation for using wider spacing between electrode contacts is to increase the number of perceivable channels and to minimize potential interaction between electrodes. The implant processor can be programmed with either a high-rate CIS strategy or a high-rate spectral-maxima strategy. The Med-El processor has the capability of generating 18,180 pulses/sec for a high-rate implementation of the CIS strategy in the 12 channel C40+ implant. The amplitudes of the pulses are derived as follows. The signal is first preemphasized, and then applied to a bank of 12 (logarithmically-spaced) bandpass filters. The envelopes of the bandpass filter outputs are extracted using the Hilbert transform [70]. Biphasic pulses, with amplitudes set to the

14 mapped filter outputs, are delivered in an interleaved fashion to 12 monopolar electrodes at a default rate of of 1,515 pulses/sec per channel. The latest Med-El device (PULSARci 100 ) supports simultaneous stimulation of 12 electrodes. Higher (than the COMBI40+) stimulation rates are supported with aggregate rates up to 50,704 pulses/sec. For a 12-channel processor, rates as high as 4,225 pulses/sec/channel can be supported. Different stimulation techniques, including the use of triphasic pulses, are currently being explored by Med-El to reduce or minimize channel interaction associated with simultaneous stimulation. 5.0 Strategies designed to enhance F0 information The above strategies were originally designed to convey speech information but fall short on many respects in conveying adequate vocal pitch (F0) information. Speakers of tonal languages, such as Cantonese and Mandarin, make use of vocal pitch variations to convey lexical meaning. Several researchers have demonstrated that CI users fitted with current strategies have difficulty discriminating between several tonal contrasts [71,72]. Also, CI users are not able to perceive several aspects of music including identification of familiar melodies and identification of musical instruments [73,74]. Hence, strategies designed to improve coding of F0 information are critically important for better tonal language recognition and better music perception. Pitch information can be conveyed in cochlear implants via temporal and/or spectral (place) cues [52,75-79]. Temporal cues are present in the envelope modulations of the band-pass filtered waveforms (see example in Figure 7). Pitch can be elicited by varying the stimulation rate (periodicity) of a train of stimulus pulses presented on a single electrode, with high pitch percepts being elicited by high stimulation rates, and low pitch percepts being perceived by low stimulation rates. Once the stimulation rate increases beyond 300 Hz, however, CI users are no longer able to utilize such temporal cues to discriminate pitch [77]. Pitch may also be conveyed by electrode place of stimulation due to the tonotopic arrangement of the electrodes in the cochlea. Stimulation of apical electrodes elicits low pitch percepts while stimulation of basal electrodes elicits higher pitch percepts. Access to spectral cues is limited however, by the number of electrodes available (ranging from in commercial devices), current spread causing channel interaction and possible pitch reversals due to suboptimal electrode placement. A number of strategies have been proposed to enhance spectral (place) cues and/or temporal cues, and these strategies are described next Enhancing spectral (place) cues Two different strategies have been explored to improve place coding of F0 information. The first approach is based on the use of virtual channels via the means of dual-electrode (simultaneous) stimulation. By properly

15 manipulating (or steering) the current delivered simultaneously to adjacent electrodes, it is possible to elicit pitches intermediate to the pitches elicited by each of the electrodes alone. These intermediate pitches may introduce intermediate virtual channels of information. The virtual-channel approach is still in its infancy stages, and is currently being evaluated by several labs. The second approach is based on modifying the shape of the filter response and/or the filter spacing. Such an approach was taken in [48,51,52]. A new filter bank was proposed by Geurts and Wouters [51] based on a simple loudness model used in acoustic hearing. The filter was designed such that the loudness of a pure tone sweeping through the filter increased linearly with frequency from the lower 3-dB cutoff frequency to the center frequency of each band, and decreased linearly from the center frequency to the upper boundary frequency. The resulting shape of the filters was triangular, with considerable overlap between adjacent filters. More filters were allocated in the low frequencies compared to a conventional filter-bank which was based on log spacing. The new filter-bank was tested on an F0 detection task in the absence of temporal cues by applying a 20-Hz low pass filter to the filter bank envelope signals. The new technique provided lower detection thresholds to F0 for synthetic vowel stimuli compared to a conventional filter bank approach. However, when temporal cues to F0 were reintroduced, differences in detection thresholds between filter banks were reduced indicating that the temporal cues also provided some information about F0. Kasturi and [48] proposed the use of semitone-based filter spacing for better music perception. Results with cochlear implant simulations indicated that the semitone filter spacing consistently yielded better performance than the conventional filter spacing. Nearly perfect melody recognition was achieved with only four channels of stimulation based on the semitone filter spacing. Subsequent studies with Clarion CII users indicated that some subjects performed significantly better with 6 channels based on semitone spacing than with 16 channels spaced logarithmically as used in their daily strategy Enhancing temporal cues The strategies designed to enhance temporal cues can be divided into two main categories: those that explicitly code F0 information in the envelope and those that aim to increase the modulation depth of the filtered waveforms in hopes of making F0 cues perceptually more salient. The idea of modulating the extracted envelope by explicit F0 information is not new and dates back to the original channel vocoder synthesizer (Figure 2), which was based on a source-filter excitation approach. In channel-vocoded speech, voiced segments of speech are generated by exciting the vocal tract by a periodic (glottal) pulse train consisting of pulses spaced 1/F0 secs apart. Note that the F0-modulation idea was initially used in feature extraction strategies in the Nucleus device and later abandoned because of the inherent difficulty in extracting reliably F0 from the acoustic signal, particularly in noise. Jones et al. [80] investigated a

16 strategy that provided F0 timing information on the most apical electrode. Results from several pitch perception tasks did not demonstrate any advantages with this approach. Green and colleagues [5,81,82] adopted a similar approach to the enhancement of temporal pitch cues, based on the principle that F0 could be automatically extracted from voiced segments of the speech signal and used to appropriately modulate the envelopes. In the proposed strategy, amplitude envelopes were effectively split into two separate components. The first component contained slow rate information of 32 Hz conveying the dynamic changes in the spectral envelope that are important for speech. The second component presented F0 information in the form of a simplified synthesized waveform. More specifically, F0-related modulation was presented in the form of a saw-tooth waveform, on the assumption that such a temporally sharpened modulation envelope, with a rapid onset in each period, would lead to more consistent inter-pulse intervals in the neural firing pattern, and therefore to more salient temporal pitch cues [5]. Implant users were required to label the direction of pitch movement of processed synthetic diphthong glides. Results indicated a significant advantage for the modified processing compared to standard CIS processing, demonstrating that the modified processing scheme was successful in enhancing the salience of temporal pitch cues. Subsequent studies by Green et al. [82], however, on tests of intonation perception and vowel perception indicated that the CI users performed worse with the F0-modified processing in vowel recognition compared to the conventional CIS strategy. The investigators concluded that while the modified processing enhanced pitch perception [5], it harmed the transmission of spectral information. The above strategies assumed access to explicit F0 information. A number of strategies were proposed that did not rely on automatic extraction of F0 from the acoustic signal. These techniques focused on sharpening the envelopes so as to make the F0 information more apparent or perceptually more salient. This was accomplished by increasing the modulation depth of the envelopes. Geurts and Wouters [83] proposed a simple modification to the estimation of the envelope. Two fourth-order low-pass filters were first employed with cutoff frequencies of 400 Hz and 50 Hz. Note that the envelope output of the 400-Hz filter contained F0 modulations, but the envelope output of the 50-Hz filter did not. The modulation depth of the envelope was increased by subtracting an attenuated version of the 50-Hz (flat) log-compressed envelope from the 400-Hz log-compressed envelope. The resulting envelope was half-wave rectified (negative values set to zero), scaled and finally encoded for stimulation (note that the envelopes were already compressed to the patient s dynamic range prior to the subtraction operation). Despite the increase in modulation depth with the modified envelope processing, no significant differences in F0 discrimination of synthetic vowels were observed compared to the conventional CIS strategy [83]. Figure 9 shows examples of envelopes extracted with the above scheme

17 and compared with envelopes extracted with conventional rectification and low-pass filtering (400 Hz). The subtraction of the 400-Hz envelope amplitude from the 50-Hz envelope is equivalent to subtraction of the mean (dc component) of the rectified envelope, and constitutes a simple method for increasing envelope modulation depth. This idea was incorporated in one of the strategies proposed by Vandali et al. [6] to increase the modulation depth of the envelopes. The so called, Multi-channel Envelope Modulation (MEM) strategy utilized the envelope of the broadband signal (input acoustic signal prior to bandpass filtering), which inherently contains F0 periodicity information, to modulate the envelopes derived from the ACE filterbank. The envelope of the broadband signal was first estimated by full wave rectifying the broadband signal and then applying a 300-Hz, fourth-order low-pass filter (LPF). The modulation depth in the envelope signal was then expanded by applying an 80-Hz, second-order high-pass filter (HPF), which effectively increased the modulation depth of the envelope signal level by removing the mean (dc) component. Note that this step is equivalent to that of subtracting the mean of the rectified signal as done in [83]. The low-pass filtered signal, obtained prior to the HPF stage, was scaled and added to the output of the HPF stage. The expanded envelope signal was then half-wave rectified, to remove any negative values, and scaled. Finally, the narrow-band envelope signals estimated by the ACE filter bank were low-pass filtered, using a 50- Hz, second-order LPF, and then modulated by the normalized F0 envelope signal derived from the broadband signal. Figure 10 shows an example of the processed signal at different stages of the algorithm. As can be seen the derived envelope has large modulation depth and the F0 periodicity is evident in the envelopes. Note also that the envelopes are temporally synchronized (across all electrodes) with the input (broadband) waveform. The second strategy (termed Modulation Depth Enhancement MDE- strategy) evaluated by Vandali et al. [6] provided explicit modulation expansion by decreasing the amplitude of the temporal minima of the envelope. Modulation depths smaller than a specified level were expanded using a third-order power function, and modulation depths above this level, but below an upper limit of 20 db, were linearly expanded. The modulation depth expansion was implemented by decreasing the amplitude of temporal minima in the signal while preserving the peak levels (a sliding time window, of duration 10 ms, was employed to track the peaks and minima). The modified envelope signals replaced those of the original envelope signals derived from the filter bank, and processing continued as per the normal ACE strategy. Comparison of the above strategies with the conventional ACE strategy indicated significantly higher scores with the MDE and MEM strategies on pitch ranking tasks. Comparison of the new strategies, however, on speech recognition tasks indicated no significant differences in scores with the conventional ACE strategy. In brief, most of the above F0-enhancement strategies have been shown to improve pitch perception on tasks requiring discrimination of small