Cortical Coding of Auditory Features

Transcription

1 Annu. Rev. Neurosci : The Annual Review of Neuroscience is online at neuro.annualreviews.org Copyright c 218 by Annual Reviews. All rights reserved Annual Review of Neuroscience Cortical Coding of Auditory Features Xiaoqin Wang Laboratory of Auditory Neurophysiology, Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Maryland 2125, USA; xiaoqin.wang@jhu.edu Tsinghua Laboratory of Brain and Intelligence (THBI) and Department of Biomedical Engineering, Tsinghua University, Beijing 184, China Keywords auditory cortex, speech, hearing, pitch, music, neural coding Abstract How the cerebral cortex encodes auditory features of biologically important sounds, including speech and music, is one of the most important questions in auditory neuroscience. The pursuit to understand related neural coding mechanisms in the mammalian auditory cortex can be traced back several decades to the early exploration of the cerebral cortex. Significant progress in this field has been made in the past two decades with new technical and conceptual advances. This article reviews the progress and challenges in this area of research. 527

2 Contents INTRODUCTION STIMULUS SELECTIVITY AND NEURAL FIRING PATTERNS INTHEAUDITORYCORTEX NONLINEARITYOFAUDITORYCORTEXNEURONS Unresponsive Neurons in the Auditory Cortex Complex Feature Selectivity of Neurons in the Primary Auditory Cortex andunderlyingnonlinearinteractions NONLINEAR REPRESENTATIONS OF SOUND LEVEL REPRESENTATIONSOFTIME-VARYINGSIGNALS SynchronizedandNonsynchronizedCorticalResponses PositiveandNegativeMonotonicRateTuning Comparison of Temporal and Rate Representations at the Primary Auditory CortexandtheMedialGeniculateBody Functional Implications of Temporal-to-Rate Transformation in the Auditory Cortex REPRESENTATIONSOFPITCHANDHARMONICS Harmonically Related Frequency Tuning in the Auditory Cortex Harmonic Template Neurons of the Auditory Cortex Pitch Processing by the Auditory Cortex TheHarmonicOrganizationHypothesis CONCLUSIONS INTRODUCTION The pursuit to understand neural coding principles in the mammalian auditory cortex can be traced back several decades to the early exploration of the cerebral cortex. Progress in understanding this sensory cortical area, however, has been much slower than that for other sensory cortices such as the visual and somatosensory areas. In contrast to the visual system, the auditory system has a longer subcortical pathway and more spiking synapses between the peripheral receptors and the cortex. This unique organization reflects the need of the auditory system to extract behaviorally relevant information from a complex acoustic environment by using strategies different from those used by other sensory systems. It has also presented challenges to researchers studying neural coding mechanisms at the central auditory processing stations, including the auditory cortex. The difficulties have resulted, in part, from the complexity and nonlinearity of the mammalian auditory cortex, as we discuss in this article. In addition, many cortical neurons, particularly those that are highly selective for complex stimulus features, may be rendered unresponsive by anesthesia, a fact that has long been recognized by pioneers in this field (Davies et al. 1956, Goldstein & Abeles 1975). When David Hubel ventured into the auditory cortex before his more famous work on the visual cortex, he noticed that some neurons in the cat auditory cortex were particularly difficult to drive, as shown by a neuron in Figure 1a (Hubel et al. 1959). This neuron almost never responded to clicks, tones, or noise from a nearby loudspeaker but was responsive to squeaks emitted when a toy mouse was squeezed. This intriguing observation has led to important discoveries many decades later, as illustrated in this article. Sounds such as human speech (Houtgast & Steeneken 1973, Rosen 1992), music (Peretz & Zatorre 25), and animal vocalizations (Agamaite et al. 215, Singh & Theunissen 23) 528 Wang

3 contain acoustic information distributed across multiple frequencies that span timescales from a few milliseconds to hundreds of milliseconds or seconds. The peripheral auditory system encodes acoustic signals containing features of speech, music, and vocalizations such as pitch, formants (spectral envelope), amplitude modulation (temporal envelope), frequency modulation (FM), and sound level dynamics. These features are progressively processed by successive stations along the ascending auditory pathway and transformed into neural representations of the central auditory a 1 s Spikes/s b c Firing rate (spikes/s) A1 neuron, stimulus: BF tone 1, ms 5 1,5 2,5 3,5 4,5 5, Time (ms) A1 (n = 72 neurons) Preferred Nonpreferred d Auditory cortex Medial geniculate body Inferior colliculus Cochlear nucleus Auditory nerve Nonpreferred stimuli (onset firing) Preferred stimuli (sustained firing) Multidimensional RF Ascending auditory pathway 5 1, 1,5 Time (ms) (Caption appears on following page) Cortical Coding of Auditory Features 529

4 Figure 1 (Figure appears on preceding page) (a) The upper trace shows the response of a neuron in the cat auditory cortex studied by Hubel et al. (1959). The lower trace shows sounds (squeaks) emitted when a toy mouse was squeezed. This neuron almost never responded to clicks, tones, or noise from a nearby loudspeaker. Modified from Hubel et al. (1959). (b) An example of sustained firing evoked by long-duration stimuli recorded from an A1 neuron in awake marmosets. Stimulus is a pure tone at the neuron s BF (9.3 khz), SPL is 8 db, rise/fall time is 1s, duration is 5 s (thick bar below x-axis). (Top) Raw recording trace of the response to one presentation of the stimulus. (Bottom) PSTH computed from responses to 5 repetitions of the same stimulus (binwidth: 2 ms). Modified from Wang et al. (25). (c) Temporal firing patterns evoked by preferred and nonpreferred stimuli. Mean PSTHs calculated from a population of A1 neurons in response to each neuron s preferred and nonpreferred stimuli, respectively. The preferred stimulus is a temporally modulated signal (sam, sfm, or nam) and set at the BMF. The nonpreferred stimulus is set at the modulation frequency corresponding to the minimum firing rate above the BMF. PSTH binwidth: 5 ms (smoothed by a 5-point moving triangular window). The thick bar below the x-axis indicates the stimulus duration (1 s). Modified from Wang et al. (25). (d ) Illustration of progressively increasing stimulus selectivity along the ascending auditory pathway and the relationship between the stimulus selectivity and sustained and onset firings. The ellipse represents the RF of a neuron in the acoustic parameter space, which is multidimensional but illustrated here on a two-dimensional plane. The black-filled, smaller ellipse represents the sustained firing region (corresponding to preferred stimuli) of a neuron s RF. The white area within the large ellipse represents the onset firing region (corresponding to nonpreferred stimuli) of a neuron s RF. A neuron exhibits sustained or onset firing depending on which region of its RF is stimulated. The neuron does not fire if stimuli fall outside its RF. Modified from Wang (27). Abbreviations: A1, primary auditory cortex; BF, best frequency; BMF, best modulation frequency; nam, amplitudemodulated noise; PSTH, peristimulus time histogram; RF, receptive field; sam, sinusoidal amplitude-modulated tone; sfm, sinusoidal frequency-modulated tone; SPL, sound pressure level. system including the auditory cortex. This article reviews recent progress in cortical coding of auditory features of complex sounds and includes historical perspectives where appropriate. STIMULUS SELECTIVITY AND NEURAL FIRING PATTERNS IN THE AUDITORY CORTEX Neurons in the auditory cortex of anesthetized animals generally display transient responses to acoustic stimulation and typically respond to a brief stimulus with one or a few spikes (Calford & Semple 1995, DeWeese et al. 23, Eggermont 1997, Heil 1997, Phillips 1985). A long steady-state stimulus does not evoke continuous neural activity over its duration in the auditory cortex under anesthesia (decharms & Merzenich 1996). These observations have long puzzled researchers and have raised serious questions about the role of the auditory cortex in encoding ongoing acoustic signals. The transient nature of auditory cortical responses and the lack of neural firing throughout stimulus duration have prompted researchers to propose various theories to explain neural coding strategies. For example, it has been suggested that neurons in the primary auditory cortex (A1) are specialized to respond to brief stimulus events (Phillips 1993) and that correlated firings between neurons, instead of firing rates of individual neurons, signal the presence of steady-state sounds (decharms & Merzenich 1996, Eggermont 1997). However, new experimental evidence accumulated over the past decade has now demonstrated that neurons recorded from the auditory cortex of awake animals exhibit both onset and sustained discharges in response to continuous acoustic stimulation (Bieser & Müller-Preuss 1996, Chimoto et al. 22, Lu et al. 21, Malone et al. 22, Mickey & Middlebrooks 23, Recanzone 2, Wang et al. 25). An example of A1 neurons responding continuously to a long-duration pure tone is shown in Figure 1b. Although some neurons exhibited sustained firings in response to long-duration pure tones or broadband noises, most A1 neurons did so when they were stimulated by their preferred (nearly optimal) stimuli that had greater temporal and spectral complexity. Recent studies of the auditory cortex of awake marmosets (Callithrix jacchus) showed that when neurons were driven by their preferred stimuli, they responded not only with higher discharge rates but also with sustained firings throughout the duration of the stimulus. In contrast, responses became more transient (or phasic) when auditory cortex neurons responded to nonpreferred stimuli (Figure 1c). 53 Wang

5 These observations demonstrate a general principle of neural firing in the auditory cortex; namely, whether a neuron responds to a stimulus with sustained firing depends crucially on the optimality of the stimulus (Wang et al. 25). A marked feature of auditory neurons at processing stations along the ascending pathway is their progressively increased stimulus selectivity. The increased stimulus selectivity is also accompanied by changes in a neuron s firing pattern. An auditory nerve is primarily selective for one dimension of a stimulus, the frequency of a pure tone. For neurons in the auditory cortex, their stimulus selectivity may be defined in a multidimensional acoustic space, for example, frequency, spectral bandwidth, sound intensity, and amplitude or frequency modulation. In the auditory cortex of awake animals, neurons (especially those in superficial cortical layers) are often highly selective for acoustic stimuli, and as such, the preferred stimulus of a neuron occupies only a small region of its receptive field (RF) in the multidimensional acoustic space (Figure 1d). We suggested that the RF of a cortical neuron contains a smaller sustained firing region (corresponding to the preferred stimulus) and a larger onset firing region (corresponding to the nonpreferred stimuli) (Wang 27). The sustained firing region becomes increasingly smaller from the auditory nerve to the auditory cortex. This explains why it is common for experimenters to observe mostly transient (onset) responses in the auditory cortex. As we discuss later in this article, many neurons in the auditory cortex are selective for stimuli more complex than the examples shown in Figure 1, such as combinations of harmonically related tones or pitch. The overall picture elucidated by these experiments is that when a sound is heard, the auditory cortex first responds with transient (onset) discharges across a relatively large population of neurons. As time passes, the activation becomes restricted to a smaller population of neurons that are preferentially driven by the sound, which results in a selective representation of the sound across both neuronal population and time (Wang et al. 25). Because each neuron has its own preferred stimulus, which differs from the preferred stimuli of other neurons, the neurons in the auditory cortex collectively cover the entire acoustic space with their sustained firing regions. Therefore, any particular sound can evoke sustained firing throughout its duration in a particular population of neurons in the auditory cortex. That is, the region of the auditory cortex activated by acoustic stimulation in whole-brain imaging measurements (e.g., functional MRI, positron emission tomography) corresponds to neurons that are preferentially driven by the acoustic stimulus. The discovery of how sustained firing can be evoked in the auditory cortex is important because it provides a direct link between neural firing and the perception of a continuous acoustic event. Such sustained firing patterns evoked by a neuron s preferred stimulus have been observed only in awake animals. In comparison, auditory nerve fibers typically respond to various acoustic signals with sustained firing as long as a stimulus s spectral energy falls within a neuron s RF, under either anesthetized or unanesthetized conditions. When Hubel et al. (1959) ventured into the auditory cortex more than half a century ago, they were puzzled by the difficulty of driving neurons in the cat auditory cortex (Figure 1a). Now we know they were likely recording from some highly selective neurons and stimulating the onset firing regions with nonpreferred stimuli. The availability of digital technology since then has made it possible to create and test a large battery of acoustic stimuli in search of the preferred stimulus of a neuron. NONLINEARITY OF AUDITORY CORTEX NEURONS Unresponsive Neurons in the Auditory Cortex In the auditory cortex of awake animals, a substantial number of neurons do not respond to pure tones, which has been documented since the early explorations of this cortical area. These neurons Cortical Coding of Auditory Features 531

6 have historically been classified unresponsive and even speculated as being nonauditory (Evans & Whitfield 1964, Hubel et al. 1959). Studies have estimated that 25 5% of A1 neurons in awake rats are unresponsive to auditory stimulation (Hromadka et al. 28) and that 2 3% of A1 neurons in awake marmosets are unresponsive to pure tones (Sadagopan & Wang 28). These studies typically used relatively simple auditory stimuli such as pure tones and noises. These unresponsive or non-tone-responsive neurons have been largely ignored by researchers in the past. Most published data from A1 have therefore been restricted to tone-responsive neurons likely located in the middle, or thalamo-recipient, cortical layers (monkeys: Philibert et al. 25, Recanzone 2; cats: Merzenich et al. 1975, Moshitch et al. 26, Schreiner & Mendelson 199; rats: Kilgard & Merzenich 1999; mice: Linden et al. 23). Recent studies of awake marmosets have shown, however, that many A1 neurons were in fact highly selective for complex sound features (Sadagopan & Wang 29). The non-tone-responsive A1 neurons in awake marmosets exhibited nonlinear combination-sensitive responses that require precise spectral and temporal combinations of two-tone pips and were commonly encountered in superficial cortical layers. In the thalamo-recipient layers of A1, sounds may be represented by neurons that are tuned to pure tones or individual frequency components (Kilgard & Merzenich 1999, Linden et al. 23, Merzenich et al. 1975, Moshitch et al. 26, Philibert et al. 25, Recanzone 2, Sadagopan & Wang 28). In secondary auditory cortical areas, neurons are typically not responsive to pure tones but exhibit tuning to complex features such as noise bandwidth (Rauschecker et al. 1995), species-specific vocalizations (Tian et al. 21), and behaviorally relevant sounds (Hubel et al. 1959). This marked increase in RF complexity suggests that an intermediate stage might exist within A1, providing a bridge between tone-tuned thalamo-recipient A1 neurons and neurons in the secondary auditory cortex that exhibit complex response properties. One likely candidate for such an intermediate stage is the population of non-tone-responsive neurons that is often encountered in the superficial layers of A1 in awake animals (Evans & Whitfield 1964, Hromadka et al. 28, Sadagopan & Wang 28). Such neurons are ideally positioned to form a stage of intermediate complexity in an auditory processing hierarchy, bridging the gap in RF complexity between tone-responsive A1 neurons and neurons in secondary auditory cortical areas that encode complex sound features (Hubel et al. 1959, Rauschecker et al. 1995, Tian et al. 21). Complex Feature Selectivity of Neurons in the Primary Auditory Cortex and Underlying Nonlinear Interactions In the superficial layers of A1 of awake marmosets, neurons that exhibited high stimulus selectivity for particular features of complex sounds such as marmoset vocalizations were often encountered. Figure 2a illustrates the responses of such a neuron that showed significant responses to 6 of 4 vocalization tokens consisting of 2 natural (forward) calls and their reversed versions. In addition to an overall preference for the natural over the reversed vocalization tokens, a close examination showed that this neuron reliably responded only when a particular feature (an upward FM trill element) of the natural trill-twitter call occurred (Figure 2b). Despite its location within A1, this neuron was unresponsive to tones (Figure 2e). However, the kind of data shown in Figure 2a,b by itself does not reveal what causes such high stimulus selectivity. Sadagopan & Wang (29) applied a two-pip stimulus set to explore underlying mechanisms. A nonlinear (second-order) two-pip interaction map based on the neuron s responses to many two-pip combinations was constructed (Figure 2c). It revealed that the minimal effective stimulus (referred to as subunits) for this neuron was the combination of a 5.8-kHz tone pip followed 75 ms later by a 6.9-kHz BF tone pip, with both pips at 2-dB sound pressure level (SPL). When this neuron was tested with upward and downward linear FM sweeps spanning from Wang

7 to 7.2 khz, it showed tuning (maximal firing rate) for the 8-ms-long upward sweep (Figure 2d), as would be predicted from the two-pip interaction map in Figure 2c. The exquisite sharpness of this neuron s tuning to this feature should be emphasized; at half-maximal response, the neuron responds to FM sweep velocities ranging from 3.63 octaves (oct.)/s (16 Hz/ms) to 6.4 oct./s (26.7 Hz/ms). The two-pip interaction map (Figure 2c) also explains this neuron s responses to vocalizations (Figure 2a,b). Qualitatively, one can observe that the initial upward-going trill part of the vocalization in Figure 2b is an 1-ms-long, upward FM fragment of approximately 6 khz that overlays the subunits of the two-pip interaction map shown in Figure 2c. However, in the reversed version of this vocalization, the upward FM fragment becomes downward and no longer overlays the subunits of the two-pip interaction map. As shown in Figure 2b, this neuron did not a Vocalization token # c Relative frequency (octaves) f Firing rate (spikes/s) 2r 2n 1r 1n Time (s) u 6.9 khz/2 db 18% Relative onset time (ms) d Stimulus type d Firing rate (spikes/s) nstim = 831 FRA Col. Voc. 2pip lfm lfm BPN BW Tone 2pip 2tone sam sfm click FM Up b Frequency (khz) Reversed lfm: khz/2 db Down Sweep length (ms) g Absolute depth (mm) *.31 Natural Time (s) e Frequency (khz)/level (db) Relative depth (mm) 14 khz 8 db 6.9 khz 1.1 Tones 3.5 khz db 1 2 2pip responsive Pure tone responsive * 1.16 Time (ms) (Caption appears on following page) Cortical Coding of Auditory Features 533

8 Figure 2 (Figure appears on preceding page) Selectivity for complex features in A1. (a) Raster of a neuron s responses to 2 pairs of marmoset vocalizations. Each pair contains a natural (forward) and a reversed vocalization token. Gray shading corresponds to vocalization duration; different vocalization tokens have different lengths. Gray dots and black dots correspond to spontaneous spikes and spikes falling within the analysis window (from 15 ms after stimulus onset to 5 ms after stimulus offset), respectively. (b) An expanded view showing the preferential responses of the neuron shown in panel a to a natural trill-twitter call over its reversed version. Among 4 vocalization tokens tested, the natural trill-twitter call elicited the maximal response in this neuron. The dot raster is overlaid on the spectrogram of the trill-twitter call, with spikes plotted around the BF of this neuron (6.9 khz). Note that the maximal response occurs immediately following the initial upward trill segment. (c) The neuron shown in panel a responded strongly to a specific combination of two-tone pips a 5.8-kHz tone pip followed 75 ms later by a 6.9-kHz BF tone pip (red disk; both tone pips at 2-dB SPL). A secondorder interaction (nonlinear) map is plotted (Sadagopan & Wang 29); image is smoothed for display. Colormap indicates the percentage of facilitation over the sum of first-order responses; dark-red contour and pink contour denote significance at p <.1 and p <.5 (modified permutation test), respectively. The nonlinear component was 18% the sum of the linear components. Gray lines represent lfm sweep stimuli tested in panel d; color intensity corresponds to response strength (lightest gray, spikes/s; darkest gray, 12.5 spikes/s). (d ) The neuron shown in panel a strongly responded to upward FM sweeps that connected the RF subunits in the nonlinear map but not to downward FM sweeps that spanned the same frequency range (mean rate plotted, error bars are 1 SD). The neuron was tuned to an 8-ms-long upward lfm sweep spanning khz (darkest gray line in panel c), precisely connecting the subunit. (e) The neuron shown in panel a was unresponsive to pure tones over a wide range of frequencies (2 octaves) and levels around estimated BF and BL (raster shown; frequency and level are interleaved on the y-axis). ( f ) Example of a highly nonlinear A1 neuron that only responded to a few of the 831 stimuli tested. Each dot is the driven response rate (after subtracting spontaneous rate) of an individual stimulus belonging to that particular stimulus set. ( g) Distributions of absolute depth (from dural surface) and relative depth (from first neuron encountered) of nonlinear (non-tone-responsive) (black dots) and tone-responsive ( gray dots) neurons. Nonlinear neurons were located at shallower cortical depths compared with tone-responsive neurons (histograms of absolute and relative depths are on margins, numbers are medians; and denote p <.5 and p <.1, respectively, from a Wilcoxon rank-sum test). This suggested that nonlinear neurons may be localized to superficial cortical layers. Median spontaneous firing rate was significantly lower in the nonlinear neurons than in the pure-tone-responsive neurons (.9 spikes/s versus 2.7 spikes/s, p <.1 Wilcoxon rank-sum test). Figure modified from Sadagopan & Wang 29. Abbreviations: 2pip, two-pip; 2tone, two-tone; A1, primary auditory cortex; BF, best frequency; BL, best level; BPN, band-pass noise of varying bandwidths; BW, bandwidth; FM, frequency modulation; Col., colony noise (environmental sounds from monkey colony); d, downward lfm sweep; FRA, frequency response area (tones); lfm, linear FM; n, natural; nstim, number of stimuli; r, reversed; RF, receptive field; sam, sinusoidal amplitude-modulated tone; sfm, sinusoidal frequency-modulated tone; SPL, sound pressure level; u, upward lfm sweep; Voc., marmoset vocalizations. respond to the reversed trill-twitter call. These examples of responses illustrate how this neuron s selectivity for FM sweep fragments and preference for forward over reversed calls are related to response properties, and how they arise from a simple underlying nonlinear computation. Figure 2f shows another single-unit example of complex tuning properties observed in nontone-responsive A1 neurons. This neuron cannot be driven by a large number of stimuli from an extensive repertoire (831 stimuli of 15 different types), including species-specific marmoset vocalizations, environmental sounds, FM sweeps, random FM contours, amplitude-modulated tones, noise of different bandwidths, and click trains. Only a few two-pip stimuli (4-ms pips) elicited strong and robust responses (Sadagopan & Wang 29). These highly nonlinear A1 neurons were commonly found in superficial cortical layers (Figure 2g). In contrast, the tone-responsive neurons were more commonly found at deeper cortical depths. The nonlinear neurons typically exhibited lower spontaneous firing rates than pure-tone-responsive A1 neurons, suggesting that they are likely to be missed if only spontaneous spikes or simple acoustic stimuli are used to search for these neurons. Data in Figure 2 demonstrate that non-tone-responsive neurons are in fact highly selective for complex acoustic stimuli. The broad distributions of BF and preferred 534 Wang

9 sound level of these nonlinear neurons suggest that they may serve as feature detectors for various complex sounds such as marmoset vocalizations (Sadagopan & Wang 29). The high selectivity, non-tone responsiveness and low spontaneous firing rates of these neurons make them difficult to isolate and drive, which may explain why earlier studies have not encountered a large number of such neurons. Combination-sensitive neurons at both cortical and subcortical levels in the auditory system of echolocating bats have been well described (Esser et al. 1997; O Neill & Suga 1979; Razak & Fuzessery 28; Suga et al. 1978, 1983). Combination-sensitive neurons specific to song sequences have been observed in songbirds (Lewicki & Konishi 1995, Margoliash 1983). The observations obtained from awake marmosets support the notion that the combination selectivity initially demonstrated in echolocating bats is a general organizational principle for cortical neurons across many species. The marmoset data differ from the data of these previous studies in that they show a greater diversity of BFs, best levels (BLs), frequency differences, and longer timescales of combination selectivity. NONLINEAR REPRESENTATIONS OF SOUND LEVEL In auditory nerve fibers and successive processing stations, neural firing rates typically increase with increasing sound level until saturation, a pattern referred to as a monotonic rate-level function (Sachs & Abbas 1974). In the auditory cortex of the echolocating bat, neurons narrowly tuned to sound level were observed (Suga 1977). Such response patterns are called nonmonotonic ratelevel functions. In the auditory cortex of awake primates, neurons with nonmonotonic rate-level functions spanning a wide range of BLs have been observed (Brugge & Merzenich 1973, Pfingst & O Connor 1981). In particular, 8% of neurons in the auditory cortex of behaving macaques had nonmonotonic rate-level functions (Pfingst & O Connor 1981) (Figure 3a). Such nonmonotonic rate-level functions indicate nonlinear responses to increasing sound level and are a markedly transformed property from the peripheral auditory system. A neuron with a monotonic rate-level function would have a V-shaped or I-shaped frequency response area (FRA), which characterizes a neuron s response to pure tones of varying frequency and level. By contrast, a neuron with a nonmonotonic rate-level function would have an O-shaped FRA. Figure 3b shows examples of both types of FRAs. In A1 of awake marmosets, most neurons ( 6%) (Sadagopan & Wang 28) had O-shaped FRAs, whereas in A1 of anesthetized animals, most neurons exhibited V-shaped FRAs (Moshitch et al. 26, Schreiner et al. 2, Schreiner & Mendelson 199). A population analysis of FRA types shows a bimodal distribution of V-shaped and O-shaped A1 neurons in awake marmosets (Figure 3c). The fraction of nonmonotonic neurons observed in A1 of awake marmosets ( 64% over entire response duration, 76% during sustained response) was similar to that found in A1 of behaving macaques (78%) (Pfingst & O Connor 1981). The number of O-shaped neurons observed in awake primates was substantially greater than what has previously been reported for anesthetized animals. For example, Sutter (2) reported that 2% of neurons in anesthetized cat A1 had circumscribed (O-shaped) FRAs. Similarly, an earlier study (Phillips & Irvine 1981) reported that 23% of neurons in anesthetized cat A1 had nonmonotonic rate-level functions. The O-shaped neurons differed from V-shaped neurons in other physiological properties, which may explain why the predominance of O-shaped neurons in A1, even in unanesthetized preparations (Recanzone 2), had not been reported previously. In awake marmosets, the median spontaneous firing rate and the maximum driven rate of O-shaped neurons (1.74 spikes/s, 22 spikes/s, respectively) were approximately half of the corresponding measures in V-shaped neurons (4 spikes/s, 42 spikes/s, respectively) (Sadagopan & Wang 28). These properties suggest that O-shaped neurons would be more difficult to find than V-shaped Cortical Coding of Auditory Features 535

10 neurons if the strategy for isolating a neuron during an experiment involved listening for spontaneous spikes while advancing electrode position. Because of their narrower frequency and sound level tuning and lower maximum driven rate, O-shaped neurons would also be harder to drive if experimental stimuli did not finely sample the frequency and sound level axes. In comparison, Recanzone (2) reported a mean spontaneous rate of 8.2 spikes/s and a mean peak driven rate of 39.3 spikes/s for A1 neurons in awake and behaving macaques. These values resemble those of V-shaped neurons found in awake marmosets, lending support to our hypothesis that O-shaped neurons may have been previously missed owing to search biases. We reduced the magnitude of these biases in our experiments in marmosets by not relying on a fixed set of search stimuli while isolating single neurons, by sampling the frequency axis at a relatively high density, and by testing a Percentage of maximum discharge rate Stimulus intensity (db SPL) c Number of units O (n = 175) I, V (n = 1) Shape index d Population I/V FRA (n = 1) b Sound level (db) Number of units O Frequency (khz) ** Monotonicity index e Population O FRA (n = 175) I/V Level rel. thresh. (db) Normalized response Level rel. BL (db) Normalized response Relative frequency (octaves) Relative frequency (octaves) (Caption appears on following page) 536 Wang

11 Figure 3 (Figure appears on preceding page) (a) Examples of nonmonotonic rate-level functions recorded from neurons in the auditory cortex of awake and behaving macaques. Modified from Pfingst & O Connor (1981). (b) Examples of RFs of I-, V-, and O-shaped neurons. I-shaped neurons have narrower RF shapes than V-shaped neurons. (Top) Lines correspond to RF extents of randomly selected I- and V-shaped neurons at half-maximal firing rate. This represents a traditional view of the auditory cortex whereby the stimulus amplitude is represented by neurons with different thresholds. Frequency resolution decreases with increasing level in V-shaped neurons. (Bottom) Ellipses correspond to O-shaped neuron RFs at half-maximal firing rate. This representation of frequency-level space is different from the traditional view; the amplitude in each frequency range is encoded by multiple neurons tuned to a smaller range of levels and frequency tuning width is independent of level (colors of lines and ellipses correspond to BF for clarity). Modified from Sadagopan & Wang (28). (c) Shape and monotonicity indices. (Left) The distribution of the SI computed from the FRAs of 275 tone-responsive A1 neurons was bimodal. We used SI =.75 as a boundary to separate O-shaped neurons from I- and V-shaped neurons. (Right) MI was strongly correlated with FRA shape. Most O-shaped neurons (black bars) were strongly nonmonotonic (MI =.25) and most V-shaped neurons ( gray bars) had monotonic rate-level functions (MI = 1). Modified from Sadagopan & Wang (28). (d ) Population average FRA of I- and V-shaped neurons (n = 1) centered on BF and threshold. (e) Population average FRA of O-shaped neurons (n = 175) centered at BF and BL. Monotonically increasing inhibition, over a wide frequency range enveloping BF, is apparent at loud levels. Modified from Sadagopan & Wang (21). Abbreviations: A1, primary auditory cortex; BF, best frequency; BL, best level; FRA, frequency response area; MI, modulation index; RF, receptive field; SI, shape index. stimuli across a wide range of sound levels (Sadagopan & Wang 28, Wang et al. 25). Most studies of A1 in anesthetized animals were restricted to layer 4 because of the difficulty of driving layer 2/3 neurons under anesthesia, whereas studies by Sadagopan & Wang (28, 29, 21) extensively sampled the supragranular layers in the awake marmoset. Another interesting issue is how O-shaped FRAs are generated. It remains unclear to what extent O-shaped FRAs are created at A1 or inherited from subcortical stations. For example, a large proportion of nonmonotonic neurons have been found in the auditory thalamus (Bartlett & Wang 211, Rouiller et al. 1983). Alternatively, a cortical inhibition source could act as a candidate for shaping O-shaped FRAs. Figure 3d shows the population average FRA of the V- shaped neurons, aligned at BF and threshold sound level (defined as the sound level evoking 2% of the response). The FRAs were on average asymmetric with a long low-frequency tail. However, when the population average of O-shaped FRAs was plotted, aligned at BF and BL, we observed strong suppression at loud sound levels both on- and off-bf over the entire two-octave range of the sampled frequencies (Figure 3e), which suggests cortical inhibition as a likely source to form O-shaped FRAs. Another possibility is a combination of both factors. Namely, thalamocortical inputs that are weakly nonmonotonic may be further sharpened by cortical inhibition, leading to the strongly level-tuned O-shaped neurons in A1. Why are O-shaped neurons needed for auditory processing and how do they fit into the processing hierarchy? Monkeys such as marmosets make both loud (e.g., phee calls) and quiet (e.g., trill calls) vocalizations that are highly tonal in nature (Agamaite et al. 215). One might hear the same sound in a variety of listening conditions, such as distance from the source, occlusion by intervening objects, and changing ambient noise. Variability in all these conditions determines the effective intensity at which the sound is heard, but the quality of the sound remains unaffected. Similar situations exist for human speech processing. Sadagopan & Wang (28) suggested that O- shaped neurons may reflect an internal representation of this statistical independence of frequency and level, exploiting this property to code sounds efficiently. One may argue that implementing level-invariant coding at an early stage of cortical processing is a highly desirable property that can simplify computational goals of the auditory cortex, such as feature or object recognition. Level invariance at the population level based on the O-shaped neurons in A1 might provide information to cortical areas higher in the auditory processing hierarchy where level invariance could be achieved by single neurons. Cortical Coding of Auditory Features 537

12 REPRESENTATIONS OF TIME-VARYING SIGNALS Synchronized and Nonsynchronized Cortical Responses Researchers have long noticed that neurons in the auditory cortex do not faithfully follow rapidly changing stimulus components (de Ribaupierre et al. 1972, Goldstein et al. 1959, Whitfield & Evans 1965). Many previous studies have shown that discharges of cortical neurons can entrain to temporal modulations only at rates far less than 1 Hz (Bieser & Müller-Preuss 1996; de Ribaupierre et al. 1972; Eggermont 1991, 1994; Gaese & Ostwald 1995; Lu & Wang 2; Phan & Recanzone 27; Schreiner & Urbas 1988; Wallace et al. 22), compared with a limit of >1kHz for the auditory nerve ( Johnson 198, Joris & Yin 1992, Palmer 1982). At subsequent processing stations beyond the auditory nerve, the upper limit of the stimulus-synchronized temporal representation progressively decreases [e.g., cochlear nucleus: Blackburn & Sachs 1989, Frisina et al. 199, Rhode & Greenberg 1994; inferior colliculus (IC): Batra et al. 1989, Krishna & Semple 2, Langner & Schreiner 1988, Liu et al. 26, Müller-Preuss et al. 1994; medial geniculate body (MGB): Bartlett & Wang 27, Creutzfeldt et al. 198, de Ribaupierre et al. 198, Preuss & Müller-Preuss 199, Rouiller et al. 1981] owing to biophysical properties of neurons and temporal integration of converging inputs from one station to the next (Wang & Sachs 1995). By the time neural signals encoding acoustic information reach the auditory cortex, temporal firing patterns alone are inadequate to represent the entire range of time-varying sounds that are perceived by humans and animals. The upper limit of synchronized auditory cortical responses appeared to be similar when measured by amplitude-modulated sounds (Schreiner & Urbas 1988) or acoustic pulse trains (Lu & Wang 2) in neurons tuned to high BFs or when measured by pure tones in neurons tuned to low BFs (Wallace et al. 22). The lack of synchronized cortical responses to rapid but perceivable temporal modulation has been puzzling. Because most previous studies of this subject over the past three to four decades were conducted in anesthetized animals, with a few exceptions (Bieser & Müller-Preuss 1996, Creutzfeldt et al. 198, de Ribaupierre et al. 1972, Evans & Whitfield 1964, Goldstein et al. 1959, Whitfield & Evans 1965), investigators speculated that the reported low temporal response rate in the auditory cortex might be caused partially by anesthetics, which was shown to alter the temporal response properties of the auditory cortex (Goldstein et al. 1959, Zurita et al. 1994). This issue was reexamined in A1 of awake marmosets with the use of periodic acoustic pulse trains whose repetition rates were systematically varied (Lu et al. 21). Two types of cortical responses to periodic click trains were observed (Figure 4a). The first type of cortical response exhibited significant stimulus-synchronized responses to click trains at long interclick intervals (greater than 25 ms) but diminished at shorter interclick intervals. The second type of cortical response did not exhibit stimulus-synchronized discharges but instead showed monotonically changing discharge rates at short interclick intervals. The synchronized and nonsynchronized populations (Lu et al. 21) of A1 neurons appeared to encode repetitive stimuli by spike timing and average discharge rate, respectively (Figure 4b). Neurons in the synchronized population showed stimulus-synchronized discharges at long interstimulus intervals but few responses at short interclick intervals. This population of neurons can thus represent slowly occurring temporal events explicitly using a temporal code. The representation of interclick intervals by the synchronized population is therefore isomorphic because it is a faithful replica of a stimulus parameter. The nonsynchronized population of neurons did not exhibit stimulus-synchronized discharges at either long or short interclick intervals. This population of neurons can implicitly represent rapidly changing interclick intervals by their average discharge rates. The representation by the nonsynchronized population is nonisomorphic because 538 Wang

13 it has transformed a stimulus dimension into an internal representation. The overlap between the encoding domains of these two populations of neurons allows the auditory cortex to represent a wide range of repetition rates (Figure 4b). Neural responses to click trains observed in A1 of awake marmosets (Lu et al. 21) differ fundamentally from those observed in anesthetized animals ( Joris et al. 24). The most crucial difference between the two experimental conditions lies in the nonsynchronized responses that have been observed only in awake animals (Figure 4a). Nonsynchronized responses to repetitive stimuli have also been observed in A1 of species other than marmosets in awake conditions (cats: Dong et al. 211; rats: Gao & Wehr 215) and appear to be a common property of A1 across species. We discuss in the sections below the functional implications of the nonsynchronized responses in A1. b Interclick interval (ms) a Interclick Interval (ms) Synchronized responses ,5 Vector strength Time (ms) Interclick interval (ms) Subthreshold.5 1. nsync+ Time (s) Spike.5 1. Rayleigh statistic Interclick Interval (ms) Nonsynchronized responses ,5 Discharge rate (sp/s) Time (ms) Interclick interval (ms) Subthreshold nsync Spike c Normalized firing rate MGB: VS (n = 56) MGB: Rate (n = 35) A1: VS (n = 36) A1: Rate (n = 5) Interclick interval (ms) d Proportion of units Synchronized Mixed Nonsynchronized IC (n = 47) from Batra et al. (1989) MGB (n = 92) from Bartlett & Wang (27) A1 (n = 94) from Lu et al. (21) Temporal response type Vector strength (VS) (Caption appears on following page) Cortical Coding of Auditory Features 539

14 Figure 4 (Figure appears on preceding page) (a) Two distinct types of cortical responses to periodic click trains. (Left) An example of stimulus-synchronized responses to click trains recorded from A1 of awake marmosets. (Top) Dot raster plot of synchronized responses. The horizontal bar below the x-axis indicates the stimulus duration (1, ms). (Bottom) Vector strengths (dashed line) and Rayleigh statistics (solid line) analyzed for the stimulus-synchronized responses shown in the top plot. The dotted line (at the Rayleigh statistics of 13.8) indicates the threshold for statistically significant stimulus-synchronized activity ( p <.1). The arrow indicates the calculated synchronization boundary. (Right) An example of nonsynchronized responses to click trains recorded from A1 of awake marmosets. (Top) Dot raster plot of nonsynchronized responses. (Bottom) Driven discharge rate is plotted against the interclick interval for the nonsynchronized responses shown in the top plot. Vertical bars represent SEM. The arrow indicates the calculated rate-response boundary. Modified from Lu et al. (21). (b) Examples of subthreshold and spiking responses of positive monotonic (nsync+) (left) and negative monotonic (nsync ) (right) neurons to click trains recorded intracellularly in A1 of awake marmosets. Stimulus duration is indicated by the gray-shaded area in all plots. Modified from Gao et al. (216). (c) Comparison of temporal response properties of A1 and MGB. Stimulus-synchronized firings are quantified by the VS (A1, thin, green dashed curve; MGB,thick, green solid curve, including both synchronized and mixed neurons). Nonsynchronized firings are quantified by the normalized firing rate (A1, thin, blue dashed curve; MGB, thick, blue solid curve). Error bars represent SEM. Modified from Bartlett & Wang (27). (d ) Comparison of synchronized and nonsynchronized responses of the auditory cortex and subcortical regions. MGB mediates the transformation from a synchronized representation in IC to a combination of synchronized and nonsynchronized representations in A1. The proportion of neurons in each auditory region (IC, purple bars; MGB,green bars; A1, blue bars) is plotted for each temporal response type (synchronized, mixed, and nonsynchronized). IC data are from Batra et al. (1989). MGB data are from Bartlett & Wang (27). A1 data are from Lu et al. (21). The total number of neurons from each region is shown in parentheses. Modified from Bartlett & Wang (27). Abbreviations: A1, primary auditory cortex; IC, inferior colliculus; MGB, medial geniculate body; SEM, standard error of the mean; VS, vector strength. Positive and Negative Monotonic Rate Tuning The study by Bendor & Wang (27) showed that auditory cortex neurons could respond with unsynchronized responses to decreasing or increasing interclick intervals (or equivalently to increasing or decreasing repetition rates) with monotonically increasing firing rates. These two types of neurons were called positive monotonic and negative monotonic neurons, respectively (Bendor & Wang 27). These two types of unsynchronized responses were later observed again in an intracellular recording study of awake marmosets (Gao et al. 216). Figure 4b shows two examples of these two types of neurons with both spiking and subthreshold responses. Positive monotonic neurons showed a similar trend as the nonsynchronized neurons reported by Lu et al (21) [i.e., increasing firing rate with decreasing interclick interval (or increasing repetition rate)]. Negative monotonic neurons, by contrast, showed the opposite trend as the positive monotonic neurons. These observations suggest that firing rate based coding mechanisms play a much greater role in representing time-varying sounds than previously thought, even within the interclick interval range where stimulus synchronization can occur in cortical neurons. The properties of the positive and negative monotonic neurons are reminiscent of similar responses observed in the somatosensory cortex of awake and behaving macaque monkeys (Romo & Salinas 23, Salinas et al. 2). Comparison of Temporal and Rate Representations at the Primary Auditory Cortex and the Medial Geniculate Body Studies of the auditory cortex of awake marmosets demonstrated a shift of neural coding from temporal representations to firing rate based representations (Wang et al. 28). To what extent are the cortical representations different from the preceding stage, the MGB of the thalamus? Bartlett & Wang (27) studied MGB responses to periodic click trains in awake marmosets and found that MGB neurons had much lower synchronization boundaries than A1 neurons (i.e., MGB neurons were able to synchronize to much faster repetition rates than A1 neurons) (Figure 4c).The median synchronization boundary for MGB neurons was 5.2 ms (Bartlett & Wang 27), whereas A1 neurons had a median synchronization boundary of 21.3 ms (Lu et al. 21). By contrast, the boundaries of nonsynchronized and positive monotonic responses in MGB neurons also shifted 54 Wang

15 to shorter interclick intervals compared with that in A1 neurons. The median nonsynchronized rate-response boundary of MGB neurons was 5.7 ms (Bartlett & Wang 27), whereas A1 neurons had a median nonsynchronized rate-response boundary of 12.9 ms (Lu et al. 21). As shown in Figure 4c, the MGB temporal and rate representations are clearly segregated at smaller interclick intervals compared with A1 temporal and rate representations. The data from MGB and A1 of awake marmosets show that there is a greater degree of temporal-to-rate transformation in A1 than in MGB. The data also show that there were clearly nonsynchronized responses in MGB, although only positive monotonic neurons were observed (Bartlett & Wang 27). Little evidence of nonsynchronized responses in the IC exists in the literature. Figure 4d compares the single-neuron data recorded from IC (Batra et al. 1989, rabbits), MGB (Bartlett & Wang 27, marmosets), and A1 (Lu et al. 21, marmosets) in awake animals. The main differences between IC, MGB, and A1 representations of the repetition rate appear to be in the creation of separate synchronized and nonsynchronized responses at MGB and A1 but not at IC. The MGB acts as a transition stage in this transformation with the emergence of nonsynchronized responses. An interesting and puzzling observation is the relatively large proportion of mixed responses in MGB, in which both synchronized and nonsynchronized responses were present in the same neuron at long and short interclick intervals, respectively (Bartlett & Wang 27). In contrast, only a small proportion of A1 neurons were found to have mixed responses (Figure 4d). It remains unclear whether the nonsynchronized and mixed responses observed in MGB are the result of a corticothalamic feedback loop or are generated within MGB, given that they do not appear to exist in IC according to previous studies of unanesthetized animals (e.g., Batra et al. 1989). These important issues require further studies of awake animals in which corticothalamic feedback circuits are experimentally manipulated. Functional Implications of Temporal-to-Rate Transformation in the Auditory Cortex The prevalence of rate-coding neurons in the auditory cortex has important functional implications. It shows that considerable temporal-to-rate transformations have taken place by the time auditory signals reach the auditory cortex. The importance of the unsynchronized neural responses is that they represent processed instead of preserved temporal information. It suggests that cortical processing of sound streams operates on a segment-by-segment basis rather than on a momentby-moment basis as found in the periphery auditory system (Wang 27). Segment-by-segment processing is necessary for complex integration to take place at this level of the auditory system, because higher-level processing tasks require temporal integration over a time window preceding and following a particular time of interest. The reduction in the temporal limit on stimulussynchronized discharges in the auditory cortex is a prerequisite for multisensory integration in the cerebral cortex. Auditory information is encoded at the periphery auditory system at a much higher temporal modulation rate than visual or tactile information, but discharge synchrony rates are similar across primary sensory cortical areas. The slowdown of the temporal response rate along the ascending auditory pathway and the accompanying temporal-to-rate transformation are necessary for rapid auditory information and information from other sensory modalities processed at slower rates to be integrated in the cerebral cortex. Whereas we have emphasized the importance of firing rate based codes in this article, several studies have suggested roles for spike timing in information coding in the auditory cortex under various physiological and behavioral conditions. Ter-Mikaelian et al. (27) showed that anesthesia can in fact significantly increase the temporal precision of A1 neurons to both tones and amplitudemodulated stimuli, suggesting that the role of spike timing must be considered in the absence Cortical Coding of Auditory Features 541

16 of anesthesia. Lu & Wang (24) showed that the spike timing on the occurrence of acoustic events was more precise at the first event than at successive events and more precise at sparsely distributed events (longer time intervals between events) than at densely packed events. Therefore, how auditory thalamus and cortex neurons encode time-varying signals is not simply an issue of whether spike timing is important, but when it is important (i.e., under what stimulus conditions and in which population of neurons). REPRESENTATIONS OF PITCH AND HARMONICS Harmonically Related Frequency Tuning in the Auditory Cortex A hallmark of neurons throughout the ascending auditory systems is frequency tuning. The auditory nerve fiber is tuned to only a single frequency (Kiang et al. 1967). Beginning from the cochlear nucleus, some neurons show a secondary frequency tuning in addition to BF (Marsh et al. 26). The prevalence of the neurons tuned to more than one frequency increases along the ascending auditory pathway. Although most neurons in A1 appear to be tuned to a single frequency, a significant proportion of neurons have multipeak frequency tuning. Multipeak cortical neurons are found in various mammalian species, including bats (Fitzpatrick et al. 1993, Suga et al. 1983), cats (Abeles & Goldstein 197, Phillips & Irvine 1981, Sutter & Schreiner 1991), and nonhuman primates (marmoset: Aitkin & Park 1993, Kadia & Wang 23, Sadagopan & Wang 29; macaques: Rauschecker et al. 1997). When a neuron is tuned to more than one frequency, the relationship between these frequencies is sometimes harmonic. In echolocating bats, combination-sensitive neurons outside A1 usually show multiple excitatory peaks at harmonics of BF (e.g., 2BF, 3BF, 4BF), corresponding to harmonically related spectral components of ultrasonic calls emitted by the bats (Suga et al. 1983). Some A1 neurons in echolocating bats show multipeak frequency tuning that is not at harmonics of BF (Kanwal et al. 1999). In the marmoset, approximately 2% of A1 neurons sampled under awake conditions have multipeak frequency tuning (Kadia & Wang 23). Similar to bat echolocating ultrasonic calls, the spectral components of marmoset vocalizations typically contain multiples of a fundamental frequency, which is between 4 and 8 khz for marmosets (Agamaite et al. 215) and 3 khz for mustache bats (Suga 1989). In some cases, the frequencies of the excitatory peaks of the multipeak neurons found in marmoset A1 fall into the frequency range of marmoset vocalizations ( 4 16 khz) (Agamaite et al. 215), but in many cases harmonically related excitatory peaks are outside the frequency range of marmoset vocalizations (Kadia & Wang 23). In mammalian species other than echolocating bats, the proportion of multipeak neurons is relatively small (less than 2% of A1 neurons by some estimates; Kadia & Wang 23, Sutter & Schreiner 1991), of which only a portion shows harmonically related excitatory peaks. Approximately 8% of A1 neurons in marmosets are considered single-peak neurons when tested by a single pure tone; that is, they exhibited frequency tuning to a BF. However, when these neurons were tested by two tones presented simultaneously (with one tone placed at a neuron s BF), a significant proportion of the single-peak neurons showed facilitation or inhibition in their two-tone responses (Kadia & Wang 23), suggesting that the neurons receive inputs at frequencies other than their BFs. Using the two-tone paradigm, Kadia & Wang (23) found that many single-peak neurons in marmoset A1 exhibited harmonically related facilitation. The two-tone combination can also generate inhibitory responses, some of which are harmonically related. This type of inhibition is called distant inhibition because it is distinctly different from the sideband inhibition flanking the excitatory region near BF (Schreiner et al. 2). Distant inhibition is interesting and important because it is often harmonically related to BF. A larger proportion of single-peak neurons in 542 Wang

17 marmoset A1 show harmonically related distant inhibition than facilitation. These observations suggest that A1 neurons receive harmonically related excitatory and inhibitory inputs. Neurons with harmonically related facilitatory frequency peaks can function to extract harmonic components embedded in complex sounds as a unitary object. Kadia & Wang (23) showed that harmonically related multipeak neurons exhibited nonlinear two-tone facilitation. The harmonically related distant inhibition, in contrast, could serve as a mechanism to remove unwanted harmonic artifacts in a natural environment. It is also conceivable that, upon combination of harmonically related facilitation and inhibition, the auditory cortex could determine whether a sound is harmonic, a function that is important for many auditory perceptions. As we discuss below, the recently discovered harmonic template neurons in the marmoset auditory cortex may serve such functions. Harmonic Template Neurons of the Auditory Cortex The auditory cortex of humans and nonhuman primates contains a core region comprising A1, R, and rostral-temporal area (RT). The core region is surrounded by belt and para-belt regions (Bendor & Wang 28, Hackett et al. 21, Kaas & Hackett 2). The core region of the auditory cortex in marmosets contains a unique class of harmonic template neurons that do not respond or respond only weakly to pure tones or two-tone combinations but that respond strongly to particular combinations of multiple harmonics (harmonic templates), as shown in Figure 5a,b (Feng & Wang 217). This neuron responded weakly to pure tones near 7.49 khz (determined as BF). When the BF tone was presented simultaneously with a second tone at varying frequency ( f 2 ), this neuron s response was maximally enhanced at two f 2 frequencies of 6.28 and 8.57 khz, respectively. The response of this neuron was further enhanced when harmonic complex tones (HCTs) with varying fundamental frequency ( f ) values were presented. At some f values (e.g., 1.5, 1.26 khz), the responses were much stronger than the maximal responses to pure tones or two-tone stimuli and, noticeably, became sustained through stimulus duration, which is consistent with the notion that sustained firing in the auditory cortex is evoked by the preferred stimulus (or stimuli) of a neuron (Wang et al. 25). A close examination shows that the HCTs at f of 1.5 and 1.26 khz contain both the three spectral components that aligned respectively with the BF (7.49 khz) and the two facilitatory peaks of the two-tone response profile (6.28, 8.57 khz). The HCT at f of 1.26 khz is slightly misaligned from the HCT at f of 1.5 khz, resulting in a smaller firing rate. These observations suggest that these neurons are tuned to particular harmonic templates. Responses of the harmonic template neurons show nonlinear facilitation to harmonic complex sounds over inharmonic sounds and selectivity for particular harmonic structures. The harmonic template neurons are distributed across A1 and R and have BFs ranging from 1 to 32 khz, which covers the entire hearing range of marmosets (Figure 5c). The harmonic template neurons found in the auditory cortex can represent combinations of multiple harmonics and may play an important role in processing sounds with harmonic structures such as animal vocalizations, human speech, and music. The broad distribution of the harmonic template neurons suggests that the primate auditory cortex utilizes a generalized harmonic processing organization across a species entire hearing frequency range to process sounds rich in harmonics, including not only species-specific vocalizations but also sounds produced by other species or devices such as musical instruments. Pitch Processing by the Auditory Cortex Pitch perception is crucial for speech and music perception and auditory object recognition in a complex acoustic environment. Pitch is the percept that allows sounds to be ordered on a Cortical Coding of Auditory Features 543

18 musical scale, and is closely associated with the perception of harmonically structured or periodic sounds (Plack et al. 25). Our auditory system relies on pitch, among other attributes of sounds, to accomplish such tasks as identifying a particular voice at a cocktail party. An important phenomenon of pitch perception is the percept of the missing fundamental, also called the residue pitch (Oxenham 212). When the harmonics of a fundamental frequency are played together, the pitch is still perceived as matching the fundamental frequency even if the fundamental frequency component is missing. Missing fundamental perception plays an important part in the invariance of pitch perception, allowing for the same pitch to be perceived for many different combinations of harmonics. The ability to perceive pitch is not unique to humans; it has been shown in several animal species, including birds (Cynx & Shapiro 1986), cats (Heffner & Whitfield 1976, Whitfield 198), and monkeys (Tomlinson & Schwarz 1988). Recent studies have shown that marmosets, whose hearing range is similar to that of humans (Osmanski & Wang 211), exhibit humans-like a Fundamental frequency (khz) Harmonic composition d e M R C L RT f HCTs f = 1.26 khz f = 1.5 khz Time (ms) R 1 mm AI 5 1,5 2,5 Frequency (Hz) b Rate (spikes/s) Fundamental frequency (khz) BF f 2 (khz) Spectra of HCTs Frequency (khz) BF (khz) Recording site (519 total units) Pitch neuron site (19 pitch units) ,2 Time (ms) f (khz) f Pure tone Two tone (7.49 khz + f2) Response to 7.49 khz 1 2 Rate (spikes/s) M R C L Insula f = 1.26 khz Marmoset Insula STG Retracted edges of the lateral sulcus f = 1.5 khz Core 5 mm LS (retracted) HH H L RT R LL Belt A1 Parabelt STG Pitch center R M STS ITG L C ITG c LS A1 g R M C R L Harmonic template neurons Human M A P L Insula Retracted edges of the lateral sulcus SI Belt HS astg L R H HG L STG HG SI HS PT FTS A1 Pitch center PT HG 1cm H CS mm ITG A L M P BF (khz) (Caption appears on following page) 544 Wang