Analysis of sensory coding with complex stimuli Jonathan Touryan* and Yang Dan

Transcription

1 443 Analysis of sensory coding with complex stimuli Jonathan Touryan* and Yang Dan Most high-level sensory neurons have complex, nonlinear response properties; a comprehensive characterization of these properties remains a formidable challenge. Recent studies using complex sensory stimuli combined with linear and nonlinear analyses have provided new insights into the neuronal response properties in various sensory circuits. Addresses *Group in Vision Science, School of Optometry, University of California, Berkeley, CA 94720, USA Division of Neurobiology, Department of Molecular and Cell Biology, University of California, Berkeley, CA 94720, USA; ydan@uclink4.berkeley.edu Correspondence: Yang Dan Current Opinion in Neurobiology 2001, 11: /01/$ see front matter 2001 Elsevier Science Ltd. All rights reserved. Abbreviations ncrf non-classical receptive field RF receptive field STRF spatiotemporal receptive field (visual or somatosensory) spectrotemporal receptive field (auditory) SVD singular value decomposition Introduction One of the major challenges in studying the function of a sensory circuit is to understand what features of the stimuli are encoded in the spiking of its neurons [1]. Traditionally, sensory neurons have been studied with small sets of simple stimuli specifically designed to probe certain aspects of their response properties. For example, the receptive fields of retinal ganglion cells can be mapped with small spots of light at different locations [2 4], and the frequency tuning of auditory neurons can be measured with pure tones at different frequencies [5]. Although this approach has provided insights into the functions of the early sensory neurons, it has limited applicability to neurons in higher sensory areas. Many high-level sensory neurons are selectively driven by unknown complex features [6], and simple stimuli that do not contain the relevant features will not be effective in probing their response properties. In addition, most of these neurons exhibit highly nonlinear stimulus response relationships. Studies using simple stimuli may not reveal nonlinearities in the responses to conjunctions of features. An alternative approach that may help to alleviate these problems is to use large ensembles of complex stimuli to probe the stimulus response relationship. By using large stimulus ensembles, the experimenter reduces a priori assumptions about which sensory features are relevant to the neuron, thus allowing a less biased characterization of the response properties. The use of complex stimuli allows one to probe certain aspects of the nonlinearity that cannot be detected in studies using simple stimuli. The distinction between simple and complex stimuli is somewhat arbitrary, and it is difficult to give a rigorous definition that will allow unambiguous categorization of the wide range of stimuli that have been used by sensory physiologists. Here, we use the term complex stimuli to refer to stimuli that contain rich structures in the stimulus domain(s) of interest, such as spatial position, orientation and frequency, within the integration time window of the neuron (Figure 1). Note that by this definition the measure of complexity depends on which domain is under investigation. For example, when the domain of interest is spatiotemporal frequency, a sinusoidal grating drifting at a constant velocity is considered a simple stimulus even though the luminance signal varies continuously in space and time (Figure 1b). By contrast, a superposition of multiple gratings [7,8] is considered more complex. The use of superposed multiple Fourier components in studying sensory neurons has been thoroughly reviewed recently [9 ]. Here we will focus on two types of complex stimuli: timevarying random stimuli, including, but not restricted to, white noise; and natural stimuli, which are loosely defined as the stimuli that are behaviorally relevant and are found within the animal s natural sensory environment. Linear analysis of responses to complex stimuli Random stimuli and reverse correlation The simplest method for analyzing the responses to timevarying random stimuli is reverse correlation [10], or spike-triggered average for spiking neurons. With unbiased random stimuli, this method computes the best linear fit of the stimulus response transformation of the neuron, such as the spatiotemporal receptive fields (STRFs) of visual neurons [11] or the spectrotemporal receptive fields (STRFs) of auditory neurons [12]. An important advantage of this approach is that the resulting receptive fields (RFs) often exhibit richer structures than those obtained from responses to simple stimuli. For example, to study processing of sound features in the awake primate auditory cortex, decharms, Blake and Merzenick [13] measured STRFs of the neurons using rapidly presented random chords and asynchronous random tone progressions, both of which approximate white noise in the spectrotemporal domain. They found that many STRFs contained structures ideal for detecting spectrotemporal edges or spectral motion, resembling the STRFs of V1 neurons selective for luminance-defined stationery or moving edges. Such auditory feature selectivity was not detected with the standard pure-tone stimuli because they do not contain the relevant spectrotemporal features. A more recent example illustrating the advantage of using white-noise stimuli has come from a series of studies in the somatosensory cortex (area 3b) of the awake monkey

2 444 Sensory systems Figure 1 (a) Complex stimuli Simple stimuli (b) (c) (d) Space Spatiotemporal frequency Orientation touching the fingerpad of the animal, and the RFs were computed using a modified reverse correlation method (see section below). The resulting RFs had distinct excitatory and inhibitory subregions, which were also more complex than most of the RFs measured with the simple, punctate stimuli used in previous studies [17]. Thus, in both these studies [14,15,16 ] and those of decharms [13], the rich structure of white-noise stimuli has proved advantageous for revealing the spatiotemporal or spectrotemporal feature selectivity of cortical neurons. In addition to white-noise stimuli that sample an entire stimulus space, reverse correlation can also be applied to random stimuli drawn from a restricted stimulus set. In a series of elegant experiments, Ringach et al. [18 20] presented sinusoidal gratings flashed at a random sequence of orientations and spatial frequencies, and applied reverse correlation to measure the dynamic tuning of V1 neurons. These studies revealed complex temporal dynamics of orientation and spatial-frequency tuning that had eluded previous studies using the conventional drifting gratings. A key feature of these experiments is that the stimuli were randomly drawn from a set of oriented gratings. Other stimuli, such as white-noise stimuli or random stimuli in the spatial domain (e.g., sparse noise, in which the position of a luminance spot jumps randomly over time [11]), generally do not evoke robust responses from the orientation-selective V1 cells. More importantly, for the majority of cortical cells (complex cells) the relationship between the neuronal firing rate and the light signals in the spatial domain is highly nonlinear [21,22]; reverse correlation in the spatial domain would reveal very limited information. In this study, choosing the proper stimulus set markedly increased the efficiency for driving the cells, and reverse correlation with the corresponding stimulus parameters (orientation and spatial frequency) revealed interesting response dynamics. A similar technique has been applied to study the temporal dynamics of chromatic tuning of primate V1 neurons [23]. Frequency Current Opinion in Neurobiology Examples of simple and complex stimuli. Simple stimuli: (a) light spot and (b) drifting grating. Complex stimuli: (c) random orientation and (d) natural sound. The left column illustrates the stimuli, whereas the right column shows the representation of each stimulus along the stimulus domain of interest and over time. The length of the time axis shown on each plot represents the memory of the system (the duration over which the stimulus affects the response of the cell). The two types of simple stimuli shown here can each be approximated as a delta function in the parameter domain, which remains constant over time. By contrast, the complex stimuli have either richer structures along the parameter domain (natural sound), or simple structures that change in a complex manner over time (random orientation). [14,15,16 ]. In these studies, the white-noise stimuli were implemented as random raised dots on a rotating drum Another recent example involving restricted random stimuli comes from Walker, Ohzawa and Freeman [24], who have investigated the spatial organization of the non-classical receptive fields (ncrfs) of V1 neurons. While an optimal drifting grating was presented in the classical RF, a second stationary grating patch at the optimal spatial frequency and orientation was randomly flashed at each of the 144 positions, which uniformly covered both the classical RF and ncrf. When reverse correlation was applied to analyze the relationship between the response and the positions of the flashed grating patch, it revealed much more complex spatiotemporal structures of the ncrfs than previously known. In this example the use of optimal gratings in the second patch was crucial, because the effect of a less optimal stimulus may have been too weak to detect. In general, random stimuli drawn from a properly designed stimulus set can be much more effective than white-noise

3 Analysis of sensory coding with complex stimuli Touryan and Dan 445 stimuli in probing particular aspects of the neuronal response properties, because they reflect certain prior knowledge of what types of stimuli are relevant to the neuron. Compared with the conventional simple stimuli, the richer temporal structures of these random stimuli allow them to probe more complex response properties. Natural stimuli and modified reverse correlation Natural stimuli are another type of complex stimuli that have been used for the analysis of sensory coding [25 27,28 ]. Because sensory circuits evolve and develop in the natural environment, they may be specifically tuned for efficient coding of natural stimuli [1,29 34]. For studying the neuronal response properties, it seems reasonable to assume that natural stimuli provide a rich source of relevant sensory features. For a linear neuron, the STRF mapped with natural stimuli should be the same as that mapped with random stimuli sampling the same stimulus space, provided that enough data are available to average out the noise. But most high-level sensory neurons exhibit various forms of nonlinearity, therefore the STRF mapped with natural stimuli can be significantly different from that mapped with random stimuli. To estimate STRF with natural stimuli, however, the reverse correlation method must be modified to remove the stimulus-derived correlation in the result [12,35]; the cross-correlation between the stimuli and the responses must be normalized by the auto-correlation of the natural stimuli (also see [15]). In practice, the auto-correlation matrix of natural stimuli may not be invertible, and noise in the estimated cross-correlation can be greatly amplified by the normalization procedure. Therefore, special care must be taken in the data analysis. An excellent example illustrating the use of natural stimuli to characterize the neuronal response properties comes from a recent study of the auditory forebrain of the zebra finch [36 ]. Theunissen, Sen and Doupe [36 ] used two types of auditory stimuli to estimate the STRFs of the neurons: an ensemble of conspecific bird songs, which are behaviorally relevant to the bird, and a sequence of random tone pips with the same average power spectrum as the songs. They found that the songs were much better than tone pips for driving these high-level auditory neurons. More importantly, the STRFs estimated from the responses to bird songs exhibited structures strikingly different from those measured with the tone pips. Overall, the song-based STRFs are much better models for predicting the neuronal responses. Here, the difference between the two estimates of the STRF clearly indicates nonlinearity in these neurons, and the superior performance of the song-based STRFs attests to the advantage of using natural stimuli in studying the stimulus-response functions. In addition to these auditory neurons, natural stimuli have also been used to characterize the response properties of visual neurons [37]. In this experiment, the monkey freely viewed image patches sampled from natural scenes while the eye positions were monitored and the images impinging on the RF of the cell were deducted. A reverse correlation method similarly modified to correct for the correlation in the stimulus ensemble was used to compute the STRFof both striate and extrastriate cortical neurons. Studies with such an approach may potentially reveal novel feature selectivity of extrastriate cortical neurons, which has been notoriously difficult to study with the conventional approach using simple stimuli. Nonlinear analysis techniques An immediate extension of the linear analysis is to incorporate the second-order nonlinearity, which can be studied by measuring the responses to two-element stimuli [22,38] (which will not be discussed further in this review), or by performing the second-order Wiener (or Volterra) kernel analysis of the responses to white noise [39 41]. A secondorder kernel usually consists of a large two-dimensional matrix of parameters (for example, if the first-order kernel consists of 100 parameters, the second-order kernel will consist of /2 parameters), which can be difficult to interpret in terms of the encoding mechanism. In a recent study of the non-phase-locking auditory neurons in the American bullfrog, however, Yamada and Lewis [42] reduced the dimensionality of the problem by performing a singular value decomposition (SVD) of the second-order Wiener kernel. They found that for each neuron, typically one or a few pairs of the most significant singular vectors carry most of the information in the second-order Wiener kernel. The reduced kernel, which was based on a small number of these singular vectors, well predicted the responses to novel test stimuli. Interestingly, each pair of these vectors represent two band-pass filters of the acoustic waveform that are almost identical in amplitude and shape, but are phase shifted by approximately a quarter of a cycle, thus forming a quadrature pair. These observations lead to a simple cascade model, each stage of which corresponds to a known biophysical mechanism of these auditory units. Here, the application of SVD greatly simplified the second-order Wiener kernel and made it much more interpretable. Another method for studying the stimulus response relationship of a sensory neuron is to characterize the probability distribution of the stimuli that evoke a certain type of response (e.g. a spike) [43]. In this framework, each stimulus waveform is represented as a vector in a multidimensional parameter space. For example, imagine a hypothetical neuron whose response depends on the light signals in two pixels (the temporal dimension is ignored for simplicity). Each stimulus waveform for this neuron can be represented as a point in a two-dimensional space, with each dimension representing the luminance value at each of the two pixels (Figure 2a). The whole stimulus ensemble used in the experiment is

4 446 Sensory systems Figure 2 (a) (c) Representation in stimulus space S2 S1 S2 S1 Stimulus sequence (b) (d) Prior distribution S2 Response-conditional distribution S2 S1 S1 Diagram showing a method used to analyze the response-conditional distribution of the stimuli. (a) Representation of the stimulus in the multi-dimensional parameter space. Here, each stimulus waveform consists of light signals in two pixels, S1 and S2. Shown on the plot are three example waveforms, each represented as a single point in the twodimensional space. (b) Distribution of the whole stimulus ensemble. The two arrows represent the eigenvectors of the covariance matrix, and their lengths represent the square roots of the corresponding eigenvalues. For Gaussian white noise, the two eigenvalues are equal. (c) Stimulus sequence and the response of a model nonlinear cell, which has a spiking probability that is proportional to (S1 + S2) 2. As a result, it responds maximally when the two pixels are both bright or both dark, but minimally when they have the opposite polarities. (d) Response-conditional distribution of the stimulus for this cell. The arrows represent the eigenvectors of the response-conditional covariance matrix. In this case, the eigenvalue of one of the vectors (upper right quadrant) is larger, as the variance along this dimension has increased owing to the feature selectivity of the cell. Spike train Current Opinion in Neurobiology distributed in this space with a density function determined by the statistics of the stimuli. In the case of Gaussian white noise, this distribution is circularly symmetric around zero (Figure 2b). If the neuron is selectively driven by a certain feature of the stimulus (Figure 2c), the response-conditional ensemble (the ensemble of all the waveforms that evoke spiking) should have a different probability distribution from the whole stimulus ensemble (Figure 2d). The task of studying the stimulus response relationship is then to compare the difference between the probability distributions of the whole stimulus ensemble and the response-conditional ensembles. Of course, comparing probability distributions in a multi-dimensional space can be difficult. One method to reduce the dimensionality is to look for dimensions along which there is an obvious change in variance, as stimulus features represented by these dimensions are likely to be relevant to the response of the neuron. This is achieved by identifying the eigenvectors of the response-conditional covariance matrix that have different eigenvalues from the covariance matrix of the whole stimulus ensemble. This technique has been used recently by Brenner, Bialek and de Ruyter van Steveninck [44 ] to study the motion sensitive H1 neurons in the fly lobula plate. The authors identified two significant eigenvectors, representing filters measuring the velocity and the acceleration of the stimulus. Identification of a small number of relevant stimulus features then allowed a detailed study of the relationship between the presence of these features in the stimuli and the response of the neuron, and how such a relationship was dynamically adapted to maximize information transmission by the system. Interestingly, although this dimensionality reduction technique originated from a different theoretical framework, in practice it is similar to applying SVD to the second-order Wiener kernel used by Yamada and Lewis [42]. The type of nonlinear methods mentioned above are readily applicable to analyzing responses to Gaussian white noise, but are much more difficult to use with natural stimuli, because of their complex statistics. An alternative technique is to use artificial neural networks and supervised learning algorithms to model the responses of sensory neurons [45,46]. As this technique does not require input with specific statistics, it can be easily applied to analyze responses to natural stimuli. A comprehensive characterization of the response properties of sensory neurons, especially under natural stimulation, will provide crucial insights into the role of each neural circuit in sensory processing. Conclusions The studies described above have demonstrated the advantage of using large ensembles of complex stimuli in

5 Analysis of sensory coding with complex stimuli Touryan and Dan 447 studying sensory neurons. Despite the prevalence of nonlinearity in the responses of high-level sensory neurons, linear analyses used in conjunction with proper stimulus ensembles can still reveal new response properties of these neurons. In addition, nonlinear methods are being used to identify sensory features that contribute to the neuronal responses. In many cases, surprising insights were gained by simply adapting an existing analytical method to study a different dimension of the system, or a different sensory modality. New breakthroughs in understanding sensory coding by high-level neurons are likely to come from studies using properly designed stimulus ensembles, especially those bearing perceptual relevance in the natural environment, coupled with robust analysis techniques that are reasonably tolerant to imperfection of the data such as noise and nonstationarity. Acknowledgements We thank Brian Lau and Frederic Theunissen for helpful comments on the manuscript. This work is supported by grants from the National Eye Institute (R01 EY ) and the Office of Naval Research (N ). References and recommended reading Papers of particular interest, published within the annual period of review, have been highlighted as: of special interest of outstanding interest 1. Barlow HB: Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1972, 1: Hartline HK: The receptive fields of optic nerve fibers. Am J Physiol 1940, 130: Kuffler SW: Discharge patterns and functional organization of mammalian retina. J Neurophysiol 1953, 16: Barlow HB: Summation and inhibition in the frog s retina. J Physiol (Lond) 1953, 119: Liberman MC: The cochlear frequency map for the cat: labeling auditory-nerve fibers of known characteristic frequency. J Acoust Soc Am 1982, 72: Tanaka K: Inferotemporal cortex and object vision. Annu Rev Neurosci 1996, 19: Bishop PO, Coombs JS, Henry GH: Receptive fields of simple cells in the cat striate cortex. J Physiol (Lond) 1973, 231: Victor JD, Shapley RM, Knight BW: Nonlinear analysis of cat retinal ganglion cells in the frequency domain. Proc Natl Acad Sci USA 1977, 74: Klein DJ, Depireux DA, Simon JZ, Shamma SA: Robust spectrotemporal reverse correlation for the auditory system: optimizing stimulus design. J Comput Neurosci 2000, 9: This paper first provides a clear overview of the various methods for characterizing the input output transformation of auditory neurons. It convincingly advocated and thoroughly developed the use of sum-of-ripples, a type of stimuli consisting of the superposition of multiple Fourier components in the spectrotemporal domain. 10. de Boer E, Kuyper P: Triggered correlation. IEEE Trans Biomed Eng 1968, 15: Jones JP, Palmer LA: The two-dimensional spatial structure of simple receptive fields in cat striate cortex. J Neurophysiol 1987, 58: Aertsen AM, Johannesma PI: The spectro-temporal receptive field. A functional characteristic of auditory neurons. Biol Cybern 1981, 42: decharms RC, Blake DT, Merzenich MM: Optimizing sound features for cortical neurons. Science 1998, 280: DiCarlo JJ, Johnson KO, Hsiao SS: Structure of receptive fields in area 3b of primary somatosensory cortex in the alert monkey. J Neurosci 1998, 18: DiCarlo JJ, Johnson KO: Velocity invariance of receptive field structure in somatosensory cortical area 3b of the alert monkey. J Neurosci 1999, 19: DiCarlo JJ, Johnson KO: Spatial and temporal structure of receptive fields in primate somatosensory area 3b: effects of stimulus scanning direction and orientation. J Neurosci 2000, 20: The last of the three papers [14,15,16 ] on RF structures in the somatosensory area 3b, examining the dependence of RF structures on scanning direction. A three-component model was constructed to account for the observed spatiotemporal characteristics of the RFs. 17. Mountcastle VB, Powell TP: Neural mechanisms subserving cutaneous sensibility, with special reference to the role of afferent inhibition in sensory perception and discrimination. Bull Johns Hopkins Hosp 1959, 105: Ringach DL, Hawken MJ, Shapley R: Dynamics of orientation tuning in macaque primary visual cortex. Nature 1997, 387: Ringach DL, Sapiro G, Shapley R: A subspace reverse-correlation technique for the study of visual neurons. Vision Res 1997, 37: Bredfeldt CE, Ringach DL: Dynamics of V1 neurons in the Fourier domain. Soc Neurosci Abstr 2000, 26: Hubel DH, Wiesel TN: Receptive fields, binocular interaction and functional architecture in the cat s visual cortex. J Physiol (Lond) 1962, 160: Movshon JA, Thompson ID, Tolhurst DJ: Receptive field organization of complex cells in the cat s striate cortex. J Physiol (Lond) 1978, 283: Cottaris NP, De Valois RL: Temporal dynamics of chromatic tuning in macaque primary visual cortex. Nature 1998, 395: Walker GA, Ohzawa I, Freeman RD: Asymmetric suppression outside the classical receptive field of the visual cortex. J Neurosci 1999, 19: Passaglia C, Dodge F, Herzog E, Jackson S, Barlow R: Deciphering a neural code for vision. Proc Natl Acad Sci USA 1997, 94: Gallant JL, Connor CE, Van Essen DC: Neural activity in areas V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing. NeuroReport 1998, 9: Stanley GB, Li FF, Dan Y: Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J Neurosci 1999, 19: Vinje WE, Gallant JL: Sparse coding and decorrelation in primary visual cortex during natural vision. Science 2000, 287: The authors show that in awake, behaving monkeys stimulation of the nonclassical receptive fields of V1 neurons increases the sparseness of the responses and decorrelates between neuron pairs. This effect was observed for both natural stimuli and random dynamic grating stimuli. 29. Attneave F: Some informational aspects of visual perception. Psychol Rev 1954, 61: Atick JJ: Could information theory provide an ecological theory of sensory processing? Network Comput Neural Systems 1992, 3: Dan Y, Atick JJ, Reid RC: Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J Neurosci 1996, 16: Rieke F, Bodnar DA, Bialek W: Naturalistic stimuli increase the rate and efficiency of information transmission by primary auditory afferents. Proc R Soc Lond B 1995, 262: Olshausen BA, Field DJ: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381: Nelken I, Rotman Y, Bar Yosef O: Responses of auditory-cortex neurons to structural features of natural sounds. Nature 1999, 397:

6 448 Sensory systems 35. Eggermont JJ, Johannesma PM, Aertsen AM: Reverse-correlation methods in auditory research. Q Rev Biophys 1983, 16: Theunissen FE, Sen K, Doupe AJ: Spectral-temporal receptive fields of nonlinear auditory neurons obtained using natural sounds. J Neurosci 2000, 20: A clear demonstration of the advantage of using natural stimuli for mapping STRFs of high-level sensory neurons, and an excellent source of useful tricks to deal with the practical problems in data analyses. 37. Mazer JA, David SV, Gallant JL: Spatiotemporal receptive field estimation during free viewing visual search in macaque striate and extrastriate cortex. Soc Neurosci Abstr 2000, 26: Rybicki GB, Tracy DM, Pollen DA: Complex cell response depends on interslit spacing. Nature 1972, 240: Gaska JP, Jacobson LD, Chen HW, Pollen DA: Space-time spectra of complex cell filters in the macaque monkey: a comparison of results obtained with pseudowhite noise and grating stimuli. Vis Neurosci 1994, 11: van Dijk P, Wit HP, Segenhout JM, Tubis A: Wiener kernel analysis of inner ear function in the American bullfrog. J Acoust Soc Am 1994, 95: Rotman Y, Bar-Yosef O, Nelken I: Relating cluster and population responses to natural sounds and tonal stimuli in cat primary auditory cortex. Hear Res 2001, 152: Yamada WM, Lewis ER: Predicting the temporal responses of non-phase-locking bullfrog auditory units to complex acoustic waveforms. Hear Res 1999, 130: de Ruyter van Steveninck R, Bialek W: Real-time performance of a movement-sensitive neuron in the blowfly visual system: coding and information transfer in short spike sequences. Proc R Soc Lond B 1988, 234: Brenner N, Bialek W, de Ruyter van Steveninck R: Adaptive rescaling maximizes information transmission. Neuron 2000, 26: The stimulus response relationship of H1 neurons is found to change according to the statistics of the motion stimuli. The authors show that the adaptive rescaling results in an optimal match between the dynamic range of the stimuli and the response property of the neuron to maximize information transmission. A dimensionality reduction technique is used to identify a small number of stimulus features relevant for the neuron s response. 45. Lehky SR, Sejnowski TJ, Desimone R: Predicting responses of nonlinear neurons in monkey striate cortex to complex patterns. J Neurosci 1992, 12: Bankes SC, Margoliash D: Parametric modeling of the temporal dynamics of neuronal responses using connectionist architectures. J Neurophysiol 1993, 69: