Efficient Encoding of Natural Time Varying Images Produces Oriented Space-Time Receptive Fields

Transcription

1 Efficient Encoding of Natural Time Varying Images Produces Oriented Space-Time Receptive Fields Rajesh P. N. Rao and Dana H. Ballard Department of Computer Science University of Rochester Rochester, NY 17 Technical Report 97. National Resource Laboratory for the Study of Brain and Behavior Department of Computer Science, University of Rochester August 1997 Abstract The receptive fields of neurons in the mammalian primary visual cortex are oriented not only in the domain of space, but in most cases, also in the domain of space-time. While the orientation of a receptive field in space determines the selectivity of the neuron to image structures at a particular orientation, a receptive field s orientation in space-time characterizes important additional properties such as velocity and direction selectivity. Previous studies have focused on explaining the spatial receptive field properties of visual neurons by relating them to the statistical structure of static natural images. In this report, we examine the possibility that the distinctive spatiotemporal properties of visual cortical neurons can be understood in terms of a statistically efficient strategy for encoding natural time varying images. We describe an artificial neural network that attempts to accurately reconstruct its spatiotemporal input data while simultaneously reducing the statistical dependencies between its outputs. The network utilizes spatiotemporally summating neurons and learns efficient sparse distributed representations of its spatiotemporal input stream by using recurrent lateral inhibition and a simple threshold nonlinearity for rectification of neural responses. When exposed to natural time varying images, neurons in a simulated network developed localized receptive fields oriented in both space and space-time, similar to the receptive fields of neurons in the primary visual cortex. 1 Introduction Since the seminal experiments of Hubel and Wiesel over 3 years ago[hubel and Wiesel, 19; 19], it has been known that neurons in the mammalian primary visual cortex respond selectively to stimuli such as edges or bars at particular orientations. In many cases, the neurons are directionally selective i.e. they respond only to motion in a particular direction. An especially useful concept in characterizing the response properties of visual neurons has been the notion of a receptive field. The receptive field of a neuron is classically defined as the area of visual space within which stimuli such as bars or edges can elicit responses This research was supported by NIH/PHS research grant 1-P1-RR93. 1

2 from the neuron [Hartline, 19]. Although they are a function of both space and time, early depictions of visual receptive fields were confined to spatial coordinates. In recent years, new mapping techniques have allowed the characterization of receptive fields in both space and time[emerson et al., 197; McLean and Palmer, 199; Shapley et al., 199; DeAngelis et al., 1993a] (see [DeAngelis et al., 199] for a review). The new mapping results indicate that in most cases, the receptive field of a visual neuron changes over time. It has been noted that while the spatial structure of a receptive field indicates neuronal attributes such as preference for a particular orientation of bars or edges, the spatiotemporal structure of a neuron s receptive field governs important dynamical properties such as velocity and direction selectivity[adelson and Bergen, 19; Watson and Ahumada, 19; Burr et al., 19]. In particular, orientation of a neuron s receptive field in space-time indicates the preferred direction of motion while the slope of the oriented subregions gives an estimate of the preferred velocity[mclean and Palmer, 199; Albrecht and Geisler, 1991; Reid et al., 1991; DeAngelis et al., 1993b; McLean et al., 199]. An attractive approach to understanding the receptive field properties of visual neurons is to relate them to the statistical structure of natural images. Motivated by the property that natural images possess a power spectrum [Field, 197], Atick and Redlich [Atick, 199; Atick and Redlich, 199] provided an explanation of the center-surround structure of retinal ganglion receptive fields in terms of whitening or decorrelation of outputs in response to natural images. Several Hebbian learning algorithms for decorrelation have also been proposed [Bienenstock et al., 19; Williams, 19; Barrow, 197; Linsker, 19; Oja, 199; Sanger, 199; Foldiak, 199; Atick and Redlich, 1993], many of which perform Principal Component Analysis (PCA). Although the PCA of natural images produces lower order components that resemble oriented filters [Baddeley and Hancock, 1991; Hancock et al., 199], the higher order components are unlike any known neural receptive field profiles. In addition, the receptive fields obtained are global rather than localized feature detectors. Recently, Olshausen and Field showed that a neural network that includes the additional constraint of maximizing the sparseness of the distribution of output activities develops, when trained on static natural images, synaptic weights with localized, oriented spatial receptive fields [Olshausen and Field, 199] (see also [Harpur and Prager, 199; Rao and Ballard, 1997a] and related work on projection pursuit [Huber, 19] based learning methods [Intrator, 199; Law and Cooper, 199; Shouval, 199]). Similar results have also been obtained using an algorithm that extracts the independent components of a set of static natural images [Bell and Sejnowski, 1997]. These algorithms are all based directly or indirectly on Barlow s principle of redundancy reduction[barlow, 191; 197; 199; 199], where the goal is to learn feature detectors whose outputs are as statistically independent as possible. The underlying motivation is that sensory inputs such as images are generally comprised of a set of independent objects or features whose components are highly correlated. By learning detectors for these independent features, the sensory system can develop accurate internal models of the sensory environment and can efficiently represent external events as sparse conjunctions of independent features. In this paper, we explore the possibility that the distinctive spatiotemporal receptive field properties of visual cortical neurons can be understood in terms of a statistically efficient strategy for encoding natural time varying images [Eckert and Buchsbaum, 1993; Dong and Atick, 199]. We describe an artificial neural network that attempts to accurately reconstruct its spatiotemporal input data while simultaneously reducing the statistical dependencies between its outputs, as advocated by the redundancy reduction principle. Our approach utilizes a spatiotemporal generative model that can be viewed as a simple extension of the spatial generative model used by Harpur and Prager[Harpur and Prager, 199], Olshausen and Field [Olshausen and

3 Field, 199], Rao and Ballard [Rao and Ballard, 1997a], and others. The spatiotemporal generative model allows neurons in the network to perform not just a spatial summation of the current input, but a spatiotemporal summation of both current and past inputs over a finite spatiotemporal extent. The network learns efficient sparse distributed representations of its spatiotemporal input stream by utilizing lateral inhibition[foldiak, 199] and a simple threshold nonlinearity for rectification of neural responses [Lee and Seung, 1997; Hinton and Ghahramani, 1997]. When exposed to natural time varying images, neurons in a simulated network developed localized receptive fields oriented in both space and space-time, similar to the receptive fields of neurons in the primary visual cortex. Spatial Generative Models The idea of spatial generative models has received considerable attention in recent studies pertaining to neural coding [Hinton and Sejnowski, 19; Jordan and Rumelhart, 199; Zemel, 199; Dayan et al., 199; Hinton and Ghahramani, 1997], although the roots of the approach can be traced back to early ideas in control theory such as Wiener filtering[wiener, 199] and Kalman filtering [Kalman, 19]. In this section, we first consider a class of spatial generative models that have previously been used in the neural modeling literature for explaining spatial receptive field properties[harpur and Prager, 199; Olshausen and Field, 199; Rao and Ballard, 1997a]. This will serve to motivate the spatiotemporal models we will be concerned with later. Assume that an image, denoted by a vector I of pixels, can be represented as a linear combination of a set of basis vectors : I (1) The coefficients can be regarded as an internal representation of spatial characteristics of the image I, as interpreted using the internal model defined by the basis vectors. In terms of a neuronal network, the coefficients correspond to the activities or firing rates of neurons while the vectors in the basis matrix correspond to the synaptic weights of neurons. It is convenient to rewrite the above equation in matrix form as: I r () where is the matrix whose columns consist of the basis vectors and r is the vector consisting of coefficients. The goal is to estimate the coefficients r for a given image and, on a longer time scale, learn appropriate basis vectors in. A standard approach is to define a least-squared error criterion of the form:! r" (3) I! r"$# I! r" () where denotes the % th pixel of I and denotes the % th row of. Note that is simply the sum of squared pixel-wise errors between the input I and the image reconstruction r. Estimates for and r can be obtained by minimizing [Williams, 19; Daugman, 19; Pece, 199; Harpur and Prager, 199; Olshausen and Field, 199; Rao and Ballard, 1997a]. 3

4 One can obtain a probabilistic generative model of the image generation process by utilizing a Gaussian noise process n to model the differences between I and r. The resulting stochastic generative model becomes: I r n () If a zero mean Gaussian noise process n with unit covariance is assumed, one can show that is the negative log likelihood of generating the input I (see, for example,[bryson and Ho, 197]). Thus, minimizing is equivalent to maximizing the likelihood of the observed data. Unfortunately, in many cases, minimization of a least-squares optimization function such as without additional constraints generates solutions that are far from being adequate descriptors of the true input generation process. For example, a popular solution to the least-squares minimization criterion is principal component analysis (PCA), also sometimes referred to as eigenvector or singular value decomposition (SVD) of the input covariance matrix. PCA optimizes by finding a set of mutually orthogonal basis vectors that are aligned with the directions of maximum variance in the input data. This is perfectly adequate in the case where the input data clouds are Gaussian and capturing pairwise statistics suffices. However, statistical studies have shown that natural image distributions are highly non-gaussian and cannot be adequately described using orthogonal bases [Field, 199]. Thus, additional constraints are required in order to guide the optimization process towards solutions that more accurately reflect the input generation process. One way of adding constraints is to take into account the prior distributions of the parameters r and. Thus, one can minimize an optimization criterion of the form: r" " () where " and " are terms related to the prior distributions of the parameters r and. In particular, they denote the negative log of the prior probabilities of r and respectively. When viewed in the context of information theory, these negative log probability terms in can be interpreted as representing the cost of coding the parameters in bits (in the base ). Thus, the function can be given an interpretation in terms of the minimum description length (MDL) principle[rissanen, 199; Zemel, 199], namely, that solutions are required not only to be accurate but also to be cheap in terms of coding length. This formalizes the wellknown Occam s Razor principle that advocates simplicity over complexity among solutions to a problem. One may also note that minimizing is equivalent to maximizing the posterior probability of input data (maximum a posteriori (MAP) estimation). Specific choices of and determine the the nature of internal representations that will be learned. For example, Olshausen and Field [Olshausen and Field, 199] proposed functions of the form " ", " and "! to encourage sparseness in r. Alternately, one can use a zero-mean multivariate Gaussian prior on r[rao and Ballard, 1997a] to yield the negative log of the prior density: r" r# r (7) where is a positive constant and denotes a set of lateral weights. The matrix represents the inverse covariance matrix of r. We show in the next section that this choice enforces lateral inhibition among the output neurons, thereby encouraging sparse distributed representations, and leads to an anti-hebbian learning rule for the lateral weights equivalent to Foldiak s well-known adaptation rule[foldiak, 199].

5 Generative Weights U Spatial Generative Model A I Image r Spatial Response Vector I U r B Image Sequence I(k) I() I(1) U(k) U() U(1) r Spatiotemporal Response Vector Spatiotemporal Generative Model I(k) U(k) I() U() I(1) U(1) r Figure 1: Generative Models. (A) Linear spatial generative model used in[harpur and Prager, 199; Olshausen and Field, 199; Rao and Ballard, 1997a]. A given input image I is assumed to be generated by multiplying a basis vector matrix with a set of hidden causes represented by the spatial response vector r. (B) Spatiotemporal generative model used in this paper. An input image sequence I,, is assumed to be generated by multiplying a set of basis matrices,, with a spatiotemporal response vector r. In the case of, for the sake of simplicity, we will assume a Gaussian distribution with a covariance, yielding the following value for : " () where denotes the sum of squares of the elements of the given matrix. A final constraint is motivated both by biology and by information coding concerns. We will constrain the network outputs (coefficients ) to be non-negative, acknowledging the fact that the firing rate of a neuron cannot be negative. The non-negativity constraint is especially attractive in information coding terms since it causes an infinite density at for the rectified and a consequent low coding cost at [Hinton and Ghahramani, 1997], which encourages sparseness among the outputs of the network. We are overlooking the possibility that a single neuron can signal both positive and negative quantities by raising or lowering its firing rate with respect to a fixed background firing rate corresponding to zero.

6 " " " 3 Spatiotemporal Generative Models In the previous section, the input data consisted of static images I, and we used a single basis matrix to capture the statistical structure of the space of input images. Furthermore, the spatial structure given by the pixels of the image I were internally represented by a single spatial response vector r. We now draw an analogy between time and space to define a spatiotemporal generative model. Suppose our training set consists of different sequences of images each, a given sequence being denoted by the vectors I ",. We will use a set of basis matrices ",. For each, " will be used to capture the statistical structure of the images occurring at the time step in the training sequences. As will become clear in the next section, the underlying motivation here is that the neurons in the network perform not just a spatial summation, but a space-time summation of inputs over a finite spatiotemporal extent (see Figure ). Thus, since inputs are weighted differentially depending on their spatiotemporal history, we need to learn a set of synaptic weights, one for each time instant. These spatiotemporal synaptic weights, as given by the ", in turn determine the spatiotemporal receptive fields of the neurons in the network. A single spatiotemporal response vector r will be used to characterize a given spatiotemporal image sequence for, in much the same way as a single spatial response vector was previously used to characterize the spatial structure for % pixels of a given static image. We thus obtain the following space-time analog of Equation : I " " r Note that in the special case where, we obtain Equation. From a probabilistic perspective, one can rewrite Equation 9 in the form of the following stochastic generative model: I " I ". I ". (9) r n () where n is a stochastic noise process accounting for the differences between I " and " r. Once again, it is easy to see that Equation is a special case of the above generative model, where. We can now define the following space-time analog of the optimization function : "! " r" I "! " r" # I "! (11) " r" (1) This is simply the sum of squared pixel-wise reconstruction errors across both space and time. As in the previous section, if n is assumed to be a zero mean Gaussian with unit covariance, one can show that is the negative log likelihood of generating the inputs I " I ". Thus, minimizing is equivalent to maximizing the likelihood of the spatiotemporal input sequence.

7 As in the previous section, by assuming Gaussian priors on the parameters r, following MDL-based cost function: r# r " and, we obtain the (13) We will additionally constrain the elements of r to be non-negative as discussed in the previous section. ", and. This can be achieved by Thus, we are now left with the task of finding optimal estimates of r, minimizing using gradient descent as discussed in the next section. Network Dynamics and Learning Rules In this section, we describe gradient-descent based estimation rules for obtaining optimal estimates of r, " and. In the case where the input consists of batch data, as in Section.1, one may alternate between the optimization of r for fixed " and, and the optimization of " and for fixed r, thereby implementing a form of the expectation-maximization (EM) algorithm[dempster et al., 1977]. In the case of on-line data, which is considered in Section., the optimization of r occurs simultaneously with " and. However, the learning rates for " and are set to values much smaller than the adaptation rate of r..1 Estimation of r An optimal estimate of r can be obtained by performing stochastic gradient descent on with respect to r: where r! r " # "! " r"! r (1) governs the rate of descent towards the minima. The above equation is relatively easy to interpret: in order to modify r towards the optimal estimate, we need to obtain the residual error "! " r" between the input at time and its reconstruction " r. The residual errors for the various time steps are then spatiotemporally weighted by their corresponding weights "# to obtain a spatiotemporal sum, which is modified by lateral inhibition due to the term! r. In a neural implementation, the individual rows of the matrices " # would comprise the synaptic weights of a single spatiotemporally summating neuron. Thus, the % th row of " # would represent the effect of the synapses of the % th neuron for time instant. Figure A shows a network implementation of the above dynamics. A possible problem with this implementation is the need for computing global residual errors at each iteration of the estimation process. This becomes especially problematic in the case where the data is being obtained on-line since one would need to keep the past images in memory and in addition, use separate sets of neurons representing the matrices " for generating the signals " r at each iteration. The dynamics can be implemented more locally by rewriting Equation 1 in the following form: r " # "! r! r (1) where " # ". Note that this form of the dynamics does not require that residual errors be computed at each iteration. Rather, we simply perform a spatiotemporal filtering of the inputs I " using the synaptic weight matrices " # for and then subtract two lateral terms, one involving 7

8 A I(k) Residual Error Signals T U(1) T U(k) T U() Spatiotemporal Filtering of Errors + + Lateral Inhibition I(1) I() Image Sequence U(k) U() U(1) Rectify r Spatiotemporal Response Vector L B I(1) I() I(k) Spatiotemporal Filtering of Inputs T U(1) T U(k) T U() + Lateral Inhibition L + Rectify r W Recurrent Excitation/Inhibition Figure : Alternate Network Implementations of the Response Vector Dynamics. (A) shows a globally recurrent architecture for implementing the dynamics of the response vector r (Equation 1). The architecture requires the calculation of residual error signals [Mumford, 199; Barlow, 199; Pece, 199; Rao and Ballard, 1997a] between predicted images and actual images. These errors are filtered by a set of spatiotemporally summating neurons whose synapses represent the matrices. The result is used to correct r in conjunction with recurrent lateral inhibition and rectification. (B) shows a biologically more plausible locally recurrent architecture for implementing the dynamics of r. Rather than computing global residual errors, the inputs are directly filtered by a set of spatiotemporally summating neurons representing. The result is then further modified by recurrent excitation/inhibition due to the weights (see text) as well as lateral inhibition due to the weights.

9 and the other involving. While the term involving vector! the synapses of spatiotemporally summating neurons, with the % th row of is exclusively inhibitory, the components of the r may be excitatory or inhibitory. Once again, the rows of the matrices "# correspond to "# representing the synaptic weights of the % th spatiotemporal neuron for time instant. Expressing the above dynamics in its discrete form and adding a final constraint of non-negativity results in the following equation for updating r until convergence is achieved for the given input sequence: r % " r % " " # "! r % "! r % " " (1) where is a threshold nonlinearity for rectification: given vector. It is interesting to note the similarity between the dynamics as given by the stochastic gradient descent rule above and those proposed by Lee and Seung for their Conic network[lee and Seung, 1997]. In particular, the above equation can regarded as a spatiotemporal extension of the dynamics used in the Conic network. A network implementation of this equation is shown in Figure B.. On-Line Estimation ", applied to all components of the The dynamics described above can be extended to the more realistic case where the inputs are being encountered on-line. Let represent the current time instant. The on-line form of Equation 1 is then given by: r! "! " # r! r (17) Expressing the above dynamics in its discrete form and enforcing the constraint of non-negativity yields the following rule for updating r at each time instant: where the operator r " r " " #! "! r "! r " " (1) again denotes rectification. In summary, the current spatiotemporal response is determined by three factors: the previous spatiotemporal response r ", the past inputs I " I! ", and lateral inhibition due to and. These three factors at time instant are combined via summation, followed by rectification, to yield the responses at time instant. The consideration of only the past inputs rather than the entire input history in the equation above is consistent with the observation that cortical neurons process stimuli within restricted temporal epochs of time (see, for example,[deangelis et al., 199])..3 Learning Rules A learning rule for determining the optimal estimate for each descent on with respect to ", for each : "! " "! 9 " can be obtained by performing gradient " r" r#! " (19)

10 where is the learning rate parameter. Note that for the on-line case, I " is replaced by its on-line counterpart I! " in the above equation. Similarly, a learning rule for the lateral weights can be obtained by performing gradient descent on with respect to :! rr#! () Note that this is a Hebbian learning rule with decay. In conjunction with the inhibition in the dynamics of r (Equation 1), the above learning rule can be seen to be equivalent to Foldiak s anti-hebbian learning rule [Foldiak, 199], if the diagonal terms of, which implement self-inhibition, are set to zero. Experimental Results The algorithms derived in the previous section were tested on a set of five digitized natural images from the Ansel Adams Fiat Lux collection at the UCR/California Museum of Photography (Figure 3A, images reproduced with permission). The images were of size pixels with grayscale pixel values between and. Each image was preprocessed by filtering with a circularly symmetrical zero-phase whitening/low-pass filter with the spatial frequency profile[olshausen and Field, 199; Atick and Redlich, 199]: " where the cut-off frequency cycles/image. As described in [Olshausen and Field, 1997], the whitening component of the filter " performs sphering for natural image data by attenuating the low frequencies and boosting the higher frequencies. The low-pass exponential component " helps to reduce the effects of noise/aliasing at high frequencies and eliminates the artifacts of using a rectangular sampling lattice. Figure 3B shows the frequency profile of the whitening/low-pass filter in the 1-D case while Figure 3C shows the -D case. The corresponding spatial profile obtained via inverse Fourier transform is shown in Figure 3D. The spatial profile resembles the well-known center-surround receptive fields characteristic of retinal ganglion cells. Atick and Redlich [Atick and Redlich, 199; Atick, 199] have shown that the measured spatial frequency profiles of retinal ganglion cells are well approximated by filters resembling ". Figure 3E shows the results of filtering an image from the training set using ". The training data comprised of sequences of contiguous image patches extracted from the filtered natural patch is shown as a box labeled RF in Figure 3E. Starting from an image patch extracted at a randomly selected location in a given training image, the next training patch was extracted by moving in one of directions as shown in Figure 3F. This was achieved by incrementing or decrementing the and/or coordinate of the current patch location by one according to the current direction of motion. After every image patches, the current direction was set randomly to one of the possible directions. The direction was also randomly changed in case an image boundary was encountered. A new training image was obtained after every patches, thereby cycling through the five preprocessed training images. In the first experiment, a network of neurons was trained on sequences of natural image patches of size. The temporal extent of processing was set to, resulting in sets of synaptic weight vectors. These vectors, which form the rows of " #,, were initialized to uniformly random values and normalized to length one. After the presentation of each training image patch, the response vector images. The relative size of a (1)

11 A K(f) IFT FT 3 3 B f f y C f x D RF Image E F Figure 3: Training Paradigm. (A) The five natural images used in the experiments. The original negatives are from the Ansel Adams Fiat Lux collection at the UCR/California Museum of Photography (images reproduced with permission). (B) 1-D spatial frequency response profile of zero-phase whitening/low-pass filter used for preprocessing the natural images. (C) The full -D response profile of the same filter. (D) The profile in the space domain obtained by inverse Fourier transform (IFT), showing the familiar center-surround receptive field characteristic of retinal ganglion cells (see text). (E) A natural image after being filtered with the center-surround filter in (D). The relative size of a receptive field (same size as the input image patches) is shown as a box labeled RF on the bottom right corner. (F) depicts the process of extracting image patches from the filtered natural images for training the neural network. Patches are extracted from the square window as it moves in one of directions within a given training image (see text from more details). 11

12 t=1 t= t=3 t= t= t= t=7 t= A Initial Synaptic Weights t=1 t= t=3 t= t= t= t=7 t= B x y C y t=1 x Synapses After Learning Figure : Example of Receptive Field Dynamics after Learning (RF size = ). (A) shows the initial set of synaptic weights (or receptive fields) for a single model neuron. The weights shown comprised the first row of for. The components of these vectors were initialized to random values according to the uniform distribution and the resulting vectors were normalized to length one. (B) shows the same set of synaptic weights after training the -neuron network on natural images (see text for details). The synaptic profiles at each time step resemble oriented Gabor wavelets[daugman, 19; Marcelja, 19]. (C) depicts these synaptic weights using a classical D receptive field representation, where the dark regions are inhibitory and the bright regions are excitatory. The entire sequence of synaptic weights suggests that this neuron is tuned towards dark bars moving diagonally from the bottom left corner of the receptive field to the top right corner r was updated according to the on-line estimation rule given by equation 1 with and. The weights " were adapted at each time step using equation 19 with and. Similarly, the lateral weights were adapted using equation with and. The diagonal of, representing self-inhibitory decay terms or the leakiness parameters in the leaky integrator equation 1, was fixed at, although qualitatively similar results were obtained if the diagonal was also additionally adapted along with the lateral weights. The learning rate parameters and were gradually decreased by dividing with every image presentations and a stable solution was arrived at after image presentations. Figure shows a set of synaptic weight vectors (first row of "# for ) for a model neuron before and after learning (A and B respectively). Since these synaptic vectors form the feedforward weighting function of neurons in the network (see Figure B), they can be roughly interpreted as receptive fields or spatial impulse response functions for time steps. The receptive fields after learning resemble localized Gabor wavelets which have previously been shown to well approximate the receptive field weighting profiles of simple cells in the mammalian primary visual cortex[daugman, 19; Marcelja, 19; Olshausen and Field, 199]. In addition, the model neuron can be seen to be tuned towards dark bars moving diagonally from the bottom left corner of the receptive field to the top right corner. A 1

13 (a) t=1 3 7 t=1 3 7 (b) (c) (d) (e) (g) (f) (h) (i) (j) (k) (l) (m) (n) (o) (p) Figure : Further Examples of Receptive Fields (RF size = ). (a) through (p) show examples of synaptic weights for sixteen other model neurons from the network described in Figure. Several varieties of orientation selectivity can be discerned, with each neuron being tuned towards a particular direction of motion. Other neurons, such as those depicted in (o) and (p), appear to be selective for oriented image structures that are approximately stationary or contain motion along the preferred orientation. number of other examples of receptive fields developed by the network are shown in Figure. The set of spatiotemporal synaptic weights together form an overcomplete set of basis functions for representing spatiotemporal input sequences. In a second set of experiments, we investigated the importance of rectification and lateral inhibition in learning visual cortex-like receptive fields. Three networks using the same set of parameters, initialization conditions, and training patches as the one used in Figures and were subjected to three different conditions. In the first network, rectification was removed but the lateral weights were learned and used as before while in the second network, the lateral weights (including the diagonal terms) were disabled but rectification was retained. The third network was trained without either lateral inhibition or rectification of responses. Figure compares the results obtained after image presentations for the same model neuron as in Figure. This specific example as well as the results obtained for other model neurons in the networks suggest that both lateral inhibition and rectification are necessary for obtaining synaptic profiles resembling visual cortical receptive fields. In a third set of experiments, the results obtained for receptive fields were verified in the case. The training paradigm involving the natural image patches was identical to the case. A network of neurons was trained on sequences of natural image patches of size. The temporal extent, of processing was set to. The parameters were set as follows: 13,,

14 y t=1 x 3 7 With Rectification and Lateral Inhibition Lateral Inhibition Only Rectification Only Without Rectification or Lateral Inhibition Figure : The Need for Lateral Inhibition and Rectification during Learning. The figure shows the synaptic profiles obtained after training for the same neuron (from Figure ) with identical initial conditions and identical training regimes but with lateral inhibition and/or rectification removed. As can be seen from this specific example, which is typical of the result obtained for other neurons in the networks, both lateral inhibition and rectification appear to be necessary for obtaining synaptic profiles resembling visual cortical receptive fields. Neither rectification nor lateral inhibition alone seemed to suffice to produce the desired neural receptive fields.,, and. The diagonal of was fixed at. The learning rate parameters and were gradually decreased by dividing with every image presentations and a stable solution was arrived at after image presentations. Figure 7 shows a set of synaptic weight vectors (or receptive fields) for a model neuron before and after learning. The model neuron can be seen to be tuned towards dark horizontal bars moving downwards. Several other examples of receptive fields developed by the network are shown in Figure. A majority of these neurons appear to be tuned towards oriented image structures moving in a particular direction. Some neurons, such as the one shown in (k), exhibit more complex dynamics, involving two or more inhibitory subregions coalescing into one while other neurons, such as the one shown in (l), appear to be tuned towards oriented image structures that are either approximately stationary or contain some form of motion along the preferred orientation. The space-time receptive fields for the model neurons in Figure 7 and are shown in Figure 9. These - plots were obtained by integrating the 3-D spatiotemporal receptive field data along the neuron s preferred orientation, as illustrated in the top two rows of the figure. The inverse of the slope of the oriented subregions in the space-time receptive field provides an estimate of the neuron s preferred velocity (see, for example, [Adelson and Bergen, 19]). Thus, in the case of the top two rows in the figure, the slope is approximately indicating a preferred speed of approximately pixel/time step for these two neurons in their respective directions. The space-time receptive fields in Figure 9 (b) through (l) are those for the model neurons in Figure (b) through (l). Note that even though the training image window moved at pixel/time step, in some cases, such as (d) and (h), the preferred speed is less than pixel/time step due to the well-known aperture effect [Adelson and Movshon, 19]. In the extreme case of an approximately stationary receptive field as in (l), the space-time receptive field indicates a preferred speed of zero. In order to evaluate the nature of internal representations used by the network, we exposed the trained 1

15 t=1 t= t=3 t= t= t= A t=7 t= t=9 t= t=11 t= Initial Synaptic Weights t=1 t= t=3 t= t= t= B t=7 t= t=9 t= t=11 t= y x Synapses After Learning C y x t= Figure 7: Example of Receptive Field Dynamics after Learning (RF size = ). (A) shows the initial set of random and normalized synaptic weights for a single model neuron in a -neuron network. (B) shows the same set of synaptic weights after training the network on the filtered natural images (see text for details). (C) depicts these synaptic weights using a classical D receptive field representation (dark regions are inhibitory, bright regions are excitatory). The sequence of synaptic weights suggests that this neuron is tuned towards dark horizontal bars moving downwards. 1

16 t= (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure : Further Examples of Receptive Fields (RF size = ). (a) through (l) show examples of synaptic weights for twelve other model neurons from the network used in Figure 7. A majority of these neurons are tuned towards oriented image structures moving in a particular direction. Some neurons, such as the one shown in (k), exhibit more complex dynamics, for example, involving two or more inhibitory subregions coalescing into one. Other neurons, such as the one shown in (l), appear to be tuned towards oriented image structures that are either approximately stationary or contain some motion along the preferred orientation. 1

17 t= t x t= t (a) x (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) t x Figure 9: Space-Time Receptive Fields (size = ). The top two rows depict the process of constructing space-time receptive field profiles from temporal sequences of -D spatial receptive fields. The top row is from Figure 7 while the second row is from Figure (a). The - plots are obtained by integrating the 3-D spatiotemporal data along the neuron s preferred orientation (in these two cases, along the horizontal direction). Note that orientation in - space indicates the neuron s preferred direction of motion (in this case, upwards or downwards). In addition, the slope of approximately in both cases indicates a preferred speed of approximately pixel/time step for these neurons (since slope is inverse with the preferred velocity[adelson and Bergen, 19]). (b) through (l) show the space-time receptive fields for the model neurons in Figure (b) through (l). In some cases, such as (d) and (h), the preferred speed is less than pixel/time step due to the aperture effect[adelson and Movshon, 19]. In the case of (l), the preferred speed is zero. 17

18 network and a companion network without the lateral inhibitory weights to a sequence of images depicting a bright vertical bar moving to the right on a dark background (Figure ). In both cases, the responses of the model neurons in each network were plotted as histograms at each time step, with the first vertical bar corresponding to the response of the first neuron, the second to that of the second neuron, and so on. As seen in Figure, the trained network generates sparse distributed representations of the spatiotemporal input sequence (only a few neurons are active at each time step). On the other hand, disabling the lateral inhibitory connections results in a much larger number of neurons being active at each time step, suggesting that the lateral connections play a crucial role in generating sparse distributed representations of input stimuli. In the final set of experiments, we analyzed the direction selectivity of model neurons in the trained -neuron network. Figure 11 illustrates the experimental methodology used. Each neuron in the network was exposed to oriented dark/bright bars on a bright/dark background moving in a direction perpendicular to the bar s orientation (bright or dark bars were chosen depending on which case elicited the largest response from the neuron). The direction of motion was varied from to in steps of. Figure 11B shows the cases where the direction of motion is (downwards) and (upwards) respectively. Figure 11C shows the response of the model neuron in these two cases as a function of the time from stimulus onset. As expected from the structure of its space-time receptive field, the neuron exhibits a significant response for a bar moving downwards (preferred direction) while a bar moving upwards (null direction) elicits little or no response. A direction selectivity index (DSI) was defined for each model neuron as: DSI! Peak Response in Null Direction Peak Response in Preferred Direction The preferred direction was taken to be the direction of motion eliciting the largest response from the model neuron and the null direction was set to the opposite direction. Thus, a neuron with a peak response of zero in the null direction and a nonzero response in the preferred direction has a DSI of or. A neuron with equal responses in both directions have a DSI of zero. Figure 1A shows polar response plots of four model neurons with increasing DSI from left to right. The rightmost plot is for the neuron in Figure 11 and its DSI of confirms its relatively high direction selectivity. Figure 1B shows the population distribution of direction selectivity in the -neuron network. A relatively large proportion of the neurons ( of the or ) had a direction selectivity index of or greater. Discussion The results suggest that many of the important spatiotemporal properties of visual cortical neurons such as velocity and direction selectivity can be explained in terms of learning efficient spatiotemporal generative models for natural time varying images. Efficiency is defined not only in terms of reducing image reconstruction errors but also in terms of reducing statistical dependencies between network outputs. This redundancy reduction is implemented via recurrent lateral inhibition as derived from the MDL principle. The network also utilizes other lateral connections which are derived from the generative model. These lateral excitatory/inhibitory connections play a role similar to those used in Bayesian belief networks for facilitating the phenomenon of explaining away [Pearl, 19]. The network additionally includes rectification of outputs to encourage sparseness among the network activities. Although rectification helps in the development of sparse distributed representations, rectification alone was found to be insufficient in producing satisfactory space-time receptive fields. Likewise, lateral inhibition alone was also found to be insufficient. 1 ()

19 Input Responses (Without L. Inhibition) Responses (With L. Inhibition) Input Responses (Without L. Inhibition) Responses (With L. Inhibition) Figure : Lateral Inhibition and Sparse Distributed Representations of Input Stimuli. To test the role of the lateral inhibitory weights (see Figure ) in generating sparse distributed representations, these weights were removed (after learning) and the network was exposed to a bright vertical bar moving to the right. The responses of the neurons in the network at the various time steps are shown as histograms (the first vertical bar is the response of the first neuron, the second is that of the second neuron, and so on). The histograms have been normalized such that the maximum response in each graph is one. The responses with lateral inhibition intact are shown to the right at each time step. As is evident from the relatively sparse number of neurons active at each time step in the with lateral inhibition case, the existence of lateral inhibitory connections in the network appears to be crucial for generating sparse distributed representations of the spatiotemporal input stimuli. 19

20 A B C..1 Response Preferred Direction. t. 1 x Space-Time Receptive Field Response Null Direction. Input Stimulus 1 Time from Stimulus Onset Figure 11: Example of Direction Selectivity. The model neuron from Figure 7 was tested for direction selectivity. (A) shows the space-time receptive field of this neuron. (B) depicts the two input stimuli used for testing: a dark horizontal bar in a white background, moving either downwards or upwards. (C) shows the response of the neuron as a function of the time from stimulus onset. As expected from the structure of its space-time receptive field, the neuron exhibits a significant response for a bar moving downwards (preferred direction), reaching a peak response of units at 11 time steps from stimulus onset. On the other hand, a bar moving upwards (null direction) elicits little or no response.

21 DSI = 1.77 % DSI =. % DSI = 3. % DSI = 9. % A Number of Neurons Direction Selectivity Index (DSI) % B Figure 1: Analysis of Direction Selectivity. (A) shows polar response plots of four model neurons when exposed to bars oriented at a particular angle and moving in a direction perpendicular to the orientation. The radius of the plot indicates the strength of the response while the angle indicates the direction of motion of the bar. The sequence of plots is arranged from left to right in the order of increasing direction selectivity, as given by the direction selectivity index DSI (see text for definition). The polar plot on the extreme right is for the neuron in Figure 11, confirming its relatively high direction selectivity. (B) shows the distribution of direction selectivity in the population of neurons in the trained network. A relatively large proportion of the neurons ( of the ) had a direction selectivity index of or greater. 1

22 Several previous models of direction selectivity have utilized recurrent lateral interactions and rectification [Suarez et al., 199; Maex and Orban, 199; Mineiro and Zipser, 1997], although without the statistical perspective pursued herein. In addition, the receptive fields in many of these approaches were hard-wired by hand rather than learned. An interesting issue is how the spatiotemporally varying synaptic weights of neurons in the present model can be implemented biologically by intrinsic synaptic mechanisms and axonal-dendritic interactions over time. In this regard, it is possible that space time receptive fields similar to those obtained by the present method may also arise in other neural circuits that do not explicitly use spatiotemporal synaptic weights. Most of the synaptic weights in the trained networks were found to be space-time inseparable. The lack of larger numbers of separable receptive fields in the trained networks suggests an alternative mechanism for the generation of such receptive fields. Most cortical receptive fields that are space-time separable have a temporal weighting profile that approximates a derivative in time. We have previously shown [Rao and Ballard, 1997b] that a generative model based on a first-order Taylor series approximation of an image produces localized oriented filters that compute spatial derivatives for estimating translations in the image plane. A possibility that we are currently investigating is to use a Taylor series expansion of an image in both space and time, and to ascertain whether such a strategy produces separable filters that compute derivatives in both space and time. Other issues being pursued include explaining contrast normalization effects[albrecht and Geisler, 1991; Heeger, 1991] and recasting the hierarchical framework proposed in[rao and Ballard, 1997a] to accommodate the spatiotemporal generative model proposed herein. References [Adelson and Bergen, 19] E.H. Adelson and J. Bergen. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A, (): 99, 19. [Adelson and Movshon, 19] E.H. Adelson and J.A. Movshon. Phenomenal coherence of moving visual patterns. Nature, 3:3, 19. [Albrecht and Geisler, 1991] D.G. Albrecht and W.S. Geisler. Motion sensitivity and the contrast-response function of simple cells in the visual cortex. Visual Neurosci., 7:31, [Atick and Redlich, 199] J.J. Atick and A.N. Redlich. What does the retina know about natural scenes? Neural Computation, ():19, 199. [Atick and Redlich, 1993] J.J. Atick and A.N. Redlich. Convergent algorithm for sensory receptive field development. Neural Computation, :, [Atick, 199] J.J. Atick. Could information theory provide an ecological theory of sensory processing. Network, 3:13 1, 199. [Baddeley and Hancock, 1991] R.J. Baddeley and P.J.B. Hancock. A statistical analysis of natural images matches psychophysically derived orientation tuning curves. Proc. R. Soc. Lond. Ser. B, :19 3, [Barlow, 191] H.B. Barlow. Possible principles underlying the transformation of sensory messages. In W.A. Rosenblith, editor, Sensory Communication, pages Cambridge, MA: MIT Press, 191.

23 [Barlow, 197] H.B. Barlow. Single units and cognition: A neurone doctrine for perceptual psychology. Perception, 1:371 39, 197. [Barlow, 199] H.B. Barlow. Unsupervised learning. Neural Computation, 1:9 311, 199. [Barlow, 199] H.B. Barlow. What is the computational goal of the neocortex? In C. Koch and J.L. Davis, editors, Large-Scale Neuronal Theories of the Brain, pages 1. Cambridge, MA: MIT Press, 199. [Barrow, 197] H.G. Barrow. Learning receptive fields. In Proceedings of the IEEE Int. Conf. on Neural Networks, pages 11 11, 197. [Bell and Sejnowski, 1997] A.J. Bell and T.J. Sejnowski. The independent components of natural scenes are edge filters. Vision Research (in press), [Bienenstock et al., 19] E. L. Bienenstock, L. N. Cooper, and P. W. Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci., :3, 19. [Bryson and Ho, 197] A.E. Bryson and Y.-C. Ho. Applied Optimal Control. New York: John Wiley and Sons, 197. [Burr et al., 19] D.C. Burr, J. Ross, and M.C. Morrone. Seeing objects in motion. Proc. R. Soc. Lond. Ser. B, 7:9, 19. [Daugman, 19] J.G. Daugman. Two-dimensional spectral analysis of cortical receptive field profiles. Vision Research, :7, 19. [Daugman, 19] J.G. Daugman. Complete discrete -D Gabor transforms by neural networks for image analysis and compression. IEEE Trans. Acoustics, Speech, and Signal Proc., 3(7): , 19. [Dayan et al., 199] P. Dayan, G.E. Hinton, R.M. Neal, and R.S. Zemel. The Helmholtz machine. Neural Computation, 7:9 9, 199. [DeAngelis et al., 1993a] G.C. DeAngelis, I. Ohzawa, and R.D. Freeman. Spatiotemporal organization of simple-cell receptive fields in the cat s striate cortex. I. General characteristics and postnatal development. J. Neurophysiol., 9(): , [DeAngelis et al., 1993b] G.C. DeAngelis, I. Ohzawa, and R.D. Freeman. Spatiotemporal organization of simple-cell receptive fields in the cat s striate cortex. II. Linearity of temporal and spatial summation. J. Neurophysiol., 9(): , [DeAngelis et al., 199] G.C. DeAngelis, I. Ohzawa, and R.D. Freeman. Receptive-field dynamics in the central visual pathways. Trends in Neuroscience, 1:1, 199. [Dempster et al., 1977] A.P. Dempster, N.M. Laird, and D.B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistical Society Series B, 39:1 3,