To compliment the population decoding approach (Figs. 4, 6, main text), we also tested the

Transcription

1 Supplementary Material, Page 1 of 18 Single-cell neurometric (ROC) analysis To compliment the population decoding approach (Figs. 4, 6, main text), we also tested the ability of individual MSTd neurons to account for cue reweighting behavior. Specifically, we used receiver operating characteristic (ROC) analysis to quantify the behavior of an ideal observer performing the same task as the animal but using only the firing rate responses of a given neuron. Response distributions for each nonzero heading were compared to the distribution for heading = 0 (taken as a proxy for the internal reference) to compute the proportion of rightward choices made by the ideal observer. As with the psychophysical analysis (Fig. 1b-d), these neurometric choice data were fit with cumulative Gaussian functions, and the neuronal PSE and neuronal threshold were taken as the mean and standard deviation of the fitted function, respectively. For the example neuron shown in Fig. 3 (replotted in Fig. S1a-c), neuronal thresholds in the single-cue trials were 6.7, 13.6, and 2.1 for the vestibular, low-coherence visual, and high-coherence visual conditions, respectively (Fig. S1d). These thresholds quantify the neuron s sensitivity to small variations in heading for each modality, which is primarily dictated by the slopes of the respective tuning functions. Using Eq. 2, single-cue neuronal thresholds were used to generate optimal vestibular weights for the ideal observer (here, 0.80 and 0.08 for low and high coherence, respectively). These were compared to observed weights computed with Eq. 4, using the neurometric PSEs from the combined, cue-conflict trials (Fig. S1e,f). Note that the reference distribution (responses at heading = 0 ) was taken from the Δ = 0 trials, regardless of the value of Δ. The use of non-conflict responses for the reference is a sensible strategy because the monkey cannot predict whether a trial will be a cueconflict trial, and may not even detect the conflict when present 1. Because the firing rates during cueconflict (green and blue traces in Fig. 3b,c and Fig. S1b,c) were compared with the non-conflict trials (orange traces), the shift of the tuning curves led to shifts of the neurometric functions, in a manner 1

2 Supplementary Material, Page 2 of 18 analogous to the psychometric functions (Fig. 1c,d). For this neuron, the observed weights corresponding to these PSE shifts were 0.76 and 0.16 for low and high coherences, respectively. A second example cell is shown in Fig. S1g-l. Because the ROC analysis approach depended on reliable PSE estimates from neurometric functions, it could only be performed on neurons with clear, monotonic tuning around straight ahead. 2

3 Supplementary Material, Page 3 of 18 Otherwise, the neurometric functions were too flat to provide reliable estimates of the neuronal thresholds and PSEs. Thus, we required the slope of the linear regression of firing rate vs. heading during the discrimination task to be significantly greater or less than zero for at least one single-cue modality, as well as the combined modality at both coherence levels. These criteria resulted in a sample of 45 neurons (30 from monkey Y, 15 from monkey W). This sample excluded most neurons with visual and vestibular tuning that was clearly opposite in slope ( opposite cells 2, see Fig. 5), as the combined tuning for these neurons was often too flat or non-monotonic. Of the 23 neurons classified as opposite (congruency index (CI) < -0.4), all but one failed to meet the criteria for ROC analysis, whereas 34 of the 46 congruent cells (CI > 0.4) passed these criteria. Weights for this subpopulation of MSTd neurons are summarized in Fig. S2a, with each neuron represented by two data points (one for each coherence, connected by a thin line) and plotted as observed vs. optimal vestibular weight. The main result to take away from this figure is that most cells showed large changes in observed weights during combined trials (ordinate) as a function of coherence. Median values for low (0.691) and high (0.195) coherence were highly significantly different (Wilcoxon matched-pairs signed rank test, p < 10-7 ). The paired difference in observed weights (low minus high coherence) was significantly greater than zero (p < 0.05, two-tailed bootstrap test) for 17 of 45 neurons (38%) and not significantly less than zero for any neurons (Fig. S2c). These results suggest that MSTd neurons respond robustly to trial-by-trial changes in cue reliability. Notably, these effects occurred for changes in reliability that were behaviorally relevant: the two coherence levels were chosen to be effective at driving behavioral reweighting (i.e., Fig. 2a,b), and were not tailored to drive changes in neurometric weights. Psychophysical weights from these 45 sessions are shown in the same formats for comparison (Fig. S2b,d). 3

4 Supplementary Material, Page 4 of 18 4

5 Supplementary Material, Page 5 of 18 The second main result from Fig. S2a involves the comparison between optimal and observed weights. Many of the data points and connecting lines trend along the unity-slope diagonal, indicating weights that were close to optimal predictions. Optimal and observed weights were significantly correlated for both coherences together (Spearman s rank correlation, rho = 0.73, p < ) and for each coherence separately (16%: rho = 0.49, p = ; 60%: rho = 0.77, p < 10-9 ). However, many data points also lie well above the unity-slope diagonal (observed weight > optimal weight; Wilcoxon matched-pairs signed rank test: p < for both monkeys together and monkey W individually; p = 0.21 for monkey Y individually), some with nearly vertical connecting lines. These cells tended to have relatively flat vestibular tuning and therefore low optimal vestibular weights. Yet, the vestibular cue nonetheless had a sizeable influence on the combined tuning, resulting in observed vestibular weights that were greater than the prediction. This apparent deviation from optimality (as defined by Eq. 2) is intriguing because it parallels the over-weighting of the vestibular cue at low coherence, observed in behavior (Fig. 2a,b and Fig. S2b). Statistical comparisons from Fig. S2a also held for Fig. S2b (observed weight > optimal weight; p < for both monkeys together and monkey W individually; p = 0.28 for monkey Y individually), although a notable difference is the vestibular under-weighting (visual over-weighting) for monkey Y at 60% coherence (Fig. S2b, red circles below the diagonal; also evident in Fig. 2a). In summary, individual MSTd neurons reweight visual and vestibular cues on a trial-by-trial basis, supporting a possible role in perceptual reweighting and consistent with the population decoding results (Figs. 4, 6). Many cells perform nearly optimally, while a subset appears to over-weight vestibular cues, similar to the monkeys behavior at low coherence. This unexpected link between neuronal activity and a modality-specific perceptual bias adds support for the hypothesis that MSTd neurons contribute to visual-vestibular cue integration for heading perception. Lastly, neuronal 5

6 Supplementary Material, Page 6 of 18 thresholds for this sample (Fig. S3) decreased under combined stimulation, in accordance with optimal predictions (Eq. 3) and consistent with our previous work 2. 6

7 Supplementary Material, Page 7 of 18 ROC analysis approach: assumptions and caveats The neurometric analysis described above assumes that (a) the ideal observer knows the preferred heading (sign of tuning slope) for each neuron and modality, and (b) the response variability at zero heading is a reasonable approximation of the uncertainty in the internal reference against which the monkey is discriminating. The first assumption is common to all similar approaches and is necessary for the observer to interpret firing rates as signaling a particular stimulus in the world. The second assumption is an alternative to the neuron-antineuron framework used in some studies for measuring neuronal sensitivity with ROC analysis, including our own previous work 2. In that procedure, the response distribution for a particular heading would be compared with the distribution for the corresponding heading of opposite sign, thereby invoking an antineuron with mirror-symmetric tuning. Instead, we found that using the zero-heading responses as the reference distribution was more intuitive when the goal was to use neuronal PSEs during cue-conflict as a measure of cue reweighting. The main effect of changing our analysis to a neuron-antineuron approach would be to increase neuronal sensitivity (decrease neuronal thresholds) across the board; i.e., for monotonic tuning functions, there will generally be greater separation between the firing rate distributions for +θ vs. -θ for a given θ, as opposed to each θ vs. zero heading. However, this change would not alter the relative neuronal thresholds across modalities (i.e., optimal weights), nor the PSEs during cue-conflict (observed weights), provided that the non-conflict (Δ = 0º) responses were still used as the reference. Reweighting index To ensure that the results of Fig. S2a,c were not dependent on any assumptions of ROC analysis or the fitting of neurometric functions, we computed a model-free reweighting index (Fig. S2e) for the same sample of neurons, as follows. First we computed the mean normalized change in firing rate 7

8 Supplementary Material, Page 8 of 18 ( delta-fr ) caused by the cue-conflict (*see note below), separately for each coherence level, and collapsed across positive and negative Δ: delta-fr = (S1) Reweighting Index = delta-fr coh=60% delta-fr coh=16% (S2) In this equation, the index h refers to heading, t refers to each of N trials, r t (h,δ) is the single-trial firing rate in the combined modality for Δ = ±4º, and r-bar is the mean combined firing rate for Δ = 0º. The normalization factor in the denominator is the total firing rate modulation (max-min) for the Δ = 0º tuning curve. Sgn is the sign function (signum), and slope refers to the slope of the combined tuning curve based on linear regression. These sign functions preserve the expected relationship between cue reliability and firing rate as a function of Δ and tuning slope i.e., for the examples in Fig. S1c and Fig. S1i the tuning curve for Δ = +4º (green) shifts downward, but would have shifted upward for that same Δ and coherence if the tuning slope was positive instead of negative, and we wanted the delta-fr metric to have the same sign in both cases. The reweighting index itself was then computed as the difference between the delta-fr values for high versus low coherence, giving positive values for shifts of the tuning curve in the predicted direction. Statistical significance of the reweighting index was determined by bootstrapping the delta-fr metrics 1000 times (resampling across trials and heading values with replacement), recomputing the reweighting index each time, and determining whether more than 95% of the bootstrapped index values were greater or less than zero (percentile method at p < 0.05). The resulting distribution of reweighting index values was strongly shifted toward positive values (t test, p < ), with 26/45 (58%) of neurons having values significantly greater than zero (p < 0.05, bootstrap) and none significantly less than zero (Fig. S2e). 8

9 Supplementary Material, Page 9 of 18 [*Note: the effect of cue-conflict on firing rates may be more aptly described as a lateral tuning shift, since the conflict manipulation itself is a lateral one (change in heading specified by the constituent cues). However, for the monotonic tuning curves in our sample, this was largely indistinguishable from a vertical delta-fr, and we found the latter to be more reliable than common methods for estimating lateral shifts, such as curve fitting or cross-correlation (particularly for cells with relatively shallow tuning).] Likelihood-based decoder: assumptions and caveats Similar to the ROC analysis method for single neurons (see above), the decoder assumes knowledge of the neuronal tuning curves (f i in Eq. 14) at the readout stage. In previous theoretical treatments 3-6, the choice of what to use for f i was clear, mainly because only single-modality likelihood functions (or log-likelihoods) were being computed. In contrast, we are decoding from real neurons in 9 distinct trial types (3 stimulus modalities, 2 coherence levels, and 3 conflict angles), and therefore must make a decision about which tuning curves to substitute for f i when computing the likelihood functions. We chose to allow the decoder full knowledge of the neuronal tuning curves in every condition i.e., f i (θ) was taken as the tuning curve in the particular condition (stimulus modality and coherence level) being simulated on that trial. An important exception to this was the combined, cue-conflict (Δ = ±4º) trials. These trials were decoded with respect to the non-conflict (Δ = 0º) tuning curve, to allow the simulation to produce a shift in the PSEs similar to the monkeys behavior (e.g., see Fig. S4). While we cannot be certain that the brain is capable of such a dynamic readout strategy, it may be feasible in our task because the subject knows the modality (and possibly the coherence level) fairly early in the trial. The presence or absence of platform motion is easily detected from vibration and auditory cues, and the approximate coherence ( low or high, given that only 2 coherence levels are 9

10 Supplementary Material, Page 10 of 18 possible) is apparent from the dynamics of the random dots before they begin to clearly simulate a particular heading trajectory (see Fig. S5 for stimulus temporal profile). The alternative strategy of using a fixed tuning curve for all conditions (e.g., the single-cue vestibular tuning) produced similar results with regard to cue reweighting. However, doing so hampered our attempts to analyze populations with substantial numbers of opposite cells. For example, when 10

11 Supplementary Material, Page 11 of 18 decoding relative to the vestibular tuning curves, the simulation made too many errors in the visual modality, and the choice data could not be fit with a monotonically increasing function such as a cumulative Gaussian. Furthermore, even for congruent or mixed populations, using a tuning curve from one stimulus modality to decode another can produce large choice biases due to offsets between the two tuning curves. Thus, to ensure an unbiased and complete picture of the information contained in our MSTd population, we felt the full knowledge approach was best suited for our purposes. 11

12 Supplementary Material, Page 12 of 18 The response of each neuron in the population (r i ) can in principle be drawn from the actual single-trial responses for the desired heading and trial type. However, because population sensitivity (i.e., signal to noise ratio) increases with N, we found that for larger population sizes (N > ~40) the simulated psychometric functions (e.g., Fig. 4c,f; note abscissa scale) became too steep to be reliably fit using only the heading angles used in the experiments (0º, ±1.23º, ±3.5º, and ±10º). Thus, it became necessary to interpolate the tuning curves (linear, 0.1º resolution) and generate simulated firing rate distributions at this finer heading resolution. Responses were typically generated by sampling a Poisson process (poissrnd in Matlab) with the λ parameter (mean and variance) equal to the mean (interpolated) response. The results of the decoding model using this procedure were compared to the method of drawing real single-trial firing rates (when smaller population sizes permitted it), and no significant differences were found. As the Poisson method restricts the variance-to-mean ratio (Fano factor) to be 1, we also tested versions of the model using Gaussian noise and various Fano factors ranging from 1 to 2. We found no substantial differences between the two noise models with regard to the main conclusions of the study. Increasing the Fano factor only had a noticeable effect on the absolute thresholds for a given population size, not on the pattern of relative thresholds across modalities (optimal weights), nor the PSEs during cue-conflict (observed weights). An additional assumption of this version of the decoder is that each neuron is independent, i.e. that noise correlations are zero. Employing the method of Shadlen et al. 7 for generating arbitrary correlation structure in simulated data, we tested other versions that included small positive correlations characteristic of MSTd neuron pairs recorded in the heading discrimination task 8. As expected from the known effects of noise correlations on the fidelity of population codes 9, 10, positive correlations increased the simulated thresholds for a given population size. However, similar to the case of 12

13 Supplementary Material, Page 13 of 18 increasing Fano factor, the relative single-cue thresholds and the combined PSEs were largely unaffected by correlations, and thus the conclusions with regard to cue reweighting did not change. The one case where the basic Poisson decoding method seemed to fail was with artificial combined responses, generated by taking weighted sums of single-cue responses (Fig. 8b-e). Initially, we performed these simulations by generating combined tuning curves (mean firing rates) as weighted sums of single-cue tuning curves, then making draws from a Poisson process at the resulting means. This procedure failed to reproduce appropriate distributions of single-trial firing rates, as evidenced by the fact that even the decoding of optimal responses (i.e., generated with weights according to Eq. 7) did not result in optimal behavior according to Eqs. 2 and 3 (not shown). This problem was remedied by taking weighted sums of single-trial responses (rather than mean responses) to generate the artificial combined data. Since the weights in the summation were not fixed at 1, the resulting response distributions were typically not Poisson, requiring us to use an ad-hoc decoding method in place of Eq. 14. This ad-hoc method began with defining the distribution P(r i θ) empirically, as a normalized histogram of single-trial firing rates for a given heading and cell. We then selected the probability value of the nearest bin corresponding to the particular r being simulated, and repeated this for all neurons and headings to essentially bootstrap the full likelihood function P(r θ). This procedure was validated by the near-perfect match between optimal and observed weights and thresholds in Fig. 8b,c, and was also able to reproduce the basic pattern of results shown in Fig. 6e,f. Variance as a signature of cue combination? The fact that our decoder failed when taking weighted sums of mean single-cue responses and then drawing Poisson random deviates raises an interesting set of questions about the variance of multisensory responses. Assuming, as we have shown (Fig. 7 and ref. 11 ), that mean combined responses 13

14 Supplementary Material, Page 14 of 18 are reasonably well modeled as weighted sums of single-cue responses, we can ask whether the variance of the combined response behaves as predicted from applying these weights. We tested this by applying the weights (derived from fits to the tuning curves; see main text) to the individual trial firing rates in the unimodal vestibular and visual conditions, then comparing the variance of these artificial weighted-sum trials against the measured variance in the combined conditions. This comparison is plotted in Fig. S6a,b for low and high coherence, respectively (one data point per heading per neuron, Δ=0º trials only; sample includes only congruent cells that were significantly well fit by the linear model, as in Fig. 8b-e (N=43)). The predicted (artificial) variance was well correlated with, and not significantly different from, the measured variance across both levels of reliability (Spearman s rho = 0.75 for 16% coh., rho = 0.77 for 60% coh., p << for both; Wilcoxon matched pairs test, p = for 16% coh., p = 0.78 for 60% coh.). This match is not trivial because the weights change rapidly with cue reliability (Fig. 7c,d) in a manner consistent with a network nonlinearity such as divisive normalization 12 (see also Discussion and Fig. S5). With such a nonlinearity at play, one might expect some degree of information loss during cue combination, which would imply greater response variance than predicted by a weighted sum of singlecue trials. The fact that combined variance is as predicted by the weighted sum model lends further support for the role of MSTd in contributing to optimal cue integration, as it suggests the combination can occur without substantial information loss. This apparent summation of variance during cue integration is roughly analogous to a recent result 13 showing an increase in variance in area LIP across time within a trial, due to the accumulation of momentary evidence during perceptual decision-making. Although a detailed examination of this phenomenon is beyond the scope of our study, it may be that variance becomes another useful tool for diagnosing whether and how candidate multisensory neuronal populations carry out the necessary computations for sensory cue integration. 14

15 Supplementary Material, Page 15 of 18 Derivation of optimal neural weights This section describes a derivation of the optimal weight ratio ρ opt (Eq. 7 in main text), which we compared to the measured weight ratio in Fig. 8a. Let f ves (θ), f vis (θ,c), and f comb (θ,c) represent the mean response (tuning function) of a particular cell for the three respective stimulus modalities as a function of heading (θ) and coherence (c). For mathematical convenience, we replace the neural weights A ves and A vis in Eq. 5 with a weight ratio ρ = A ves / A vis and include a scaling constant a. We then have (dropping the θ and c dependence for clarity): 15

16 Supplementary Material, Page 16 of 18 (S3) where f' refers to the derivative of the tuning functions. If we assume that the spike count variability on single-cue trials is Poisson-like (i.e., can be well approximated by a broad family of functions known as the exponential family with linear sufficient statistics 14-16, a type of variability believed to be common across cortical areas 15, 17-19, including in our MSTd recordings (Y. Gu, CRF, DEA and GCD, unpublished observations)), the variance σ 2 will be proportional to the mean, where the proportionality constant is the Fano factor (F = variance/mean). Because the variance of a sum of independent random variables is the sum of their respective variances, and applying the identity Var(aX) = a 2 Var(X), we have: (S4) What we seek is the value of ρ that maximizes linear Fisher information (the information that can be recovered by an optimal linear decoder) from the combined responses (I comb ), which is defined as 20 : (S5) To find the weight that maximizes information we need to solve: (S6) Solving this equation (note that the constants a and F drop out) gives the following expression for the optimal neural weight ratio ρ opt : (Eq. 7 in text) 16

17 Supplementary Material, Page 17 of 18 (with θ and c dependence restored, to illustrate that this ratio is evaluated at θ = 0 and computed separately for the two coherence levels (Fig. 8a)). Note: although it appears that we have derived the weights maximizing linear Fisher information only, Eq. 7 in fact maximizes the full Fisher information. Indeed, it is easy to show that for the weight ratio above, the combined responses also belong to the Poisson-like family, and for the Poisson-like family it turns out that Fisher information is equal to linear Fisher information 20. As mentioned in the main text, Eq. 7 describes the neural operation one would expect to see in an area performing optimal multisensory integration within a probabilistic population code (PPC) framework 16. By taking into account the observed tuning parameters (which define the mean of the likelihood as a function of heading), this optimal weight ratio allows a population to implicitly encode the posterior P(θ r), even if tuning is not simply scaled multiplicatively by coherence (see Eq. 6). Thus, the fact that measured weights vary with coherence (Fig. 7c,d) in a manner roughly similar to optimal weights (Fig. 8a) constitutes indirect evidence for a probabilistic coding scheme such as PPC. To estimate the terms on the right side of Eq. 7 (i.e., for Fig. 8a-c), responses in each single-cue condition were fit separately via linear regression, and the slope and intercept were taken as f' and f, respectively. Because the derivation above assumes linear tuning, these linear fits were used in place of real single-cue data for the decoding simulations depicted in Fig. 8b,c; this accounts for the slight difference in single-cue thresholds (and thus optimal weights) in Fig. 8b,c versus d,e. 17

18 Supplementary Material, Page 18 of 18 References 1. Fetsch, C.R., Turner, A.H., DeAngelis, G.C. & Angelaki, D.E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J Neurosci 29, (2009). 2. Gu, Y., Angelaki, D.E. & Deangelis, G.C. Neural correlates of multisensory cue integration in macaque MSTd. Nat Neurosci 11, (2008). 3. Jazayeri, M. & Movshon, J.A. Optimal representation of sensory information by neural populations. Nat Neurosci 9, (2006). 4. Sanger, T.D. Probability density estimation for the interpretation of neural population codes. J Neurophysiol 76, (1996). 5. Dayan, P. & Abbott, L.F. Theoretical Neuroscience (MIT press, Cambridge, MA, 2001). 6. Foldiak, P. The 'ideal homunculus': statistical inference from neural population responses. in Computation and Neural Systems (ed. F.H. Eeckman & J.M. Bower) (Kluwer Academic Publishers, Norwell, MA, 1993). 7. Shadlen, M.N., Britten, K.H., Newsome, W.T. & Movshon, J.A. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J Neurosci 16, (1996). 8. Gu, Y., et al. Perceptual learning reduces interneuronal correlations in macaque visual cortex. Neuron 71, (2011). 9. Zohary, E., Shadlen, M.N. & Newsome, W.T. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature 370, (1994). 10. Averbeck, B.B., Latham, P.E. & Pouget, A. Neural correlations, population coding and computation. Nat Rev Neurosci 7, (2006). 11. Morgan, M.L., Deangelis, G.C. & Angelaki, D.E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, (2008). 12. Ohshiro, T., Angelaki, D.E. & DeAngelis, G.C. A normalization model of multi-sensory integration. Nat Neurosci In Press. (2010). 13. Churchland, A.K., et al. Variance as a signature of neural computations during decision making. Neuron 69, (2011). 14. Beck, J., Ma, W.J., Latham, P.E. & Pouget, A. Probabilistic population codes and the exponential family of distributions. Prog Brain Res 165, (2007). 15. Beck, J.M., et al. Probabilistic population codes for Bayesian decision making. Neuron 60, (2008). 16. Ma, W.J., Beck, J.M., Latham, P.E. & Pouget, A. Bayesian inference with probabilistic population codes. Nat Neurosci 9, (2006). 17. Shadlen, M.N. & Newsome, W.T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J Neurosci 18, (1998). 18. Carandini, M. Amplification of trial-to-trial response variability by neurons in visual cortex. PLoS Biol 2, E264 (2004). 19. Tolhurst, D.J., Movshon, J.A. & Dean, A.F. The statistical reliability of signals in single neurons in cat and monkey visual cortex. Vision Res 23, (1983). 20. Beck, J., Bejjanki, V.R. & Pouget, A. Insights from a simple expression for linear fisher information in a recurrently connected population of spiking neurons. Neural Comput 23, (2011). 18