Encoding model of temporal processing in human visual cortex

Transcription

1 Encoding model of temporal processing in human visual cortex Anthony Stigliani a, Brianna Jeska a, and Kalanit Grill-Spector a,b, PNAS PLUS a Department of Psychology, Stanford University, Stanford, CA 95; and b Stanford Neurosciences Institute, Stanford University, Stanford, CA 95 Edited by Thomas A. Carlson, University of Sydney, and accepted by Editorial Board Member Marlene Behrmann November, 7 (received for review March, 7) How is temporal information processed in human visual cortex? Visual input is relayed to V through segregated transient and sustained channels in the retina and lateral geniculate nucleus (LGN). However, there is intense debate as to how sustained and transient temporal channels contribute to visual processing beyond V. The prevailing view associates transient processing predominately with motion-sensitive regions and sustained processing with ventral stream regions, while the opposing view suggests that both temporal channels contribute to neural processing beyond V. Using fmri, we measured cortical responses to time-varying stimuli and then implemented a two temporal channel-encoding model to evaluate the contributions of each channel. Different from the general linear model of fmri that predicts responses directly from the stimulus, the encoding approach first models neural responses to the stimulus from which fmri responses are derived. This encoding approach not only predicts cortical responses to time-varying stimuli from milliseconds to seconds but also, reveals differential contributions of temporal channels across visual cortex. Consistent with the prevailing view, motion-sensitive regions and adjacent lateral occipitotemporal regions are dominated by transient responses. However, ventral occipitotemporal regions are driven by both sustained and transient channels, with transient responses exceeding the sustained. These findings propose a rethinking of temporal processing in the ventral stream and suggest that transient processing may contribute to rapid extraction of the content of the visual input. Importantly, our encoding approach has vast implications, because it can be applied with fmri to decipher neural computations in millisecond resolution in any part of the brain. fmri V V hmt transient How does the visual system process the temporal aspects of the visual input? In the retina () and lateral geniculate nucleus (LGN) ( ), temporal processing is thought to be mediated predominately by a magnocellular (M) pathway distinguished by its large transient responses (, ) and a parvocellular (P) pathway, which has larger sustained responses than the M pathway (, ) [in addition to a smaller koniocellular (5) pathway]. While M and P pathways remain segregated up to striate cortex (V), there is intense debate as to how these pathways contribute to visual processing in extrastriate cortex. The prevailing view suggests that the dorsal stream, particularly motionsensitive middle temporal (MT), is M-dominated (6 8), and that the ventral stream, particularly V, is P-dominated (9). However, based on evidence for M and P contributions to both V (5, 9) and MT (, ), an opposing view suggests that these pathways are not segregated in extrastriate cortex (5, 8). Since M and P pathways are associated with transient and sustained responses, respectively, these theories make predictions regarding temporal processing in human visual cortex. The prevailing view predicts that human MT complex () will have large transient but small sustained responses and conversely, that human V (hv) will have large sustained but small transient responses. However, the opposing view predicts substantial transient and sustained responses in both and hv. These predictions are derived from studies of the macaque brain; whether the same predictions can be made for the human brain is uncertain, because the organization of human visual cortex differs from the macaque in three notable ways. (i)vand MT neighbor in the macaque brain, but hv and are separated by cm in the human brain; (ii) whether macaque V and hv are homologous is subject to debate ( 5); and (iii) the human brain contains several additional visual regions neighboring hv and that are not found in the macaque [VO-/VO- () and LO-/LO- (6), respectively]. Thus, generating a complete model of temporal processing in human visual cortex necessitates measurements in humans. Understanding temporal processing in human visual cortex has seen little progress for two main reasons. First, the temporal resolution of noninvasive fmri measurements is in the order of seconds (7), an order of magnitude longer than the timescale of neural processing, which is in the tens to hundreds of milliseconds range. Second, while fmri responses are largely linear for long stimulus presentations (8, 9), they exhibit marked nonlinearities for short and transient stimuli (8 ). Since the general linear model (GLM) for fmri (8, ) is thought to be inadequate for modeling responses to such stimuli and fmri is slow, the temporal processing characteristics of human visual cortex has remained elusive. If the observed nonlinearities are of neural [rather than blood oxygen level dependent (BOLD)] origin, a new encoding approach applied to fmri (5 7), which uses computational models to predict neural responses (even if they are nonlinear), Significance How is temporal information processed in human visual cortex? To address this question, we used fmri and a two temporal channel-encoding model. This approach not only explains cortical responses for time-varying stimuli ranging from milliseconds to seconds but finds differential temporal processing across human visual cortex. While motion-sensitive regions are dominated by transient responses, ventral regions that process the content of the visual input surprisingly show both sustained and transient responses, with the latter exceeding the former. This transient processing may foster rapid extraction of the gist of the scene. Importantly, our encoding approach marks a transformative advancement in the temporal resolution of fmri, as it enables linking fmri responses to the timescale of neural computations in cortex. Author contributions: A.S. and K.G.-S. designed research; A.S. and B.J. performed research; A.S. and K.G.-S. analyzed data; and A.S. and K.G.-S. wrote the paper. The authors declare no conflict of interest. This article is a PNAS Direct Submission. T.A.C. is a guest editor invited by the Editorial Board. This open access article is distributed under Creative Commons Attribution-NonCommercial- NoDerivatives License. (CC BY-NC-ND). Data deposition: The data and code reported in this paper are available via the Open Science Framework (OSF) at To whom correspondence should be addressed. kalanit@stanford.edu. This article contains supporting information online at 7/pnas.7877/-/DCSupplemental. PSYCHOLOGICAL AND COGNITIVE SCIENCES PNAS Published online December 5, 7 E7 E56

2 A Experimental design Experiment : continuous image per trial Experiment : flashed images per trial Experiment : continuous images per trial B Trial durations s s 8 s,, 8, 5, or s per image ms per image; ISI =,,, 67, or 967 ms... 67,, 67 5, or ms per image 5 s s Example images Experiment Experiment Experiment Fig.. Measuring brain responses to combinations of sustained and transient visual stimuli. (A) Participants fixated centrally and viewed phasescrambled gray-level images that were presented in trials of different durations interleaved with -s periods of a blank screen. The same fixation task (detecting change in fixation color) was used in all three experiments. Experiment : a single phase-scrambled image was shown for the duration of a trial. Experiment : briefly presented images ( ms each), each followed by a blank screen, were presented in each trial. As the trial duration lengthens, the gap between images increases, causing the fraction of the trial containing visual stimulation to decrease. Experiment : continuous images (with no gaps between consecutive stimuli) were presented in each trial. As the block duration lengthens, the duration of each image progressively increases. (B) The same trial durations (,, 8, 5, or s) were utilized across all three experiments, while the rate and duration of visual presentation varied between experiments. Corresponding trials in experiments and have the same overall duration of stimulation but different numbers of stimuli, whereas trials in experiments and have the same number of stimuli but different durations of stimulation. (Upper) Stimulation durations for example trials in each experiment; (Lower) zoomed-in view of the - and -s trials. could surmount these issues. Different from the GLM method, which predicts fmri signals directly from the stimulus, the encoding approach first models neural responses to the stimulus and then, from the predicted neural responses, calculates fmri responses. The encoding approach (5 9) has been influential for three reasons: (i) it provided an important insight that accurately modeling neural responses at a subvoxel resolution better predicts fmri responses at the voxel resolution, (ii) it has advanced understanding of neural mechanisms by building explicit quantitative models of neural computations, and (iii) it can be applied to predict fmri responses to dynamic stimuli, such as movies, by using millisecond variations in the stimulus to precisely model neural responses (, ). Here, we sought to leverage the encoding approach to characterize temporal processing in human visual cortex. Thus, we built a temporal encoding model of neural responses to timevarying visual stimuli in millisecond resolution and used this model to predict fmri responses in second resolution. The model is based on estimation of the transient and sustained neural channels impulse response functions (IRFs) from measurements in macaque V ( ) and psychophysics in humans (5 8). To determine temporal processing in human visual cortex, we implemented three experiments aimed to measure fmri responses to time-varying visual stimuli that were sustained (one continuous image per trial, durations ranging from to s) (Fig., experiment ), transient [ flashed, -ms long images per trial, interstimulus intervals (ISIs) ranging from to 967 ms] (Fig., experiment ), or contained both transient and sustained components ( continuous images per trial, durations ranging from 67 to, ms per image) (Fig., experiment ). We first determined if millisecond temporal variations in visual stimuli generate substantial modulations of fmri responses in visual cortex. Then, we used experiments and data to estimate the parameters of the two temporal channel-encoding model. Finally, we evaluated how well the model predicts fmri responses to stimuli that vary in their temporal properties from milliseconds to seconds in data from experiment. Additionally, to verify the utility of the two-temporal channel model, we compare its performance with that of several other models of fmri responses (7, 8, 9, ) featuring different channel architectures and nonlinearities. After we established the model s validity, we derived the contributions of sustained and transient channels to neural responses across striate and extrastriate visual cortex to test the competing theoretical hypotheses. Results Do Millisecond Temporal Variations in the Visual Stimulus Modulate V Responses? To test the feasibility of this approach, we first examined V responses during the three experiments. Predicted fmri responses from a GLM depend only on the type and duration of stimuli. Thus, the GLM approach predicts longer responses for longer trials and identical responses in experiments and (Fig. A, blue and green, respectively), which use the same visual stimuli and trial durations and just vary by the number of images per trial ( vs., respectively). Furthermore, the model predicts that the amplitude of responses in experiments and will increase from - to 8-s trials and will remain largely the same for longer trials. While this model predicts the same response durations in experiment, it predicts substantially lower response amplitudes in experiment than experiments and, because the transient visual stimuli are presented for only a fraction of each trial. Furthermore, the model predicts a progressive decrease in response amplitude during experiment from - to -s trials as the fraction of the trial in which visual stimuli are presented decreases (from / to / of the trial) (Fig. A, red). While V responses to sustained visual stimulation in experiment largely followed the predictions of a GLM (Fig. A, blue), responses in experiments and deviated from GLM predictions. First, responses in experiment (Fig. B, green) were higher than responses in experiment for all trial durations. Second, responses to transient stimuli in experiment (Fig. B, red) were substantially higher than predicted by a GLM. In fact, V responses during - to 8-s trials of experiment were equal or higher than those of experiment, although the cumulative duration of stimulation across images in experiment was a fraction of the duration of stimulation in experiment. Third, different from the predictions of the GLM, response amplitudes in experiment did not systematically decline with trial duration but instead, peaked for the -s trials. These data show that (i) varying the temporal characteristics of visual presentation in the millisecond range has profound effects on V fmri responses and (ii) the GLM approach is inadequate in predicting measured fmri responses for these stimuli, in agreement with prior data. Furthermore, the higher responses in experiment (which has both sustained and transient visual stimulation) compared with experiments and (which have either sustained or transient stimuli, respectively) suggest that both transient and sustained components of the visual input contribute to the fmri signals, consistent with our hypothesis. An Encoding Model for Temporal Processing in Visual Cortex. To accurately predict fmri responses in all three experiments, we built a temporal encoding model of neural responses in millisecond resolution and used this model to predict fmri responses in second resolution (Fig. ). Our model consists of two neural temporal channels, each of which can be characterized by a linear systems approach using a temporal IRF (, 6 8). The sustained channel is characterized by a monophasic IRF (Fig. B, blue channel IRF), peaking at around ms and lasting E8 Stigliani et al.

3 A B Predicted V responses by general linear model 5 Exp. Exp. Exp. Trial (s): 8 5 Measured V responses (N = ) 5 Exp. Exp. Exp. Trial (s): 8 5 Time (s) Fig.. V responses to transient stimuli differ from the predictions of a GLM. (A) The GLM approach predicts the same response in trials of the same duration across experiment (blue) and experiment (green), since both present stimuli continuously for the same total duration in each trial. However, responses in experiment (red) are predicted to be much lower, because stimuli are spaced apart and are only presented for a fraction of each trial duration. Illustrations of the stimulus are schematic representations. (B) The mean V response in experiments averaged across participants. Each curve is data from a different experiment. Shaded regions around the curves indicate ± SEM across participants. In both A and B, the onsets and lengths of the trials are illustrated as thick black bars below each graph, and curves extend s before the onset and s after the offset of each trial. 5 ms; convolving this channel with a visual stimulus will produce a sustained neural response for the duration of the stimulus. The transient channel is characterized by a biphasic IRF, akin to a derivative function, with the positive part peaking at around 5 ms and the negative part peaking at around 7 ms (Fig. B, red channel IRF). A quadratic nonlinearity (squaring) is added, as both stimulus onset and offset lead to increased neural firing and consequently, increased metabolic demands (8, ). This nonlinearity produces a positive neural response when there is an onset or offset of the visual stimulus but zero response in between when the stimulus is presented for durations longer than the duration of the IRF. The predicted fmri response is generated by convolving the output of each neural channel with the hemodynamic response function (HRF) and summing the responses of the two temporal channels (Fig. C). Our procedure for evaluating the two temporal channelencoding model had three stages. First, we estimated the contributions of the two temporal channels to fmri signals using concatenated data from experiments and that were designed to largely drive sustained or transient channels, respectively. Second, we cross-validated the model by testing how well it predicted data from experiment that had both transient and sustained visual stimulation. Third, we compared the performance with two other families of models: a family of three hemodynamic models and a family of three nonlinear single-channel neural models implementing compressive temporal summation (CTS). Hemodynamic models are (i) the standard GLM approach (8, ), (ii) the hemodynamic temporal derivative (HTD) model that has been used to model temporal variability of hemodynamic responses across cortex (9) and contains two linear channels the canonical HRF and its derivative and (iii) the balloon model (7), which models changes in blood flow, volume, and oxygenation to account for nonlinearities in fmri signals. CTS single-channel neural models that were derived to explain subadditive temporal summation and adaptation effects in fmri responses are (i) compressive temporal summation with a static power law (CTS-p), (ii) compressive temporal summation with divisive normalization (CTS-n) (), and (iii) dynamic compressive temporal summation (dcts), which has a prominent onset response and a subsequent declining continued response for the duration of the stimulus (). Does a Two-Temporal Channel Model Explain V fmri Responses to Time-Varying Stimuli? Comparing the predictions of the twotemporal channel model with V responses reveals three findings. First, the two-temporal channel model containing one sustained predictor (weighted by ) and one transient predictor (weighted by β T ) generated fmri signals that tracked both the duration and amplitude of V responses in experiments and [Fig. A, compare model prediction (black) with measured V data (gray)]. Consistent with our predictions, the sustained channel accounted for the majority of responses in experiment (blue in Fig. A, Upper), while the transient channel contributed the bulk of the response in experiment (red in Fig. A, Lower). Second, the two-temporal channel model with and β T fit from experiments and (Fig. C) accurately predicted independent data from experiment (Fig. B). The sustained contribution (Fig. A and B, blue) in experiment was comparable with experiment (these experiments have the same total duration of stimulation per trial), and the transient contribution (Fig. A and B, red) in experiment was similar to experiment (these experiments have the same number of transients per trial). Since both temporal channels provided a significant contribution to V and the contributions of the two channels are additive, experiment responses were higher than both experiments and across all trial durations. Analysis of cross-validated R showed that the two-temporal channel model explained an average of 7 ± % of variance in experiment (Fig. F), although the channel weights were estimated from responses to independent data with different temporal characteristics. Third, the twotemporal channel model outperformed the other models that we tested. Notably, while all models predicted V responses to the long and sustained stimulus presentations in the first experiment, hemodynamic models failed to capture responses to A Stimulus Time (ms) 5 off on Operators Convolve Multiply Sum Channel IRFs Sustained Transient 5 Time (ms) B Neural responses nonlinearity Sustained Transient HRF Time (s) C fmri responses Sustained Transient Fig.. The two-temporal channel model. (A) Transitions between stimulus and baseline screens are coded as a step function representing when a stimulus was on vs. off with millisecond temporal resolution. In the example illustrated here, each stimulus is presented for ms and followed by a -ms blank screen. (B) Separate neural responses for the sustained (blue) and transient (red) channels are modeled by convolving the stimulus vector with IRFs for the sustained and transient channels, respectively, estimated from human psychophysics. A squaring nonlinearity is applied in the transient channel to rectify offset deflections (Materials and Methods). (C) Sustained and transient fmri response predictors are generated by convolving each channel s neural responses with the HRF and down-sampling to match the temporal acquisition rate of fmri data. The total fmri response is the sum of the weighted sustained and transient fmri predictors for each channel. To estimate the contributions (β weights) of the sustained ( )andtransient(β T ) channels in V, we fit the two-temporal channel model to data concatenated across experiments and. β T PNAS PLUS PSYCHOLOGICAL AND COGNITIVE SCIENCES Stigliani et al. PNAS Published online December 5, 7 E9

4 A Model fit Exp. ( continuous image per trial) 5 V Sustained (S) Transient (T) S + T Trial (s): Trial (s): ISI (ms): 5 B Model validation Trial (s): Img. (ms): 67 Exp. ( flashed images per trial) Exp. ( continuous images per trial) C Channel weights β T Model comparison Hemodynamic models Single-channel models Dual-channel models D Exp. =.8 R E Exp. = 8.* R F Exp. =.8* Validated R noise ceiling transient channel (Fig. D F and Fig. S). Thus, a two-temporal channel model with a linear sustained channel and a linear nonlinear transient channel predicts fmri responses to visual stimuli across a threefold range of presentation durations ranging from tens of milliseconds to tens of seconds with greater accuracy than several alternative models. Do Temporal Processing Characteristics Differ Across Intermediate Visual Areas? We next examined hv and responses to the time-varying visual stimuli in experiments, as the competing theories make different predictions regarding the contributions of sustained and transient channels to these regions. hv and illustrated distinct patterns of responses. Like V, hv showed higher responses in experiment ( continuous images per trial) than either experiment ( continuous image per trial) or experiment ( flashed images per trial). Different from V, hv exhibited equal or stronger responses to the brief transient visual stimuli in experiment than the sustained single images in experiment (Fig. 5A). Different from both V and hv, exhibited close to zero evoked responses for the sustained stimuli in experiment (except for Fig.. Sustained and transient contributions to V fmri responses. (A) Measured V responses in experiments and are plotted as the mean (white) ± SD (gray) across participants for each trial duration. Superimposed are the predictions of the two-temporal channel model (quadratic nonlinearity) (as in Fig. ) fit across data from both experiments. Blue, sustained predictor weighted by ; red, transient predictor weighted by β T ; black, prediction of the two-temporal channel model, which is the addition of the two channels. (B) Measured V responses and cross-validated model prediction for experiment. The sustained and transient predictors are weighted with and β T, respectively, fitted from experiments and. Trial durations in A and B are illustrated below the x axis, and curves extend s before the onset and s after the offset each trial. (C) The model solution ( and β T ) for V fit with the two-temporal channel model using data concatenated across experiments and. (D F) Comparison of the variance explained (R ) across three families of models for each experiment with results of corresponding ANOVA indicating the main effect of model. White, hemodynamic models (, GLM;, HTD model;, balloon model); light gray, single-channel models (, CTS-p; 5, CTS-n; 6, dcts); dark gray, dual channel models (7, two-temporal channel model with rectification; 8, two temporal channel with quadratic power law). Dashed lines indicate the noise ceiling. Open circles indicate models with significantly worse performance vs. dual channel models (P <.5). Error bars in C F indicate ± SEM across participants. F value of an ANOVA comparing model family performance (hemodynamic, single channel with a compressive nonlinearity, two temporal channel with a nonlinearity) for each experiment is indicated. *Significant difference across models (P <.). transient stimuli in the second experiment (Fig. A and Fig. SA). That is, the two-temporal channel model fit to both experiments explained an average of 65 ± % (mean ± SEM across participants) of V response variance in experiment (Fig. D) and 5 ± % of the variance in experiment (Fig. E). In contrast, the hemodynamic models fit to both experiments explained an average of 6 ± % of the variance in experiment (Fig. D) but less than ± % of the variance in experiment (Fig. E). Furthermore, although CTS models explained an average of 6 ± % of the variance in experiment (Fig. D) and 8 ± % of the variance in experiment (Fig. E), all CTS models underestimated responses in experiment (Fig. SB). Indeed, in experiment, the average cross-validated R of the two-temporal channel models was significantly higher than each of the hemodynamic or CTS models (ts >.6, ps <., paired t tests) (Fig. F). In general, the dual channel models outperformed the other families of models in all visual areas tested (Figs. S and S), with no significant differences in performance between the quadratic and rectification nonlinearities on the A Measured hv responses Exp. Exp. Exp. Trial (s): B Measured responses E Model comparison Hemodynamic models Single-channel models hv Dual-channel models.8.6 R.6.. noise ceiling Exp. =.5 Exp. = 55.9* Exp. = 6.7* R.. Exp. = 6.9* C D hv channel contributions Exp. =.* β T channel contributions β T Exp. = 7.7* Fig. 5. Differential sustained and transient contributions across hv and. (A) hv and (B) responses for experiment (blue), experiment (red), and experiment (green). Curves show the mean (solid line) ± SEM across participants (shaded area). Trial durations are indicated by the thick black bars below the x axis, and curves extend s before the onset and s after the offset each trial. The two-temporal channel model solution ( and β T ; quadratic nonlinearity) for (C) hv and (D) is plotted with error bars representing ± SEM across participants. (E) Comparison of the variance explained (R ) by each of the models tested. All models were fit using data from experiments and. ANOVAs for each experiment for hv (Left) and (Right) indicate significant differences in performance across models. White, hemodynamic models (, GLM;, HTD model;, balloon model); light gray, single-channel models (, CTS-p; 5 CTS-n; 6 dcts); dark gray, dual channel models (7, two-temporal channel model with rectification; 8, two temporal channel with quadratic power law). Dashed lines indicate the noise ceiling. Open circles indicate models with significantly worse performance vs. dual channel models (P <.5). Error bars in C F indicate ± SEM across participants. F value of an ANOVA comparing model family performance for each experiment is indicated. *Significant difference across model family (P <.). E5 Stigliani et al.

5 onset and offset responses that are visible in trials of 8 s and longer) (Fig. 5B). However, showed substantial responses for transient stimuli in experiment that were comparable with experiment, which had both transient and sustained stimulation. Together, these data suggest differences in temporal processing across hv and. Next, we quantified hv and responses with the two temporal channel-encoding model. The model fits revealed (i) that, in hv, both channels contributed to responses, with the contribution of the transient channel about double that of the sustained channel (Fig. 5C) and (ii) that, in, the transient channel substantially contributed to responses but that the sustained channel had close to zero contribution (Fig. 5D). Across both hv and, the two-temporal channel models fit data from experiment better than the hemodynamic models [ts >.88, ps <., paired t tests on R values for each region of interest (ROI)] and predicted data from experiment equally or better than all hemodynamic and CTS models (Fig. 5E). These results show that not only does the two-temporal channel model perform significantly better than a variety of alternate models at intermediate stages of the visual hierarchy but that the contributions of transient and sustained channels differ across hv and. What Is the Topology of Sustained and Transient Channels Across Visual Cortex? To complement the ROI approach, we next visualized the spatial topology of the sustained and transient channels across visual cortex, which enables mapping channel contributions at the voxel level. Examining the contribution of sustained and transient channels across ventral and lateral occipitotemporal cortex revealed two main findings. First, lateral occipitotemporal cortex was devoid of contributions from the sustained channel but had substantial contributions from the transient channel (Fig. 6 A and B). This effect was widespread and not only included voxels in, as predicted by the prior analysis, but also, extended (i) posteriorly into portions of lateral occipital areas LO- and LO- and (ii) ventrally into the inferior occipital gyrus and lateral fusiform gyrus. Dorsal regions along the intraparietal sulcus also showed negligible sustained responses (Fig. 6B). Second, in ventral occipitotemporal cortex, regions along the posterior collateral sulcus and medial fusiform gyrus (where hv, VO-, and VO- are located) showed both transient and sustained responses, with larger contributions from the transient than sustained channel (Fig. 6A). Third, a similar pattern of results was observed when participants viewed complex stimuli, such as faces, bodies, and pseudowords (Fig. S). This indicates that the differential contributions of the sustained and transient channels to early and intermediate regions generalize across stimuli. We quantified the mean contributions of transient and sustained channels across visual areas spanning occipitotemporal cortex. Our results showed differences in the contributions of sustained and transient channels across early visual cortex (V V), ventral occipitotemporal cortex (hv, VO-, and VO-), lateral occipitotemporal cortex (LO-, LO-, and ), and dorsal occipitotemporal cortex [VA and VB; significant temporal channel by cluster interaction, F, =.9, P <., two-way ANOVA on β weights with factors of temporal channel (sustained/transient) and visual cluster (early/ventral/lateral/ dorsal)] (Fig. 6C). From V to higher-order areas, there was a larger drop in the contribution of the sustained channel than the transient channel (Fig. 6C). Nevertheless, there were significant differences among clusters: in ventral areas, both sustained and transient channels contributed to responses, but in lateral areas, responses were largely dominated by the transient channel (Fig. 6C). As the overall amplitude of responses varied across regions, we quantified the relative contribution (ratio) of sustained to transient channel across regions (Fig. 6D). Dorsal and lateral regions have a larger transient preference than V, as predicted by prior research. However, it is surprising that ventral regions hv, VO-, and VO- also have a greater transient preference than V (Fig. 6D and Fig. S). A Channel contributions across ventrolateral visual cortex Transient contribution (β T ) Sustained contribution ( ) β - - VO- VO- hv LO- LO- LO- IPS- VA VB LO- VO- VO- hv LO- LO- LO- B Channel contributions across dorsolateral visual cortex Transient contribution (β T ) Sustained contribution ( ) C β Sustained - - Channel contributions across visual field map clusters D LO- Early visual Ventral Lateral Dorsal LO- VB T hv VO- VA VO- V V V IPS- VA LO- VB Contribution ratio ( / β T ).5 V V V hv VO- VO- LO- LO- VA VB / β T V / β T hv / β T Fig. 6. Differential transient and sustained contributions across visual regions. (A) Ventrolateral view of occipitotemporal cortex (Inset) depicting groupaveraged (n = ) maps of the contributions of transient (Left) and sustained (Right) channels. We estimated β weights for each channel in each voxel in each participant s native brain space; β-weight maps were then transformed to the FreeSurfer average brain using cortex-based alignment and averaged across participants in this common cortical space. The resulting group maps were thresholded to exclude voxels with weak contributions (. > β >.). Boundaries of ventral and lateral regions (black) are drawn from the Wang atlas (69), with as the union of TO- and TO-. (B) SameasA for the dorsolateral view (Inset). (C) Contributions (β weights) of transient (x axis) and sustained (y axis) channels to each visual area asestimated by the two-temporal channel model. Marker size spans ± SEM across participants in each axis, and β weights were solved by fitting the two temporal channel using data concatenatedacross experiments and. (D) The ratio of sustained to transient channel contributions was computed for each subject and visual area. Data show the average across subjects per area. A ratio of one indicates no preference, a ratio of less than one indicates a transient preference, and a ratio equal to zero indicates no sustained responses. Colored dots indicate a significantly (P <.5) different ratio vs. V (gray), hv (purple), and (orange). PNAS PLUS PSYCHOLOGICAL AND COGNITIVE SCIENCES Stigliani et al. PNAS Published online December 5, 7 E5

6 The spatial topology of sustained and transient channels also revealed differences in temporal processing within regions. Specifically, in early visual cortex (V V), the sustained channel was robust in eccentricities < but declined in more peripheral eccentricities (Fig. 7A, Right). In contrast, the transient channel contributed to responses across a larger range of eccentricities that extend farther into the periphery (> ) (Fig. 7A, Left). We quantified these effects by measuring the contributions of the two channels across eccentricities using uniformly sized disk ROIs defined along the horizontal meridian representations in V and V/V (Fig. 7B). This quantification showed that, in early visual areas, the magnitude of the sustained channel declined more rapidly with eccentricity than the transient channel to the extent that, at eccentricities of, there still was a.9 ±.7% transient response but less than.6 ±.7% of a sustained response (Fig. 7B). Furthermore, the decline of the sustained channel with eccentricity occurred more rapidly in V/V than V. Together, we find a differential contribution of transient and sustained channels across eccentricities and areas [significant three-way interaction of temporal channel (sustained or transient), visual area (V or V/V), and eccentricity (5,,, or ), F, =.8, P <.5, three-way ANOVA on β weights]. Discussion Our results show that a two-temporal channel model of neural responses, containing a sustained linear channel and transient channel with a nonlinearity, is a parsimonious encoding model that predicts fmri responses in human visual cortex to visual stimuli across a broad range of durations from tens of milliseconds to tens of seconds. Critically, our data address the ongoing debate regarding the contribution of sustained and transient channels in extrastriate cortex. Consistent with the prevailing view (, 6, 7), we find that the transient channel dominates responses and peripheral eccentricity representations of V (8). Importantly, we show that this temporal processing characteristic extends to human lateral occipitotemporal cortex as well as peripheral eccentricity representations of V and V. A Channel contributions across V-V β - Transient contribution (β T ) V Vv Vv 5 Sustained contribution ( ) V Vv Vv 5 B Channel contributions across eccentricities in early visual cortex Sustained V V/V 5 5 Transient β T Fig. 7. Differential transient and sustained contributions across central and peripheral eccentricities. (A) Medial cortical surface zoomed on the occipital lobe (Inset) depicting group-averaged (n = ) maps of the contributions of transient (Left) and sustained (Right) channels. We first estimated β weights of each channel in each voxel in each participant s native brain space; β-weight maps were transformed to the FreeSurfer average brain using cortex-based alignment and averaged across participants in this common cortical space. The resulting group maps were thresholded to exclude voxels with weak contributions (. > β >.). Regional boundaries (black) and eccentricity bands (white) of early visual areas are derived from the Benson atlas (7). (B) Contributions of transient (x axis) and sustained (y axis) channels across eccentricities along the horizontal representation in V (grays) and V/V (blues) as estimated by the two-temporal channel model. Eccentricities range from 5 (lightest markers) to (darkest markers). Marker size spans ± SEM across participants in each dimension, and β weights were solved by fitting the two temporal channel using data concatenated across experiments and. In contrast to the prevailing view, we find that both sustained and transient channels drive responses in not only hv but also, ventral occipitotemporal regions VO- and VO-, with a surprisingly larger transient preference in ventral regions compared with V. This finding argues against the view that the ventral stream primarily codes static visual information and suggests a rethinking of the role of transient processing in the visual system. One prior experiment (8) had used a similar two-temporal channel approach to model luminance transients in visual cortex during an experiment in which subjects were placed in the scanner with semitransparent, hemisphere-shaped diffusers over their eyes and presented with alternating -s blocks of low and high luminance with no spatial contrast. Because the study by Horiguchi et al. (8) contained a single experimental condition, their data do not enable making inferences about the validity of the model to other conditions or stimuli. In contrast, our study used an innovative experimental design that revealed four important findings. (i) A key finding of our study is that the twotemporal channel model explains the observed nonlinearities in fmri responses for transient and rapid visual stimuli, resolving an outstanding gap in knowledge in understanding BOLD signals. We made the data and the code implementing this model freely available: (ii) We report differences in the contributions of the sustained and transient channels across multiple visual areas in the human brain. (iii) We tested a range of conditions within and across experiments and show that the two-temporal channel model not only predicts the responses to data (experiment ) but that it generalizes across a variety of low- and high-level visual stimuli. (iv) We quantitatively evaluated the performance of this model compared with other models (7, 8, 9, ) and show that it outperforms them, especially for conditions containing brief and rapid visual stimulation. Differential Transient and Sustained Responses Across Visual Cortex. Our research fills a large gap in knowledge regarding temporal processing in human visual cortex by showing (i) that the twotemporal channel model is applicable to at least additional visual areas beyond V (, 8, ) and (ii) that temporal processing is a key functional attribute that differentiates visual areas. Our observation that responses to sustained stimuli are close to zero is consistent with the prevailing view (i) that is involved in processing visual dynamics rather than static information and (ii) that inputs to are M-dominated (7). Notably, we found that neighboring regions, LO- and LO-, also have close to zero sustained responses. This finding is interesting, because LO-, which is thought to be involved in visual processing of objects (6, ) and body parts (), shows more robust responses to rapidly presented stimuli compared with nearby category-selective regions (5). These data suggest that this characteristic may be an outcome of a dominant transient channel a hypothesis that can be tested in future research using more optimal stimuli to drive this region. Inconsistent with the prevailing view, we found that hv showed not only sustained responses as expected (8,, 6, 7) but also, large transient responses. While this observation is consistent with reports that macaque V receives both P and M inputs (5, 9), it is unexpected that the transient preference in hv is larger than V. Furthermore, unlike the macaque brain, where portions of V are adjacent to MT, hv is cm away from, indicating that this finding cannot be explained by the proximity of these two regions. While the sustained channel is often associated with coding static visual input and the transient channel is often associated with coding visual dynamics (6, 8, 9), visual transients can also indicate changes to the content of the visual input. Indeed, in our experiments, transients occurred when stimuli changed (i.e., when a new image was shown or an image was replaced by a uniform gray screen). Since the E5 Stigliani et al.

7 function of the ventral stream is to derive the content of the visual input, fast, transient visual processing in ventral stream regions may enable rapid processing of visual changes (5), which in turn, may foster detection of novel stimuli and rapid extraction of the gist of the visual scene. It is interesting that temporal processing in dorsolateral occipitotemporal regions, like that of far peripheral eccentricities (> ) in early visual cortex, was dominated by the transient channel, and temporal processing in ventral occipitotemporal regions, like central eccentricities in early visual cortex, showed a dual channel contribution. These functional characteristics may be anatomically supported by white matter connections from peripheral representations of early visual areas to MT and nearby regions (5) that are separate from white matter connections from central representations of early visual cortex to ventral occipitotemporal regions (5). Furthermore, our observations of diminished sustained responses in the periphery of early visual cortex are consistent with prior findings showing reduced P inputs (5) and diminished sustained luminance responses (8) in peripheral compared with central V as well as faster perception in the periphery (5). Implications for Modeling fmri Signals: Millisecond Timing Matters. Our data have important implications regarding modeling fmri signals and understanding temporal processing in the human brain, because they show (i) that varying the temporal characteristics of the visual stimulus in the milliseconds range has observable effects on fmri responses in the seconds range and (ii) that, by considering the contribution of a transient neural channel, an encoding model can account for nonlinearities in fmri responses for rapid and short visual stimuli (8,, 55). It would be interesting to extend our design using rapid eventrelated paradigms and accelerated MRI acquisitions to measure and model responses to single brief images in future studies. Our data extend the original GLM of fmri signals (8, ) by showing the importance of modeling the temporal properties of neural responses at millisecond resolution to accurately predict fmri signals. In their original study, Boynton et al. (8) noted that, for short durations ( 6 s), fmri responses deviated from the predictions of the linear model, which underestimated fmri signals. These nonlinearities are exacerbated in experiments using even shorter stimuli [/ s (,, 55)]. Likewise, in experiment of our study, linear hemodynamic models fail to account for the large fmri responses to brief, transient stimuli (Fig. and Figs. S S). Boynton et al. (8) suggested that nonlinearities in BOLD responses, neural adaptation, or transient neural responses may explain deviations from linearity in fmri responses. We favor the interpretation that transient responses account for nonlinearities for four reasons. (i) Taking into account the neural transient channel resolves this nonlinearity and can predict not only our fmri measurements but also, prior data showing nonlinearities () (Fig. S5). (ii) Adaptation would have resulted in declining responses during long trials of continuous presentation of a single stimulus (5). However, we observed negligible adaptation in striate (56) and extrastriate (e.g., hv) areas, even during the -s single continuous image trials. (iii) While the family of single-channel CTS models examined here was developed to model subadditive accumulation of fmri responses caused by adaptation (), their generalization across a range of temporal conditions is more limited than the two-temporal channel model (Fig. and Figs. S S). (iv) The balloon model, explicitly derived to model neurovascular nonlinearities in the BOLD signal, did not account for experimental conditions containing transient stimuli. Of the other models tested, dcts () came closest in performance to the two-temporal channel model. Like the two-temporal channel model, dcts has a prominent onset response. Different from the two-temporal channel model, it predicts a declining continuous response across the duration of the stimulus and predicts no offset response. Notably, the two-temporal channel model has several advantages over dcts. (i) It is a more parsimonious model: across ROIs and experiments, its performance is equal or better than the dcts. (ii) It is a more interpretable model, as it is clear which component of the response is caused by the transient and which is caused by the sustained part, which is difficult to untangle with the dcts model. (iii) It is faster and easier to solve, as the solution is analytical rather than requiring complex nonlinear optimization. While the two-temporal channel model provides a significant improvement in modeling fmri signals, we acknowledge that this model does not explain the entire variance of fmri signals (Fig. S) and does not account for all nonlinearities. For example, in experiment, the model overestimates responses in extrastriate regions during short trials (durations <8 s)(fig. S). Furthermore, temporal processing dynamics may differ across brain areas (, 5, 57 59), which may require developing area-specific temporal encoding models with area-specific neural IRFs. Additionally, while we did not find differences between two-temporal channel models with rectification vs. quadratic nonlinearities, our experiments were not designed to discriminate between specific nonlinearities. Thus, future experiments optimized for isolating on and off neural responses could distinguish these alternatives. Another direction for future research would be to implement models that consider both nonlinear neural responses and nonlinearities between neural activity and BOLD responses (6), as presently, we examined each type of nonlinearity separately. Finally, an important goal for future research would be to combine the two-temporal channel model with a spatial receptive field model (5 7) to generate a complete spatiotemporal understanding of visual responses. Given the pervasive use of the GLM approach in fmri research, our results have broad implications for fmri studies in any part of the brain. We find that timing of stimuli in the millisecond range has a large impact on the magnitude of fmri responses, which has important implications for interpreting results of studies that vary the temporal characteristics of stimuli across conditions (6, 6). Critically, we show that, rather than ignoring fast cortical processing because of nonlinearities that are not accounted for in the GLM, it is possible to generate neural predictions at subsecond resolution and use them to accurately predict fmri responses (). These encoding approaches open exciting opportunities for investigating fast cortical mechanisms using fmri in many domains, including somatosensory, auditory, and high-level cognitive processing. In sum, our experiments elucidate the characteristics of temporal processing across human visual cortex. These findings are important, because they (i) explicate the contribution of transient and sustained visual responses across human visual cortex beyond V and (ii) show that accounting for neural responses in the millisecond range has important consequences for understanding fmri signals in the second range in any part of the brain. Materials and Methods Participants. Twelve participants (six males, six females) with normal or corrected to normal vision participated in this study. All participants provided written informed consent, and the experimental protocol was approved by the Stanford University Institutional Review Board. Each individual participated in three fmri sessions: two used to fit and validate the two-temporal channel model and one session in which we conducted population receptive field (prf) mapping (5) to define retinotopic cortical regions and another experiment to define human motion-sensitive area () (6 65). Detailed materials and methods can be found in SI Materials and Methods. Temporal Channels Experiments. To obtain data that can be used to estimate and test the two temporal channel-encoding model, we introduce an fmri paradigm that estimates independent sustained and transient contributions to fmri responses across visual cortex using three experiments. All three experiments used the same stimuli, trial durations, and task and only varied PNAS PLUS PSYCHOLOGICAL AND COGNITIVE SCIENCES Stigliani et al. PNAS Published online December 5, 7 E5