Testing the Nature of the Representation for Binocular Rivalry Master s Thesis Presented to The Faculty of the Graduate School of Arts and Sciences Brandeis University Department of Psychology József Fiser, Advisor In Partial Fulfillment of the Requirements for Master s Degree by Yang Chen February, 2012
ABSTRACT Testing the Nature of the Representation for Binocular Rivalry A thesis presented to the Psychology Department Graduate School of Arts and Sciences Brandeis University Waltham, Massachusetts By Yang Chen Recently, several studies proposed a probabilistic framework for explaining the phenomenon of binocular rivalry, as an alternative to the classic bottom-up or eyedominant interpretation of it. According to the new framework, perception is generated from the observer s internal model of the visual world, based on sampling-based probabilistic representations and computations in the cortex. To test the validity of this proposal, we trained participants with four patterns of non-rival Gabor patches corresponding to four percepts in binocular rivalry settings where the diagonally oriented Gabor patches where presented at two locations. The probability distribution of the appearance of the trained patterns was set as 10%, 40%, 15% and 35% for the four percepts. We tested participants prior and posterior distributions of these four perceptions in both binocular rivalry and non-rivalry situations, where they either made judgments by what was perceived in rivalry or guessed what could possibly be the answers of Gabor orientation pairs when they saw only non-rivalry Gaussian noise. ii
Kullback Leibler divergence and resampling methods were used to compare the pretraining and post-training distributions from each individual participant. For the nonrivalry inference, three out of five participants showed significant difference between pre and post-training distributions of the four possible answers. Compared with the pre, the post-training distribution all shifted towards the target distribution manipulated in the training session in these participants. The initial dominance in binocular rivalry revealed learning effect for the same participants who learned the target distribution from nonrivalry training. In contrast, for binocular rivalry, none of the participants showed change in the distribution of four perceptions overall from pretest to posttest, suggesting no learning effect transferred from non-rivalry training. These results provided no decisive evidence that binocular rivalry is a visual process based on probabilistic representation, but suggested that the onset and subsequent periods of binocular rivalry might need to be distinguished and might result from different mechanisms. iii
Table of Contents I. Introduction...1-3 II. Methods A. Apparatus...4 B. Stimuli...4 C. Procedure...5-6 D. Participants...7 III. Results...7-10 IV. Discussion...11-12 V. Reference...13-14 iv
List of Tables and Figures Table 1...10 Figure 1...10 Figure 2...11
Testing the Nature of the Representation for Binocular Rivalry Introduction Binocular rivalry occurs when two different images are shown to two eyes and the brain cannot integrate them into one interpretation. The perception alternates between all possible interpretations, with brief transitions where overlapping disparate objects are perceived. Because of the disassociation of unvarying visual input and changing perception, binocular rivalry provides a powerful tool to investigate the uncertainty in the representation of the environment and how the brain generates conscious experience (Blake & Logothetis, 2002). Numerous studies have explored the temporal dynamics of binocular rivalry and found that they could be modulated by several low-level stimulus features, including contrast, brightness, contours and context, as well as high-level factors, such as attention and emotion (Blake & Logothetis, 2002). It is well documented that the durations of alternating dominance follow a gamma distribution (Logothetis, Leopold & Sheinberg, 1996) and that the dominance occurs in a traveling wave fashion (Blake & Logothetis, 2002). With help of these regularities and variability of the properties binocular rivalry, a number of intriguing questions can be explored to find out the mechanism behind binocular rivalry: when is the competition between perceptions solved? Is it at an early or late level of processing? Is information processed during unconsciousness? What drives the alternation of perceptions? According to the classic and conventional accounts for binocular rivalry, rivalry happens at an early stage of visual processing. A neural theory by Blake (1989) assumes that the reciprocal interocular suppression in primary visual cortex leads to the perceptual 1
dominance: the population of neurons coding dominant image inhibits that coding the suppressed image; the switch of dominance is produced by changes in the inhibitory strength due to neuronal fatigue. This hypothesis received support from psychophysical studies showing effects from the eye-of-origin information (Ooi & He, 1999, Silver & Logothetis, 2007), as well as imaging studies showing neural activity changing with the perception in monocular regions in V1 (Tong & Engle, 2001). Another neural theory (Pettigrew, 2001) proposed that the perceptual switches are driven by oscillatory activity from the brainstem that generates rhythmic fluctuations in the activity throughout the brain. Whether neural system s fatigue or oscillator activity, the low-level or bottom-up accounts for binocular rivalry implies the rhythm or stochastic property of the internal visual system drives the perception in binocular rivalry. However, growing evidence suggests a late stage of visual processing involved in resolving rivalry. Kovacs and colleagues (1996), using complementary patchworks of rival images, found that pattern coherency could drive perceptual alternations. Logothetis and colleagues (1996) observed slow alternations in perceptual dominance when rival stimuli were swapped between two eyes at a high frequency, also indicating the existence of stimulus-based competition rather than eye competition. These findings, from spatial and temporal aspect of rivalry respectively, were supported by neural evidence that binocular neurons were found at several levels of the cortex that fire following perceptual alternations of rival stimuli (Logothetis & Schall, 1989, Leopold et al., 1996). Together they imply that rival input is likely to be fully represented in the primary visual cortex and used for generating visual awareness (Kovacs et al., 1996). 2
Regarding the evidence of both top-down and bottom-up influences on perceptual multistability including binocular rivalry, Sundareswara and Schrater (2007, 2008) proposed a Bayesian inference framework for multistable perception, as opposed to a mere low-level mechanism. In a unifying Bayesian view of the brain, the percepts represent a posterior probability distribution of hypotheses of the environment, and the degree of the beliefs of these hypotheses is determined by the sensory input (likelihood) combined with a prior assumption of the world. From this perspective, the samplingbased model by Sundareswara and Schrater (2007, 2008) produced the same quantitative characteristics as human data on Necker Cube bistability, another phenomenon commonly used for bistable perception, and the variability of the model captured the between-subject variation. A recent study on multistability and perceptual inference (Gershman, Vul & Tenenbaum, 2012) suggests various characteristics in binocular rivalry, such as gamma-like alternations, piecemeal percepts and traveling waves can be explained by a visual system that approximates the posterior over all hypotheses with a set of samples. In the present study, in order to test the validity of the Bayesian inference proposal of the representation for binocular rivalry, we performed an experiment to see whether (or how) human participants distribution of interpretations in binocular rivalry changes after implicit learning of a certain distribution under a non-rivalry condition. This non-rivalry learning plays the role of the sensory input in real life that helps change the prior of the world structure, which is used later in following rivalry situations. If the bistable perception in binocular rivalry is the result of Bayesian inference, we should observe a transition of the distribution of percepts (posterior) in rivalry towards the 3
targeted distribution trained in non-rivalry situations, comparing the distributions before and after training. However, non-bayesian models will not predict such trend of distribution switch. This hypothesis is based on the assumption of the effectiveness of the learning in non-rivalry training and the transfer of the learning effect from non-rivalry to binocular rivalry, which we will also test empirically. Methods Apparatus Stimuli were displayed on a View Sonic LCD Monitor with refresh rate at 120 Hz. With the NVIDIA 3D Vision shutter glasses and paired infrared emitter, the left and right eyes received different stimuli alternatively at 60 Hz, rendering binocular rivalry. In other occasions when stimuli to the left and right eyes were identical, participants saw normal non-rival images. The resulting viewing distance was approximately 60 cm. All stimuli and the experiment were created and carried out using MATLAB (R2010a) on a PC. Stimuli The stimuli shown to each eye comprised of two Gabor patches (200*200 pixel) at two spatial locations, upper and lower (25 pixels) from the central point of the screen. Each of the Gabor patch had an orientation, either tilted 45 degree clockwise or counterclockwise from the vertical line. There was grid frame surrounding the Gabor patches, aiming at helping participants to attend within the targeted visual field. A black dot was presented in the middle of the screen while stimuli were on, to prevent participants from attending on only one Gabor patch during the experiment. Apart from 4
the Gabor patches with full contrast, another set of Gabor patches covered with Gaussian noise were used in a threshold test and mixed with rivalry trials in pre and post-training tests. The Gaussian noise was made by adding random values from a normal distribution onto each pixel of a given Gabor image. In these noisy Gabor images, only one Gabor patch at one of the two locations was presented. Procedure Threshold test A staircase threshold test starting with no noise was conducted prior to the main experiment sessions to find out the threshold level of Gaussian noise on top of Gabor patches that renders visual unawareness of participants. Participants pressed keys to indicate the location and orientation of the present Gabor patch and got feedback of the correctness until their performance reached a chance level. Pre-training and post-training test The tests of perception distribution before and after training procedure were identical. Participants viewed three types of visual stimuli, which served three difference purposes, and pressed keys to make judgment accordingly. Rivalry trials: for 16 rivalry trials, participants were shown full-contrast Gabor patches at two locations, as described above. In these 30-second-long trials, the orientations for two eyes at the same location were always orthogonal so that there was always rivalry at each location. The numbers of presented orientation combinations were equal and counterbalanced. Auditory cues for key pressing, (beeps at 600 Hz, 0.2 s long), 5
were generated and the interval lengths from a normal distribution (Mean = 3 s, SD = 0.3 s). Upon the beeps, participants pressed two keys corresponding to their present perceptions for the two Gabor patches. Non-rival noise trials: for another 48 trials (2 s long per trial), participants viewed non-rival single Gabor patch at one of the two locations. These Gabor patches were covered with Gaussian noise at a threshold level from the threshold test so that they were not aware of the orientations underneath the noise while pressing keys to judge the orientations of the noise-masked Gabors at two locations. These trials were designed to test the participants internal prior (and posterior) knowledge of the probabilistic distribution of Gabor orientations. Detect trials: there were 48 detect trials (2 sec long) where participants viewed full-contrast Gabor patches identical to both eyes (non-rival). The detect trials were used as a quality control of the performance in the other two types of trials. Performance with correctness higher than 90% was considered valid. Non-rival training procedure During the training session, participants were shown to both eyes four pairs of Gabor patches, which equal to the four possible percepts of orientation combinations (2 upper *2 lower locations). These Gabor patches were covered by a medium level of Gaussian noise and were clearly recognizable to all participants. The durations of images were generated from a gamma distribution (mean=6 s, variance=0.8) to simulate the commonly reported gamma distribution of dominance duration in binocular rivalry studies. Due to such property, the majority of images were shown with durations around 6
2 s, with a small number shown very briefly (< 1 s) and some others longer (> 4 s). The distribution of image durations was manipulated in the way that the proportion of summed time of each image in total time was set (10%, 40%, 15% and 35% respectively for the four combinations). Beeps were generated in the same way as in the test sessions, as indicators for participants to press keys for Gabor orientations at two locations. With 640 trials in total, the training session took about 20 minutes. Participants Five naïve participants were recruited from the introductory psychology course and paid subject recruiting website at Brandeis University. All participants had normal or corrected-to-normal vision. Upon the completion of the experiment, they received extra credit or money as reward. Results Accuracy analysis for the detect trials in both pre and post training tests showed that the participants were dedicated to the experiment, with correctness being higher than 90%. The accuracy for the non-rivalry noisy trials suggested that the noise covered single Gabor patches were at a threshold level to participants (correctness ranges from 33% to 60%). Comparing pre and post training distributions of percepts For the responses between two beeps, if no key or only one key was pressed, or the keys for the two Gabor patches were none of the four combinations, they were regarded as invalid and were excluded from the analysis. The distribution of percepts 7
was calculated as frequencies of the four possible percepts over the total number of valid responses. To quantify the difference between the distributions of percepts pre and post training for each individual participant, Kullback-Leibler Divergence (D KL ) (Kullback & Leibler, 1951) and resampling method were applied. Kullback-Leibler Divergence, from information theory, is a measure of the difference between two probability distributions P and Q (see equation below). Here P and Q were replaced by pre and post distributions of perceptual interpretations and D KL (Pre, Post)=1/2 [D KL (Pre Post)+ D KL (Post Pre)]. D KL (P Q)=Σ i P(i) ln (P(i)/Q(i)). (P>0, Q>0) From the data for binocular rivalry trials in the pretest, the resampling procedure generates a new set of data from the original one. Using the new probability distribution calculated from the resampled data, a distribution of D KL values between the original pretest distribution and each resampled distribution could be computed. This distribution represents how far the pretest distribution can vary from itself and helps to determine whether the posttest distribution is significantly different from the pretest distribution, compared with the D KL (Pre, Post). The significance of D KL (Pre, Post) was defined as D KL (Pre, Post) being larger than 95% of the D KL s in the distribution resulted from resampling procedure. Figure 1 shows the percepts distributions in pre and posttest and in both non rivalry and binocular rivalry conditions. Three out of five participants (Y.T., J.L., A.A. ) showed a significant change from pretest distributions to posttest distributions under non rivalry condition (Table 1, Y.T.: D KL (Pre, Post)=.8353, p<.0001; J.L.: D KL (Pre, Post)=.1935, p=.01; A.A. : D KL (Pre, Post)=.1139, p=.07, marginally significant). 8
However, no participant showed significant difference between pre and posttest distributions of percepts in binocular rivalry (ps>.05). Some researchers argue that the onset phase of rivalry is independent of sustained perceptual switches, regarding the distinctive effects shown on the two phases from both low-level stimulus features and high-level cognitive or affective factors (Stanley, Forte, Cavanagh & Carter, 2011). Analysis of the onset dominance suggests significant change from pre to post tests in the same participants that showed learning effect in the non-rivalry condition (Table 1, Y.T.: D KL (Pre, Post)=.2465, p=.06, marginally significant; J.L.: D KL (Pre, Post)=.2948, p=.05; A.A. : D KL (Pre, Post)=.5250, p<.0001). To see if the direction of changes were towards target, a measure termed here as absolute change in probability was used as an indicator of the distance between target distribution and pre (or post) test distribution. The absolute change in probability was computed as the summed absolute proportion difference between Pre (or Post) distribution and Target distribution ([0.1,0.4,0.15,0.35]) so that a smaller value means being closer to the target. Figure 2 shows such absolute changes in individual participants in all conditions. The decrease found in onset dominance in rivalry matches that in the non rivalry except for participant B.S.. No obvious decrease appeared in the sustained binocular rivalry condition, which was compatible with the results from the D KL analyses. 9
Figure 1 Perception Distributions in Five Individuals Table 1 D KL (Pre, Post) in Binocular Rivalry and Non Rivalry Conditions (Resampling size=1000) Participants Y.T. J.L. M.M. A.A. B.S. Binocular Rivalry D KL.0058.0134.0067.0213.0070 Sustained P.6190.2560.5580.0920.5090 Binocular Rivalry D KL.2465.2948.1146.5250.1539 Onset P.0640.0530.3160 <.0001.2140 Non Rivalry D KL.8353.1935.0343.1139.0344 P <.0001.0120.6510.0720.7050 10
Figure 2 Absolute Changes in Probability from the Target Distribution Discussion In the present study I found that for participants who effectively learnt the target probability distribution from non-rivalry training session, the learning effect transferred and could be reflected in the initial dominance in binocular rivalry, but not in the average dominance in subsequent periods of rivalry. However, neither conventional interocular suppression model nor the probabilistic inference model distinguishes the initial and sustained phases of rivalry. So far, no decisive evidence was found in this study to verify the probabilistic representation for binocular rivalry. At least for the sustained period of 11
rivalry, internal rhythm or stochastic properties of neural system seems to override the perceptual experience in generating the interpretations of bistable visual stimuli. The experimental design of this study had certain limitations that may explain the absence of expected results. The training session in the non-rivalry condition was relatively short (20 mins) and different in the visual form the rivalry condition (noise covered vs. fullcontrast Gabor patches). The limited time of being exposed to a target probability distribution may explain the result of no learning effect in two participants in the nonrivalry tests. Competing with a prior that results from the visual experience since eye opening, the learning effect needs to be powerful enough to overwrite the prior and even stronger to be transferred to rivalry situations. Meanwhile, there are fairly large individual variations in the perceptual dynamics of multistable vision, regarding the variations of both the internal visual systems and visual experiences among individuals. These variations make learning different for individuals, in terms of time to acquire and depth. Apart from the limited training time, the change in visual form of stimuli from training to testing (non-rival-noisy Gabor patches vs. full-contrast rival Gabor patches) might make the transfer of learning effect difficult as the visual system might treat the two types of visual experience distinctively. The reason for using such stimuli was that (1) for the rivalry condition, it has been well documented that higher contrast leads to stronger dominance, so the full-contrast Gabor patches were used to get more stable clearly distributed durations of dominance and less time of percepts of mixed patches. (2) for the training condition, the Gaussian noise covered Gabor patches were to increase the demanding level of the task so that participants would pay more attention to what was presented and that the learning would be enhanced; as to the testing sessions, noisy Gabor 12
patches at a threshold gave participants information while making perceptual decisions, which simulated the real world perceptual decision making process more closely. It has been found that task utility or relevance helped to increase initial dominance (Chopin & Mamassian, 2010), indicating the possibility of effect transfer between different tasks and visual forms. Regarding the above explanations of the absence of learning effect in rivalry, future studies will attempt using rivalry stimuli and longer duration for training, expecting an easier and more direct transfer of learning. The present results supported the idea of distinguishing the initial phase and subsequent phase of rivalry. Reviewed by Stanley et al (2011), the onset stage of rivalry biases more easily while the average of dominance in subsequent periods tend to have high unpredictability. High-level effects such as task relevance and attention increase dominance corresponding to relevant or attended target, but such effects were not assessed in sustained rivalry. Combined with our results, it seems that experience or learning plays an important role only in the decision making process upon the onset of ambiguous stimuli. For the onset dominance, one may argue that it is the perceptual memory that determines the initial reaction to ambiguity: repeated exposure to some stimuli drives the preference of the interpretations corresponding to those stimuli when encountering ambiguity; in other words, people pick the most frequently viewed choice when immediate decisions need to be made, in my case, the patterns presented 40% and 35% of the total time. Nevertheless, there is evidence that the onset dominance is not equivalent to perceptual memory (Stanley el al, 2011): in absence of perceptual memory, with longer than 10s stimulus intervals, biases in onset dominance stay stable and constant for a long period of time (Carter & Cavanagh, 2007). One may also argue that it 13
is possible to describe the onset rivalry effect from either the perspective of a low-level mechanism or a sampling-based inference process: the neurons getting adapted to the trained percepts or the internal model learning the distribution of non-rivalry patterns. But if such effect occurred for the onset perception of a rivalry stimulus, there should not be a reason it disappeared if looking at sustained rivalry. The sustained rivalry is merely a continuous form of the onset rivalry, with the same physical presentation and ambiguity as input to the visual system. Additional work is needed to explore the complex time course of binocular rivalry and it is too early to say whether the mechanisms of onset dominance and subsequent alternations of binocular rivalry are completely independent of each other. However, given this widely observed phenomenon of onset versus average dominance, the existing models should be looked into more attentively to explain how the phenomenon merges. 14
Reference Blake, R. (1989). A neural theory of binocular rivalry. Psychological Review, 96, 145-167. Blake, R., & Logothetis, N. K. (2002). Visual competition. Nature Reviews Neuroscience, 3, 13 23. Carter, O. L., & Cavanagh, P. (2007). Onset rivalry: brief presentation isolates an early independent phase of perceptual competition. PLoS ONE 2, e343. Chopin, A., & Mamassian, P. (2010). Task usefulness affects perception of rivalrous images. Psychological Science, 21, 1886-1893. Gershman, S.J., Vul, E., & Tenenbaum, J.B. (2012). Multistability and perceptual inference. Neural Computation, 24, 1-24. 4. Kovács, I., Papathomas, T. V., Yang, M., & Fehér, A. (1996). When the brain changes its mind: Interocular grouping during binocular rivalry. Proceedings of the National Academy of Sciences of the United States of America, 93, 15508 15511. Kullback, S.; Leibler, R.A. (1951). "On Information and Sufficiency". Annals of Mathematical Statistics, 22 (1): 79 86. Leopold, D., & Logothetis, N. (1999). Multistable phenomena: Changing views in perception. Trends in Cognitive Sciences, 3, 254 264. Logothetis, N. K., Leopold, D. A., & Sheinberg, D. L. (1996). What is rivalling during binocular rivalry? Nature, 380, 621 624. Logothetis, N. K., & Schall, J. D. (1989). Neuronal correlates of subjective visual perception. Science, 245, 761-763. Ooi, T. L., & He, Z. J. (1999). Binocular rivalry and visual awareness: The role of attention. Perception, 28, 551 574. Pettigrew, J. D. (2001). Searching for the switch: Neural bases for perceptual rivalry alternations. Brain and Mind, 2, 85 118. Schrater, P., & Sundareswara, R. (2007). Theory and dynamics of perceptual bistability.in B. Scho lkopf,j.platt, & T.Hoffman(Eds.), Advances in neural information processing systems, 19 (pp. 1217 1224). Cambridge, MA: MIT Press. Silver, M. A., & Logothetis, N. K. (2007). Temporal frequency and contrast tagging bias the type of competition in interocular switch rivalry. Vision Research, 47, 532 543. Stanley, Forte, Cavanagh & Carter (2011). Onset rivalry: the initial dominance phase is independent of ongoing perceptual alternations. Frontiers in Human Neuroscience, 5:140, 1-9. 15
Sundareswara, R., & Schrater, P. (2008). Perceptual multistability predicted by search model for Bayesian decisions. Journal of Vision, 8(5), 1 19. Tong, F., & Engel, S. A. (2001). Interocular rivalry revealed in the human cortical blindspot representation. Nature, 411, 195 199. 16