1 More complex brains are not always better: Rats outperform humans in implicit categorybased generalization by implementing a similarity-based strategy Ben Vermaercke 1*, Elsy Cop 1, Sam Willems 1, Rudi D Hooge 1, & Hans P. Op de Beeck 1 SUPPLEMENTARY ONLINE MATERIALS Table of contents A) Supplementary Methods Rat experiment - Matching stimulus contrast over a range of spatial frequencies - Initial shaping phase - Training phase - Testing phase part 1 - Testing phase part 2 - Discussion of 2AFC advantages - Main data analysis - Additional data analysis Human experiment - Training Procedure - Matching of stimuli between rats and humans - Data analysis B) Supplementary control analysis - Relationship between speed of learning in training phase and degree of generalization - Control data: Can rats go back to chance performance? - Possible learning effects during generalization phase C) Supplementary figures - Figure S1: Images of behavioural setups - Figure S2: Example of learning curves with sigmoidal fit D) Supplemental References
2 A) Supplementary Methods Rat experiment Matching stimulus contrast over a range of spatial frequencies Spatial frequencies ranged from a minimum of 0.05 cycles per visual degree (cpd) to a maximum of 0.30 cpd. This range was selected based on the contrast sensitivity values of rats obtained by Silveira and colleagues (Silveira et al. 1987). They found that between 0.05 and 0.30 cpd, the visual evoked cortical potential responses of pigmented rats is of the same order of magnitude with only small differences in sensitivity. We compensated for the remaining difference in sensitivity by reducing the contrast of the lower spatial frequencies. In the final stimuli, the Michelson contrast for 0.05 cpd would be 0.45 and for 0.30 cpd would be approximately 1, which would for rats result in stimuli that perceptually appear to have similar contrast (probably more so than for humans, given that this factor is never taken into account in human studies). Initial shaping phase For the shaping procedure, which is not part of the actual experiment, we used two very easy stimuli (black vs. white screen) and first dropped the animal right in front of the platform. In this phase they had to learn that a platform can be found somewhere and that this is the only way out of the water maze. Consequently, we dropped them gradually further away from the screen, until they were placed beyond the divider. At this time, the animal had to make a decision in which arm to look first. The platform was always in front of the white screen, but platform positions were chosen randomly on each trial. All rats learned to solve this task after a week of two times 10 trials per day. Training phase After the animals learned the shaping task requiring them to discriminate a white from a black screen (94% correct averaged across all animals, worst animal: 88 % correct), we started the first phase of the actual experiment. Each animal was presented with the first pair of gratings, taken from the centre of the stimulus space, which was rotated according to the condition (RB or II; see Fig. 1A). 28 sessions of 12 trials were needed to get all rats to a steady performance level above the 85 percent correct criterion. We set this criterion fairly high, because we expected the rats to show a decrease in performance in the next phase for some combinations of stimuli. Most of this time the gratings had a constant phase (so non-variable position of whiter and darker regions), but at the end of training we randomized phase. Rats generalized to the random phase without response decrement, indicating that they were not focusing upon specific screen locations but processing the stimuli in terms of global dimensions such as orientation and spatial frequency.
3 Testing phase part I In this phase we first expanded the stimulus set with 4 new gratings which were located most distantly from the centre (scheme shown in Fig. 1B; exact stimuli shown in the bottom row of Fig. 1A). These 6 gratings were combined in 9 pairs that were presented in every session. These fixed pairs were interleaved with 3 pairs resulting from a combination of stimuli that had a position in stimulus space random in between the original central stimulus and the extremes. We included these random pairs to avoid that the animals would learn simple stimulus-response mappings. Part I of the testing phase was stopped after 10 sessions (total of 120 trials per rat). These are the data reported in the main paper. Testing phase part II The difference between the two categories along the relevant direction in stimulus space was now reduced to only 44% of the original difference. In addition, we also completely randomized stimulus values on the orthogonal dimension (irrelevant direction in stimulus space), resulting in a more continuous distribution of stimuli (as is also typically done in published human studies, see e.g. Helie et al. 2010). Even in these more challenging conditions, rats did not show any difference in performance between RB and II conditions (unpaired t-test, t(13)=-0.77; p=0.451). These data are not discussed in the main paper. Advantages of the V-Maze setup from a signal detection point of view The setup essentially implements a two-alternative forced choice (2AFC) task, a common paradigm in the context of signal detection theory (Green and Swets 1966). The main advantages of using this task is that subjects are less likely to show a large response bias and that it yields better performance than identification experiments with equal stimulus differences (Macmillan and Creelman 2005). Main data analysis Trials were scored automatically using a top-view camera. The computer scored the response of the animal when it fully crossed an imaginary line drawn at the divider. If the animal entered the correct arm, the trial was considered to be solved successfully. In the training phase, we analysed the learning curve of individual animals across sessions (12 trials each). The data points (average performance per session) were used to fit a sigmoid function. Further analyses of the training phase include the data of all sessions of a rat after the point at which his individual sigmoid function reached 85% correct performance. For the testing phase (part 1), we used averaged data from all 10 sessions per animal.
4 We calculated proportion correct for each animal and grouped them in two conditions: RB versus II category learning tasks. These data were analysed in a separate ANOVA for both training and test phases. To check whether animals were using a similarity based strategy, we divided all stimulus pairs into five distinct subsets (see Fig. 1B). Subset 1 contained the original training pair. Subset 2 contained pairs where the original target was used, while the distractor was unseen. Subset 3 contained pairs that shared the distractor with the original training pair, while the target was unseen. Subset 4 contained pairs that consisted of two unseen stimuli that shared the same value on the irrelevant direction in stimulus space (straight pairs). Subset 5 contained pairs that consisted of two unseen stimuli that differed maximally on the irrelevant direction in stimulus space. Additional data analysis To assess possible effects of over-training on generalization, we correlated the amount of sessions every animal performed above 85% (ranging between 12 and 24 sessions) with the average generalization performance on the subset of pairs that contained one, two and either one or two unseen stimuli (resp. subsets 2-3, 4-5 and 2-5, described above and indicated in Fig. 1B). Human experiment Training Procedure Human subjects performed 600 trials in one single session; of these only 156 trials (36 train + 120 test) were used, the remaining trials corresponded to testing phase part II which is not further discussed in this study. First, we included a short training phase with 36 trials that, as in rats, included a basic discrimination task between two central stimuli, one for each category (same design used for the rats as explained in Fig. 1A). Instructions were given by the experimenter that there were good (target) and bad (distractor) patterns and they had to detect these by trial-and-error. Subjects were given feedback (correct: green triangle pointing up; incorrect: red triangle pointing down) whether they responded correctly, thus allowing them to learn which prototype constituted the target. Because of a bug in the program, there was one difference between human and rat stimuli, which is that configuration 4 was presented to all human subjects in the information integration condition (and never configuration 2); there is no reason to expect this to affect our results because there was no difference in generalization between rats trained with configuration 2 (98.5% correct) and rats trained with configuration 4 (95.1% correct; t(6)=1.79; p=0.122). At the end of this training, most human subjects typically mastered this discrimination as well as the rats did at the end of the training phase. Then, human subjects completed the testing phase, during which we presented 120 patterns (cf. 10 sessions of 12 trials that were collected from the rats) and which also included feedback. The
5 number of trials and the stimuli in this human testing were perfectly matched with Testing Phase Part I in rats. Matching of stimuli between rats and humans First, we calculated that the acuity in terms of cycles per stimulus for the rats, given their reported visual acuity of 1.5 cpd at best (Prusky et al. 2002), equals at most 40 cycles per stimulus. In a grating detection experiment, we determined the threshold acuity for humans at about 12 peripheral using a staircase procedure aiming at a performance level of 75%. The cut-off frequency was about 5cpd for the timing settings used in all our human experiments, which is consistent with the literature (Thomas 1978). This yields an estimated 30 cycles per stimulus for human subjects with the stimulus size used in the experiment. This value was taken lower for humans on purpose, because we wanted to make sure the stimulus acuity for humans was equal or worse than for the rats, so that the stimulus visibility for rats would be in the range covered by our present human study (humans slightly worse than rats) and by previous human studies (much better stimulus visibility than for rats). Analysis See analysis methods section for the rats, as we used the same statistical techniques to analyse the data from human subjects. For the training phase, we divided the trials in sessions of 12 trials as for the rats (even though humans completed all these sessions successively in one day), so that the exact same procedure of fitting a sigmoid function could be used. Two of the human subjects were clear outliers and did not reach significant above chance performance in the training phase, and were excluded from all further analyses. Given the low difficulty of the discrimination task in this training phase, and the fact that the participating students receive the course credits for this experiment independently from their performance, this low performance was most likely due to lack of motivation.
6 B) Supplementary control analyses Relationship between speed of learning in training phase and degree of generalization In rats there was a large inter-individual variability in the number of sessions needed to reach the criterion performance (range 5-17 sessions); two representative examples are shown in Fig. S2. Given that we switched all rats at the same time to the test phase, some rats were much longer at the desired performance level than other rats and than any of the humans (the total duration of the training phase was much shorter for humans). Might the variability in learning speed and the associated variability in time since the task was mastered be related to the performance of the rats in the generalization phase? To answer this question, we searched for significant correlations between the number of sessions performance of animals was above criterion and performance in the generalization test phase on pairs consisting of one (subsets 2&3) unseen stimulus (r = -0.12, p =.65), two (subsets 4&5) unseen stimuli (r = 0.20, p =.45), and the average of all these subsets of pairs (r = 0.09, p =.74). Given that none of these correlations approaches significance, we conclude that the learning speed of the animals or the associated overtraining in some animals did not influence generalization behaviour during the test phase. This finding provides empirical evidence against the possibility that overtraining might explain the superior generalization of rats in the II conditions. Evidence in the literature already suggested that this would be unlikely. Overtraining in learning situations typically results in worse generalization instead of better (this phenomenon is often referred to as over-fitting in the neural networks literature, see e.g. Weigend 1994; Sarle 1995).
7 Control data: Can rats go back to chance performance? Although our results were consistent with the predictions derived from the COVIS framework, we wanted to exclude two possible alternative explanations. First, maybe rats had discovered a bug in our set-up. Maybe the stimuli were not the only cue for which arm contained the platform. The most likely bug would be that the platform might not have been perfectly invisible. The effect of similarity is an assurance that the rats were still paying attention to the stimuli, but nevertheless part of their performance might be related to such a confound. Second, maybe the rats had suddenly become extremely fast learners. Given that the new stimuli in the Testing phase were rewarded from the first trial on (always one platform present), a perfect learner would only need one trial to pick up the category associated with each stimulus (note that feedback was also presented to human subjects, and they did reach perfect categorization performance for all stimuli). Both possibilities were extremely unlikely given the data from the shaping phase and the first part of the Training phase, and given previous experiments that we had already done using this same set-up. Nevertheless, to be absolutely sure, the same animals started with a new discrimination experiment in which they had to discriminate a square from a triangle which is a task that rats can readily learn (e.g., Minini and Jeffery 2006). This control experiment started immediately after the end of the category learning experiments, and was performed by the same experimenters. The average luminance of these new stimuli was the same as for the gratings and they were also presented through a circular aperture (see Fig. S1), so platform (in)visibility should be the same. The performance of each of the 16 rats fell back to around 50% chance performance on the first day with the new stimuli. This excludes the possibility that the rats could see the platform or that they were using a cue not related to the exact stimuli. In addition, the rats needed many training sessions before their performance rose above 50% with an average performance of only 67% on the 6th session and of 87% on the 9th session. Thus, the rats are still slow learners after the category learning experiment. In sum, the results of this control experiment strengthen our conclusion that the surprisingly good performance of the rats in the test phase is due to how they generalize from the old to the new stimuli.
8 Possible learning effects during human generalization We investigated the possibility that our animal and human subjects showed learning effects during the generalization phase. Ideally of course, no feedback would be given during this test phase, to avoid any new learning. But in our case, it was impossible to withhold feedback from the rats, there would always have to be a platform in order for them to escape. For this reason, we also included feedback during the generalization phase of the human experiment. We looked at groups of 12 trials, which corresponds to actual experimental sessions for the rats, while for humans this grouping is more artificial as all trials were collected within the one session. Then we calculated average performance of all subjects and compared RB and II groups. We used a 2 (CatType: RB or II) by 10 (SessionNr: 1-10) repeated measures ANOVA. The number of subjects was 22 for the human analysis and 16 for the rat analysis. For the humans, we found a main effect of CatType (F(1,200)=90.1429; p<0.0001), no main effect of SessionNr (F(9,200)=0.4956; p=0.8765) and no interaction effect between both (F(9,200)=0.4113; p=0.9282). For completeness, we repeated the same procedure for rats, here we found no main effect of CatType (F(1,140)=3.6228; p=0.0590), no main effect of SessionNr (F(9,140)=1.4244; p=0.1831) and no interaction effect between both (F(9,140)=0.3408; p=0.9598). Taken together, we find no evidence for learning effects in any of the two groups during this phase of the experiment. We cannot exclude that after many more sessions the difference between RB and II human groups would eventually disappear (Helie et al. 2010) due to improvements in the II group, however, this was beyond the scope of this study.
9 C) Supplementary Figures Figure S1: Images of behavioural setups for rats (panels A and C) and humans (panels B and D). Figure S2: Learning curves in the Training phase with sigmoidal fit for two example rats, (A) a fast learner and (B) a slow learner. Grey rectangles indicate the training sessions used to calculate performance as shown in Fig. 2.
10 D) Supplemental References Green DM, Swets JA. 1966. Signal detection theory and psychophysics. John Wiley, Oxford, England. Helie S, Waldschmidt JG, Ashby FG. 2010. Automaticity in rule-based and information-integration categorization. Atten Percept Psychophys 72: 1013-1031. Macmillan NA, Creelman CD. 2005. Detection Theory: A User's Guide (2nd ed.). Lawrence Erlbaum Associates, Mahwah, N.J. Minini L, Jeffery KJ. 2006. Do rats use shape to solve "shape discriminations"? Learn Mem 13: 287-297. Prusky GT, Harker KT, Douglas RM, Whishaw IQ. 2002. Variation in visual acuity within pigmented, and between pigmented and albino rat strains. Behav Brain Res 136: 339-348. Sarle WS. 1995. Stopped Training and Other Remedies for Overfitting. Proceedings of the 27th Symposium on the Interface of Computing Science and Statistics: 352-360. Silveira LC, Heywood CA, Cowey A. 1987. Contrast sensitivity and visual acuity of the pigmented rat determined electrophysiologically. Vision Res 27: 1719-1731. Thomas J. 1978. Normal and amblyopic contrast sensitivity function in central and peripheral retinas. Invest Ophthalmol Vis Sci 17: 746-753. Weigend A. 1994. On overfitting and the effective number of hidden units. Proceedings of the 1993 Connectionist Models Summer School: 335-342.