Bryan, Adams & Monin; Online Supplemental Material 1 Online Supplemental Material Pilot time trial Because online samples often include participants who rush through studies without paying appropriate attention (Johnson, 2005), we established a minimum completion time (MCT) criterion for inclusion in the final sample for our two online experiments (2 and 3). One conventional procedure for eliminating outliers uses distributional criteria (e.g., 2 SDs above or below the mean). This method is not suitable here, however, because completion times are limited on the low end but not on the high end there is a minimum time in which it is possible to complete an experiment in good faith but there is no practical maximum. Instead, we conducted a pilot test to determine the shortest time in which one could reasonably participate in good faith. We asked 5 colleagues who were unfamiliar with the experiment to complete the online experiment as quickly as possible while still reading all essential instructions and questions but skipping the consent form, having a coin immediately accessible for flipping, and ignoring two open-ended questions at the end of the study to ensure they completed the study as quickly as possible. The mean completion time by our rushed testers was 3.09 minutes (SD = 0.55); this was set as our MCT. We emphasize that this is not the mean time in which a good faith participant would be expected to complete the experiment. Rather, it is the mean time in which a good faith participant could possibly be expected to complete the experiment. An even more conservative criterion (but, we believe, an unrealistic one given the already conservative instructions pilot participants were given) is the time in which our fastest rushed tester completed the experiment: 2.38 minutes. We report results of analyses using the 3.09-minute MCT in the main text. Analyses using the highly conservative 2.38-
Bryan, Adams & Monin; Online Supplemental Material 2 minute criterion yield similar results and are reported in the Additional Analyses section below, as are results with no time-based exclusions. In Experiment 2, the 3.09-minute criterion excludes 5 people from the sample and the 2.38-minute criterion excludes 3. In Experiment 3, those two criteria exclude 32 and 19 people, respectively. The higher rate of time-based exclusions in this study was likely due to the ad hoc nature of the sample: Whereas participants in Experiment 2 had registered to take part in surveys regularly and had presumably set time aside for the study, the Experiment 3 sample consists of casual internet browsers who landed on the survey somewhat unexpectedly, clicking an ad they discovered while browsing Facebook. Therefore, the sample likely included a contingent of people who just clicked through quickly to see what the study was about. Analyses Using Different MCT Criteria Analyses using the 2.38-minute completion-time criterion In Experiment 2, using the highly conservative 2.38 minute time criterion, participants in the cheating condition claimed, on average, to have obtained 5.48 heads (SD=1.24) while those in the cheater condition claimed to have obtained 4.90 heads (SD=1.37), t(79)=1.97, p=0.053, d=0.44. The number of heads reported in the cheating condition was still significantly higher than the 5.00 expected by chance, t(39)=2.42, p=0.020, d=0.39, and the number of heads reported in the cheater condition was still not different from chance, t(40)=0.45, p>0.6. In Experiment 3, using the highly conservative 2.38-minute criterion, the omnibus effect of condition remained significant, F(2, 109)=4.62, p=0.012, as did the differences between the cheating and cheater conditions, t(109)=2.56, p=0.012, d=0.67, and between the baseline and cheater conditions, t(109)=2.81, p=0.006, d=0.71. The
Bryan, Adams & Monin; Online Supplemental Material 3 difference between the baseline and cheating conditions remained non-significant, t(109)=0.25, p>0.80. Further, the number of heads claimed in both the cheating and baseline conditions remained significantly higher than chance, t(39)=4.81, p<0.0005, d=0.76 and t(41)=5.15, p<0.0005, d=0.79, respectively. The number of claimed heads in the cheater condition was still not significantly different from chance, t(29)=1.47, p>0.15. Analyses with no time-based exclusions In Experiment 2, including in the sample the 3 people who completed the experiment more quickly even than our highly conservative 2.38-minute criterion also did not change the results meaningfully. Participants in the cheating condition claimed, on average, to have obtained 5.59 heads (SD=1.41) while those in the cheater condition claimed to have obtained 4.86 head (SD=1.36), t(82)=2.40, p=0.019. The number of heads reported in the cheating condition was still significantly higher than the 5.00 expected by chance, t(40)=2.65, p=0.011, and the number of heads reported in the cheater condition was still not different from chance, t(42)=0.68, p>0.5. In Experiment 3, including in the sample the 19 people who completed the experiment more quickly even than our highly conservative 2.38-minute criterion did mask our effect, however. Including them, the omnibus effect of condition is no longer significant, F(2, 128)=0.94, p=0.394, nor are the differences between the cheating and cheater conditions, t(128)=1.25, p=0.215, or between the baseline and cheater conditions, t(128)=1.16, p=0.248. The difference between the baseline and cheating conditions remained non-significant, t(128)=0.11, p>0.91. Finally, the number of heads claimed in the cheater (M=5.82), cheating (M=6.27) and baseline (M=6.23)
Bryan, Adams & Monin; Online Supplemental Material 4 conditions were all significantly higher than chance, t(38)=5.82, p<0.005, t(45)=6.27, p<0.0005, and t(48)=6.23, p<0.0005, respectively, suggesting that the 19 people who completed the experiment so quickly cheated at a much higher rate than our good-faith participants. Tests of Distribution in Studies 2 and 3 Please refer to Figures S1 and S2 for the full distribution of responses in Experiments 2 and 3. Below we present statistical analyses of these distributions beyond the mean differences reported in the main text. Test of Standard Deviations It might appear, on first glance, that the standard deviation in the cheater condition in Experiment 3 (SD=1.18) is low compared to the expected value if participants were indeed reporting their coin tosses honestly (E(s)=sqrt[Np(1- p)]=sqrt(10*.5*.5)=1.58), raising the possibility that participants in that condition opted to report the most honest-sounding number of heads (i.e., 5) without even tossing a coin. However, a computer simulation with 5,000 samples of 27 participants, each flipping a fair coin 10 times yields a 95% confidence interval for the SD of {1.16; 2.00}, which includes the observed SD. Thus the observed variability is within the range of what can be reasonably expected, and we cannot reject the null to assume a restriction of the range. Participants seem indeed to be reporting their tosses honestly in the cheater condition. Moreover, similar simulations show that the other two SDs in this study (as well as all SDs observed in Experiment 2) fall well within the expected 95% confidence intervals corresponding to those cell sizes.
Bryan, Adams & Monin; Online Supplemental Material 5 Skewness and Kurtosis To further probe this question, we tested the distributions of numbers of claimed heads in the cheater conditions of both Experiments 2 and 3 (see supplementary figures S1 and S2). There is no significant skew or kurtosis in the cheater condition distributions in either experiment, z skew =0.47, p=0.64, z kurtosis =-0.47, p=0.64 (Experiment 2) and z skew =1.57, p=0.12, z kurtosis =1.59, p=0.11 (Experiment 3). Likelihood of Reporting Exactly 5 Heads Finally, we examined directly whether there was a difference between conditions, in either Experiment 2 or 3, in the likelihood of reporting exactly 5 heads. In Experiment 2, 35.0% of cheater condition participants reported exactly 5 and 35.9% of cheating condition participants did, χ 2 (1)=0.007, p=0.93. In Experiment 3, the difference was somewhat larger but still did not approach significance: 32.4% of cheater condition participants reported exactly 5 heads, 23.3% of cheating condition participants did, and 17.0% of baseline condition participants did, χ 2 (2)=2.73, p=0.26.
Bryan, Adams & Monin; Online Supplemental Material 6 Reference Johnson, J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39, 103-129. doi:10.1016/j.jrp.2004.09.009
Bryan, Adams & Monin; Online Supplemental Material 7 Figure S1. Distributions of the numbers of heads participants claimed to have obtained in the cheating and cheater conditions in Experiment 2.
Bryan, Adams & Monin; Online Supplemental Material 8 Figure S2. Distributions of the numbers of heads participants claimed to have obtained in the cheater, cheating, and baseline conditions in Experiment 3.