Title:Mixed-strain Housing for Female C57BL/6, DBA/2, and BALB/c Mice: Validating a Split-plot Design that promotes Refinement and Reduction

Transcription

1 Author's response to reviews Title:Mixed-strain Housing for Female C57BL/6, DBA/2, and BALB/c Mice: Validating a Split-plot Design that promotes Refinement and Reduction Authors: Michael Walker Mr (mwalk04@uoguelph.ca) Carole Fureix Dr (carole.fureix@bristol.ac.uk) Rupert Palme Prof (rupert.palme@vetmeduni.ac.at) Jonathan Newman Prof (jnewma01@uoguelph.ca) Jamie Ahloy Dallaire Mr (jdallair@uoguelph.ca) Georgia Mason Professor (gmason@uoguelph.ca) Version:2Date:2 November 2015 Author's response to reviews: see over

2 Reviewer:Karl Broman Reviewer's report: The authors argue for the use of split-plot mouse experiments with co-housing of multiple strains. To provide evidence for the value of such designs, they conducted an experiment on the effects of strain and enrichment material on a broad range of outcomes. Major comments - Discussion of degrees of freedom in the background section (lines ) would probably best be moved to a subsection of the results. The typical reader will not appreciate these calculations. We have kept a simplified overview of this in the introduction as it forms the foundation of our paper (see lines ). However, the in-depth information has been removed from the introduction and put into the supplementary information online for anyone who is more interested in the math behind the derivation of the denominator degrees of freedom. - Give some intuition for the advantage of the split-plot design. I take it to be that when there are big cage effects, with the split-plot design, the advantage is having the different strains paired on cage? The advantage is that, for any given sample size, the critical F-value is lowered for assessing strain, cage level treatments, and their interaction in the mixed-strain design as a result of the different methods for calculating the denominator degrees of freedom. This is now made explicit at lines I don't like the power calculations. First, doing a t-test to compare power (line 222 and )? "No significant difference" and p-values are not helpful. Provide a figure with estimated power for the different phenotypes. We aren t sure what kind of analyses you would prefer but we have made figures so that they can now be visually compared. Please see supplementary figures Power in Tables S2-S4, in which the simulated effects were allowed to be different between the single-strain-per-cage and multiple-strains situations (see lines ), are not informative about the advantage of the design but are really just about the observed estimated effects. I think it would be better to investigate the parameters in a more systematic way, and show which parameters matter, in the choice of design. Yes, the power simulations shown in Tables S2-S4 show the effect of the design AND the observed estimated effects. The benefits solely due to the design are not dependent on the parameter in question, but are intrinsic to the split-plot design itself. This is outlined in the introduction (lines ) and shown in Fig. 1. It is

3 then explained using examples in Supplementary Figs. 3&4. Minor comments - I would much rather see figures in place of the tables of p-values. We thought very hard about doing this but respectfully decided not to, as we think it is impossible to present anything clear and helpful for this number of dependent variables and this complexity of design. The challenge is coming up with a reasonable way of conveying all of the complex information in figures: it would require 26 figures (one for each dependent variable), each further broken down by each strain*enrichment*cage type group (= 12 LS means in each one of the 26 figures). This would take up a good deal of space and also be very overwhelming. Furthermore, the data were transformed for analysis and so the least squared means are not that informative on their own. We have, however, made all of the raw data publicly available for anyone who wants to see the raw data (see e.g. for anyone who wants to see for themselves what the effect of X was on any given dependent variable. - The experiment considered only females. There is some discussion of this (lines ), but it could be highlighted further. Can males generally be co-housed? We have now included more discussion of this, on lines I don't like the use of term "trend" (for non-significant effect) (line 173). Do you have a preferred term? We wanted to be transparent about our results and not look like we were trying to hide anything. It is not uncommon for researchers to identify trends in BMC journals (p-values between 0.1 and 0.05; e.g. Dudzic et al. 2015) and anyone who does not like this convention is free to ignore it when reading. However, treating them as simply non-significant may look as though we are ignoring the possible hints that mixed-strain housing does perhaps influence some variables. - I'd usually use "GLM" for generalized linear model. Here it'd be better to just say mixed model. (line 508) This has been changed throughout the text. - I believe the step-down procedure mentioned in the methods (line 530) is seeking to control the family-wise error rate, not the false discovery rate. The technique that we used (as was described in the paper; see references there too) corrects for the expected proportion of false positives (i.e. the false discovery rate).

4 In contrast, the Bonferroni correction is often used to correct for the chance that there is even one false positive (i.e. the family-wise error rate). - Line numbers are good; I'd also like page numbers. Done. Reviewer:Erin Austin Reviewer's report: Thank you to the authors for the opportunity to review their work. Please consider the following comments: 1. (Discretionary Revision) A strength of the article is the power gains made possible with a straightforward statistical design that is not usual employed. However, as someone not versed in the context, this reviewer thought the discussion was missing more about why this method isn't already commonplace (which again is maybe why this article is new and important). That is, this reviewer is left wondering why doesn't everyone use this (again, a strength of the article), which makes me think that the ability to translate this to practice is limited. It would be helpful to have more on the generalizability of the results. This is a difficult point to address scientifically and also tactfully. We think there are several reasons that this method may not be commonplace as of yet. The first is that knowledge of statistics and experimental design is often poor in biomedical research (e.g. see good reviews/analyses by Kilkenny et al & Button et al. 2013), and so some may not appreciate the benefits of this type design. The second reason stems from the first: cage is generally analyzed as a fixed effect in biomedical research (effectively treating the mouse, not cage, as the unit of replication even though cagemates are clearly non-independent [see recent discussion by Lazic 2010]). We believe that this is a widespread mistake that perpetuates itself because it is easier to obtain significant results this way. The benefits of our mixed-strain design rely on cage correctly being analyzed as a random effect (see lines refs for justification). Thirdly, this approach would require each research group to do a pilot study using strains and dependent variables of interest to it since our validation obviously cannot generalize to all possible cases. Some people may not be willing to make this investment. Finally, it is possible that many scientists just resist new ideas and don t discriminate between standard practice and best practice (for example, when we first submitted a validation for co-housing just two strains [since published in PLoS One], the reviews were shockingly emotional!). We have tried to highlight cases where our approach would be useful, as well as those where it wouldn t, at the end of the discussion (lines ). However, we do not wish to criticize the biomedical community s use of statistics in the manuscript itself. 2. (Major Compulsory Revision) One major concern I had was related to the discussion of power. First, I think the article is missing evidence that the current validation design would have the power to find cage type effects. For example, the power calculations relating to the supplementary tables don't directly assess

5 if there is enough power to capture a cage effect. In fact I think that the results show that for some phenotypes, there wasn't enough power (using the required cage numbers from the individual cage methods). Essentially, this reviewer would request a stronger demonstration that large phenotype effects (e.g. those that would used have raised concerns about single versus mixing strain) could have been found in this study. Thank you for pointing this out< that s a very valid issue. We have added a further analysis to our manuscript to address this point, showing empirical effect sizes and the estimated effect sizes we would have needed in order to find a significant effect. Please see all details now included in the manuscript on lines , , , and Table 3. These demonstrate that we did have the power to detect large effects sizes; and that, although we did not have the power to detect medium or smaller effect sizes, empirically all our effect sizes were small: so we do think are lack of Cage Type effect is not a misleading Type II error, and that if Cage Type does have any effects (we cannot say as nothing was significant), they are only small. 3. (Major Compulsory Revision) A second concern is related to the tests of variance. I apologize if it is from a lack of understanding, but this reviewer didn't fully understand the approach taken in this article. It would have been helpful to see a table of the actual variances. Another question is about the use of CV. There was no differences in means from the other analyses; thus, testing variances directly would seem like a stronger approach. Also, this reviewer wasn't able to figure out what exactly the model on line 542 is showing and the intuition for why it is a good way to assess differences in variances (and is there a concern about correlation among differences such as among phenotypes). This reviewer would request more discussion on why the current methods fully address the issue of changes in variance related to cage type. A single table for all of the variances was just too large to create. All of these data have however been made publically available though for anyone who wants to see it ( We also removed the CV analyses and replaced them with analyses done using standard deviations. The results were largely unchanged, with the exception of one easy-to-understand interaction (the standard deviation of the negligible amount of stereotypic behaviour in the EE was lower). See lines We have also explained the model on line 625 in more detail and stated why it was a useful approach for assessing differences in variation between designs (lines ). We hope this is better now. 4. (Discretionary Revisions) This reviewer would suggest adding more detail to the statistical methods descriptions. It would also be helpful to see more in the "Statistical Analysis" section. For example, it is not clear if "Cage" is a fixed effect in this model. The authors describe it as a random effect, but this reviewer would request more clarity and/or detail. Overall, the manuscript is about a split plot design, but the validation example is not framed well in this context. As a last example, what was the method used to assess Inter-observer reliability? As a result of both referees comments on this issue, we have added more

6 information to the statistical methods descriptions. For your specific points: - Is there a notation you would prefer that we use to label cage as a random effect in the model? As you noted yourself, it is already described as a random effect in the text (i.e. only the residual term was used to form the denominator mean square for cage, whereas the other terms used cage AND the residual to form the denominator mean square). - The validation example has been made clearer and now has accompanying supplementary figs. 3&4 to highlight the benefits of the split-plot design. We have also made it clear how this design is a split-plot design (line ). - This information is now included on line (Major Compulsory Revision) One final statistical question. It appears that the multiple correction is essentially accounting for 182 tests, but to this reviewer the correction should be based on the 26 phenotypes. Please provide insight into why single models had multiple testing corrected p-values. This is explained on line 612. The idea behind our decision is that every individual p-value is a test of the hypothesis that mixed-strain housing affects phenotypes. We don t have 26 different hypotheses (1 for each dependent variable): we have only have one overarching hypothesis, that each p-value is aimed at assessing. 6. (Minor Essential Revision) This reviewer found a very small number of needed corrections. For example "one" or "once" in the first sentence of the abstract's conclusions (line 60). Another example is that this reviewer did not think that Figure 2 in the Supplement matched its usage in the manuscript text (line 164). These have been corrected, thanks for spotting.