Empirical answer (based on experimental data, statistical analyses)

Transcription

1 Blockkurs Zoology and Evolution FS 2013 Experimental Design Mathias Kölliker Observation Scientific theory Research Question? Empirical answer (based on experimental data, statistical analyses)

2 (Working) Hypothesis ( a clear statement articulating a plausible candidate explanation for observations ; Ruxton & Colgrave) Prediction (a clear statement for an expected outcome if the hypothesis is correct) (the plan for how to collect and analyse data to test the predictions) Data (Collection of measurements/data following experimental design with independent replication) Statistical Analysis (evaluate support for predictions based on collected data) Interpretation (ideally: the answer to your research question) Example Observation: speed on ski-pists is increasing Research question: Why is the speed on ski-pists increasing? Hypotheses: faster material (better skis and snowboards) harder snow (less snow and more snow-canons) global warming (sunnier weather) more ski-lifts at high altitudes (steeper slopes) helmet wearing. Observation: sharp increase of helmet-wearing on ski-pists over the last few years). But why should that be?

3 Level of fun / Level of risk Net fun Intuitive thought No Helmet Helmet Downhill skiing / snowboarding speed 100 km/h Hypothesis: The higher speed on skipists is at least partly due to higher prevalence of ski-helmet wearing ( Risk compensation hypothesis ) Model = A simplified, but carefully worked out argument (for complex arguments, mathematics is necessary to keep track of the logics connecting an argument s components) Assumption: skiers and snowboarders behave in a way that maximizes the amount of fun per unit risk ( economics of behaviour) Predictions: Helmet-wearing increases the downhill skiing speed Helmet-wearing reduces the absolute risk taken, but by less than is usually thought The net fun is enhanced with helmets!

4 Experimental design: Two experimental designs chosen by scientists to test the prediction that helmet wearing enhances downhill skiing/snowboarding speed 1 Scott, M.D., Buller, D.B., Andersen, P.A., Walkosz, B.J., Voeks, J.H., Dignan, M.B. And Cutter, G.R Testing the risk compensation hypothesis for safety helmets in alpine skiing and snowboarding. Injury Prevention 13, Support for hypothesis : Survey/interview of downhill skiers/snowboarders that do/don t wear helmets about their estimated speed (n=1779) 2 Shealy, J.E., Ettlinger, C.F. And Johnson, R.J How fast do winter sports participants travel on alpine slopes? Journal of ASTM International 2, Support for hypothesis : Measure downhill speed of skiers/snowboarders that do/don t wear helmets using radar (n=650) Observational designs Manipulative designs Mean number of nests per male Shortened Control Elongated Tail treatment Kleven et al Behav. Ecol. Sociobiol. 59, Barn swallow (Hirundo rustica) Long-tailed widow bird (Euplectes progne) Feuerweber: Hahnenschweifwida Anderson Nature 299,

5 Observational designs Pros Often easier to carry out Not much previous knowledge of organism required Low risk for artefacts Usually ethically rather unproblematic Cons low capacity to reveal causalities reverse causation third variable causation conclusions of the sort A affects B are usually not possible/allowed Correlation Causation Manipulative designs Pros Requires careful thinking about hypothesis and predictions before data collection Revelation of causal relationships Control over reverse and third variable causation Conclusions of the sort A affects B are possible Cons Risk of artefacts (experimental controls!!) Good experimentation requires good prior knowledge of organism More time and resource investment (e.g., pilot studies) Ethical concerns possible

6 Randomization in space, time and across treatments (avoid confounding effects) Space Treatment Control Treatment Control Here, position in the laboratory confounds the experimental treatment. The defense that this is unlikely to bias your results is not a valid one. Recall that you are a human being, and cannot be aware of all the relevant environmental influences on your study organism Time Randomization in space, time and across treatments (avoid confounding effects) Morning Afternoon Treatment Control

7 Randomization in space, time and across treatments (avoid confounding effects) Markings/dyes Treatment A Treatment B Treatment A Treatment B Also applies to e.g. micro-array studies where dyeswitches are required True, independent replication (avoid pseudoreplication) Pseudoreplication is a problem because the likelihood of finding a statistically significant effect if the effect is truly there (i.e., statistical power ) depends on sample size used for the statistical test. This is why it is central that the sample size in the statistical test truly reflects the number of units of independent replication.

8 True, independent replication (avoid pseudoreplication) Control Treatment Control Treatment Control Individual Individual Individual Individual Individual Rate of true replication (equal statistical sample size n): 1 Rate of pseudoreplication (not equal statistical sample size): 5 Individuals would need to be replicated in this example. Sometimes, it makes sense to have experimental manipulations or multiple measurements within individuals ( repeated-measures design ), but independent replicates (individuals) are still required There are many ways of generating pseudoreplication in your data. Consider carefully if you are collecting data from truly independent subjects or not Reliability of observations/measurements Whenever possible, the observer collecting data should be blind with respect to experimental treatments Possibilities to avoid this problem use subject ID s that do not contain information about experimental treatments data-sheets for observations/measurements only display the ID, no information about treatments different persons setting up treatments and making observations/measurements

9 Reliability of observations/measurements Bad examples: Good example: Measurement list Treatment list ID Measurement A A A B A B-006 A-007 B-008 B-009 ID Treat Measure 001 A A A B A B 007 A 008 B 009 B ID Measurement ID Treatment 001 A 002 A 003 A 004 B 005 A 006 B 007 A 008 B 010 B Only Measurement list is used/visible during data collection Reliability of observations/measurements Imprecise Inaccurate Precise Inaccurate Potential solutions: - Multiple measurements - Calibration - Improved standardisation Imprecise Accurate Precise Accurate Within/between-observer reliability -Assessement/training: make multiple measurements and test their degree of fit (e.g., by regression/correlation analysis) Usually, it is necessary to tolerate some level of measurement error. The goal is to minimize it as much as possible, and to make sure it does not confound your results (i.e., is random with respect to treatments see observer bias). From Martin & Bateson (2007)

10 Reliability of observations/measurements Low measurement error (good precision) Large measurement error (poor precision) Measurement 2 Measurement 2 Measurement 1 Measurement 1 Informal guideline (Martin & Bateson, 2007): r 2 > 0.5 (r > 0.7) is ok for difficult measurements. But the higher the better. Not all measurements can be equally well standardized (e.g., weight of a dead animal is easier to measure than weight of a live, moving animal). Reliability of observations/measurements Two observers: precise but biased Two observers: imprecise but unbiased Two observers: precise and unbiased Observer 2 Observer 2 Observer 2 bias Observer 1 Observer 1 Observer 1

11 Some final considerations Think about required/possible sample size Logistic constraints: available space number of replicates that can be dealt with simultaneously nr of wells in PCR plates, vial slots in GC trays (technical limitations) or available cameras Good experimentation requires focused and repetitive work in order to obtain meaningful and informative replication : summary Have a precise research question Have a well-defined hypothesis that could answer the research question. In biology, we often check if a hypothesis passes the scrutiny of evolutionary principles Have a clear-cut prediction for an outcome in an experiment when the considered working hypothesis (H 1 ) applies. A good and clear answer to a single question is scientifically more valuable than weak answers to many questions at a time

12 : summary Decide on observational versus manipulative design Incorporate necessary controls in manipulative designs Randomization of treatments Avoid observer bias (blind observation; whenever possible, test reliability of measurements and multiple observers) Think of statistical analysis of your experimental design. How will you test your predictions statistically, based on which data and what statistical test?