Study Designs and Systematic Reviews of Interventions: Building Evidence Across Study Designs

Transcription

1 Zoonoses and Public Health REVIEW ARTICLE Study Designs and Systematic Reviews of Interventions: Building Evidence Across Study Designs J. M. Sargeant 1,2, D. F. Kelton 1,2 and A. M. O Connor 3 1 Centre for Public Health and Zoonoses, University of Guelph, Guelph, ON, Canada 2 Department of Population Medicine, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada 3 Department of Veterinary Diagnostic and Production Animal Medicine, Iowa State University College of Veterinary Medicine, Ames, IA, USA Impacts A large variety of study designs are available in animal agriculture and veterinary medicine to address different types of research questions. Evidence pyramids provide a means of visualizing the evidentiary value of different study designs for evaluating interventions. When ethical and feasible to perform, RCTs provide the highest level of evidence for evaluating interventions. Keywords: Study designs; evidence pyramids; systematic review; veterinary medicine Correspondence: J. M. Sargeant. 103 MacNabb House, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada N1G 2W1. Tel.: (519) Ext ; Fax: (519) ; sargeanj@uoguelph.ca Received for publication September 5, 2013 doi: /zph Summary This article is the second article in a series of six focusing on systematic reviews in animal agriculture and veterinary medicine. This article addresses the strengths and limitations of study designs commonly used in animal agriculture and veterinary research to assess interventions (preventive or therapeutic treatments) and discusses the appropriateness of their use in systematic reviews of interventions. Different study designs provide different evidentiary value for addressing questions about the efficacy of interventions. Experimental study designs range from in vivo proof of concept experiments to randomized controlled trials (RCTs) under real-world conditions. The key characteristic of experimental design in intervention studies is that the investigator controls the allocation of individuals or groups to different intervention strategies. The RCT is considered the gold standard for evaluating the efficacy of interventions and, if there are well-executed RCTs available for inclusion in a systematic review, that review may be restricted to only this design. In some instances, RCTs may not be feasible or ethical to perform, and there are fewer RCTs published in the veterinary literature compared to the human healthcare literature. Therefore, observational study designs, where the investigator does not control intervention allocation, may provide the only available evidence of intervention efficacy. While observational studies tend to be relevant to real-world use of an intervention, they are more prone to bias. Human healthcare researchers use a pyramid of evidence diagram to describe the evidentiary value of different study designs for assessing interventions. Modifications for veterinary medicine are presented in this article. Introduction The overarching methodology for conducting a systematic review includes defining a targeted question, identifying relevant primary studies (i.e. reports of original research), extracting relevant information from studies, assessing the risk of bias in the studies, summarizing the evidence from the studies that are included in the systematic review, and presenting and interpreting the findings (EFSA, 2010; Higgins and Green, 2011). A consideration of study design, and the appropriate study design(s) for addressing the question posed by a systematic review, is therefore central to the systematic review methodology. The types of research questions that are commonly addressed in human and veterinary medicine and public health include those related to the efficacy and safety of interventions: the prevalence or incidence of diseases or conditions; aetiology and risk factors; prediction and Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014) 10 17

2 J. M. Sargeant et al. Overview of Study Designs diagnosis; diagnostic accuracy and other questions including hypothesis generation (Glasziou et al., 2001). Although all of these questions are of relevance in veterinary medicine, the remainder of this article will focus on studies to evaluate interventions. In this context, intervention refers to a deliberately applied therapeutic or preventive treatment, and includes drug therapies, surgical or physical therapies and dietary or management changes used to prevent, reduce or treat an adverse health outcome or event. In the context of therapeutic interventions, the animal already has the disease, and interventions are applied to cure the disease, improve or prolong the quality of life, or maintain or restore production. Interventions may be applied to individuals or to groups (e.g. pens, rooms or herds). For preventive interventions, the animal or group of animals does not have the outcome at the time of the intervention, and interventions are given to prevent the event or illness, or to reduce the severity of incident cases. Often, interventions are given at the group level for trials of preventive interventions in food animal species. Study Designs Used in Veterinary Medicine and Issues of Validity Study designs can be broadly classified into descriptive studies or analytical (hypothesis-testing) studies. Hypothesis-testing studies may be experimental or observational in nature, depending on whether the investigator has control over allocation to exposure or intervention groups. Depending on the research question being posed, different study designs have advantages and disadvantages. Common study designs for addressing intervention questions in animal agriculture and veterinary medicine are described briefly in the following sections. There are numerous resources available that provide detailed information about each of these study designs (for examples, see Dohoo and Waltner-Toews, 1985a,b,c; Trevejo, 2007; Holmes and Cockcroft, 2008; Dohoo et al., 2010). When interpreting research results, it is important to consider both internal and external validity. External validity refers to the representativeness (generalizability) of the study results from the study population to the population in which the intervention is intended to be used (Rothwell, 2005). Assessing external validity may be subjective as the end-user may have in mind a different target population for potential application of the intervention than was considered by the researcher. Internal validity means that any differences observed between the intervention groups can be attributed to the intervention under investigation or to random error (Juni et al., 2001). Random error is accounted for in statistical tests. Systematic error (bias) also needs to be considered. This can be defined as an effect at any stage of a study that produces results that systematically depart from the true value, that is the estimate of intervention efficacy differs from the true value in the source population (Glasziou et al., 2001; Juni et al., 2001). Bias may lead to either an over- or under-estimation of the intervention effect. Although there is an extensive nomenclature for biases, most biases can be summarized by three types: selection bias, information bias and confounding bias (Rothman et al., 2008; Dohoo et al., 2010). Selection bias results from systematic differences between characteristics of the study population and the source population from which it was drawn, and is caused by the method of selecting the study population from the source population (Thrusfield, 2007). Information bias, sometimes denoted as misclassification bias for categorical variables [variables that have a fixed number of possible values], occurs when a variable is measured with error. For instance, misclassification of an outcome occurs when either diseased animals are classified as non-diseased (less than perfect test sensitivity), or animals without a particular disease are classified as disease positive (less than perfect test specificity). Measurement errors may be non-differential, that is not related to intervention status or outcome status, or differential, that is the degree of error in measurement differs between the intervention groups and/or outcome status, introducing information bias (Delgado-Rodriguez and Llorca, 2004). In general, non-differential measurement errors tend to bias associations between intervention and outcome towards the null hypothesis of no effect. In contrast, differential information bias can bias the association between intervention and outcome towards or away from the null hypothesis. The third type of bias is confounding bias. A confounding variable (confounder) is any factor that is either positively or negatively associated with the outcome (disease) in the source population and also associated with the explanatory variable (intervention), but is not a consequence of exposure to the explanatory variable (Thrusfield, 2007). Confounding bias occurs when the effect of the confounding variable is not accounted for when describing the association between the exposure of interest and the outcome. The direction of confounding bias depends upon the relationship between the confounder, the outcome of interest and the exposure of interest [in randomized controlled trials (RCTs), the intervention]. Confounding bias is of particular concern in observational studies. Descriptive studies Descriptive studies, as the name suggests, are used to describe phenomena. They are useful for answering questions related to prevalence or incidence of a disease or condition, or to describe features related to clinical presentation 2014 Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014)

3 Overview of Study Designs J. M. Sargeant et al. or disease progression and prognosis. Descriptive studies do not have a comparison (control) group and are not appropriate for testing hypotheses related to disease causation, risk factors for disease or efficacy of interventions (Dohoo et al., 2010). The main types of descriptive studies are surveys, case reports and case series. Experimental studies The key characteristic of experimental study designs is that the allocation of study units to intervention groups is under the control of the investigator (Dohoo et al., 2010). Study units may be individuals, or animals in groups such as in pens, rooms, barns or herds. Experimental studies can be classified into those that are carried out under artificial conditions (usually considered a classical experiment ) and those that are conducted under real-world conditions (controlled trials, field trials or clinical trials). In the development and evaluation of interventions, laboratory experiments include hypothesis testing using in vitro or in vivo models. In vitro means that the experiment is performed in an artificially created environment outside of a living organism, whereas in vivo means that the experiment was conducted using a living organism. In vitro experiments may provide important proof of concept for efficacy of interventions, but are a preliminary step in the research process and are not discussed further in this article. In vivo experiments can be conducted in the species of interest or in animal models where the experimental unit may or may not be the same species as that in which the intervention ultimately is intended to be used. As an example, if an investigator was interested in using an experiment to evaluate the efficacy of an intervention for viral pneumonia in cats, s/he could conduct that experiment in cats, or using laboratory mice as a model of the disease in cats. There is no consistent terminology to differentiate animal models in the target species from an animal model for the species of interest, although the implications for external validity will clearly differ between these types of studies. For this article, we will refer to in vivo animal experiments where the experimental unit of interest is not the target species as animal models. When an experiment is conducted in the species of interest with a deliberate disease induction (e.g. exposure to an infectious disease agent), we will refer to the study design as a challenge study. For intervention assessment in challenge studies, induction of the disease outcome is under the control of the investigator, as is allocation to intervention group. For infectious diseases, disease induction may involve the use of laboratory-adapted strains of the pathogen. The intervention of interest could be intended as a therapeutic or preventive intervention. For instance, if the objective of the experiment was to evaluate the efficacy of a vaccine to prevent disease caused by a specific virus, the study subjects would be allocated to either receive or not receive the vaccine, and, after an appropriate time period, all animals would be deliberately exposed to the virus (challenged). If the objective of the experiment was to evaluate the efficacy of an intervention to treat clinical disease caused by a specific virus, all of the study subjects would be exposed to the virus, and, after the onset of clinical illness, would either receive or not receive the intervention. Deliberate disease induction does not always involve an infectious disease outcome; for instance, metabolic diseases may be induced to evaluate therapeutic interventions, or induction may be attempted in all study subjects following a prevention intervention. In addition to controlling exposure to a disease outcome of interest, challenge trials also control the animal used (e.g. by restricting the study subjects to animals of the same breed, age or genetic lineage) and the environmental conditions (e.g. by housing all study subjects in the same barn or room and feeding them the same diet). The advantage of these restrictions on the study population is a reduction in the potential for confounding bias by known confounders such as breed or age. The disadvantage is that the conditions under which these studies are conducted, including the nature of the disease challenge, the study population and/or the environment, are artificial, with the consequence being that the results may have low external validity. However, challenge studies are useful designs for initial testing of product safety and for proof of concept for the efficacy of an intervention. It is possible to include these restrictions in trials with a natural disease exposure, or to incorporate them into the population eligibility criteria for an observational study, with the same advantages and disadvantages. The concept is presented in this section, as this is a common feature of challenge trials. Controlled clinical/field trials differ from challenge trials in several important ways. In a field trial, the investigator controls the allocation of study units to intervention groups, but the study takes place under real-world conditions, with natural development of the disease of interest. The ideal method of allocation of study units to intervention groups is randomized allocation, that is where allocation is based on a formal chance (random) process (Schulz and Grimes, 2002). Proper randomization helps to prevent systematic bias by attempting to make the intervention groups comparable based on known or unknown confounding variables, facilitates blinding of intervention allocation and allows probability theory to be used to evaluate whether differences between interventions groups are likely to be due to chance (Schulz and Grimes, 2002). Random allocation to group is necessary for establishing exchangeability (i.e. comparability) of the groups and therefore causal inference for intervention effects, provided the study Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014) 10 17

4 J. M. Sargeant et al. Overview of Study Designs population is sufficiently large. Randomization methods range from a simple flipping of a coin to random number tables to computer-generated randomization schedules. In some instances, particularly with smaller sample sizes, randomization may occur within blocks or stratifications, where the blocking or stratification factor is thought to be an important confounder (e.g. by weight or by gender). In field trials, blocking by unit of housing may be used to take into account for differences between housing units, such as temperature, diet or management, which might influence the outcome. The advantage of RCTs with natural exposure to the disease is that this approach most closely resembles the conditions under which the intervention would actually be used and therefore often has high external validity compared to trials with an artificial disease challenge. However, even RCTs with natural disease exposure have exclusion criteria for the study population that can limit external validity. Additionally, RCTs may be conducted in referral hospitals or research herds (which may or may not house animals at the same animal density as commercial herds), which also will impact external validity. RCTs may be more costly to perform and may not be ethically appropriate if there is no proof of concept to suggest that the intervention is safe and at least potentially efficacious. Further, RCTs may not be ethical if there is no support for the notion of equipoise, that is the idea that the null hypothesis could be possible (Freedman, 1987; Lilford and Jackson, 1995). If there is no reasonable belief that the null hypothesis is true, such as the tongue-in-cheek example of the absence of RCTs to support the efficacy of parachutes in preventing deaths when falling out of a plane (Smith and Pell, 2003), an RCT is unethical. Quasi-experimental studies Quasi-experimental studies include a range of non-randomized intervention studies that are used when it is not feasible or ethical to conduct a RCT (Harris et al., 2006). While these studies lack randomization, they are often able to measure changes in populations as a result of policy, regulatory or other control interventions. An example from the veterinary literature was a study evaluating changes in average bulk milk somatic cell counts (BMSCC, a measure of milk quality) following regulatory changes in the province of Ontario, Canada, in the maximum allowable BMSCC without financial penalties (Sargeant et al., 1998). This policy change was instituted province-wide, precluding the ability to compare among farms affected and those unaffected by the change. Therefore, a quasi-experimental design was used to compare average BMSCC before and after the policy change. However, an inherent issue with before and after comparisons (sometimes called historical controls ) is that factors other than the factor of interest can produce changes in an outcome over time. In the aforementioned example, climatic changes are known to affect somatic cell counts in dairy cattle, and if there were significant climatic differences in the before period versus the after period, it would be difficult to know whether to attribute the changes in BMSCC to the policy or the weather. Therefore, if experimental or observational studies with concurrent comparison groups are possible, the quasi-experimental approach is inappropriate. Another group of non-randomized trials is experiments where the allocation to group is still under the control of the investigator, and parallel groups are assessed (unlike the before and after example above), but allocation is not random. For example, in poultry slaughter houses, random allocation of carcasses to intervention may not be feasible, but the investigator is still able to design an allocation system that he or she believes will maximize exchangeability of the groups. For instance, a systematic allocation system may be used. It is important that the allocation system not be discernable to those assessing outcomes and not be associated with the outcome. Observational study designs In observational study designs, the allocation of study units to intervention groups is not under the control of the investigator and therefore cannot be randomized; observational studies relate naturally occurring exposures to natural disease occurrence (Mann, 2003). Observational studies may be used to evaluate individual characteristics, behaviours, environmental conditions or interventions as exposures that may be associated with or modify the risk of disease or response to intervention. In general, observational studies are good at establishing associations among exposures and outcomes, but not for proving a causal relationship. There are several observational study designs, which are differentiated by the method of selecting the study population, including cohort, case control and cross-sectional designs. In cohort studies, study units are selected based on exposure status or as a cohort (group) of individuals who are disease-free at the start of the study. Study units are then followed over time to compare the incidence of disease between exposure groups. In cross-sectional studies, study units are selected without regard to exposure or outcome. In prevalence case control studies, study units are selected based on outcome status, and exposure status is determined after selection of cases and controls. Cross-sectional studies use prevalent cases rather than incident cases, and case control studies also often use prevalent cases. It is possible to conduct case control studies using incident cases: often referred to as density case control designs, nested 2014 Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014)

5 Overview of Study Designs J. M. Sargeant et al. case control designs or case-cohort designs (Pearce, 2012). The distinction between the use of incident and prevalent cases is important, as a temporal relationship between an exposure and an outcome is a key element of causality. Observational studies are common in the veterinary literature and may be the only design choice for some types of exposures, particularly when allocation by the investigator to exposure group would be unethical or impractical (e.g. when evaluating differences in an outcome among different housing systems). However, observational designs are particularly prone to confounding bias and do not provide the strength of evidence of efficacy for interventions that can be obtained using RCTs. Although cohort studies are potentially less prone to selection bias than other observational study designs, the results may still be affected by confounding by indication, where the decision to use a particular intervention is related to clinical indications that are also prognostic factors, that is related to the risk of the outcome (Rothman et al., 2008). Cross-sectional and (prevalence) case control studies may be more appropriate for generating hypotheses regarding interventions than for testing them to infer a causal relationship (Mann, 2003). Hierarchies of Research Evidence Scientific evidence is the product of appropriately designed and carefully controlled research studies. Evidence for or against an intervention should ideally be derived from multiple studies investigating the same research question. Although a single study can contribute to a body of knowledge, it only provides preliminary evidence for an intervention. Evidence-based medicine in human health care categorizes different types of clinical evidence and ranks them according to their risk of bias (or their freedom from the various biases that may occur). For example, the strongest evidence for intervention efficacy is provided by a systematic review of multiple RCTs when it is ethical and feasible to use this design to address the question of interest (Clancy, 2002). In contrast, testimonials, case reports and expert opinion have limited value as proof because of the potential for placebo effect, the biases inherent in observation and reporting of cases (such as recall bias), difficulties in ascertaining who constitutes an expert and more. In the human healthcare literature, one way to describe the hierarchy of evidence is through a framework commonly known as the pyramid of evidence (Fig. 1). The pyramid of evidence provides a framework to stimulate thinking about the evidentiary value, in terms of field efficacy, of different research designs and is based on the concept of causation and the need to control bias (Roudebush et al., 2004). In progressing from the bottom towards the top of the pyramid, the number of studies available in the literature tends to decrease. However, the relevance to answering clinical Randomized Controlled Double Blind Studies Cohort Studies Case Control Studies Case Series Case Reports Ideas, Editorials, Opinions Animal research In vitro ( test tube ) research Systematic Reviews and Meta-analyses Fig. 1. Pyramid of evidence in human health research. SUNY Downstate Medical Center, 2004, library.downstate.edu/ebmdos/2100.htm. [NB: We would advocate the use of appropriately blinded, rather than double blind. There is evidence in human and animal health research that there are different interpretations of the terminology used to describe blinding such as double ; (Devereaux et al., 2001; Giuffrida et al., 2012)]. questions increases. At the base of the pyramid, in vitro research and non-target species animal models might provide important proof of concept for an intervention, but do not provide strong evidence of efficacy in real-world settings. Cross-sectional and prevalence case control studies may be used to test hypotheses related to intervention efficacy under field conditions, but because of the use of prevalent cases and absence of a temporal relationship between exposure and outcome, these designs do not provide as high a level of evidence as RCTs. Cohort studies or incidence case control designs suffer from the potential for confounding by indication and therefore do not provide the same level of evidence as RCTs. Well-executed meta-analyses (the statistical combination of data from multiple studies) and systematic reviews are at the top of the evidence pyramid. An adaptation of the human healthcare model for use in veterinary medicine was proposed by Holmes and Cockcroft (2008), who separated systematic reviews as higher quality than meta-analyses alone. This approach distinguishes between meta-analyses where the data came from a systematic review and meta-analyses that were conducted based on a convenient, non-comprehensive or undescribed group of research studies. Meta-analyses not conducted as a component of a systematic review likely are more prone to bias. In their evidence pyramid, Roudebush et al. (2004) placed systematic reviews at the top of the pyramid, followed by RCTs, but combined cohort studies with case control studies as epidemiologic studies followed by models of disease, case series, case reports, research in other species, pathophysiological rationale, ideas editorials and opinions, and finally in vitro research. Clearly, there is no consensus about the exact ranking of the various types of evidence, but the relative rank of the major study designs is relatively consistent Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014) 10 17

6 J. M. Sargeant et al. Overview of Study Designs There are, however, additional considerations that may need to be incorporated into veterinary medical evidence hierarchies, particularly in relation to food animal species. These relate to unique study designs in veterinary medicine and the relevance to efficacy under field (real-life) conditions. Deliberate challenge trials evaluating interventions in the species of interest may provide important preliminary evidence for the efficacy of an intervention under artificial conditions. However, an artificial disease challenge/induction may not adequately mimic natural exposure and, with infectious diseases, laboratory-adapted strains of the disease agents are often used. The animals in these trials are often housed individually or in small groups. This may not represent natural disease pressures experienced by large groups of animals housed together and may impact generalizability of interventions normally allocated at the group level. Therefore, the challenge study design provides proof of concept for efficacy of an intervention in the species of interest but is not as strong as RCTs in providing evidence of efficacy under real-world conditions. A second type of trial unique to veterinary medicine is the trial conducted in small groups, often on research farms. In this type of study, the disease exposure is natural, but the animals may be housed in smaller groups, or under lower animal density conditions, than is typical under field conditions. Thus, disease dynamics and disease pressures may differ from field settings. While the small-group trial is more relevant to efficacy under real-world conditions compared to the challenge study design, it still does not provide the level of evidence provided by trials conducted under field conditions. Quasi-experiments are also used for evaluating some veterinary interventions. Although this design is not unique to veterinary medicine, it is not included in the evidence pyramid shown in Fig. 1. This type of trial is often the only means of evaluating interventions that are implemented as mandatory policy changes at a regional level. In this instance, they can provide important evidence of effectiveness. However, when random allocation is possible, RCTs are a much stronger design. Therefore, a veterinary pyramid of evidence will need to consider the evidentiary value of multiple types of RCTs and of non-randomized studies. A proposed modification of the classic pyramid of evidence that incorporates modification for veterinary research is shown in Fig. 2. In this figure, we differentiate between studies involving incident cases and those involving prevalent cases, with incidence studies having higher evidentiary value. We include multiple study designs within each of incidence studies and prevalence studies classifications. Within these categories, the specific ordering of study designs may depend on the specific research question and the appropriateness of various designs in that context. Fig. 2. Proposed pyramid of evidence for studies in veterinary medicine. 1 Randomized controlled trials. 2 Before/after trials or trials where the investigator has control over intervention allocation, but random allocation was not used. 3 Trials with induced disease conducted in the target species. Studies on the same level may be ranked or interchangeable depending on appropriateness to address the specific research question. 4 Trials with induced disease conducted in a different species than the target species. Study Designs Used in Systematic Reviews Systematic reviews can make use of research data from numerous study designs. However, when conducting systematic reviews of interventions, studies conducted using low evidentiary designs for evaluating real-world efficacy are generally not appropriate for inclusion. For example, in vitro experiments and animal models using other species offer important data for proof of concept during the development of potential interventions but do not provide direct evidence as to the efficacy under field conditions and should not be included in systematic reviews that address clinical intervention questions. In addition, descriptive studies should not be included in systematic reviews of interventions, as they are used to provide baseline or preliminary data, and are not appropriate for hypothesis testing. For most interventions in human healthcare, randomized and appropriately blinded controlled trials represent the minimum level of evidence to be included in a systematic review (Higgins and Green, 2011). For questions that cannot be addressed by RCTs, guidelines are available for conducting reviews of non-randomized studies (for example, see Higgins and Green, 2011). The human healthcare research community is also considering how best to incorporate post-marketing surveillance data into the evidence pyramid, in the light of the need for real-world data for a range of very expensive, novel therapeutics for diseases such as cancer, cardiovascular disease and inflammatory arthritis (Santaguida et al., 2005) Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014)

7 Overview of Study Designs J. M. Sargeant et al. Therefore, if there are RCTs available in the literature to address the efficacy of a specific intervention, it is generally recommended that the systematic review includes only this study design. If RCTs are unavailable, then observational studies and challenge trials may be used, although depending upon the disease and intervention, multiple observational studies or challenge trials may not provide as high a level of evidence for field efficacy as a single large well-designed and well-executed RCT. For exposures that cannot be assessed with an RCT, cohort studies where a meaningful attempt has been made to control for potential confounders by design or analysis are preferred to the other observational study design approaches. Summary A variety of study designs are available, providing different evidentiary value for addressing questions about the efficacy of interventions. A blinded RCT conducted in real-world conditions with study subjects that are representative of the target population for the interventions is considered the gold standard for evaluating the efficacy of interventions. For some veterinary and food safety interventions, few RCTs are published. Therefore, observational study designs, where the investigator does not control intervention allocation, may provide the only available evidence of intervention efficacy. Pyramids of evidence provide a useful framework for considering the relative evidentiary value of a study design for evaluation realworld efficacy. Modification of existing human healthcare pyramids for veterinary medicine and food safety will need to consider the use of deliberate disease challenge models in the species of interest and RCTs in animals housed under non-commercial conditions. Acknowledgements and Funding Support The authors thank Annette Wilkins for assistance with this manuscript. Partial funding was obtained from the Laboratory for Foodborne Zoonoses, Public Health Agency of Canada and the Canadian Institutes of Health Research (CIHR) Institute of Population and Public Health/Public Health Agency of Canada Applied Public Health Chair. References Clancy, M. J. 2002: Overview of research designs. Emerg. Med. J. 19, Delgado-Rodriguez, M., and J. Llorca, 2004: Bias. Epidemiol. Community Health 58, Devereaux, P. J., B. J. Manns, W. A. Ghali, H. Quan, C. Lacchetti, V. M. Montori, M. Bhandari, and G. H. Guyatt, 2001: Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 285, Dohoo, I., and D. Waltner-Toews, 1985a: Interpreting clinical research. Part I. General considerations. Compend. Contin. Educ. Pract. Vet. 7, S473 S478. Dohoo, I., and D. Waltner-Toews, 1985b: Interpreting clinical research. Part II. Descriptive and experimental studies. Compend. Contin. Educ. Pract. Vet. 7, S513 S519. Dohoo, I., and D. Waltner-Toews, 1985c: Interpreting clinical research. Part III. Observational studies and interpretation of results. Compend. Contin. Educ. Pract. Vet. 7, S605 S613. Dohoo, I., W. Martin, and H. Stryhn, 2010: Veterinary Epidemiologic Research, 2nd edn. VER Inc., Charlettetown, PEI. European Food Safety Authority (EFSA), 2010: Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J. 8, Freedman, B. 1987: Equipoise and the ethics of clinical research. N. Engl. J. Med. 317, Giuffrida, M. A., K. A. Agnello, and D. C. Brown, 2012: Blinding terminology used in reports of randomized controlled trials involving dogs and cats. J. Am. Vet. Med. Assoc. 241, Glasziou, P., L. Irwig, and C. E. A. Bain, 2001: Systematic Reviews in Healthcare: A Practical Guide. Cambridge University Press, Cambridge, UK. Harris, A. D., J. C. McGregor, E. N. Perencevich, J. P. Furuno, J. Zhu, D. E. Peterson, and J. Finkelstein 2006: The use and interpretation of quasi-experimental studies in medical informatics. J. Am. Med. Inform. Assoc. 13, Higgins, J. P. T., and S. Green (eds), 2011: Cochrane Handbook for Systematic Reviews of Interventions Version (updated March 2011). The Cochrane Collaboration. Available at: Holmes, M., and P. Cockcroft, 2008: Handbook of Veterinary Clinical Research. Blackwell Publishing, Oxford, UK. Juni, P., D. G. Altman, and M. Egger, 2001: Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323, Lilford, R. J., and J. Jackson, 1995: Equipoise and the ethics of randomization. J. R. Soc. Med. 88, Mann, C. J. 2003: Observational research methods. Research design II: cohort, cross sectional, and case control studies. Emerg. Med. J. 20, Pearce, N. 2012: Classification of epidemiological study designs. Int. J. Epidemiol. 41, Rothman, K. J., S. Greenland, and T. L. Lash, 2008: Modern Epidemiology, 3rd edn. Lippincott Williams and Wilkins, Philadelphia, PA. Rothwell, P. M. 2005: External validity of randomised controlled trials: to whom do the results of this trial apply? Lancet 365, Roudebush, P., T. A. Allen, C. E. Dodd, et al., 2004: Application of evidence-based medicine to veterinary clinical nutrition. J. Am. Vet. Med. Assoc. 224, Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014) 10 17

8 J. M. Sargeant et al. Overview of Study Designs Santaguida, P. L., M. Helfand, and P. Raina, 2005: Challenges in systematic reviews that evaluate drug efficacy or effectiveness. Ann. Intern. Med. 142, Sargeant, J. M., Y. H. Schukken, and K. E. Leslie, 1998: Ontario bulk milk somatic cell count reduction program: progress and outlook. J. Dairy Sci. 81, Schulz, K. F., and D. A. Grimes, 2002: Generation of allocation sequences in randomised trials: chance, not choice. Lancet 359, Smith, G. C. S., and J. P. Pell, 2003: Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ 327, Thrusfield, M. 2007: Veterinary Epidemiology, 3rd edn. Blackwell Publishing, Oxford, UK. Trevejo, R. T. 2007: A small animal clinician s guide to critical appraisal of the evidence in scientific literature. Vet. Clin. North Am. Small Anim. Pract. 37, Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014)