Guidelines for Reporting Non-Randomised Studies

Transcription

1 Revised and edited by Renatus Ziegler B.C. Reeves a W. Gaus b a Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, Great Britain b Biometrie und Medizinische Dokumentation, Universität Ulm, Germany Key Words Non-randomised studies Cohort studies Randomised controlled trials CONSORT Standards of reporting Bias Summary Non-randomised studies (NRSs) are useful because they allow interventions to be evaluated that are difficult to investigate by randomised controlled trials (RCTs). However, NRSs are more susceptible to bias. The Consolidated Standards of Reporting Trials (CONSORT) statement was established to ensure that researchers report features of RCTs that must be considered when appraising their quality. CONSORT has improved the reporting of key information, highlighting missing key information for users. Researchers have a responsibility to report essential information that allows users to assess the susceptibility of NRS to selection, performance, detection and attrition bias. This paper considers criteria for reporting cohort studies: the rationale behind the CONSORT criteria for reporting of RCTs will be applied to cohort studies. Many of the criteria need no modification but application of others raise difficult issues for cohort studies, e. g.: description and standardisation of control and intervention treatments; description of the method of allocation; choice of prognostic factors to be collected; distinguishing between intended and provided treatments; collection of data on adverse and longterm outcomes; establishing a priori plans for analysis. The CONSORT Statement The CONSORT (Consolidated Standards of Reporting Trials) statement, revised in 2001 [1, 2], consists of a checklist of items to be reported by researchers when writing up the findings of randomised controlled trials (RCTs) for publication. The aim of the statement is to ensure that researchers report key information that allows readers to appraise the quality of the trial. However, this list of items is also important for researchers when they are designing (considering items to be included in a protocol) and conducting as well as reporting trials, since they now need to plan their trials in order to be able to report the items (there is a special website: see also the chapter on quality assessment of RCTs in the Cochrane Reviewers Handbook at The CONSORT statement has 22 items and is divided into 5 sections: title (1 item), introduction (1 item), methods (10 items), results (7 items), discussion and comment (3 items). It is still evolving and will certainly be improved in the future. The primary aim is to help authors improve the quality of reports of simple 2-group parallel RCTs. However, the basic ideas and procedures can be applied to any type of cohort study design. In order to keep the statement simple, the item asking for the unit of randomisation (e. g. cluster) has been dropped from the revised CON- SORT statement because specific checklists have been developed for reporting cluster RCTs and other design types (see In the rest of this paper, the term cohort studies is used loosely to include all comparisons between groups of exposed and unexposed people, i. e. controlled clinical trials (CCT), non-randomised studies (NRS), observational studies of variations in everyday care, etc. The intention is to consider how the CONSORT items can be applied to such studies, and how the items may need to be modified in order to do so. Fax Information@Karger.de S. Karger GmbH, Freiburg Accessible online at: Barnaby C. Reeves, PhD Department of Public Health and Policy, London School of Hygiene and Tropical Medicine Keppel Street, GB-London WC1E 7HT, UK Tel , Fax barney.reeves@lshtm.ac.uk

2 Title and Introduction: Items 1 and 2 Item 1: Title and Abstract Describe how participants were allocated to interventions, i. e. state explicitly random allocation, randomised, randomly assigned. This description means that: a) the study can be readily indexed as an RCT in bibliographic databases (e. g. in MEDLINE); b) the study can be readily identified as an RCT, c) the study can be added to trial registers (e. g. CENTRAL of the Cochrane Library); d) the study can be included in systematic reviews. 1. Identify the study clearly as non-randomised, with no random allocation ; 2. clearly describe the study as comparative, with i) two or more groups (comparison between groups) and/or ii) measurement of outcomes at two or more time points, proving before and after comparison(s) (within a group). These requirements make the nature of the comparison investigated by the researchers explicit. They also mean that the study can be readily indexed as a cohort study in a bibliographic database (e. g. in MEDLINE), and the study can be readily identified as not being an RCT. This should prevent the study being mistakenly added to trial registers (e. g. CENTRAL). Item 2: Introduction and Background Describe the scientific background and explain the rationale of the study. For example: a) describe the nature, scope and severity of the problem that led to the RCT being carried out and explain the rationale for the study; b) describe other studies of the same research question; c) ideally, cite or carry out a systematic review to demonstrate that the research question has not already been answered; d) demonstrate collective equipoise about the research question (uncertainty about the answer); e) justify the trial both from logistical and ethical points of view; f) suggest a plausible explanation for how the intervention under investigation might work, especially if there is little previous experience with the intervention. 1. Items (a) to (f) apply as above; 2. explain why an RCT is not possible; 3. give reasons why a non-randomised study is likely to provide a sufficiently valid answer, e. g. precautions against bias or reasons why bias is unlikely. The modifications are required because reasons for not doing RCTs are often poorly described. The reasons should be explicitly stated so that they can be considered critically. The fact that an RCT is not possible is insufficient justification to invest scarce resources in a cohort study; researchers also need to be confident that a cohort study will provide a valid answer. Methods: Items 3 12 Item 3: Participants and Setting Describe eligibility criteria for participants and the settings and locations where the data were collected. For example: a) describe explicitly the eligible reference population ; patients to whom clinicians aim to provide the interventions (inclusion criteria); b) describe the method of recruitment, i. e. referral or selfselection; c) describe how the study population was sampeled to make it representative, e. g. consecutive or random sample; d) describe specific exclusion criteria, either for clinical or logistical reasons. Detailed information about the participants and settings allows the reader to assess the applicability of results of the trial (external validity / generalisability item 21). Item 4: Interventions Provide precise details of the interventions intended for each group and how and when they were actually administered. For example: a) characterise both new and control (e. g. current best practice, usual care, no treatment) interventions thoroughly and carefully; b) describe placebo-details (formulations, method of disguise) if applicable; c) allow user/reader to replicate the interventions as described. Detailed information about the interventions allows the reader to assess the applicability of results of the trial. 47

3 Item 5: Objectives Describe specific objectives and hypotheses the trial was designed to answer. (The position of this item as number 5 in this list emphasises that objectives must be specific, i. e. apply to the reference population and the interventions described.) Objectives should be clearly defined and reasonable in number. Item 6: Outcomes Define clearly primary and secondary outcome measures and, when applicable, any methods used to enhance quality of measurements (e. g. multiple observations, training of assessors). For example: a) aim to characterise the full range of effects of interventions, as well as the exact methods used to measure them (e. g. validated scales, consensus guidelines) and methods used to increase the reliability of the measurements; b) state clearly the primary outcome, i. e. the prespecified outcome of greatest importance for which the target sample size was calculated; c) describe secondary outcomes, i. e. other outcomes of interest as well as unanticipated or unintended beneficial or harmful effects of the intervention; d) provide a list of specific adverse events, checked for each participant; e) assess the resources consumed by participants in each group, unless one intervention is dominant (for economic evaluation). (a) to (e) apply, as above. It should be noted that the choice / definition of outcomes and the nomination of primary and secondary outcomes are often poorly specified in cohort studies. Outcomes must be named and defined before analyses are carried out to avoid data mining. Item 7: Sample Size Provide details of how the sample size was determined: a) specify the target difference for the primary outcome; b) justify the clinical relevance of the target difference; c) specify acceptable type 1 and type 2 error rates ( and ); d) specify the control outcome frequency (or SD for continuous outcomes); e) provide details of any allowance made for attrition during the study; f) when applicable, describe any interim analyses (time points, type of statistical analysis), stopping rules and adjustment for multiple analyses; g) state clearly whether the items (a) to (f) above were decided before the trial began or some time thereafter. These issues have important safety and ethical implications. Items (a) to (e) and (g) above apply to cohort studies. The role of interim analyses and stopping rules in prospective cohort studies is uncertain, although there is no intrinsic reason why they should not be applied as for RCTs. Arguably, sample size issues are more important for cohort studies than for RCTs, since (small) cohort studies with negative findings may be more difficult to identify from the literature for systematic reviews. Item 8: Randomisation Sequence Generation How was the randomisation sequence generated? Was it truly a random method? Where there any restrictions or constraints? Describe details of the procedure, for example: a) simple or constrained (blocking, stratification or minimisation) randomisation; b) the unit of randomisation. See below. Item 9: Randomisation Allocation Concealment Specify the method used to implement the random allocation sequence in the enrolment process (e. g. numbered containers, central telephone, etc.). State clearly whether the allocation was concealed until after a participant was recruited and, if yes, whether concealment was successful. As Altman et al. explain [1, p. 673], allocation concealment should not be confused with blinding (item 11). Allocation concealment prevents selection bias, protects the assignment sequence before and until allocation, and can always be successfully implemented. In contrast, blinding seeks to prevent performance and ascertainment/ detection bias, protects the sequence after allocation, and cannot always be implemented. Without adequate allocation concealment, random allocation can be subverted. See below. Item 10: Randomisation Implementation Describe who generated the random allocation sequence, who recruited patients and who allocated patients to treatment. Ideally different people should be responsible for each of these tasks. 48 Reeves/Gaus

4 Two different aspects of enrolling participants into trials should be kept in mind, i. e. generation and implementation. The first encompasses the preparation of the random sequence and concealment of allocation; the second encompasses the enrolment of patients by assessing eligibility, discussing the trial, obtaining informed consent and, subsequently, ascertaining the treatment assignment (e. g. telephoning a central number, opening the next envelope) and administering the intervention. See below. Substitution of Items 8 10 for Cohort These items clearly cannot be applied directly to cohort studies. However, the principle can be applied, i. e. researchers should give as much information as possible about what they did to create groups with alternative treatments for comparisons or what other people (including patients, doctors and nurses, etc.) did to create groups for comparison. Researchers often assume that this information is implicit in the design label used to describe a study. However, labels are often ambiguous and may be incorrectly applied. Hence the emphasis above on the concrete actions that people took to create groups. For example, the following features of studies are relevant: were comparisons made between different groups or within the same group at two or more time points? were the groups generated by the actions of researchers, by differences in time, by geographic differences, by the decisions of people caring for patients, by patient preferences or both? what parts of the study were prospective: identification of participants, assessment of baseline, treatment allocation, assessment of outcomes, generation of hypotheses? Ideally, a list of features is needed that can be applied to any cohort study. The modification of items 8 10 of CONSORT would then simply require researchers to comply with identifying the features that applied to their study. This concept is further discussed elsewhere [3]. Item 11: Blinding (Masking) Specify the blinding process. For example: a) describe whether or not participants, those administering the interventions, those monitoring the data flow, those assessing the outcomes and those analysing the results were aware of group assignment; b) if any of the above were blinded, describe explicitly the mechanism of blinding; c) if any of the above were not blinded, explain why they were not blinded; d) if any of the above were blinded, describe how the success of masking was assessed. In principle, the same issues apply to cohort studies, especially with respect to blinding of the assessment of outcomes. It is less clear how participants and their carers can be blinded in a cohort study; the extent to which blinding is possible is likely to depend on the specific features of a study. However, given that performance and detection biases are associated with knowledge about the treatment comparison, these should be minimised by keeping this information secret. (This might be an advantage of a retrospective cohort study, in which the treatment comparison of interest is only established after the data have been collected. Of course, such a study may be subject to other biases because of its retrospective nature.) Keeping information about the treatment comparison secret does not avoid the placebo effect, if one group of participants perceives that they are not having an active treatment. The implications for reporting of cohort studies are that researchers should report: whether outcome assessment was blinded; how much information patients and their carers had about the research question being investigated. Item 12: Statistical Methods Researchers should: a) describe the statistical methods used to compare groups for primary outcome(s); b) describe the methods for additional analyses, such as secondary outcomes, subgroup analyses and adjusted analyses; c) state whether subgroup and adjusted analyses were specified before the study began: analysis methods should be stated (choice of variables, handling of continuous variables). Note that subgroup analyses should not be data driven (post hoc); researchers should not carry out analyses because there might be something there or that looks interesting. Also, adjusted analyses are not necessary for RCTs; there should be a clear justification for adjusting for anything other than baseline outcome measurements or stratification variables. The methods for adjusted analyses should be prespecified in the study protocol. It is good practice to submit an analysis plan to a data and safety monitoring committee, or to an organisation such as Biomed Central, ideally when the protocol is first written but certainly prior to carrying out any comparative analyses. If an independent body logs and dates the submission of an analysis plan, this allows published and planned analyses to be reconciled [4]. Items (a) to (c) above also apply to cohort studies. However, because cohort studies are susceptible to selection bias, adjusted analyses are likely to be normal, rather than exceptional, practice. Researchers should describe in detail: how the analyses were planned to assess susceptibility to selection bias and to adjust for potential confounding (in the design, 49

5 by matching or stratification; in the analysis: by regression, propensity scores, sensitivity analyses); how they identified potential confounders; whether the assessment of the comparability of treatment groups considered all known confounders / prognostic factors (see item 15). The issues described above about prespecifying analyses and not carrying out exploratory analysis all apply to cohort studies. Because analyses may be more difficult to specify in advance, researchers should describe explicitly how the final analyses differed from the a priori analysis plan. Results: Items Item 13: Participant Flow Describe the flow of participants through each stage of the trial. Researchers: a) are strongly recommended to use a diagram; b) must report for each group of participants the numbers assigned, receiving treatment as intended, completing the study protocol, analysed for the primary outcome; c) should describe protocol deviations, together with subsequent actions; d) should ideally also report the numbers of people who were approached for the trial, who were ineligible and who were unwilling to participate. (a) to (d) all apply to cohort studies, with special emphasis on the reporting of the number of patients receiving treatment as intended and on reporting the numbers approached, ineligible and unwilling. Item 14: Recruitment Report dates that define the periods of recruitment and follow-up; also, state clearly if the trial was stopped early, e. g. because of the results of interim analyses. The same points apply to cohort studies, although it is not clear whether interim analyses and stopping rules have ever been established for prospective cohort studies. Item 15: Baseline Data Describe the baseline demographic and clinical characteristics of each group. This information: a) characterises the study sample, so the reader can assess whether the sample is representative of the reference population and hence whether the results are applicable; b) describes the distribution of key prognostic factors, so the reader can be reassured that randomisation has worked and that these factors are similarly distributed in each group. Note that statistical tests of the comparability of groups on key prognostic factors should not be carried out. The same points apply to cohort studies, although the susceptibility of cohort studies to selection bias makes the description of the distribution of prognostic factors in each group more critical. Researchers need to satisfy the reader that all important confounders have been taken into account; this may require a systematic review of potential confounding factors. Statistical tests of the comparability of groups on key prognostic factors may be carried out; however, evidence that groups do not differ significantly does not rule out the possibility of confounding. Item 16: Number of Participants Analysed Specify the number of participants (denominator) in each group included in the analysis and whether the analysis was by intentionto-treat and/or by protocol. State the results in absolute numbers (e. g. 10/20, not 50%); this requirement allows the reader to compare the outcome frequencies with those observed in other research reports or settings. The same issue applies to cohort studies. However, the distinction between the intended treatment and treatment received by a patient is rarely made in cohort studies. Researchers should try to establish this distinction as part of data collection, including the date when the decision (intention) to provide a particular treatment was made. Item 17: Outcomes and Estimation For each primary and secondary outcome the report should include: a) a summary of the results for each group, i. e. the number of participants, the mean and SD (or the number and proportion/percentage); 50 Reeves/Gaus

6 b) effect size estimates for each outcome, i. e. the difference in means or percentages, or the relative risk (e. g. risk, odds or hazard ratio); c) the precision of the effect size estimates, i. e. 95% confidence intervals or standard error; p-values may also be included but should not be used instead of confidence intervals; d) report all pre-specified outcomes and analyses, even if they are not significant; e) do not report non-specified outcomes and analyses, even if they are significant, unless they are clearly identified as being exploratory (see item 18); For binary outcome data, it may be helpful for interpretation of the results of the trial to express the results in terms of the number needed to treat for benefit (NNT) or harm (NNH). However, NNT and NNH can be misleading, if the trial population is unrepresentative; usually it is the relative effect that can be applied with greater confidence. All of the above points apply to cohort studies. However, the analysis of a cohort study is potentially much more complex. As a minimum, researchers should: Provide unadjusted and adjusted results (if adjusted analyses were carried out); Provide adjusted results according to different methods (if different methods were used); For adjusted results, describe which confounding factors were taken into account. Because of the complexity of analyses of observational data, other relevant factors are likely to emerge. Item 18: Ancillary Analyses Address multiplicity by reporting any other analyses performed, including subgroup analyses and adjusted/unadjusted analyses, indicating which were pre-specified and which were exploratory. In general: a) avoid carrying out exploratory (data driven) analyses unless there are very good reasons for them; b) report both significant and non-significant findings. (It is often difficult to publish non-significant effects but it is very important that they are publicly reported. Otherwise systematic reviews will be affected by reporting biases, which can ultimately harm future patients. Institutional websites now provide one way of publicly reporting studies in greater detail, if paper journals limit the available space.) All of the above points apply to cohort studies. However, because of the complexity of analyses of observational data, other relevant factors may emerge. Item 19: Adverse Events List and describe all important adverse events or harmful (and beneficial but unintended) side effects in each group. Provide operational definitions for the measures of the severity of adverse effects. The same conditions apply for reporting these outcomes as for other outcomes. Comment and Discussion: Items Item 20: Interpretation The following issues should be considered: a) consider the findings in relation to the study hypotheses, while keeping in mind the difference between statistical significance and clinical importance; b) specify the limitations of the study, in particular with respect to the susceptibility of the trial to biases (see below) and the methods used to minimise bias; c) consider alternative possible mechanisms and explanations for the main effects as well as side effects; d) describe the findings in the context of results from other relevant studies; e) consider sources of measurement error (imprecision), e. g., measures of primary outcomes, and the likely consequences; f) keep in mind the dangers associated with multiplicity of analyses and outcomes; g) report the researchers insights about the trial. In addition, discuss in detail the internal validity of the trial, i. e. the potential for systematic error (bias) arising from: selection bias: biased allocation to treatment groups; performance bias: unequal provision of care apart from treatment under study; detection bias: biased assessment of outcome(s); attrition bias: biased handling of deviations from protocol and losses to follow-up. All of the above points apply to cohort studies. However, because of the complexity of analyses of observational data and the increased susceptibility of cohort studies to biases, other relevant factors may emerge. Item 21: Generalisability Discuss the applicability (generalisability / external validity) of the trial: 51

7 a) How widely can the findings of the study be applied? b) What reasons are there for limiting the applicability of the findings? c) What are the likely consequences of specific exclusions? d) Is there a sound basis for generalisation? This is a matter of judgement about distortion of assembly of the study population. e) Can the results be applied to other circumstances? Consider: populations (age, gender, severity of disease, risk factors, co-morbidity), settings (level of care (primary, secondary, tertiary), experience and specialisation of provider), treatment variables (dosage, timing, type of treatment, other treatments), measurement variables (type or definition of outcomes and duration of follow-up). Item 22: Overall evidence Give a general interpretation of the results in the context of current evidence. Specifically, relate the findings to findings of other studies (preferably by updating a previous systematic review). Conclusions From the above discussion, it is clear that the majority of CON- SORT items (items 3 7, 13 16, 19 22) do not require any modification for their application to cohort studies. Other items require only minor modification (items 1, 2, 11 and 18). Only items relating to the ways in which comparison groups were formed (items 8 10), description of statistical methods (item 12) and the reporting of outcomes and the estimation of treatment effects (item 17) require substantial modification. Researchers reporting cohort studies should be encouraged to apply the CONSORT guidelines as closely as possible, taking into consideration the points discussed in this paper. Methodologists should urgently work on a modified checklist of items for cohort studies. References 1 Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Ann Inter Med 2001;134: Moher D, Schulz KF, Altman DG: The CON- SORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285: Also published in: Ann Inter Med 2001;134: and Lancet 2001;357: Reeves BC: A framework for classifying study designs to evaluate health care interventions. 2004;11(suppl 1):13 17 (this issue). 4 Chan AW, Hrobjartsson A, Haar MT, Gotzsche PC, Attman DG: Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 2004;291: Reeves/Gaus