PROTOCOL: Reporting Guidelines Systematic Review 1. Dr. David Moher

Transcription

1 PROTOCOL: Reporting Guidelines Systematic Review Dr. David Moher Project relevance Effective knowledge translation of primary health research and knowledge syntheses is likely to be most successful when there is adequate reporting of their methods and results. When knowledge translation is compromised by inadequate reporting, scarce resources are compromised, and patient care may suffer. Mental health continues to be a major health concern in many parts of the world. It is estimated that 5% to 10% of people are affected by depression (1). In 2006 the United States National Institutes of Health spent about a third of a billion dollars on research into depression. Yet reports of randomized trials evaluating interventions to optimally manage individuals with depression are disturbingly inadequate, likely making their results of limited use to healthcare professionals, other decision makers and patients. Hotopf and colleagues (2) examined reports of 122 randomized controlled trials (RCTs) evaluating medical interventions for managing individuals with depression and found that only one provided any details about the randomization process used, fundamental to any RCT, and a process that should be included in a report of every trial. Such problems are not unique to reports from depression trials. Inadequate reporting is pervasive to almost every area of health research (3 9). Without complete, clear and transparent reports, readers cannot judge the reliability and usefulness of health research. Optimal reporting of all health research is an attainable goal for all researchers. Inadequate methods and/or reporting are also associated with biased estimates of treatment effectiveness. In a landmark study published in 1995 (10) Schulz, Altman (members of the present application team), and colleagues examined 250 reports of randomized trials included in 33 meta analyses examining a variety of treatments in obstetrics and gynecology. They found that trials that used and/or reported inadequate methods of allocation concealment, compared to those trials where this information was adequate, exaggerated the estimates treatment s effectiveness by about 30%, on average. Since that publication several other researchers have examined the relationship between the quality of reporting and bias. The results of a recent systematic review (11) of these papers substantiated the results first reported by Schulz and colleagues. A simple way to improve the clarity, completeness, and transparency of reporting is to provide authors and journal editors with a mechanism to follow so that their reports will be of value to readers. Reporting guidelines are one way to achieve this goal. If developed appropriately reporting guidelines provide authors with a checklist of items that should be addressed when reporting a study. Some reporting guidelines also recommend a flow diagram so that authors can report the stages of their research project including the process of participants (or study reports) throughout the study s conduct. PROTOCOL: Reporting Guidelines Systematic Review 1

2 Few reporting guidelines specifically aimed at improving the reporting of healthcare research existed prior to the mid 1900s. The International Committee of Medical Journal Editors (ICMJE) developed reporting guides almost 30 years ago (12). However, they were focused on the format of health research reports rather than the scientific details. In 1983 the British Medical Journal (13) developed guidance for authors on statistical issues. However, it aimed to help peer reviewers assess manuscripts and so did not distinguish between conduct and reporting and thus might have confused potential users. The British Journal of Obstetrics and Genecology (14), and the Canadian Medical Association Journal (15), in 1989 and , respectively, published a series of reporting recommendations for different types of designs. For example, for reporting randomized controlled trials (RCTs) the CMAJ asked authors to provide sufficient details about the control and intervention treatments so that readers can apply the same intervention to their patients (15). Unfortunately the guidance did not address several points, such as asking authors to describe issues related to the randomization and allocation of participants. With funding from the CIHR the nominated principal applicant (of this application) convened a meeting, in Ottawa, in 1993, that ultimately resulted in the CONSORT Statement (16), an evidence based reporting guideline consisting of a 22 item checklist and flow diagram, for reporting RCTs. CONSORT is perhaps the most widely endorsed reporting guideline, to date. More than 300 journals endorse the CONSORT Statement and ask authors, usually in their Instructions to Authors section of their journals, to report their randomized trials according to the criteria set out in CONSORT checklist and flow diagram. The results of a recent systematic review (17) of studies evaluating the use of the CONSORT Statement indicates that it an effective intervention for helping to improve the quality of reporting of RCTs in journals endorsing this reporting guideline. While some healthcare journals advocate for using reporting guidelines (18) as well as endorsement and adherence to them, fewer than 50% of journals endorse the CONSORT Statement and less than 10% of these journals endorse any CONSORT extensions (19), of which there are several. Since the publication of the CONSORT Statement in 1996 an increasing number of reporting guidelines have been developed; at least 80 of them now exist. They cover a broad spectrum of health research. Examples include recommendations for specific study designs (e.g., diagnostic accuracy studies 20), types of data (e.g., harms assessed in randomized trials 21) and sharing of data for microarray experiments (22). However, no formal systematic review of reporting guidelines exists today, except for reporting of searches for systematic reviews (23). A core set of steps are likely required to optimally develop any reporting guideline. These include: a review of existing literature to help identify previous relevant guidance, existing literature on the quality of reporting of the content area the guidance is aimed at improving, a face to face meeting, drafting of the guidance, including checklist, and accompanying explanatory paper, strategy for endorsement and adherence within journals, evaluating the guideline, and updating the guidance (24). PROTOCOL: Reporting Guidelines Systematic Review 2

3 However, a recent survey of 30 developers of reporting guidelines suggests that most guidelines are created idiosyncratically (25). This may result in some reporting guidelines being more robustly developed, and therefore useful, than others. For example, results of surveying guideline developers indicate inconsistency in the process of their development and not all of them went through the steps identified (e.g., steps identified above). Similarly, the survey (25) noted inconsistencies in how the guideline developers disseminated their guidance: 57% of respondents (n=17) did not use any implementation strategies to increase the uptake of the guidance by journals; 80% (n=24) have not performed any formal evaluation of the impact of the guidance on the quality of reporting of research it was aimed at improving; and 83% indicate the need to update the reporting guideline. To date, no formal systematic review identifying, documenting, and characterizing reporting guidelines have been published. A review of the Cochrane library (August 2008) indicates that no title and/or protocol for such a systematic review is registered. Given the dramatic increase in the number of guidelines developed over the last few years we believe the results of the proposed systematic review will provide guidance to editors in several relevant areas: identifying and documenting existing reporting guidelines, characterizing existing reporting guidelines including their: topic area, development, endorsement, uptake and evaluation; provide a better understanding as to the development of reporting guidelines. The results of such a systematic review will also provide invaluable data on any efforts to develop a system to appraise reporting guidelines. A similar approach has been successfully developed for appraising clinical practice guidelines (26) Our team includes several editors of general medical and specialty journals, experts in the design, conduct, and reporting of systematic reviews, knowledge translation, and an experienced information specialist. Members of the research team have contributed to the reporting guidelines literature in several areas: developed the CONSORT Statement, and several CONSORT extensions, an early evidence based reporting guideline; have been substantively involved in the development of many other reporting guidelines (e.g., QUOROM for reporting meta analysis of randomized trials; STROBE for reporting guideline for observational studies; REMARK for Reporting recommendations for tumor marker prognostic studies), completed primary evaluations of reporting guidelines including a systematic review of the effectiveness of CONSORT, developed guidance on how to develop health related reporting guidelines, and developed the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network, network.org (27). This new international initiative was developed to improve the quality of scientific publications by promoting transparent and accurate reporting. The network is developing resources and training related to the reporting of health research and will assist in the development, dissemination and implementation of robust reporting guidelines. One of the Network s major goals is to build a comprehensive web based resource centre to develop and maintain up to date information, tools and other materials related to reporting health research. The proposed systematic review will help meet this goal. PROTOCOL: Reporting Guidelines Systematic Review 3

4 Research Plan The proposed knowledge synthesis will be completed using a conventional systematic review approach (28). The objectives of the systematic review are to identify, document, and characterize all existing reporting guidelines developed to improve the clarity, completeness and transparency of health research. We are unaware of any formal definition of a reporting guideline. A working definition (24) is that the principal objective of the researchers is to systematically use a methodology whose primary outcome is a checklist, flow diagram, or explicit text to guide authors reporting a specific type of research. One example is the CONSORT Statement (16). This systematic approach needs to be differentiated from other efforts even if a checklist is produced, such as the recent effort to improve the design and conduct of self administered surveys (30). This definition would also exclude formatting guidance for reporting health research, such as some journals Instructions to Authors. The systematic review will be completed based on an a priori protocol, the basis of which will be the research plan described below. Our scope will be health research, namely, clinical, basic science, and laboratory research. The following inclusion criteria will be used: the report must be a reporting guideline using the definition described above. The reporting guideline can document using any methodology for its development. We will search for reports in any language and publication status. However, due to limited resources we will only include reporting guidelines published in English or French. Editorials, commentaries and reports referring to a reporting guideline will be excluded. At the beginning of the project (Spring 2009) we will search the following electronic databases (Ovid interface) from their inception to December 2008: MEDLINE, EMBASE, Cochrane Library, CINAHL, Web of Science, PsycINFO, Google, and the Instructions to Authors sections of 166 general and specialty journals (19). Additionally, we will review the EQUATOR Network website to help identify reporting guidelines. We will also contact the corresponding author of each reporting guideline included in the review to ascertain the existence of other reporting guidelines. We will update the electronic searches prior to publication of the systematic review, namely, during any publication revision phase, thus providing the most up to date data sources. We will use the following electronic search strategy (see Appendix 1) to search MEDLINE. The search strategy will be peer reviewed prior to implementation (29). This search strategy will be translated for each electronic database included in the review. Following the execution of all of our searches the identified records (titles, and abstracts, whenever available) will be collected together in a database (Reference Manager) for de duplication. The resulting unique records will be exported to an Internet based software (SRS) for screening. Two members of the research team, independently, will screen all the records using the following broad criteria: reported in English or French (reporting guidelines in other languages will be documented and not included any further in the systematic review); appear to be a reporting guideline, and not a commentary/editorial. Disagreements between both screeners will be passed to the next level, namely, the eligibility criteria. The full text reports of those records passing the board screen will be PROTOCOL: Reporting Guidelines Systematic Review 4

5 retrieved and uploaded to SRS. Two members of the research team, independently, will apply the eligibility criteria (described above) to the full text articles. Any disagreements will be resolved through consensus and/or third party (another member of the research team) arbitration. All articles passing the eligibility criteria will be considered included in the systematic review. To document, describe and characterize the reporting guidelines we will extract the information collected on specifically designed data abstraction forms. These forms will be pilot tested prior to completing the formal data abstraction. The pilot test will be completed by the research associate (hired for this project) and a member of the research team. Following the pilot test the research associate will complete all of the data abstraction of all the included reporting guidelines. For quality assurance purposes a 10% random sample will be data abstracted, independently, by a member of the research team. Each completed data abstraction form will be sent to the respective reporting guideline corresponding author to confirm our data extraction process and to respond to any data queries. We will document the title, language of publication, name (and coordinates) of the corresponding author, the number of items included in any checklist, the use of a flow diagram. We will also abstract data from each reporting guideline based on four phases we believe are relevant when developing a reporting guideline (24): pre meeting activities; the meeting; post meeting activities leading to publication; and post publication activities. For the pre meeting activities we will abstract whether the following information was reported: rationale for developing the reporting guideline (review the literature to identify previous relevant guidance, relevant evidence on the quality of reporting in published research articles, and key information related to the potential sources of bias in such studies); identification of stakeholders; whether and how funding was obtained; the number of participants invited and attending any face to face meetings; the meeting venue; a list of participants invited and attended any meetings; inclusion and/or reference to existing literature; whether a Delphi exercise was conducted; whether a program for any meeting was prepared; whether presentations on relevant background topics, including summary of evidence (present results of Delphi exercise, if done; discuss potential checklist items; and invite session chairs) formed part of the main face to face meeting; whether meeting logistics were documented; and whether any prepared materials were sent to participants prior to meeting; and whether the main meeting was recorded. For the meeting phase we will abstract: whether the meeting objectives were clarified; the presentation and/or discussion of any pre meeting activities (presentation and discussion of relevant evidence, develop specific guidelines by structured discussion, discuss strategy for producing documents; identify who will be involved in which activities; discuss authorship, and reported discussing dissemination strategy). For the post meeting activities leading to publication we will abstract: details on the drafting of the guidance; how feedback was incorporated; piloting of the guidance, such as the checklist and diagram; the development of an explanatory document; publication strategy; and website development. PROTOCOL: Reporting Guidelines Systematic Review 5

6 And for the final phase post publication strategies we will abstract: journal endorsement; evaluation of guidance; and handling of criticism. To ascertain information on evaluations of the reporting guidelines we will use develop formal searching criteria and use the related articles feature in PubMed. We are unable to identify of any method developed to assess the risk of bias (quality) of reporting guidelines. Given this content area is new the empirical literature underpinning it is limited. As part of our ongoing research program we have developed guidance on issues to consider when developing a reporting guideline (24). As part of this process we developed a 24 item checklist. A subset of these items is deemed essential to include when developing a reporting guideline, such as face to face meeting. We will use this subset of 11 checklist items as a surrogate for a risk of bias assessment. We will report the numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, with a flow diagram. For each included study we will present characteristics for which data were extracted (see details above). We will describe the reporting guidelines in a series of tables and a narrative summary. We are not proposing any quantitative analyses. We will report the systematic review using the PRISMA Statement (31), a reporting guideline for reporting systematic reviews of healthcare interventions. This guidance is a substantive very recent update of the QUOROM Statement (32). PROTOCOL: Reporting Guidelines Systematic Review 6

7 References 1. Agency for Healthcare Research and Quality: Depression in Primary Care. Washington: US Department of Health and Human Services, Hotopf M, Lewis G, Normand C: Putting trials on trial the costs and consequences of small trials in depression: a systematic review of methodology. Journal of Epidemiology and Community Health 1997; 51: Zhang D, Yin, Freemantle N, Jordan R, Zhong N, Cheng KK. An assessment of the quality of randomized controlled trials conducted in China. Trials 2008, 9:22 4. Chan, AW, Altman, DG, Epidemiology and Reporting of Randomised Trials Published in PubMed Journals, Lancet, 2005, 365, Moher D, Tetzlaff J, Tricco A, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007; 4(3):e78.doi: /journal.pmed Chan S, Bhandari M. The quality of reporting of orthopaedic randomized trials with use of a checklist for nonpharmacological therapies. The Journal of Bone & Joint Surgery 2007;89: Smith BA, Lee HJ, Lee JH, Choi M, Jones DE, Bausell RB, Broome ME. Quality of reporting randomized controlled trials (RCTS) in the nursing literature: application of the consolidated standards of reporting trials (CONSORT). Nursing Outlook 2008;56: Yesupriya A, Evangelou E, Kavvoura FK, Patsopoulos NA, Clyne M, Walsh MC, et al. Reporting of Human Genome Epidemiology (HuGE) association studies: An empirical assessment. BMC Medical Research Methodology 2008;8 9. Lee W, Bindman J, Ford T, Glozier N, Moran P, Stewart R, Hotopf M. Bias in psychiatric case control studies: literature survey. British Journal of Psychiatry. 2007;190: Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273: Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta analyses of randomized trials. International Journal of Epidemiology 2007; 36: PROTOCOL: Reporting Guidelines Systematic Review 7

8 12. International Committee of Medical Journal Editors, Uniform Requirements for Manuscripts Submitted to Biomedical Journals, (add access date) 13. Gardner MJ, Machin D, Campbell MJ. Statistical checklists. In: Statistics with Confidence. 2nd edition. Edited by Altman DG, Machin D, Bryant TN, Gardner MJ. London: BMJ Books, 2000: Grant A. Reporting controlled trials. British journal of Obstetrics and Gynaecology 1989;96: Squires BP, Elmslie TJ: Reports of randomized controlled trials: what editors want from authors and peer reviewers. Canadian Medical Association Journal 1990; 143: Moher D, Schulz KF, Altman DG, for the CONSORT group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. Annals of Internal Medicine 2001;134: Plint AC, Moher D, Morrison A, Schulz KF, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomized controlled trials: a systematic review. Medical Journal of Australia 2006; 185: The PLoS Medicine editors (2008). Better reporting, better research: guidelines and guidance in PLoS Medicine. PLoS Med 5(4):e Hopewell S, Altman DG, Moher D, Schulz KF. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal Instructions to Authors. Trials 2008,9: Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW for the STARD group: Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Annals of Internal Medicine 2003; 138: Ioannidis JPA, Evans S, Gotzsche PC, O Neill RT, Altman DG, Schulz KF, Moher D, for the CONSORT Group. Better Reporting of Harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine. 2004;141: Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C, Aach J, Ansorge W, Ball CA, Causton HC, Gaasterland T, Glenisson P, Holstege FC, Kim IF, Markowitz V, Matese JC, Parkinson H, Robinson A, Sarkans U, Schulze Kremer S, Stewart J, Taylor R, Vilo J, Vingron M: Minimum information about a microarray experiment (MIAME) toward standards for microarray data. Nature Genetics 2001; 29: PROTOCOL: Reporting Guidelines Systematic Review 8

9 23. Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus of search reporting methods for systematic reviews. Journal of Clinical Epidemiology 2008;61: Moher D, Schulz KF, Altman DG. Considerations when developing guidelines for reporting health research: a structured approach. Submitted for publication consideration. 25 Simera I, Altman DG, Moher D, Schulz KF, Hoey J. Guidelines for reporting health research: the EQUATOR Network s survey of guideline authors. PLoS Medicine 2008 Jun 24;5(6):e The AGREE Collaboration. Writing Group: Cluzeau FA, Burgers JS, Brouwers M, Grol R, Mäkelä M, Littlejohns P, Grimshaw J, Hunt C. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality and Safety in Health Care 2003; 12: Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. The Lancet 2008;31: Tetzlaff J, Tricco A, Moher D. Knowledge Creation: Knowledge synthesis, In: Knowledge to action: the science and practice. Eds Straus SE, Tetroe J, Graham ID. Oxford: Blackwell/Wiley/BMJ Publishing 29. Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; Burns KEA, Duffett M, Kho ME, Meade MO, Adhikari NKJ, Sinuff T, Cook DJ, for the ACCADEMY Group. A guide for the design and conduct of self administered surveys of clinicians. Canadian Medical Association Journal 2008;179: Moher D, Liberati A, Tetzlaff J, Altman DG, and the PRISMA Group. Preferred reporting items for systematic reviews and meta analyses: the PRISMA Statement. Submitted of publication consideration 32. Moher D, Cook DJ, Eastwood S. Olkin I, Rennie D, Stroup D. for the QUOROM group. Improving the quality of reporting of meta analysis of randomized controlled trials: the QUOROM statement. The Lancet 1999;354: PROTOCOL: Reporting Guidelines Systematic Review 9

10 Appendix 1 Medline search strategy for reporting guidelines 1. GUIDELINE$.TI,AB. 2. CHECKLIST$.TI,AB. 3. RECOMMENDATION$.TI,AB. 4. STANDARD$.TI,AB. 5. REQUIREMENT$.TI,AB. 6. INSTRUCTION$.TI,AB. 7. GUIDANCE$.TI,AB. 8. POLICIES.TI,AB. 9. POLICY.TI,AB. 10. QUALITY.TI,AB. 11. RESEARCH.TI,AB. 12. (1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11).TI,AB. 13. (12 NEAR REPORTING).TI,AB. 14. (12 NEAR PUBLISHING).TI,AB. 15. (12 NEAR PUBLICATION).TI,AB. 16. (12 NEAR GOOD ADJ PRACTICE).TI,AB. 17. ((REPORTING OR PUBLISHING OR PUBLICATION) NEAR GOOD ADJ PRACTICE).TI,AB. 18. (13 OR 14 OR 15 OR 16 OR 17).TI,AB YEAR > YEAR = 1996 PROTOCOL: Reporting Guidelines Systematic Review 10