Methods to select results to include in meta-analyses deserve more consideration in systematic reviews

Transcription

1 Journal of Clinical Epidemiology 68 (2015) 1282e1291 Methods to select results to include in meta-analyses deserve more consideration in systematic reviews Matthew J. Page a, *, Joanne E. McKenzie a, Marisa Chau b, Sally E. Green a, Andrew Forbes c a Australasian Cochrane Centre, School of Public Health and Preventive Medicine, Monash University, Level 1, 549 St Kilda Road, Melbourne, Victoria 3004, Australia b National Trauma Research Institute, Central Clinical School, Monash University, Commercial Road, Melbourne, Victoria 3004, Australia c Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University, Level 6, The Alfred Centre,99 Commercial Road, Melbourne, Victoria 3004, Australia Accepted 23 February 2015; Published online 6 March 2015 Abstract Objectives: To investigate how often systematic reviewers encounter multiple trial effect estimates that are available for inclusion in a particular meta-analysis (multiplicity of results) and the methods they use to select effect estimates. Study Design and Setting: We randomly sampled Cochrane and MEDLINE-indexed non-cochrane reviews published between January 2010 and January The first presented meta-analysis of an effect measure for a continuous outcome in each review was identified, and methods to select results to include in this meta-analysis were extracted from review protocols and reviews. All effect estimates that were available for inclusion in the meta-analyses were extracted from trial reports. Results: We examined 44 reviews. Multiplicity of results was common, occurring in 49% of trial reports (n 5 210). Prespecification of decision rules to select results from multiple measurement scales and intervention/control groups (in multi-arm trials) was uncommon (19% and 14% of 21 review protocols, respectively). Overall, 70% of reviews included at least one randomized controlled trial with multiplicity of results, but this occurred less frequently in reviews with a protocol (risk difference, 25%; 95% confidence interval: 52%, 1%). Conclusion: Systematic reviewers are likely to encounter multiplicity of results in the included trials. We recommend that systematic reviewers always consider predefining methods to select results to include in meta-analyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be most valuable. Ó 2015 Elsevier Inc. All rights reserved. Keywords: Systematic review; Meta-analysis; Randomized controlled trials; Reporting; Bias; Research methodology Conflict of interest: M.J.P. has roles in The Cochrane Collaboration including systematic review trainer for the Australasian Cochrane Centre; Methodological Editor for the Depression, Anxiety, and Neurosis Group; member of the Bias Methods Group, Statistical Methods Group, and Trainer s Network; and author of Cochrane systematic reviews. J.E.M. has roles in The Cochrane Collaboration including Co-convenor of the Statistical Methods Group; member of the Methods Executive, Methods Board, and the Bias Methods Group; Statistical Editor for the Consumers and Communication Review Group; Editor of Cochrane Methods; and author of Cochrane systematic reviews. M.C. has a role in The Cochrane Collaboration as author of Cochrane systematic reviews. S.E.G. has roles in The Cochrane Collaboration including Co-Director of the Australasian Cochrane Centre; past editor of the Cochrane Handbook for Systematic Reviews of Interventions; and author of Cochrane systematic reviews. A.F. has a role in The Cochrane Collaboration as member of the Statistical Methods Group. The views expressed in this article are those of the authors and not necessarily those of The Cochrane Collaboration or its registered entities, committees, or working groups. * Corresponding author. Australasian Cochrane Centre, School of Public Health and Preventive Medicine, Monash University, Level 6, The Alfred Centre, 99 Commercial Road, Melbourne, Victoria 3004, Australia. Tel.: þ ; fax: þ address: matthew.page@monash.edu (M.J. Page). 1. Introduction Systematic reviews of randomized controlled trials (RCTs) of health care interventions have the potential to have a major impact on patient health, research agendas, and policy making. However, the validity of systematic review findings can be compromised by challenges in undertaking meta-analysis. One challenge is that multiple effect estimates in a trial report may be available for inclusion in a particular meta-analysis [1,2]. For example, a trial report may present effect estimates for two depression scales, at week three, six, and nine, each analyzed as unadjusted and adjusted for covariates. Multiplicity of effect estimates may lead to selective inclusion of results, whereby the process for selecting the trial effect estimates for inclusion in a meta-analysis is based on the estimates themselves, which may, in turn, result in biased meta-analytic effects [3]. Several organizations that produce systematic reviews (e.g., [4e6]) have recommended methods that aim to /Ó 2015 Elsevier Inc. All rights reserved.

2 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e What is new? Key findings Authors of Cochrane and non-cochrane systematic reviews can expect to commonly encounter multiple eligible effect estimates in trials when they do not prespecify methods to select results to include in meta-analyses. Reporting of particular methods to select results to include in meta-analyses (e.g., predefining which measurement scales or time points are preferred for inclusion when multiple are available) varied across systematic reviews. There was a mismatch between the types of multiplicity that were commonly encountered in the trials (i.e., measurement scales and intervention/ control groups) and the decision rules reported in the review protocols. What this adds to what was known? Previous studies have found that multiplicity of results was common in trials included in Cochrane reviews, although methods to select results to include in meta-analyses were rarely predefined in review protocols. We explored, in both Cochrane and non-cochrane reviews, the frequency and types of multiplicity of results that arise in trials, and the frequency and types of methods to select results to include in meta-analyses that are reported in both review protocols and reviews. We also explored whether these frequencies were modified by the existence of a review protocol and the clinical condition of the review. What is the implication and what should change now? In systematic review protocols, we recommend that authors more frequently consider predefining methods to select results to include in metaanalyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be of most value. In systematic reviews, we recommend that authors more frequently report whether multiplicity of results was encountered in trial reports, the methods used to select results to include in meta-analyses, and whether these methods were developed a priori or post hoc. reduce selective inclusion of results. The methods (specified a priori) aim to uniquely identify results that will be included in a meta-analysis and can be placed in two broad categories, which we label eligibility criteria to select results and decision rules to select results. Eligibility criteria to select results include specifying lists of measurement scales, intervention/control groups, time points, and analyses that systematic reviewers consider eligible to include in the review (ideally based on some clinical or methodological rationale). Providing specific criteria discourages the use of broad outcomes such as pain, and instead encourages specification of details such as the eligible pain measurement scales and time points of interest to the review [1,2]. Predefining eligibility criteria to select results is an effective method to minimize the number of effect estimates available for inclusion in a particular metaanalysis. However, this method may not always identify a single eligible effect estimate per trial, and in such cases, the addition of decision rules is useful. Decision rules are strategies to either select one effect estimate, or combine effect estimates, when multiple are available. An example of a decision rule to select one effect estimate is when commonly encountered measurement scales for a particular outcome domain (e.g., depression) are ranked based on their psychometric properties, and for trials that report the results of more than one scale, the results for the tool with the best measurement properties are selected. Such a strategy has previously been referred to as an outcome data hierarchy [2,7,8]. An example of a decision rule to combine effect estimates is when a trial includes more than one active treatment arm (e.g., placebo vs. high-dose drug vs. low-dose drug), and rather than selecting data from only one of the active arms (e.g., only one dosage group), data from all active treatment arms are combined (e.g., any dosage vs. placebo) [9,10]. To our knowledge, only two previous studies have investigated multiplicity of results in trial reports or the methods systematic reviewers use to select results to include in meta-analyses [2,11]. In the first study, that examined interobserver variation in results extracted from trials for use in meta-analyses, decision rules to select final vs. change from baseline values were reported in 4 of 10 review protocols [11]. In the second study [2], that examined the impact of multiplicity of trial results on metaanalysis results, multiplicity was found to be common, but methods to select results to include in meta-analyses were rarely predefined. In 83 RCTs included in 19 Cochrane reviews published from 2006 to 2007, 35% of the RCTs had multiple measurement scales, 29% had multiple intervention/control groups (i.e., in multi-arm RCTs), and 36% had multiple time points that were available for inclusion in a particular meta-analysis. In all review protocols, eligibility criteria for measurement scales and intervention/control groups were always defined, and eligibility criteria for time points were defined in eight (42%). In contrast, decision rules to select measurement scales or intervention/control groups were not reported in any of the review protocols, whereas a decision rule to select time points was reported in one review protocol (5%) [2].

3 1284 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e1291 To inform methods guidance regarding inclusion of results when there is multiplicity, several issues still require exploration. First, the protocols in Tendal et al. studies were published before 2006, and it is unclear whether reporting of eligibility criteria and decision rules to select results has changed over time. Second, most systematic reviewers do not report working from a review protocol [12,13], and the methods used to select results to include in such reviews have not been examined. Third, there has been no investigation of the frequency of other types of multiplicity which may arise in RCTs [e.g., reporting of results from intentionto-treat (ITT) and per-protocol or unadjusted and covariateadjusted analyses]. Fourth, no one has examined whether multiplicity of results and reporting of methods to select results to include in meta-analyses differs between clinical conditions. It may be hypothesized that there may be less multiplicity of results for clinical conditions that have core outcome measurement sets available [14e16]. Core outcome measurement sets are measurement scales recommended for use in RCTs and systematic reviews of a particular health condition and are designed to increase consistency in scale selection. Our aim was to investigate multiplicity of results in trial reports and methods systematic reviewers use to select results to include in meta-analyses. The primary objectives were to investigate the frequency and types of: (1) multiplicity of results that arise in RCTs and (2) eligibility criteria and decision rules to select results, which are reported in review protocols and reviews. Secondary objectives were to examine how the extent of multiplicity of results was modified by the existence of a review protocol and the clinical condition of the review and how the reporting of eligibility criteria and decision rules to select results was modified by the clinical condition of the review. We also plan to investigate whether there is evidence of selective inclusion of results in the sample of reviews and what impact this may have on meta-analytic effect estimates [17]; the results of this research will form a subsequent article. continuous outcome meta-analysis, the summary statistics (e.g., means, standard deviations, sample sizes) or effect estimate and precision of each included RCT, and the meta-analytic effect estimate and its precision. We excluded systematic reviews which (1) reported no meta-analyses of continuous outcomes, (2) included nonrandomized studies in all continuous outcome meta-analyses, or (3) used nonstandard meta-analytical methods (e.g., Bayesian, multiple treatments, or individual patient data meta-analyses). We selected the conditions RA/OA and depressive/anxiety disorders because we aimed to explore whether the existence of a core outcome measurement set for RA and OA trials [8,18] impacted on the frequency of multiplicity and methods to select results. Depressive/anxiety disorders reviews, which do not have a core outcome measurement set, were selected because of our familiarity with the measurement scales typically used in this specialty. This familiarity facilitates identification of scales that were eligible for inclusion in a particular meta-analysis. Furthermore, we only focused on continuous outcomes because there is greater scope for multiplicity of continuous outcomes in these clinical areas (e.g., arising from multiple measurement scales, final vs. change from baseline values, adjusted vs. unadjusted means, subscale scores) compared with dichotomous outcomes Search for relevant articles We searched the Cochrane Database of Systematic Reviews and MEDLINE (PubMed interface). Our search strategies (see [17]) included clinical terms recommended by The Cochrane Collaboration Musculoskeletal Review Group [19] and Depression, Anxiety and Neurosis Review Group [20]. For the PubMed search strategy, we combined the clinical search terms with a systematic review search filter used in a previous study investigating the epidemiology and reporting characteristics of systematic reviews [12]. 2. Methods Our study protocol that describes the eligibility criteria, search strategies, selection of systematic reviews, data extraction, and planned analyses is published elsewhere [17]. An overview of the methods is provided here Eligibility criteria We included systematic reviews meeting the following criteria: (1) Cochrane or non-cochrane systematic review published between January 2010 and January 2012; (2) focusing on any intervention(s) for rheumatoid arthritis (RA), osteoarthritis (OA), depressive disorders or anxiety disorders; (3) written in English; (4) reporting references of all included RCTs; and (5) reporting, for at least one 2.3. Sample size and study selection We originally planned to include 40 reviews (see sample size calculation in [17]). During data extraction (before undertaking any data analysis), we increased the target sample size to 44 reviews as we had the resources to complete this additional work. One author (M.J.P.) screened all titles and abstracts identified from the searches and retrieved the full-text reports of potentially eligible records. The citations of full-text reports were imported into Microsoft Excel (Redmond, WA), a random number was assigned to each citation, and the citations were sorted based on the random numbers. One author (M.J.P.) then screened the full-text reports until at least 10 Cochrane RA/OA reviews, 10 non-cochrane RA/OA reviews, 10 Cochrane depressive/ anxiety disorder reviews, and 10 non-cochrane depressive/ anxiety disorder reviews that met the eligibility criteria

4 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e were identified. Screening of full-text reports ceased once 44 eligible reviews were included. Any difficulties in determining whether a review met the inclusion criteria were resolved by discussion with a second author (J.E.M.). The first presented meta-analysis of an effect measure for a continuous outcome [either the mean difference or standardized mean difference (SMD)] in each review which met eligibility criterion 5 (see Section 2.1) was selected for examination (henceforth referred to as the index meta-analysis ). The index meta-analysis may have been identified from the abstract, summary of findings table, or results section of the review, depending on where the meta-analytic effect estimate was first reported in the publication Data extraction Protocols of included reviews and reports of RCTs included in the index meta-analyses were retrieved. One author (M.J.P.) extracted data from all review protocols, reviews, and RCTs using a standardized pilot tested form created in Microsoft Excel. To assess the accuracy of the data extraction, a second author (M.C.) independently extracted data from a random sample of 14 (32%) reviews, including the corresponding review protocols and RCTs. The reviews were selected (by M.J.P.) via assigning a random number to each review, sorting the reviews based on the random number, and selecting the first 14. Discrepancies were resolved via discussion. From each review protocol and review, we extracted descriptions of: (a) the types of participants, interventions, comparisons, and outcomes of interest to the review; (b) all eligibility criteria to select results that were relevant to the index meta-analysis; (c) all decision rules to select results that were relevant to the index meta-analysis. From the RCT reports, we extracted all outcome data that were eligible for inclusion in the index metaanalysis, where eligibility was determined by how the outcome was specified in the review and the methods in the review protocol. For example, if the index metaanalysis outcome was specified in the review as depression and the only predefined method to select results to include in the meta-analysis was that the results of only one scale (e.g., the Beck Depression Inventory (BDI)) were eligible, we extracted all results for the BDI (e.g., at all time points, both unadjusted and covariate-adjusted analyses). In this example, however, we would not have extracted results for any other depression scales reported in the RCTs. As another example, if the index meta-analysis was specified as pain intensity at 12 weeks and the systematic reviewers had not predefined any method to select results to include in the meta-analysis, we extracted all results for pain intensity (e.g., based on any measurement scale, ITT, and per-protocol analyses) from each RCT but at 12 weeks only. Additional limits on data extraction were imposed according to the meta-analytic effect measure used (mean difference or SMD). All trial effect estimates included in mean difference meta-analyses must be in units of one particular scale, although estimates can comprise a mixture of final values and change from baseline values. In contrast, trial effect estimates included in SMD meta-analyses may vary according to the unit of measurement, although it is recommended that all estimates are final values, or change from baseline values, not a mixture [21,22]. Therefore, for mean difference meta-analyses, we only extracted data for the particular scale included by the systematic reviewer but extracted final and change from baseline values when available. We did not extract change from baseline values from trials included in SMD meta-analyses that only included final values (and vice versa for SMD meta-analyses that only included change from baseline values). For reviews without a publicly available protocol, we assumed that no methods to select results to include in metaanalyses were prespecified ( worst-case scenario assumption) and extracted all results based on how the outcome was specified in the review (as per the second example mentioned previously). We believe that this approach was appropriate because few systematic reviewers report working from an unpublished review protocol [12,13]. Even if we could have obtained unpublished review protocols, it would have been difficult to verify whether the methods were specified before or after commencing the review. We only extracted RCT results that were completely reported, where this was defined as reporting sufficient data for inclusion in a meta-analysis (i.e., means, measures of variability and sample sizes per group, or the effect estimate and a measure of precision of the effect estimate) [23]. Unpublished data (e.g., missing standard deviations) were not sought from trialists Analysis Results were summarized using frequencies and percentages [with 95% confidence intervals (CIs)] for binary outcomes and the median and interquartile range (IQR) for continuous outcomes. Risk differences (and 95% CIs) were calculated to examine whether the existence of a review protocol or the clinical condition of the review explained differences in percentages. We used alternative methods to calculate confidence limits for percentages and differences in percentages when it was not appropriate to use asymptotic methods. For percentages, we calculated exact binomial confidence limits [24] using the statistical package Stata (College Station, Tx) [25]. For differences in percentages, we used the method of Mee with the Miettinen and Nurminen modification [26], as implemented in the library PropCIs [27] in the statistical package R (Vienna, Austria) [28].

5 1286 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e Results A flow diagram of the identification, screening, and inclusion of systematic reviews is presented in Fig. 1. Searching yielded a total of 2,590 records. A full-text report was retrieved for 264 records. Of these, 145 were screened and excluded (the most common reasons for exclusion were that no metaanalyses were conducted or no continuous outcomes were analyzed in the review). The target sample size was reached after screening 189 randomly sorted full-text articles (leaving 75 full-text articles that did not require screening). We included 44 systematic reviews, of which 10 were Cochrane RA/OA reviews, 13 were non-cochrane RA/OA reviews, 11 were Cochrane depressive/anxiety disorder reviews, and 10 were non-cochrane depressive/anxiety disorder reviews Characteristics of the included reviews and index meta-analyses Twenty-one reviews (48%) had a publicly available protocol (published between 2006 and 2011), two of which Records identified through database searching (n = 2590) Records screened (n = 2590) Full-text articles retrieved for eligibility assessment (n = 264) Systematic reviews included (n = 44) Duplicates removed (n = 0) Records excluded (n = 2326) Full-text articles screened and excluded (n = 145) No meta-analyses reported (n = 41) No continuous outcomes (n = 30) Not a systematic review (n = 17) No RCTs identified (n = 15) No RCT effect estimates or measures of variability reported (n = 10) Both RCTs and non-randomised studies included in meta-analyses (n = 9) Ineligible clinical condition (n = 9) Secondary publication of a Cochrane review (n = 7) Non-standard meta-analysis methods used (n = 5) No references of included studies reported (n = 1) Language other than English (n = 1) Screening of full text not necessary because target sample size reached (n = 75) Fig. 1. Flow diagram of identification, screening, and inclusion of systematic reviews. RCT, randomized controlled trial. were non-cochrane reviews. The most common outcome domains analyzed in the index meta-analyses were depression or pain. Eleven other outcome domains were analyzed in at least one index meta-analysis. There was an approximately equal distribution of index meta-analyses that were labeled primary vs. nonprimary outcomes and that investigated the efficacy/effectiveness of a pharmacologic vs. nonpharmacologic intervention. Most index meta-analyses used the SMD effect measure, fitted a random-effects model, and examined a placebo/no intervention controlled comparison. A third of the index meta-analyses were defined by a single measurement scale (e.g., Hamilton Rating Scale for Depression score ), and just over half were defined by a single time point (e.g., pain at 6 weeks ; Table 1). A total of 276 RCTs were included in the 44 index meta-analyses, with a median of five RCTs (IQR 3e8; range 2e21) per meta-analysis Multiplicity of results in RCT reports Two-hundred ten (of the 276) RCTs were included in reviews without any prespecified methods to select results to include in meta-analyses. We used this subset of RCTs to estimate the extent of multiplicity of results that systematic reviewers can expect to encounter when no eligibility criteria or decision rules to select results are predefined. Of the 210 RCTs, 49% (95% CI: 42%, 56%) had multiple effect estimates that were available for inclusion in a particular meta-analysis, with a median of one (IQR 1e3; range 1e21) eligible effect estimate per RCT (see Supplementary Table 1/Appendix at The most common types of multiplicity arose from multiple measurement scales and intervention/control groups (in multi-arm RCTs; Table 2). The least common types of multiplicity arose from analyses of both final and change from baseline values and both unadjusted and covariateadjusted analyses. The association between clinical condition and overall multiplicity of results was uncertain. The RA/OA RCTs were slightly less likely to have any multiplicity than depressive/anxiety disorder RCTs, although the 95% CI included the possibility of more likely and no difference (risk difference 7%; 95% CI: 20%, 7%). When stratified by type of multiplicity, RA/OA RCTs were less likely to have analyses undertaken on multiple samples (e.g., ITT and per-protocol) and multiple measurement scales (although the 95% CI for the latter type included no difference) but were more likely to have multiple time points available for inclusion in a particular metaanalysis Eligibility criteria and decision rules reported in systematic review protocols Of the review protocols (n 5 21), all included at least one eligibility criterion and 81% included at least one

6 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e Table 1. General characteristics of systematic reviews (n 5 44) Characteristics n (%) a Review protocol status Published or available from a Web site 21 (48) Unavailable or no mention that a protocol was used 23 (52) Clinical condition RA or OA 23 (52) Depressive or anxiety disorder 21 (48) Included RCTs Total number in systematic review, median (IQR) 13 (7e25) Number in index meta-analysis, median (IQR) 5 (3e8) Outcome domain of index meta-analysis Depression 18 (41) Pain 12 (27) Function 3 (7) Swollen joint count 2 (5) Other (anxiety, obsessive compulsive symptoms, 9 (20) posttraumatic stress disorder symptoms, aerobic capacity, fatigue, joint space narrowing, physical activity, quality of life, range of motion) Outcome label of index meta-analysis Primary 21 (48) Secondary 9 (20) No label 14 (32) Index meta-analysis effect measure Mean difference 13 (30) Standardized mean difference 31 (70) Index meta-analysis model Fixed effect 10 (23) Random effects 32 (73) Not reported 2 (5) Type of comparison in index meta-analysis Placebo/no intervention controlled comparison 39 (89) Head-to-head comparison 5 (11) Type of active intervention in index meta-analysis Pharmacologic 19 (43) Nonpharmacologic 25 (57) Index meta-analysis specification RA/OA meta-analyses n 5 23 Defined by scale (e.g., Health Assessment 7 (30) Questionnaire score ) Not defined by scale (e.g., Disability ) 16 (70) Defined by time point (e.g., Pain at 6 weeks ) 11 (48) Not defined by time point (e.g., Pain ) 12 (52) Depressive/anxiety disorder meta-analyses n 5 21 Defined by scale (e.g., Hamilton Rating Scale for 8 (38) Depression score ) Not defined by scale (e.g., Depression score ) 13 (62) Defined by time point (e.g., Anxiety at 3 months ) 13 (62) Not defined by time point (e.g., Anxiety ) 8 (38) Abbreviations: RA, rheumatoid arthritis; OA, osteoarthritis; RCT, randomized controlled trial; IQR, interquartile range. a All values given as n (%) except where indicated. decision rule to select results (Table 3). Eligibility criteria for measurement scales and intervention/control groups were predefined in nearly all review protocols (95% and 100%, respectively), whereas decision rules to select results from multiple measurement scales and multiple intervention/control groups were infrequently predefined (19% and 14%, respectively). Decision rules to select results when analyses have been undertaken on multiple samples (e.g., ITT vs. per-protocol) and at multiple time points were most commonly predefined (71% and 62%, respectively). The content of the decisions rules varied considerably across the review protocols (see Supplementary Table 2/ Appendix at The association between clinical condition and prespecification of methods to select results to include in metaanalyses was uncertain. The RA/OA review protocols were more likely to include prespecification of at least one decision rule compared with depressive/anxiety disorder review protocols, although the 95% CI included the possibility of less likely and no difference (risk difference 31%; 95% CI: 7%, 58%). The magnitude and direction of differences in the percentages of particular types of decision rules specified in the RA/OA vs. depressive/anxiety disorder review protocols varied. The largest difference arose in the prespecification of decision rules for multiple measurement scales, where despite a wide 95% CI, the difference was suggestive of decision rules for measurement scales being more commonly reported in the RA/OA review protocols (Table 3) Eligibility criteria and decision rules reported in systematic reviews Of the reviews (n 5 44), all included at least one eligibility criterion and 95% included at least one decision rule to select results (Table 4). In approximately half the reviews, decision rules to select results from multiple measurement scales (55%), multiple intervention/control groups (50%), and analyses undertaken on multiple samples (e.g., ITT vs. per-protocol) (50%) were reported. Decision rules to select results from multiple time points were most commonly reported (73%), whereas decision rules to select results from unadjusted vs. covariate-adjusted analyses were least commonly reported (5%). Similar to the review protocols, the content of the decisions rules varied considerably across the reviews (see Supplementary Table 3/Appendix at For example, of eight reviews where the index meta-analysis outcome was a depression score, each reported a different decision rule to select results from multiple depression scales. Most types of decision rules were less common in the RA/OA reviews compared with the depressive/anxiety disorder reviews, although the 95% CIs of these differences in percentages were wide and included no difference as a possible estimate (Table 4). Of the 21 reviews with a protocol, 19% (95% CI: 5%, 42%) had at least one discrepant eligibility criterion and 52% (95% CI: 30%, 74%) had at least one discrepant decision rule between the review protocol and review (see Supplementary Table 4/Appendix at Addition of a new decision rule was more common than omission or modification of an existing decision rule. The most common type of discrepancy was the addition of a new decision rule to deal with multiple intervention/control groups in multi-arm RCTs.

7 1288 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e1291 Table 2. Number (%) of RCTs with different types of multiplicity of results, in systematic reviews without any prespecified methods to select results a Total n (%), n [ 210 RA/OA n (%), n [ 101 Depression/anxiety n (%), n [ 109 Risk difference (95% CI), % Type of multiplicity Any 103 (49) 46 (46) 57 (52) 7 ( 20, 7) Measurement scales 60 (29) 23 (23) 37 (34) 11 ( 23, 1) Intervention/control groups 35 (17) 16 (16) 19 (17) 1 ( 12, 8) Time points 23 (11) 20 (20) 3 (3) 17 (9, 25) Final and change from baseline values 0 (0) 0 (0) 0 (0) 0 ( 3.4, 3.7) b Analyses undertaken on multiple samples 22 (10) 6 (6) 16 (15) 9 ( 17, 1) (e.g., ITT and per-protocol) Unadjusted and covariate-adjusted analyses 7 (3) 3 (3) 4 (4) 1 ( 7, 5) b Period and paired analyses in crossover RCTs 4 (2) 4 (4) 0 (0) 4 (0, 10) b Abbreviations: RCT, randomized controlled trial; RA, rheumatoid arthritis; OA, osteoarthritis; CI, confidence interval; ITT, intention-to-treat. a Estimates based on 210 RCTs in 27 systematic reviews either: (a) with a protocol without any methods to select results or (b) without a protocol. b Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification [26] Investigation of factors associated with multiplicity of results Overall, 70% of reviews included at least one RCT that had multiplicity of results, but this occurred less frequently in reviews with a protocol (12 of 21 with protocol vs. 19 of 23 without protocol; risk difference 25%; 95% CI: 52%, 1%). Encountering multiplicity of results in RA/ OA and depressive/anxiety disorders reviews was similar, although the 95% CI of the difference in percentages was wide (15 of 23 RA/OA reviews vs. 16 of 21 depressive/anxiety disorder reviews; risk difference 11%; 95% CI: 38%, 16%). 4. Discussion Our investigation of multiplicity of results demonstrates that systematic reviewers can expect to commonly encounter multiple eligible effect estimates in trials when they do not predefine methods to select results to include in meta-analyses. Multiple measurement scales and intervention/control groups (in multi-arm RCTs) were the most common types of multiplicity. At least one eligibility criterion and decision rule to select results were reported in more than 80% of review protocols and reviews; however, the frequency of different types of decision rules varied. In 70% of systematic reviews, multiplicity of results in trial Table 3. Number (%) of systematic review protocols reporting eligibility criteria and decision rules to select results Type of eligibility criterion and decision rule Total n (%), n [ 21 RA/OA n (%), n [ 8 Depression/anxiety n (%), n [ 13 Risk difference (95% CI) a,% Total At least one eligibility criterion 21 (100) 8 (100) 13 (100) 0 ( 34, 24) At least one decision rule 17 (81) 8 (100) 9 (69) 31 ( 7, 58) Measurement scales Eligibility criteria 20 (95) 8 (100) 12 (92) 8 ( 27, 34) Decision rule 3 (19) b 3 (75) c 0 (0) 75 (29, 96) Intervention/control groups Eligibility criteria 21 (100) 8 (100) 13 (100) 0 ( 34, 24) Decision rule 3 (14) 0 (0) 3 (23) 23 ( 51, 14) Time points Eligibility criteria 18 (86) 7 (88) 11 (85) 3 ( 36, 34) Decision rule 13 (62) 4 (50) 9 (69) 19 ( 57, 23) Analyses Eligibility criteria for any type of analysis 17 (81) 8 (100) 9 (69) 31 ( 7, 58) Decision rule for final vs. change from baseline values 8 (38) 3 (38) 5 (38) 0 ( 40, 41) Decision rule for analyses undertaken on multiple samples 15 (71) 7 (88) 8 (62) 26 ( 16, 57) (e.g., ITT vs. per-protocol) Decision rule for unadjusted vs. covariate-adjusted analyses 0 (0) 0 (0) 0 (0) 0 ( 24, 34) Decision rule for period vs. paired analyses in crossover RCTs 10 (48) 4 (50) 6 (46) 4 ( 37, 44) Other decision rule 0 (0) 0 (0) 0 (0) 0 ( 24, 34) Abbreviations: RA, rheumatoid arthritis; OA, osteoarthritis; CI, confidence interval; ITT, intention-to-treat; RCT, randomized controlled trial. a Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification [26]. b Five protocols (24%) were marked not applicable as the index outcome did not have more than one possible measurement scale, so the denominator for this percentage is 16. c Four RA or OA protocols (50%) were marked not applicable as the index outcome did not have more than one possible measurement scale, so the denominator for this percentage is 4.

8 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e Table 4. Number (%) of systematic reviews reporting eligibility criteria and decision rules to select results Total n (%), n [ 44 RA/OA n (%), n [ 23 Depression/anxiety n (%), n [ 21 Risk difference (95% CI), % Type of eligibility criterion and decision rule Total At least one eligibility criterion 44 (100) 23 (100) 21 (100) 0 ( 15, 16) a At least one decision rule 42 (95) 21 (91) 21 (100) 9 ( 27, 8) a Measurement scales Eligibility criteria 33 (75) 16 (70) 17 (81) 11 ( 37, 14) Decision rule 16 (55) b 8 (53) c 8 (57) d 4( 40, 32) Intervention/control groups Eligibility criteria 42 (95) 22 (96) 20 (95) 1 ( 17, 19) a Decision rule 22 (50) 12 (52) 10 (48) 4 ( 25, 34) Time points Eligibility criteria 39 (89) 18 (78) 21 (100) 22 ( 42, 4) a Decision rule 32 (73) 14 (61) 18 (86) 25 ( 50, 0) Analyses Eligibility criteria for any type of analysis 36 (82) 19 (83) 17 (81) 2 ( 22, 26) a Decision rule for final vs. change from baseline values 18 (41) 10 (43) 8 (38) 5 ( 24, 34) Decision rule for analyses undertaken on multiple samples 22 (50) 8 (35) 14 (67) 32 ( 60, 4) (e.g., ITT vs. per-protocol) Decision rule for unadjusted vs. covariate-adjusted analyses 2 (5) 1 (4) 1 (5) 1 ( 19, 17) a Decision rule for period vs. paired analyses in crossover RCTs 14 (32) 7 (30) 7 (33) 3 ( 30, 25) Other decision rule 0 (0) 0 (0) 0 (0) 0 ( 15, 16) a Abbreviations: RA, rheumatoid arthritis; OA, osteoarthritis; CI, confidence interval; ITT, intention-to-treat; RCT, randomized controlled trial. a Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification [26]. b Fifteen reviews (34%) were marked not applicable as the index outcome did not have more than one possible measurement scale, so the denominator for this percentage is 29. c Eight RA or OA reviews (35%) were marked not applicable as the index outcome did not have more than one possible measurement scale, so the denominator for this percentage is 15. d Seven depression/anxiety reviews (33%) were marked not applicable as the index outcome did not have more than one possible measurement scale, so the denominator for this percentage is 14. reports was encountered, but the percentage was less for reviews that had a protocol. A key finding of our study was that there was a mismatch between the types of multiplicity that were commonly encountered in the RCTs (i.e., measurement scales and intervention/control groups) and the decision rules reported in the review protocols. It is not known if reviewers did not anticipate finding multiple scales and intervention/control groups in the trial reports or if they had difficulty predefining decision rules for multiple scales and intervention/control groups. Predefining decision rules for measurement scales and intervention/control groups is complex, requiring consideration of several clinical and methodological issues. A large number of measurement scales for a particular outcome may exist, each with different item content and psychometric properties (e.g., construct validity and interrater reliability) [29]. Systematicreviewersmayalsoneedtoconsiderwhich variants of an intervention (e.g., dosage, duration, method of administration) are most relevant to their review question, in anticipation of finding multi-arm trials [30]. With all the other methods that are recommended for reporting in a review protocol, consideration of decision rules may not be seen as a priority. However, the effort in developing decision rules at the protocol stage can reduce the amount of work required for data extraction and analysis, and more importantly, reduces the risk of data-driven selection of results [1,2]. Our results suggest that core outcome measurement sets for a particular clinical condition may reduce the extent of multiplicity of measurement scales in trial reports and help systematic reviewers use consistent methods to select results from multiple scales. The RA/OA RCTs were less likely to have multiple scales than the depressive/anxiety disorder RCTs. We anticipated this finding because for some outcomes in RA RCTs, only one measurement scale is recommended as part of the core set (e.g., the Health Assessment Questionnaire for disability [31]). Of five index meta-analyses of interventions for OA pain, three used the same hierarchy of pain scales previously developed for use in OA systematic reviews [8]. In contrast, depressive and anxiety disorder RCTs have no such core set [32,33], which may have contributed to the variation in decision rules for depression scales that we encountered. Gaining consensus on which scales should be prioritized for inclusion in meta-analyses of a particular outcome, underpinned by a systematic review of the psychometric properties of the scales, may be a helpful step toward reducing the work required by systematic reviewers to develop decision rules for measurement scales. Compared with a previous study [2], we found higher percentages of review protocols with decision rules for multiple measurement scales (19% vs. 0%), intervention/ control groups (14% vs. 0%), and time points (62% vs. 5%). Two-thirds of our review protocols were published after the 2008 edition of the Cochrane Handbook for

9 1290 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e1291 Systematic Reviews of Interventions, which was the first edition to include guidance on dealing with multiplicity of results [34]. Thus, review protocol authors in our study may have had greater awareness of the potential for multiplicity within RCTs and the value of decision rules. It will be interesting to explore what impact the 2012 Methodological Expectations of Cochrane Intervention Reviews standards, which describes decision rules for measurement scales, intervention/control groups, and time points as highly desirable [4], has on the reporting of decision rules in Cochrane review protocols. None of the reviews without protocols stated whether the eligibility criteria or decision rules to select results which they reported were predefined or developed post hoc. This is not ideal because readers cannot judge the risk of bias due to selective inclusion of results in the review. Development of criteria to assess the risk of selective inclusion bias in systematic reviews may be warranted. For example, it may be reasonable to suspect selective inclusion bias if reviewers reported using a decision rule that prioritizes inclusion of a nonvalidated scale over validated scales. However, in circumstances where it is unclear whether multiplicity was encountered in the RCTs and how effect estimates were selected, it is difficult to judge whether selective inclusion of results occurred without referring to the reports of the included RCTs [17]. Therefore, we encourage systematic reviewers to report the methods they used to select trial effect estimates in instances when multiple were available and to note whether these methods were developed a priori or post hoc. Our study has some limitations. Results of our study are a reflection of what was reported, but this may not reflect practice. In the sample of systematic reviews included in our study, the methods to select results may have been less likely to be reported because at that time, reporting guidelines for systematic reviews did not emphasize the importance of specifying these methods. Furthermore, systematic reviewers may assume that there is already consensus within a specialty about which particular scale or time point is the most important to include in a review and therefore not state this. In this circumstance, we would have overestimated the extent of unanticipated multiplicity of results. Conversely, because we only counted effect estimates that were completely reported in the RCT reports, we may have underestimated the extent of unanticipated multiplicity. Many systematic reviewers may seek unpublished data from trialists (e.g., missing standard deviations and effect estimates), which could increase the pool of effect estimates available for inclusion in a particular meta-analysis. In addition, our analysis of multiplicity of results in RA/OA RCTs compared with depressive/anxiety disorder RCTs may be confounded by differences in the specification of the index meta-analyses. In particular, more depressive/ anxiety disorder meta-analyses than RA/OA metaanalyses were defined by a single time point (e.g., specified as depression at 3 months rather than just depression ) (62% vs. 48%, respectively). This difference may partly explain why fewer depressive/anxiety disorder RCTs than RA/OA RCTs had multiple time points available for inclusion. Other potential limitations are that only one author screened the systematic reviews for inclusion and double data extraction was only undertaken for a third of the reviews. Single screening of systematic reviews may have led to errors in applying review eligibility criteria; however, we do not believe single screening is likely to have biased the results of our study. At the screening stage, included RCTs had not been retrieved so review inclusion/exclusion decisions were unable to be influenced by knowledge of the extent of multiplicity in RCTs. Furthermore, based on the results of the independent double data extraction, we believe our estimates of multiplicity are unlikely to have been notably affected by data extraction errors. Results of the double data extraction identified no errors in the classification of text as an eligibility criterion or decision rule, and the number of effect estimates extracted were consistent except for three occasions where one author incorrectly extracted more effect estimates from a trial report than was necessary. There are several factors that limit the generalizability of our results. First, Cochrane reviews comprised roughly half of our sample, yet comprise a minority of all published systematic reviews [12]. We oversampled Cochrane reviews because we wished to investigate whether review protocols affected our estimates, and when we commenced our study, only a minority of non-cochrane reviews had a publicly accessible protocol [13]. Second, we only searched MEDLINE to identify non-cochrane reviews and restricted inclusion to English language publications, so our estimates of the frequency of methods to select results may not generalize to non-medlineeindexed reviews and reviews published in other languages. Finally, we only focused on meta-analyses of continuous outcomes so were unable to estimate the frequency of types of multiplicity of results unique to dichotomous outcomes (e.g., how often multiple effect estimates were available from measurement scales that have been dichotomized using different cut points). In conclusion, as it is likely that systematic reviewers will encounter some type of multiplicity of results in the included trials, we recommend that consideration should always be given to predefining methods to select results to include in meta-analyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be of most value. At the review stage, clear reporting of whether multiplicity of results was encountered in trial reports, the methods used to select results to include in meta-analyses, and whether these methods were developed a priori or post hoc would be beneficial. Reporting such information can aid in a reader s assessment of the potential risk of bias due to selective inclusion of results in a review.

10 M.J. Page et al. / Journal of Clinical Epidemiology 68 (2015) 1282e Acknowledgments This work was conducted as part of a PhD undertaken by M.J.P., which is funded by an Australian Postgraduate Award administered through Monash University, Australia. J.E.M is supported by an NHMRC Australian Public Health Fellowship ( ). Supplementary data Supplementary data related to this article can be found at References [1] Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol 2008;61:857e65. [2] Tendal B, N uesch E, Higgins JP, J uni P, Gøtzsche PC. Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study. BMJ 2011;343:d4829. [3] Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol 2013;66:524e37. [4] Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention Reviews. Version : Available at cochrane.org/mecir. Accessed October 9, [5] Institute of Medicine (IOM). Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; [6] AHRQ Publication No. 10(14)-EHC063-EF. Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2014:Chapters available at Accessed October 9, 2014 [7] Reichenbach S, Sterchi R, Scherer M, Trelle S, B urgi E, B urgi U, et al. Meta-analysis: chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 2007;146:580e90. [8] J uni P, Reichenbach S, Dieppe P. Osteoarthritis: rational approach to treating the individual. Best Pract Res Clin Rheumatol 2006;20(4): 721e40. [9] Hasselblad V. Meta-analysis of multitreatment studies. Med Decis Making 1998;18:37e43. PubMed PMID: eng. [10] Higgins JPT, Deeks JJ. Chapter 7: selecting studies and collecting data. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version [updated March 2011]. The Cochrane Collaboration. Available at Accessed October 9, [11] Tendal B, Higgins JPT, J uni P, Hrobjartsson A, Trelle S, N uesch E, et al. Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. BMJ 2009; 339:b3128. [12] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4:e78. [13] Turner L, Galipeau J, Garritty C, Manheimer E, Wieland LS, Yazdi F, et al. An evaluation of epidemiological and reporting characteristics of complementary and alternative medicine (CAM) systematic reviews (SRs). PLoS One 2013;8:e [14] Boers M, Kirwan JR, Wells G, Beaton D, Gossec L, d Agostino M-A, et al. Developing core outcome measurement sets for clinical trials: OMERACT Filter 2.0. J Clin Epidemiol 2014;67:745e53. [15] Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L. OMER- ACT: an international initiative to improve outcome measurement in rheumatology. Trials 2007;8(1):38. [16] Altman R, Brandt K, Hochberg M, Moskowitz R, Bellamy N, Bloch DA, et al. Design and conduct of clinical trials in patients with osteoarthritis: recommendations from a task force of the osteoarthritis research society: results from a workshop. Osteoarthritis and Cartilage 1996;4:217e43. [17] Page MJ, McKenzie JE, Green SE, Forbes AB. An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol. Syst Rev 2013;2:21. [18] Kirkham J, Boers M, Tugwell P, Clarke M, Williamson P. Outcome measures in rheumatoid arthritis randomised trials over the last 50 years. Trials 2013;14:324. [19] CMSG Search Strategies. Availabe at o/cochrane/clabout/articles/muskel/frame.html. Accessed October 9, [20] CCDAN Search Strategies. Availabe at com/o/cochrane/clabout/articles/depressn/frame.html. Accessed October 9, [21] Deeks JJ, Higgins JPT, Altman DG. Chapter 9: analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version [updated March 2011]. The Cochrane Collaboration. Available at Accessed October 9, [22] Fu R, Vandermeer BW, Shamliyan T, O Neil ME, Yazdi F, Fox SH, et al. Chapter 14: handling continuous outcomes in quantitative synthesis. AHRQ Publication No. 10(14)-EHC063-EF. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; Chapters available at Accessed October 9, [23] Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457e65. [24] Clopper C, Pearson E. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934;26:404e13. [25] StataCorp. Stata statistical software: release 12. College Station, TX: StataCorp LP; [26] Newcombe RG. Interval estimation for the difference between independent proportions: comparison of eleven methods. Stat Med 1998; 17:873e90. [27] Scherer R. PropCIs: various confidence interval methods for proportions. R package version : Available at Accessed October 9, [28] Team RC. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; [29] McDowell I. Measuring health: a guide to rating scales and questionnaires. 3rd ed. New York: Oxford University Press; [30] Glasziou PP, Chalmers I, Green S, Michie S. Intervention synthesis: a missing link between a systematic review and practical treatment(s). PLos Med 2014;11:e [31] Fries JF, Spitz P, Kraines RG, Holman HR. Measurement of patient outcome in arthritis. Arthritis Rheum 1980;23:137e45. [32] Demyttenaere K, De Fruyt J. Getting what you ask for: on the selectivity of depression rating scales. Psychother Psychosom 2003;72(2): 61e70. [33] Balon R. Rating scales for anxiety/anxiety disorders. Curr Psychiatr Rep 2007;9:271e7. [34] Higgins JPT, Deeks JJ, Altman DG. Chapter 16: special topics in statistics. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. West Sussex, England: John Wiley & Sons Ltd; 2008.