Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

Transcription

1 Annals of Oncology 28: , 2017 doi: /annonc/mdw691 Published online 30 December 2016 SPECIAL ARTICLE Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews D. Herrmann, P. Sinnett, J. Holmes, S. Khan, C. Koller & M. Vassar* Oklahoma State University Center for Health Sciences, Tulsa, USA *Correspondence to: Dr Matt Vassar, Oklahoma State University Center for Health Sciences, 1111 West 17th Street, Tulsa, OK 74107, USA. Tel: þ ; Background: Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews. Design: We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and Systematic reviews were screened for eligibility and qualifying reviews (n ¼ 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger s regression, trim-and-fill, and selection models. Results: Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80%, 41/51) followed by Egger s regression (59%, 30/51) and Begg s test (43%, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed. Conclusion: Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias. Key words: publication bias, systematic reviews, oncology, Egger s, funnel plot, meta-analysis Introduction Research indicates that systematic reviews in oncology may have serious methodological flaws leading to a high risk of bias, especially among reviews published in peer reviewed journals [1]. One form of bias, known as publication bias, occurs to the extent that the sample of studies included in a systematic review is not representative of the population of completed studies on a particular relation of interest [2]. Publication bias may occur when investigators, peer reviewers, or journal editors do not submit or accept a manuscript for publication based on the strength or direction of their findings [3]. Such bias compromises the validity of systematic review outcomes and may lead to inaccurate clinical decisions [4]. For example, Simes examined meta-analyses comparing combination therapy with alkylating agents as treatment of advanced ovarian cancer and reported that when only registered trials were included, patient survival did not differ between the two treatments [5]. In contrast, when only published trials were examined, a statistically significant benefit was found for the combination therapy. VC The Author Published by Oxford University Press on behalf of the European Society for Medical Oncology. All rights reserved. For Permissions, please journals.permissions@oup.com.

2 Special article Cancer trials provide unique challenges due to the high number of studies that never reach publication. In a review of adult cancer clinical trials from ClinicalTrials.gov (n ¼ 7776 trials), authors found a 7-year cumulative incidence of failure to complete of 20%. These studies were comprised of patients and contributed little to the scientific knowledge base. Poor accrual was the most common reason for failure to complete, followed by logistics (e.g. cancellation by the trial sponsor, or inadequate budget), and unacceptable toxicity or poor interim results [6]. A second challenge is the scientific impact of positive results of cancer clinical trials. Recent evidence indicates that trials with positive results are published in journals with higher impact factors and cited two times as often as cancer trials with negative results [7]. Given the lower probability of cancer trials reaching publication as well as the scientific importance associated with positive outcomes, publication bias may be an overlooked contributor toward poor study completion and publication failure. In this study, we examine systematic reviews published in oncology journals to determine the extent to which PB is discussed, planned for, and evaluated. We also analyze a subset of systematic reviews in our sample that failed to report such evaluations to examine the presence (or absence) and degree of PB within these reviews. Methods We conducted a study of systematic reviews to evaluate the use of methods to address publication bias practices. This study did not meet the regulatory definition of human subject research as defined in 45 CFR (d) and (f) of the Department of Health and Human Services Code of Federal Regulations and, therefore, was not subject to Institutional Review Board oversight. We applied relevant PRISMA guidelines for systematic reviews and SAMPL guidelines for reporting descriptive statistics. Data from this study are publically available on figshare ( Search criteria A PubMed search of six oncology journals from 2007 to 2015 was conducted using the following search string: (((((( Journal of clinical oncology: official journal of the American Society of Clinical Oncology [Journal] OR Nature reviews. Cancer [Journal]) OR Cancer research [Journal]) OR The Lancet. Oncology [Journal]) OR Clinical cancer research: an official journal of the American Association for Cancer Research [Journal]) OR Cancer cell [Journal]) AND ( 2007/01/01 [PDAT]: 2015/12/31 [PDAT]) AND humans [MeSH Terms]) AND (meta-analysis [Title/Abstract] OR systematic review [Title/Abstract]). This search strategy was adapted from a previously established method that is sensitive to identifying SRs and meta-analyses [8]. Journals were selected based on 2014 h5-index from Google Scholar metrics: oncology sub-category. Searches were conducted on 18 May and 26 May Review selection and data extraction We used Covidence (covidence.org), a systematic review platform, to initially screen articles based on the title and abstract. To qualify as a SR, articles had to summarize evidence across multiple studies and provide information on the search strategy, such as search terms, databases, and inclusion/exclusion criteria. Meta-analyses were classified as quantitative syntheses of results across multiple studies [9]. All articles that did not meet the requirements for classification as a SR or meta-analysis were excluded. Two authors (DH and MV) independently categorized all articles for inclusion, and a follow up meeting was held to discuss differences Annals of Oncology in screening. All disagreements were settled by consensus. Full-text versions of included articles were obtained via EndNote. To ensure the accuracy of the data abstraction process, an abstraction manual was developed and piloted on a small subset of reviews. The authors conducted a training session to familiarize themselves with the process, and subsets of reviews were jointly coded as a group. After training, authors (DH and JH) were given three new reviews to code independently. These data were analyzed for inter-rater agreement by calculating Cohen s kappa statistic. As inter-rater agreement was high (k ¼ 0.86; agreement ¼ 91%), each author was assigned an equal subset of articles for data abstraction. Data elements for this study included (i) the article title, (ii) authors, (iii) year published, (iv) journal name, (v) whether PB was discussed, (vi) whether PB was assessed, (vii) what method was used for assessment, (viii) whether a funnel plot was published as a figure in the SR, (ix) whether PB, if evaluated, was present, (x) number of studies, (xi) which guidelines were followed, (xii) whether foreign languages were searched, (xiii) whether a hand search of references was carried out, (xiv) whether gray literature was searched, and (xv) whether clinical trials registries were searched. After the initial abstraction process, validation checks were conducted such that each data element was verified by the other author. After these checks were carried out, authors met to discuss disagreements and settle them by consensus. Analysis of descriptive data was conducted using STATA Assessment of unreported publication bias Additional analyses were carried out to evaluate PB among SRs failing to report such an assessment. PB was assessed only from SRs that met the following criteria: PB was not originally assessed in the SR, the SR included a meta-analysis, data from greater than 10 primary studies were reported, data from primary studies were listed in the SR in a usable form, and the SR did not include evaluation of diagnostic accuracy studies. A minimum of 10 studies was selected because with fewer studies the power of the statistical tests is too low to distinguish chance from asymmetry [10]. If SRs contained multiple meta-analyses, or if the SR split meta-analysis into several groups, they were individually evaluated and must have included more than ten studies per group. Diagnostic accuracy studies were excluded since PB methods using traditional funnel-plots and regression models are not recommended for diagnostic accuracy outcomes [11]. We evaluated for publication bias using three methods: Egger s regression, Duval and Tweedie s trim-and-fill, and Copas selection models. We used Egger s regression and trim-and-fill because they are among the most commonly reported statistical methods for detection of PB in SRs [12, 13]. We used trim-and-fill for re-analysis, because it is an established method used by systematic reviewers in oncology and allows for the estimation of an adjusted summary effect having removed the influence of publication bias. Trim-and-fill adjusts for the potential effect that non-published studies might have had on the measured outcome [14, 15]. Egger s test is a regression-based approach in which the standard normal deviate is regressed against its precision. For both analyses, the point estimates from the data were calculated using Comprehensive meta-analysis (CMA) version 3 and then compared with the point estimates presented in the original meta-analysis to ensure accuracy of our entered data. In some cases, confidence intervals had to be slightly modified to achieve symmetry of the interval. This was carried out in four metaanalyses due to rounding differences. Point estimates were compared with published reports to ensure accuracy. A fixed or random effects model was selected based on the model reported in the original SR. If the choice of model was not provided, we analyzed the results using both methods and chose the one that produced an aggregate effect size that most closely matched the SR s point estimate. The direction of the bias (left or right), number of studies trimmed, point estimate, confidence intervals, adjusted point estimate and adjusted confidence intervals were all recorded from the trim-and-fill method for each meta-analysis. The t-value and intercept were recorded from the Egger s regression calculations, and P-values were considered significant at P < Herrmann et al. Volume 28 Issue

3 Annals of Oncology The third method we used to evaluate for the likelihood of PB was the Copas selection model [16]. Selection models correct for bias using a weight function to represent the selection process. There are two aspects to selection models. The effect size model (sometimes called data models) specifies what the distribution of effect sizes would be in the absence of selection. The selection model specifies how the effect size model is altered by the selection process. A correlation parameter between these two models indicates the extent of publication bias, such that the higher the correlation, the greater the chance that only more extreme treatment effects are selected for publication [17 19]. For analysis, we used the R-package metasens. Results The PubMed search resulted in 337 articles from four journals. After screening titles and abstracts, 79 were excluded because they were not SRs or meta-analyses. An additional 76 articles were excluded after full text screening. Two articles could not be retrieved after multiple attempts. A total of 182 systematic reviews were included for data abstraction (Figure 1). We found that 40% of SRs (73/182) discussed PB and 28% of SRs (51/182) included an assessment of PB. Across journals, Clinical Cancer Research had the largest percentage of SRs that discussed (59%, 10/17) and evaluated (41%, 7/17) for PB. The Journal of Clinical Oncology (39%, 41/106 discussed and 27%, 29/106 evaluated) and The Lancet Oncology (38%, 22/58 discussed and 26%, 15/58 evaluated) both had similar rates. Cancer Research contained only one systematic review. Of the SRs that did assess PB, most used multiple methods (69%, 35/51). The most common assessments utilized are shown in Figure 2. Of the 51 studies that carried out an assessment of PB, 40 found no evidence of PB, 9 found evidence of PB, and 2 had inconclusive results. Of the non-evaluating articles, 3 reported results of publication analysis but never mentioned performing an evaluation in their methods. From the subset of 51 studies, 10 included a funnel plot as a figure in the article. Adherence to PRISMA guidelines was reported in 38 SRs, MOOSE in 10 reviews, and QUOROM in 10. The percent of SRs that actually followed the guidelines when assessing PB was 45% (17/38) for PRISMA, 70% (7/10) for MOOSE, and 50% (5/10) for QUOROM. We looked for evidence of mitigating PB by examining search strategies of SR authors. Over half (52%, 94/182) included a hand search of the references of included articles. Conference abstracts were the most common form of gray literature searched (27%, 49/ 182) followed by clinical trials registries (8%, 15/182). The most common registry searched was clinicaltrials.gov (67%, 10/15) followed by WHO International Trials Registry (13%, 2/15). Thirty percent (55/182) of articles expanded their search to including articles in foreign languages, 42% (77/182) did not, and 27% (50/ 182) did not mention language in the search strategy. Twelve SRs containing 31 meta-analyses were selected to assess PB (Table 1). These SRs did not report a formal PB evaluation, so we re-analyzed the data from each SR to determine whether PB might be present. According to results from the trim-and-fill analysis, the number of studies trimmed ranged from 0 to 7. Of these, nine meta-analyses imputed one additional study to the funnel plot, and two meta-analyses imputed two studies. The 16 remaining meta-analyses imputed 3 or more studies to the funnel plot. The Egger s test found evidence for publication bias in three studies. Selection models found evidence of publication bias in one meta-analysis. Chow (2007) was the only meta-analysis where publication bias was indicated by all three methods. Discussion Special article Our results show PB assessment among SRs in oncology journals could be greatly improved. Despite being recognized as an important evaluation, 72% (131/182) of the reviews included in our study did not include an assessment. The 28% (51/182) of reviews that did assess PB was much lower than the 69% (80/116) found by Onishi and Furukawa [9] in high impact factor general medical journals. Our results more closely match Moher et al. [14]thatreportedonly 23% (68/294) of systematic reviews comprising their sample included such an evaluation. Parekh-Burke et al. [15] reported similar findings [30% (91/300)]. Our study, in accordance with other literature, points to the same conclusion: authors of SRs should perform an evaluation for PB, if appropriate, to strengthen the quality of their reviews. In cases where PB evaluations are not possible, review authors should acknowledge this in the published report and cite relevant references for not performing these evaluations. We examined the degree to which reported adherence to SR reporting guidelines was associated with an increased likelihood of PB evaluations. Our results call into question actual adherence practices of such reporting guidelines. For example, although PRISMA item 15 calls for indicating any evaluation of risk of bias that may affect the cumulative evidence, over half of the articles that reported use of the guidelines (55%, 21/38) did not conduct this evaluation. Discussion of and evaluating for PB was consistent between oncology journals; however, very few SRs or meta-analyses were found in Cancer Cell, Cancer Research, ornature Reviews Cancer. The search process is an important component of systematic reviews, and comprehensive strategies are needed to ensure that the final sample of primary studies accurately reflects the true nature of the field. Searches that report attempts to locate unpublished data, so as not to contribute to misrepresented summary effects from only published reports, should be preferred. In our study, only 30% (55/182) searched in foreign languages, 52% (94/182) hand searched references, 32% (58/182) searched gray literature, and 8% (15/182) searched trials registries. Recent evidence suggests that searching clinical trials registries may yield additional data for inclusion in systematic reviews [20, 21]. Therefore, the authors encourage systematic reviewers to incorporate trial registries into systematic review searches, although the limitations of trial registries are not fully known. Of the studies that conducted a PB assessment, only 19% (10/ 51) found PB to exist in their study. This is similar to Onishi and Furakawa s [9] finding (19%, 7/36) but consistent with Atakpo and Vassar s [22] study of publication bias in dermatology (15.3%). When we evaluated for the likelihood of PB across nonreporting SRs we found notable variability depending on the method used. Using Egger s test, we found that approximately one-fifth of the meta-analyses we analyzed showed evidence for PB (21%, 9/42), which is comparable to Onishi and Furakawa [9], in which 7 of the 36 contained PB. The trim-and-fill method evaluations in many cases imputed one or two studies to the funnel plot. It is likely in these cases that the one or two imputed studies could be attributed to chance, and these results should be Volume 28 Issue doi: /annonc/mdw

4 Special article Annals of Oncology Identification Records identified through database searching (n = 337) Screenin Records screened (n = 337) Non-systematic reviews and meta-analyses excluded (n = 79) Eligibility Full-text articles assessed for eligibility (n =258) Full-text articles excluded: Genetic studies (40) Individual patient data (20) Primary study (6) Genomic study (3) Narrative (3) Article could not be retrieved (2) Letter to the editor (1) Histologic (1) (n = 76) Inclusion Studies included in qualitative analysis (n = 182) Full-text articles excluded for quantitative analysis: Systematic reviews without Meta-analyses (56) Meta-analyses with fewer than 10 studies (41) Diagnostic accuracy studies (7) Provided evaluation of PB (51) Data not compatible with CMA or R re-analysis (15) (n = 170) Studies included in auantitative analysis (n=12) Figure 1. PRISMA flow diagram. interpreted in this context. The selection model found evidence for PB in a single meta-analysis. In a study comparing Copas selection models to trim-and-fill, investigators found that trimand-fill is more likely to find missing studies than Copas selection models, and our findings confirm this [23]. Therefore, the method selected by investigators may have important implications regarding the likely extend of PB. As previously indicated, funnel plot asymmetry methods were most commonly used by systematic reviewers in oncology. It should be noted that asymmetry in the funnel plot may be attributed to other factors instead of publication bias such as small study bias. In drug trials, for example, large magnitude effects may be estimated based on small samples conducted early and made up of high-risk patients likely to receive the greatest benefit from the drug [2, 24]. Asymmetry may also be caused by true heterogeneity in the underlying treatment effects, selective outcome reporting, or chance [25]. Adding contour lines to the funnel plot that correspond to common statistical significance thresholds (such as P < 0.05 and P < 0.10) may help distinguish publication bias from other causes of funnel plot asymmetry [2]. Duval and Tweedie s trim and fill method was also frequently reported in our systematic reviews. This method adjusts the summary effect estimate after removing (trimming) small studies until the funnel plot is symmetrical. The funnel plot is then replicated with the trimmed studies replaced plus their missing counterparts (filling) [20]. This method is based on the assumption 934 Herrmann et al. Volume 28 Issue

5 Annals of Oncology Special article Number of studies Funnel plots Egger's test Begg and mazumdar's test Trim-and-fill Method of assessment Fail-safe-N Gleser and olkin's method Figure 2. Methods of assessment used to detect publication bias. Table 1. Re-analysis of publication bias among non-reporting meta-analyses Article Figure/ table Fixed/ random Trim and fill Eggers Copas selection Studies trimmed Observed point estimate (CI) Adjusted point estimate (CI) Egger s intercept t-value RR (CI) Number of unpublished studies Flaherty (2014) a Table 2 A Fixed (0.65, 0.75) 0.69 (0.65, 0.74) (-.031, -0.07) 0 Table 2 B Fixed (0.76, 0.89) 0.88 (0.81, 0.87) ( 0.23, 0.06) 0 Niraula (2012) Figure 2A Fixed (1.18, 2.49) 2.46 (1.75, 3.46) (0.63, 0.85) 0 Figure 2B Fixed (1.77, 2.51) 1.58 (1.09, 2.28) (0.67, 1.21) 0 Figure 3 Fixed (1.17, 1.52) 1.32 (1.15, 1.50) (0.89, 2.62) 0 Horgan (2012) Figure 2 Random (0.46, 1.07) 0.91 (0.61, 1.36) (0.91, 2.09) 0 Algra (2012) Figure 3A Fixed (0.75, 0.80) 0.88 (0.67, 1.17) (0.72, 1.08) 0 Figure 3B Fixed (0.65, 0.67) 0.55 (0.54, 0.56) * 0.75 (0.65, 0.86) 0 Figure 3C Fixed (0.70, 0.90) 0.79 (0.70, 0.90) (0.76, 1.11) 0 Figure 3D Fixed (0.74, 0.92) 0.82 (0.74, 0.91) (0.70, 0.88) 0 Bourhis (2011) Figure 2 Fixed (0.76, 1.29) 0.99 (0.76, 1.29) (0.91, 1.07) 0 Figure 4 Fixed (0.77, 1.34) 0.98 (0.75, 1.29) (0.94, 1.07) 0 Jamieson (2011) a Table 2A Fixed (1.56, 1.97) 1.69 (1.51, 1.89) (0.12, 0.35) 0 Table 2B Fixed (1.44, 1.81) 1.54 (1.38, 1.72) (0.14, 0.36) 0 Table 2C Fixed (1.14, 1.50) 1.29 (1.13, 1.47) ( 0.05, 0.33) 0 Cocks (2011) Table 2 Random (0.26, 0.43) 0.31 (0.24, 0.39) (2.02, 2.93) 0 Vale (2008) Figure 1 Fixed (0.61, 0.92) 0.72 (0.59, 0.87) (0.76, 0.96) 0 Figure 3 Fixed (0.65, 0.89) 0.74 (0.64, 0.86) (0.77, 0.93) 0 Fairchild (2008) a Figure 5 Random (0.73, 0.93) 0.86 (0.76, 0.98) (0.75, 0.96) 0 Figure 6 Random (0.63, 1.07) 0.80 (0.62, 1.04) (0.90, 1.55) 0 Ives (2007) Figure 2 Fixed (0.59, 1.11) 0.81 (0.59, 1.11) (0.89, 1.68) 0 Figure 3 Fixed (0.24, 0.47) 0.34 (0.24, 0.47) (1.75, 5.33) 0 Figure 4 Fixed (0.46, 0.80) 0.61 (0.46, 0.80) (1.24, 2.18) 0 Trivella (2007) a Figure 2 Random (0.98, 1.09) 1.05 (0.99, 1.11) ( 0.08, 0.10) 0 Figure 3 Random (0.76, 1.57) 1.10 (0.76, 1.57) ( 0.18, 0.20) 0 Chow (2007) Figure 1 Random (0.95, 1.03) 1.01 (0.97, 1.05) * 0.99 (0.95, 1.03) 1 Figure 2 Random (0.88, 1.06) 1.00 (0.91, 1.09) (0.88, 1.06) 0 Figure 3 Random (1.76, 3.53) 2.28 (1.65, 3.15) (1.78, 3.46) 0 Figure 7 Random (0.92, 0.99) 0.97 (0.94, 1.01) * 0.96 (0.93, 1.00) 0 Figure 8 Random (0.85, 1.02) 0.96 (0.88, 1.04) (0.85, 1.02) 0 Figure 9 Random (1.74, 3.45) 2.29 (1.67, 3.14) (1.75, 3.35) 0 a Confidence intervals slightly adjusted to run in CMA. *P-value (two-tailed) <0.10. Volume 28 Issue doi: /annonc/mdw

6 Special article that the most negative studies are missing, and this assumption is often not realistic [26]. While this method adjusts meta-analytic results for publication bias based on imputing fictional studies, we do not recommend relying on such adjustments for final decision-making. Rather, any adjustment methods should be used as a form of sensitivity analysis to evaluate the potential effect of missing studies on meta-analytic results [27]. Owing to the limitations of the PB methods commonly reported in the oncology literature, we recommend systematic reviewers consider more robust methods to evaluate for the likelihood of publication bias. Selection models are a class of approaches that model the selection process by which studies are selected for publication. These models allow investigators to evaluate the possible effect of missing studies had they been included in the meta-analysis. As with the trim and fill method, selection models are best used as a sensitivity analysis tool to examine the potential effects of unpublished studies on summary effects. One study compared the Copas selection model with the trim-and-fill method on 157 meta-analyses with binary outcomes [28]. Findings from this study indicated standard errors and P- values were larger for the trim-and-fill method while point estimates were similar. The authors ultimately recommended the Copas selection model over the trim-and-fill method. Perhaps the greatest limitation of selection models, at present, is the lack of user-friendly software which complicates their use in practice (see Jin, Zhou, and He). An R-package, metasens (formerly the copas R-package), has been developed to apply the Copas selection model to meta-analysis. Systematic reviews of diagnostic test accuracy were excluded from this study, but publication bias methods for these types of reviews do warrant attention. Funnel plot methods (such as Egger s, Harbord s, Begg s, and Peter s tests) are not suitable for such studies since diagnostic odds ratios often have magnitudes much higher than clinical odds ratios and may lead to incorrect indications of publication bias by these tests too often [2]. Guidance on methods to evaluate for publication bias in diagnostic accuracy reviews is limited and conflicting. The Cochrane Handbook for Diagnostic Test Accuracy Reviews discusses the Deeks test as more suitable to diagnostic accuracy reviews but cautions that this test has low power to detect small study effects when there is heterogeneity in the diagnostic odds ratio and should therefore be interpreted carefully [29]. Bürkner and Doebler conducted a simulation study of PB methods and concluded that common tests, such as Egger s, Begg s, and Macaskill s had low power when using a random effects model [30]. The Deeks test also suffered from low power under conditions of high between-study heterogeneity, and none of these tests were recommended for use in diagnostic test reviews. Instead, the authors recommended applying the trim and fill method combined with the log diagnostic odds ratio to detect publication bias. A recent meta-epidemiological study examined diagnostic test accuracy reviews to determine the statistical methods reported when evaluating publication bias [31]. This study reported that Egger s, Deeks, and Begg s tests were most commonly used, and a re-analysis of published data from 92 meta-analyses indicated that concordance between tests was moderate. Ultimately, these authors call for using the Deeks test for diagnostic test accuracy reviews, but the basis for this recommendation is unclear. Further research is needed to understand the contributing factors of publication bias in diagnostic accuracy studies and ways to assess its likelihood. We note some limitations to the current study. We selected systematic reviews published in highly ranked journals, so results from this study may not be generalizable to other oncology journals. Furthermore, there are likely additional oncology journals that routinely publish systematic reviews, and these were not included in our analysis. It is possible that other oncology journals have different journal policies regarding the conduct and reporting of systematic reviews that could increase the rates with which publication bias is handled. In the current study, we found that there exists an underuse of PB assessments by oncology systematic reviewers. When PB was assessed, the most common method was a funnel plot complemented with Egger s regression or Begg s test. Using these assessments, PB was a routine finding in the SRs. We also found other areas of weakness such as adhering to reported guidelines and searching the gray literature, including trials registries. These results should inform future SR methodologies and create awareness that PB should be routinely reported among SRs in oncology. Funding None declared. Disclosure The authors have declared no conflicts of interest. References Annals of Oncology 1. Lundh AS, Knijnenburg L, Jorgensen AW et al. Quality of systematic reviews in pediatric oncology a systematic review. Cancer Treat Rev 2009; 35(8): Kepes S, Banks G, McDaniel M et al. Publication bias in the organizational sciences. Organ Res Methods 2012; 15(4): Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990; 263: Dwan K, Gamble C, Williamson P, Kirkham J. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS ONE 2013; 8(7): e Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986; 4: Stensland K, McBride R, Latif A et al. Adult clinical trials that fail to complete: an epidemic? J Natl Cancer Inst 2014; 106(9): dju Unger J, Barrlow W, Ramsey S et al. The scientific impact of positive and negative phase 3 cancer clinical trials. JAMA Oncol 2016 [Epub ahead of print], doi: /jamaoncol Ramsey S, Scroggins J. Practicing on the tip of an information iceberg? Evidence of underpublication of registered clinical trials in oncology. Oncologist 2008; 13(9): Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol 2014; 67: Montori VM, Wilczynski NL, Morgan D et al. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005; 330: Higgins JPT, Green S, Addressing Reporting Biases. In: Cochrane Handbook for Systematic Reviews of Interventions. West Sussex: Wiley 2008; Egger M, Davey SG, Schneider M et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: Herrmann et al. Volume 28 Issue

7 Annals of Oncology 13. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56: Moher D, Tetzlaff J, Tricco AC et al. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007; 4: e Parekh-Bhurke S, Kwok CS, Pang C et al. Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol 2011; 64: Copas J, Shi J. A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res 2001; 10(4): Rothstein H, Sutton AJ, Borenstein M, Publication Bias in Meta-analysis: Prevention, Assessment and Adjustments. Chichester, England: Wiley Guido S, Carpenter J, Rücker G. Empirical evaluation suggests Copas selection model preferable to trim-and-fill method for selection bias in meta-analysis. JAMA Oncol 2010; 63(3): Guido S, Carpenter J, Rücker G. Meta-analysis with R. Print. Switzerland: Springer International Publishing. 20. Sinnett PM, Carr B, Cook G et al. Systematic reviewers in clinical neurology do not routinely search clinical trials registries. PLoS ONE 2015; 10(7): e Bibens ME, Chong AB, Vassar M. Utilization of clinical trials registries in obstetrics and gynecology systematic reviews. Obstet Gynecol 2016; 127(2): Atakpo P, Vassar M. Publication bias in dermatology systematic reviews and meta-analyses. J Dermatol Sci 2016; 82(2): Special article 23. Carpenter J, Rucker G. Schwarzer Assessing the sensitivity of metaanalysis to selection bias: a multiple imputation approach. Biometrics 2011; 67(3): Guyatt GH, Oxman AD, Montori V et al. GRADE guidelines: 5. Rating the quality of evidence publication bias. J Clin Epidemiol 2011; 64: Mavridis D, Salanti G. Exploring and accounting for publication bias in mental health: a brief overview of methods. Evid Based Ment Health 2014; 17: Rothstein HR, Sutton AJ, Borenstein M, Publication bias in meta-analysis. In Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. John Wiley 2006; Duval S, The trim and fill method. In Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. John Wiley 2006; Schwarzer G, Carpenter J, Rücker G. Empirical evaluation suggests Copas selection model preferable to trim-and-fill method for selection bias in meta-analysis. J Clin Epidemiol 2010; 63: Handbook for DTA Reviews j Cochrane Screening and Diagnostic Tests. 2016; (26 October 2016, date last accessed). 30. Bürkner P-C, Doebler P. Testing for publication bias in diagnostic metaanalysis: a simulation study. Stat Med 2014; 33: Van Enst WA, Ochodo E, Scholten RJPM et al. Investigation of publication bias in meta-analyses of diagnostic test accuracy: a metaepidemiological study. BMC Med Res Methodol 2014; 14. Volume 28 Issue doi: /annonc/mdw