Selective reporting, quality of reporting and statistical significance chasing in prognostic marker studies

Transcription

1 Selective reporting, quality of reporting and statistical significance chasing in prognostic marker studies Panayiotis A. Kyzas 1,2 1: Research Fellow, Department of Hygiene and Epidemiology, University of Ioannina, Medical School, Greece. 2: Specialist trainee, Oral and Maxillofacial Surgery, North Manchester General Hospital, Manchester, UK.

2 Prognostic markers studies Frequent problems Variation in study design Variation in inclusion criteria Variation in methods of analysis Variation in adjusting for covariates Variation in measurement techniques Most of them are retrospective. BMJ Jul 28;323(7306):224 8.

3 Prognostic markers studies Possible threats Selective reporting Poor reporting quality Statistical significance

4 Prognostic markers studies Selective reporting eta analysis of the prognostic significance of TP53 status in HNSCC esign year mortality as main outcome (RRs) EDLINE and EMBASE literature search

5 Prognostic markers studies Selective reporting tandardizations mmunohistochemistry (IHC) vs. PCR HC cut off 10 % 4 months mortality, 2 x 2 tables

6 Prognostic markers studies Selective reporting rognostic markers meta analyses reviously published meta analyses ssues of retrieval and standardized information: Are they addressed?

7 Prognostic markers studies Selective reporting vailable Data

8 Prognostic markers studies Selective reporting eta analysis results

9 Prognostic markers studies Selective reporting rognostic markers meta analyses 8 published meta analyses, on 37 prognostic markers 8 markers (76%): Statistically significant 3 meta analyses (72%) did not use standardized definitions ( )

10 Prognostic markers studies Reported Quality Quality of reporting of prognostic marker studies Reported quality: Association with the magnitude of the estimated prognostic effect?

11 Prognostic markers studies Reported Quality Prognostic marker meta analyses identified in MEDLINE / EMBASE Primary studies retrieved Estimates of the prognostic effect recorded

12 Prognostic markers studies Reported Quality Studies evaluated for 8 quality measures (Study design assay method of REMARK) Blinding Prospective vs. retrospective design Power calculations Outcomes definitions Time of enrollment Reporting of variables Assay description Assay reference J Natl Cancer Inst. 2005;97(16):

13 Prognostic markers studies Reported Quality Ratios of relative risks (RRRs) Poor quality vs. Good quality (i.e. no blinding vs. blinding) Summary ratios of relative risks

14 Prognostic markers studies Reported Quality Power calculations: 3 studies (0.9%) Blinded: 129 studies(39.0%) Prospective: 73 studies (21.5%) Time of enrollment: 232 studies(70.0%) Candidate variables: 234 studies (70.7%) Outcomes definition: 254 studies (76.7%) Assay description: 317 studies (95.8%) Assay reference: 177 studies (53.5%)

15 Prognostic markers studies Reported Quality Summary ratios of relative risks of poor versus good quality studies : 0.95 to 1.26 None was statistically significant.

16 Prognostic markers studies Reported Quality

17 Prognostic markers studies Reported Quality

18 Prognostic markers studies Reported Quality

19 Prognostic markers studies Statistical Significance Given the previously described biases, an excess of statistically significant findings is expected in the prognostic literature.

20 Prognostic markers studies Statistical Significance BUT, what is the extend of this excess???

21 Prognostic markers studies Statistical Significance Search strategy and eligibility criteria for cancer prognostic marker studies Database 1: Studies included in prognostic marker metaanalyses (J Natl Cancer Inst Feb 7;99(3):236-43) Database 2: Articles on cancer prognostic markers published in 2005 PubMed, readily available high specificity prognosis search algorithm (Clinical Queries/Prognosis/narrow, specific search)

22 Prognostic markers studies Statistical Significance Assessment of statistical significance (SS) Abstract At least one SS prognostic effect/any marker/any outcome SS was assessed: p-value, 95% CI language ( statistical(ly) significant(ly) ) SS articles: positive

23 Prognostic markers studies Statistical Significance Assessment of statistical significance (SS) Negative articles Claiming significance for other analyses Expanding on non-significant trends Apologies Exact phrasing was recorded

24 Prognostic markers studies Statistical Significance

25 Prognostic markers studies Statistical Significance

26 Prognostic markers studies Statistical Significance Further studies, on other uniform populations, with tumour features different from those described here, are necessary in order to reveal the prognostic significance of the molecules discussed The cohort examined was relatively small and with larger patient numbers, MDM2 over-expression may emerge as a more significant covariate HIF-1 alpha does not appear to predict survival; however, this study suggests that bioreductive drugs should be investigated in clinical trials These results vary from our previous study on the expression of the differentiation marker cytokeratin 18, which showed that positive staining of tumour cells was associated with a statistically significant poorer prognosis at stage I regardless of histological types

27 Prognostic markers studies Discussion Almost everything is statistically significant Even in the absence of SS, authors find ways to defend the importance of their markers Publishable unit: At least 1 SS prognostic relation!!! Research culture driven by the pursuit of SS

28 Prognostic markers studies Discussion Selective reporting Poor quality of reporting Different analyses Reported analyses Unreported analyses Analyses performed on the same sample in other articles and on other samples on the same associations

29 Prognostic markers studies Discussion Clinical application: Tiny fraction of the proposed markers

30 Thank you