Critical Thinking A tour through the science of neuroscience

Transcription

1 Critical Thinking A tour through the science of neuroscience NEBM 10032/5 Publication Bias Emily S Sena Centre for Clinical Brain Sciences, University of Edinburgh slides

2 Bias Biases in research and reporting are presumably exacerbated by systems of publication and career evaluation that reward impact and productivity over quality and ability to replicate studies Systematic reviews Can only include data made available If unpublished data are systematically different to the published literature then the summarised data, opinions and understanding will be biased.

3 Growing Research Area Increased efforts in the study of conscious and unconscious biases in research Threaten human health Waste economic resources Threaten scientific progress Fanelli and Ioannidis, 2013

4 Reporting bias Neutral and negative studies Publication bias Time lag bias Language bias Vibration effects Selective analysis reporting Selective outcome reporting

5 Publication Bias Neutral and negative studies remain unpublished less likely to be identified in systematic review leads to the overstatement of efficacy in meta-analysis Hot topic for clinical trials summary results of clinical trials conducted in Europe publicly available

6 How to assess for it? Effect size and corresponding standard error To assess for its presence Funnel plot Egger regression Estimate efficacy in the absence of publication bias Trim and Fill

7 Funnel Plots Work on the basis that small studies are likely to be more spread around the mean Small studies that are significant are more likely to be published inverted funnel Funnel plots are visually assessed and most useful with a large number of studies Relative measures plotted on a log scale ensures that effects of the same magnitude but opposite directions are equidistant from 1.0 Used to examine whether smaller studies report larger treatment effects

8 Funnel Plot Precision Effect Size

9 Egger Regression Egger regression statistically assess asymmetry of the funnel plot The standardised effect is regressed on the precision A weighted regression of the effect size on it standard error with the weight equal to the precision With no asymmetry the regression line and its 95% CI will pass through the origin Again, not proof of bias but does raise the questions regarding the interepration of results

10 Egger Regression Precision Effect Size/Standard Error

11 Trim and Fill Missing studies are iteratively identified ( trimming ) and are replaced ( filling ) in order to calculate an adjusted estimate of effect size Studies with the largest deviation from the mean are trimmed The remaining symmetrical plot is used to recalculate the summary effect Trimmed studies are replaced and their counterparts filled to correct variance The mirror axis is placed along the adjusted estimate

12 Trim and Fill Overall efficacy was reduced from; 32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%)

13 Publication bias in experimental stroke Trim and Fill suggested 16% of experiments remain unpublished Best estimate of magnitude of problem Overstatement of efficacy 31% Only 2% publications reported no significant treatment effects

14 Cancer Induced Bone Pain - animals 103 filled studies 2.79 ( ; n=257) to 1.6 ( ; n=360) 42.3% relative overstatement

15 Publication bias 20% - 32% n expts Estimated unpublished Reported efficacy Corrected efficacy Stroke infarct volume % 27.5% EAE - neurobehaviour % 15.0% EAE inflammation % 37.5% EAE demyelination % 30.5% EAE axon loss % 41.7% AD Water Maze sd sd AD plaque burden sd sd

16 Few Studies 16 Studies OR 6.7 ( )

17 Few Studies 7 filled trials OR 6.7 to % relative overstatement in effect?

18 More Data 165 comparisons OR 1.8 ( )

19 More Data 34 filled trials OR 1.8 to % relative overstatement in effect

20 Limitations Too few studies, ideally want >25 Need reasonable dispersion of sample size For sparse data the Mantel-Haenszel methods are probably more appropriate for weighting but Trim and Fill function only has the inverse variance method enabled Small study effects may have other causes ORs rather than NNT Measure of variance difficult to derive for NNT OR are more extreme compared to RR if the event rate is high so asymmetry may observed with OR

21 Note of caution. Funnel plot asymmetry has a number of potential sources Selection bias (Publication bias, reporting bias, biased inclusion criteria) True heterogeneity (study effect differs according to study size, differences in underlying risk) Data irregularities (poor methodological design, inadequate analysis, fraud) Artefact (poor choice of effect measure) Chance

22 Is the effect an artefact of bias? We checked for its presence and impact but not whether the overall effect is robust Rosenthal s fail-safe N How many studies do we need to nullify the observed effect? Statistical rather than substantiative Assumes missing studies have an effect of zero rather than negative or small positive effects Orwin s Fail-safe N Attempts to address both issues

23 Other options Cumulative meta-analysis by size This allows you to see whether adding smaller studies shifts a stable effect size of larger studies Smaller studies are generally given less weight but are they influencing the pooled estimate Used a fixed model (Mantel-Haenszel) where less weight is given to smaller studies. Will not be thrown by few aberrant studies

24 Cumulative MA by size

25 Cumulative MA by size

26 Other options - Excess Significance To identify whether there is an excess of significant studies within previously conducted systematic reviews of neurological disorders

27 Excess Significance Test Examines whether too many individual studies in a meta-analysis report statistically significant results Driven by selective reporting of analyses or outcomes Low power to detect bias in a single meta-analysis with few studies but is applicable across many metaanalyses in a field

28 Methods Dataset 6 diseases Alzheimer s disease, EAE, focal ischaemia, intracerebral haemorrhage, Parkinson s disease, and spinal cord injury 60 meta-analyses 4,445 experiments Median sample size = 16 (IQR 11-20)

29 Methods Data extraction Year of publication Intervention Effect size + standard error Sample size Quality score items Peer review, randomisation, allocation concealment, blinded assessment of outcome, sample size calculation, conflict of interest Funnel plot asymmetry

30 Methods Excess Significance Test [(O)>(E)] Observed nominally positive studies p<0.05 Binomial test Expected positive studies is power True effect size Fixed effect size of the most precise study Wilcoxon rank sum test

31 Assessed excess significance In aggregate across the six disease Within each disease Subgroup Amount of heterogeneity Significant summary fixed effect Egger Regression Reporting of measures to minimise bias (quality score)

32 Expected significant results = 919 (21%) Observed significant results = 1719 (39%) Excess significance was present across all six neurological disorders in all subgroups defined by methodological or reporting characteristics The strongest excess significance was observed in meta-analyses where small study effects were also observed with the least precise studies

33 Some research areas may be more susceptible to bias Softer sciences Proportion of studies reporting positive results increases moving from physical sciences to medical to social sciences. Research choices are more influenced by a scientist s own beliefs, expectations and wishes. Behavioural studies Regional Publish or perish long characterised US-based research Research from Asian and developing countries tends to be rejected unless they report extraordinary results Fanelli and Ioannidis, 2013

34 Further analysis of research bias 82 meta-analyses 1,174 primary outcomes Samples from health related biological and behavioural research Measured how individual results deviated from the overall summary estimate of effect from within their respective meta-analysis Stratified by research type and country of origin Fanelli and Ioannidis, 2013

35

36 Hypotheses In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity Leads to a body of evidence with an inflated proportion of published studies with statistically significant results This potentially compromises the utility of animal models and contributes to translational failure

37 What happens Small (underpowered), poorly conducted studies reach spurious (falsely positive) conclusions but are published because they are seen to be interesting. Small (perhaps) poorly conducted (sometimes) studies not reaching the same conclusions are not published. Investigators become conditioned by the apparent success that comes from conducting small underpowered studies Investigators keep trying to replicate the positive studies Exacerbated by the combination of pressures to publish and winner-takes-all system of rewards.

38 What can we do? Problem: Not all outcomes are reported A priori analyses not always reported What we need: We need to know if scientists report what they set out to report Solution: Published Protocols Registries

39 Thanks to... Edinburgh Malcolm Macleod Kieren Egan Hanna Vesterinen Gill Currie Rustam Al-Shahi Salman Joseph Frantzis Project Students Melbourne David Howells Ana Antonic Peter Batchelor Taryn Wills Sarah McCann Stanford/Ioannina John Ioannidis Kostad Tsilidis Orestis Panagiotou Eleni Aretouli Vangelis Evangelou Utrecht Bart van der Worp Nottingham Philip Bath Translators NHS R&D methodology program Chief Scientist Office e_drugs_they_prescribe.html