7/25/2014. Data Sharing & Reproducible Research: Council of Science Editors May Reproducible Research

Transcription

1 Data Sharing & Reproducible Research: Council of Science Editors May 2014 Christine Laine, MD, MPH Editor in Chief, Annals of Internal Medicine Reproducible Research Scientific community arrives at the truth by independent verification of new observations Replicationby independent scientists in independent settings using independent data provides the best assurance of validity Reproductionusing the same data set by independent scientists is next best Reproduction requires sharing of protocol, statistical code, and data Reproducible Research is a model for communicating research that promotes transparency of methods 2 Authors of original research must declare whether they will share their protocol, statistical code, and data Annals does not require sharing, just documentation of whether (and under what conditions) authors would share Published at end of every research article 1

2 Annals Reproducible Research: Authors Willingness to Share % Willing to share Willingness to share 2012 vs. 2008: Protocol OR 0.39 (0.18,0.85) Statistical code OR 1.07 (0.50,1.39) Data OR 0.53 (0.28,1.04) 5 Annals Reproducible Research: Willingness to Share by Study Characteristics Study Design RCT (n=110) Observational (n=225) Decision/Cost Analysis (n=25) Other (n=29) Funding Industry (n=73) Non-Industry (n= 316) Conflicts Disclosed Yes (n=303) None disclosed (n=86) Protocol Stat. Code Data 85% 67% 58% 50% 72% 70% 67% 85% 73% 65% 72% 77% 56% 53% 52% 55% 52% 54% 50% 60% OVERALL 72% 54% 6 2

3 Annals Observations The majority of authors stated that they would make study materials available to others. However, most would do so only if others contacted them and attached requirements to the sharing of this information. Willingness to share was fairly stable Researchers were most willing to fully share protocols and least willing to share data. 7 What Data Sharing Can Achieve Lessons from Annals/YODA rhbmp-2 Project 8 Annals/YODA rhbmp-2 Project rhbmp-2 is widely used in spinal fusion surgery Systematic review published in the journal Spine in 2011suggested adverse effects not adequately reported in clinical trials, but reviewers did not have patient-level data Medtronic made patient level data from all their rhbmp-2 trials available to the Yale Open Date Access Project (YODA) for patient-level meta-analysis YODA decided to contract with two independent systematic review groups to conduct their own systematic review and meta-analyses of these data YODA approached Annals to peer review the metaanalyses and publish both if suitable for publication even if they ended up being identical 9 3

4 Annals Review Process Split the editorial team in half, each with equal numbers of physician and statistical editors Handled outside of our electronic system Separate external reviewers Each paper underwent 3 rounds of revision Neither team looked at the other s paper until after both were in final accepted form 10 Annals of Internal Medicine Volume June 2013 Published with: -All versions of manuscripts -Reviews -Editors correspondence -Editorials by YODA team Medtronic Orthopedic surgeons Annals editors rhbmp-2: Evidence Search and Selection Data Search York Evidence Center Medtronic, literature search through 2012 Oregon Evidence Center Medtronic, literature search through 2012 Evidence Identified Studies Excluded 11 Medtronic RCTs (n=1302) 1 other RCT (n=106) 35 observational studies MedWatch reports 4 single group trials 1 trial that did not use iliac crest bone grafting as the comparator 12 Medtronic RCTs (n=1879) 1 other RCT (n=102) 31 cohort studies 47 intervention series 34 case reports 1 Medtronic trial (n=3) 4 other Medtronic trials (not specified why) 12 4

5 rhbmp-2: Outcomes/Analysis Outcomes York Evidence Center Primary: pain (Owestry Disability Scale) Oregon Evidence Center Primary: Overall success as defined by trials (mostly fusion) Secondary: fusion, adverse events Secondary: Pain, disability, neurologic status, function, adverse events Time 6 wks, 3, 6, 12, and 24 months 6 wks, 3, 6, 12, and 24 months Analytic Approach 2 stage meta-analysis using random effects as the primary analysis Mixed effects model as a sensitivity analysis Mixed effects models by spine area and surgical approach for effectiveness, but pooled for adverse events 2-stage meta-analysis as a sensitivity analysis 13 rhbmp-2: Conclusions Effectiveness Safety York Evidence Center Pain not clinically significantly different with rh-bmp-2 vs. bone graft Radiographic fusion was 12% more common with rhbmp-2 Heterotopic bone formation, osteolysis, retrograde ejaculation more common with rhbmp-2 Cancer was infrequent but twice as common with rhbmp-2 (RR 1.98[ ] Oregon Evidence Center rhbmp-2 and bone graft had similar effects on overall success (fusion) AEs high (77-93%), similar in both groups rhbmp-2 assoc with urogenital problems in anterior lumbar fusion, wound problems/dysphagia in anterior cervical fusion Cancer was infrequent but higher with rhbmp-2 (RR3.45[1.98,6.0] Reporting bias Did not assess Selective, duplicate, & under-reporting 14 rhbmp-2: Conclusions In spinal fusion surgery, when compared to iliac crest bone grafting, rhbmp-2: results in no or modest improvement in radiologic fusion does not yield a clinically significant improvement in clinical outcomes (pain) at 2 years is associated with adverse events, possibly including cancer, but safety issues seemed less marked than suggested by initial Spine review 15 5

6 rhbmp-2: Conclusions In spinal fusion, rhbmp-2 yields clinical outcomes (pain, function) the same as with iliac crest bone grafting despite higher fusion rates with rhbmp- 2 Higher risk of adverse events, including possibly cancer BOTTOM LINE: Don t routinely use rhbmp-2 in spinal fusion surgery 16 Lessons Learned The sum is more than its parts. Peer review matters what readers see is not what the journal got. Reproducible research is valuable. Question of effectiveness/safety could not be answered adequately without pt-level data from multiple trials. 17 What contributed to the rhbmp-2 mess? Most trials focused on radiographic fusion rather than on outcomes important to patients Many trials were underpowered to detect clinically important differences in effectiveness or safety Trials failed to follow standards for reporting adverse events Many of the trialistshad conflicts of interest, many of which were not properly disclosed Trial reports did not include sufficient detail to enable comprehensive systematic reviews 18 6

7 Other Recent Data Sharing Efforts, Proposals, & Activities European Medicines Agency: Draft policy to next be discussed March 2014 Johnson & Johnson: Jan 2014 agreement with Yale Open Data Access Project to review equess from those seeking access to anonymized trials data and make final decisions on sharing GSK: established a site and system that enables researchers to submit research proposals and request anonymized data from clinical trials listed on the site or to ask about studies not on the site, Independent Review Panel considers whether to issue a data sharing agreement and provide access to the data on the protected site Boerhringer Ingelheim, Roche, Sanofi, and ViiV Healthcare have committed to contribute data to the GSK site 19 7