Genetic risk prediction for CHD: will we ever get there or are we already there?

Transcription

1 Genetic risk prediction for CHD: will we ever get there or are we already there? Themistocles (Tim) Assimes, MD PhD Assistant Professor of Medicine Stanford University School of Medicine WHI Investigators meeting May 7, 2015 Disclosure: CRA with Telomere Diagnostics Inc.

2 Initial GWAS: Complex diseases have more complex genetic architectures then expected not the best situation for risk prediction Adapted from Thanassoulis, G. and R.S. Vasan, Genetic cardiovascular risk prediction: will we get there? Circulation, (22): p

3 Most widely used and recognized test of discrimination C statistic AKA Receiver-operating-curve (AUC) Sensitivity vs 1-Specificity Probability among a randomly selected case and control, that the case will have a higher modelbased predicted probability of an event 0.5 = chance 1.0 = perfect standard metric for binary outcomes Limitations Big differences in risk = small differences in risk Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29 36.

4 AUC tough to budge when its already reasonably good Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17-23.

5 Pitfall of relying only on AUC: some TRFs would not be included in current scores + indicates the addition of each variable separately to the model with age, SBP, smoking only Adapted from Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:

6 General concept of reclassification Who moves and to where with new model? But what if you moved a subject inappropri ately? E.g. move case into a lower category of risk Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:

7 New discrimination tests NRI and cnri introduced in 2008 Consider absolute predicted risk of individuals ( reclassification ) 2 category Net reclassification index (NRI) clinical NRI (cnri) The NRI for the intermediate category of risk only US Preventative Services Task Force endorsed concept of reclassification Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72; discussion Cook, N.R., Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine. Stat Med, (2): p Helfand M, Buckley DI, Freeman M, et al. Emerging risk factors for coronary heart disease: a summary of systematic reviews conducted for the U.S. Preventive Services Task Force. Ann Intern Med 2009;151:

8 NRI for adding HDL to Framingham AUC: (without HDL), (with HDL), ΔAUC p-value= Adapted from Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157-72; discussion

9 Bias in the cnri Clinical NRI widely used but found to be biased Overall NRI could be negative and the clinical NRI highly positive (including several GRS papers) Correction for Clinical NRI correction Wiped out signal in 2 high profile CVD risk predictions papers Paynter, N.P. and N.R. Cook, A Bias-Corrected Net Reclassification Improvement for Clinical Subgroups. Med Decis Making, 2012

10 cnri bias in CHD risk prediction Ripatti S, Tikkanen E, Orho-Melander M, Havulinna AS, Silander K, Sharma A, et al. A multilocus genetic risk score for coronary heart disease: case-control and prospective cohort analyses. Lancet Oct 23;376(9750): PubMed PMID:

11 Questioning the utility of the NRI NRI is not a proportion Only the NRI events and NRI nonevents Combining as a simple sum or not appropriate and potentially misleading If one weights overall prevalence of events and non-events, NRI can easily move from positive to negative territory look at each separately and consider clinical consequences 3 category NRI doesn t consider large jumps in risk any differently than small jumps Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology Jan;25(1): PubMed PMID: Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med Jan 21;160(2): PubMed PMID:

12 Large and Small Values for NRI >0 Are Undefined Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, (1): p

13 Questioning that statistical properties of NRI >0 high false positive rate even with independent test set Can Make Uninformative New Markers Appear Predictive Especially if models not well calibrated But not the case for AUC or for likelihood ratio testing Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, (1): p Pepe, M.S., H. Janes, and C.I. Li, Net risk reclassification p values: valid or misleading? J Natl Cancer Inst, (4): p. dju041

14 Concerns with testing the nulls for NRI H0 : NRI = 0, z-statistic has never been validated t t For 2-category NRI event or NRI non-event at a given risk threshold cannot reject H 0 : NRI event = 0 and H0 : NRI non-event = 0 on the basis of Y being a risk factor. Tests not yet established for these nulls Kerr, K.F., et al., Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology, (1): p Pepe, M.S., H. Janes, and C.I. Li, Net risk reclassification p values: valid or misleading? J Natl Cancer Inst, (4): p. dju041

15 Back to the AUC?? recent insights on testing whether a new model is better than the old one EQUIVALENT NULL HYPOTHESES H 0 : risk (X,Y) = risk (X) H 0 : AUC (X,Y) = AUC (X) Recommend standard regression statistics No need to test null > 1 x Superior power with Highly developed likelihoodbased tests Avoid inconsistent results from inference that has not been worked out as well for other methods Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, (9): p

16 Back to the AUC?? Testing the AUC Much more work needed re: properties of tests Delong or resampling based tests do not adjust for variability in est. regression coefficients VERY CONSERVATIVE (low power) - even after bootstrap Is this why AUC is insensitive to improvements in prediction performance? Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, (9): p

17 Performance of GRS for CHD today Many examples of robust and relatively consistent association with GRS using ~45-50 GWAS SNPs Cohort GRS RR (95% CI) Comparison # events ARIC 1.29 ( ) Per SD GRS 620 Finnish Cohorts 1.27 ( ) Per SD GRS Swedish Cohorts 1.54 ( ) 4 th quartile to 1 st 781 Goldstein, B.A., et al., Simple, standardized incorporation of genetic risk into non-genetic risk prediction tools for complex traits: coronary heart disease as an example. Front Genet, : p. 254 Tikkanen, E., et al., Genetic risk prediction and a 2-stage risk screening strategy for coronary heart disease, in Arterioscler Thromb Vasc Biol p Ganna, A., et al., Multilocus Genetic Risk Scores for Coronary Heart Disease Prediction. Arterioscler Thromb Vasc Biol, (9): p

18 Then what? Need to estimate the extent of improvement Big debate as to how to quantify improvement One strong recommendation: net benefit (NB) Good news, if you reject H 0 : risk (X,Y) = risk (X) You also reject H 0 : NB (X,Y) (t) = NB (X) (t) where t is risk threshold Test of equality of decision curves Pepe, M.S., et al., Testing for improvement in prediction model performance. Stat Med, (9): p Vickers, A.J. and E.B. Elkin, Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making, (6): p

19 Net benefit analysis of treating with rosuvastatin in the JUPITER trial Dorresteijn, J.A., et al., Estimating treatment effects for individual patients based on the results of randomised clinical trials. BMJ, : p. d5888.

20 Comparison of HRs for the last CHD risk factor added to the model Goldstein, Salfati, Yang, Assimes, under preparation

21 My optimistic viewpoint for genetic risk prediction in CHD ++ Markers reproducible and stable ++ Safe and effective interventions We have already overcome initial analytic challenges GRS with robust association comparable to other risk factors, will continue to improve Net benefit likely present when it comes to statin Rx clinical trial to test GRS? Value to just having a better calibrated model to convey risk? the main impediment to implementation is cost genotyping / sequencing Technical genetic data & lgorithms into EHRs