TRIPODS Workshop: Models & Machine Learning for Causal I. & Decision Making

Transcription

1 TRIPODS Workshop: Models & Machine Learning for Causal Inference & Decision Making in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD Department of Biostatistics Brown University School of Public Health in Medical Decision Making : Jan 14-18, 2019 Providence, RI, USA and Predictive Accuracy text Stavroula Chrysanthopoulou,

2 Outline Definition Description The MILC model Comparative Analysis Simulation Study in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

3 Definition MSMs in medical decision making Micro-simulation models (MSMs): Complex predictive models aimed at simulating individual disease trajectories using Markov Chain Monte Carlo methods. Incorporate combined information from several sources Predict outcomes of interest under different medical interventions Compare findings to decide on the best practice (cost-effectiveness analysis) Inform health policies in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

4 Description The MILC model The MIcrosimulation Lung Cancer (MILC) Model : an MSM describing the natural history of lung cancer Chrysanthopoulou SA (2017). MILC: A Microsimulation Model of the Natural History of Lung Cancer. International Journal of Microsimulation. 10(3):5-26 in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

5 Description The MILC model Micro-Simulation Model for lung cancer Streamlined MSM model Continuous time Dynamic Natural history of lung cancer (no screening component) Covariates Age Gender Smoking: Status (current, former, never) Start and quit smoking age Intensity (cigarettes/day) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

6 Description The MILC model Markov State Diagram Figure: Markov State Diagram of the continuous time in Medical Decision Making : S 0 : Disease-free state S 1 : Local state (onset of the 1st malignant cell) S 2 : Regional state (lumph nodes) S 3 : Distant state (distant metastases) S 4 : Death state h ij : the hazard of moving from state i to j and Predictive Accuracy text Stavroula Chrysanthopoulou,

7 Description The MILC model Components Onset of the 1st malignant cell TSCE carcinogenesis model (Moolgavkar 1990) Tumor growth Gompertz function (Laird 1964) Disease progression log-normal distribution (Spratt 1964) Survival competing risks (CIF estimates) NHIS and SEER data in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

8 Description The MILC model Structure of the MILC model in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

9 Description The MILC model R package MILC : in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

10 Comparative Analysis methods in MSMs: a comparative analysis in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

11 Comparative Analysis vs Estimation in statistical theory Statistical modeling: calibration estimation model fitting Here: calibration model tuning pertains to the specification of those sets of values for the model parameters that can result in predictions close to the observed, pre-specified target quantities. in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

12 Comparative Analysis Why calibration? MSM s complexity no closed form of the model s outputs (e.g., likelihoods, hazard rates, transition probabilities, etc.) Latent variables + high dimensionality identifiability problems Multiple sets of parameter values underlying correlation structures parameter uncertainty in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

13 Comparative Analysis methods for MSMs (overview) Directed optimum set (Nelder-Mead, simulation annealing, etc) Undirected acceptable sets (exhaustive or sampling design based grid search) Bayesian joint posterior distribution in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

14 Comparative Analysis Comparative analysis Objective: Comparative analysis of two calibration methods Bayesian: MCMC (Rutter et at, 2009) Empirical: Grid Search using Latin Hypercube Sampling (GSLHS) Implementation: MILC model Comparative analysis: Quantitative and qualitative in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

15 Comparative Analysis Methods Method Bayesian Empirical (Rutter et al. 2009) (Grid Search using LHS design) Goal draw values from the draw sets of acceptable joint posterior h(θ Y) values for θ Specifics Gibbs sampler Latin Hypercube Sampling appr. MH algorithm LR test Distribution of (multiple draws for) θ in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

16 Comparative Analysis Comparative Analysis Study Design I 1. Parameters to calibrate: tumor growth (θ 1 =m) disease progression (θ 2 =mdiagn, θ 3 =mreg, θ 4 =mdist) 2. Input data: sample (smpl.c N, N=5000) from 1980 US population of males, current smokers 3. targets, Y clbr =[Y <60 Y Y >80 ] T : lung cancer incidence rates by age group (2006 SEER data) 4. Size (total number of micro-simulations): M Bayes = (!!!) (sequential) M Emp = (!!!) (parallel) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

17 Comparative Analysis Comparative Analysis Study Design II Terms of comparison: Overlap calibrated values for θ predictions Model validation (predictions Ỹ vs Y clbr ): Internal (Ỹ = M50 (Θ, smpl.c 5000)) External (Ỹ = M50 (Θ, smpl.v 5000)) Efficiency computational time in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

18 Comparative Analysis Comparative Analysis Study Design III Tools: Graphical ways (density, contour, and box plots) Discrepancy measures Mean Absolute Deviations (MAD) MAD = 1 V V v=1 ỹ vj y j y j (1) Mean Squared Deviations (MSD) MSD = 1 V ) 2 (ỹvj y j (2) V v=1 Euclidean & Mahalanobis distances D M = (Ỹ Y clbr ) T S 1 (Ỹ Y clbr ) (3) y j in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

19 Comparative Analysis Density plots: marginal distributions of the calibrated MSM parameters. in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

20 Comparative Analysis Contour plots: bivariate distributions of the calibrated parameters in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

21 Comparative Analysis in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

22 Comparative Analysis Discrepancy measures: predictions vs calibration targets Internal Validation Bayesian Empirical <60 yrs yrs >80 yrs Overall <60 yrs yrs >80 yrs Overall MAD Y clbr MSD External Validation Bayesian Empirical <60 yrs yrs >80 yrs Overall <60 yrs yrs >80 yrs Overall MAD Y clbr MSD in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

23 Comparative Analysis High Performance Computing (HPC) using R I Total micro-simulations: M Bayes = , M Emp = MSM Embarrassingly Parallel Computations Solutions: Code profiling Rprof library Parallel computing snow & Rmpi libraries Code for parallel processing: for each θ = [θ 1, θ 2, θ 3, θ 4 ] T update, Bayesian method: m=4 50,000 (sequential) Empirical method: m=50,000 (simultaneous checks) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

24 Comparative Analysis High Performance Computing (HPC) using R II Algorithm efficiency improvement (m=50,000) Profiling Parallel Nodes Type Time ( % impr.) computing (secs) SOCK SOCK SOCK MPI MPI MPI Overall Improvement: /1.98 = faster (!!!) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

25 Comparative Analysis Discussion Overlapping distributions of the calibrated parameters (Θ BAYES vs Θ EMP ): univariate (density plots, MAD, MSD) multivariate (contour plots, Mahalanobis distances) Predictions (Ỹ BAYES vs Ỹ EMP): overall comparable Ỹ EMP less dispersed Ỹ BAYES better for rare events Computational burden: Emprical(parallel) more efficient than Bayesian(sequential) methods undirected(parallel) more efficient than directed(sequential) methods in R HPC techniques (!!!) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

26 Simulation Study Assessing the predictive accuracy of MSMs in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

27 Simulation Study Performance of predictive models Explained variation (R 2 statistics) (GoF statistics) Discrimination (C-statistics) Implementation: type of predictions (e.g., continuous, ordinal, nominal, survival data) : How close individual predictions are to observed data of MSMs: Important though not explored yet incorporation of individual level characteristics between individuals variability in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

28 Simulation Study Assessing predictive accuracy of MSMs Output of interest: survival time (event or censoring) common survival models (Cox-PH, AFT, etc.): predicted risk VS observed survival time MSMs (special type of survival predictive models): predicted survival time VS observed survival time Proposed methods: Concordance statistics: correct classification, given a set of covariates, based on the predictions (enough?) Harrell s index (Harrell et al 1996) Uno s index (Uno et al 2011) Hypothesis testing: Predicted vs observed survival function log-rank Renyi type statistics Cramer von Mises statistics in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

29 Simulation Study Simulation Study Objective: Compare the proposed methods for the assessment of the predictive accuracy of MSMs Input sample (N=5000) of males, current smokers (US 1980 population) Simulate observed lung cancer incidence (Gompertz distribution) Predict lung cancer incidence (calibrated MILC model) Ỹ BAYES = M(Θ BAYES, smpl ), Ỹ EMP = M(Θ EMP, smpl ) for V={200, 400, 600, 800, 1000} vectors for θ Apply C-statistics and Hypothesis testing methods for the predictive accuracy of the two MSMs (predicted vs observed survival data) Evaluate the performance of the proposed methods in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

30 Simulation Study in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

31 Simulation Study Calibrated MSM Method Bayesian Empirically C-statistic Harrell ± ± Uno ± ± Hypothesis Test Log-Rank (non-rejection Renyi rate %) C-M (Q1) C-M (Q2) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

32 Simulation Study Discussion Conclusions: Runs of the MSM for V=400 vectors for θ (randomly selected from Θ) adequate for assessing the predictive accuracy of the model C-statistics: almost identical results (estimates & conclusions) cannot capture differences in the accuracy of the individual predictions Hypothesis testing similar results (analogous estimates & same conclusions) log-rank & Renyi type tests more sensitive in detecting differences between observed and predicted individual survival data preferable for the assessment of the predictive accuracy of an MSM in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

33 Simulation Study Acknowledgments Collaborators Constantine Gatsonis, PhD (Brown University) Carolyn Rutter, PhD (RAND - CISNET, colorectal cancer group) Matthew Harrison, PhD (Brown University) Joseph Hogan, PhD (Brown University) in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,

34 Simulation Study References Gatsonis, C., et al (2011), The National Lung Screening Trial: Overview and Study Design, Radiology, 258, Laird, A. K. (1964), Dynamics of Tumor Growth British Journal of Cancer, 18, McMahon, P. M. (2005), Policy assessment of medical imaging utilization: methods and applications, [doctoral thesis] Meza, R., Hazelton, W. D., Colditz, G. A., and Moolgavkar, S. H. (2008), Analysis of lung cancer incidence in the nurses health and the health professionals follow-up studies using a multistage carcinogenesis model, Cancer Causes & Control, 19, Moolgavkar, S. H. and Luebeck, G. (1990), Two-Event Model for Carcinogenesis: Biological, Mathematical, and Statistical Considerations, Risk Analysis, 10, Rutter CM, Miglioretti DL, Savarino JE., (2009), Bayesian calibration of microsimulation models, JASA;104(488): Rutter CM, Zaslavsky AM, Feuer EJ, (2010), Dynamic Microsimulation models for health outcomes: a review, MDM Spratt, J. S. and Spratt, T. L. (1964), Rates of Growth of Pulmonary Metastases and Host Survival Annals of Surgery, 159, Steyerberg, E. W., et al (2010), Assessing the Performance of Prediction Models A Framework for Traditional and Novel Measures, Epidemiology, 21, Stout N.K., et al (2009), Methods used in Cancer Simulation Models and Suggested Reported Guidlines, Pharmacoeconomics, 27, Vanni T., et al (2011), Calibrating models in economic evaluation: a seven-step approach, Pharmacoeconomics, 29, in Medical Decision Making : and Predictive Accuracy text Stavroula Chrysanthopoulou,