2014 Modern Modeling Methods (M 3 ) Conference, UCONN

Transcription

1 2014 Modern Modeling (M 3 ) Conference, UCONN Comparative study of two calibration methods for micro-simulation models Department of Biostatistics Center for Statistical Sciences Brown University School of Public Health May 20-21, 2014

2 Outline MSM for lung cancer Description Discussion

3 MSMs in medical decision making Micro-simulation models (MSMs): Complex predictive models aimed at simulating individual disease trajectories using Markov Chain Monte Carlo methods. Incorporate combined information from several sources Predict outcomes of interest under different medical interventions Compare findings to decide on the best practice

4 Description Discussion The MIcrosimulation Lung Cancer (MILC) model Streamlined MSM model Continuous time Dynamic Natural history of lung cancer (no screening component) Covariates Age Gender Smoking: Status (current, former, never) Start and quit smoking age Intensity (cigarettes/day)

5 Description Discussion Markov State Diagram of the MILC model S 0 : Disease-free state S 1 : Local state (onset of the 1st malignant cell) S 2 : Regional state (lumph nodes) S 3 : Distant state (distant metastases) S 4 : Death state h ij : the hazard of moving from state i to j

6 Description Discussion Components Onset of the 1st malignant cell TSCE carcinogenesis model (Moolgavkar 1990) Tumor growth Gompertz function (Laird 1964) Disease progression log-normal distribution (Spratt 1964) Survival competing risks (CIF estimates) NHIS and SEER data

7 Description Discussion Software: R Library: MILC S. A. Chrysanthopoulou., MILC: MIcrosimulation Lung Cancer (MILC) model, R package version 1.0. URL:

8 Description Discussion Discussion Summary: New streamlined MSM, natural history of lung cancer (best practices in lung cancer modeling) implementation in R (transparency) tool for studying the statistical properties of MSMs Limitations: no screening or treatment component level of complexity Future work: include screening and treatment components increase complexity to describe lung cancer course in more detail

9 methods in MSMs: a comparative analysis

10 vs Estimation in statistical theory Statistical modeling: calibration estimation model fitting Here: calibration model tuning pertains to the specification of those sets of values for the model parameters that can result in predictions close to the observed, pre-specified target quantities.

11 Why calibration? MSM s complexity no closed form of the model s outputs (e.g., likelihoods, hazard rates, transition probabilities, etc.) Latent variables + high dimensionality identifiability problems Multiple sets of parameter values underlying correlation structures parameter uncertainty

12 methods in MSMs Specification of parameter values (Stout 2009, Vanni 2011): Directed optimum set (Nelder-Mead, simulation annealing, etc) Undirected acceptable sets (exhaustive or sampling design based grid search) Bayesian joint posterior distribution

13 Comparative analysis Objective: Comparative analysis of two calibration methods Bayesian: MCMC (Rutter et at, 2009) Empirical: Grid Search using Latin Hypercube Sampling (GSLHS) Implementation: of the MILC model Comparative analysis: Quantitative and qualitative

14 Method Bayesian Empirical (Rutter 2009) (new) Goal draw values from the draw sets of acceptable joint posterior h(θ Y) values for θ θ : parameters Y : predictions Specifics Gibbs sampler Latin Hypercube Sampling appr. MH algorithm LR test V draws for θ

15 Bayesian Set up: Priors π(θ) Data Y j f(y j g j (θ)), (e.g., lung cancer cases Poisson(λ j = g j (θ))) Joint posterior h(θ Y j ) π(θ) f(y j g j (θ)) How? Gibbs sampler to sequentially draw from h(θ k Y j, θ ( k) ) g( ) unknown approximate MH algorithm Ỹjm, m=1, 2,..., M runs of the MSM given θ ĝ j (θ) = T (Ỹjm) substitute ĝ j (θ) = T (Ỹjm) in: r k (θ k, θ k) = π k(θ k ) J j=1 f j(y j ĝ j (θ k, θ ( k))) π k (θ k ) J j=1 f j(y j ĝ j (θ)) (1)

16 Empirical (example: 2-dimensions) I Set up: Plausible values for θ 1, θ 2 (e.g., literature, ad-hoc analysis) Data Y j f( Y j g j (θ)), (e.g., lung cancer cases Poisson) Joint distribution of calibrated MSM parameters, estimated by the joint empirical of the acceptable sets of parameter values How? Repeated Latin Hypercube Sampling (V=R N LHS ) design to search the parameter space Deviance statistic criterion for acceptable parameter values θ

17 Empirical (example: 2-dimensions) II How? (continue) Single LHS N LHS draws of θ Parameter space N LHS N LHS grid randomly draw N LHS parts of the grid randomly draw θ1, θ 2 within each part g( ) unknown ĝ j (θ) = T (Ỹjm) acceptable values Deviance statistic: J D = 2 [l(ĝ(θ) y j ) l(y j y j )] (2) j=1 Accept θ for which H 0 is not rejected at α% Figure: Single LHS of size 20

18 I 1. Parameters to calibrate: θ1 =m (tumor growth) θ 2 =mdiagn, θ 3 =mreg, θ 4 =mdist (disease progression) 2. Input data: sample (smpl.c 5000, N=5000) from 1980 US population of males, current smokers baseline characteristics = {age, smoking intensity} 3. targets, Y clbr =[Y <60 Y Y >80 ] T : lung cancer incidence rates by age group SEER data ( ) 4. output: V=1,000 vectors for θ

19 II Terms of comparison: Overlap calibrated values for θ predictions Model validation (predictions Ỹ vs Y clbr ): (Ỹ Internal = M50 (Θ, smpl.c 5000)) (Ỹ External = M50 (Θ, smpl.v 5000)) Efficiency computational time

20 III Tools: Graphical ways (density, contours, and box plots) Discrepancy measures Mean Absolute Deviations (MAD) MAD = 1 V V v=1 ỹ vj y j y j (3) Mean Squared Deviations (MSD) V ) 2 (ỹvj y j (4) MSD = 1 V v=1 Euclidean & Mahalanobis distances D M = y j (Ỹ Y clbr ) T S 1 (Ỹ Y clbr ) (5)

21 Density plots for the marginal distributions of the calibrated MSM parameters.

22 Contour plots for the bivariate distributions of the calibrated parameters

23

24 Box-Plots of the multivariate distances of the MILC predictions from the calibration targets

25 Discrepancy measures: predictions vs calibration targets Internal Validation Bayesian Empirical <60 yrs yrs >80 yrs Overall <60 yrs yrs >80 yrs Overall MAD Y clbr MSD External Validation Bayesian Empirical <60 yrs yrs >80 yrs Overall <60 yrs yrs >80 yrs Overall MAD Y clbr MSD

26 High Performance Computing (HPC) using R I Total micro-simulations: M Bayes = , M Emp = MSM Embarrassingly Parallel Computations Solutions: Code profiling Rprof library Parallel computing snow & Rmpi libraries Code for parallel processing: for each θ = [θ 1, θ 2, θ 3, θ 4 ] T update, Bayesian method: m=4 50,000 (sequential) Empirical method: m=50,000 (simultaneous checks)

27 High Performance Computing (HPC) using R II Algorithm efficiency improvement (m=50,000) Profiling Parallel Nodes Type Time ( % impr.) computing (secs) SOCK SOCK SOCK MPI MPI MPI Overall Improvement: /1.98 = faster (!!!)

28 Discussion Overlapping distributions of the calibrated parameters (Θ BAYES vs Θ EMP ): univariate (density plots, MAD, MSD) multivariate (contour plots, Mahalanobis distances) Predictions (Ỹ BAYES vs Ỹ EMP): overall comparable Ỹ EMP less dispersed Ỹ BAYES better for rare events Computational burden: Emprical(parallel) more efficient than Bayesian(sequential) methods undirected(parallel) more efficient than directed(sequential) methods in R HPC techniques (!!!)

29 Future work Comprehensive calibration of the MILC model gender (males/females) smoking status (never/former/current) Multiple calibration targets (lung cancer incidence and mortality, tumor size at diagnosis, etc.) Use of the MILC model to assess results from the National Lung Screening Trial (NLST) study

30 Thesis committee Advisor: Constantine Gatsonis, PhD (Brown University) Readers: Carolyn Rutter, PhD (Group Health Research Institute - CISNET, colorectal cancer group) Matthew Harrison, PhD (Brown University) Xi Luo, PhD (Brown University)

31 References Gatsonis, C., et al (2011), The National Lung Screening Trial: Overview and Study Design, Radiology, 258, Laird, A. K. (1964), Dynamics of Tumor Growth British Journal of Cancer, 18, McMahon, P. M. (2005), Policy assessment of medical imaging utilization: methods and applications, [doctoral thesis] Meza, R., Hazelton, W. D., Colditz, G. A., and Moolgavkar, S. H. (2008), Analysis of lung cancer incidence in the nurses health and the health professionals follow-up studies using a multistage carcinogenesis model, Cancer Causes & Control, 19, Moolgavkar, S. H. and Luebeck, G. (1990), Two-Event Model for Carcinogenesis: Biological, Mathematical, and Statistical Considerations, Risk Analysis, 10, Rutter CM, Miglioretti DL, Savarino JE., (2009), Bayesian calibration of microsimulation models, JASA;104(488): Rutter CM, Zaslavsky AM, Feuer EJ, (2010), Dynamic Microsimulation models for health outcomes: a review, MDM Spratt, J. S. and Spratt, T. L. (1964), Rates of Growth of Pulmonary Metastases and Host Survival Annals of Surgery, 159, Steyerberg, E. W., et al (2010), Assessing the Performance of Prediction Models A Framework for Traditional and Novel Measures, Epidemiology, 21, Stout N.K., et al (2009), used in Cancer Simulation Models and Suggested Reported Guidlines, Pharmacoeconomics, 27, Vanni T., et al (2011), Calibrating models in economic evaluation: a seven-step approach, Pharmacoeconomics, 29, 35-49