Markov Decision Processes for Chronic Diseases Lessons learned from modeling type 2 diabetes Brian Denton Department of Industrial and Operations Engineering University of Michigan
Agenda Models for study of diabetes treatment decisions Methods for sensitivity analysis Examples: HbA1C control Cholesterol and blood pressure control
Industrial and Operations Engineering IOE Department Statistics: Awarded over 6,500 bachelor's degrees Awarded more than 2,577 master's degrees Awarded 476 doctoral degrees 528 undergraduates 208 graduate students 31 faculty members (many interested in healthcare) Department ranked #2 in US News
My Research Interests Development and validation of quantitative models for comparative effectiveness Cost-Effectiveness of new technologies Predictive models for medical decision making
Why resort to models? Privacy, Ethics, and Cost
PubMed Search Results 8000 7000 6000 5000 4000 3000 2000 1000 0
What is a Markov Decision Process? Starts with a Markov model for a disease (states, transition probabilities, rewards) Overlays a decision process on the model that: Defines allowable actions at each time period and each state Goal is to find the optimal action in each state at each period to maximize rewards
State Transition Diagram r(e,i) Non-Fatal Events On Treatment r(s,i) Health States before an event has occurred. r(l,w) L r(m,w) M H r(h,w) V r(v,w) r(d,d) Death
Optimality Equations Health status: Treatment Status (on or off medication): Action: a ( st s t, m ) S { 1,2,3,..., L} m {0,1} I, W if m 0 W if m 1 Optimal Reward to Go 1 Period Reward Expected Future Reward v t ( s t, m) max R( st, I), r( s t, W ) p( st ' st, W ) vt ( st ', m) 1 s t Discounted future rewards on treatment starting at age t Transition probabilities IOE512 Dynamic Programming offered every Fall Semester
Decision Process Choose the best action each year to achieve a goal such as the following: (Willingness to Pay) (Life Years) Costs Expected benefit of treatment Expected benefit of treatment Expected benefit of treatment Initiate or Delay Treatment? Change in Health Status Initiate or Delay Treatment? Change in Health Status Initiate or Delay Treatment? Change in Health Status Age 40 Age 41 Age 42
States for Diabetes HbA1c Cholesterol: Total Cholesterol HDL Triglycerides LDL Blood Pressure Health History Medication History
Example: Cholesterol States Total Cholesterol Level High-density Lipoprotein Level L / L L / M L / H L / V M / L... V / V TC and HDL have four possible levels each, so there are 16 states in total. L M H V TC <160 160-200 200-240 >240 HDL < 40 40-50 50-60 >60
Example: Total Cholesterol
Cholesterol Computing Treatment Effects Treatment options: Statins Fibrates Ace Inhibitors ARBs Calcium Channel Blockers Thiazide μ NT Treatment Initiation Time Treatment Effect μ T
Computing Treatment Effects Electronic medical record data Selection bias Published randomized trials Adherence bias
Decision Maker Perspectives Patient Maximize expected quality adjusted life years (QALYs) Third-party Payer Minimize expected costs of treatment and health services Society Maximize a weighted combination of expected patient rewards for QALYs minus costs of treatment and health services
Societal Perspective Objective function includes rewards for quality adjusted life years (QALYs) and costs One-time Costs Follow-up Costs r( s, a ) t t R( s, a ) ( C t t S ( s ) C t CHD ( s )) t ( CF S ( s ) CF t CHD ( s )) t mc ST Weighted Benefit Statin Cost
Weighted Annual Benefit to the Patient Stroke Decrement Factor Medication Decrement Factor R( s, a ) R (1 d ( s ))(1 d ( s ))(1 d ( a )) S CHD ST t t 0 t t t Reward in Dollars, i.e. Willingness to Pay CHD Decrement Factor
Reward Parameters Systematic review of the literature via Pubmed Insurance claims data Pharmacy Redbook drug costs Cost Effectiveness Registry: https://research.tufts-nemc.org/cear4/home.aspx
Study Cohort Practice setting: Type 2 diabetes patients seen in 6 primary care sites at Mayo Clinic Rochester Sample definition: 663 patients with: Research authorization No prior hx: stroke-chd 10+ years of follow-up Patient Attribute Study Cohort Age, years 52.46 (8.83) Diagnosis, years 3.24 (5.33) % Female 39.67 Total Chol mgm% 216.27 (51.61) HDL mgm% 43.65 (11.58) LDL mgm% 126.98 (37.31) SBP mm Hg 139.11 (19.75) HbA1c 8.01 (2.38)
Treatment Effect Mean treatment effects for study cohort Costs based on 2010 Redbook Metabolic Factors Therapy SBP DBP Tot Chol HDL ACEI/ARB -3.72-5.48 Thiazide -4.97-3.73 β Blocker -4.64-4.17 CCBlocker -2.49-4.76 Statin -13.97 7.28 Fibrate -3.91 4.73
U.S. ATP III Guideline Diabetes is a CHD risk equivalent Source: Third report on the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III), NIH Publication No. 01-3670, 2001
U.S. JNC 7 Source: The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure, NIH Publication No. 03-5233, 2003
Life Years to Event (yrs.) Policy Evaluation Males 70.5 70 69.5 69 Australian Canadian European U.S. U.S. (ATPIII*) 68.5 68 67.5 67 66.5 No Treatment 0 5000 10000 15000 20000 25000 30000 35000 Cost ($)
Life Years to Event (yrs.) Optimal policy for varying willingness to pay MDP Optimal Tradeoff Curve Males 70.5 70 69.5 69 Australian Canadian European U.S. U.S. (ATPIII*) Maximum LYs 68.5 68 67.5 67 66.5 No Treatment 0 5000 10000 15000 20000 25000 30000 35000 Medication Costs ($)
Life Years to Event (yrs.) Optimal policy for varying willingness to pay MDP Optimal Tradeoff Curve Females 74.5 74 73.5 Canadian U.S. U.S. (ATPIII*) Australian European Maximum LYs 73 72.5 72 No Treatment 71.5 0 5000 10000 15000 20000 25000 30000 35000 Medication Costs ($)
Are Newer Drugs Better?
Are Newer Drugs Better? Men Women
Basic idea: TPM Sampling Method Random-direction algorithm 1 for sampling random vectors over convex region Sample each row of the TPM independently from intersection of uncertainty set, U, and standard simplex, Δ 1 : Smith, R.L, Efficient Monte Carlo procedures for generating points uniformly distributed over bounded regions, Operations Research, 32(6) p 1296-1308, 1984
Algorithm Choose initial point X 0 in the uncertainty set, U For j = 1,, W + M samples Sample d such that X j 1 + λd Δ Find λ and λ such that X j 1 + λd U Sample λ uniformly in interval [λ, λ ] While(X j 1 + λd U) if λ 0 then λ λ else λ λ Sample λ uniformly in interval [λ, λ ] End While X j X j 1 + λd; j j + 1 End For
Estimated expected QALYs Sensitivity Analysis TPM for Glycemic Control 64.44 64.42 64.4 64.38 64.36 64.34 27 27.5 28 28.5 29 29.5 30 Estimated expected total medication costs ($,in thousands) Working paper and Matlab code available for use upon request
Sensitivity Analysis Medication Disutility -0.06-0.05 0.05 0.06 HbA1c TPM -0.04-0.04 0.03 0.04 Medication Effect on HbA1c -0.01-0.01 0.01 0.01 Monthly Medication Cost 0.00 0.00-0.08-0.06-0.04-0.02 0.00 0.02 0.04 0.06 0.08 Absolute changes in the Expected QALYs (QALYs)
Conclusions Treating risk instead of risk factors has the potential for better health outcomes Low variation in optimal sequence of medication; optimal tradeoff differentiated by timing of treatment for men and women Treatment significantly influenced by individual risk factors
Acknowledgements Jennifer Mason, University of Virginia Lauren Steimle, University of Michigan Jim Wilson, NC State University Yuanhui Zhang, CDC Nilay Shah, Mayo Clinic Steven Smith, Mayo Clinic This work was supported by the National Science Foundation CMMI 1462060. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Thank You Slides posted on my website: http://umich.edu/~btdenton Steimle, L.N., Denton, B.T., Markov Decision Processes for Screening and Treatment of Chronic Diseases, Working Paper Brian Denton Industrial and Operations Engineering University of Michigan btdenton@umich.edu 36
Recent Work Mason, J., Denton, B.T., Shah, N., Smith, S., Optimizing the Simultaneous Management of Cholesterol and Blood Pressure Treatment Guidelines for Patients with Type 2 Diabetes, European Journal of Operational Research, 233, 727-738, 2013. Zhang, Y., McCoy, R.G., Mason, J., Smith, S.A., Shah, N., Denton, B.T., Second-line agents for glycemic control for type 2 diabetes: are newer agents better?, Diabetes Care, 37:5 1338-1345, 2014. Zhang, Y.,, Wu, H., Denton, B.T., Wilson, J.R., Lobo, J.M., Conducting Probabilistic Sensitivity Analysis for Markov Decision Processes, Working paper Zhang, Y., Denton, B.T., Robust Markov Decision Processes for Medical Treatment Decisions, Working Paper, 2015 (available at Optimization Online: http://www.optimization-online.org/db_html/2015/10/5134.html) Steimle, L.N., Denton, B.T., Markov Decision Processes for Screening and Treatment of Chronic Diseases, Working Paper