Lecture 10: Learning Optimal Personalized Treatment Rules Under Risk Constraint

Transcription

1 Lecture 10: Learning Optimal Personalized Treatment Rules Under Risk Constraint

2 Introduction

3 Consider Both Efficacy and Safety Outcomes Clinician: Complete picture of treatment decision making involves both efficacy and safety Most efficacious treatment for a patient may also lead to greater safety concern, e.g., escalating dosage of insulin may increase risk of hypoglycemia Patients with chronic disease (e.g., diabetes) and long duration treatment exposed to higher risk of adverse events (e.g., severe hypoglycemia) Regulator/Industry: Important for clinical drug development to characterize both the efficacy and risk profiles among patient populations (e.g., FDA guideline on evaluating cardiovascular risk for new antidiabetic therapies)

4 Consider Both Efficacy and Safety Outcomes in Personalized Medicine Goal: need to control average risk when estimating optimal treatment regimen to maximize efficacy. Challenge: when maximizing the benefit function, how should safety outcomes be incorporated? Presence of heterogeneity: Well known example: abundance of drug-metabolizing enzymes (cytochrome P45) varies across subjects, and thus adverse reactions to the same drug dosage Risk of hypoglycemic events depends on patient characteristics and choice of treatment regimen (Sinclair et al., 2015)

5 Learning Method

6 Framework for Personalized Benefit-Risk Analysis Notation: Y: efficacy outcome (e.g., symptom reduction; change in HbA1c) R: risk outcome (hypoglycemia episodes) Two treatment arms A { 1, 1} Subject-specific covariates X Treatment rule D(X): mapping from X to { 1, 1}.

7 Utility Function with Risk Constraint Goal: estimate optimal treatment rule D such that { maxd E D (Y), s.t. E D (R) τ, E D [ ]: conditional expectation under probability measure P D for (Y, R, A, X) given A = D(X) τ: pre-specified tolerance threshold of the risk

8 Theoretical Optimal Treatment Rules Under Risk Constraint Using data (Y, R, A, X) collected from RCT, equivalent to: { } E I(A=D(X)) Y, max D s.t. E { p(a X) I(A=D(X)) p(a X) } R τ. With some arithmetics and define D(X) = sign(f (X)), the above is equivalent to { maxf E {δ Y (X)I(f (X) > 0)}, s.t. E[δ R (X)I(f (X) > 0)] α, where δ Y (X) = E[Y X, A = 1] E[Y X, A = 1], δ R (X) = E[R X, A = 1] E[R X, A = 1], and α = τ E[R A = 1].

9 Theoretical Optimal Treatment Rules Under Risk Constraint Key result: The optimal treatment rule under risk constraint is D (X) = sign(f (X)), where { f sign(δy (X)), X A (X) = sign (δ Y (X) λ δ R (X)), X A c and A = {X : δ Y (X)δ R (X) 0}. Here, λ = 0 if E [ δ + R (X) X Ac] α ; otherwise, λ solves equation E [δ R (X)I{δ R (X) > 0, δ Y (X)/δ R (X) > λ} X A c ] +E [δ R (X)I{δ R (X) < 0, δ Y (X)/δ R (X) < λ} X A c ] = α, with α = α E[δ R(X)I(δ Y (X)>0,X A)] P(X A c ). Remark 1. Solving for D is analogous to finding the optimal rejection region at a given type I error rate as in the Neyman-Pearson lemma. Remark 2. When no treatment heterogeneity on safety outcomes, apply with δ R (X) = c.

10 Estimating Constrained Optimal Treatment Rules Proposed Method 1: BR-Q learning Regression-based learning algorithm (in line with Q-learning in the absence of R) Step 1. Fit regression model for Y given (A, X), obtain δ Y (X) = Ê[Y X, A = 1] Ê[Y X, A = 1] Step 2. Fit regression model for R given (A, X), obtain δ R (X) = Ê[R X, A = 1] Ê[R X, A = 1] Step 3. Apply key result: { sign( δy (X)), X f  (X) = sign ( δy (X) λ δ ) R (X), X Âc.

11 Estimating Constrained Optimal Treatment Rules Method 2: BR-O learning Directly estimate D under risk constraint without posing a regression model (in line with O-learning in the absence of R): max D s.t. E { I(A=D(X)) p(a X) E { I(A=D(X)) p(a X) R } Y, } τ. Maximizes empirical value function under constraint: n max f n 1 Y i P(A i X i ) I (A i = sign(f (X i ))), s.t. n 1 n i=1 i=1 R i P(A i X i ) I (A i = sign(f (X i ))) τ.

12 Implementation of BR-O learning Challenges: constrained optimization with non-convex objective function and non-convex constraint. Solution: approximate I (A i sign(f (X i ))) in objective function by a surrogate hinge loss, and approximate I (A i = sign(f (X i ))) in the constraint by a shifted ψ loss as upper bound ψ δ (u) = f 1 δ (u) f 0 δ (u) = δ 1 (u + δ) + δ 1 (u) +. Loss Shifted psi-loss 0-1 loss f1 f u

13 Implementation of BR-O learning The optimization solved by difference of convex functions algorithm (DCA) (Tao and An 1998) and quadratic programming: min f s.t. n R i i=1 C n i=1 Y p i ξ i βt (0) Kβ (0), [ p i δ 1 {A i f (X i ) + δ} + δ 1 ] {A i f (X i )} + nτ, ξ i 1 A { i β0 + n j=1 β jk(x i, X j ) }, ξ i 0 i. Tuning parameters C and δ selected by cross validation.

14 Numeric Results

15 Simulation Design 20 covariates as X 1,..., X 20 i.i.d. U(0, 1), n = 300, 100 replications Efficacy responses are continuous variables, Y = 1 2X 1 + X 2 X 3 + h Y (X, A) + ɛ h Y = 2 (1 X 1 X 2 ) A Safety responses are from Poisson distribution log(m R ) = 1 + X 1 2X 2 X 3 + h R (X, A) h R = (1 + X 1 X 2 ) A BR-Q: linear regression; BR-O: linear kernel

16 Theoretical Optimal Treatment Decision Boundaries Figure: Regions of Optimal Treatments with and without Risk Constraint and Relationship with Average Benefit and Risk (τ = 1): linear boundary

17 Simulation Results Figure: Average efficacy and safety outcome estimated by theoretical formula, BR-Q learning and BR-O learning as a function of pre-specified τ Safety Outcome Efficacy Outcome R Theoretical BR-Q BR-O Y Theoretical BR-Q BR-O τ τ : Optimal average Y without safety constraint = 0.662

18 Simulation Results Table: Estimated average risk and optimal benefit. Safety outcome R Efficacy outcome Y % Correct τ Theo BR-Q BR-O Theo BR-Q BR-O BR-Q BR-O : Average safety outcome is 1.503, and optimal value function without safety constraint is : Theo : computed from theoretical formula, BR-Q : Risk constrained Q-learning, and BR-O : Risk constrained O-learning.

19 Theoretical Optimal Treatment Decision Boundaries: Nonlinear Case Figure: Regions of Optimal Treatments with and without Risk Constraint and Relationship with Average Benefit and Risk (τ = 1.75): nonlinear boundary

20 Application

21 Application to DURABLE Trial DURAbility of Basal Versus Lispro Mix 75/25 Insulin Efficacy (DURABLE) Trial (Buse et al., 2009): Randomized trial to compare the ability of two starter insulin regimens (once-daily basal insulin Glargin or twice-daily premixed insulin Lispro 75/25) to achieve glycemic control in patients with type 2 diabetes 2,091 insulin-naive patients with type 2 diabetes who did not achieve adequate control with oral antihyper-glycemic drugs Efficacy outcome: glycemic control (change in HbA1C from baseline to end point) Safety outcomes: hypoglycemia (a plasma glucose value 70 mg/dl or presence of typically associated symptoms)

22 Application to DURABLE Trial Overall analyses results (Buse et al., 2009): Efficacy: Lispro 75/25 better (p = 0.005) control on glycemic than GL Safety: Lispro 75/25 higher (p = 0.007) hypoglycemia rate compared to GL (A): black Lispro, white GL; (D): filled hypoglycemia rate (triangle: Lispro, square: GL).

23 Application to DURABLE Trial Application Data Description: Sample size: 965 patients on Lispro Mix and 980 patients on insulin Glargin. Efficacy endpoint: A1c change from baseline after 24 weeks treatment. Safety endpoint: Hypoglycemic event rate per day. Covariates: 18 baseline covariates (weight, BMI, blood pressure, heart rate, 7 points blood glucose values, fasting blood glucose, fasting insulin etc.).

24 Application to DURABLE Trial Risk(τ) Method Risk-Training Risk-Testing Benefit-Training Benefit-Testing BR-Q (0.005) (0.004) (0.142) (0.049) BR-O (0.003) (0.006) (0.142) (0.042) BR-Q (0.006) (0.004) (0.141) (0.050) BR-O (0.003) (0.006) (0.135) (0.050) BR-Q (0.006) (0.004) (0.142) (0.050) BR-O (0.003) (0.006) (0.135) (0.051) BR-Q (0.006) (0.004) (0.146) (0.051) BR-O (0.003) (0.006) (0.135) (0.046) BR-Q (0.006) (0.004) (0.148) (0.052) BR-O (0.003) (0.006) (0.131) (0.048) BR-Q (0.01) (0.003) (0.153) (0.048) BR-O (0.01) (0.005) (0.146) (0.052) Conclusion: BR-O controls risk below τ with similar benefit as BR-Q

25 Ranks of Predictive Biomarkers τ = Fasting Blood Glucose Adiponectin Fasting Insulin Glucose:Morning before meal Glucose:Morning 2 hours after meal Glucose:Noon before meal Glucose:Noon 2 hours after meal Glucose:Evening before meal Glucose:Evening after meal Glucose: 3am at night Body Weight Height BMI Diastolic blood pressure Systolic blood pressure Heart rate Duration of diabetes Baseline A1c

26 Conclusion

27 Extension We propose two methods for estimating optimal individualized treatment rules while controlling for average risk. Extensions: Multiple efficacy and safety outcomes Multiple group-dependent thresholds Multi-stage trials (SMART, Lavori & Dawson 2000, 2004; Murphy 2005) Identifying safest individualized treatment while maintaining minimal benefit