Lecture 5: Sequential Multiple Assignment Randomized Trials (SMARTs) for DTR. Donglin Zeng, Department of Biostatistics, University of North Carolina

Transcription

1 Lecture 5: Sequential Multiple Assignment Randomized Trials (SMARTs) for DTR

2 Introduction

3 Introduction Consider simple DTRs: D = (D 1,..., D K ) D k (H k ) = 1 or 1 (A k = { 1, 1}). That is, a fixed treatment is assigned to any patients at stage k. How shall we evaluate V(D)? Conduct a single-arm randomized trial. How shall we compare two and more different DTRs: D 1 vs D 2 (not necessarily simple ones)? Conduct a multiple-arm randomized trial.

4 Randomization Is Key Why randomization is needed? It provides representative coverage of the patient population. It controls all potential confounders which may bias the effects of comparing treatments. One essential mathematical relationship links observed outcome to potential outcome: E[R(d) H] = E[R(d) A = d, H] by randomization = E[R A = d, H] by consistency assumption. Consistency/SUTVA: R = R(d) if A = d and is not affected by the particular treatment assignments to the others.

5 Difficulty in Design Studies for Optimal DTRs Even for simple DTRs and only two treatment options at each stage, the total number of DTRs is 2 K. The optimal DTRs permit treatment decisions to depend on patient characteristics, H k, at each stage k so the total number of DTRs is infinite. Some smart design is needed.

6 Learn from Artificial Intelligence

7 What is Reinforcement Learning in AI Reinforcement learning (RL) is a machine learning method in artificial intelligence. Its history dates back to decades ago when AI engineers tried to mimic learning process in human brains. Different from commonly known machine learning, RL is a learning method of interactions between agent and unknown environment with feedback loop. It consists of exploitation and exploration. Exploitation: apply already learnt decisions Exploration: trial and learn from new environment.

8 Link between RL and DTR They both involve sequential decisions. They share the same goal: find optimal decisions to maximize some value. Equivalence between DTR and RL terminologies: treatment == action state/covariate==state reward outcome==reward treatment rule==policy

9 How RL Designs Studies in AI In artificial intelligence (AI), a single agent (Robot) trains and learns by itself for a particular task, for example, play chess to win. A typical design for this learning is a trial and error learning process. The single agent tries many different episodes, each with a given policy. It leans from these episodes and also updates policies in new episodes.

10 How Different RL Should Be Used in Medical Trials Instead of training one single agent with many policies, we train many patients, each given potentially different treatments. Randomization is necessary to have sufficient representation and avoid selection for treatments. This is essentially the idea of Sequential Multiple Assignment Randomized Trial (SMART).

11 SMART

12 SMART SMART: Sequential Multiple Assignment Randomized Trial (Lavori & Dawson 2000, 2004; Murphy 2005) Patients are sequentially randomized at each critical decision stage. Randomization probability may depend on current states of patients. Practical SMART Adaptive Pharmacological and Behavioral treatments for ADHD; Sequenced Treatment Alternatives to Relieve Depression (STAR*D) ; CATIE for schizophrenia; ExTENd for alcohol dependence; Adaptive therapy for androgen independent prostate cancer

13 Example: Two-Stage SMART Study The study (Kasari et al., 2014) was designed to study communication intervention for minimally verbal children with autism. The study aimed to test the effect of SGD, each stage lasting 12 weeks. SGD: speech-generating device; (JASP+EMT): blended developmental/behavioral intervention The second stage had another 12 week follow-up. The study started with 61 eligible children and 46 completed both stages.

14 was conducted by an independent data-coordinating center. Diagram of The Autism Study parent-report and observation during study adminis tration of the NLS. FIGURE 1 Participant flow through trial. Note: JASPþEMT ¼ spoken mode of JASPER plus Enhanced Milieu Teaching; JASPþEMTþSGD ¼ spoken mode of JASPER plus Enhanced Milieu Teaching plus Speech Generating Device. Figure: SMART Design of Autism Study (Kasari et al. 2014) JOURNAL OF THE AMERICAN ACADEMY OF CHILD & ADOLESCENT PSYCHIATRY VOLUME 53 NUMBER 6 JUNE

15 Advantages of SMART Research questions to be answered from a SMART Main effects of treatments What is the better initial treatment, JASP+EMT or JASP+EMT+SGD? What about the slow-responders: intensify or not? Effects of embedded DTR JASP+EMT intensify vs JASP+EMT+SGD intensify vs JASP+EM JASP+EMT+SGD Exploring optimal treatment strategy (deep tailoring) intensify or not in the second stage dependent on additional intermediate outcomes?

16 General Advantages Valid comparisons of different treatment options at different stages due to the virtue of randomization. Discover adaptive treatment strategies that are embedded in the SMART trial. Inform the development of adaptive and deeply tailored treatments (using potentially high-dimensional biomarkers).

17 Unbiased Value Estimation under SMART Recall the value function associated with D = (D 1,..., D K ) V(D) = E[R(D)] = E[R(D 1,..., D K )]. Let π k (a, h) be the randomization probability P(A k = a H k = h). Start from stage K: E[R(D 1,..., D k 1, D K )] { [ ]} = E E R(D 1,..., D K ) H K, A K = D K (H k ) { [ = E E R(D 1,..., D k 1, A K ) I(A K = D K (H K )) ]} H K π K (A K, H K ) [ = E R(D 1,..., D k 1, A K ) I(A ] K = D K (H K )). π K (A K, H K )

18 Continue to Prior Stages Now at stage K 1, we continue and repeat the same derivation to obtain E[R(D 1,..., D k 1, D K )] [ = E R(D 1,..., A K 1, A K ) I(A ] K 1 = D K 1 (H K 1 ), A K = D K (H K )). π K 1 (A K 1, H K 1 )π K (A K, H K ) Continue backwards till stage 1: V(D) = E [ R(A 1,..., A K 1, A K ) I(A 1 = D 1 (H 1 ), A K = D K (H K )) K k=1 π k(a k, H k ) [ ] = E R I(A 1 = D 1 (H 1 ), A K = D K (H K )) K k=1 π. k(a k, H k ) The value function for D can be estimated using the average reward outcomes from the patients whose treatments follow D, weighted by their randomization probabilities. ]

19 An Alternative Interpretation In SMART, a particular treatment assignment takes probability K π k (A k, H k ). k=1 If we run a single-arm trial for a given D, the treatment assignment is K k=1 I(A k = D k (H k )). To use SMART to estimate the expected reward in the D-specified trial, important sampling theory gives V(D) = E [ R I(A 1 = D 1 (H 1 ), A K = D K (H K )) K k=1 π k(a k, H k ) In conclusion, SMART provides unbiased estimation and comparisons between DTRs so potentially leads to valid estimation of optimal DTRs. ].

20 Demo of SMART Trial

21 NSCLC trial In treating advanced non-small cell lung cancer, patients typically experience two or more lines of treatment, and many studies demonstrate that three lines of treatment can improve survival for patients. 1st-line 2nd-line 3rd-line

22 1st-line therapy Scagliotti et al., 2008 { Cisplatin + Gemcitabine Cisplatin + Pemetrexed Sandler et al., 2006 { Paclitaxel + Carboplatin Paclitaxel + Carboplatin + Bevacizumab (Avastin) Pirker et al., 2008 { Cisplatin + Vinorelbine Cisplatin + Vinorelbine + Cetuximab (Erbitux)

23 2nd-line therapy Docetaxel Approved regimens Pemetrexed Erlotinib (approved for third-line, too) Timing issue Immediate or delayed docetaxel? (Fidias et al., 2007; Ciuleanu et al., 2008) Comparison of PFS suggests immediate, but OS did not reach significant difference. Merits of pemetrexed and erlotinib remain unclear.

24 Important clinical questions 1. Among many approved 1st-line treatments, what treatment to administer? 2. Then, at the end of the 1st-line treatment Among approved 2nd-line treatments, what treatment to administer? When to begin the 2nd-line of treatment? Immediate Progression Death 1st-line 2nd-line Possible treatments Possible treatments and initial timings

25 Reinforcement Learning Has a well-known framework for optimizing sequence of treatments in an evolving, time-varying system. Discovers individualized treatment regimens for cancer. Selects treatments that improve outcomes even when the relationship between treatments and outcomes is not fully known. Can evaluate treatments based on immediate and long-term effects. Applications: Behavioral disorders (Pineau et al., 2007). Sequential multiple assignment randomized trial (SMART) designs for developing dynamic treatment regimes (Murphy et al., 2007).

26 Clinical Reinforcement Trial Drafted Protocol 1. A finite, reasonably small set of decision times is identified. 2. For each decision time, a set of possible treatments to be randomized is identified. 3. A utility function is identified which can be assessed at each time point. 4. Patients are then recruited into the study and randomized to the treatment set under the protocol restrictions at each decision point. 5. The patient data is collected and Q-learning is applied, in combination with SVR applied, to estimate the optimal treatment rule as a function of patient variables and biomarkers, at each decision time.

27 Reinforcement Learning Trial Y D =0,δ =1,T 0 = T 1 t 1 T1 Y D =0,δ =0,T 0 = C t 1 C (t 2 + T M) T P T2 Y D =1,δ =1,T 0 = t 2 + T 2 t 1 t2 T 1 = T D t 2, Y D = I(T D C t 2 ) Y D =1,δ =0,T 0 = C (t 2 + T M) T P t 1 t2 T 2 = (T D t 2 )I(T D t 2 ) = (T D t 2 )I(T 1 = t 2 ) C 2 = (C t 2 )I(C t 2 ) T D = T 1 + T 2, T 0 = T D C = T 1 C + Y D (T 2 C 2 ) C

28 Learning Algorithm 1. At t 1, (w 1, m 1, d 1, T 1 C, ) n i=1 ; at t 2, (w 2, m 2, d 2, T M, T 2 C 2, ) n j=1, where n n. 2. Q 2, the expected reward at stage 2, is estimated using regression methods with outcome T 2 C Q 1, the expected optimal reward at stage 1, is estimated using regression methods with outcome T 1 C + I(T 1 = t 2 ) max d 2,T M Q In step 2 & 3, regression methods can include machine learning algorithms (SVR). 5. Given Q 1 (ˆθ 1 ) and Q 2 (ˆθ 2 ), compute D 1 and D 2 by maximizing Q 1 and Q 2, respectively.

29 Simulation Settings 1st-line treatment regimens: A 1 or A 2 2nd-line treatment regimens: A 3 or A 4 initiation time for 2nd-line treatment: random time between 2.8 and 4.8 months from 1st-line treatment prognostic factors: W t (quality of life), M t (tumor size) four groups are generated along with their optimal treatment regimens

30 Simulation scenario table Table 1: The scenarios studied in the simulation. Sample size = 100/group. Group State Variables Status Timing Optimal Regimen W 1 N(0.25, σ 2 ) 1 M 1 N(0.75, σ 2 ) W 1 M 1 A 1 A W 1 N(0.75, σ 2 ) M 1 N(0.75, σ 2 ) W 1 N(0.25, σ 2 ) M 1 N(0.25, σ 2 ) W 1 N(0.75, σ 2 ) M 1 N(0.25, σ 2 ) W 1 M 1 A 1 A 4 1 W 1 M 1 A 2 A 3 3 W 1 M 1 A 2 A 4 2

31 Performance of the optimal regimen A 1A 31 A 1A 32 A 1A 33 A 1A 41 A 1A 42 A 1A 43 A 2A 31 A 2A 32 A 2A 33 A 2A 41 A 2A 42 A 2A 43 optimal Overall Survival

32 Performance of the optimal regimen Table: Comparisons between true optimal regimens and estimated optimal regimens for overall survival (month). Each reinforcement trial is of size 100/group with 10 simulation runs. The confirmatory trial is of size 100/group. Optimal True Predicted survival Group regimen survival Min Mean Max 1 A 1 A A 1 A A 2 A A 2 A Average

33 A 1A 32 A 1A 41 A 2A 33 A 2A 42 Frequency Frequency Frequency Frequency Group 1 (92%, 100%) Group 2 (100%, 100%) Group 3 (100%, 100%) Group 4 (100%, 81%)

34 Sensitivity of the predicted survival to the sample size Overall Survival Sample Size for Each Group

35 ɛ-svr-c performance Predicted Overall Survival Predicted Overall Survival Predicted Overall Survival None 25% delete None 50% delete None 75% delete (a) (b) (c)

36 Summary Identified optimal treatment strategies tailored to proper subpopulation of NSCLC patients. Solved timing problem of initiating second-line therapy in NSCLC. Handled right censored data.

37 Design Limitations

38 Practical limitations of SMART Operation cost of administrating multiple stage studies and multiple treatments is high. The length of trial period is long (March et al. 2010). Study dropout or compliance is common even in regular RCTs: In the CATIE study, 705 of 1460 patients stayed for the entire 18 months of the study. In ExTENd, the drop-out rate was 17% (52 out of 302) in the first-stage treatment and an additional 13% (41 out of 302) during the second stage.