Automatic learning of adaptive treatment strategies. using kernel-based reinforcement learning

Transcription

1 using kernel-based reinforcement learning Presented by:, School of Computer Science, McGill University Joint work with: Marc G. Bellemare, Adrian Ghizaru, Zaid Zawaideh, Susan Murphy (Univ. Michigan), John Rush (UT Southwestern Medical Center) Biostatistics Seminar, McGill University March 7, 2007

2 Outline Background on adaptive treatment design Reinforcement learning primer Automatic construction of treatment strategies Preliminary results from the STAR*D trial Discussion

3 Adaptive treatment strategies Individually tailored treatments, where treatment type, dosage and duration varies according to patient outcomes. Goal is to improve longer-term outcomes, as opposed to focusing on only short-term benefits, for patients with chronic disorders. Recently developed for treatment of chronic illnesses, e.g.» Brooner et al. (2002) Treatment of Cocaine Addiction» Breslin et al. (1999) Treatment of Alcohol Addiction» Prokaska et al. (2001) Treatment of Tobacco Addiction» Rush et al. (2003) Treatment of Depression

4 Example of an adaptive treatment strategy Following graduation from an intensive outpatient program, alcohol dependent patients are provided naltrexone (Treatment 1). In the ensuing two months patients are monitored. If during that time the patient has 5 or more heavy drinking days (Nonresponder), then the medication is switched to acamprosate (Treatment 2a). If the patient reports no more than 4 heavy drinking days (Responder), then the patient is continued on naltrexone and monitored monthly for signs of relapse (Treatment 2b).

5 Why use adaptive treatment strategies? In Prevention Individuals at risk of problem behaviors for different reasons (multiple causes). Periods of high and low risk. In Treatment High heterogeneity in response to any one treatment.» What works for one person may not work for another. Improvement often marred by relapse.» What works now for a person may not work later. Co-occurring disorders may be common.

6 The big questions What is the best sequencing of treatments? What information do we use to make these decisions?

7 SMART Trials SMART = Sequential Multiple Assignment Randomized Trial [Murphy, 2005] These are multi-stage trials; conceptually a randomization takes place at each stage. Goal is to inform the construction of an adaptive treatment strategy.

8 Example of a SMART study Legend: MedA = Naltrexone MedB = Acamprosate CBT= Cognitive Behavioral Therapy TDM = Telephone Disease Management EM = Motivation therapy for adherence

9 Question Why not use data from multiple trials to construct the dynamic treatment regime? Choose best initial treatment on the basis of a randomized trial of initial treatments. Choose the best secondary treatment on the basis of a randomized trial of secondary treatments. Etc.

10 Negative synergies: Delayed effects Initial treatment may produce higher proportion of responders, but also result in side effects that are sufficiently high so that nonresponders are less likely to adhere to subsequent treatments. Positive synergies: A treatment may not be best initially but may have enhanced long term effectiveness when followed by a particular maintenance treatment. Diagnostic effects: Initial treatment may elicit informative patient response that permits better choice of subsequent treatment.

11 Examples of SMART designs: Thall et al. (2000) Treatment of Prostate Cancer CATIE (2001) Treatment of Psychosis in Schizophrenia STAR*D (2003) Treatment of Depression Oslin (on-going) Treatment of Alcohol Dependence

12 STAR*D STAR*D = Sequenced Treatment Alternatives to Relieve Depression Largest study of depression (14 regional centers, 4041 patients). Longitudinal (5 years), NIMH-sponsored. Multiple patient randomizations (up to 4 treatments: medication or psychotherapy). Entrance evaluation First treatment First evaluation Fourth treatment Fourth evaluation

13 STAR*D Randomizations (slightly simplified!) Level 1 Treatment CIT Level 2 Treatment options SER BUP VEN CT CIT+BUP CIT+BUS CIT+CT Treatment strategy Switch Augment Level 3 Treatment options MRT NTP SER+Li BUP+Li VEN+Li CIT+Li Treatment strategy Switch Augment Level 4 Treatment options TCP VEN+MRT Treatment strategy Switch

14 Clinical data Each evaluation measures 20+ variables, e.g. Quick inventory of depressive symptoms (QIDS) Frequency and intensity of side-effects Work and productive activity impairment Quality of life enjoyment and satisfaction Medication compliance Research outcomes measured (during clinic visit or telephone interview) at: beginning of treatment level, every ~6 weeks during treatment, exit from treatment level, every month during follow-up. Remission = QIDS at end of treatment level falls below threshold (QIDS 5).

15 Research objectives for computer scientists Long-term: Develop new methodologies for automatic learning of adaptive treatment strategies. Current project: Automatically construct decision rules from the STAR*D trial dataset using reinforcement learning techniques.

16 Decision rules Adaptive treatment strategies are composed of a series of decision rules, one per treatment step. Decision rule: IF baseline assessments = Z 0 AND prior treatments at Steps 1 through t = {A 1,, A t } AND clinical assessments at Steps 1 through t = {Z 1, Z t } THEN at Step t+1 apply treatment A t+1 Goal is to learn such rules directly from data.

17 An adaptive treatment strategy in STAR*D Patient is diagnosed with depression. Treat with citalopram. If the patient s depression remits and medication is tolerated, then retain patient on citalopram. If the patient does not remit, or cannot tolerate citalopram, then ascertain patient s preference for a medication switch treatment augmentation. If the patient prefers a switch in treatment then provide sertraline. If the patient prefers an augmentation and the patient was poorly adherent to citalopram then provide cognitive therapy in addition to citalopram. If the patient prefers an augmentation and the patient was adherent to citalopram, then provide bupropion in addition to citalopram.

18 Learning decision rules Specify set of candidate decision rules. For each decision rule, evaluate: its expected reward (or cost), its expected effect. Select sequence of decision rules with highest expected cumulative reward (or lowest cost). Formal framework for such problem: Reinforcement learning.

19 What is reinforcement learning (RL)? Inspired by trial-and-error learning studied in the psychology of animal learning: good actions are positively reinforced, poor actions are negatively reinforced. Formalized in computer science and operations research [Bellman, 1957; Sutton&Barto, 1998].

20 What is reinforcement learning (RL)? (cont d) Mathematical framework to estimate usefulness of taking sequences of actions in an evolving, time varying, system. Intuition: A decision-maker tries (random) actions and observes their effect until it gradually learns when to apply which action. Used to evaluate treatment based on immediate and long-term effects. Can evaluate both therapeutic and diagnostic effects.

21 A reinforcement learning problem is described by: States: What we know History of measurements and medications Actions: What we can do Treatment choice: CIT, CIT+CT, BUP, psychotherapy, Effects: How the action affects the state E.g. QIDS t=1 = 20 + A=CIT QIDS t=2 =15 Rewards: Time-dependent outcomes Duration of treatment, final outcome (remission, no remission, left trial), side effect burden,

22 A graphical model of the STAR*D trial a 1 =CIT a 2 =BUP s 0 Pretreatment data: Demographics Screening assessments Baseline assessments Baselinee IVRs s 1 Level 1 data: Treatment preference Assigned treatment Clinical assessments IVR assessments Duration e s 2 Level 2 data: Treatment preference Assigned treatment Clinical assessments IVR assessments Duration e r 0 r 1 r 2 r2 where s = state a = action e = effect r = reward

23 Reinforcement learning objective To learn the best treatment strategy for each state. best = the one that maximizes a given reward function. The reward function r(s) is a numerical indicator of whether a state is good or bad, capturing time-varying outcomes, e.g.: Depression score (e.g. QIDS) [Lower score = higher reward] Severity of side-effects [Lower score = higher reward] Length of treatment [Shorter length = higher reward]

24 Reinforcement learning in STAR*D a 1 =CIT a 2 =?? s 0 Pretreatment data: Demographics Screening assessments Baseline assessments Baselinee IVRs s 1 Level 1 data: Treatment preference Assigned treatment Clinical assessments IVR assessments Duration e s 2 Level 2 data: Treatment preference Assigned treatment Clinical assessments IVR assessments Duration e r 0 r 1 r 2 r2 Q: Which treatment a 2 has highest total (immediate + future) reward?

25 Maximizing the reward function To maximize the expected total reward: E [ r (s 0 ) + r (s 1 ) + r (s 2 ) + r (s 3 ) + r (s 4 ) ] We define the value function: V t (s) = max a [ r (s,a,s ) + s P a ss V t+1 (s ) ] where P a ss captures the effect of treatment on state. If we know P a ss, r(s,a,s ), then we can evaluate by dynamic programming. Problem: we lack a model to estimate P a ss!

26 Kernel-based Reinforcement Learning Consider value function: V t (s) = max a s P a ss [ r(s,a,s ) + V t+1 (s ) ] Approximate using kernel-based averaging: V t (s i ) = max a <s,a,s > w a (s i, s) [ r(s,a,s ) + V t+1 (s ) ] where <s, a, s > is a given data instance and w a (s i, s j ) = φ( s i -s j /b) is a weight kernel <s,a,s > φ( s i -s /b) Provides a natural way of incorporating measures of patient similarity.

27 Specifics of kernel regression We assume a Gaussian kernel: φ( s i -s j /b) = exp(-d 2 (s i,s j )/2σ 2 ). We allocate one kernel function per training example <s, a, s >. The width of the kernel σ 2 controls the neighbourhood of influence of each training sample. σ 2 =0 means no generalization to unseen states. σ 2 = means every state has (identical) mean value. The distance function d 2 (s i,s j ) measures similarity between patients via their values on the independent variables.

28 Bias-variance trade-off Width of the kernel, σ 2, controls the neighbourhood of influence of each training sample. Small bandwidth = more variance. Large bandwidth = more bias. Bandwidth should be chosen to minimize bias-variance trade-off. Need to shrink bandwidth as sample size increases. With the right shrinking-rate, we can guarantee: V t - V t * P 0 as m

29 Instance-based Reinforcement Learning Estimate non-parametric value function using kernel regression: V t (s) = max a s w a ss V t+1 (s ) Patient n Pretreatment data CIT Level 1 data consider all acceptable treatments s sweeps over patient database Pretreatment data w ss a weight CIT proportional to similarity Level 1 data of measurements Patient n Pretreatment data CIT Level 1 data BUP V t+1 V t+1 Patient n Pretreatment data CIT Level 1 data VEN... Patient 1 Patient 2 Patient 3 Patient 4... Pretreatment data CIT Level 1 data BUP Level 2 data Pretreatment data CIT Level 1 data BUP Level 2 data BUP+LI Level 3 data Pretreatment data CIT Level 1 data VEN Level 2 data Patient n Pretreatment data CIT Level 1 data CIT+BUP Patient N Pretreatment data CIT Level 1 data SER Level 2 data NTP Level 3 data

30 Caveat #1: Feature selection State representation at level i: s i = pretreatment data + treatment 1 + measurements at level treatment i + measurements at level i Total state representation: dozens of variables, multi-valued (up to 60+ values in some cases) Urgent need for feature selection! But Data is extremely sparse. Feature selection in decision-making is an open research question. Currently, select features by hand based on discussion with clinicians.

31 Caveat #2: Reward elicitation Reward function should be provided by clinicians. How do we elicit this information from them? Reward needs to have sufficient discriminative information. Reward is an abstract concept for a clinician. Clinicians may not be consistent in reward assessment. Results presented today assume hand-crafted reward function. Ongoing work on decision-theoretic method for eliciting preferences [Zawaideh & Pineau, submitted].

32 Current implementation State variables: Pre-treatment depression score QIDS-C 0 ={1,.., 25} Switch/Augment preference during levels 1 t. Gradient over depression scores during levels 1 t. Treatments at levels 1 t. Decisions: Level 2 and 3 treatments Reward function: r = 1 if QIDS-C 5 r = 0 otherwise

33 Level 2 treatment - Switch group Level 1 QIDS-C Entrance QIDS-C

34 Level 2 treatment - Augment group

35 Level 2 myopic treatment - Augment group

36 Value function at level 2 Recall value function: V t (s) = max a s w a ss V t+1 (s ) QIDS at exit from level 1 QIDS at exit from level 0

37 Estimated Level 3 value for patients with pretreatment QIDS-C = 15 and BUP as Level 2 treatment QIDS Level at exit 2 QIDS-C from level 2 QIDS Level at exit 1 from QIDS-C level 1

38 Estimated Level 3 value for patients with pretreatment QIDS-C = 15 and SER as Level 2 treatment QIDS Level at exit 2 QIDS-C from level 2 QIDS Level at exit 1 from QIDS-C level 1

39 Summary Adaptive treatment strategies can be constructed using: data from SMART studies reinforcement learning analysis. The analysis highlights the role of treatment as a diagnostic tool. The analysis presented is generating new hypothesis regarding the treatment of depression.

40 Clinical challenges Establishing close level of collaboration between clinical researchers and methodologists. Designing confirmatory trial for learned adaptive strategies. Performing preference (reward) elicitation studies.

41 Methodological challenges Extending RL methods to provide confidence measures. Using RL with data from observational studies. Improving problem design:» variables that are predictive of costs/rewards,» variables that interact with treatment.

42 Related ongoing project Learning adaptive treatment strategies for epilepsy (joint work with Massimo Avoli, MNI) Similar RL framework, different function approximation (not kernel RL). Gather data from generative model of epilepsy (rather than SMART trial). Faster treatment cycle, strategy can adapt to patient over time.

43 Follow-up Recent paper (proofs available by Pineau, Ghizaru, Bellemare, Rush & Murphy. ``Improving the management of chronic disorders by learning adaptive treatment strategies". Drug and Alcohol Dependence, Supplemental Issue on Adaptive Treatment Strategies. Elsevier. In press. Grad student position available for this project. Acknowledgements: Data generously provided by the STAR*D team. Partial funding provided by NIH (grant R21 DA019800). Some slide material borrowed from: