Dan Lizotte, Lacey Gunter, Eric Laber, Susan Murphy Department of Statistics - University of Michigan

Transcription

1 REINFORCEMENT LEARNING FOR CLINICAL TRIAL DATA Dan Lizotte, Lacey Gunter, Eric Laber, Susan Murphy Department of Statistics - University of Michigan

2 it s not what you think I say: Reinforcement Learning You think: Ah yes, Reinforcement Learning... a 1 0 a 1 a 1 a a 0 a 1.5 a 1 No. a 0 2 a 0 4 a 0 Well yes, but your pre-conceived notions may lead you astray Grid-world and mountain car intuitions may not apply

3 a different picture F0 F1 F2... FN F0 F1 F2... FN A0 G0 G1... GM F0 F1 F2... FN A0 G0 A0=1 A0=5 F0 F1 F2... FN A0 G0 G1... GM A1 H1 H2... HK F0 F1 F2... FN A0 G0 G1... GM A1 H1 H2... HK A2 J1 J2... A1=B A2=Δ

4 unusual things about this setting More (different?) generality than is assumed in toy problems State includes past history: observations, actions and rewards s k S = s k (o 0,a 0,r 0,o 1,a 1,r 1,..., o k ) The set of permissible actions depends on the current state A = A(s)

5 familiar things about this setting Our objective is to maximize the expected sum of future rewards We will accomplish this by learning and using a Q function, computing expectations and maximizations appropriately Q (s, a) =E s [r(s, a, s ) + max a Q (s,a )] We then recover the optimal policy from Q* This is off-policy: Data were collected using a random policy.

6 the data: STAR*D Sequenced Treatment Alternatives to Relieve Depression. NIMHfunded study involving ~4000 patients Intended to discover effective, patient-tailored treatment strategies (i.e. a good policy) for treating clinical depression STAR*D has many layers of complexity Actions: Treatments for depression: Drugs, cognitive therapy Observations: Demographic information, depression assessment,... Rewards: Unclear. We must combine data about therapeutic benefits, side-effects, costs, and unforeseen things

7 STAR*D terminology Level A period of time over which a patient adheres (theoretically) to one of a Level-specific set of possible treatments There are 4(-ish) Levels in STAR*D that a patient may participate in, in sequence. Hence maximum depth is 4(-ish) Observations are made at intervals during each level Remission Different measures of symptoms and side-effects If a patient achieves a sufficiently good score on a measure of clinical depression (QIDS), he or she may go to follow-up Otherwise, the patient is supposed to proceed to the next Level

8 more STAR*D terminology Treatment Preference: When moving from one Level to the next, a patient provides a treatment preference Augment : Patient prefers to add a drug/therapy to the current drug/therapy the patient is receiving Switch : Patient prefers to discontinue the current treatment, and substitute a new one These state variables define the possible action sets When transitioning to a new level (for whatever reason): Patient is randomized among the set of treatments that are consistent with a patient s preference

9 Level 1 Max 12 Weeks CIT QIDS! 5 QIDS > 5 Follow-up Level 2 Max 12 Weeks SER BUP VEN CIT+BUS CIT+BUP Preference to Switch QIDS > 5 Preference to Augment QIDS! 5 Follow-up Level 3 Max 12 Weeks MIRT NTP L2+Li L2+THY Preference to Switch QIDS > 5 Preference to Augment QIDS! 5 Follow-up Level 4 Mandatory 12 Weeks TCP MIRT+VEN QIDS! 5 Follow-up

10 Example STAR*D Patients Starting at Level Patient Level Level 3 Level QIDS Weeks

11 weird things about this setting More (different?) generality than is usually assumed in an MDP State includes past history: observations, actions and rewards s k S = s k (o 0,a 0,r 0,o 1,a 1,r 1,..., o k ) The set of permissible actions depends on the current state A = A(s)

12 weird things about this setting More (different?) generality than is usually assumed in an MDP State includes past history: observations, actions and rewards s k S = s k (o 0,a 0,r 0,o 1,a 1,r 1,..., o k ) Which treatment did the patient receive previously? How well did they do on that treatment? The set of permissible actions depends on the current state A = A(s) Did the patient consent to augment or switch treatment? The various o and a may have different domains

13 familiar things about this setting Our objective is to maximize expected sum of future rewards We will accomplish this by learning and using a Q function, computing expectations and maximizations appropriately Q (s, a) =E s [r(s, a, s ) + max a Q (s,a )] We then recover the optimal policy from Q* This is off-policy: Data were collected using a random policy.

14 familiar things about this setting Our objective is to maximize expected sum of future rewards Reward will be some sort of cost-benefit analysis of treatment We will accomplish this by learning and using a Q function, computing expectations and maximizations appropriately Q (s, a) =E s [r(s, a, s ) + max Q (s,a )] a Short horizon means we can compute Q* exactly (except for approximation error) Q* at the end states depends only on r... We then recover the optimal policy from Q* This is off-policy: Data were collected using a random policy.

15 this domain is easy because The domain is episodic Horizons are short (i.e. 2, 3, 4...) Q-learning type analysis can be done exactly i.e. online methods (in the stochastic gradient sense) are unnecessary

16 this domain is hard because We only have 1292 interesting trajectories There are dozens (hundreds?) of interesting features The types of observations are unusual for RL e.g. variable-length vectors, where the length is meaningful Many observations are missing and/or lies We need bias that is both useful and defensible How sensitive are analyses w.r.t. bias? By bias I mean everything that goes into choosing a class of Q functions, including variable selection, choice of regressor, etc.

17 the truth about STAR*D STAR*D is a remarkable resource STAR*D is a mess The action sets are complicated There is an enormous amount of missing data Some scheduled appointments just don t happen Only limited data were collected during follow-up Many patients just disappear Side effects, didn t like the doctor, felt better,...

18 this domain is dangerous because Opportunities for attribution bias are absolutely rampant Suppose: Reward = negative QIDS score at end of study We will not use any data from patients who disappeared There are missing state and reward variables After analysis, exotic-drug-x looks really great Exotic-drug-X works really well!...or exotic-drug-x has horrendous side effects in patients with severe depression, so that group disappears Eliminating incomplete cases induces bias

19 types of missing data MCAR - Missing Completely At Random Missingness does not depend on any of the data values M_ OF TH E _ATA ARE _IS_IN_ MAR - Missing At Random Missingness can depend on the observed data SO_E O_ THESE DA_A A_E MISSI_G If the data are not MCAR, then assuming the missing data are i.i.d. (or equivalently, throwing out missing data) induces bias However, if we use a richer model, we can correct for this. A bigraph model will give correct results in this case Letters 25% Missing Letters after Vowels 25% Missing

20 types of missing data NMAR - Not Missing At Random Missingness depends on the missing value S_ME _F TH_S_ D_T R_ MISS_NG Okay so now we re hosed. Vowels 75% Missing Another feature could fix this problem - like This letter is a vowel S_ME _F TH_S_ D_T R_ MISS_NG This this would make the data MAR Side information could at least help, like These words are English A richer model could provide us with almost the same benefit

21 dealing with missing data When data are not MCAR or MAR, you can t tell. The better our model can predict the missing values, the more bias we will remove. Conventional wisdom is to use as rich a model as is feasible to model the values of the missing portion of our data given the observed portion. When we really don t know about how variables relate, the General Location Model is a popular choice.

22 the General Location Model Basically an LDA model Don t know Linear Discriminant Analysis? I will explain. Each exemplar is a vector X = [W1, W2,... Wp, Z1, Z2,... Zk] W are discrete, Z are continuous Each possible configuration of Ws is given a probability Define W to be a single discrete variable taking a value for each Z are multivariate normal with mean μ(w) and shared Σ Marginal distribution of W is multinomial Marginal distribution of Z is a mixture of MVNs

23 the General Location Model cartoon X = [ W, Z1, Z2 ] W { 1, 2, 3 } P(W) = [ ] μ(1) = [ ] μ(2) = [ ] μ(3) = [ ] Σ = [ ] [ ]

24 why are we doing this again? We want to model the missing data given the observed data. No problem! Given any pattern of missingness, we can compute the distribution of the missing data given the observed data...if we knew the parameters.

25 why are we doing this again? But we don t know the parameters Curses! We ll learn them. Great! We can t; we have missing data. Curses! Those of you who are in-the-know realize what I m about to do...

26 building the model We will run MCMC. 0) Make up an initial set of parameters along with priors on them 1)Sample missing data given observed data and parameters 2)Sample parameters given the complete data you just made up 3) Repeat What do we get? A bunch of parameter sets, and a bunch of imputed data sets These data sets have the missing bits drawn from mis obs, with the parameters integrated out --- it s like *magic*!

27 why all this trouble? We want to use methods, like Q-learning, that rely on complete input data. We can t just throw out the incomplete exemplars; we know the data are not MCAR. Instead, we do Bayesian multiple imputation, get m imputed data sets, and run whatever we want on each one. How do we combine results? Right now we average them to estimate the expected Q function, where the expectation is over the missing data. We might want the MAP Q-function; this is future work

28 back to STAR*D 150 variables Many are missing; we *know* the data are not MCAR Compare: Complete Case Analysis (a.k.a. Throw out incomplete exemplars ) Bayesian multiple imputation Assess confidence using the bootstrap Repeat 1000 times: Draw N exemplars from the training set with replacement For each bootstrapped dataset, learn your Q function, look at the variability among these 1000 Q functions

29 1000 ˆQ f rom CCA ˆQ using C C A f or L e ve l 2 Sw itch Tre atments Voting re sults f rom b ootstrapp e d datase ts SER VEN BUP Vote s Value Initial QIDS Sc ore 0

30 1000 ˆQ ob s ˆQ ob s f or L e ve l 2 Sw itch Tre atments Voting re sults f rom b ootstrapp e d datase ts SER VEN BUP Vote s Value Initial QIDS Sc ore 0

31 1000 ˆQ f rom CCA ˆQ using C C A f or L e ve l 2 Sw itch Tre atm e nts Voting re sults f rom b ootstrapp e d datase ts SER VEN BUP ˆQ ob s ˆQ ob s f or L e ve l 2 Sw itch Tre atments Voting re sults f rom b ootstrapp e d datase ts SER VEN BUP Vote s Value Vote s Value Initial QIDS Sc ore Initial QIDS Sc ore 0 CCA analysis is confident, and wrong. Bayesian multiple imputation still provides information but better reflects uncertainty Clinical significance? We re not sure yet.

32 summary We re doing RL But not your garden-variety RL Easy because of the short-horizon, batch scenario Hard because of: Limited data Complicated data Missing data and attribution bias Ambiguous rewards We are also looking at CATIE, a schizophrenia study

33 Thanks to John Rush and the STAR*D team Lacey Gunter Eric Laber Susan Murphy See also: J. Pineau, M.G. Bellemare, A. J. Rush, A. Ghizaru, S.A. Murphy (2007). Constructing evidence-based treatment strategies using methods from computer science. Drug and Alcohol Dependence, 88, Supplement 2:S52-S60. S.A. Murphy, L.M. Collins, A.J. Rush (2007). Customizing Treatment to the Patient: Adaptive Treatment Strategies (Editorial). Drug and Alcohol Dependence, Drug and Alcohol Dependence. 2007;88(2):S1-S72.