Factorial HMMs with Collapsed Gibbs Sampling for Optimizing Long-term HIV Therapy

Transcription

1 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy Amit Gruber 1,, Chen Yanover 1, Tal El-Hay 1, Aners Sönnerborg 2 Vanni Borghi 3, Francesca Incarona 4, Yaara Golschmit 1 1 IBM Research 2 Karolinska Institute 3 Moena University Hospital 4 EuResist Network GEIE Haifa Research Lab Karolinska University Hospital InformaPro S.r.l. Abstract Combine antiretroviral therapies (CART) can successfully suppress HIV in the serum an bring its viral loa below etection rate. However, rug resistance remains a major challenge. As resistance patterns vary between patients, personalize therapy is require. Automatic systems for therapy personalization exist an were shown to better preict therapy outcome than HIV experts in some settings. However, these systems focus only on selecting the therapy most likely to suppress the virus for several weeks, a choice that may be suboptimal over the longer term ue to evolution of rug resistance. We present a novel generative moel for HIV rug resistance evolution. This moel is base on factorial HMMs, applying a novel collapse Gibbs Sampling algorithm for approximate learning. Using the suggeste moel, we obtain better therapy outcome preictions than existing methos an recommen therapies that may be more effective over the long term. We emonstrate our results using simulate ata an using real ata from the EuResist ataset. 1 Introuction Much progress has been mae in recent years in treating HIV an moern Combine Antiretroviral Therapies (CART) can successfully suppress the virus an prolong patients life. These therapies consist of multiple rugs, typically 2-3, that target ifferent stages of the HIV Proceeings of the 21 st International Conference on Artificial Intelligence an Statistics (AISTATS) 2018, Lanzarote, Spain. PMLR: Volume 84. Copyright 2018 by the author(s). reprouction mechanism. When taken properly, these rugs can suppress the virus an reuce its level in the bloo below etection level. However, the virus is suppresse but not eraicate from the patient s boy, an it still resies in viral reservoirs Pierson et al. [2000], also calle latent reservoirs. These are cells infecte by the virus but not actively proucing it, an they are not affecte by rugs targeting the virus replication process. These cells contain integrate HIV-1 DNA. When they become active, they replicate the virus, contributing to its persistence in the boy espite it being suppresse in the bloo. If the therapy is stoppe, or if the virus evelops resistance to the rug, its level in the bloo will rise again. When HIV replicates, new mutations are ranomly create. Some of these mutations may show rug resistance, an prevail uner therapy. HIV may acquire rug resistance through such mutations over time, an moreover, an initial infection by a rug resistant strain can also be transmitte from another person cfemia et al. [2015], Van e Vijver et al. [2006]. Due to the prevalence of rug resistant mutations, care must be taken when selecting a therapy. As resistance patterns vary between ifferent patients, therapy selection nee to be personalize. Such personalization is one by following guielines AIDSInfo [2016], an recently also using genotypic resistance tests. Due to the ifficulty an importance of selecting an effective therapy, machine learning ecision support systems for therapy selection have been propose. Rosen- Zvi et al. [2008] an Prosperi et al. [2010] use the EuResist ataset 1 to train a system combining 3 preiction engines. Each of these engines applies logistic regression on a set of clinical an emographic features to preict the probability of success for a given treatment. A stuy by Zazzi et al. [2011] shows that this system is better at preicting therapy outcome than human HIV experts. After ranom forests were Author is currently with Facebook Inc. 1

2 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy shown to outperform SVM an artificial neural networks in Wang et al. [2009], Revell et al. [2014] use the HIV-RDI ataset 2 to train ranom forest using clinical features to preict therapy outcome. They showe that preiction accuracy is not harme when genotype information is not available. A similar result was publishe by Prosperi et al. [2010]. While these systems are successful in optimizing therapy outcome in a range of several weeks (8-24 weeks), their performance significantly eteriorates when preicting therapy outcome in a longer range (48 weeks). Further, these systems o not consier the effect of the selecte therapy on the evolution of future rug resistance an may recommen therapies that are suboptimal in a longer term. In a recent stuy, Prosperi et al. [2016] stuy therapy time to failure. Aiming to optimize treatments outcome in the long term, in this paper we suggest a novel generative moel for rug resistance evolution. With the increasing availability of electronic health recors (EHR), graphical moels have been suggeste to moel isease progression: Krishnan et el. [2016] moel isease progression uner therapy using an HMM (Figure 1a). The hien states in this HMM correspon to the patient s health status an the observations are clinical measurements inclue in the EHR. Patient s health is affecte by the known therapy the patient receives, whose effect is noticeable after some perio of time. El-Hay et al. [2014] propose structure proportional jump process for non-homogenous ata, in which the moel factorizes into an element that epens on time an an element that epens on system configuration. They apply their moel to iabetes an HIV ata. Arora an Dixit [2009] use a HMM to moel a certain aspect of HIV isease progression. Specifically, they moel the evolution of rug resistance against a single specific rug. We follow a similar approach an moel rug resistance evolution in HIV patients using an HMM. Unlike Arora an Dixit [2009], we consier CART therapies consisting of multiple rugs. Using a stanar HMM to moel resistance against multiple rugs woul require an exponentially large state space to represent all the combinations of possible virus resistance against each one of these rugs. Further, as therapy may be altere over time, such an HMM woul nee to represent the possible virus resistance against each one of the rugs in the arsenal. Instea of using a stanar HMM, we use a factorial HMM Ghahramani et al. [1997] consisting of multiple chains interconnecte through the observation (Figure 1b). In our case, each chain correspons to a possible resistance with respect to a specific rug an 2 its evolution over time, an the observation connecting the chains is the therapy outcome. Using this factorize representation, we avoi the exponential state space representation. The main contributions of this paper are: (1) A novel graphical moel for moeling isease progression uner combine rug therapy. Specifically, we apply this moel to the evolution of rug resistance in HIV patients taking CART (therapy) an moel patient s aherence to the therapy. (2) We use this graphical moel to recommen a series of treatments that are effective over the long-term. (3) A novel collapse Gibbs Sampling algorithm for Factorial HMM in general an for the suggeste moel specifically, ifferent from the previously suggeste approximate learning approaches incluing Gibbs sampling (non-collapse). 2 FResist: A Generative Moel for Drug Resistance Evolution HIV is typically treate with combine antiretroviral therapy (CART): a combination of multiple (usually 2-3) compouns given in a single pill that nees to be taken aily by the patient. If taken properly, an if the virus is sensitive to at least one of the compouns in the CART, the therapy may succee an prevent the virus from multiplying (but not eraicate it from reservoirs). If the virus is resistant to all the compouns in the CART, the therapy is expecte to fail. We suggest a novel generative moel for the evolution of HIV rug resistance uner CART. This moel, which we call FResist, is a special case of factorial HMM where each of its K chains represents the sensitivity or resistance of the virus to a specific compoun k an its possible evolution. We first escribe the components of a single chain an the relation between them. Then we escribe the interchain relation to the treatment outcome an provie a full generative process. In Section 2.1 we exten the moel to account for patient aherence. HIV acquires rug resistance through mutations. When the virus replicates, mutations are ranomly create. When no treatment is taken by the patient, the wil type, which is avantageous to other mutations, outgrows an ominates other mutants. But in the presence of rugs, the wil type is suppresse an rug resistant mutations prevail Clavel an Hance [2004]. nce rug resistant mutants are present in a significant amount in the bloo stream, they may form latent reservoirs - collections of immune cells infecte by the virus. These latent reservoirs are not affecte by anti-hiv rugs as these only target the virus replication process Pierson et al. [2000]. nce rug resistant mutants are present in the reservoirs, this may be viewe as acquisition of permanent resistance. We begin with a etaile

3 A. Gruber, C. Yanover, T. El-Hay, A. Sönnerborg, V. Borghi, F. Incarona, Y. Golschmit s s s t-1 t s s s o o o s s s s s s o o o S S K (a) (b) (c) Figure 1: a. Markov moel of a patient uner therapy. The therapy affects the patient s health after a perio of time; this effect may be observe through clinical measurements. b. Factorial HMM Ghahramani et al. [1997]. c. Factorial HMM in plates notation. escription of this process for a single patient, omitting the patient specific inex for simplicity of notation. We escribe how mutations may be create, how resistance may be acquire, the relation to the therapy outcome an the transition to time t + 1. Then we escribe the entire generative process. Focusing on a single chain k, let t,k be a binary variable enoting whether rug k was taken at time t an let t be the multi-rug treatment outcome ( t = 1 for a successful treatment). Let m t,k be a binary variable representing the existence of mutations resistant to rug k in the serum, an let R t,k be a binary variable representing whether permanent resistance to rug k alreay exists (through viral reservoirs) at time t. If permanent resistance was alreay acquire at time t (i.e. R t,k = 1), then the rug resistant strain may also be foun in the serum: m t,k = 1. therwise, when R t,k = 0, when rug k is taken ( t,k = 1), new rug resistant mutations may appear with probability p M k. If the rug is not taken at time t, no such mutations will prevail. Equation 1 summarizes this relation: 1 if R t,k = 1 Pr(m t,k = 1 R t,k, t,k ) = p M k if R t,k = 0, t,k = 1 0 otherwise (1) When rug resistant mutations appear in the serum (m t = 1) an are not suppresse ( t = 0), these may form reservoirs, an the virus will consequently acquire permanent resistance to rug k (R t+1,k = 1). We enote the probability for this event by p C. There are two conitions for this situation: (1) Mutations resistant to rug k have emerge, an (2) The treatment fails, i.e., no other rug taken at time t is successful in suppressing the virus. It is also possible, that viral reservoirs resistant to rug k ha alreay existe at time t an will be preserve at time t + 1. Finally, there is also a low probability for a new infection by a strain resistant to rug k between time t an t + 1 leaing to the creation of a new viral reservoir. We enote the probability for this event by p I. In practice, we assume p C = 1 ɛ an p I = ɛ for a small fixe ɛ. 1 ɛ if R t,k = 1 1 ɛ if R t,k = 0, Pr(R t+1,k = 1 R t,k, m t,k, t ) = m t,k = 1, t = 0 ɛ otherwise (2) We place a prior p R0 k = Pr(R 1,k = 1) on resistance that ha alreay been acquire by the patient before the first recore treatment. Such resistance may be acquire ue to past treatments missing from our ata before t = 1, or ue to an initial infection by a rug resistant strain cfemia et al. [2015], Van e Vijver et al. [2006]. In combine antiretroviral therapy, the therapy is expecte to succee if the virus is sensitive to at least one of the compouns in it, for at least one k, m t,k = 0. To account for eviations from this moel an since meical recors are prone to errors, we moel the observe outcome as a noisy version of this expecte outcome: Let t E be the expecte therapy outcome an let t be the therapy outcome observe in the EHR ata, then t E = {k: t,k =1} m t,k Pr( t = E t m t, t ) = 1 p N (3) At any given time t, the observe treatment outcome, t, epens only on a small number of variables m t,k (typically 2 3) associate with the rugs in the CART. The generative process is as follows: 1. For all rug compouns k 1,..., K (a) raw mutation probability p M k Beta(β) (b) raw prior resistance probability p R0 k Beta(γ) 2. raw outcome noise probability p N Beta(η)

4 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy 3. For all patients i=1,...,n (a) for t=1,...,t i. for all rugs k=1,..,k A. if t = 1, raw R i,1,k p R0 k, else raw R i,t,k Pr(R i,t,k R i,t 1,k, i,t 1, m i,t 1,k ) B. raw m i,t,k conitione on R i,t,k, i,t,k ii. raw treatment outcome t conitione on m i,t, i,t where N is the number of patients an β, γ, an η are Beta priors that we treat as hyper-parameters of the moel. This moel is epicte in Figure 2(b). 2.1 Patient Aherence The generative process in Section 2 assumes full knowlege of the rugs taken by the patient, i.e. that patients take rugs as prescribe. However, in real life, patients often fail to ahere to the prescribe rug regimen Kim et al. [2014], Glass et al. [2006]. Therefore, when a therapy fails, there is now uncertainty if this is ue to rug resistance or ue to lack of aherence. We exten the moel escribe in the previous section to account for patient aherence: Let a i,t be a binary variable enoting whether patient i has taken the prescribe rugs at time t, an let p A i = P r(a i,t = 1) be the patient specific aherence probability. Since moern CART therapies typically use a single pill that contains all therapy rugs, all rugs can either be taken or not at a specific time t, an a single aherence variable a i,t for all rugs is sufficient. We exten the generative process to inclue aherence by aing the following steps as a prior process before the generative process escribe in Section 2: 1. For all patients i 1,..., N (a) raw p A i Beta(α) (b) for t=1,...,t i. raw a i,t p A i ii. set i,t,k = a i,t s i,t,k for all k = 1... K where s i,t,k is a binary variable inicating whether rug k was prescribe to patient i uner the therapy at time t an α is a Beta hyper-parameter. Note that these aitional steps only set i,t,k to the rugs taken in practice by patient i at time t which are now unobserve. nce these rugs have been set, the generative process procees as before. This extene moel is epicte in Figure 2(c). This extension also moifies the factorial structure of the chain by aing a new interchain epenency at each time point, as the aherence a i,t affects all i,t,k for all therapy rugs (for which s i,t,k = 1). 2.2 Relate Moels FResist is closely relate to Factorial HMM Ghahramani et al. [1997], with the aition of observe treatment, an an effect of the observe therapy outcome at time t on the hien states at time t + 1. FResist specifies a structure on the hien states of each one of the chains inuce by the relations between R an m. Further, since CARTs consist of only 2-3 rugs, the observation in FResist at each time (therapy outcome) epens on only 2-3 chains. The two moels are epicte in figures 1(c) an 2(b). Moeling patient aherence as a epenency between the chains, this time through a latent variable (Figure 2(c)). Couple HMMs Bran et al. [1997] are a relate class of moels where the chains interact uring their progression. However, each chain has its own observations: s t,i, the state of chain i at time t epens on the states of other chains, but the observation o t,i epens only on s t,i. Pan et al. [2012] moel influence by human interaction in a social network using couple HMMs an employ variational EM algorithm for approximate learning. Dong et al [2012] an Fan et al. [2015] moel isease infection in a social network using Graph-couple HMM, a special case of couple HHM. They take avantage of a sparse network structure to erive a Gibbs Sampling algorithm for a isease sprea moel. Dong et al [2016] evelope variational inference methos for a class of couple HMM moels for sprea of an epiemic. ur work iffers from those mentione above in the structure of the moel, the algorithm we use for approximate learning an the application. 3 Approximate Learning Similar to Factorial HMM an couple HMM, exact learning in FResist is intractable. Various approaches have been suggeste for approximate learning in these moels, incluing EM an variational EM Ghahramani et al. [1997], Pan et al. [2012], Dong [2016], EM with Gibbs sampling Ghahramani et al. [1997], Fan et al. [2015] an Gibbs Sampling for graphs with sparse connectivity between the chains Dong et al. [2012]. In this paper we suggest a novel Collapse Gibbs Sampling algorithm for factorial HMMs in general an for FResist in particular. It iffers from Gibbs sampling as in Dong et al. [2012] by integrating out the continuous variables rather than sampling them. This approach is commonly applie in topic moels such as LDA Griffiths an Steyvers [2004] an in the context of a single HMM Griffiths et al. [2004]. We erive the sampling equations for the general case of factorial HMM an we provie them in Appenix A. Here we escribe the

5 A. Gruber, C. Yanover, T. El-Hay, A. Sönnerborg, V. Borghi, F. Incarona, Y. Golschmit S t-1 t S K R t-1 t m m R K a s R t-1 t a m s m R K n n n n (a) (b) (c) Figure 2: a. Multi-rug therapy moel. Notice the aition of rugs () an the effect of treatment outcome on the patients health. b. The suggeste moel, FResist, is a special case of the moel in (a). c. FResist with patient aherence (a). sampling equations for the special case of FResist. 3.1 Collapse Gibbs Sampling for FResist In collapse Gibbs Sampling Liu [1994], the iscrete variables are sample while the continuous variables are integrate out. In the case of FResist, the iscrete variables nee to be sample are m an R, an in the aherence moel of Section 2.1 also a 3. The continuous variables that integrate out are p M, p R0, p N an in the case of aherence also p a. We assume that the Beta priors are known an set ɛ = We iterate over all patients an their treatments t, an at each iteration sample R t, m t, a t of a specific patient conitione on all other variables (omitting again the patient specific subscripts for ease of notation). Due to the eterminstic relations between R t, m t, a t in some cases, (e.g. m t,k = 1 an R t,k = 0 implies a t,k = 1) we sample these 2K + 1 variables together as a block. We use the factorization of the joint posterior istribution given in equation 4 to sample them efficiently (in time linear, rather than exponential, in the number of prescribe rugs). Rt, m t, a t are sample conitione on all other variables, observations an hyper parameters. To simplify an shorten the equations below, we only explicitly write the variables on which R t, m t, a t epen (an omit all others. We also omit the hyper parameters an observe prescriptions on which we always conition). Pr( R t, m t, a t R t 1, R t+1, t 1, t ) (4) = Pr( R t m t, a t, R t 1, R t+1, t 1 ) Pr( m t R t 1, R t+1, a t, t 1, t, ) Pr(a t t 1, t, R t 1, R t+1 ) 3 t,k is simply compute from a t an the observe s t, ; formally a t an t are block sample together. 4 ɛ (or p C an p I ) coul be integrate out exactly like the other probabilities in the moel. In practice, setting ɛ = 0.01 prove sufficient an simplifies the presentation of the sampling algorithm. The factorization given by equation 4 implies that we first sample a t averaging over the 2K variables R t, a t, then sample m t conitione on the alreay sample a t an average over the R t, an finally sample R t conitione on a t, m t. We begin with sampling R t : The iniviual elements R t,k of R t are conitionally inepenent given a t, m t, R t+1, R t 1, t 1, t an we can sample them iniviually: Pr(R t,k R t 1,k, R t+1,k, m t, a t, t 1, t ) (5) Pr(R t+1 R t,k, t, m t,k ) Pr(m t,k R t,k, a t ) Pr(R t,k R t 1,k, t 1 ) where the terms in equation 5 can be compute using equations 2, an 11. For sampling m t, we use the factorization Pr( m t...) = k Pr(m t,k m t,k <k,...) to iniviually sample m t,k conitione on m t,k for all k < k an average over m t,k for all k > k as escribe in equations 6-7. Pr(m t,k R t 1, R t+1, a t, t, t 1, m t,k <k) (6) R t,k Pr(R t+1,k m t, t, R t,k ) Pr( t m t,k k, R t 1,k >k, R t+1,k >k) Pr(m t,k a t, R t,k ) Pr(R t,k R t 1,k, t 1 ) Computing Pr( t m t,k, R t 1, R t+1, m t,k <k) requires averaging over all configurations of m t,k >k (an exponentially large number). We show in equation 7 how to compute it in linear time. The other factors from equation 6 are compute using equations 2 an 11. Pr( t m t,k k, t 1, R t 1,k >k, R t+1,k >k) (7) [ Pr( t E ( m t = o)) o=0,1 m M o ] Pr( m t,k >k a t, R t 1,k >k, t 1 ) Pr(R t+1,k >k t, m t,k )

6 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy where M 1 = {m : E (m) = 1} an M 0 = {m : E (m) = 0} are the sets of all configurations of m that yiel the specific expecte outcome E. The sum over M 0 inclues the single term where all m k = 1 (for rugs prescribe at time t). The other sum, over M 1 inclues all other configurations. We compute it in time linear in the number of rugs in the therapy using the factoriaztion: Pr( m t R t 1, a t ) Pr( R t+1 t, m t, a t ) = (8) [ Pr(m t,k R t,k, a t ) Pr(R t+1,k t, m t,k, R t,k ) k R k ] Pr(R t,k R t 1,k, t 1 ) Similarly, a t is sample while averaging over m t, R t in linear time: Pr(a t t 1, t, R t 1, R t+1 ) (9) Pr(a t ) Pr( t a t, R t 1, R t+1 ) (10) where Pr( t a t, R t 1, R t+1 ) is compute using equation 7 setting k = 0 (i.e. averaging over all m k, R k ). The other probabilities require for sampling accoring to equations 5-7 are compute from counts of sample variables: Pr(m t,k = m R t,k = 0, a t = 1) = C M k,m + β m CM k,m + 2β (11) Pr( t m t ) = C t=t E(m) + η n C (12) n + 2η C R0 k,r Pr(R 1,k = r) = + γ r CR0 k,r + 2γ (13) Ci,a A Pr(a i,t = a) = + α (14) a CA i,a + 2α Where Ck,m M, CR0 k,r, CA i,a are the counts of the sample variables m,k, n i,t, R 0,k, a i respectively excluing m i,t, R 0,i,k, a i,t, an C n counts the number of times t is equal or ifferent from t E (compute from the sample m s). To summarize, espite the coupling of the chains through patient aherence an therapy outcome, we have erive a sampling algorithm linear in the number of variables to be sample. This is in contrast to sampling in a general factorial HMM (see appenix A in the supplementary material for etails). 4 Planning Long-Term Therapies Having moele rug resistance evolution using FResist an having learne the moel ynamics, we procee to using the moel for therapy esign. ur goal is to plan an entire series of treatments a 1,..., a T, possibly ifferent from each other, that woul optimize patient s health over the entire time perio (in contrast to optimizing only for the first treatments). Due to the Markovian structure of FResist, optimizing such a sequence of treatments is a finite Markov Decision Process (MDP) that can be solve using ynamic programming to fin a policy that woul maximize the total cumulative rewar 5. Let q t = (m, R, a) t the state at time t, a t A the action (a combination of rugs) we take at time t an t (q t, a t ) the outcome of the treatment a t if the patient is at state q t at time t. We efine the immeiate rewar of a treatment a t to be 1 if the treatment is successful an 0 otherwise, an the expecte rewar of a state q t an action a t is E( t q t, a t,..., a T ). We efine the value of state q t at time t to be the expecte outcome of the best treatment (optimize over all possible actions): ur goal is to fin the sequence of actions (therapies) yieling an overall maximal rewar (maximal cumulative patient health along the entire time perio). V t (q t ) = max at,...,a T T E( t q t, a t,..., a T ) (15) t =t With this efinition, it follows that: V T (q T ) = max E( T (q T, a)) (16) a A V t 1 (q t 1 ) = max E( t 1 (q t 1, a) (17) a + qt p(q t q t 1, a)v (q t )) Equations can be solve by ynamic programming with a single backwar pass. 5 Experiments We empirically evaluate an analyse the performance of FResist using synthetic an real ata sets. We use preictive log likelihoo on a previously unseen test set as our means of valiating the moel an comparing to other methos. We compare it to logistic regression an ranom forest as these are the methos currently being use in state of the art ecision support systems for selecting a therapy for HIV patients Revell et al. [2016, 2014], Rosen-Zvi et al. [2008]. In a previous work, Wang et al. [2009] compare ranom forest to artificial neural networks an SVM with RBF kernel an conclue that ranom forest outperforme the other methos. The systems in Revell et al. [2016, 5 For optimizing the total cumulative rewar inefinitely, one can efine a ecay factor an use value iteration to solve the optimization problem.

7 A. Gruber, C. Yanover, T. El-Hay, A. Sönnerborg, V. Borghi, F. Incarona, Y. Golschmit 2014], Rosen-Zvi et al. [2008] use a iverse range of features incluing emographic features an have shown them to be very informative for preicting treatment outcome. However, in the comparisons we inclue in this paper we use only clinical ata in the form of therapy an its outcome, as the focus of this paper is on the rug resistance evolution. As a part of ecision support system, the metho suggeste in this paper can be use as an aitional feature to logistic regression or ranom forest similar to the use of a Bayesian network in Rosen-Zvi et al. [2008]. Following Rosen- Zvi et al. [2008], the clinical features that we use for both logistic regression an ranom forest were: rugs of the current therapy, rugs previously taken by the patient, number of previous treatments, outcome of last treatment. In the first experiment, we generate a family of atasets with varying outcome noise levels (p N from equation 3) in the range There were 1000 patients in all the atasets, all of them with full aherence (a i = 1). Each patient receive two ranomly selecte combination therapies, each of which was a series of 10 continuous treatments, to a total of 20 treatments per patient. Each therapy was a ranom combination of 3 antiretroviral compouns out of a total of 5 available compouns to choose from (yieling ( 5 3) possible therapies). We split each of these atasets to a train set with 500 patients an a test set with the remaining 500. Figure 3(a) shows preictive log likelihoo of the three methos as a function of outcome noise. As expecte, FResist outperforms the other methos on ata generate accoring to the moel. It is interesting to see that as we a noise to the ata, the performance of all methos roppe rastically, well beyon the ecrease in likelihoo expecte only ue to introucing noisy outcomes in the test set (where ranom values nee to be preicte). The reason for this ecrease is that observing a noisy therapy failure, even if just one, is a strong inication that persistent resistance has evolve. In the secon experiment, we teste the effect of patient aherence on preicting therapy outcome. Similar to the first experiment, we generate a family of atasets with the same characteristics, but without any outcome noise (p N = 0) an we varie the aherence level a i between 1 an 0.8. Figure 3(b) shows preictive log likelihoo of the three methos as a function of patient s aherence. Again, FResist outperforms the other two methos (the performance of FResist ecrease compare to the previous experiment as aherence was sample rather than set). nce again, we see a significant rop in performance for all methos as patient aherence ecreases. The reason is the increase uncertainty when observing a treatment failure regaring its cause - whether it is ue to rug resistance or because the patient simply i not take the prescribe rugs. The EuResist ataset 6 is an integrate atabase containing clinical an emographic ata of more than patients in Europe. From this ataset, we extracte all patients with at least 30 viral loa measurements. From these, we selecte a ranom subset of 1000 patients an split them to train an test sets of 500 each. The ataset that we extracte inclue clinical samples taken between 1997 an Similar to Rosen-Zvi et al. [2008] an others, we associate a clinical sample with a therapy if the sample was taken at least 8 weeks after the therapy starte an before the therapy was stoppe. We use viral loa as the therapy outcome. A treatment was consiere successful when the viral loa was below 500 copies/ml an failure otherwise (see aitional iscussion below). Figure 4 compares the preictive log likelihoo of FResist, logistic regression an ranom forest on the 500 patients from the EuResist test set. FResist outperforms the other two methos. To better unerstan FResist s performance we inspecte the samples rawn by it from the posterior istribution. For example, estimating an average patient aherence p A by averaging all variables a p,t, the value learne by FResist is 0.87, which suggests that moeling patient aherence is important. To verify that, we compare the performance of FResist with a range of pre-set values for patient aherence, an we inee see its importance as shown in Figure 6 (in this experiment, each aherence value was set to all patients, which performs worse than an iniviually learne value on the train set, but generalizes better to the test set in this setting where aherence is not iniviually fit to the patients in the test set). Another interesting observation is the importance of moeling the prior resistance (Figure 5): In our ataset (train an test combine), 473 out of 1000 of the first treatments faile. It is important to note some limitations of the above empirical valiation: The EuResist ata is retrospective (in contrast to ata from clinical trials): Drugs were selecte by physicians who expecte them to be effective for a specific patient. This means that a therapy prescribe for patient i is more likely to succee for i than for others. However, the above methos ignore this bias an assume that apriori (before observing any treatment outcome) there is no ifference between the patients with respect to therapy chance of success. We i not ajust for this bias in our stuy an this is a subject for future work. Another limitation of the above analysis is the significant progress mae in HIV research an treatment uring the years in which the EuResist ata has been col- 6

8 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy Synthetic Data: Noise Synthetic Data: Aherence log likelihoo (test) FResist LR RF log likelihoo (test) FResist LR RF noise level aherence level (a) (b) Figure 3: a. Test log likelihoo of FResist (black), logistic regression (LR, re) an ranom forest (RF, green) as a function of the noise ae to synthetic ata. b. Test log likelihoo as a function of patient aherence. log likelihoo (test) EuResist FResist LR RF Figure 4: Log likelihoo (test) of FResist, logistic regression an ranom forest on the EuResist ataset. lecte: availability of rugs an guielines has change with the introuction of new rugs an new rug classes. Moreover, the ability to etect HIV in the bloo has improve significantly. While in the past viral loa below 500 copies/ml coul not be etecte, it is now possible to etect the virus even when its level in the bloo is below 50 copies/ml. To avoi an inconsistent efinition of treatment success in ifferent years, we chose 500 as our threshol for therapy success (for untreate patients, viral loa is in the range of tens - hunres of thousans). 6 Summary EuResist: Importance of Prior Resistance log likelihoo (test) FResist FResist No Prior Resistance Figure 5: EuResist: Moeling prior rug resistance greatly outperforms assuming full initial rug sensitivity. log likelihoo (test) Figure 6: than 1. EuResist: Patient Aherence Aherence (pre set) Patient Aherence: ptimal value is smaller In this paper we presente FResist, a novel probabilistic graphical moel for the evolution of rug resistance in HIV. The moel erives from Factorial HMM an specifies relations between therapy rugs, mutations, persistent resistance an therapy outcome. It further extens Factorial HMMs by moeling patient aherence. Approximate learning in FResist is one using a novel Collapse Gibbs Sampling algorithm that we evelope, new also in the context of Factorial HMM. Compare to state of the art ecision support systems for HIV therapy selection, FResist more accurately preicts therapy outcome, an gains insight regaring patient aherence to the therapy. Further, it plans a series of treatments optimizing patient s health in the long term, while existing systems provie greey therapy recommenations optimize for the success of a single treatment at a time. While motivate by rug resistance in HIV, it woul be interesting to apply FResist to other iseases with emerging rug resistance an multi-rug therapies such as certain types of cancer. ther irections for future research inclue ajusting for bias in the clinical ata use for training, taking possible sie effects into account when planning a series of treatment or consiering rug cross resistance.

9 A. Gruber, C. Yanover, T. El-Hay, A. Sönnerborg, V. Borghi, F. Incarona, Y. Golschmit References AIDSInfo. Guielines for the use of antiretroviral agents in hiv-1-infecte aults an aolescents. Panel on Antiretroviral Guielines for Aults an Aolescents. Department of Health an Human Services., URL ContentFiles/AultanAolescentGL.pf. P. Arora an N. M. Dixit. Timing the emergence of resistance to anti-hiv rugs with large genetic barriers. PLoS Comput Biol, 5(3):e , M. Bran, N. liver, an A. Pentlan. Couple hien markov moels for complex action recognition. In Computer vision an pattern recognition, proceeings., 1997 ieee computer society conference on, pages IEEE, F. Clavel an A. J. Hance. Hiv rug resistance. New Englan Journal of Meicine, 350(10): , W. Dong. Variational inference with agent-base moels. In Proceeings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages International Founation for Autonomous Agents an Multiagent Systems, W. Dong, K. Heller, an A. Pentlan. Graph-couple hmms for moeling the sprea of infection. In Uncertainty in Artificial Intelligence, T. El-Hay,. Weissbro, E. Eban, M. Zazzi, an F. Incarona. Structure proportional jump processes. In Proceeings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pages AUAI Press, K. Fan, M. Eisenberg, A. Walsh, A. Aiello, an K. Heller. Hierarchical graph-couple hmms for heterogeneous personalize health ata. In Proceeings of the 21th ACM SIGKDD International Conference on Knowlege Discovery an Data Mining, pages ACM, Z. Ghahramani, M. I. Joran, an P. Smyth. Factorial hien markov moels. Machine learning, 29(2-3): , T. R. Glass, S. De Geest, R. Weber, P. L. Vernazza, M. Rickenbach, H. Furrer, E. Bernasconi, M. Cavassini, B. Hirschel, M. Battegay, et al. Correlates of self-reporte nonaherence to antiretroviral therapy in hiv-infecte patients: the swiss hiv cohort stuy. JAIDS Journal of Acquire Immune Deficiency Synromes, 41(3): , T. L. Griffiths an M. Steyvers. Fining scientific topics. Proceeings of the National acaemy of Sciences, 101 (suppl 1): , T. L. Griffiths, M. Steyvers, D. M. Blei, an J. B. Tenenbaum. Integrating topics an syntax. In NIPS, volume 4, pages , S.-H. Kim, S. M. Gerver, S. Filer, an H. War. Aherence to antiretroviral therapy in aolescents living with hiv: systematic review an meta-analysis. Ais, 28(13): , R. G. Krishnan, U. Shalit, an D. Sontag. Structure inference networks for nonlinear state space moels. arxiv preprint arxiv: , J. S. Liu. The collapse gibbs sampler in bayesian computations with applications to a gene regulation problem. Journal of the American Statistical Association, 89(427): , M. cfemia, A. Hernanez, N. Sauvala, A. ster, an H. Hall. Epiemiology of hiv-1 transmitte rug resistance among men who have sex with men in the unite states. In XXIV International HIV Drug Resistance Workshop, W. Pan, W. Dong, M. Cebrian, T. Kim, J. H. Fowler, an A. S. Pentlan. Moeling ynamical influence in human interaction: Using ata to make better inferences about influence within social systems. IEEE Signal Processing Magazine, 29(2):77 86, T. Pierson, J. McArthur, an R. F. Siliciano. Reservoirs for hiv-1: mechanisms for viral persistence in the presence of antiviral immune responses an antiretroviral therapy. Annual review of immunology, 18(1): , M. Prosperi, A. Pironti, F. Incarona, G. Traigo, an M. Zazzi. Preicting human-immunoeficiency virus reboun after therapy initiation/switch using genetic, laboratory, an clinical ata. In Proceeings of the 7th ACM International Conference on Bioinformatics, Computational Biology, an Health Informatics, pages ACM, M. C. Prosperi, M. Rosen-Zvi, A. Altmann, M. Zazzi, S. Di Giambeneetto, R. Kaiser, E. Schülter, D. Struck, P. Sloot, D. A. Van De Vijver, et al. Antiretroviral therapy optimisation without genotype resistance testing: a perspective on treatment history base moels. PloS one, 5(10):e13753, A. Revell, M. Boy, D. Wang, S. Emery, B. Gazzar, P. Reiss, A. Sighem, J. Montaner, H. Lane, an B. Larer. A comparison of computational moels with an without genotyping for preiction of response to secon-line hiv therapy. HIV meicine, 15 (7): , A. D. Revell, D. Wang, R. Woo, C. Morrow, H. Tempelman, R. L. Hamers, P. Reiss, A. I. van Sighem, M. Nelson, J. S. Montaner, et al. An upate to the hiv-treps system: the evelopment an evaluation

10 Factorial HMMs with Collapse Gibbs Sampling for ptimizing Long-term HIV Therapy of new global an local computational moels to preict hiv treatment outcomes, with or without a genotype. Journal of Antimicrobial Chemotherapy, page kw217, M. Rosen-Zvi, A. Altmann, M. Prosperi, E. Aharoni, H. Neuvirth, A. Sönnerborg, E. Schülter, D. Struck, Y. Peres, F. Incarona, et al. Selecting anti-hiv therapies base on a variety of genomic an clinical factors. Bioinformatics, 24(13):i399 i406, D. Van e Vijver, A. M. Wensing, C. A. Boucher, et al. The epiemiology of transmission of rug resistant hiv-1. HIV sequence compenium, 2007:17 36, D. Wang, B. Larer, A. Revell, J. Montaner, R. Harrigan, F. De Wolf, J. Lange, S. Wegner, L. Ruiz, M. J. Pérez-Elías, et al. A comparison of three computational moelling methos for the preiction of virological response to combination hiv therapy. Artificial intelligence in meicine, 47(1):63 74, M. Zazzi, R. Kaiser, A. Sönnerborg, D. Struck, A. Altmann, M. Prosperi, M. Rosen-Zvi, A. Petroczi, Y. Peres, E. Schülter, et al. Preiction of response to antiretroviral therapy by human experts an by the euresist ata-riven expert system (the eve stuy). HIV meicine, 12(4): , 2011.