A Simplified Approach to Teaching Markov Models

Transcription

1 Notes A Simplified Approach to Teaching Markov Models C. Daniel Mullins College of Pharmacy, University of Maryland, 100 Penn Street, Baltimore MD Erik S. Weisman Department of Economics, Duke University, Box 90097, Durham NC Markov Models are used with increasing frequency to describe disease progression. The Markov process is a fairly complex tool and a power mathematical technique. By delineating the logic behind the process, pharmacy faculty can prepare students to understand the Markov process and to appreciate the usefulness of Markov modeling for pharmaceutical research and the development of clinical guidelines. This article provides an introduction to the Markov process and Markov models, with suggestions for including Markov modeling in pharmacy instruction. Following a brief introduction to Markov modeling in a general setting with six attributes of Markov processes and explains the usefulness of Markov modeling in a general setting with the aid of two generic examples. Next, a specific Markov model of a terminal illness is presented, and applications of this example are explored. A brief listing of software used in Markov modeling is presented in the appendix. INTRODUCTION Understanding complex problems often can be facilitated by transforming the global question into a series of manageable components. Each component represents a simplified problem offering unique and necessary information for evaluating the broader, more complex issue. In the area of health, examples include optimizing the selection of drug protocols, the guidelines for radiation therapy, and the use of experimental procedures or products. In each example, the decision occurs in practice at the individual patient level: the care provider determines a treatment regimen at the initial visit, then modifies treatment at subsequent visits based on how the patient responds to care. At the population level, treatment regimens are based on the observed outcomes across a welldefined group of patients(1,2). The results of optimization problems often are measured in terms of output or outcomes, such as number of prescriptions filled, quality-adjusted life years, or reduction in diastolic blood pressure. For theoretical analysis, calculus-based techniques provide efficient means for determining maximized values of the desired output. However, calculus-based techniques require that the function being optimized be differentiable over the decision or input variables. Markov models use a dynamic programming technique that does not restrict the analysis to continuous and differentiable decision variables. Furthermore, the stepwise logic embedded in the dynamic programming methodology makes Markov models particularly adept at handling complex optimization problems with many variables in which time plays a crucial role(3,4). ATTRIBUTES OF THE MARKOV DECISION PROCESS The Markov process, named after the Russian mathematician A.A. Markov, , is a special case of a more general category of processes called stochastic processes. Doob(5) defines a stochastic process as the mathematical abstraction of an empirical process whose development is governed by probabilistic laws. Markov Models examine the condition of the stochastic process at various points in 42 American Journal of Pharmaceutical Education Vol. 60, Spring 1996

2 time, categorize these conditions, and examine the effects of external influences on the stochastic process. As such, Markov models incorporate a multi-stage decision process. As with all multi-stage decision processes, the Markov process is distinguished from other stochastic processes by the characterization of six basic attributes: states, stages, actions, rewards, transitions, and constraints(3). States The states of a Markov model describe the complete set of mutually-exclusive alternative conditions under which a system operates. One can think of a system as a sample space that is partitioned into j states. Each state fully represents a distinct situation, and therefore, the states cannot overlap. In addition, the j states cover the entire sample space. In the medical and pharmacoeconomics literature, these states often represent various levels of disease progression. The number of states commonly is assumed to be finite for purposes of mathematical simplification and because a complete listing of the description of each state generally is provided. A state may be defined as either transient or absorbing. A transient state is, by definition, temporary. Once entered, it will be vacated with certainty. An absorbing state is one from which there is no escape. Once entered, no other states are possible(6). In terms of disease progression, a transient state might be a clinically-diagnosed early onset of cancer. Upon entering this state, a patient either will progress to an advanced state of cancer, go into remission, or make the transition to the absorbing state of death. Stages The stages of a Markov model are the points in time when the system is observed and data are collected. Note that stages in a Markov model differ from disease stages described in the medical literature, which are similar to the states of a Markov Model. Markov modeling can accommodate either constant or random time intervals between stages. Referring to the cancer patient, the first observation stage may show a cancerous stage later. Alternatively, a patient may remain in the same cancerous state for several consecutive stages. Once an absorbing state, e.g., death, is entered, each successive stage will, by definition, show the patient in the same state. Actions/Decisions At each stage, an action may be taken from a finite set of alternatives. The choice of actions may be state specific. In terms of a disease, the appropriate therapy for a patient may involve a drug regimen, surgery, dietary modification, or no action at all. It should be noted that the decision process often is absent from Markov models which are designed to study the natural history of a disease. For example, see Longini(7,8), Nagelkerke(9) and Mariotto(10) regarding early studies of HIV. As research of the disease evolves and knowledge is translated into clinical practice, treatment protocols are developed. The effect of these protocols on disease progression can be analyzed using Markov models. Rewards/Benefits The objective of the sequence of actions is to maximize a sequence of observable rewards or potential benefits. These rewards may be speed of recovery, extended years of life, or increased quality of life. The Markov process results in a sequence of rewards as the patient makes the transition from state to state. This reward stream is a random variable, the probability distribution of which is reflected by the nature of the Markov process itself and the sequence of actions taken(6). (See the State Transitions/Laws of Motion attribute below.) Different state of a disease may dictate different actions in order to maximize the rewards. An optimal course of action may be devised by comparing the stream of rewards resulting from different decisions or treatment protocols. State Transitions/Laws of Motion The transitions, or laws of motion, of a Markov process describe the likelihood of being in a particular state at some future time given the current state. Thus, the law of motion determines the probability distribution of the Markov random variable. Consider an example in which the complete set of alternatives is described by six mutually-exclusive states denoted s j where j=1,2,3,4,5,6. For some given state s j, there are six associated probabilities, p ij (t), that describe the probability of making the transition from state i to each of the six states over an interval of time equal to t. One of these probabilities, p ij (t), is the probability of remaining in the same state for a time period of t. Each probability p ij must be between zero and one, inclusive. Further, the Markov processes considered in this article involve a finite number of mutually-exclusive states where the set of states completely describes the system, the sum of the probabilities must equal one. That is, Markov models generally rely on the assumption that the set of transition probabilities depends only on the current state (and current action) and not on any previous history of the process(6). In other words, a patient s progression rate to the next state is independent of his or her progression rate through previous states(8). In addition, the Markov process is time homogeneous: the transition probabilities depend only on the length of the time period, t, and not on the initial time(11). Constraints Constraints may be introduced into the Markov process to incorporate computational or medical/scientific logic within the system. Constraints also are added to the process to simplify and generalize the progression of disease states or to concentrate on typical behavior by eliminating the effect of outliers. For the study of a terminal illness such as HIV, a common constraint is that state transitions are irreversible and sequential(7,8,10,12,13). That is, once a patient has progressed to a subsequent disease state, it is assumed that s/he will not revert to an earlier state. Furthermore, some researchers apply a constraint that each state of the disease must be passed through in order to enter a more progressed state. States that meet both of these criteria are referred to as tunnel states because an individual passes through them in a set sequence, analogous to passing through a tunnel(4). It may be the case that, because of the timing of the observations, a patient appears to have skipped a state from one stage to the next. In such instances, it is assumed American Journal of Pharmaceutical Education Vol. 60, Spring

3 Table I. Terminal illness transition matrix Current Period Next Period Terminal illness Death Terminal illness 1-p(t) p(t) Death 0 1 Table II. Multi-state transition matrix Next period Current Disease Symptomatic Advanced period free disease disease Death Disease free p 11 (t) p 12 (t) 0 0 Symptomatic p 21 (t) p 22 (t) p 23 (t) p 24 (t) disease Advanced p 31 (t) p 32 (t) p 33 (t) p 34 (t) disease Death Fig.1. Simple terminal illness Markov moel. that the patient moved through the skipped state during the interval. GENERIC MARKOV MODELS Once student have grasped a basic understanding of the six attributes of the Markov process, a generic example can assist in synthesizing the information. It may be helpful to begin with a simple example involving only two states and excluding the decisions and rewards attributes. For instance, consider a cohort of patients with some specific terminal illness. If one does not distinguish between early and advanced states of terminal illness, terminally ill patients either will remain in the current state, i.e., continue to be terminally ill, or die. Assume that the probability that a living person will make the transition to death in any given period is p, where 0<p<1. Because of the sum of the probabilities must equal one, the remaining probability, 1-p, represents the probability that the patient will continue to be terminally ill. Once dead, the probability of changing to any other state is zero as the individual remains dead with a probability of one. These probabilities can be described by a transition matrix, as illustrated in Table I. The transition matrix defined in Table I describes the likelihood of being in either the state of terminal illness or death given the current state. Thus, the probability of Terminal Illness in the next period given Terminal Illness in the current period is equal to 1-p while the probability of being in the state of Death in the next period given Terminal Illness in the next period given Death in the current period is 0, while the probability of being in the state of Death in the next period given Death in the current period is 1. Because the transition from one state to another depends on the length of time between observations, the probability often is denoted by p(t). The information in Figure 1 is identical to the information contained in the transition matrix in Table I. Most Markov models are more complex than the one represented by Figure 1. Typically, models involve several different disease states with a set of probabilities describing Fig 2. Multi-state disease Markov model. The likelihood of being in each of the states at a specified future time given the current state. As with the first example, the set of probabilities can be described either by a transition matrix or a graphic representation. An example of a more complex Markov model is presented in Figure 2 and Table II, both of which present identical information. Again, the decisions and rewards attributes are absent. The Markov process described in Table II and Figure 2 incorporates four disease states: State 1 is an initial state in which individuals are disease free; State 2 is a symptomatic disease state in which patients demonstrate characteristics of illness; State 3 is an advanced disease state in which patients have progressed to a more profound, perhaps critical, condition; and State 4 is an absorbing state of death. In this example, an individual may move as shown by the arrows in Figure 2. The state are not sequential, i.e., states may be skipped. For example, a patient may make the transition from State 2 to State 4, bypassing State 3, but s/he may not move from State 1 to State 3 directly. The Markov process describing this disease also is reversible. The transition process does permit an individual to revert to an earlier state. For instance, symptomatic individuals in State 2 can revert to a disease-free state presumably, these individuals are cured and individuals in the advanced disease state can revert to a less advanced (symptomatic) disease state or become disease free. In this example, the probability of making a transition from state i to state j over a period of length t is written as p jj (t). For example, the probability of transition from State 2 to State 3 is p 23 (t), the probability of transition from State 3 to State 1 is p 31 (t) and the probability of remaining in State 2 is p 22 (t) 44 American Journal of Pharmaceutical Education Vol. 60, Spring 1996

4 TRANSITION INTENSITIES AND TRANSITION PROBABILITIES Reversible, nonsequential processes, such as the one illustrated in Figure 2, describe the transition probabilities of moving between any two states. A more relevant factor in an irreversible, sequential process is the likelihood of exiting a particular state and, by necessity, moving to a more advanced state. Consider an expansion of Figure 1 with several irreversible, sequential disease states prior to the terminal state. The probability that a patient moves out of a particular tunnel state, i, during some standard time interval (often one month) is called the transition intensity and is denoted λ i. The transition probability of moving, for example, from State 1 to State 2 during this standard time interval, p 12 (l), where time is denoted in months, is not the same as the transition intensity, λ i. The transition probability p ii+1 includes both a starting and ending state over the one month interval. This concept is more restrictive than λ i, which indicates only a starting state for the same time interval. For example, over the course of a month, the patient may have moved from State 1, through State 2, and finished in State 3. This outcome is included in λ 1, but not in and thus, p ii+1 (l) λ i. The λ i s are monthly progression rates, i.e., the probability that, over the course of one month, the patient exits state i by the end of the month. The transition probabilities, p ij (t), are the probabilities that a patient begins in state i and ends in state j over a time interval of t months. The transition probabilities are a function of the transition intensities and, because of the time homogeneity of the Markov process, the λ i s are constants(14). The transition intensities also can be used to predict the length of time a typical patient will spend in each tunnel disease state. These time intervals are called mean waiting times(7). For example, if the monthly progression rate, λ i, is 0.4, then the probability that the typical patient exits from stage i over the course of a month is 40 percent and the mean waiting time for state i is 1/λ i or 2.5 months. TERMINAL ILLNESS MARKOV MODEL Referring to Figure 3, consider the natural history of a disease described by a Markov model in which individuals progress irreversibly and sequentially through disease states. The model in Figure 3 contains five states, which describe a general model of disease progression, and four transition intensities, λ i >0, i=1,2,3,4. The initial state of this model is one in which the individual is disease free. An intermediate tunnel state follows in which the individual has a subclinical disease. The illness may be detectable through routine screening in a clinical setting, but the individual is asymptomatic during this state(15). The onset of symptoms defines the third state in a general model of terminal disease progression. The advanced disease state is the last state for which there remain transition alternatives (i.e., the individual can remain in the advanced state or make the transition Fig.3. Irreversible multi-state terminal disease Markov model. to death). The final state is not only irreversible, but also absorbing. APPLICATIONS OF THE TERMINAL ILLNESS MODEL The terminal illness Markov model presented in Figure 3 describes the natural history of a disease and therefore does not include a decision process (nor the resultant reward sequence) in the analysis. For teaching purposes, this setup allows students to focus on the transition component of the model. This example is useful for enhancing the knowledge of disease state transitions for specific illnesses and has been used extensively for cancer(16) and AIDS(7,8,10,12). While Markov models offer a powerful instructional aid for explaining state transitions, these models are not limited to describing the natural history of a disease. Once the natural history has been established, these estimation techniques also allow comparisons of alternative treatments. Separate data sets on similar cohorts of patients can be examined using Markov models to compare the effects of various interventions. In the same way that a control group serves as a benchmark for examining the effects of an experimental therapy, a model of the natural progression of a disease can be used as a base case to measure the outcomes and effectiveness of these interventions. Therefore, the Markov model is useful for expanding the knowledge of treatment protocols by examining the effects of clinical practice on the transition probabilities. Markov models also can assist in understanding the effect of risk factors associated with the onset or progression of illness. Risk factors include predisposing factors such as gender, age, and previous illness, enabling factors such as poor nutrition, restricted access to medical care, and poverty, precipitating factors such as exposure to a viral, carcinogenic, or other noxious agent, or reinforcing factors such as repeated exposure and stress(17). DATA AND MARKOV MODELING Data concerning the progression of a disease often are characterized by a high degree of censoring. Because an investigator only observes subjects at discrete points in time (the stages), the exact transition time between states is not known. If a transition from one state to another has occurred between two subsequent stages, the data are said to be interval censored(7). The midpoint between the two stages of observation often is used as the transition date. In addition, subjects usually are followed for only a portion of their disease history. This leads to both left and right censored data. If, at the time of the first stage, a subject has been in a given infected state for an indeterminate amount of time, the data are said to be left censored. If, at the last stage, a subject has not yet made the transition to the terminal state, the data are said to be right censored(7). The fact that the data are censored does not imply that only intermediate endpoints are observed or that complete information regarding disease states is unavailable for all patients. Rather, it means that data are not complete for all subjects. Another attribute of data concerning disease progression is that a subject usually is not observed at constant intervals, but rather intermittently(18). Markov models are nicely suited for censored and intermittent data, because the primary interest is in predicting the transition probabilities from State i to State j during some interval and not in the probabilities of exact transition times. American Journal of Pharmaceutical Education Vol. 60, Spring

5 All that is necessary to predict the transition probabilities using a Markov model is knowledge of subsequent states and the time elapsed between observations(9). The ability to accommodate intermittent data and censored data is heavily dependent on the Markov assumption of time homogeneity mentioned above. It should be noted that, while data are collected at various stages or points in time, these data can be used to predict transition probabilities for any time interval the researcher chooses. That is, one is not restricted in the time intervals of the transition probabilities by the timing of the stages. This situation is analogous to regression analysis where the data are used to predict values of Y for chosen values of X i. PARAMETER ESTIMATION Parameter estimation for Markov analyses involving hundreds or thousands of patients across multiple attributes requires the use of appropriate computational aides. (Several computer software packages that calculate parameter estimates for Markov models are referenced in the Appendix.) While the primary concern of an investigator may be to estimate the transition intensities and transition probabilities, staged stochastic models also can provide probability density functions for virus incubation periods, cumulative distribution functions describing the fraction of the population that is diagnosed or symptomatic, and survival curves indicating the fraction of the population that has not yet reached the terminal state. For example, a Markov model can estimate the mean time form exposure to seroconversion of HIV, or the average time between AIDS diagnosis and death. The model also provides statistical information necessary to plot survival curves for HIV + individuals. Much of the research on Markov models focuses on maximum likelihood estimates of the transition intensities, λ i, by numerically maximizing the natural logarithm of the likelihood function. Because the data often are non-continuous variables, this estimation typically is accomplished through a derivative-free, pseudo-gauss-newton algorithm(7,10) or a similar quasi-newton approach(18). Reporting of confidence intervals is problematic, because studies use non-continuous data that, at best, can assume asymptotic multivariate normality. As a result, the delta method or various bootstrap techniques are used (12,13). DISCUSSION The Markov processes described above provide guidance for macro analysis of disease progression and can assist in producing general guidelines for treatment. The description is intended to demonstrate the wide variety of applications of Markov models. In contrast to previous work by Sonnenberg and Beck(4) and Pauker and Kassirer(19) which examine the use of Markov models in clinical decision making, this paper focuses on the natural progression of illness as the baseline measure against which treatment alternatives can be compared. Pharmacy instruction that introduces the Markov model according to the outline provided above should be followed by specific examples from the medical decision making literature. For HIV and AIDS treatments, good examples include analysis of early zidovudine treatment(20) and zidovudine in combination with Pneumocystis carinii pneumonia prophylaxis(12,21). Although Markov modeling can be a very powerful tool, there is one severe restriction that should be considered. This restriction is the basic Markovian assumption of time homogeneity, which states that the transition probabilities of the model are constant over time and are independent of the history of the process. This simplifying assumption is made to obtain tractable results for the transition probabilities in an event of heavily censored data(8). Thus, it is assumed that the length of time spent in any given state does not affect the chance of exiting that state. Depending on the number of states and the frequency of stages, this assumption may be violated for disease progression and might lead to biases from model mis-specification(7,18). It may be the case that either the length of time elapsed from the beginning of the process or the length of time spent in the current state affects the transition probabilities. Unfortunately, the appropriateness of the timehomogeneity assumption is difficult to examine because of the heavily censored data (7,16,18). Another problem with Markov models is the often seemingly arbitrary method of choosing states. An example of this is the common use of CD4+ lymphocyte count intervals to define states of HIV status (8,12,13,18,22). It is difficult to defend the position that an individual with a CD4 + lymphocyte count of 199 is different from that of an individual with a CD4 + lymphocyte count of 200. An additional concern with Markov models is censoring methodology(7,10,18). When various researchers follow different methodologies, it becomes difficult to compare results from separate studies. The same difficulty arises with regard to persistence criteria(8,12). It is not uncommon for researchers to apply some criterion to ensure that a patient has, in fact, made a transition to another tunnel state. This criterion might include observing the patient in the same or more advanced state for two consecutive stages, thus excluding anomalies where a patient is observed as having advanced for one stage only to have reversed in the following stage. It is difficult to compare studies using a specific persistence criterion with those not using the same criterion (or those using none at all). Nonetheless, Markov models provide valuable scientific and medical information, as evidenced by the increasing number of published studies in peer-reviewed literature. CONCLUSION Markov models allow health researchers to evaluate complex data concerning disease progression and intervention using a manageable mathematical process. Pharmacy instruction can introduce students to the Markov process by describing the six attributes of Markov models: states, stages, actions/decisions, rewards/benefits, state transitions/laws of motion, and constraints. The use of general examples provide future practitioners with the ability to develop their knowledge of Markov modeling that then can be applied to specific examples. The examples provided in this article bring students from a basic understanding of modeling the probability of survival to a more advanced focus on the relevant parameters of disease progression. These estimation techniques then can be incorporated into research that estimates the effect of various interventions on specific disease progression. Markov models offer an efficient means of evaluating protocols and can incorporate quality of life measures to assist in pharmacoeconomic analysis. 46 American Journal of Pharmaceutical Education Vol. 60, Spring 1996

6 Am J. Pharm. Educ., 60, 42 47(1996); received 9/19/95, accepted 1/16/96. References (1) Lohr, K.N., Outcome measurement: Concepts and questions, Inquiry, 25, 37-50(1988) (2) Mullins, C.D., Baldwin, R. and Perfetto, E.M., Outcomes: A yardstick for health care, J. Amer. Pharm. Assoc., NS36, 39 49(1996). (3) Lippman, S.A., Dynamic programming and Markov decision processes, in The New Palgrave: A Dictionary of Economics, Vol. 1, (edits. Eatwell, J., Milgate, M. and Newman P.), The Macmillian Press Limited, London (1987), pp (4) Sonnenberg, F.A. and Beck, J.R., Markov models in medical decision making: A practical guide, Med. Decis. Making, 13, (1993). (5) Doob, J. L., Stochastic Processes, John Wiley & Sons, Inc., New York (1953). (6) Howard, R.A., Dynamic Programming and Markov Processes, The M.I.T. Press, Cambridge, MA (1960). (7) Longini, I.M. Jr., Clar, W.S., Byers, R.H., Ward, J.W., Darrow, W.W., Lemp, G.F. and Hethcote, H.W., Statistical analysis of the stages of HIV infection using a Markov model, Stat. Med., 8, (1989). (8) Longini, I.M. Jr., Clark, W.S., Gardner, L.I. and Brundage, J.F., The dynamics of CD4+ t-lymphocyte decline in HIV-infected invididuals: A Markov modeling approach, J. Acquir. Immune Defic. Syndr., 4, (1991). (9) Nagelkerke, N.J.D., Plummer, F.A., Holton, D., Anzala, A.O., Manji, F., Ngugi, E.N. and Moses S., Transition dynamics of HIV disease in a cohort of African prostitutes: A Markov model approach, AIDS 4, (1990). (10) Mariotto, A.B., Mariotti, S., Pezzotti, P., Rezza, G. and Verdecchia, A., Estimation of the Acquired Immunodeficiency Syndrome incubation period in intravenous drug users: A comparison with male homosexuals, Am. J. Epidemiol., 135, (1992). (11) Bhurucha-Reid, A.T., Elements of the Theory of Markov Processes and their Applications, McGraw-Hill Book Company, Inc., New York NY (1960). (12) Longini, I.M. Jr., Clark, W.S. and Karon, J.M., Effect of routine use of therapy in slowing the clinical course of Human Immunodeficiency Virus (HIV) infection in a population-based cohort, Am. J. Epidemiol., 137, (1993). (13) Statten, G.A. and Longini, I.M. Jr., Estimation of incidence of HIV Infection using cross-sectional marker surveys, Biometrics, 50, (1994). (14) Keeler, E. Decisions tree and Markov models in cost-effectiveness research, in Valuing Health Care, (edit. Sloan, F.A.), Cambridge University Press, New York (1995), pp (15) Hennekens, C.H. and Buring, J.E., Epidemiology in Medicine, Little Brown and Company, Boston (1987). (16) Kay, R., A Markov model for analyzing cancer markers and disease states in survival studies, Biometrics, 42, (1986). (17) Beaglehole, R., Bonita, R. and Kjellstrom, T., Basic Epidemiology, World Health Organization, Geneva (1993). (18) Gentleman, R.C., Lawless, J.F., Lindsay, J.C. and Yan, P., Multi-state Markov models for analyzing incomplete disease history data with illustrations for HIV disease, Stat. Med., 13, (1994). (19) Pauker, S.G. and Kassirer, J.P., Decision analysis, N. Engl. J. Med., 316, (1987). (20) Oddone, E.Z., Cowper, C., Hamilton, J.D., Matchar, D.B., Hartigan, P., Samsa, G., Simberkoff, M. and Feussner, J.R., Cost effectiveness analysis of early zidovudine treatment of HIV infected patients, Br. Med. J., 307, (1993). (21) Graham, N.M.H., Zeger, S.L., Park. L.P., Phair, J.P., Detels, R., Vermund, S.H., Ho, M., Saah, A.J. and the Multicenter AIDS Cohort Study, Effect of zidovudine and Pneumocystis carinii pneumonia prophylaxis on progression of HIV-1 infection to AIDS, Lancet, 338, (1991). (22) Wilson, S.R. and Solomon P.J., Estimates for different stages of HIV/AIDS disease, Cabios, 10, (1994). APPENDIX. COMPUTER SOFTWARD A variety of software packages easily can handle the complex calculations required to produce parameter estimates for Markov models. A number of researchers use statistical packages such as BMDP(7), DATA, SMLTREE, or Decision Maker(19). Others have found it more efficient to tailor their estimation procedures by writing computer programs in FORTRAN(9) or SPSS(10). Wilson and Solomon(22) have devised a Unix-based disease specific software designed for HIV/AIDS progression. American Journal of Pharmaceutical Education Vol. 60, Spring