Tutorial: Introduction to Markov modelling

Transcription

1 Tutorial: Introduction to Markov modelling Mats O Karlsson Pharmacometrics Research Group Department of Pharmaceutical Biosciences Uppsala University Sweden

2 Coin toss Assume a coin has been tossed n times P tails =/2 (if perfect coin) Q: What is the probability of tails at the next toss P n+ ( tails ) = P tails Outcome is dependent on P tails only and not dependent on outome of previous tosses 2

3 Coin toss Assume a coin has been tossed n times P tails =/2 (perfect coin) Q: What is the total number of tails (S n+ ( tails )) after the next toss P n+ (S n+ ( tails )<S n ( tails )) = P n+ (S n+ ( tails )=S n ( tails )) = P tails P n+ (S n+ ( tails )=S n ( tails )+) = P tails P n+ (S n+ ( tails )>S n ( tails )+2) = Outcome is dependent on P tails and outomes of previous tosses S n ( tails ) contains all necessary information about prior history 3

4 Definition Markov property The Markov property, proposed by A.A. Markov ( ), asserts that the distribution of future outcomes depend only on the current state and not on the whole history 4

5 Modeling of sleep/wake state Example: Nighttime observations of awake or sleep every 5th minute in an insomniac patient Asleep (=) Awake (=) Time (h) 5

6 Logistic model NMTRAN $PROB Logistic model for sleep and awake $DATA data $INPUT DV $PRED P = THETA() IF(DV.EQ.) Y=P IF(DV.EQ.) Y=-P $THETA (,.5,) ; PROBABILITY OF BEING ASLEEP $ESTIM LIKE 6

7 P Two-state Markov model (Awake) P P (Asleep) P P = P P = - P 7

8 Markov model NMTRAN code $PROB Transition probabilities between sleep and awake $DATA data $INPUT DV PDV ;PDV=Previous DV ;PDV = Value of immediately preceding observation $PRED P = THETA() P = THETA(2) IF(PDV.EQ..AND.DV.EQ.) Y=P IF(PDV.EQ..AND.DV.EQ.) Y=-P IF(PDV.EQ..AND.DV.EQ.) Y=P IF(PDV.EQ..AND.DV.EQ.) Y=-P $THETA (,.,) ; PROB AWAKE GIVEN ASLEEP $THETA (,.,) ; PROB ASLEEP GIVEN AWAKE $ESTIM LIKE 8

9 Same model MLXTRAN code DESCRIPTION: Categorical data model with Markovian dependence, Binomial distribution INPUT: parameter = {p, p} OBSERVATION: Y = { } type = categorical categories = {,} dependence = Markov P(Y= Yp=) = p P(Y= Yp=) = p 9

10 Results Logistic model P =.43.5 OFV 32.7 Markov model P =..4 OFV 72.4 P =.4.5

11 P Two-state Markov model (Awake) P P (Asleep) P Transition Matrix PDV= PDV= DV=.89.4 DV=..86

12 Simulations Original data Time (h) Simulation from logistic model Time (h) Simulation from Markov model Time (h) 2

13 P Discrete-time Markov model (Awake) P P (Asleep) P Continuous-time Markov model (Awake) K K (Asleep) 3

14 Continuous-time Markov model $PROB Continuous-time two-state Markov model for awake/asleep $DATA data $INPUT DV PDV DELTAT $PRED K = THETA() K = THETA(2) P = (EXP(-(K+K)*DELTAT)*K+K)/(K+K) P = (EXP(-(K+K)*DELTAT)*K+K)/(K+K) IF(PDV.EQ..AND.DV.EQ.) Y=P IF(PDV.EQ..AND.DV.EQ.) Y=-P IF(PDV.EQ..AND.DV.EQ.) Y=P IF(PDV.EQ..AND.DV.EQ.) Y=-P $THETA (,) ; K ASLEEP GIVEN AWAKE $THETA (,) ; K AWAKE GIVEN ASLEEP $ESTIM LIKE 4

15 Continuous time Markov model - differential eqn parametrization $PROB Transition probability between awake and asleep $DATA data $INPUT DV TIME AMT CMT EVID ; Complex data set to initiate and empty compartments ; to reset compartment amounts after each observation $MODEL COMP=PRWAKE COMP=PRSLP $PK K = THETA() K = THETA(2) $DES DADT() = - K*A() + K*A(2) ;Represents Probability of - awake DADT(2) = K*A() K*A(2) ;Represents Probability of - asleep $ERROR IF(DV.EQ.) Y = A() IF(DV.EQ.) Y = A(2) $THETA (,) ; K ASLEEP GIVEN AWAKE $THETA (,) ; K AWAKE GIVEN ASLEEP $ESTIM LIKE 5

16 Data set for differential eqn solution to Markov model $PROB Two-state Markov model ; Entire system updated after each observation $DATA data $INPUT DV TIME AMT CMT EVID ;. ;awake at start - initialization ;. ;obs awake at T= ;. 4 ;set probabilities to P(awake)= ; 2. ;obs asleep at T=2 ; ;set probabilities to P(asleep)= 6

17 Data set for differential eqn solution to Markov model $PROB Two-state Markov model ;Data set structure with updating compartment by compartment ;May be needed when not entire system is to be updated (e.g. PKPD model) $DATA data $INPUT DV TIME AMT CMT EVID ;. ;awake at start - initialization ;. ;awake at T= ;. - 2 ; empty compartment ; ; empty compartment 2 ;. ; reinitialize comp to ;. 2 2 ; restart comp 2 with ; 2. ;asleep at T=2 ;. 2-2 ; empty compartment ; ; empty compartment 2 ;. 2 2 ; reinitialize comp 2 to ;. 2 2 ; restart comp with 7

18 Probability Probabilities following an awake observation at time= No additional observations made p = K K + K + K K + K e K+K t P P Time (h) 8

19 Probability Probabilities following an awake observation at time= Additional observations made at h (awake) and 2 h (asleep) Time (h) P (awake) P (asleep) 9

20 Probability Probability Probability Probability Introducing a treatment effect Without treatment With treatment P (awake) P (asleep) Time (h) Time (h) Promote falling asleep K K K+K Inhibit waking up K K K+K (Awake) K K (Asleep) Time (h) Time (h) Both effects K K K+K 2

21 Clues about mechanism in data Mechanism Total sleep time Average time of sleep episodes Average time of awake episodes # of arousals # of times falling asleep Total # of State transitions Promote falling asleep Inhibit waking up Both effects K+K 2

22 Repeated time to event analysis an alternative to Markov model Assume transitions as events Assume no unobserved transitions Separate (constant) hazards for waking up and falling asleep Asleep (=) I I I I I I Awake (=) X X X X X Time (h) X X event during awake period (falling asleep) censoring during awake period I event during sleeping period (arousal) 22

23 RTTE model NMTRAN $PROB Repeated Time To Event data - constant hazard $INPUT ID DV TYPE DVID $DATA data $PRED IF(TYPE.EQ.) HAZ= THETA() ;hazard for waking up IF(TYPE.EQ.) HAZ= THETA(2) ;hazard for falling asleep CUMHAZ = HAZ*DV ;cumulative hazard SUR = EXP(-CUMHAZ) ;survival probability IF(DVID.EQ.) Y = HAZ*SUR ;event IF(DVID.EQ.2) Y = SUR ;censoring $THETA (,.46) ; HAZ_WK (/5min) $THETA (,.9) ; HAZ_SL (/5min) $ESTIM LIKE 23

24 No of abstracts Markov PAGE Markov models Hidden Markov models

25 Bergstrand Bizzotto Pharmacometric Markov models GI movement Sleep Non-ordered categorical Non-ordered categorical Compartmental chain 29 CPT st order + stagetime 28 PAGE Girard Adherence 3-state categorical Discrete-time 998 Stat Med Henin Hand-and-foot syndrome Ordered categorical st order 28 CPT Ito Dizziness Ordered categorical st order 28 CPT Karlsson Sleep Non-ordered categorical st order + stagetime 2 CPT Karlsson Sedation Ordered categorical st order +stagetime 22 Kjellsson Sleep Non-ordered categorical Measurement and kinetics of in vivo drug effects st order + stagetime 26 PAGE Lacroix ACR2 Binary + dropout st order 29 CPT Maas Migraine Ordered categorical Hidden Markov 26 Cephalalgia Plan None Count st order 29 JPKPD Snoeck Seizures Count st order 27 PAGE Troconiz Seizures Count st order 27 PAGE Zandvliet Follicules Multinomial count Compartmental chain 28 PAGE Zingmark Side-effect Ordered categorical st order 25 JPKPD 25

26 Markov model for responder, nonresponder and dropout Ex, ACR2 score in Rheumatoid Arthritis Pr of remaining non-responder Non- Responder Pr of becoming responder Pr of becoming non-responder Responder Pr of remaining responder Pr 2 of dropping out Pr 2 of dropping out Absorbing state cannot leave the state Absorbing chain absorbing state can be reached from all other states 2 Dropped out Lacroix et al., Clin Pharmacol Ther, 29 26

27 NMTRAN code Responder, non-responder and dropout $PRED ;----transition from being a responder to non- responder--- LGT=LOG(THETA()/(-THETA())) + ETA() P=EXP(LGT)/(+EXP(LGT)) ;----transition from responder to dropout--- LGT2=LOG(THETA(2)/(-THETA(2))) P2=EXP(LGT2)/(+EXP(LGT2)) ;-- transitions from being a non-responder to a responder--- LGT=LOG(THETA(3)/(-THETA(3))) + ETA(2) P=EXP(LGT)/(+EXP(LGT)) ; transition's probabilities IF (PREV.EQ..AND.DV.EQ.) Y=P IF (PREV.EQ..AND.DV.EQ.2) Y=P2*(-P) IF (PREV.EQ..AND.DV.EQ.) Y=-P-P2*(-P) IF (PREV.EQ..AND.DV.EQ.) Y=P IF (PREV.EQ..AND.DV.EQ.2) Y=P2*(-P) IF (PREV.EQ..AND.DV.EQ.) Y=-P-P2*(-P) $EST METH= LAPLACE LIKE ;---transition from non-responder to dropout--- LGT2=LOG(THETA(2)/(-THETA(2))) P2=EXP(LGT2)/(+EXP(LGT2)) 27

28 Diagnostics ACR2 model - proportion responders Lacroix et al., Clin Pharmacol Ther, 29 28

29 Diagnostics ACR2 model - transitions 29

30 Concentration (nmol/l) Score Concentration (nmol/l) Score Concentration (nmol/l) Score Concentration (nmol/l) Score Spontaneous reporting of a side-effect Concentration Score Concentration Score Time (h) Concentration Score Time (h) Concentration Score Time (h) Time (h) Zingmark et al., JPKPD 25 3

31 Side-effect score Alternative assumptions regarding nature of transitions Time (h) For RTT(C)E implementation see Plan et al., CP&T 2 3

32 Parameterization of the model D b b pre l D b pre l D b b pre l D b pre l D b b pre l D b pre l S S S S S S Ce TC C EC Ce e E D tol pre i ) ( 5 5,,2 max lx lx x e e PC p S= = -PC S p S= = PC S -PC S=2 p S=2 = PC S=2

33 Spontaneous reporting of a side-effect - model simulations Proportional odds model Markov model Number of transitions min 3 min 6 min min 3 min 6 min Observed data (249; 335) (2; 33) (62; 82) (9; 4) (8; 5) (8; 5) (33; 223) (64; 95) (39; 59) (2; 29) (2; 29) (2; 29) The results shown are the average and ( th and 9 th percentiles) from simulated datasets The performance of the proportional odds model, but not the Markov model, is dependent on the choice of observation frequency 33

34 Sleep scoring non-ordered categorical data 2 Awake REM Stage - Stage 2-2 Stage 3-3 Stage Time (h) (Karlsson et al., CPT 2) 34

35 Choice of transitions to model To reduce the number of transitions to model, three criteria were defined to identify the transitions of interest representing: (i) >% of all observations in a stage, (ii) >% of all transitions from a stage (iii) >% of all transitions to a stage A transition was modeled, if at least one of these criteria was fulfilled (Kjellsson et al., PAGE 27) 35

36 Overall sleep pattern (Karlsson et al., CPT 2) 36

37 Probability of falling asleep Probability of falling asleep Probability of falling asleep Transition dependences Time of night Time in stage Concentration 37

38 What previous info to condition on? First-order Markov models often sufficient Sometimes biological processes dictate use of higher-order Markov elements 2nd (3rd etc) order elements Time since entering present stage (stagetime) Prior stage(s) 38

39 How to start a Markov chain? Sometimes obvious Sleep-wake data that starts at bed-time Responder status for ACR2 Sometimes screening data provide information about initial state Transform first observation(s) to a covariate for starting the chain 39

40 Sedation score Sedation scores following stroke Time (h) Karlsson et al., in Measurement and Kinetics of in vivo drug effects, 22 4

41 Alternative assumptions regarding nature of transitions Many transitions to model No assumption about intermediate transition states Fewer transitions to model Assumption about intermediate transition states 4

42 Sedation score Sedation score Sedation score Individual time courses of sedation Observations Time (h) Markov model simulations Time (h) Ordered categorical model simulations Time (h) 42

43 Model for GI transit A K FA K AF F Stomach F = proximal stomach A = distal stomach K AS K SI K SI K SI Small intestine SI:-4 transit SI: SI:2 SI:3 SI:4 Colon AC = Ascending colon TC = Transverse colon DC = Descending colon DC K TC TC K AC AC K SI K DC SCR Sigmoid colon / Rectum Bergstrand et al., CPT 29 43

44 Probability Probability GI position Proximal stomach Distal stomach Small intestine Colon Time after dose (h) Time after dose (h) 44

45 Count data - daily seizure scores in epileptic patients Time (days) Markov model; Troconiz et al., JPKPD 36:46-77 (29) Hidden Markov model; Delattre et al., JPKPD 39:263-7 (22) 45

46 Consequences of ignoring Markov properties - estimation Information content in data overestimated SEs underestimated Hypothesis tests inappropriate Interindividual variability overestimated Potential structural model misspecification No info on time-course of dependence 46

47 Consequences of ignoring Markov properties simulation & design Duration of state periods too low Inflated number of transitions Inflated number of extreme value occurrencies E.g. distribution of maximum severity score in population Individualisation strategy suboptimal Value of more frequent observations overrated Positioning of observation times Optimal design results inappropriate* *Optimal design w/ Markov models; Nyberg & Hooker PAGE 2 47

48 Markov model or not? Start with Markov Frequent observations Many consequtive same-state observations Many levels of response Non-ordered categorical data Start without Markov and diagnose Check number of transitions Check average duration of same-state periods 48

49 Thank you! 49