Computational methods for evidence-based decision support

Transcription

1 CIHC 2010 Computational methods for evidence-based decision support in pharmaceutical decision making Tommi Tervonen Econometric Institute, Erasmus University Rotterdam Eindhoven, 20/10/2010

2 Outline 1 Introduction to regulatory drug benefit-risk analysis Evidence-based medicine Benefit-risk analysis 2 Evidence synthesis methods for clinical trials Meta-analysis Network meta-analysis 3 Decision aiding for pharmaceutical decisions Utility theory Value functions MAVT SMAA 4 Aggregate Data Drug Information System (ADDIS) Case study

3 A simple example of evidence-based medicine Q: Should we advise parents to administer over the counter cough medicines for acute cough? Aims: To determine the effectiveness of over the counter (OTC) cough medicines for acute cough in children (...) Methods: Systematic review of randomised controlled trials (RCTs) (...) Results: Six trials involving 438 children met all inclusion criteria. Antitussives, antihistamine decongestant combinations, other fixed drug combinations, and antihistamines were no more effective than placebo in relieving symptoms of acute cough (...) Most drugs appeared to be well tolerated with a low incidence of mostly minor adverse effects. Conclusion: OTC cough medicines do not appear more effective than placebo in relieving symptoms of acute cough (...) Schroeder & Fahey, BMJ, 2002

4 Randomized Controlled Trial (RCT) Properly designed RCT s provide the highest quality evidence on the treatments effects

5 Evidence-Based Medicine (EBM) Evidence-based medicine aims to apply the best available evidence gained from scientific research to medical decision making A large share of decisions made by health care professionals are informed by evidence-based medicine, e.g. prescription, regulatory- and reimbursement policy decisions Although the scientific evidence is transparent and achieved with methodological rigour, the actual decisions are often unstructured, ad hoc and lack transparency as the treatment benefit-risk valuation is not explicit

6 Application of EBM in regulatory drug benefit-risk analysis For a drug to be granted marketing authorization, it must be proven efficant, safe, and have a sufficient benefit-risk (BR) profile compared to other drugs already in the market a c Knowledge b Time Eichler & al., Nature Drug Disc, 2008

7 Drug benefit-risk analysis BR analysis should include all relevant evidence, and therefore apply (network) meta-analysis

8 Drug benefit-risk analysis Problems 1 Inclusion of all relevant evidence in the meta-analysis is not guaranteed 2 The BR analysis is unstructured and non-transparent

9 Approach taken in ADDIS Separate clinical data (measurements) from the value judgements (MCDA) Include all data present in the original analysis (imprecise measurements) Provide metrics for decision uncertainty Enable model generation for re-applicability

10 Outline 1 Introduction to regulatory drug benefit-risk analysis Evidence-based medicine Benefit-risk analysis 2 Evidence synthesis methods for clinical trials Meta-analysis Network meta-analysis 3 Decision aiding for pharmaceutical decisions Utility theory Value functions MAVT SMAA 4 Aggregate Data Drug Information System (ADDIS) Case study

11 Meta-analyses Often clinical trials are under-powered to reach statistical significance Sometimes different trials give different results (e.g. due to heterogenous population) need to synthesize, that is, to meta-analyze the existing evidence base

12 Example meta-analysis Hansen et al. Ann Intern Med 2005;143:

13 Fixed effect meta-analysis A fixed effect model is based on assumption that every study is evaluating a common treatment effect. That means the effect of treatment, allowing for the play of chance, was the same in all studies. Another way of explaining this is to imagine that if all the studies were infinitely large they d give identical results. The resulting summary treatment effect estimate is this one true or fixed treatment effect, and the confidence interval describes how uncertain we are about the estimate.

14 From fixed to random effects Sometimes this underlying assumption of a fixed effect meta-analysis (i.e. that diverse studies can be estimating a single effect) is too simplistic The alternative approaches to meta-analysis are (i) to try to explain the variation or (ii) to use a random effects model A group effect is random if we can think of the levels we observe in that group to be samples from a large population (e.g. when collecting data from different medical centers, the center can be thought of as random)

15 Random effects meta-analysis Assumes that the true treatment effects in the individual studies may be different from each other. That means there is no single number to estimate in the meta-analysis, but a distribution of numbers. The most common random effects model assumes that these different true effects are normally distributed. The meta-analysis therefore estimates the mean and standard deviation of the different effects. In inverse variance random effects model, the weight of each study is proportional to the inverse of its variance

16 Fixed- and random effects models

17 Meta-analysis limits Hansen et al. (2005) systematic review: 46 studies comparing n = 10 second-generation AD In total, 20 comparisons are available Out of n(n 1) 2 = 45 possible comparisons 3 meta-analyses are performed

18 Meta-analysis limits Fluoxetine (8) (5) (6) Paroxetine Sertraline Venlafaxine Hansen et al. Ann Intern Med 2005;143:

19 Meta-analysis limits Fluoxetine (8) Paroxetine (1) (7) (1) (6) (1) (2) (2) (3) Duloxetine Venlafaxine Citalopram (1) (2) (1) (2) (1) (2) (1) (2) Sertraline Escitalopram (3) (1) (2) Bupropion Mirtazapine Fluvoxamine

20 Meta-analysis with more than 2 treatments? Fluoxetine (baseline) 1.00 ( ) Paroxetine 1.09 ( ) Sertraline 1.10 ( ) Venlafaxine 1.12 ( ) How likely is it that Venlafaxine has the greatest efficacy? What happens if we choose another baseline? Other studies included possibly different results (2) Fluoxetine (8) Paroxetine (1) (7) (1) (6) (1) (2) (3) Duloxetine Venlafaxine Citalopram (1) (2) (1) (2) (1) (2) Sertraline Escitalopram (1) (2) Not all drugs can be included (escitalopram) (3) (1) (2) Bupropion Mirtazapine Fluvoxamine We re double counting multi-arm trials

21 Network meta-analysis Fluoxetine (8) Paroxetine (1) (7) (1) (6) (1) (2) (2) (3) Duloxetine Venlafaxine Citalopram (1) (2) (1) (2) (1) (2) (1) (2) Sertraline Escitalopram (3) (1) (2) Bupropion Mirtazapine Fluvoxamine Include all evidence in one analysis

22 Network meta-analysis Is an extension of normal meta-analysis Allows comparison of 2 alternatives Integrating direct and indirect evidence While checking for (in-)consistencies A.K.A.: Mixed/Multiple Treatment Comparison (MTC)

23 Network meta-analysis models Is a Bayesian hierarchical model That can be estimated with a Markov Chain Monte Carlo software (BUGS, JAGS) ADDIS ( can do it for you

24 Network meta-analysis: model Main components are: 1 Individual effect estimates 2 Linkage of the effect estimates (relative effects) 3 Random effects for the relative effects 4 Priors 5 Linkage of the relative effects

25 Level 1 (likelyhood) Modeling the effect r ik /n ik, for treatment k of study i, as a binomial proccess with success probability p ik : r ik Bin(p ik, n ik ) Using MCMC simulation, the model converges to maximum joint likelyhood estimate of these p ik given the data on r ik and n ik.

26 Linkage to relative effects Apply a transformation to obtain normally distributed variable: θ ik = logit(p ik ) = log p ik 1 p ik ; p ik = logit 1 (θ ik ) And choose a baseline b(i) and define θ ik (the log odds) in terms of a baseline µ i and relative effect δ ib(i)k (the log odds-ratio): θ ik = µ i + δ ib(i)k

27 Level 2 (random effects) We assume the relative effects (LORs) to be normally distributed: δ ixy = N (d xy, σ 2 xy) And, for a three-arm trial i with treatments x, y, z, b(i) = x: ( ) (( ) ( δixy dxy σ N, 2 )) xy ρ xy,xz σ xy σ xz δ ixz d xz ρ xy,xz σ xy σ xz σ 2 xz

28 Level 2 (random effects) We assume the relative effects (LORs) to be normally distributed: δ ixy = N (d xy, σ 2 ) And, for a three-arm trial i with treatments x, y, z, b(i) = x: ( ) (( ) ( δixy dxy σ 2 σ N, 2 )) /2 σ 2 /2 σ 2 δ ixz d xz Under the assumption of equal variances

29 Level 3 (Priors) For each of the parameters of interest, µ i, d jk and σ, specify priors: µ i N (0, 1000) d jk N (0, 1000) σ U(0, 2)

30 Overview So far, this is just a random effects meta-analysis!

31 Consistency assumption d yz = d xy d xz Borrow strength from indirect evidence. The left-hand term (d yz ) is a functional parameter The right-hand terms (d xy, d xz ) are basic parameters Only basic parameters are stochastic

32 Inconsistency Factors Introduce an inconsistency factor for each functional parameter: d yz = d xy d xz + w xyz After the model has converged, we test w xyz = 0 Comparing inconsistency and consistency models on model fit is also useful Correctly specifying the model is tricky (but possible with 15 definitions, 5 lemmas, 3 corollaries and 4 theorems)

33 Example: evidence network Cipriani & al., Lancet 2009

34 Example: results (LOR)

35 Example: results (rank probabilities)

36 Outline 1 Introduction to regulatory drug benefit-risk analysis Evidence-based medicine Benefit-risk analysis 2 Evidence synthesis methods for clinical trials Meta-analysis Network meta-analysis 3 Decision aiding for pharmaceutical decisions Utility theory Value functions MAVT SMAA 4 Aggregate Data Drug Information System (ADDIS) Case study

37 What is a decision? In our context: a choice of the best treatment from a set of competing ones There exists uncertainty about the outcomes... and how can we formally support such decisions?

38 St. Petersburg paradox Consider the following game We flip a coin. If head comes up, the game ends. If tail comes, we flip again. You begin the game with 1 euro. After each head, we double it. On head, you get the current pot. How much are you willing to pay to play this game? A poor person might be unwilling to pay more than a euro, the minimum possible pay-off of the game.

39 St. Petersburg paradox Expected value: E = = k=1 = 1 2 But still, no-one would be willing to stake e.g e Why don t people behave as to maximize expected wealth?

40 Utility theory Utility is dependent on the particular circumstances of the person making the estimate. U X X (U = utility, X = total current wealth) Individuals (should) behave as to maximize expected utility Utilities elicited with lotteries:

41 Utility functions Figure: Risk averse Figure: Risk seeking

42 Utility functions in healthcare HRQOL, CUI,... Quality Adjusted Life Year (QALY) the most common measure, 1.0 = a year of full health, 0.0 = dead. E.g. a life year in wheel chair could have a value of 0.7. The QALY model requires utility independent, risk neutral, and constant proportional tradeoff behaviour QALY is often used as a scale that is translatable to monetary terms, which can be incorrect!

43 Problems with utility theory Entire risk profile cannot be captured with a single number (expected utility) Utile has no meaning to most people No natural utility functions People violate axioms (e.g. framing / prospect theory)

44 Back from utility to value functions Utility is a measure of preference under uncertainty Value is a measure of preference under certainty

45 Multi-attribute value theory (MAVT) Functional representation F (x 1,..., x n ) = f (v 1 (x 1 ),..., v n (x n )) In practice we usually apply an additive form F = n v i (x i )w i i=1 Applicable when all attributes are mutually preferentially independent

46 Preferential independence Attribute specific value functions are meaningful only if preferences for performance levels in one attribute do not depend on the performance levels of other attributes This should not to be confused with a possible correlation of alternatives scores! Is a property of preferences, not that of alternatives

47 Weights v(x) = n v i (x i )w i i=1 The weights in MAVT represent (ONLY) the importance of scale swings How many times more important is to increase salary from 1000 to 2000 euros than to have office in 16th floor instead of 2nd floor? As only the ratio is meaningful, they can be normalized to sum to e.g. 1

48 Existing BR models O Brien & Lynd s benefit-risk plane for 2-criteria stochastic outcomes Mussen & al s regulatory BR assessment with MAVT Felli & al s with binning of continuous data to categories + MAVT Tervonen & al s with data in original format and ordinal preferences

49 Drug BR model with clinical data Evaluate a set of alternative treatments with respect to outcome measures in the trials Benefits = clinically relevant endpoints, Risks = adverse drug reactions (side effects) Measurements directly from trial results: Single-study BR model: beta-distributed incidence rate Network BR model: Log-normal distributed relative effects For odds-ratios we need to estimate/assume baseline, as the scale swing is meaningless Preferences with ordinal swing weighting ( would you rather decrease the incidence rate of diarrhea from 0.6 to 0.1 or increase HAM-D responder rate from 0.2 to 0.5? )

50 SMAA - an inverse approach to MAVT Figure: Traditional MCDA Lahdelma & Salminen, Springer, 2010

51 SMAA - an inverse approach to MAVT Figure: SMAA Lahdelma & Salminen, Springer, 2010

52 Weight space Total lack of preference information is represented by a uniform joint probability distribution of the weight space If some preference information is available, the weight space can be restricted with linear constraints

53 Rank acceptability index Describes the share of different weights and criteria measurements ranking an alternative on a certain rank bi r = ξ χ f χ (ξ) f W (w) dw dξ w Wi r (ξ)

54 Central weight vector & confidence factor Central weight vector describes the preferences of a typical DM supporting this alternative with the assumed preference model CW s are used for inverse approach: instead of asking preferences and giving results, answers the question which preferences support an alternative to be the most preferred one? Confidence factor is the probability for an alternative to be the preferred one with the preferences expressed by its central weight vector CF measures whether the criteria measurements are accurate enough to discern the efficient alternatives

55 Computation Analytical techniques based on discretizing the integrals with respect to each dimension are infeasible, so the integrals are estimated through Monte Carlo simulation simulations provide sufficient accuracy for the indices Algorithm has less-than squared mean complexity and is very fast in practice Tervonen & Lahdelma, EJOR, 2007

56 Outline 1 Introduction to regulatory drug benefit-risk analysis Evidence-based medicine Benefit-risk analysis 2 Evidence synthesis methods for clinical trials Meta-analysis Network meta-analysis 3 Decision aiding for pharmaceutical decisions Utility theory Value functions MAVT SMAA 4 Aggregate Data Drug Information System (ADDIS) Case study

57 What is ADDIS? Cross platform open source software for managing and analyzing aggregate clinical trial results Implements random effects- and network meta-analyses as well as benefit-risk analyses Current version 1.0, new releases every few months. Been developed for 1,5 years, and is still under development at RuG/UMCG

58 Problems in clinical data models Same clinically relevant endpoint can be measured in different times (e.g. 3 or 6 weeks) and on different scales (e.g. HAM-D or MADRS) Standardization of the data structures haven t aimed for model generation need for a minimal data model

59 ADDIS data model Indication Name: String (Key) StudyCharacteristic Name: String (Key) OutcomeMeasure Definition: String (Key), BenDir: {H, L}, Type: {R, C} Study Name: String (Key) ActualStudyCharacteristic Value: String Treatment Definition: String (Key) StudyOutcomeMeasure Arm ArmCharacteristic Name: String (Key) Measurement SampleSize: Natural ActualArmCharacteristic Value: String RateMeasurement Count: Natural ContinuousMeasurement Mean: Real, StdDev: Real

60 ADDIS data model: example instantiation OutcomeMeasure Definition BDir Type Responders... Change from... Headache Nausea Chest pain H... L L L R C R R R Indication Name Depression Type 2 Diabetes Study Name Indication Coleman... Depression NCT Depression StudyCharacteristic Name Study size Group allocation Treatment blinding Patient eligibility criteria ActualStudyCharacteristic Study StudyChar. Value Coleman... Group... Randomized Coleman... Treatment... Double-blind Treatment Definition Placebo Fluoxetine Rosiglitazone StudyOutcomeMeasure OutcomeMeasure Study Arm Study Treatment ArmCharacteristic Name Arm size Dosing Gender distribution Measurement S.O.M. Arm SampleSize ActualArmCharacteristic Arm ArmChar. Value... Dosing 20 mg/day RateMeasurement Measurement Count ContinuousMeasurement Measurement Mean StdDev

61 ADDIS Case study Hansen & al. (Ann Intern Med, 2005) assessed safety and efficacy of four second generation antidepressants and concluded that there are no significant differences among the drugs In general, the assessment of antidepressants is hard; placebo effect is always present causing high uncertainty on the results Q s: 1 How can the benefit-risk assessment of second-generation antidepressant be structured based on evidence from the clinical trials? 2 Can we come up with something better than no significant differences?

62 Time for ADDIS!

63 Conclusions There is a current need to support drug regulatory decision making with benefit-risk models For truly evidence-based pharmacological decision support, we need to synthesize evidence with network meta-analysis Stochastic multi-criteria decision models can be generated from clinical trial contents ADDIS is the first (and only) software supporting generation and computation of network meta-analysis- and benefit-risk models Thank you for attention!