Standards for Heterogeneity of Treatment Effect (HTE)

Transcription

1 Standards for Heterogeneity of Treatment Effect (HTE) Ravi Varadhan, PhD (Biostat), PhD (Chem.Engg) Chenguang Wang, PhD (Statistics) Johns Hopkins University July 21, 2015

2 Module 1 Introduction

3 Objectives To be able to appreciate the importance of HTE assessment To understand the challenges of HTE assessment To learn the statistical concepts pertaining to HTE assessment To learn about the various types of inferential techniques for HTE assessment To learn a software tool for conducting HTE assessment using a case study example To understand and apply the PCORI methodology standards for HTE assessment

4 PCORI Standards for HTE 1 HT-1: State the goals of HTE analyses 2 HT-2: For all HTE analyses, Pre-Specify the analyses plan; for hypotheses driven HTE analyses, pre-specify hypotheses and supporting evidence base 3 HT-3: All HTE claims must be based on appropriate statistical contrasts among comparison groups, such as interaction tests or estimates of differences in treatment effect 4 HT-4: For any HTE analysis, report all pre-specified analyses and at minimum, the number of post-hoc analyses, including all subgroups and outcomes analyzed

5 Importance of the Standards Understanding HTE is a core aspect of PCOR, and the standards are aimed at enhancing the reliability of HTE findings from PCOR studies The standards are broadly applicable All study types: experimental/observational studies, explanatory/pragmatic trials, claims/ehr databases, etc. Should be considered during all 3 stages of a study: design, analysis, and reporting stages

6 What is HTE? At-risk individuals are heterogeneous (age, sex, disease etiology and severity, comorbidities, coexposures, genetic variants) Often, individuals respond differently to the same treatment Treatment response variation = explainable variation + random fluctuation HTE is the explainable variation in treatment response that is attributable to individual characteristics

7 ITE and HTE Two treatments: Z i = 0 and Z i = 1, i = 1,..., N. Two potential responses: {Y i (Z = 0), Y i (Z = 1)}. Individual treatment effect (ITE) is a comparison of the two potential responses: e.g., θ i = Y i (1) Y i (0). A necessary condition for HTE to be present: var(θ) > 0. The condition that var(θ) = 0, is known as unit-treatment additivity = Y i (1) Y i (0) = τ, i (Cox 1984)

8 Fundamental Problem of HTE However, we can only observe either Y i (Z = 0), or Y i (Z = 1), but not both (except in very rare cases) Hence, ITE is not identifiable We can consider groups of similar individuals and estimate a group-specific treatment effect (GTE) for each group This is subgroup analyis (to be discussed in another module)

9 Reading List 1 Berry DA (1990). Subgroup analyses. Biometrics. 46: Cox DR (1984). Interaction. Intl. Statist. Rev. 52: Gabriel SE and Normand SLT (2012). Getting the methods right - the foundation of patient-centered outcomes research. NEJM. 367:

10 Module 2 Why is HTE important?

11 Individual Variability If it were not for the great variability between the individuals, medicine might as well be a science, not an art." (William Osler, 1892) One man s meat is another man s poison

12 Average Treatment Effect (ATE) The paradox of the clinical trial is that it is the best way to assess whether an intervention works, but is arguably the worst way to assess who benefits from it. (Mant, 1999) What s good for the goose is good for the gander

13 Can We Do Better Than ATE? Answers from the clinical trial represent a group reaction. They show that one group fared better than another, that given certain treatment patients, on average, get better more frequently or more rapidly, or both. We cannot necessarily, perhaps very rarely, pass from that to stating exactly what effect the treatment will have on a particular patient. But there is surely, no way and no method of deciding that. Also, it is well to remember that observation of the group does not inhibit the most scrupulous and careful observation of the individual at the same time - if it is believed that more can be learnt that way. Sir Austin Bradford Hill, 1952

14 HTE Assessment Is Perilous HTE assessment is important and dangerous Numerous examples of spurious or unreplicated findings (e.g., astrological signs in ISIS-2) Percent reduction in 5-week vascular mortality Gemini or Libra -9% (NS) Other signs 28% (p < 10 5 ) Overall 23% (p < 10 5 ) Peter Rothwell s monograph on Treating Individuals contains more examples

15 Why is HTE Assessment Perilous? ATE is a first-order contrast, i.e. comparison between two treatment groups HTE analysis involves second-order or higher-order contrasts, i.e. difference of differences (e.g., difference in treatment effect between men and women) Higher-order effects have larger variance due to reduced information content - or, equivalently, due to reduced sample size in subgroups Increased false-positive (type-i) and false-negative (type-ii) errors for identifying HTE

16 The Central Challenge of HTE Analysis To Identify which type of individuals are likely to benefit from or be harmed by the treatment, while minimizing the possibility of spurious inferences due to biases and chance associations.

17 Reading List 1 Berry DA (1990). Subgroup analyses. Biometrics. 46: Fleming TR (2010). Clinical trials: discerning hype from substance. Ann Intern Med. 153: Jones HE et al. (2011). Bayesian models for subgroup analysis in clinical trials. Clin Trials. 8: Rothwell PM (2007). The Lancet: Treating Individuals: from randomised trials to personalised medicine. Elsevier. 5 Varadhan R et al. (2013). A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research. J Clin. Epidem. 66:

18 Module 3 Subgroup Analysis

19 Subgroup Analysis We focus on the setting of a randomized clinical trial of treatment Z for response Y with K categorical baseline variables or factors, X 1,..., X K Also applies to non-experimental settings, but possibly with some additional considerations such as confounded treatment assignment It is supposed a priori that any of these factors could be a source of HTE (selection of these factors is a critical issue, which will be discussed in another module) Two types of subgroups: univariate and multivariate subgroups

20 Univariate Subgroup Analysis Each subgroup is defined on the basis of a single baseline variable For example: men and women; smoker/non-smoker; normal/overweight/obese HTE is examined separately for each subgrouping variable Note that the subgroups are not mutually exclusive

21 Subgroup-Specific Treatment Effects

22 Limitations Comparisons are indirect & stratification reduces power

23 One-by-one interaction testing (OBO) g(e[y X]) g(e[y X]) = α 01 + α 11 Z + β 1 X 1 + γ 1 X 1 Z. = α 0K + α 1K Z + β K X K + γ K X K Z Remarks A proper way to test for HTE X k is assumed to be binary Bonferroni correction for type I error inflation Increases type I error without adjustment for multiplicty Adjustment for multiplicity decreases power K

24 Univariate Subgroup Analysis Most popular approach - reported in most Phase 3 clinical trials Evaluate whether the treatment is consistent across univariate subgroups The subgroups are not mutually exclusive, therefore, subgroup-specific treatment effects are correlated Subgroups are also more similar to each other since they differ in only one characteristic - less likely to detect HTE

25 Multivariate Subgroup Analysis Each subgroup is defined on the basis of multiple baseline variables For example: men-smoker, women-smoker, men-non-smoker, and women-non-smoker Mutually exclusive subgroups, therefore, the raw subgroup effects are independent

26 Multivariate Subgroup Analysis Our interest lies in understanding how the effect of treatment Z on Y varies jointly according to baseline characteristics. We can define the treatment effect within any given stratum x = {x 1,, x K } as follows: θ(x) = g(e[y Z = 1, X = x]) g(e[y Z = 0, X = x]), where E[.] denotes the expectation (mean) of Y, and g(.) is a link function denoting the scale in which the treatment effect is quantified.

27 Multivariate Subgroup Analysis Commonly used treatment effect scales are: ˆ identity link: g(x) = x ˆ log link: g(x) = log(x) x ˆ logit link: g(x) = log( 1 x ). Curse of dimensionality severely limits the number of characteristics Some type of modeling becomes necessary to share information across subgroups (we will discuss Bayesian approaches in another module)

28 Reading List 1 Berry DA (1990). Subgroup analyses. Biometrics. 46: Fleming TR (2010). Clinical trials: discerning hype from substance. Ann Intern Med. 153: Jones HE et al. (2011). Bayesian models for subgroup analysis in clinical trials. Clin Trials. 8: Rothwell PM (2007). The Lancet: Treating Individuals: from randomised trials to personalised medicine. Elsevier. 5 Varadhan R et al. (2013). A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research. J Clin. Epidem. 66:

29 Module 4 Regression Models for HTE Analysis

30 Interaction and Effect-Modification Two types of baseline factors X (DR Cox, ISR 1984) No difference statistically, but important interpretationally When X is another treatment or a modifiable factor (e.g., smoking), we say that X interacts with treatment Z When X is an intrinsic variable or a fixed attribute (e.g., gender), we say that X modifies the effect of treatment Z

31 Risk-Based HTE Analysis In conventional analysis, each covariate might have small effect (weak analyses w/ multiplicity concerns) Covariates might combine in some fashion to reveal HTE (joint effects) Primary outcome risk is a potentially strong HTE predictor (Kent 2010) Outcome risk, P(Y = 1 Z = 0), is mathematically related to the treatment effect, for example: P(Y = 1 Z = 0) P(Y = 1 Z = 1) RRR = 1 RR = P(Y = 1 Z = 0)

32 Risk-Based HTE Analysis Outcome risk is computed for each individual Outcome risk distribution is reported separately for the two treatment arms Treatment effect (relative risk or absolute risk reduction) is estimated and reported within strata of outcome risk Formal interaction test of outcome risk with treatment is not emphasized

33 Risk-Based HTE Analysis How to estimate the outcome risk for each individual? Pre-existing, validated prognostic scores (e.g., Framingham risk score, CHADS 2, Karnofsky performance scale, FRAX, frailty index) What if validated prognostic models are not available (or if the model does not calibrate well to the present study)? We may use the control arm to develop a prognostic index, but this underestimates uncertainty and HTE estimation could be biased

34 Unstructured interaction model (UIM) g(e[y]) = α 0 +α 1 Z+β 1 X 1 + +β K X K +γ 1 X 1 Z+ +γ K X K Z Comments All possible two-way Z*X interactions Model has 2K + 2 parameters Single test of any possible interactions Power decreases with increasing K

35 Proportional interactions model (PIM) g(e[y]) = α 0 (1 Z) + α 1 Z + β X(1 Z) + θ(β X) Z A single parameter represents HTE In UIM, the γ interaction coeffiicients (γ X) Z can take any value PIM is a special case of UIM: γ = θβ, θ a scalar Main assumption: prognostic effects for treated subjects are proportional to those of control subjects Similar in spirit to risk-based HTE approach, but more powerful and broadly applicable

36 Proportional interactions model (PIM) g(e[y]) = α 0 (1 Z) + α 1 Z + β X(1 Z) + θ(β X) Z Under PIM: H 0 : θ = 1 to H 1 : θ = 1 Rejecting H 0 is evidence that proportional interaction applies Not rejecting H 0 does not necessarily imply absence of HTE More challenging to interpret HTE findings

37 Reading List 1 Cox DR (1984). Interaction. Intl. Statist. Rev. 52: Gail M and Simon R (1985). Testing for Qualitative Interactions Between Treatment Effects and Patient Subsets. Biometrics. 41: Kent DA, et al. (2010). Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 11: Kovalchik S et al. (2013). Assessing the heterogeneity of treatment effect in a clinical trial with the proportional interactions model. Stats. Med. 32: Rothwell PM (2007). The Lancet: Treating Individuals: from randomised trials to personalised medicine. Elsevier. 6 vanderweele TJ and Knol MJ (2014). A tutorial on interactions. Epidemiol. Methods. 3:

38 Module 5 Bayesian Models for HTE Analysis

39 What is a Bayesian HTE Analysis? In Bayesian analysis, all unknown parameters (e.g., main effects, interaction effects) are treated as random variables Observed data {X, Z, Y} is considered to be fixed Whereas in the more popular frequentist approaches, the unknown parameters are fixed, and the observed data is a particular instance of a random process Bayes methods require specification of prior beliefs for the unknown parameters, in the forms of prior probability distributions

40 What is a Bayesian HTE Analysis? Like frequentist methods, they require a statistical model for data generating process Using the Bayes rule, the prior distributions and data generating model are combined to produce updated, posterior distributions for the unknown parameters Powerful computational tools are often used (e.g., Markov Chain Monte Carlo techniques) to generate samples of unknown parameters from the posterior distribution Summaries of the posterior distribution comprise the main results of a Bayesian analysis

41 Why Consider Bayesian HTE Analysis? The phrase heterogeneity of treatment effects implies an underlying distribution of treatment effects Thus a Bayesian framework is natural A Bayesian analysis does not emphasize whether a statistical procedure for detecting HTE is significant or not - an arbitrarily dichotomous decision A Bayesian approach emphasizes estimation of the magnitude of HTE

42 Why Consider Bayesian HTE Analysis? Bayesian approach can exploit prior knowledge to increase the precision of subgroup-specific effects Typically, model-based Bayesian estimates will have lesser uncertainty than estimates from separate analyses of subgroups (e.g., raw subgroup-specific effects) By sharing information across subgroups, according to the model, the Bayesian approach will stabilize the raw estimate by pulling it back (shrinkage) towards the overall treatment effect This produces estimates with lower mean-squared error.

43 Why Consider Bayesian HTE Analysis? Bayes approach supports simple and direct probability statements about subgroup-specific effects For example, we can ask: what is the probability that treatment A is better than treatment B for women? Or Do men benfit more from treatment than women? Such summaries can be readily understood by patients and other stakeholders Frequentist approach, however, only permits statements about the likelihood of observed data, under the hypothesized value of the effects

44 Simple Bayesian Models

45 Simple Shrinkage

46 Simple Shrinkage Prior : θ 1,..., θ K μ, ω Data : Y k θ k, σ 2 k Posterior : θ k μ, ω, Y k N(μ, ω 2 ) N(θ k, σ 2 k ) N(.,.) ω2 Posterior mean of θ k = μ + σ 2 k + (Y ω2 k μ) Hyper-priors for μ, ω, and σ 2 k (= σ2 /n k ) μ N(0, BIG) ω N 1/2, Gamma, IG log(σ) Unif ( a, a)

47 Simple Shrinkage.

48 Regression Prior : θ k X, β, ω Data : Y k θ k, σ 2 k Posterior : θ k μ, ω, Y k N(β 0 + X k β, ω 2 ) N(θ k, σ 2 k ) N(.,.) X k are binary/categorical Hyper-priors for β, ω, and σ 2 k (= σ2 /n k ) β 0, β N(0, BIG) ω N 1/2, Gamma, IG log(σ) Unif ( a, a)

49 These models and a number of other Bayesian regression models can be fit using BEANZ software

50 Reading List 1 Berry DA (1990). Subgroup analyses. Biometrics. 46: Dixon DO and Simon R (1991). Bayesian subset analysis. Biometrics. 47: Jones HE et al. (2011). Bayesian models for subgroup analysis in clinical trials. Clin Trials. 8: Spiegelhalter DJ et al. (2004). Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Wiley.

51 Module 6 Treatment Effect Scale and Qualitative Interactions

52 HTE Depends on Effect Scale Table. Probability of an adverse event Z = 0 Z = 1 X = X = Z is the treatment and X = 0 and X = 1 are the subgroups. Relative risk reduction is same 60% for both subgroups - no HTE But, absolute risk reduction is greater for X = 1 (0.12 versus 0.06) - substantial HTE

53 HTE Depends on Effect Scale Potential responses: {Y i (Z = 0), Y i (Z = 1)}. Individual treatment effect is a contrast b/w potential responses: e.g., θ i = Y i (1) Y i (0), or θ i = Y i (1)/Y i (0). HTE present iff: θ i = θ j, for some i, j. Homogeneity on relative scale does not imply homogeneity on difference scale - this would happen when Y i (0) varies

54 Removable Interaction Interaction is removable if a transformation of Y induces additivity For continuous Y, a nonlinear variance-stabilizing transformation, h(y), can be found (Cox, ISR 1984) (interpretation is important!) Removable interactions commonly known as quantitative interactions More difficult for binary Y Qualitative interactions are scale invariant - essential interactions

55 On which scale should HTE be evaluated? The scale in which the model best fits the data?! The scale that provides a parsimonious model, i.e. without interaction terms?! The traditional scale (e.g., logit)?! In any case, the results should also be reported in terms of absolute difference on the basic scale, i.e. difference in means or risk difference Since the absolute difference scale is easy to interpret and also is more pertinent for policy or treatment decision purposes

56 Quantitative and Qualitative Interactions quantiative HTE = presence of any HTE qualitative HTE = benefit in some and harm in some subgroups

57 Gail-Simon Test for Qualitative Interaction True treatment effects in K mutually-exclusive subgroups: = {δ 1,, δ K } Positive orthant, O + = { : δ i 0, i}; negative orthant, O = { : δ i < 0, i} Null H 0 : O + O

58 Qualitative Interaction: Gail-Simon Test

59 Gail-Simon Test for Qualitative Interaction Q = I(D k > 0)(D 2 k /s2 k ) > c ; k Q + = I(D k < 0)(D 2 k /s2 k ) > c, k where D k is the treatment effect in subgroup k and s 2 is its variance. k Likelihood ratio test: min(q +, Q ) > c The critical value c is determined so as to get the desired probability of rejection under H 0

60 Gail-Simon Test: An Illustration NSABP Data: Treatment effect measured as difference in disease-free proportion at 3 years Age < 50 Age > 50 Age < 50 Age > 50 PR < 10 PR < 10 PR 10 PR 10 PF PFT PF PFT PF PFT PF PFT Proportion disease-free 0 at 3 years SE Effect, D i se, s i D 2 i /s2 i Gail Simon test statistics: Q + = 10.89, Q = 4.28 Critical value (at α =.05) = 5.43 M Gail and R Simon (Biometrics, 1985)

61 Gail-Simon Test: An Illustration Re-analysis with treatment effect measured as log hazard ratio Age < 50 Age > 50 Age < 50 Age > 50 PR < 10 PR < 10 PR 10 PR 10 Effect, D i se, s i D 2 i /s2 i Gail Simon test statistics: Q + = 14.38, Q = 6.54 Critical value (at α =.05) = 5.43 M Gail and R Simon (Biometrics, 1985)

62 Reading List 1 Cox DR (1984). Interaction. Intl. Statist. Rev. 52: Gail M and Simon R (1985). Testing for Qualitative Interactions Between Treatment Effects and Patient Subsets. Biometrics. 41: vanderweele TJ and Knol MJ (2014). A tutorial on interactions. Epidemiol. Methods. 3:

63 Module 8 HTE Analysis Plan and Results Reporting

64 Main Issues to Think About What is the goal of the HTE analyses? How will the HTE findings inform patient-centered healthcare decisions? How should I conduct HTE analyses? How should I report HTE results?

65 PCORI Standards for HTE 1 HT-1: State the goals of HTE analyses 2 HT-2: For all HTE analyses, Pre-Specify the analyses plan; for hypotheses driven HTE analyses, pre-specify hypotheses and supporting evidence base 3 HT-3: All HTE claims must be based on appropriate statistical contrasts among comparison groups, such as interaction tests or estimates of differences in treatment effect 4 HT-4: For any HTE analysis, report all pre-specified analyses and at minimum, the number of post-hoc analyses, including all subgroups and outcomes analyzed

66 Standards for HTE HT-1: State the goals of HTE analyses Most important standard - demonstrates clarity of purpose Different types of goals: 1 confirmatory (hypothesis-driven); 2 exploratory (hypothesis-generating); 3 assessing consistency of treatment effect in univariate subgroups ; 4 reporting results on important pre-specified subgroups (descriptive); 5 identification of groups most/least likely to benefit (discovery); 6 other goals

67 Standards for HTE HT-1: State the goals of HTE analyses Clearly identify each set of analyses as confirmatory, exploratory, discovery, etc. Confirmatory analysis requires a greater burden of proof Consistency analysis is common in RCT reports Sex-specific subgroup analysis (e.g., FDA-CDRH, JNCI, IOM report) Discovery analysis is aimed at identifying extreme subgroups

68 Standards for HTE HT-2: Pre-Specify hypotheses, evidence base and analyses plan Pre-specification is a critical aspect of HTE analyses Demonstrates pre-thought (really speaks to study design) Sharply defined hypotheses, including the direction, for confirmatory Provides a sampling basis for valid inference (eliminates random-high bias) A thorough analytic plan required for confirmatory Independent validation essential for discovery analysis Description of treatment selection bias adjustment for observational HTE

69 Standards for HTE HT-3: Use appropriate statistical methods HTE cannot be claimed based on signficance of subgroup effects Test of interaction is a valid approach Use of model-based estimation for multiple effect modifiers Use of prognostic score (e.g., Framingham risk score) as effect modifier Bayesian shrinkage/regression estimation of subgroup effects

70 Standards for HTE HT-4: Reporting HTE results Boonacker (AJE 2012) compared HTE analyses in grant applications to the related journal publications Discrepancy between study protocol/grant application and final publications Non-reporting of prespecified subgroup analyses and reporting of post-hoc subgroup analyses very common Even worse, 77% of the analyses didn t mention whether they were prespecified or post-hoc Justification for subgroup analyses and analytic methods were seldom reported

71 Standards for HTE HT-4: Reporting HTE results Transparency is key here Protocols and study reports describe exact procedures - both pre-specified and data-driven analyses Examples include: 1 number of endpoints examined 2 transformation of reponse variable and covariates 3 categorization of continuous covariates 4 data-mining or model selection approaches 5 type of adjustment for multiple testing 6 prior distribution and posterior computation for Bayesian methods Forest plot of subgroup effects, with adequate information, is visually appealing

72 Take-Home Points HTE analysis is challenging: rife with false positive and false negative findings Reliability enhanced by following two key principles behind HTE standards: pre-specification and transparency Standards are not prescriptive - provide guidance for sound study design, analysis, and reporting of HTE assessments. How will the HTE findings inform patient-centered healthcare decisions?

73 Reading List I 1 Berry DA (1990). Subgroup analyses. Biometrics. 46: Cox DR (1984). Interaction. Intl. Statist. Rev. 52: Dixon DO and Simon R (1991). Bayesian subset analysis. Biometrics. 47: Fleming TR (2010). Clinical trials: discerning hype from substance. Ann Intern Med. 153: Gabriel SE and Normand SLT (2012). Getting the methods right - the foundation of patient-centered outcomes research. NEJM. 367: Gail M and Simon R (1985). Testing for Qualitative Interactions Between Treatment Effects and Patient Subsets. Biometrics. 41:

74 Reading List II 7 Jones HE et al. (2011). Bayesian models for subgroup analysis in clinical trials. Clin Trials. 8: Kent DA, et al. (2010). Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 11: Kovalchik S et al. (2013). Assessing the heterogeneity of treatment effect in a clinical trial with the proportional interactions model. Stats. Med. 32: Rothwell PM (2007). The Lancet: Treating Individuals: from randomised trials to personalised medicine. Elsevier. 11 Spiegelhalter DJ et al. (2004). Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Wiley.

75 Reading List III 12 vanderweele TJ and Knol MJ (2014). A tutorial on interactions. Epidemiol. Methods. 3: Varadhan R et al. (2013). A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research. J Clin. Epidem. 66: