Adaptive Randomization:

Transcription

1 : Promise with Problems Russell W. Helms, Ph.D. Rho, Inc. Chapel Hill, NC Presented to The Annual DIA Workshop on Statistical Methodology in the Biopharmaceutical Sciences Philadelphia, PA 03 March 2004 Before we get started: STAT SIAC SIAC: Special Interest Area Community DIA reorganizing: grassroots Newly functioning Needs Assessment Volunteers? Website, newsletter, networking, programming, training Talk to me! me: DIA Copyright 2004 Rho, Inc. 2 : Promise with Problems Agenda Introduction: What and Why? Types of and Inference Advice: What (not) to Do in Practice Conclusions Special Thanks Rosenberger and Lachin (2002) McEntegart (2003 DIJ) : What & Why? What is randomization? The act of assigning treatments to observational units using (an element of) chance. and why do we do it? two adequate and well controlled studies RCT is the gold standard Control bias, basis for inference is necessary to control bias, but not sufficient Related topics: controls, masking Copyright 2004 Rho, Inc. 3 Copyright 2004 Rho, Inc. 4 Controls: What & Why? Masking: What & Why? Now a gold standard, but Salk: I found but one person who rigidly adhered to the idea of a placebo control and he is a bio-statistician who, if he did not adhere to this view, would have had to admit his own purposelessness in life. FDA helps Placebo effect, paper challenging it Better than what? New information is obtained from a comparison of alternate states Association and causation Masking = Blinding = Veiling Masking helps control sources of bias Selection bias: investigators may be able to bias composition of treatment groups by choosing (not) to enroll based on what treatment a particular patient may receive Subconscious preference more likely than deliberate dishonesty e.g., Good prognosis versus bad prognosis e.g., Likely to respond to this treatment Worse: investigators (or subjects) may grade outcomes differently Eliminates placebo effect? Copyright 2004 Rho, Inc. 5 Copyright 2004 Rho, Inc. 6

2 Masking: What & Why? Masking is not always possible. Emergency unmasking Surgery (Story: arthroscopic surgery trial for osteoarthritis) Even when planned, patients and investigators are smart, and discern. Side effects, lab tests, response (Story: Good Friday experiment) Plenty of examples in literature, my experience Implication on randomization and guessing : What & Why? Why random allocation? 1:Control Bias (throughout) e.g., selection bias Story: Teachers switching treatment groups From Chalmers et al., 1977: 32 Studies of anticoagulants for acute MI. Mortality improvement in: 15/18 studies using historical controls (900 pts), 5/8 studies using nonrandomized concurrent controls (3,000 pts).» Pooled: 50% reduction in mortality 1/6 RCT (3,800 pts).» Pooled: 20% reduction in mortality Bias > effect? Large effect versus small effect: Other publications show examples of inappropriately favoring new interventions Copyright 2004 Rho, Inc. 7 Copyright 2004 Rho, Inc. 8 : What & Why? : What & Why? Why random allocation? 1:Control Bias (throughout) e.g., accidental bias from covariates/confounders. Imbalance of important prognostic factors basic problem of observational studies e.g., Smoking and Lung Cancer (Fisher) Story: NCI HIP RCT: PE + Mammography recommendations lowered breast cancer mortality. But within test group, those who refused exam had lower breast cancer mortality.» Self-selection confounding by education level. tends to produce comparable groups: both measured and unknown prognostic factors. Size and direction of imbalances distributed over trials Small trials may have imbalances Leap from association to causation Why random allocation? 2:Basis for inference (section later) Issues specific to RCT in Drug Development Sequential appearance of potential subjects Inclusion/Exclusion criteria Multicenter trials Phases with different goals Implemented by very intelligent people Implemented by people with (sometimes poor) judgment Story: investigator too busy to call CIRS Rescue Story: investigators forgot ==> multiple randomizations Studying people with independent will: plan for some chaos Copyright 2004 Rho, Inc. 9 Copyright 2004 Rho, Inc. 10 : What & Why? Four Types of Given (what I ve just explained about) why we randomize, what do we seek in a randomization? Control bias (selection, accidental, etc.), basis for inference Stochastic versus deterministic e.g., alternating sequence (ABAB, BABA ) isn t randomization ICH E9: random element in all assignments What does adaptive mean? At simplest level, one treatment assignment is at least partially dependent on another Strict definition: can t generate a complete schedule before the first patient is randomized Complete randomization Simple coin tossing Restricted randomization (e.g., permuted blocks) Complex coin tossing: add restrictions on the coin and how it is used to ensure balance of treatment arms Covariate-adaptive randomization (e.g., Pocock and Simon) Used to ensure balance of treatment arms with respect to certain known covariates Response-adaptive randomization Treatment assignments depend upon previous patient responses Copyright 2004 Rho, Inc. 11 Copyright 2004 Rho, Inc. 12

3 Types of : Complete Complete randomization Simple coin tossing Not adaptive No selection bias: impossible to predict Chance of (possibly severe) treatment imbalance: one treatment may be assigned more frequently than another at any point in the trial, including the end. Why want equal (balanced) allocation? Max power: generally not much of an issue, but extreme imbalances, while rare, can be a serious problem. (Seek balance for insurance.) Equipoise and ethics Facilitates CTM supply management Chronological bias: patients differ over time of enrollment e.g., epidemics in vaccine or anti-infective trials Types of : Restricted Restricted randomization Complex coin tossing: add restrictions on the coin and how it is used to ensure balanced allocation of treatments. Very common. Different assignments are somewhat dependent: Adaptive at simplest level, but not by strict definition Examples: Random Allocation Rule: Urn drawing without replacement. Avoid because: Deterministic in late stages: selection bias, chronological bias, inference Truncated Binomial design Binomial until one treatment group full. Avoid because: Deterministic in late stages: selection bias, chronological bias, inference Possibility of severe covariate imbalances and resulting accidental bias Permuted blocks (more on next page) Copyright 2004 Rho, Inc. 13 Copyright 2004 Rho, Inc. 14 Types of : Restricted Permuted Block design: most common in our industry Promise: Balance at end and throughout: every n-th subject. [BABA] [AABB] [ABAB] Important when: time-heterogenous covariate related to treatment outcome: avoid chronological bias interim analyses are planned The trial might be stopped early. Problems: Overall imbalance possible with incomplete blocks inference and wasted data (more later) Large element of determinism (every block) selection bias: secret and/or variable block size (worse?) determinism and inference (potentially small reference set) Types of : Restricted (Generalized) Biased Coin Designs Examples Effron s Biased Coin design Vary p from ½ when balance gets bad Only two values of p (1/2 or other) Some more deterministic modifications aim to prevent extreme imbalance Wei s Urn design Adaptive biased coin design: p varies by degree of imbalance Urn with replacement plus a few Promise: balance, difficult to predict (selection bias) Problems: analysis complications, chronological bias Copyright 2004 Rho, Inc. 15 Copyright 2004 Rho, Inc. 16 Types of : Restricted Maximal Procedure (Berger, Ivanova, Knoll, 2003) Relatively new, relatively untried Maximal reference set subject to maximum tolerated imbalance condition Focus on terminal balance, maximum tolerated imbalance, and ignore forced returns to exact balance (as in permuted blocks) Can implement with complete randomization and replacement randomization (discarding sequences that don t fit constraints) Promise: controls balance (chronological bias), difficult to predict (selection bias), not necessarily deterministic Problems: untested, explaining to investigators, IRBs, etc. Types of : Restricted Restricted randomization summary: Complex coin tossing: add restrictions on the coin and how it is used to ensure balanced allocation of treatments. Different assignments are somewhat dependent: Adaptive at simplest level, but not by strict definition Permuted blocks is the most common, maximal procedure may be better Copyright 2004 Rho, Inc. 17 Copyright 2004 Rho, Inc. 18

4 Types of : Covariate- Often covariates or prognostic factors of interest besides treatment affect outcome. e.g., site in a multi-center trial is often the greatest source of variability e.g., infections: nosocomial versus community-acquired, resistant versus not e.g., baseline severity of disease Arthritis examples: Duration of illness: 2-3 years versus 10+ years (potentially continuous) Prior treatment history Accidental bias may result from imbalances on such factors. Especially true for permuted block randomization, as the asymptotic decline of accidental bias is slow for that design (Rosenberger and Lachin) Types of : Covariate- Covariates besides treatment affect outcome: Solution 1: Stratification = Separate randomization for each level of factor, with any randomization method so far discussed. Promise: stratification can protect from accidental bias (known covariates + correlates) and also increase power. Note: stratification on randomization requires stratification in analysis; stratification in analysis is possible without stratification in randomization, similar power gains Problems: treatment balance within strata may reduce overall balance Number of strata increases rapidly and can make overall balance impossible Copyright 2004 Rho, Inc. 19 Copyright 2004 Rho, Inc. 20 Types of : Covariate- Covariates besides treatment affect outcome: Solution 2: Covariate-adaptive randomization of subject # (k+1) depends on balance among first k subjects, assign k+1 the treatment that minimizes the imbalance Zelen s Rule: Use schedule but swap when balance bad Wei s marginal Urn design: similar to urn design with urns for margins of covariates, instead of an urn for each stratum Optimal Design: Based on linear model, minimize variance of estimated treatment effect in presence of covariates. Note different than balancing over covariates to mitigate bias. Difficult to implement and to explain to nonstatisticians. Types of : Covariate- Pocock and Simon (sometimes called minimization ) Independent generalization of Taves; incorporates stochastic element. Measures imbalance on each factor then combines to get an overall measure of imbalance associated with each treatment. Bad imbalance implies a high probability (e.g., biased coin) of assigning a treatment. Tip: Unlike stratification, can account for many levels of strata. Metrics are arbitrary, but intuitive. Little research on properties, but used with some frequency. Tip: simulation to discern properties for each implementation Tip: Balances margins, not cells. Series of adaptations: Begg & Iglewicz, Atkinson, Frane (lead to Optimal ) Story: Site-level imbalance in anti-infective. Tip: Imbalance measure based on variance better than based on range: better avoid extreme imbalances Tip: Include factors in analysis Copyright 2004 Rho, Inc. 21 Copyright 2004 Rho, Inc. 22 Types of : Covariate- Promise: Insurance: extreme covariate imbalance; while unlikely, may make a trial difficult/impossible to interpret. Enhances credibility for less statistically sophisticated reviewers who may misunderstand or mistrust model-based adjustments for imbalance Increases precision and interpretability for interim analyses and early-stopped trials Avoids analysis problems of empty cells, and Handles very many factor levels Types of : Covariate- Problems: Implementation, analysis can be complicated (inference) Possibility of selection bias increases as p 1 Intuitive method, properties not (yet) fully explored Doesn t guarantee balance of unmeasured covariates (accidental bias), though simulation studies suggest it may help (Rosenberger & Lachin) Balance may be overrated Copyright 2004 Rho, Inc. 23 Copyright 2004 Rho, Inc. 24

5 Types of : Covariate- In general, The more stochastic, the better ICH E9 Guidance: Use random element in all assignments. Forcing treatment balance among covariates may be overrated: be quite sure it s necessary before stratifying or implementing covariate-adaptive procedure. EXCEPTION: Site Many cells: stick with modified P&S: stochastic (e.g., p=0.8) great balance and difficult to predict, but unknown properties Comparison to permuted block design: P&S less chance of selection bias as long as p < 1 P&S less deterministic as long as p < 1 P&S may be better at preventing accidental, chronological bias Types of : Response- Treatment assignments depend upon previous patient responses, e.g., Play-the-winner : highly deterministic Two-armed bandit : more like generalized biased coin Promise: Ethics: maximize number of patients on superior treatment Problems: Requires rapid ascertainment of patient outcomes Lower power (larger sample size offsets ethical advantage) Chronological bias. Story: ECMO example (1 F, 9S, Stop) Analysis complicated: inference next slide! Comment: Other CT designs (NonStop designs, interim analyses) can accomplish similar goals Copyright 2004 Rho, Inc. 25 Copyright 2004 Rho, Inc. 26 and Inference and Inference: Population Model A goal of randomization is to aid in inference from the trial. Two methods: Population-Model Tests Editorial: Practical, simulation-supported hand-waving Permutation Tests ( model) Editorial: Mathematically rigorous complications Population A: Y ~ G(y θ A ) Sample at Random n A Subjects: Y Aj ~ G(y θ A ) Population B: Y ~ G(y θ B ) Sample at Random n A Subjects: Y Bj ~ G(y θ B ) H 0 : θ A = θ B Note: Adapted from (Lachin, 1988) Copyright 2004 Rho, Inc. 27 Copyright 2004 Rho, Inc. 28 and Inference: Model n eligible & consenting subjects, presenting sequentially n A subjects n B subjects H 0 :Outcomes evenly distributed among n A, n B Note: Adapted from (Lachin, 1988) Copyright 2004 Rho, Inc. 29 and Inference: Approaches address different questions (null hypotheses). Permutation: Probability that observed effect (or more) in n patients actually randomized into the trial could have occurred by chance Allows conclusions about effects of treatment in patients actually studied. Population: Probability that at the observed effect (or more) in n patients studied could have been observed in samples of n A and n B drawn at random from respective populations. Allows conclusions about effects of treatment in hypothesized general population. Thus generalization may be different, but this is really a bigger issue of trial design (inclusion/exclusion criteria, etc.) Copyright 2004 Rho, Inc. 30

6 and Inference: Approaches address design differently: Permutation tests: assumes outcomes fixed, treatment assignment is a random variable Population model: treatment (populations) fixed, outcome is a random variable Approaches have different assumptions: Permutation tests require virtually no assumptions. Population model requires faith in untestable assumptions. Population model based on Neyman-Pearson. Type of randomization matters for permutation tests, but is ignorable for population This bothers some in the strict analyze as randomize crowd. Power and sample size must be based on population Sometimes permutation test wastes data e.g., unfilled blocks Copyright 2004 Rho, Inc. 31 and Inference: For complete randomization, the approaches produce the same statistic. Not true for restricted or adaptive. In practice, a variety of authors have concluded it is rare to get a substantially different answer from the approaches. Population-model may be slightly conservative, but permutation test has discrete p-values. Be careful with the reference set! (At least 20 members ) No difference strict adaptive versus not: any dependency of treatment assignments makes the two approaches different. Decide whether or not you believe the population model assumptions. Apply this decision consistently: including permuted block Don t forget blocking factor in analysis, or the test is conservative. Choice of randomization details shouldn t influence inference preference Inference preference may influence randomization method Copyright 2004 Rho, Inc. 32 and Inference: Beware chaos and extreme complications: permutation tests can make our job interesting and exciting! Story: ice storm Remember other stories Interim analyses, stratification Permutation tests may be more appealing to statisticians. Theoretically sound, intuitive. Fun to program! Out-of-context Salk quotation from earlier: and he is a bio-statistician who, if he did not adhere to this view, would have had to admit his own purposelessness in life. Population tests may be more intuitive to non-statisticians. Beware inflicting our purpose in life on others needlessly. Recommendations: What (not) to Do should be stochastic, not deterministic (ICH E9). Mask vigorously. Mask all details, including block sizes, parameter values Stratification: Stratify on center (ICH E9) Keep it simple, adjust in analysis (Post-adjust whether or not pre-adjust) If you can t keep it simple, use Pocock and Simon instead. Worry more about selection bias than balance, especially when balancing covariates. Copyright 2004 Rho, Inc. 33 Copyright 2004 Rho, Inc. 34 Recommendations: What (not) to Do Rosenberger s and Lachin s recommendation: two stage inference: Tests are permutation tests Estimates based on population model My recommendation: Use population model through Phase-II. Before Phase-III, ask FDA. If permutation test is possible, be wary of the size of the reference set: don t use a design whose minimum p-value is > 0.05! Conclusions Promise? Bias control and inference. Problems? Bias control and inference. In practice, nearly all randomization designs are adaptive in the simplest sense: one treatment assignment is at least partially dependent on another. Permuted block designs, while common, suffer from many of the same problems as more adaptive designs. Adopt a method of inference faithfully regardless of randomization details. Have faith? Either you believe the population model or you don t. Population Model Analysis is independent of randomization details; Permutation Test Analysis depends on them absolutely. Permutation tests can be complicated, especially with chaos. Controlling selection bias is paramount. Copyright 2004 Rho, Inc. 35 Copyright 2004 Rho, Inc. 36

7 Conclusions The Powerpoint presentation will be available on the internet: questions, comments, etc. to: UNIX types: you don t have to use capital letters Thank you! Copyright 2004 Rho, Inc. 37