Adaptive designs: When are they sensible?

Transcription

1

2

3 Adaptive designs: When are they sensible? Janet Wittes Statistics Collaborative, Inc. Sensible Guidelines May 2012

4 The pitch Clinical trials are too rigid There are new classes of designs More ethical More efficient Adaptations saves money, time, etc. But, we need to be adaptive by design

5 Jerry Cornfield s admonition The statistician functions as a devil s advocate against the admission of new evidence, and, in this capacity, has an important influence on the quality and cogency of the evidence submitted. The investigator must pay close attention to this advocate, but it is his and not the advocate s responsibility to decide when he must stop listening. Am J of Mental Deficiency 64: (1959)

6 What s wrong with the pitch Not really so new

7 What s wrong with the pitch Not really so new Doesn t always save time or money May be very much less efficient I don t know what ethics means

8 What s wrong with the pitch Not really so new Doesn t always save time or money May be very much less efficient I don t know what ethics means Adaptive by design good in general Don t always need to design up front Should Columbus have declared the islands India?

9 Four kinds of adaptations Operational Fix nuisance parameters (internal pilot) Change hypothesis: external data and thoughts Change hypothesis: internal data

10 Operational Titrating the design Adding and dropping centers Encouraging adherence to study protocol Changing entry criteria

11 Not really so new Molière s Bourgeois Gentilhomme Good Heavens! For more than forty years I have been speaking prose without knowing it. M. Jourdain

12 Four kinds of adaptations Operational Fix nuisance parameters (internal pilot) Change hypothesis: external data and thoughts Change hypothesis: internal data

13 External vs. internal pilots External pilot Do a prior study prior to real study Use knowledge gained to design real study Internal pilot (aka adaptive design ) Start a real study Tweak it on basis of early data

14 External pilot studies Feasibility Questionnaires & instruments Assays Etc. Parameters Variances Event rates

15 The paranoid s view of external pilots Pilot populations: small, relatively homogeneous Therefore estimates not reliable Pilot variances too small Pilot rates and hazards are not accurate Safety: can only see frequent events Efficacy: only hope - look at surrogates & markers

16

17 Internal pilot Start study with unanswered design questions What is the variance? The drop-out rate? The (pooled) cross-over rate? The event rate? The effect size? Look at early data and change design parameters

18 The variance or rate unknown Sample size proportional to σ 2 Part-way recalculate sample size using new s 2 Worries Blinded/unblinded? When should we look? Too early, our s 2 is too imprecise Will it preserve Type I error rate?

19 Nuisance parameter (s 2, p, ) unknown Blinded/unblinded: Methods for both When to look: ~ ½ way is not a bad time Preserve Type I error rate? Methods available Conclusion: methods not controversial My experience: FDA and EMA are fine with this Hard to include in grant because of time

20 Four kinds of adaptations Operational Fix nuisance parameters (internal pilot) Change hypothesis: external data and thoughts Change hypothesis: internal data

21 Change the 1 outcome or analysis SHARP originally set out to assess the effects of lowering cholesterol on the risk of major vascular events, defined as major atherosclerotic events plus other cardiac causes of death and strokes due to bleeding. But, whilst the trial was ongoing, new information became available from other completed studies that suggested that lowering cholesterol might have little effect on these additional events. Before breaking the trial s code, therefore, the independent Steering Committee decided that analyses of major atherosclerotic events would provide the most valuable information to patients and doctors. This change was described in a paper that was published in a leading medical journal before the trial s code was broken. Taking the combination of ezetimibe and simvastatin long-term reduced the risk of heart attacks, strokes and operations to open blocked arteries by about one quarter in people with chronic kidney disease, irrespective of the severity of their disease

22 Changing primary outcome Do with trepidation There is a certain incertitude concerning the primary endpoint (apparently changed by the Steering committee before the trial ended but this change was not endorsed by the sponsor).

23 Other similar changes. Change method of analysis E.g., way of handling missing Switch from change score to longitudinal analysis

24 Four kinds of adaptations Operational Fix nuisance parameters (internal pilot) Change hypothesis: external data and thoughts Change hypothesis: internal data on effect size

25 Effect size-start with a hypothesized Frequentist Look part-way through (define when; infrequent) Calculate observed and recalculate sample size Bayesian Look part-way through (can be often) Calculate predictive power Decide whether to stop or add more

26 Problems with frequentist approach The methods can be very inefficient Data after the adjustment is downweighted Downweighting can be serious Early data imprecise Does have a clinical meaning? If we have chosen rationally, is smaller ok?

27 Problems with Bayesian approach In ordinary sequential If you don t cross boundary, you can t declare success If you do cross boundary, you don t have to stop Then you recoup you re α In Bayes, must you follow the rules? True Bayesians: no FDA: yes My experience: Bayes adaptation black boxy

28 So, the dilemma External pilot studies (Phase 2) are scientifically Important and necessary But boring Hard to fund Internal pilot studies Effective in tweaking parameters Not always accepted

29 The interesting methods (i.e., the ones popularly called adaptive ) Fundamentally change the hypothesis If we believe in Δ Are statistically inefficient Even if the new estimate is correct But it s subject to a lot of variability Require extremely careful protection of the blind

30 Conclusions I Failure to adapt when necessary risks failure Operational: doesn t need plan Nuisance parameters: probably need intent Response to outside data: cannot have plan

31 Conclusions II Failure to adapt when necessary risks failure Operational: doesn t need plan Nuisance parameters: helps to have plan Response to outside data: cannot have plan Adapting on the basis of observed effect Adds power at the expense of Large increase in sample size Changing hypothesis

32 Conclusions from a Neanderthal The questions we are asking are important! Many adaptive designs trade speed for precision We should be aiming for reasonable precision A good understanding of risks and benefits Adaptive trials are VERY operationally complex