The death of the sample size calculation. Ed Mills

Transcription

1 The death of the sample size calculation Ed Mills

2 Suicide by Sample Size Calculations Ed Mills

3 A Sackett approach: Physiological Statistics: 3 Or, the only statistic people seemed to learn but needn t have Ed Mils Emills@cfenet.ubc.ca

4 The Guyatt appraoch: It Is a Delusion to Think That Sample Size Calculations of Individual Trials Matter

5 In the era of systematic reviews, does the size of an individual trial still matter? Ed Mills MSc, PhD, LLM

6 Background Funding agencies, ethics review committees, journals and others are preoccupied with power calculations It is often the only statistical element that a reviewer remarks upon This is a willing self-deception that a simple statistical calculation is one of the most important elements of a trial.

7 Background In the majority of conditions, multiple pathogenic and behavioural mechanisms that are poorly understood determine the likelihood of an event. Thus, we can only expect small treatment effects from interventions that act on one or two of these mechanisms. This is further reduced in the presence of well-conducted head-to-head evaluations Only after a trial is completed, can we determine the power achieved and power needed

8 Reality check Most investigators assume that their intervention will create a >25% RRR Yet few interventions for common conditions are so effective In reality, most investigators assume the size of the populations they can realistically enroll and then create a power calculation that will give them that ballpark figure Multiple examples exist of compromises, such as composite endpoints and surrogate endpoints.

9 Realism Almost all clinical trials are underpowered to evaluate likely effects. Mega-trials are difficult to obtain funding for. Even very large trials, such as the CAPRIE trial (Clopidogrel versus aspirin in patients at risk of ischemic events) (n>19,000) can give indefinite results (upper 95% CI, 0.3% RRR) Sometimes the results of even very large trials proves discrepant with one another

10 Study name Statistics for each study Risk ratio and 95% CI Risk Lower Upper ratio limit limit p-value ACAPS AFCAPS/TexCAPS ALLHAT-LLT ASCOT-LLA ASPEN CAIUS CARDS FAST KAPS MEGA PREVEND IT PROSPER WOSCOPS PHYLLIS Mohler ALERT PMSG Favours statin Favours control Figure 3. Random-Effects Meta Analysis: Cardiovascular mortality

11 What is the impact of unreal expectations? Many trialists abandon efforts as they are unable to get funded Funding agencies refuse to fund because another similar trial exists Ethicists consider this unethical and thus studies are unapproved. Journals reject as underpowered even after the trials are completed

12 What can we do? At the heart of rigorous decision-making is the hope that information is available from multiple sources Thus, the systematic review/meta-analysis of all trials Meta-analysis, by their nature, include data from all available trials, not according to sample size. Meta-analysis of multiple trials allows one to inspect and infer not only a threshold p-value of effect or not, but the consistency of the trial effects, the size of the trials and effects, and the arbitrary decision-making that the data is convincing or not.

13 As trialists We should be informed decision-makers What are the realistic effects of my intervention? Can I collaborate to increase my sample size? Is my question important enough to get funded and have I convinced funders of the absence of evidence? Can I improve power, without increasing my sample size?

14 Small sample sizes can give more precise results than large sample sizes Consider two studies, both with a 33% RRR, a reduction of 1/3 of adverse events Study A has 100 patients in total Study B has 1000 patients in total Study A has 20 vs 30 events (RRR 33%, 95% CI, -8 to 59%) Study B has 2 vs 3 events (RRR 33%, 95% CI, -233 to 87%) Which study provides stronger inferences?

15 Improving power within a trial Higher risk populations Increase study duration More effective intervention *

16 In conclusion Most trials are underpowered Power calculations are usually misleading estimations Multiple trials provide the most compelling evidence Increasing power is not simply an issue of number of participants

17 We have a duty To inform funders and regulators Trial sample size is rarely enough The largest manageable trials should be our goal The deception of power calculations needs to be realized and admitted To move away from expectations of power calculations and towards important questions within representative populations