P values From Statistical Design to Analyses to Publication in the Age of Multiplicity

Transcription

1 P values From Statistical Design to Analyses to Publication in the Age of Multiplicity Ralph B. D Agostino, Sr. PhD Boston University Statistics in Medicine New England Journal of Medicine March 2, 2017

2 Introduction Controlling for multiple testing in a clinical trial is a major area of concern Why Concern? Multiple testing within a clinical trial can increase probability of false positive study or Family Wide Error Rate (FWER) Need to control this probability at desired significance level In publications need to present appropriate p value(s) for correct interpretation As an academic researcher I am often involved with the FDA, Industry, Statistics in Medicine and NEJM to address this. Emphasis today is what belongs in Journal publication 2

3 What are the Topics/Areas to consider for Publication? 1. Multiple Endpoints 1. Multiple primary 2. Multiple secondary 2. Interactions (e.g., Center by Treatment) 3. Composite Outcomes 4. Subgroup analyses and interactions 5. Multiple treatment comparisons PROTOCOL and STATISTICAL ANALYSIS PLAN SHOULD ADDRESS ALL 3

4 A Little Personal History: FDA: First Exposures to Multiplicity IPPD: Separate symptoms (fullness, bloating, distension, gas) followed one at a time. We said change to overall relief Blood pressure: Reduction of DBP to following it over day Pain: Pain Intensity every half hour to SPID (sum) and/or reduction of pain significant at every time (Sportscreme at 30 and 60 m.) 4

5 1. Multiple Endpoints 1.1.Multiple Primary Outcomes Simple Conventional Strategies exist for controlling error rates Powering study for 2 primary endpoints If win criterion is both endpoints need to show significant treatment effect: Each endpoint tested at 0.05 level controls FWER at 0.05 max. Power each endpoint at 80% using 0.05 significance level? Probability of win is then between 64% and 80% Power each endpoint at 90% (using 0.05 significance level) to ensure at least 80% probability of win. General Approach: Desired probability of false positive study = α Desired probability of study win = δ Power each endpoint at δ at α significance level 5

6 Multiple Primary Endpoints Powering study for 2 primary endpoints If win criterion is at least one endpoint needs to show significant treatment effect: Bonferroni: Each endpoint tested at level controls FWER (probability of false positive study ) at 0.05 max. Power each endpoint at 80% (using significance level)? Probability of win is at least 80%. General: Desired probability of false positive study = α Desired probability of study win = δ Power each endpoint at of δ using α/2 significance level 6

7 Multiple Primary Endpoints Example - DAPT Study: Study to assess effect of Dual Antiplatelet Therapy (DAPT) in patients receiving cardiac stent All patients receive 12 months of DAPT following stent implantation After this 12 month run-in, eligible patients randomized to further 18 months of DAPT or placebo Two primary endpoints: Incidence of stent thrombosis Incidence of Major Adverse Cerebral and Cardiovascular Events 7

8 Multiple Primary Endpoints Example - DAPT Study (continue): Benjamini - Hochberg approach used: Study is a win if both endpoints significant at 0.05 level or if at least one is significant at level. Strongly controls FWER in 2-endpoint situation For overall power of 80%, power each endpoint at 80% using level of significance for each, or power each endpoint at 90% using 0.05 level of significance for each. 8

9 Some Methods to Control FWER Sequential Testing Bonferroni Holm Hockberg Selection depends upon belief in variables and their significance and possibly correlation structure (see D Agostino and Russell in Encyclopedia of Biostatistics and SiM review and tutorials) 9

10 Multiple Primary Endpoints Li, Sankoh, D Agostino (2010) approach Group variables Example: Primary Biliary Cirrhosis 4 endpoints: cholate injury portal inflammation piecemeal necrosis fibrosis Separate endpoints into 2 families E.g., cholate injury, portal inflammation, necrosis in family 1, fibrosis in family 2 10

11 Multiple Primary Endpoints Li, Sankoh, D Agostino (2010) approach Example: Primary Biliary Cirrhosis Test endpoints in family 1 at α1 < α = 0.05 (e.g., use Bonferroni) If at least one significant result, test family 2 at α, otherwise test at, e.g., α α1, α1 =.0167 Controls FWER at 0.05 Family 1 p-values: 0.163, 0.025, Family 2 p-values: Bonferroni rejects H 0 for one endpoint in Family 1 Use 0.05 level of significance for Family 2:Reject H 0 11

12 2. Interactions (e.g., Treatment-by-Center) Primary Variables Significant Often suggest significance level to be 0.10 or 0.15 for each endpoint to circumvent low power issue Often no adjustment given for multiple comparisons Increases probability of false significant interaction, (but again, low power) Often not a major problem; if for example p-value = 0.12, often descriptive analysis shows not a marked interaction effect 12

13 Treatment by Center Interaction In many international studies there is concern that U. S. looks different than the rest of the study countries or locations How to anticipate this, test for it and explain it? At a Cardio-Renal panel was not as good as another drug except in US: aspirin use? 13

14 3. Composite Outcome Why go composite? Interest in Many Outcomes Sample size requirements for Individual Outcomes very large (Unrealistic) while sample size for composite outcome realistic Components have similar effects (biology) Example: Statin Trials: Composite is time to first CHD death, MI CV death, Myocardial infarction, Stroke (TNT study) CV death, MI, Stroke, CHF 14

15 Composite Outcome Issues of Concern 1. What if results are driven by specific components (What does significance mean) 2. Is consistency of components enough to declare all components meaningful 3. Do results apply to all components, to death 4. Is significant effect related to entry groups 5. How to obtain good data on components 6. Effect of Follow Up time 7. Components go in opposite directions (ACCORD) 15

16 Composite Outcomes Often Sponsor use composites, but want to declare clinical significance (claim in label) that each component is significant Data and analysis often does not justify this NEJM 2013 Mediterranean Diet. Composite: CV Deaths, MI, Stroke (Composite p<0.05, but Stroke alone p<0.05) 16

17 1.2.Secondary Endpoints D Agostino (2000) categorized secondary variables into six groups 1. Supply background and understanding of primary variables 2. In trials with composite endpoints, secondary separate endpoints 3. Important as primary but for which there may be low power 4. Aid in understanding the mechanism of action of a drug E.g., in a study on cardiovascular mortality, assess effect of treatment SBP, DBP, cholesterol level; perhaps reduction in these parameters is reducing cardiovascular mortality 5. Relate to subhypotheses that are important to understand but are not the major effect of treatment. 6. Variables for exploratory analysis Potential for learning something new Perhaps lay the groundwork for future confirmatory trials 17

18 Secondary Endpoints If win on primary, proceed to secondary endpoints If significant secondary endpoints to be included in label or marketing material, FDA has requested FWER control, usually at 0.05, using, e.g., Sequential testing Bonferroni Holm If secondary are exploratory, no alpha control necessary. What should be done about stating p values? 18

19 Secondary Endpoint What if primary not significant yet an important secondary has a p < 0.05? Should not claim significance What if it is death? Example: Carvediolol for CHF Primary (exercise ability) outcome not significant Mortality (secondary variable significant) Approve drug? What if primary significant and death in the wrong direction? 19

20 Secondary Endpoint: Bad Practice What I see too often Submission to Journal (e.g., NEJM) Primary outcome not significant Secondary which should be at best exploratory are stated to be exploratory but then presented with enthusiasm and with uncorrected (unadjusted) p values 20

21 4.Subgroup Analysis 1. Primary Hypothesis is for an overall statistically significant effect (all data combined) 1. What if overall test is significant? Move forward! 2. What if overall test is not significant? 2. Primary Hypothesis anticipates subgroup effects 3. Exploratory analysis for subgroups part of analysis 21

22 4.1 Primary Hypothesis is for an overall statistically significant effect Primary hypothesis is that two treatments are significantly different If yes to above, concern then is that there should be consistently seen in all relevant subgroups (THIS IS A SECONDARY ANALYSIS. It could need P value control if hypothesis driven) If no consistency, then what? 22

23 Significance: T 1 better than T 2 Events Plots over Study Time 1.00 MENT 1.75 TtreatmentEATMENT Time from randomization in years 23

24 F Overall Gender Males Females Age < 65 >65 ensder Previous Condition yes no Forest plot Ideal outcome Note: Many subgroups may not show statistical significance, but do show consistency Hazard Ratio better 2 better 24

25 Overall Gender Males G Age Females < 65 >65 Subgroup is highly significant Location of cancer No Yes Hazard Ratio better 2 better Special subgroup Responsible for significance? 25

26 Subgroup Analysis Can you make anything out of special group? If you did not anticipate this group effect you only have a consistency statement that you can make. Special effect is exploratory with confirmation needed Question: Are significance results in other subgroups due to this? 26

27 27 21

28 Relative Risk Reduction by Qualifying Condition IS n = 6431 MI n = 6302 PAD n = Total n =19185 Clopidogrel Better Aspirin Better 28

29 What do we believe about PAD? We can say we have consistency? In fact there was significant interactions between entry subgroups and outcome event FDA Cardio-Renal Advisory Committee and FDA gave approval for all entry groups 29

30 Non-small cell lung cancer and epidermal growth factor receptor (EGFR) Progression Free Survival (PFS) Paclitaxel Gefitinib 6 mths 30

31 Overall Gender M/F Age Years with Condition < 65 >65 Yes no Significances in subgroups are in opposite directions EGFR mutation Negative Positive Hazard Ratio Gefitinib better Paclitaxel better 31

32 EGFR and lung cancer treatment Confirmation is needed Consistency with genetic theory epidemiological data clinical trials What if mutation identified after start of trial (Prospective-Retrospective) What can publication say? 32

33 4.2 OVERALL TEST IS NOT SIGNIFICANT TECHNICALLY YOU CANNOT GO BEYOND THIS WITH ANY STATISTICAL STATEMENTS? USEFUL TO LOOK AT SUBSETS AS EXPLORATORY ANALYSIS (NOT EVEN APPROPRIATE TO CALL IT A SECONDARY ANALYSIS?) Actually, there exists methods to sort out useful subsets (SIM Tutorial: Lipkovich, Dmitrienko, D Agostino, 2016) in this situation 33

34 Overall Gender Age Years with Condition Males females < 65 >65 Yes no Location X NO YES Hazard Ratio better 2 better Location is significant. Was it prespecified as primary? No 35

35 4.3. Primary Hypothesis anticipates possible subgroup effect Subgroups identified by pre-randomization or post-randomization stratification Procedure 2.1 Subgroups tested for equal vs. unequal effects. This is done formally by INTERACTION TEST If significant interaction, do not pool data, rather test within groups If no significant interaction pool, test overall data May need to add variable in analysis for groups 36

36 4.2. Primary Hypothesis anticipates subgroup effect (continue) PROCEDURE Interest primarily in subgroups. INTERACTION TEST may be avoided and subgroups can be tests separately with control of error rates For example, if there are two groups then each can be tested at level of significance For example, groups can be tested sequentially with error rate control Also can test overall data and then groups with alpha control (Put explicitly in protocol) 37

37 RECENT EXAMPLE GIK for developing MI in EMS Want to reduce infarct size What people who have STEMI MIs Cannot identify these in ambulance ITT group all suspected mitt groups are those with STEMI Can we use STEMI as primary group? What if treatment groups are not balanced? 38

38 What of the 0.95 Confidence Intervals Often Presented? All the above methods can produce confidence intervals with the correct alpha control Compute [m root of 100(1-α)]% for Bonferroni

39 MULTIPLE TREATMENTS Above can be extended to multiple treatments, both within a clinical trial or an Observational study (propensity matching, covariate analysis, etc.

40 Closing Comments Multiplicity is an issue in Clinical Trials(and Observational Studies) Need to control FWER, or chance of a false positive study/result Whether to control FWER & how to control FWER depends on study objectives State clearly in protocol/stat plan how multiplicity will be addressed and controlled The p values remains a tool. Make clear what is being done Play by the rules and KEEP IT SIMPLE AND UNDERSTANDABLE 47

41 DECISION vs. CONCLUSION Statistical tests and procedures deal with decisions about variables To reach a conclusion we need Strength of association Specificity Coherence Biologic plausibility Consistency Boston University Statistics and Consulting Unit 48

42 Some References