Supplementary Online Content Piaggio G, Elbourne DR, Pocock SJ, Evans SJW, Altman DG; for the CONSORT Group. Reporting of noninferiority and equivanlence randomized trials: extension of the CONSORT 2010 statement. JAMA. doi: 10.1001/jama.2012.87802 eappendix. This supplementary material has been provided by the authors to give readers additional information about their work.
eappendix Methodological considerations in noninferiority trials Hypotheses. In a noninferiority clinical trial the aim is to establish that the effect of the new treatment, when compared to the reference treatment, is not below a pre-stated noninferiority margin (alternative hypothesis). The null hypothesis is that its effect is above this margin. A trial may be planned to assess noninferiority for a primary endpoint and superiority for other endpoints typically related to safety. 1 Design. Sample size depends on the level of confidence chosen, the desired power, the assumed percentages of success (binary outcomes) or the variability (continuous outcomes), and. 2 3 Δ can be specified as a difference in means or the logarithm of an odds ratio, risk ratio, or hazard ratio. It is often chosen as the smallest value that would be a clinically important effect. 4 Δ should be particularly small to guard against the acceptance of inferior treatments. The required size of noninferiority trials is therefore usually larger than that for superiority trials because of the small size of the required margin. 5 Sample size calculations are often based on the assumption that the difference in proportions or means between treatments is 0. [Ref 62] If instead we assume that the new treatment is less effective than the reference treatment, then the sample size has to be larger for a given power. Conversely, if the new treatment is more effective, the needed sample size for the same power is smaller. Conduct. Trial conduct should closely match that of any trial that demonstrated efficacy of the reference treatment, provided they were of high quality. 6 One should avoid features that might dilute true differences between treatments, thereby enhancing the risk of erroneously concluding noninferiority, 7 8 such as poor adherence, dropouts, recruitment of patients unlikely to respond, and treatment crossovers. If there is an interest in showing superiority on another endpoint (or on the one for which demonstrating noninferiority is the main purpose of the study), then there is a clear incentive to conduct a trial that strives to separate the two treatments. Analysis. Although a modified hypothesis testing framework exists, 9 10 a more informative CI approach is preferred in the design, analysis, and reporting of noninferiority and equivalence trials. 11 For superiority trials, intention-to-treat (ITT) analysis (analyzing all patients within their randomized groups, regardless of whether they completed allocated treatment) is recommended as it is least biased analysis for assessing the effect of two policies. 12 Full ITT analyses may not be possible when primary outcome data are missing, and when there are differences between policy and actual practice. ITT analysis often leads to smaller observed treatment effects than if analysis is limited to patients adhering to treatment (often called perprotocol (PP) analysis). PP analysis may be helpful as a sensitivity analysis. In noninferiority trials, however, ITT analysis may increase the risk of falsely claiming noninferiority, 5 and the addition of the non-itt analyses might be desirable as a protection from this risk. 13 ITT analyses are not always anti-conservative in noninferiority trials, 14 and alternative analyses that exclude patients not taking allocated treatment or otherwise not protocol-adherent could bias the trial in either direction. 15 Whichever is chosen as primary, the other should be presented as a secondary sensitivity analysis. Given that both ITT and PP analyses may be problematic for type 1 error rates and bias in noninferiority trials, minimizing protocol deviations may be particularly important. In the presence of non-adherence, obtaining reliable data on its extent is important, 16 and it has been suggested that a hybrid ITT/PP analysis might be preferable 17. Interim analyses in noninferiority trials have some differences in rationale from superiority trials. If noninferiority is established before the trial is completed, there may be no ethical requirement to stop early because of lack of efficacy. 18 However, other advantages (adverse effects, cost) could justify stopping the trial, to expedite availability of the new treatment. If a treatment is clearly inferior, then stopping the trial (or a particular trial arm) is ethically justified. 19-21 Stopping rules might be asymmetric, a trial being allowed to continue longer if the new treatment appears superior, 22 although this result is unlikely. 18 Confidence intervals for treatment effects should be adjusted for interim analyses, much as the criteria for statistical significance should be adjusted to control type I error rates. Interpretation. Interpreting a non- inferiority trial s results depends on where the CI for the treatment effect lies relative to both the margin of noninferiority and a null effect. The upper bound of the 2-sided (1 2α) 100% CI or that of the 1-sided (1 α) 100% CI for the treatment effect has to be below the margin Δ to declare that noninferiority has been shown, with a significance level α. Both Δ and α should be prespecified in the noninferiority hypothesis.
Many noninferiority trials based their interpretation on the upper limit of a 1-sided 97.5% CI, which is the same as the upper limit of a 2-sided 95% CI. Although both 1-sided and 2-sided CIs allow for inferences about noninferiority, we suggest that 2-sided CIs are appropriate in most noninferiority trials. 23 If a 1-sided 5% significance level is deemed acceptable for the noninferiority hypothesis test, a 90% 2-sided CI could then be used. Figure 1 interprets several possible scenarios with 2-sided CIs for a noninferiority trial. Once noninferiority is evident, it is acceptable to then assess whether the new treatment appears superior to the reference treatment, using an appropriate test or CI, with a significance level defined a priori and with an ITT 2 24 25 analysis. It is inappropriate to claim noninferiority post hoc from a superiority trial unless the noninferiority hypothesis was stated in the protocol and clearly related to a predefined margin of equivalence. That is, both superiority and noninferiority hypotheses need explicit specification in the trial protocol and appropriate methods applied for sequential testing. 26,27 It is, however, always reasonable to interpret a CI as excluding an effect of a particular prestated size. 28 Choice of the noninferiority margin The choice of should be justified on statistical and clinical grounds. The noninferiority margin has been given different names: irrelevant difference 29, clinical acceptable amount 30. It is often determined to preserve a pre-specified fraction of the reference treatment effect relative to placebo. However, if the reference treatment has a large effect over placebo it does not mean that large differences are unimportant, therefore it might be unwise to decide to preserve a fixed fraction without considering the resulting margin with a clinical perspective 1 31. The choice of needs to take into account the uncertainty of the difference between the reference treatment and placebo. However, there are situations where no such past placebo-controlled evidence exists, e.g. in device or surgical trials or where the reference treatment is so long established that no relevant data exist. In that case, other judgements come into play as to what is a sensible choice of Δ. The FDA guidance 32 about the design of noninferiority trials are generally conservative in their use of these trials to provide sufficient evidence to grant a license. They therefore look to narrow margins for the noninferiority and good evidence of assay validity. The EMA might be seen as slightly more liberal, and their Points to Consider on switching from Superiority to Noninferiority (2000) and their Guidance on choice of NI margin (2005) 1 2 concentrate on these issues. The EMA emphasizes confidence intervals more than hypothesis testing in this context, but also give importance to assay validity. If past placebo-controlled trials exist establishing the efficacy of the reference treatment, then there are two steps in the determination of : 1. Estimate the difference between the reference treatment and placebo. Given several previous trials, the effect of the reference treatment can be estimated from a meta-analysis. For the fixed margin approach the Federal Drug Administration of the United States (FDA) suggests taking the lower bound of the 95% confidence interval of the effect size as the estimate 32. In doing this, it is assumed that there is assay sensitivity, e.g. assurance that the reference treatment would be superior to a placebo if a placebo is employed in the current trial; and that the constancy assumption holds (eg that the difference between the reference treatment and placebo in those trials will hold in the setting of the new trial if a placebo control is used). In many circumstances this is too stringent a criterion, leading to an unrealistically small Δ, e.g. in the PROFESS trial. 33 2. Once the effect of the reference treatment is estimated, should be determined using clinical judgment so that at least a certain amount of the superiority of the reference treatment over placebo is retained 32 (e.g. 50% to 80% depending on the seriousness of the outcome). It needs to be recognized that such statistical logic can lead to an unachievable sample size. Hence choice of Δ needs to take account of such practicality. Approaches to analysis of a noninferiority trial if past placebo-controlled trials exist 1) Fixed margin or two confidence interval method: Conclude noninferiority if the upper bound of the 95% confidence interval for the difference between the reference treatment and the new treatment (for a beneficial outcome) is below Δ in the NI trial. This method is called two confidence interval because it assumes that Δ has been determined using the lower bound of the 95% confidence interval to estimate the effect of the reference treatment as described above. 32 2) Synthesis method: combines the estimate of the effect in the NI trial with the estimate of the reference treatment effect from a meta-analysis of historical trials, making use of the variability from both. With this method, Δ is not predetermined. It is controversial, since it gives an estimate of the difference between the new treatment and placebo using indirect evidence. It is not usually recommended as a primary analysis. 32
Noninferiority and effectiveness of the new treatment To demonstrate noninferiority and effectiveness of the new treatment compared to the reference treatment, the following are needed 18 30 : 1. Noninferiority margin was defined a priori on the basis of statistical reasoning and clinical judgment; the trial was planned with adequate power and appropriate significance level. 2. The effectiveness of the reference treatment has been established, assay sensitivity and constancy assumptions are valid. 3. The new treatment should be shown to be beneficial over the reference treatment regarding at least one nonefficacy outcomes, such as safety, tolerability, cost, or convenience. This is the basis for the rationale of conducting a noninferiority trial. If the other benefits of the new treatment have not been shown in the past, there should be power to show them in the current noninferiority trial, using a superiority hypothesis. 4. An attempt to demonstrate superiority of the new treatment to a putative placebo might be required by some regulatory authorities if at least one placebo-reference treatment trial has been conducted in the past. It is calculated by combining evidence from a) the difference between the new and the reference treatment in the present trial and, b) the difference between the reference treatment and placebo in historical trials. This method is subjected to all the caveats of indirect comparisons and the potential bias induced by any lack of assay constancy. 34 More detailed lists of essential elements to assess noninferiority have been provided. 35 Recent research in methodology Determination of the margin of noninferiority Methods to determine the margin of noninferiority for a mean difference of a continuous measurement have been discussed and extended to the ratio of two means or the hazard ratio. 36 To address the caveats involved in the determination of a fixed Δ, methods have been developed for the determination of the margin of noninferiority based on adaptive testing, for both continuous 37 and binary outcomes, 38 that allow it to vary as a function of the reference treatment response. This strategy is useful to assure that the new treatment has no adverse impact on a main outcome (e.g. in oncology trials, assuring that survival of patients administered the new therapy (is) at least 95% of survival with standard care ), while offering advantages in safety or other clinical outcomes. 38 It also has the advantage of being a more efficient testing procedure, thus saving on sample size. The situation in which the reference treatment response cannot be predicted accurately presents difficulties. Tests of noninferiority have been developed for the situation in which the noninferiority margin is a function of unknown parameters. 39 Estimation of the effectiveness of the new treatment in relation to placebo In recent years it has become clear that it is sometimes not sufficient to demonstrate noninferiority of the new treatment compared to the reference treatment, but it is also required to demonstrate the effectiveness of the new treatment (see section above about Demonstrating noninferiority and effectiveness of the new treatment ). Unlike superiority trials, interpretation of noninferiority trials can depend on information that is not measured in the trial, namely that the reference treatment is effective compared to placebo. 32 If this evidence is not available, a noninferiority trial might be showing noninferiority of a new treatment with respect to an ineffective reference treatment. This point has been raised mainly in the context of drug development, but it is relevant also for other treatments or interventions. A unified approach for establishing the efficacy of a new treatment, with the fixed margin approach and the synthesis approach being particular cases, has been proposed. This approach allows to control the impact of departures from the assumptions of assay sensitivity and constancy. 40 Analysis A method of analysis that preserves the integrity of randomization more effectively than a per-protocol analysis has been developed for time-to-event outcomes. 41 Different methods to calculate confidence intervals exist and they could result in different conclusions regarding the noninferiority hypothesis. 42
ereferences 1. EMEA. Committee for medicinal products for human use (CHMP) guideline on the choice of the non-inferiority margin. Stat Med 2006;25:1628-1638. 2. European Medicines Agency (EMEA). Points to Consider on Switching Between Superiority and Noninferiority. London, England, 2000. 3. Tamayo-Server JH, Albert JM, Tamayo-Server M. Advanced statistics: how to determine whether your intervention is different, at least as effective as, or equivalent: a basic introduction. Acad Emerg Med 2005;12:536-542. 4. Wiens BL. Choosing an equivalence limit for noninferiority or equivalence studies. Control Clin Trials 2002;23(1):2-14. 5. Jones B, Jarvis P, Lewis JA, Ebbutt AF. Trials to assess equivalence: the importance of rigorous methods. BMJ 1996;313(7048):36-39. 6. ICH Steering Committee. Harmonised Tripartite Guideline: Choice of Control Group and Related Issues in Clinical Trials (E10): Geneva, Switzerland: International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use Avalable at: http://www.ich.org/fileadmin/public_web_site/ich_products/guidelines/efficacy/e10/step4/e10_guidel ine.pdf. Accessed November 17, 2011, 2000. 7. Senn S. Statistical Issues in Drug Development: John Wiley & Sons, 2002. 8. Kim MY, Goldberg JD. The effects of outcome misclassification and measurement error on the design and analysis of therapeutic equivalence trials. Stat Med 2001;20(14):2065-2078. 9. Dunnett CW, Gent M. Significance testing to establish equivalence between treatments, with special reference to data in the form of 2X2 tables. Biometrics 1977;33(4):593-602. 10. Dunnett CW, Gent M. An alternative to the use of two-sided tests in clinical trials. Stat Med 1996;15(16):1729-1738. 11. Rothman KJ. Significance questing. Ann Intern Med 1986;105(3):445-447. 12. Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134(8):663-694. 13. ICH Steering Committee. Statistical Principles for Clinical Trials (E9)Statistical Principles for Clinical Trials (E9): Geneva, Switzerland: International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use. Available at: http://www.ich.org/fileadmin/public_web_site/ich_products/guidelines/efficacy/e9/step4/e9_guidelin e.pdf. Accessed November 17, 2011., 1998. 14. Brittain E, Lin D. A comparison of intent-to-treat and per-protocol results in antibiotic non-inferiority trials. Stat Med 2005;24(1):1-10. 15. McAlister FA, Sackett DL. Active-control equivalence trials and antihypertensive agents. Am J Med 2001;111(7):553-558. 16. Sheng D, Kim MY. The effects of non-compliance on intent-to-treat analysis of equivalence trials. Stat Med 2006;25(7):1183-1199. 17. Sanchez MM, Chen, X. Choosing the Analysis Population in Non-Inferiority Studies: Per Protocol or Intent-to- Treat. Stat Med 2006;25:1169-1181. 18. D'Agostino RB, Sr., Massaro JM, Sullivan LM. Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics. Stat Med 2003;22(2):169-186. 19. Lallemant M, Jourdain G, Le Coeur S, et al. A trial of shortened zidovudine regimens to prevent mother-to-child transmission of human immunodeficiency virus type 1. Perinatal HIV Prevention Trial (Thailand) Investigators. N Engl J Med 2000;343(14):982-991. 20. Prandoni P, Bruchi O, Sabbion P, et al. Prolonged thromboprophylaxis with oral anticoagulants after total hip arthroplasty: a prospective controlled randomized study. Arch Intern Med 2002;162(17):1966-1971. 21. Chan FK, Chung SC, Suen BY, et al. Preventing recurrent upper gastrointestinal bleeding in patients with Helicobacter pylori infection who are taking low-dose aspirin or naproxen. N Engl J Med 2001;344(13):967-973. 22. Durrleman S, Simon R. Planning and monitoring of equivalence studies. Biometrics 1990;46(2):329-336. 23. Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 2006;295(10):1152-1160. 24. Lewis JA. The European regulatory experience. Stat Med 2002;21(19):2931-2938. 25. Fleiss JL. General design issues in efficacy, equivalency and superiority trials. J Periodontal Res 1992;27(4 Pt 2):306-313; discussion 23-27. 26. Stone GW, McLaurin BT, Cox DA, et al. Bivalirudin for patients with acute coronary syndromes. N Engl J Med 2006;355(21):2203-2216.
27. SYNERGY Trial Investigators. Enoxaparin vs unfractionated heparin in high-risk patients with non st-segment elevation acute coronary syndromes managed with an intended early invasive strategy: primary results of the SYNERGY randomized trial. JAMA 2004;292:45-54. 28. Elbourne D, Dezateux C, Arthur R, et al. Ultrasonography in the diagnosis and management of developmental hip dysplasia (UK Hip Trial): clinical and economic results of a multicentre randomised controlled trial. Lancet 2002;360(9350):2009-2017. 29. Lange S, Freitag G. Choice of delta: requirements and reality--results of a systematic review. Biom J 2005;47(1):12-27; discussion 99-107. 30. Kaul S, Diamond GA. Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 2006;145(1):62-69. 31. Brown D, Volkers P, Day S. An introductory note to the CHMP guidelines: choice of the non-inferiority margin and data monitoring committees. Stat Med 2007;26:230-236. 32. Food and Drug Administration of the United States (FDA). Guidance for Industry, Non-Inferiority Clinical Trials, March 2010. 33. Sacco RL, Diener HC, Yusuf S, et al. Aspirin and extended-release dipyridamole versus clopidogrel for recurrent stroke. N Engl J Med 2008;359(12):1238-1251. 34. Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: a survey of published systematic reviews. BMJ 2009;338:b1147. 35. Kaul S, Diamond GA. Making sense of noninferiority: A clinical and statistical perspective on its application to cardiovascular clinical trials. Progress in Cardiovascular Diseases 2007;49(4):284-299. 36. Ng TH. Noninferiority hypotheses and choice of noninferiority margin. Stat Med 2008;27(26):5392-5406. 37. Laster LL, Johnson MF. Non-inferiority trials: the 'at least as good as' criterion. Stat Med 2003;22(2):187-200. 38. Laster LL, Johnson MF, Kotler ML. Non-inferiority trials: the least as good as criterion with dichotomous data. Stat Med 2006;25:1115-1130. 39. Chow SC, Shao J. On non-inferiority margin and statistical tests in active control trials. Stat Med 2006;25:1101-1113. 40. Snapinn S, Jiang Q. Controlling the type 1 error rate in non-inferiority trials. Stat Med 2008;27(3):371-381. 41. Matsuyama Y. A comparison of the results of intent-to-treat, per-protocol, and g-estimation in the presence of non-random treatment changes in a time-to-event non-inferiority trial. Stat Med 2009;29:2107-2116. 42. Flandre P. Statistical methods in recent HIV noninferiority trials: reanalysis of 11 trials. PloS one 2011;6(9):e22871.