Guidance on specifying the target difference ( effect size ) for a RCT

Guidance on specifying the target difference ( effect size ) for a RCT Jonathan Cook Centre for Statistics in Medicine & NDORMS, University of Oxford

Aim & feedback Building upon exiting DELTA guidance Limited in scope Presumed knowledge Initial feedback was (mostly) to Retain a wide scope Make it accessible to a range of individuals More practical guidance (e.g. recommendations) More examples desired Address unusual scenarios (e.g. alternative designs) Cook et al Trials, 2015

Guidance content General considerations for specification of the sample size calculation Brief summary of methods Recommendations specifying the target difference reporting items for key trial documents Case studies Appendices covering more technical/statistical topics Conventional sample size calculations Alternative approaches to sample size calculations Alternative trial designs

Available methods Diversity in conception and implementation Methods can be judgement based, data driven or a combination Identify a difference which is Important e.g. minimum clinically important difference (MCID) Realistic e.g. based upon prior evidence, or Both important and realistic

Use of methods Perspective adopted is influential (e.g. patient, clinician) Justification more difficult for some outcomes (e.g. QoL) More than one method might be appropriate

Methods for specifying the target difference 1. Anchor: The outcome is anchored by using a judgement (patient s or health professional s) to define an important difference. 2. Distribution: Methods based upon distributional variation/assumption e.g. a value that is larger than the inherent imprecision in the measurement. 3. Health economic: Assessment incorporating cost and benefit e.g. determine threshold recurrence rate based upon cost-effectiveness. 4. Opinion-seeking: Elicitation of expert opinion e.g. survey of clinicians.

Methods for specifying the target difference 5. Pilot study: A pilot study might be carried out to guide expectations. 6. Review of evidence base: Summarising current empirical evidence e.g. systematic review and meta-analysis. 7. Standardised effect size: The magnitude of the effect upon a standardised scale is used to define the value of the difference e.g. Cohen s (d) effect sizes.

Specification recommendations 1. Begin by searching for relevant literature to inform the specification of the target difference. Relevant literature can: a. relate to candidate primary outcomes and/or the comparison at hand; b. inform what is an important and/or realistic difference. 2. All candidate primary outcomes should be considered and the potential sample size calculated to help determine the appropriate final choice. Ideally, the final sample size chosen will be sufficient for all key outcomes though this is not always practical. 3. With the exception of a mortality outcome, any difference between interventions cannot be assumed to be clinically important. Therefore the specification of the target difference for all other outcomes requires additional justification to infer importance to stakeholders. 4. The target difference for a Phase III trial should be one considered to be important to at least one key stakeholder group (for example, patients and/or health professionals). It does not necessarily have to be the minimum value which would be considered important if a larger difference is considered a realistic possibility.

Specification recommendations (2) 5. Where additional research is needed to inform what would be an important difference, the opinion seeking and anchor methods are to be favoured. The distribution method for specifying a target difference should not be used. Specifying the target difference using a SES approach should be considered a last resort. 6. Where additional research is needed to inform what would be a realistic difference, the RoEB and opinion seeking methods are generally the most useful. Pilot studies typically are too small to inform on their own what would be realistic difference and primarily address other aspects of trial design and conduct (for example, recruitment). 7. Use existing literature to inform the values of key nuisance parameters which are part of sample size calculation. These include the SD for a continuous outcome, along with other relevant inputs such as the amount of loss-to-follow. 8. Specification of sample size (including the target difference) should be reported according to the guidance for reporting items in section 5.4 when preparing grant applications, and protocol and result manuscripts.

Recommended reporting items Core items for all key trial documents Primary outcome Target difference Statistical significance and power Associated parameters (e.g. SD) Adjustments (e.g. multiple testing) Final sample size Additional items by document type Basis and explanation of target difference [grant proposal/protocol] Reference protocol [results paper]

Example text continuous outcome FILMS trial: The primary outcome is ETDRS distance visual acuity. A target difference of a mean difference of 5 letters with a common standard deviation (SD) of 12 was assumed. Five letters is equivalent to one line on a visual acuity chart and is viewed as an important difference by patients and clinicians. The SD value was based upon two previous studies one RCT and one observational comparative study. This target difference is equivalent to a standardised effect size of 0.42. Setting the statistical significance to the 2 sided 5% level and seeking 90% power, 123 participants per group are required; 246 in total.

Example text continuous outcome FILMS trial: The primary outcome is ETDRS distance visual acuity. A target difference of a mean difference of 5 letters with a common standard deviation (SD) of 12 was assumed. Five letters is equivalent to one line on a visual acuity chart and is viewed as an important difference by patients and clinicians. The SD value was based upon two previous studies one RCT and one observational comparative study. This target difference is equivalent to a standardised effect size of 0.42. Setting the statistical significance to the 2 sided 5% level and seeking 90% power, 123 participants per group are required; 246 in total.

Where next? DELTA 2 Continuing process of engagement on current draft Including Delphi process (thank you to those involved!) Finalise guidance in autumn 2017 Working with funders to produce programme specific guidance Dissemination to follow

References DELTA 2 website https://www.csm.ox.ac.uk/research/methodologyresearch/delta2 Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005;365:1348 53 Cook J, et al. Specifying the target difference in the primary outcome for a randomised controlled trial - guidance for researchers. Trials 2015; 16:12. Cook J, et al. Assessing methods to specify the target difference for a randomised controlled trial DELTA (Difference ELicitation in TriAls) review. Health Technol Assess 18:28 2014. Hislop J, et al. Methods for Specifying the Target Difference in a Randomised Controlled Trial: The Difference ELicitation in TriAls (DELTA) Systematic Review. PLOS Med 11(5): e1001645. 2014. Blanton H, Jaccard J. Arbitrary metrics in psychology. Am Psychol 2006;61:27-41.