The Nonuse, Misuse, and Proper Use of Pilot Studies in Education Research

Transcription

1 The Nonuse, Misuse, and Proper Use of Pilot Studies in Education Research Elizabeth Stuart Associate Dean for Education, Professor estuart February 28, 2018

2 Outline 1 The set-up 2 Problem 1: Danger in determining whether to proceed based on effect size from pilot study 3 Problem 2: Danger in using effect size estimate to power next trial 4 So what should researchers do?

3 The big picture We want to learn what works" in education research To do that we need rigorous studies that are large enough to detect effects when they exist But large-scale rigorous studies are expensive and resource-intensive So we want to do them only when an intervention is particularly promising But how do we know that??

4 Pilot studies Smaller-scale (underpowered) pilot" studies sometimes used to demonstrate the promise" of an intervention And sometimes the direct effect size estimates are used to motivate the size of the next study (e.g., using effect size estimates). Either from development to efficacy or efficacy to effectiveness. This talk will discuss why that s a bad idea, and how pilot studies should be used. Nothing new" here but gets things out in the open and demonstrates some of the trouble we can get into if we aren t careful

5 IES context: Definition of pilot study from 2017 RFA Pilot study: A study designed to provide evidence of the promise of the fully developed intervention for achieving its intended outcomes when it is implemented in an authentic education setting. A pilot study differs from studies conducted during the development process. The latter are designed to inform the iterative development process (e.g., by identifying areas of further development, testing individual components of the intervention); therefore, they are expected to lead to further development and revision of the intervention. The pilot study is designed to help determine whether a finalized version of the intervention performs as expected. Depending on the results, pilot studies may lead to further development of the intervention, or they may lead to a rigorous evaluation of the intervention."

6 IES Goals Projects under the Development/Innovation goal will result in... Pilot data regarding the intervention s promise for generating the intended beneficial student education outcomes and reaching the level of fidelity of implementation considered necessary to generate the intended beneficial student education outcomes." (RFA p. 50) The Efficacy and Replication goal (Efficacy/Replication) supports the evaluation of fully developed education interventions with evidence of promise for improving student education outcomes... " (RFA p. 5) What is meant by promise"? Sadly sometimes people interpret as effect size estimates from small under-powered studies...

7 IES examples of pilot studies (RFA p. 54) Full scale, fully powered efficacy studies Underpowered efficacy studies (e.g., randomized controlled studies with a small number of classrooms or schools that provide unbiased effect size estimates of practical consequence which can stand as evidence of promise while not statistically significant)." Single-case studies that meet WWC standards Quasi-experimental studies

8 The problem Sometimes people use point estimates from the underpowered pilot studies to directly inform sample size calculations in a subsequent study

9 But who can blame them? We as methodologists haven t given much guidance on how to use pilot studies! Many major textbooks and other resources don t explicitly talk about pilot studies (e.g., not in index) Couldn t find any papers in education research journals that talked explicitly about how to (or not to) design and use pilot studies Empirical studies rarely explicitly talk about use of pilot studies in the design process (exception was 3 articles in JREE!) IES RFA useful but still not very specific (perhaps purposefully)

10 So what do we do in this paper/talk? Show danger in using direct results from pilot studies to determine whether to proceed with another study Using effect size estimates from pilot study to guide size of that subsequent study Clarify how pilot studies should be used, and how to think about power analyses Done using easy to use shiny web app to step through and directly illustrate the dangers

11 Outline 1 The set-up 2 Problem 1: Danger in determining whether to proceed based on effect size from pilot study 3 Problem 2: Danger in using effect size estimate to power next trial 4 So what should researchers do?

12 Simulation settings 4 settings: 1 No effect (0) 2 Small effect without practical significance (0.2) 3 Medium effect with practical significance (0.5) 4 Very effective intervention with practical significance (0.8) Assumed a threshold for practical significance" of 0.25 Here we assume pilot study of size 50 Run through various scenarios about decisions to proceed with full trial, and how to power that full trial

13 Shiny app: pilotpower.table1.org

14 Outline 1 The set-up 2 Problem 1: Danger in determining whether to proceed based on effect size from pilot study 3 Problem 2: Danger in using effect size estimate to power next trial 4 So what should researchers do?

15 Two dangers Overestimating true effect and proceeding when shouldn t (Scenarios 1 and 2) Underestimating true effect and failing to proceed when should (Scenarios 3 and 4) Consider 3 decision rules

16 Percentage of simulations where full trial conducted given pilot results True effect size 1: 0.0 2: 0.2 3: 0.5 4: 0.8 Proceed rule (Mistake) (Should proceed) If effect size est. > If effect size est. > If effect significant

17 Outline 1 The set-up 2 Problem 1: Danger in determining whether to proceed based on effect size from pilot study 3 Problem 2: Danger in using effect size estimate to power next trial 4 So what should researchers do?

18 The challenge Tempting to use effect size from trial to power next study (i.e., to power next study to be able to detect it) BUT... if effect size estimate from pilot study larger than it really is we will underpower our next study OR... if effect size smaller than true effect we waste resources

19 The set-up According to decision rules from above, power full study using effect size estimate from pilot studies Then simulate full trial of that size, see how often that full trial adequately powered to detect the true effect

20 Power results if power full trial based on pilot effect size True effect size 1: 0.0 2: 0.2 3: 0.5 4: 0.8 If full trial run, % where null rejected in full trial Among ALL simulations, % where null rejected If effect significant, % where effect overestimated Power works out as expected if trial powered to detect practically significant effect Scenarios 3 and 4: full trials end up being underpowered to detect true effects (Row 1) But this also doesn t include the scenarios where we didn t even proceed! Achieved power": how often we don t detect a true effect when there is one (Row 2) But when proceed and find significance the effects themselves are overestimated; trials are powered to detect too-large" effects and so when effects are detected they are larger than the true effect (Row 3)

21 Outline 1 The set-up 2 Problem 1: Danger in determining whether to proceed based on effect size from pilot study 3 Problem 2: Danger in using effect size estimate to power next trial 4 So what should researchers do?

22 Best practices Don t use pilot study effect sizes alone to decide about proceeding to full trial Also use qualitative information, other evidence (inc. non-experimental), theory of change, other interventions, etc. Large rigorous non-experimental studies may be more useful! Power studies based on practically significant effect sizes DO use pilot studies to examine procedures around recruitment, retention, measurement, randomization, implementation, etc.

23 IES discussion of power analyses for efficacy/replication studies Discuss the statistical power of the research design to detect a reasonably expected and minimally important effect of the intervention on the focal student education outcomes... Identify the minimum effect of the program or policy that you will be able to detect, justify why this level of effect would be expected, and explain why this would be a practically important effect. " (p. 66)

24 Conclusions Put succinctly, using pilot study effect sizes to power full trials leads to incorrectly judging effective interventions to be ineffectual, and judging effective interventions to be more effective than they are." (Westlund & Stuart, p. 13) We need to develop more guidelines and tools to help researchers plan studies!

25 Resources pilotpower.table1.org Westlund, E., and Stuart, E.A. (2017). The Nonuse, Misuse, and Proper Use of Pilot Studies in Experimental Evaluation Research. American Journal of Evaluation 38(2): Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63, 484. doi: Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004). Design and analysis of pilot studies: Recommendations for good practice. Journal of Evaluation in Clinical Practice, 10, 307?312. doi: 384.doc.x Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., & Rios, L.,... Goldsmith, C. (2010). A tutorial on pilot studies: The what, why and how. BMC Medical Research Methodology, 10, 1. doi: