Quantitative Outcomes

Transcription

1 Quantitative Outcomes Comparing Outcomes Analysis Tools in SCD Syntheses Kathleen N. Zimmerman, James E. Pustejovsky, Jennifer R. Ledford, Katherine E. Severini, Erin E. Barton, & Blair P. Lloyd The research reported in this article was supported in part by Grant R305D from the Institute of Educational Sciences, U.S. Department of Education. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.

2 Quantitative Outcomes Synthesis Considerations Procedural components Available tools Comparative meta-analyses Implications

3 Considerations 1. Visual analysis is critical 2. Conducted for designs with sufficient quality/rigor 3. Not all quantitative outcomes tools are created equal

4 Procedural Components Select tool to calculate outcomes for single designs Non-overlap measures Parametric effect sizes Regression-based approaches Select method for combining outcomes across designs Systematic review Meta-analysis

5 Procedural Components Select tool to calculate outcomes for single designs Non-overlap measures Parametric effect sizes Regression-based approaches Select method for combining outcomes across designs Systematic review Meta-analysis

6 Non-overlap measures Also called non-parametric effect sizes Do not measure magnitude of effect Procedural sensitivities Number of data points in a condition Design type Session length Measurement system Methods for meta-analysis not developed; difficult to interpret Should not be used as indices of magnitude in SCD meta-analysis Chen et al., 2016; Ledford et al., 2014; Moeyaert et al., 2018; Pustejovsky, 2015; 2016b; 2016c; Pustejovsky & Ferron, 2017; Wolery et al., 2010

7 Parametric Effect Sizes Between-Case Effect Sizes Mean difference in outcomes across cases Outcome similar to between groups ES Limited application Tiered designs across participants 3 withdrawal designs across 3 participants Within-Case Effect Sizes Magnitude of effects within a single case Not comparable to group design ES Applicable under certain conditions Between-case SMD Hedges, Pustejovksy, & Shadish, 2012, 2013 Pustejovsky, Hedges, & Shadish, 2014 Shadish, Hedges, & Pustejovsky, 2014 Log response ratio Within-case SMD Moeyaert et al., 2018; Pustejovsky & Ferron, 2017

8 Within-Case Parametric Effect Sizes Standardized Mean Difference No requirements for DV measurement (a) Average changes in measured behavior divided by estimated variance (b) Hedges g small sample correction Baseline data cannot be stable at 0 Log Response Ratio (decreasing) Log Response Ratio (increasing) DV measured using direct observation with a ratio scale Can be converted to a percentage change (a) Ratio of mean level of outcome in intervention/mean level of outcome in baseline (b) Use natural log because ratio scale is bounded (c) Add small sample bias correction (d) Estimate Mean either condition 0 Busk & Serlin, 1992; Pustejovsky, 2017; Pustejovsky & Ferron, 2017

9 Comparing Meta-Analytic Outcomes Across Tools Antecedent sensory-based interventions for young children 69 designs included in review; peer-reviewed and gray literature included Random effect meta-analysis Tau, LRRi, LRRd, SMD Simple un-weighted averages PND, IRD Non-overlap Measures Percentage non-overlapping data (PND) Improvement rate difference (IRD) Tau Within-case Parametric Effect Sizes Log response ratio (increasing) Log response ratio (decreasing) Standardized mean difference Zimmerman et al., 2018a; 2018b

10 Comparing Meta-Analytic Outcomes Across Tools Do overall conclusions about the effectiveness of an intervention vary contingent on tool selection for a common body of literature? Ineffective Percentage non-overlapping data (PND)** Tau Log response ratio (increasing) Log response ratio (decreasing) Standardized mean difference Effective Improvement rate difference (IRD)** **simple unweighted averages were calculated rather than meta-analytic statistics Zimmerman et al., 2018a; 2018b

11 Comparing Meta-Analytic Outcomes Across Tools Do conclusions about the effectiveness of subgroups of an intervention vary contingent on tool selection for a common body of literature? Ineffective LRRd, LRRi, PND Effective Tau, SMD, IRD Tau, SMD, LRRd, LRRi, PND, IRD Zimmerman et al., 2018a; 2018b

12 Comparing Meta-Analytic Outcomes Across Tools Do conclusions about the effectiveness of subgroups of an intervention vary contingent on tool selection for a common body of literature? Ineffective LRRd, LRRi, PND Tau, SMD, LRRd, LRRi, PND, IRD Effective Only PND, LRRd/LRRi detected non-significant results for both the omnibus and subgroup analyses Tau, SMD, IRD LRRd/LRRi only weighted analysis Zimmerman et al., 2018a; 2018b

13 Comparative Conclusions Using non-overlap measures fails to adhere to logic of an overall measure of the magnitude of effect Non-overlap measures should not be used in SCD meta-analysis Sensitive to procedural variations May over-estimate effects compared to parametric effect sizes Random effects meta-analysis only possible with Tau Zimmerman et al., 2018a; 2018b

14 Comparative Conclusions Log response ratio increasing/decreasing (LRRd/LRRi) may be more conservative than within-case SMD Likely to be applicable because of popularity of interval recording (Lane & Ledford, 2014) Not limited by design type or number of cases Current bounds to interpret the size of LRR outcomes have not been developed Zimmerman et al., 2018a; 2018b

15 SCD Meta-Analysis Implications When conducting single SCDs, researchers should make raw data available in open source formats

16 SCD Meta-Analysis Implications Applicability of technology to conduct SCD meta-analyses is limited Few bodies of literature use same DV measurement or design type Expertise and technology required to conduct rigorous meta-analyses may not be widespread for use without a statistician

17 SCD Meta-Analysis Implications More comparative evaluations need to be conducted to guide tool selection Mostly positive outcomes Mixed outcomes Heterogeneous designs

18 SCD Meta-Analysis Implications When conducting and reviewing SCD meta-analyses, evaluate the following: Applicability of tool as a measure of magnitude Availability of technologies required Considerations of interpretation of quantitative values

19 Discussion Questions 1. The study presented in presentation #1 did not include a gray literature search. Does this omission represent a major flaw given the purpose was related to participant characteristics and not outcomes? 2. Across-groups research syntheses have well-established guidelines. Is SCD research too heterogeneous for well-defined rules? 3. What additional resources are needed for researchers to conduct high-quality SCD syntheses? 4. Given that lower-quality syntheses are often published, how do we increase the likelihood that higher-quality reviews will be used?