Meta-Analysis: A Gentle Introduction to Research Synthesis

Transcription

1 Meta-Analysis: A Gentle Introduction to Research Synthesis Jeff Kromrey Lunch and Learn 27 October 2014

2 Discussion Outline Overview Types of research questions Literature search and retrieval Coding and dependability Effect sizes Describing results Testing hypotheses Threats to validity Reporting meta-analyses References worth pursuing

3 Overview Summarization of empirical studies using quantitative methods Results Estimated weighted mean effect size Confidence interval around mean effect size (or test null hypothesis about mean effect size) Homogeneity of effect sizes Tests of moderators

4 Overview: Why Meta-Analyze? Strength in numbers Several non-significant differences may be significant when combined Strength in diversity Generalizability across variety of participants, settings, instruments Identification of moderating variables Good way to look at the forest rather than the trees What do we think we know about a phenomenon? How well do we know it? What remains to be investigated? It s fun!

5 Overview: Stages of Meta-Analysis Formulate problem Draw sample / collect observations Measure observations Analyze data Interpret data Disseminate

6 Types of Research Questions: Treatments Does the treatment (in general) appear effective? How effective? Does treatment effectiveness vary by Participant characteristics? Treatment characteristics? Research method characteristics? Does the treatment appear ineffective in some conditions?

7 Types of Research Questions: Relationships What is the relationship (in general)? Direction? Strength? Does direction or strength of relationship vary by Participant characteristics? Treatment characteristics? Research method characteristics? Is the relationship not evident in some conditions?

8 Literature Search and Retrieval Decisions to make before searching the literature Inclusion/Exclusion criteria for sources Types of publication Language and country of publication Dissemination: Journal, presentation, unpublished Study characteristics Participant characteristics Information reported Timeframe Type of design Measures

9 Literature Search and Retrieval Decisions to make before searching the literature Search strategies Keywords Databases ERIC, PsychInfo, GoogleScholar, Web of Science Other Key researchers Listservs Websites Reference sections of articles

10 Coding of Studies Record Study inclusion/exclusion characteristics Effect size(s) Multiple measures? Subsamples? Different times? Other relevant variables Research design (sampling, controls, treatment, duration) Participant attributes (age, sex, race/ethnicity, inclusion/exclusion) Settings (geography, classrooms, laboratory) Dissemination characteristics (journal, conference, dissertation, year, Dr. B)

11 Coding of Studies (Cont d) Written codebook and coding forms The Goldilocks principle: not too coarse, not too fine. Training and calibration of coders Beware of drift Estimating reliability of coders

12 Study Coding Form Meta-Analysis Coding Part I: Increased levels of stress will reduce the likelihood of XXX treatment success. STUDY TITLE: I. Qualifying the study: Answer the following questions as either yes or no. Does the study involve women participating in an XXX treatment program? Does the study focus on the relationship between stress and XXX treatment outcomes? Was the study conducted between January 1995 and December 2013? Does the study employ a prospective design? Does the study report outcome measures of stress or anxiety as well as XXX treatment outcomes? If the answer to each of the above questions is yes, the study qualifies for inclusion in the meta-analysis.

13 II. Coding the study: A. Publication Characteristics 1. Title of the study: 2. Year of Publication: 3. Authors: B. Ecological Characteristics 1. Age of Female Participants: Mean: Range: 2. Country: 3. Race: White N: %: Black N: %: Hispanic N: %: Asian / Pacific Islander N: %: American Indian N: %: Other N: %:

14 Duration of Psychoeducational Intervention (Please choose) a. Daily for duration of XXX treatment b. 1 3 sessions during XXX treatment c. 6 weeks during XXX treatment d. 8 weeks during XXX treatment e. 10 weeks during XXX treatment f. Other: Length of Psychoeducational Intervention (Please choose) a. 1 hour b. 1.5 hours c. 2 hours d. Other: Frequency of Psychoeducational Intervention (Please choose) a. Daily b. Weekly c. Bi-Weekly d. Other:

15 Effect Size How false is the null hypothesis? How effective is the treatment? How strong is the relationship? Independent of sample size (more or less) Useful in primary studies and in metaanalysis Links to power Descriptive statistic (big enough to care?)

16 Effect Size (Cont d) Jacob Cohen Statistical Power Analysis for the Behavioral Sciences Anytime a statistical hypothesis is tested, an effect size is lurking in there somewhere Small, medium, large effects Medium effect size is big enough to be seen by the naked eye of the careful but naïve observer

17 Effect Size: Standardized Mean Difference Population effect size 1 2 Sample effect size d X X 1 2 ˆ pooled Small=.20, Medium=.50, Large=.80

18 Effect Size: Chi-square Tests Population effect size 2 aj oj j oj Sample effect size w N 2 Small=.10, Medium=.30, Large=.50

19 Effect Size: ANOVA and Regression ANOVA fˆ ( k1) F N Regression (test of R 2 ) fˆ 2 2 R 1 R 2 signal noise Regression (test of R 2 change) fˆ 2 R RL signal remaining noise

20 Effect Size: Correlation Pearson Product Moment Correlation is an effect size Commonly transformed to z for aggregation and analyses z r.5log Small=.10, Medium=.30, Large=.50 e 1 r 1 r xy xy

21 Effect Size: Computing from Reported Statistics Article Information: X ˆ , , n1 20 X ˆ , , n2 24 SS SS SS ˆ pooled 3.06 n1 n and d ˆ i 2 Knowing SS and SS n 1 n ˆ i i i i i 2 2 SS ,

22 Effect Size: Computing from Reported Statistics Article Information: t(54) = 4.52, p <.05 d 2t df d

23 Effect Size: Caveats Sensitivity to violations of assumptions VIN Robust effect sizes Improved estimation Difficulty/impossible to compute without original data Multiple effect sizes from same sample or correlated samples Special analyses

24 Describing Results: Graphical Displays Funnel Plot Display for Example Data (k=10) Effect Size Sample Size

25 Describing Results: Graphical Displays

26 Describing Results: Graphical Displays Stem and Leaf Plot

27 Describing Results: Graphical Displays

28 Describing Results: Graphical Displays

29 Testing Hypotheses d 1 All observed effect sizes from a single population d 2 d 3 d 4 d 5 d 6

30 Testing Hypotheses d 1 Females Observed effect sizes from two populations d 2 d 3 d 4 d 5 Males d 6

31 Testing Hypotheses: Fixed Effects vs. Random Effects Frequency Frequency Population Effect Size Population Effect Size

32 Testing Hypotheses: Fixed Effects vs. Random Effects Fixed Effects Assumes one population effect size Effect size variance = sampling error (subjects) Weights represent study variance due to sampling error associated with the subjects (sample size)

33 Testing Hypotheses: Fixed Effects vs. Random Effects Random Effects Assumes population effect size is a normal distribution of values (i.e. not one effect size) Effect size variance = sampling error (subjects) + random effects (study) Weights represent study variance due to sampling error associated with the subjects (sample size) and sampling of studies (random effects variance component)

34 Testing Hypotheses: Fixed Effects vs. Random Effects Fixed Effects vs. Random Effects: Which model to use? Aspects to consider: Statistics decision based on the outcome of the homogeneity of effect sizes statistic (conditionally random-effects) Desired Inferences decision based on the inferences that the researcher would like to make Conditional Inferences (fixed effect model): Researcher can only generalize to the studies included in the meta-analysis Unconditional Inferences (random effect model): Researcher can generalize beyond the studies included in the meta-analysis Number of studies when the number of studies is small fixed effects may be more appropriate

35 Testing Hypotheses: Estimation of Weights Fixed effects weight v i 1 2 i Random effects weight w 1 i 2 2 i For standardized mean difference: i 2 i n n d n n n n

36 Testing Hypotheses: Weighted Mean Effect Size Fixed effects vd i v Random effects wd i w i i i i 1 var(.. ) v i 1 var(.. ) wi

37 Testing Hypotheses: Estimates of Effect Size Variance Also called Random Effects Variance Component (REVC), symbolized with Used to calculate random effects weights Three methods to calculate Observed variance Q based Maximum likelihood 2

38 Testing Hypotheses: Estimates of Effect Size Variance Observed variance: i s k Q based: 2 Q k1 c where c v i v v 2 i i Maximum likelihood: 1 2 k k 2 1 k 1 yi ;, 2 2 exp i1 i 2 i1 i 2 i l y

39 Testing Hypotheses: Significance Testing and Confidence Intervals (CI) Significance testing: Z 0 var( ) Confidence interval (95% CI): 1.96 var( )

40 Testing Hypotheses: Mean and Individual Effect Size Differences Focused test of between group differences 1 2 Q BET ( ) j var( ) j General test of homogeneity of effect sizes Q 1 var( d ) i d 2 i

41 Testing Hypotheses: Meta-Analytic Regression Model d X X X e i p p Generalization of the Q test Continuous or categorical moderators X i are potential moderating variables j Test for moderating effect

42 Threats to Validity Sources Primary studies unreliability, restriction of range, violations of assumptions, missing effect sizes (publication bias), incompatible constructs, and poor quality Meta-analysis processes incomplete data collection (publication bias), inaccurate data collection, poor methodology, and inadequate power

43 Threats to Validity Apples and Oranges Dependent Effect Sizes File Drawer/Publication Bias Methodological Rigor Power

44 Threats to Validity Apples and Oranges Are the studies being analyzed similar regarding: Constructs examined Measures Participants (sampled from same population?) Analyses Dependent Effect Sizes Participants cannot contribute to the mean effect size more than once without special treatment (see Hedges et al., 2010; Owens et al., 2011)

45 Threats to Validity: Publication Bias Publication Bias = Studies unavailable to the meta-analyst due to lack of publication acceptance or submission (termed file drawer problem by Rosenthal, 1979) Pattern in the literature Variance Small (N=large) Large (N=small) Small Published (Stat Sig) Not Published (not Stat Sig) Effect Size Large Published (Stat Sig) Published (Stat Sig)

46 Threats to Validity: Publication Bias Publication Bias Detection Methods Visual interpretation Funnel plot display Statistical methods Begg Rank Correlation (variance or sample size) Egger Regression Funnel Plot Regression Trim and Fill

47 Threats to Validity Methodological Rigor of Primary Studies Set criteria for inclusion Include various levels of rigor; then code and use in meta-analytic analyses (moderators or quality weights) Power Enough studies collected to support the validity of hypothesis tests?

48 Reporting Meta-Analyses: Pertinent Information to Include Details regarding the search criteria and retrieval Coding process including rater reliability Describe effect sizes graphically Analyses Mean effect size (significance test and / CI) Fixed vs. Random Effects model Homogeneity of effect sizes Tests for moderators How threats to validity were addressed

49 For Further Reading & Thinking Bangert-Drowns, R.L. (1986). Review of developments in meta-analysis method. Psychological Bulletin, 99, Cafri, G., Kromrey, J. D., & Brannick, M. T. (2009). A SAS macro for statistical power calculations in meta-analysis. Behavior Research Methods, 41, Cafri, G., Kromrey, J. D., & Brannick, M. T. (2010). A meta-meta-analysis: Empirical review of statistical power, Type I error rates, effect sizes, and model selection of meta-analyses published in psychology. Multivariate Behavioral Research, 45, Cohen, J. (1992). A power primer. Psychological Bulletin, 112, Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press. Cooper H. & Hedges, L. (1994). The handbook of research synthesis. New York: Russell Sage Foundation. Cooper, H., Hedges, L., & J.C. Valentine, J. C. (2007). The handbook of research synthesis and metaanalysis (2 nd ed). New York: Russell Sage Foundation. Fern, E. F. & Monroe, K. B. (1996). Effect size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23, Gloudemans, J., Owens, C., & Kromrey, J. D. (2012, April). MV_Meta: A SAS macro for multivariate meta-analysis. Paper presented at the annual meeting of the SAS Global Forum, Orlando. Grissom R.J. & Kim J.J. (2001). Review of assumptions and problems in the appropriate conceptualization of effect size. Psychological Methods, 6(2), p

50 For Further Reading & Thinking Hedges, L.V. & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press. Hedges, L. V., Tipton, E., and Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, Hedges, L.V. & Vevea, J. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, Hedges, L. V., & T. D. Pigott The power of statistical tests in meta-analysis. Psychological Methods, 6, Hedges, L. V., & T. D. Pigott The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, Hogarty, K. Y. & Kromrey, J. D. (2000). Robust effect size estimates and meta-analytic tests of homogeneity. Proceedings of SAS Users Group International, Hogarty, K. Y. & Kromrey, J. D. (2001, April). We ve been reporting some effect sizes: Can you guess what they mean? Paper presented at the annual meeting of the American Educational Research Association, Seattle. Hogarty, K. Y. & Kromrey, J. D. (2003). Permutation tests for linear models in meta-analysis: Robustness and power under non-normality and variance heterogeneity. Proceedings of the American Statistical Association. Alexandria, VA: American Statistical Association.

51 For Further Reading & Thinking Hogarty, K. Y., Kromrey, J. D., Ferron, J. M., Hess, M. R., & Hines, C. V. (2005). Robustness in meta-analysis: A macro for computing point estimates and confidence intervals for standardized mean differences and Cliff s delta. Proceedings of the Southeast SAS Users Group. Huberty, C. J. & Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and Psychological Measurement, 60, Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd edition). Newbury Park, CA: Sage. Hillsdale, NJ. Kromrey, J. D., Ferron, J. D., Hess, M. R., Hogarty, K. Y. & Hines, C. V. (2005, April). Robust Inference in Meta-analysis: Comparing Point and Interval Estimates Using Standardized Mean Differences and Cliff s Delta. Annual meeting of the American Educational Research Association, Montreal. Kromrey, J. D. & Foster-Johnson, L. (1996). Determining the efficacy of intervention: The use of effect sizes for data analysis in single-subject research. Journal of Experimental Education, 65, Kromrey, J. D. & Hogarty, K. Y. (2002). Estimates of variance components in random effects metaanalysis: Sensitivity to violations of normality and variance homogeneity. Proceedings of the American Statistical Association. Alexandria, VA: American Statistical Association.

52 For Further Reading & Thinking Kromrey, J. D. & Hogarty, K. Y. (2006). METAPERM2: A SAS macro for permutation tests of linear models in fixed and random effects meta-analyses. Proceedings of the 31 st Annual SAS Users Group International Conference. Kromrey, J. D., Hogarty, K. Y., Ferron, J. M., Hines, C. V. & Hess, M. R. (2005, August). Robustness in Meta-Analysis: An Empirical Comparison of Point and Interval Estimates of Standardized Mean Differences and Cliff s Delta. Proceedings of the American Statistical Association Joint Statistical Meetings. Kromrey, J. D. & Foster-Johnson, L. (1999, February). Effect sizes, cause sizes and the interpretation of research results: Confounding effects of score variance on effect size estimates. Paper presented at the annual meeting of the Eastern Educational Research Association, Hilton Head, South Carolina. Kromrey, J. D. & Rendina-Gobioff, G. (2006). On knowing what we don't know: An empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66, Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-analysis. Thousand Oaks: Sage. National Research Council (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy of Science Press. Owens, C. M., Kromrey, J. D., and Gloudemans, J. (2011). Meta-Analysis of Multivariate Outcomes: A Monte Carlo Comparison of Alternative Strategies. JSM Proceedings. Miami: American Statistical Association.

53 For Further Reading & Thinking Rendina-Gobioff, G. (2006). Detecting Publication Bias in Random Effects Meta-Analysis: An Empirical Comparison of Statistical Methods. Unpublished doctoral dissertation, University of South Florida, Tampa. Rendina-Gobioff, G., Kromrey, J. D., Dedrick, R. F., & Ferron, J. M. (2006, November). Detecting Publication Bias in Random Effects Meta-Analysis: An Investigation of the Performance of Statistical Methods. Paper presented at the annual meeting of the Florida Educational Research Association, Jacksonville. Rendina-Gobioff, G. & Kromrey, J. D. (2006, October). PUB_BIAS: A SAS Macro for Detecting Publication Bias in Meta-Analysis. Paper presented at the annual meeting of the Southeast SAS Users Group, Atlanta Romano, J. L. & Kromrey, J. D. (2009). What are the consequences if the assumption of independent observations is violated in reliability generalization meta-analysis studies? Educational and Psychological Measurement, 69, Rosenthal, R. (1995). Writing meta-analytic reviews. Psychological Bulletin, 118, Sutton, A. J., Abrams, K. R., Jones, D. R., Sheldon, T. A., & Song, F. (2000). Methods of meta-analysis in medical research. New York: Wiley. Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63,

54 Now, let s just talk Thank You