Predicting Group-Level Outcome Variables: An Empirical Comparison of Analysis Strategies

Transcription

1 Predicting Group-Level Outcome Variables: An Empirical Comparison of Analysis Strategies Jeffrey D. Kromrey & Lynn Foster-Johnson University of South Florida & Geisel School of Medicine Dartmouth College 2015 Modern Modeling Methods Conference May 19-20

2 Outline Introduction & Research Problem Purpose of the Study Simulation Design Results Type I Error Control Statistical Power Bias RMSE 2

3 Introduction Familiar multi-level models are usually adequate for modeling outcome variables at the individual level (macro-micro) Not appropriate when the outcome is measured at the group level (micro-macro) Analysis strategies for these types of data have not been adequately investigated 3

4 Analysis Options Replicate the outcome value for each observation and conduct a single level analysis of participants (a REALLY bad idea) Aggregate values of individual level variables to the cluster level, then conduct a single level analysis of group means. Croon and van Veldhoven (2007) suggested a latent variable two-step approach Lüdtke et al. (2008) described a FIML method for contextual effects in multilevel models. 4

5 Purpose Empirically compare the performance of three approaches to the analysis of group level outcomes Analysis of group means Croon and van Veldhoven s latent variable approach Lüdtke et al. s FIML approach Include White s correction for heteroscedasticity in conjunction with first two methods 5

6 Simulation Study #1 A completely crossed factorial design was conducted using simulation methods Number of regressors (3, 5, and 7 at the individual level; 2 and 4 at the group level) Correlation between regressors (.10,.30, and.50 at the individual level;.20,.40, and.60 at the group level) Cross-level correlations (0,.30,.50) Reliability of regressors (.70,.90, and 1.00) Effect size for regressors (0 and.15) ICC for regressors (.10 and.20) 6

7 Simulation Study #1 (Cont d) Number of groups (25, 50, 100) Group size Fixed at 10 and 40 Random size (5 15 and 20 60) Factors were completely crossed, providing 23,328 conditions 10,000 samples per condition SAS/IML for simulation 7

8 Type I Error Control 8

9 Type I Error Control 9

10 Statistical Power 10

11 Statistical Power 11

12 Statistical Bias Distributions of Bias Estimates for Individual Level (X) Predictors Estimated Bias CV CV-W GRP GRP-W Analysis Method 12

13 Statistical Bias Distributions of Bias Estimates for Group Level (Z) Predictors Estimated Bias CV CV-W GRP GRP-W Analysis Method 13

14 RMSE Distributions of RMSE Estimates for Individual Level (X) Predictors Estimated RMSE CV CV-W GRP GRP-W Analysis Method 14

15 RMSE Distributions of RMSE Estimates for Group Level (Z) Predictors Estimated RMSE CV CV-W GRP GRP-W Analysis Method 15

16 Simulation Study #2 A completely crossed factorial design was conducted using simulation methods Number of regressors (3 and 7 at the individual level; 2 at the group level) Correlation between regressors (.10 at the individual level;.20 and.40 at the group level) Cross-level correlations (.30 and.40) Reliability of regressors (.70 and 1.00) Effect size for regressors (0 and.15) ICC for regressors (.20) 16

17 Simulation Study #2 (Cont d) Number of groups (25, 100, and 500) Group size Random size (5 15 and 20 60) Factors were completely crossed, providing 192 conditions 1,000 samples per condition SAS/IML for simulation; Mplus for FIML 17

18 Type I Error Control 18

19 Type I Error Control 19

20 Statistical Bias 20

21 Statistical Bias 21

22 RMSE 22

23 RMSE 23

24 Convergence Problems Analysis of group means: no problem Croon 7% non-convergence FIML: bigger problem With three level-1 predictors Only 44% of conditions converged with all samples Non-convergence did not exceed 10% in any condition With seven level-1 predictors Only 22% of conditions converged with all samples Non-convergence reached as high as 45% 24

25 Conclusions Analysis of group means has advantages Easy Always converges Little bias Smaller RMSE Better power Analysis of group means has disadvantages Appears to be discarding information from the data Not as sexy as latent variable approaches 25

26 Conclusions (Cont d) White s correction yields a notable boost in power for both the group mean analysis and the Croon approach FIML performed poorly in both Type I error control and convergence 26

27 Conclusions (Cont d) Although complex designs and state-of-the-art methods are sometimes necessary to address research questions effectively, simpler classical approaches often can provide elegant and sufficient answers to important questions. Do not choose an analytic method to impress your readers or to deflect criticism...occam s razor applies to methods as well as to theories. (Wilkinson and the Task Force on Statistical Inference, 1999, p. 598) 27