On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA

Transcription

1 STRUCTURAL EQUATION MODELING, 13(2), Copyright 2006, Lawrence Erlbaum Associates, Inc. On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA André Beauducel Mannheim University Philipp Yorck Herzberg Technical University Dresden The simulation study compared maximum likelihood (ML) estimation with weighted least squares means and variance adjusted (WLSMV) estimation. The simulation study was based on confirmatory factor analyses with 1, 2, 4, and 8 factors, based on 250, 500, 750, and 1,000 cases, and on 5, 10, 20, and 40 variables with 2, 3, 4, 5, and 6 categories. There was no model misspecification. The most important results were that with 2 and 3 categories the rejection rates of the WLSMV chi-square test corresponded much more to the expected rejection rates according to an alpha level of.05 than the rejection rates of the ML chi-square test. The magnitude of the loadings was more precisely estimated by means of WLSMV when the variables had only 2 or 3 categories. The sample size for WLSMV estimation needed not to be larger than the sample size for ML estimation. Maximum likelihood (ML) estimation is most commonly used in confirmatory factor analysis (CFA) and structural equation modeling (SEM). ML estimation assumes that the observed variables follow the multivariate normal distribution. However, many data in psychological research are ordinal data or do not follow the multivariate normal distribution. One approach for analyzing ordinal data is weighted least squares (WLS) estimation, which assumes that the observed ordinal Correspondence should be addressed to Dr. André Beauducel, Department of Psychology II, Mannheim University, Schloss, Ehrenhof Ost, Mannheim, Germany. beauducel@ tnt.psychologie.uni-mannheim.de

2 ML VERSUS WLSMV ESTIMATION 187 variables stem from a set of underlying latent continuous variables. Comparisons of ML estimation and WLS estimation for the analysis of ordinal data revealed high amounts of bias for WLS estimates, especially with small samples sizes and moderate loadings (Hoogland & Boomsma, 1998). However, with increased nonnormality, ML-based parameter estimates, standard errors, and factor intercorrelations also indicated high levels of bias (Dolan, 1994; Hoogland & Boomsma, 1998). Muthén (1993) and Muthén, du Toit, and Spisic (in press) introduced the weighted least squares means and variance adjusted (WLSMV) estimation as a refinement of the WLS estimator (see also Muthén & Muthén, 2004). The properties of WLSMV estimation for analysis of ordinal data have not yet been investigated extensively. However, Muthén et al. (in press) evaluated WLSMV estimation by means of a simulation study. They found that it was acceptable even for a sample size of 200 cases. Muthén et al. (in press) used a longitudinal population model as a basis for their simulations. Such a model is of course a meaningful basis for a simulation study; however, the model does not correspond to the typical simple structure CFA, as it is often used in social sciences. A typical simple structure model has only a few salient loadings for each factor (Thurstone, 1947) and comprises several factors. The nonsalient loadings, that is, the small or nonsignificant loadings, are typically assumed to be zero in CFA. Such a model is often used as a basis for test construction (e.g., Ferrando & Chico, 2001; Haynes, Miles, & Clements, 2000; Tomas & Oliver, 1999). Therefore, simple structure CFA models have regularly been used in simulation studies (Beauducel & Wittmann, 2005; Hu & Bentler, 1998, 1999). The general aim of this study was to investigate whether the promising results obtained in Muthén et al. (in press) for WLSMV estimation can be generalized to simple structure models as they are typically used for scale construction. However, Yu (2002) investigated WLSMV estimation in the context of simple structure by means of a simulation study. Yu primarily investigated the cutoff values of fit indexes with WLSMV. Marsh, Hau, and Wen (2004) discussed some problems with the rationale for the identification of cutoff values in Hu and Bentler (1999), which was also applied in Yu. To avoid these problems, this study did not investigate cutoff values but just described the differences in fit indexes as well as in model parameters. Moreover, the simple structure CFA models in Yu were based on the models used in Hu and Bentler (1998, 1999). Beauducel and Wittmann (2005) showed that, even with the same type of model, one can get different results in a simulation study when different loading magnitudes are used. It was therefore considered important to choose a loading magnitude that represented typical loadings for a given research setting. Because in many areas of psychological research, only moderate loadings will often occur, we chose a moderate loading magnitude of.50 as a basis for further simulations. This is considerably smaller than the loading magnitude in Muthén et al. (in press) and in Yu but mirrors typical CFA appli-

3 188 BEAUDUCEL AND HERZBERG cation in a variety of psychological research settings. Although Muthén et al. used a longitudinal three-factor model, their model was similar to a confirmatory factor model with three factors. However, in the confirmatory factor models investigated here, no temporal order between the factors was assumed. It was shown in Muthén et al. and in Yu that WLSMV works well with large loadings. The aim of this study was to extend the results of Muthén et al. and of Yu in that a moderate loading size was used. When the loading size decreases, one would expect an increased estimation bias both for ML estimation and for WLSMV estimation. Thus, the pivotal point of this study was to compare the results of ML estimation with the results of WLSMV estimation on the basis of a moderate loading size of.50. Muthén et al. and Yu used models with three factors for their analyses. This study investigates whether their results can be generalized across different numbers of factors. Therefore, the number of factors was manipulated between one, two, four, and eight factors. This study tried to generalize the results of Muthén et al. and Yu to models with moderate loadings and varying numbers of factors. This study was thus aimed to compare the ML and the WLSMV estimation, because ML estimation is probably the most common type of parameter estimation in SEM. The ML estimation is the method of choice when the data conform to the multivariate normal distribution. On the other hand, WLSMV may be a method of choice with categorical data. WLSMV may be an interesting alternative, because WLS as another alternative requires very large samples (more than 1,000 cases) and because WLS-based standard errors showed large amounts of negative bias (DiStefano, 2002). According to Muthén et al. (in press), the sample size required for WLSMV may be smaller than the sample size needed for WLS. However, it is not known whether WLSMV also requires a sample size as large as WLS when the size of the salient loadings is only moderate. Another aim of this study was, therefore, to investigate whether WLSMV estimation needs larger sample sizes than ML estimation in simple structure models with moderate salient loadings. When sample size is investigated, it is important to vary the number of parameters to be estimated, because they may interact with the required sample size. Therefore, the number of factors was varied in this study. In many research settings, it is not clear whether the variables should be regarded as categorical or whether the number of categories is sufficiently large to consider the variables as continuous. It is, therefore, important to investigate variables with different numbers of categories to determine with how many categories ML estimation will be superior to WLSMV estimation. The number of categories was not varied in Yu (2002), because Yu used variables with two categories or continuous variables. This study may, therefore, provide additional information on the differences between WLSMV and ML estimation. To summarize, the aims of this study were to extend knowledge on the WLSMV estimation, especially in comparison to ML estimation, and to investigate whether the promising results of Muthén et al. (in press) for WLSMV could also be found

4 ML VERSUS WLSMV ESTIMATION 189 with typical CFA models with a moderate loading size. More specifically, it was investigated whether especially large sample sizes are needed with WLSMV and how many categories the variables should have so that they cannot be treated as continuous, leading to superior performance of WLSMV estimation over ML estimation. The comparison of ML and WLSMV was performed for the model parameters, that is, the loadings, and standard errors of the loadings, as well as for those fit indexes that are available both with ML and with WLSMV estimation. This study addressed issues that were not addressed in previous simulation studies on WLSMV, but, of course, some important issues were not considered. With any simulation study, there are innumerable conditions to manipulate and choices need to be made to keep the design manageable. Therefore, there were some important limitations in this simulation study. For example, the effect of different skewness of the variables and the effect of different loading magnitudes were not addressed in this study. However, acceptable results were obtained with skewed variables in Muthén et al. (in press) with a sample size of 400 cases. Thus, skewness seems not to be a problem that is specific to WLSMV estimation. Even when not all possible aspects of empirical data sets can be covered by a single simulation study, this study provides information on the performance of WLSMV estimation in comparison to ML estimation for models that are typical in the context of social science research based in CFA. METHOD The simulation study was performed for four different samples sizes (250, 500, 750, 1000), with four different numbers of variables (5, 10, 20, and 40 with 1, 2, 4, and 8 latent factors, respectively) and five numbers of categories in the variables (2, 3, 4, 5, and 6). The distributions of the variables were generated on the basis of the binomial distribution. Binomial distributions with a larger number of categories are more similar to the normal distribution than binomial distributions with a smaller number of categories. Therefore, the similarity to the normal distribution was manipulated through the number of categories. Overall, there were 16 types of independent data sets (4 sample sizes 4 numbers of factors), and for each type of data 500 data sets were created so that 8,000 data sets containing continuous normal distributed variables were created. Because both orthogonal and oblique models were investigated, 8,000 data sets corresponding to an orthogonal population model and 8,000 data sets corresponding to an oblique model were generated. To provide a direct comparison of the effect of categorization on model parameters, the data sets with variables containing 2, 3, 4, 5, and 6 categories were derived from the 2 8,000 independent data sets. Thus, a total amount of 40,000 data sets, independent with respect to sample size and number of factors and dependent with respect to the number of categories, was created.

5 190 BEAUDUCEL AND HERZBERG Data sets were generated with SPSS for Windows (SPSS, 2001). To produce the correlation matrices, variables containing z-standardized, normally distributed random numbers were computed for the given sample size (250, 500, 750, or 1,000). Then, these variables were weighted and aggregated to produce variables having asymptotically the correlation needed according to the population models. When two aggregates have a variable in common, they will correlate because of this common variable. If, for example, four variables loading on one factor are created, it is necessary to create four aggregates, composed each by a single random variable, representing the specific variance, and a common variable. The common variable will be a fifth random variable that will be the same in the four aggregates (which will, therefore, represent the common variance). The weighting procedure necessary to produce specific covariances or correlations is described in Knuth (1981) and in Beauducel and Wittmann (2005). The standard deviation of the normally distributed measured variables created by this procedure was 10 and their mean was zero. As an example, the following formula illustrates the data generation of the two measured variables A and B that have a standard deviation of 10 (a variance of 100) and an intercorrelation of.25 leading to a common factor loading of.50 in the completely standardized solution. Three z-standardized, normally distributed random variables z1, z2, and z3 are weighted and aggregated. A = (75) 0.5 z1 + (25) 0.5 z3 (1) B = (75) 0.5 z2 + (25) 0.5 z3 (2) The variable z3 represents the common variance of A and B, z1 represents the specific variance of A, and z2 the specific variance for B. The weights are chosen in a way that the sum of the squared weights yields 100. The correlation of A and B is determined by the two weights of z3inaand in B. Because the weight of z3is5(= ) both in A and in B, the covariance of A and B is given by the weight of z3ina times the weight of z3inb (5 5) and is therefore 25, divided by 10, the standard deviations of A and B, which gives a correlation of.25. This correlation would lead to salient loadings of.50 on the corresponding factor. However,.25 is the value for the true correlation here. Due to sampling error, the exact value of the correlation between A and B will vary around this true value, with smaller variations for larger sample sizes. For each factor, five variables with population loadings of.50 were created. The nonsalient population loadings were zero on all factors. In addition to the orthogonal population model with population loadings of.50 on each factor, an oblique population model was created. The oblique model had a population loading of.55 on each factor and a population correlation of.33 between all factors. A moderate correlation between the factors was regarded as realistic for most studies in the social sciences. The number of variables per factor, the number of factors, and all other characteristics of the simulation were exactly the same as for the orthogonal model.

6 ML VERSUS WLSMV ESTIMATION 191 To investigate the effect that categorization has on the correlations and on the corresponding model parameters, the normally distributed continuous variables were categorized according to the binomial distribution. That is, the cutoff values were chosen in a way that the resulting distribution corresponds to the binomial distribution for the respective number of categories. The distribution for the variables with two categories was created by means of the following cutoff criterion: Values equal or larger than zero were set to one and values below zero were set to zero. The cutoff values were 0.67 and 0.67 for three categories, to achieve a response probability of.250 for the first category,.500 for the second category, and.250 for the third category. The cutoff values were 1.15, 0.00, and 1.15 for four categories, and the response probability from the first to the fourth category were.125,.375, 375, and.125. The cutoff values for five categories were 1.53, 0.49, 0.49, and 1.53 for five categories, and the corresponding response probabilities for the first to the fifth category were.063,.250,.375,.250, and.063. The cutoff values for six categories were 1.86, 0.89, 0.00, 0.89, and 1.86 for six categories, and the corresponding response probabilities for the first to the sixth category were.031,.156,.313,.313,.156, and.031. A binomial distribution with six categories does not correspond to a normal distribution; however, there are many instances in psychological research in which variables with six or a lower number of categories are used. The data sets created by this procedure were analyzed with Mplus 3.11 (Muthén & Muthén, 2004) to estimate the model parameters. For each of the 40,000 data sets, a model was estimated by means of ML and by means of WLSMV. The models were estimated according to the population model; thus, when salient loadings were to be expected, the loadings were freely estimated and when zero loadings were expected, the loadings were fixed at zero. For the data based on orthogonal population models, orthogonal models were specified, and for the data based on oblique models, oblique models were specified. There was no model misspecification. The parameters of these model estimations are the basis for the following analyses, which were performed with SPSS for Windows Repeated measures analysis of variance (ANOVA) was performed for those fit indexes available for both ML and WLSMV estimation in Mplus 3.11, that is, the chi-square value, the comparative fit index (CFI), Tucker Lewis Index (TLI), root mean squared error of approximation (RMSEA), and the standardized root mean squared residual (SRMR). Repeated measures ANOVA was also performed for the model parameters (completely standardized loadings and standard errors of loadings) as dependent variables. Because the ML and WLSMV estimations of model parameters were based on the same data sets, the factor Estimation Method comprising the levels ML and WLSMV was entered as within-subjects factor. Moreover, the data sets based on variables with 2, 3, 4, 5, and 6 categories were based on the same initial data sets with continuously and normal distributed variables. Therefore, the factor Categories comprising five levels was also entered as within-subjects factor. Degrees of freedom based on tests involving this repeated

7 192 BEAUDUCEL AND HERZBERG measures factor with more than two levels were Greenhouse Geyser corrected. The factors Sample Size (250, 500, 750, and 1,000 cases) and Number of Factors (1, 2, 4, and 8) were based on independent data sets so that they were entered as between-subjects factors. For both the orthogonal and the oblique simulation, the sample size for the simulated data comprises 8,000 cases for fit indexes and 40,000 cases, 80,000, 160,000, or 320,000 cases for model parameters. Therefore, even small effects would reach significance in repeated measures ANOVA. The evaluation of the results was, therefore, primarily based on η 2 as a measure for effect size. RESULTS Model Fit First, ML and WLSMV estimation were compared with respect to fit statistics. The chi-square values of ML and WLSMV cannot be compared, because they are based on different degrees of freedom (see Table 1). Degrees of freedom for ML are df = p q, where p is the number of sample moments and q is number of distinct parameters, whereas degrees of freedom could not be computed directly for WLSMV. Instead, the degrees of freedom for WLSMV estimation are calculated from the estimated covariance matrix and a diagonal weight matrix (see Muthén & Muthén, 2001, pp ). The question of different model fit can only be investigated in the p values. Therefore, the subsequent comparisons will be performed on the basis of p values. The main effect for the factor Estimation Method was significant, but the effect size was so small both for the orthogonal case, F(1, 7984) = 20.69, p <.001, η 2 =.003, and for the oblique case F(1, 7984) = 73.79, p <.001, η 2 =.009, that it could not be regarded as substantial. Thus, there is no substantial overall difference in the p values. A more relevant effect size occurred for the interaction of Estimation Method with the factor Categories: orthogonal case: F(3.69, ) = , p <.001, η 2 =.045; oblique case: F(294, ) = , p <.001, η 2 =.162. Larger p values occurred for WLSMV based on two and three categories and larger p values occurred for ML based on five and six categories. A significant and substantial main effect occurred for Number of Factors: orthogonal case: F(3, 7984) = , p <.001, η 2 =.126; oblique case: F(3, 7984) = , p <.001, η 2 =.100, indicating that the p values of both WLSMV and ML solutions decreased with an increasing number of factors (see Table 1). Sample Size had a small main effect on p values: orthogonal case: F(3, 7984) = 90.28, p <.001, η 2 =.033; oblique case: F (3, 7984) = 48.43, p <.001, η 2 =.018. This effect is nevertheless interesting, because it reflects an increase of p values with sample size. Because the statistical power increases with sample size, one would have expected a decrease of the p values with sample size. However, the sampling error de-

8 ML VERSUS WLSMV ESTIMATION 193 TABLE 1 Means for the χ² Values, the Degrees of Freedom, and Means for the p-values Orthogonal Simulation Oblique Simulation Number of Method Factors 2 (df) p 2 (df) p ML (5) (5).49 WLSMV (5) (5).49 ML (35) (34).21 WLSMV (25) (27).19 ML (170) (164).21 WLSMV (97) (117).18 ML (740) (712).21 WLSMV (261) (249).15 Note. squares. ML = maximum likelihood; WLSMV = mean and variance adjusted weighted least creases with sample size and the models had no specification error so that the increasing p values reflect the increasing similarity of the empirical data with the population model in this case. The mean of the p values is greater.05, which means that most models would be accepted at the.05 level of significance (see Table 1). This was, of course, expected, because there was no model misspecification. The effect size of the interaction of Estimation Method with Sample Size was extremely small for the p values in the orthogonal case, F(3.00, ) = 3.69, p <.05, η 2 =.001, and was a bit more substantial in the oblique case, F(3.00, ) = , p <.001, η 2 =.053. In the orthogonal case, the effect is irrelevant and in the oblique case the effect indicates that the p values increase very strongly with sample size in ML estimation, whereas there is only a minimal increase of p values with sample size with WLSMV. It is interesting to investigate the model rejection rates with respect to the number of categories, because the ML and WLSMV estimation should perform differently with respect to the number of categories. According to an alpha level of.05, one should expect that about 5% of the correctly specified models will be rejected on the basis of the chi-square test. However, the percentage of rejected models was larger than 5% for the ML estimation with all numbers of categories (both for the orthogonal and the oblique models). What was unexpected was the slight increase of the percentage of rejected models with increasing number of categories for the ML estimation of orthogonal models, because ML estimation was expected to work more convincingly when the distributions of the variables become more close to the normal distribution. The rejection rates for WLSMV estimation were close to the 5% level for two and three categories and nearly the same as for ML estimation for six categories (see Table 2). However, with six categories and oblique

9 194 BEAUDUCEL AND HERZBERG TABLE 2 Model Rejection Rates at the.05 Alpha Level (in Percent Out of 8,000 Models) Number of Categories Orthogonal case ML WLSMV Oblique case ML WLSMV Note. squares. ML = maximum likelihood; WLSMV = mean and variance adjusted weighted least models, the rejection rate of the WLSMV models was even higher than the rejection rate of the ML models. It was expected that for the low number of categories the difference between the rejection rates of WLSMV and ML was larger, because the performance of ML estimation should decrease and the rejection rates of ML estimation should increase with a decreasing number of categories. For WLSMV estimation, the rejection rates were expected to be rather close to the 5% level of significance with a low number of categories, because this method should work well under these conditions. For CFI, a substantial main effect for Estimation Method occurred: orthogonal case: F(1, 7984) = , p <.001, η 2 =.108; oblique case: F(1, 7984) = , p <.001, η 2 =.167. The CFI values for WLSMV estimation were larger than for ML estimation. However, there was also an interaction of Estimation Method with Categories: orthogonal case: F(3.33, ) = , p <.001, η 2 =.075; oblique case: F(3.45, ) = , p <.001, η 2 =.174. This interaction points to the fact that the CFI based on WLSMV was larger than the CFI based on ML for variables with two and three categories but that there was no difference in CFI for five and six categories. There was a substantial main effect for Number of Factors: orthogonal case: F(3, 7984) = , p <.001, η 2 =.227; oblique case: F(3, 7984) = , p <.001, η 2 =.127, which was related to an decrease of CFI with the number of factors. The main effect for Sample Size was even larger: orthogonal case: F(3, 7984) = , p <.001, η 2 =.385; oblique case: F(3, 7984) = , p <.001, η 2 =.366, indicating that the CFI increased with sample size. For TLI, the effect size of the main effect for Estimation Method was moderate: orthogonal case: F(1, 7984) = , p <.001, η 2 =.015; oblique case: F(1, 7984) = , p <.001, η 2 =.072. The TLI based on ML was a bit smaller than the TLI based on WLSMV, but the mean difference is without any practical meaning be-

10 ML VERSUS WLSMV ESTIMATION 195 cause it was smaller than.002. The effect size of the interaction of Estimation Method with Categories was also moderate (orthogonal case: F(3.17, ) = , p <.001, η 2 =.057; oblique case: F(3.45, ) = , p <.001, η 2 =.102. This interaction points to the fact that the TLI based on WLSMV was larger than the TLI based on ML for variables with two and three categories but that it was the reverse for variables with five and six categories. There was a moderate main effect for Number of Factors: orthogonal case: F(3, 7984) = , p <.001, η 2 =.134; oblique case: F(3, 7984) = , p <.001, η 2 =.075, which was related to a decrease of TLI with the number of factors. The main effect for Sample Size had a similar size: orthogonal case: F(3, 7984) = , p <.001, η 2 =.102; oblique case: F(3, 7984) = , p <.001, η 2 =.087, indicating that the TLI increased with sample size. There was no specification error so that the models should fit in the population. In consequence, many zero values occurred for RMSEA so that the RMSEA was not normally distributed. Of course, due to the large sample size, the effects reported here were significant even when tested by means of nonparametric tests (Friedman Test, Kruskal Wallis Test). However, the nonparametric tests do not allow for the calculation of effect sizes; therefore, the ANOVA results were reported here, even when they should be interpreted cautiously. For RMSEA, the significant main effect for Estimation Method was only small in size: orthogonal case: F(1, 7984) = , p <.001, η 2 =.036; oblique case: F(1, 7984) = , p <.001, η 2 =.035. The effect size of the interaction of Estimation Method with Categories was a bit larger: orthogonal case: F(3.69, ) = , p <.001, η 2 =.068; oblique case: F(2.81, ) = , p <.001, η 2 =.099. For two categories, the RMSEA based on WLSMV was smaller than the RMSEA based on ML; for four to six categories, it was the reverse. Figure 1 represents the interaction for the orthogonal case; for the oblique case, the interaction was similar. One can see from Figure 1 that there was not much room for large effect sizes, because all RMSEA means were extremely small. This was not surprising because the RMSEA reflects the fit of the population model, which was perfect in this case, because there was no model misspecification. The main effect for Number of Factors was small: orthogonal case: F(3, 7984) = 77.80, p <.001, η 2 =.028; oblique case: F(3, 7984) = , p <.001, η 2 =.051, pointing to a slight increase of RMSEA with Number of Factors. A larger effect occurred for Sample Size: orthogonal case: F(3, 7984) = , p <.001, η 2 =.172; oblique case: F(3, 7984) = , p <.001, η 2 =.166, indicating a substantial decrease of the RMSEA with sample size. For SRMR, the main effect for Estimation Method is very large in size: orthogonal case: F(1, 7984) = , p <.001, η 2 =.979; oblique case: F(1, 7984) = , p <.001, η 2 =.961. Although the effect size appeared to be large in ANOVA, the mean differences between SRMR based on an ML and WLSMV were small (see Table 3). The large effect size in ANOVA was due to the large sample size (8,000 cases), which leads to very small standard errors. The ef-

11 196 BEAUDUCEL AND HERZBERG FIGURE 1 Mean RMSEA based on ML and WLSMV estimation for variables with two, three, four, five, and six categories. fect size of the interaction of Estimation Method with Categories was also very large: orthogonal case: F(2.06, ) = , p <.001, η 2 =.938; oblique case: F(2.56, ) = , p <.001, η 2 =.853. The interaction is related to the effect that the SRMR based on WLSMV is much larger than the SRMR based on ML for two and three categories, whereas the WLSMV based SRMR is only slightly larger for five and six categories. The main effect for Number of Factors was large: orthogonal case: F(3, 7984) = , p <.001, η 2 =.896; oblique case: F(3, 7984) = , p <.001, η 2 =.871, pointing to an increase of SRMR with number of factors. A large main effect occurred also for Sample Size: orthogonal case: F(3, 7984) = , p <.001, η 2 =.905; oblique case: F(3, 7984) = , p <.001, η 2 =.902, indicating a decrease of the SRMR with sample size. To give an impression of the overall fit of the models, the means and standard deviations of the fit indexes were reported in Table 3. Overall, one can conclude from the fit indexes that the models fit the data well, which was expected, because no model misspecification was introduced. These overall fitting models were a good basis for the comparison of WLSMV and ML with respect to the size of the model parameters. Model Parameters There was a significant and large effect for Estimation Method on the magnitude of the completely standardized loadings: orthogonal case: F(1, ) = , p <.001, η 2 =.823; oblique case: F(1, ) = , p <.001, η 2 =.674. The size of the interaction of Estimation Method and Categories was also important: orthogonal case: F(2.98, ) = , p <.001, η 2 =.694; oblique case: F(2.76, ) = , p <.001, η 2 =.620. Both the main

12 ML VERSUS WLSMV ESTIMATION 197 TABLE 3 Means and Standard Deviations of the Reported Fit Indexes Orthogonal Case Oblique Case Index Mean (SD) Mean (SD) CFI ML.98 (.03).99 (.02) CFI WLSMV.98 (.02).99 (.01) TLI ML.99 (.04).99 (.03) TLI WLSMV.99 (.03) 1.00 (.01) RMSEA ML.01 (.01).01 (.01) RMSEA WLSMV.01 (.01).01 (.01) SRMR ML.03 (.01).03 (.01) SRMR WLSMV.04 (.02).04 (.01) Note. ML = maximum likelihood; WLSMV = mean and variance adjusted weighted least squares; CFI = comparative fit index; TLI = Tucker Lewis Index; RMSEA = root mean squared error of approximation; SRMR = standardized root mean square residual. effect and the interaction were related to the effect that the mean of the WLSMV loadings was close to the population value (.50 for the orthogonal simulation,.55 for the oblique simulation) for all numbers of categories, whereas the mean of the ML loadings was much lower for two categories and increased considerably with the number of categories. Figure 2 represents the interaction for the orthogonal case, the interaction was similar for the oblique case. However, even with five and six categories the mean of the WLSMV loadings was more close to the population value than the mean of the ML loadings. This result indicates to what degree it is possible to estimate a population value for continuous variables from a sample of FIGURE 2 Mean loadings (completely standardized) based on ML and WLSMV estimation in relation to the number of categories. The standard errors of the means were not given, because they were extremely small due to a sample size of 150,000 cases for model parameters.

13 198 BEAUDUCEL AND HERZBERG categorical variables. As one can see from Figure 2, WLSMV estimation performed especially well with respect to the loading size. The sizes of the main effects of Sample Size and Number of Factors on the magnitude of the loadings were extremely small η 2 <.01, and the effect sizes of the interactions of Sample Size and Number of Factors with Estimation Method were η 2 <.001, which means that for both ML and WLSMV the size of the loadings was not substantially affected by Sample Size and the Number of Factors (both for the orthogonal and the oblique case). Another important aspect of the model parameters is the standard error of the loadings. This does not refer to the standard errors of the means across the data sets in the simulation but to the standard errors of the loadings as they are computed for each individual model. These standard errors are an indicator of the precision of the estimation procedure. When the standard errors of the loadings were entered as dependent variable into repeated measures ANOVA, a moderate main effect occurred for Estimation Method: orthogonal case: F(1, ) = , p <.001, η 2 =.153; oblique case: F(1, ) = , p <.001, η 2 =.097. The mean standard errors of the WLSMV loadings were smaller than the mean standard errors of the ML loadings. This was the case across all conditions of the simulation. One might have expected an interaction of Estimation Method with Categories for the standard errors; however, the effect size of this interaction was extremely small although significant with the large sample: orthogonal case: F(1.12, ) = , p <.001, η 2 =.001; oblique case: F(2.65, ) = , p <.001, η 2 =.002. There is a small interaction of Estimation Method with Sample Size: orthogonal case: F(3, ) = , p <.001, η 2 =.033; oblique case: F(3, ) = , p <.001, η 2 =.016, pointing to the fact that the mean difference between the WLSMV and ML standard errors decreased slightly with sample size. It was not surprising that there was also a substantial main effect of Sample Size on the standard errors with larger sample sizes leading to smaller standard errors: orthogonal case: F(3, ) = , p <.001, η 2 =.110; oblique case: F(3, ) = , p <.001, η 2 =.126. Figure 3 gives an impression of the magnitude of the mean difference of the standard errors of the loadings for ML and WLSMV estimation for different sample sizes. The size of the main effect for Number of Factors was extremely small η 2 <.001, which means that for both ML and WLSMV the size of the standard errors was not substantially affected by the number of factors. For the oblique model, the effects of the Estimation Method on the interfactor correlations was also investigated. There was a substantial main effect for Estimation Method on the size of the correlations: F(1, 69988) = , p <.001, η 2 =.335. Moreover, there was a moderate interaction between Estimation Method and Categories: F(2.81, ) = , p <.001, η 2 =.094. This effect was related to a small tendency of WLSMV to overestimate the interfactor correlations, especially when based on a large number of categories. The tendency was small,

14 ML VERSUS WLSMV ESTIMATION 199 FIGURE 3 Mean standard errors of the loadings (SE) based on ML and WLSMV estimation in relation to sample size. because an overall mean of.35 was estimated with WLSMV instead of.33, which was the population value. However, the mean interfactor correlation estimated with ML was close to.33, even with two and three categories. The effect size of the interaction of Estimation Method with Sample Size was small: F(3, 69988) = , p <.001, η 2 =.019, as well as the effect size of the interaction of Estimation Method with the Number of Factors: F(3, 69988) = , p <.001, η 2 =.012. The largest amount of overestimation occurred for WLSMV with 2 factors, 250 cases, and 6 categories, with a mean correlation of.38. The lowest amount of overestimation occurred for 8 factors, 1,000 cases, and 2 categories, with a mean correlation of.34. DISCUSSION The general aim of this simulation study was to investigate whether the promising results of Muthén et al. (in press) for WLSMV could also be found with typical CFA models with a moderate loading size and with different numbers of factors. More specifically it was investigated whether especially large sample sizes are needed with WLSMV and how many or few categories variables should have so that they cannot be treated as continuous leading to superior performance of WLSMV estimation over ML estimation. These aims were investigated by means of a simulation study, which was based on CFA models with 1, 2, 4, and 8 factors, based on 250, 500, 750, and 1,000 cases, and variables with 2, 3, 4, 5, and 6 categories. There were five variables per factor and there was no model misspecification. The simulations were performed on the basis of an orthogonal population model

15 200 BEAUDUCEL AND HERZBERG with population loadings of.50 and on the basis of an oblique population model with population loadings of.55 and interfactor correlations of.33. It was found that WLSMV estimation performed as well as ML estimation across all sample sizes, because the effect size of the interaction of Estimation Method with Sample Size was small for p values of the chi-square tests (η 2 =.001 for the orthogonal case; η 2 =.053 for the oblique case). Moreover, for all sample sizes and for all number of categories, the mean size of the WLSMV loadings was closer to the continuous variables population loading (.50 for the orthogonal case;.55 for the oblique case) than the mean size of the ML loadings. The effect size of the interaction of Estimation Method with Sample Size was extremely small (η 2 <.001) for the loadings of both the orthogonal and the oblique models, showing that sample size does not affect the loadings in WLSMV and ML estimation in a different way. Overall, these results demonstrate that WLSMV estimation needs not the large sample sizes needed for WLS estimation. Overall, a clear superiority of WLSMV over ML estimation was found for categorical variables with two and three categories. For example, the model rejection rates based on an.05 alpha level of the p values of the chi-square tests were close to the.05 alpha level for WLSMV with data based on two or three categories. This should be the case, because there was no model misspecification and because a.05 alpha level means that in 5% of the cases the correct model will be rejected by the chi-square tests. However, the rejection rates for variables with two and three categories were about twice as large for ML estimation. This means that there was a tendency to overreject correct models based on variables with two and three categories with ML estimation but not for WLSMV estimation. This result replicates the results of Yu (2002), who also found a tendency to overreject for chi-square tests based on ML estimation with two categories. In addition to Yu, a tendency to overreject correct models was also found for WLSMV estimation when based on variables with five and six categories. However, a tendency to overreject correct models was also found for ML estimation based on variables with five and six categories. The following can be summarized for the remaining fit indexes: CFI, TLI, and RMSEA indicated superior model fit when based on WLSMV and two and three categories. It should be noted that the lower CFI and TLI based on ML estimation and Pearson correlations of categorical variables is partly based on the effect that the Pearson correlations underestimate the true relation when the variables are categorical. Because ML is sometimes used with categorical variables and with Pearson correlations, the demonstration is nevertheless important. The underestimation of the true relation when Pearson correlations are used may explain the low performance of CFI and TFI when based on ML estimation and a low number of categories. Because the difference between the actual model and the null model will be underestimated when the correlations are underestimated, the ML-based CFI and TFI were especially affected by the low number of categories. When

16 ML VERSUS WLSMV ESTIMATION 201 based on five and six categories, there was no difference in ML and WLSMV based CFI, but the TLI and the RMSEA tended to indicate superior model fit when based on ML estimation and variables with five and six categories. Because there was no model misspecification, this may indicate that the performance of the ML-based fit assessment increased with five and six categories. However, the effect was the reverse for SRMR: The SRMR based on WLSMV estimation was larger than SRMR based on ML estimation when based on variables with two or three categories. With an increasing number of categories, the difference in the SRMR diminished. However, due to its sensitivity to sample size, Yu (2002) did not recommend the use of SRMR with binary variables. The sensitivity to sample size was also found in this study. However, with five and six categories, the SRMR based on WLSMV was not very different from the WLSMV based on ML. Although further studies may be necessary to give a more complete picture, it seems that SRMR may be useful at least with five or six categories. Overall, the picture for the fit indexes was less clear than for the chi-square tests. This points to the fact that different threshold values may be appropriate for fit indexes when based on ML or WLSMV estimation. This is an argument for those who challenged the tendency to develop golden rules of model fit (Marsh et al., 2004). However, no conclusion with respect to the superiority of ML or WLSMV estimation can be drawn from the results on the descriptive fit indexes. There was, however, a clear tendency to underestimate the size of the loadings with ML estimation when the variables had only two or three categories. This tendency diminished with increasing number of categories, but even with six categories, there was a slight tendency to underestimate the magnitude of the loadings with ML estimation. The effect size of the main effect corresponding to the tendency to underestimate the loadings with ML was very large (η 2 =.823 for the orthogonal case and η 2 =.674 for the oblique case). Moreover, the standard errors of the loadings were a bit smaller for WLSMV than for ML estimation across all number of categories, which means that one can trust the WLSMV parameters even a bit more than one can trust the ML parameters. Thus, if the size of the model parameters is important in a research setting and if the variables have only two or three categories, the use of WLSMV estimation instead of ML estimation can be recommended. This is in line with Dolan (1994), who found that five response categories are a minimum when ML estimation is performed. However, even with four and five categories, the performance of WLSMV estimation was slightly superior to the performance of ML estimation, especially with respect to the bias of the loadings, so that the use of WLSMV with such variables is encouraged on the basis of these results. The point of a clear superiority of ML estimation will probably be reached with a larger number of categories; however, this has to be investigated in further studies. It is clear that a method like WLSMV, which was designed to deal with categorical variables, cannot outperform ML estimation when the number of categories is very large.

17 202 BEAUDUCEL AND HERZBERG When the number of categories is large, one may pay attention to the possibility of an overestimation of the intercorrelations between factors with WLSMV. This effect may get a relevant size when the sample size is small (about 250 cases) and when the model contains only two factors. When the number of categories is small and the number of factors is large, the effect can be neglected. However, further research would be necessary to describe the effect more completely. Because WLSMV does not need a larger sample size than ML estimation and because it leads to more correct chi-square rejection rates and to more precise loading estimates with two and three categories, these results can be regarded as an extension of the promising results on WLSMV in Muthén et al. (in press) for the case of typical CFA models with moderate loading size and with a large number of factors. It seems that the WLSMV estimation compensates more effectively than the ML estimation for the bias that is due to the categorical aspects of the variables and that WLSMV does not have the disadvantages of WLS estimation. It should be noted that the simulation covered models based on rather small sample sizes of 250 cases and that it also covered rather complex models with eight factors based on only moderate salient loadings. Even in these small sample, large model, moderate loading cases WLSMV estimation performed well. It should, however, be noted that several issues remain to be investigated: For example, the effect of skewness and kurtosis of the variables, the effect of different loading magnitudes, the effects of model misspecification, and the extension of the results to different types of SEM models should be investigated in further studies. REFERENCES Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indices in confirmatory factor analysis based on data with slightly distorted simple structure. Structural Equation Modeling, 12, DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47, Ferrando, P. J., & Chico, E. (2001). The construct of sensation seeking as measured by Zuckerman s SSS-V and Arnett s AISS: A structural equation model. Personality and Individual Differences, 31, Haynes, C. A., Miles, J. N. V., & Clements, K. (2000). A confirmatory factor analysis of two models of sensation seeking. Personality and Individual Differences, 29, Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modelling. Sociological Methods & Research, 26, Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1 55.

18 ML VERSUS WLSMV ESTIMATION 203 Knuth, D. E. (1981). The art of computer programming (Vol. II, 2nd ed.). Reading, MA: Addison-Wesley. Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler s (1999) findings. Structural Equation Modeling, 11, Muthén, B. O. (1993). Goodness of fit with categorical and other non-normal variables. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp ). Newbury Park, CA: Sage. Muthén, B. O., du Toit, S. H. C., & Spisic, D. (in press). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Muthén, L. K., & Muthén, B. O. (2001). Mplus, user s guide (Version 2). Los Angeles: Author. Muthén, L. K., & Muthén, B. O. (2004). Mplus, user s guide (Version 3.11). Los Angeles: Author. SPSS. (2001). SPSS for Windows, Release Chicago: Author. Thurstone, L. L. (1947). Multiple factor analysis. Chicago: University of Chicago Press. Tomas, J. M., & Oliver, A. (1999). Rosenberg s self-esteem scale: Two factors or method effects. Structural Equation Modeling, 6, Yu, C.-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Unpublished doctoral dissertation, University of California, Los Angeles.