Evaluating Factor Structures of Measures in Group Research: Looking Between and Within

Transcription

1 Group Dynamics: Theory, Research, and Practice 2016 American Psychological Association 2016, Vol. 20, No. 3, /16/$ Evaluating Factor Structures of Measures in Group Research: Looking Between and Within Rebecca A. Janis Pennsylvania State University Gary M. Burlingame and Joseph A. Olsen Brigham Young University Data from psychotherapy groups are nested by nature. Ignoring this nesting when performing statistical tests can result in inflated Type I error rates and spurious significant results. This presents a serious problem for interpreting research that does not account for members nested within a specific group. Factor analytic methods are not immune to the negative effects of nesting, and when it is ignored, a poorly fitting factor structure can be hidden if one does not examine model fit at both the between or within-group level. Multilevel confirmatory factor analysis allows the researcher to account for nesting by separately examining both the between- and within-group structures of measures used in group research. This article presents an overview of methods for evaluating the level of group dependency using the intraclass correlation coefficient (ICC) and a comparison of 2 methods for calculating ICCs. It then provides an overview and an example of multilevel factor analysis as a method for testing the model fit at the between and within levels separately by using partially saturated models. The authors end by reviewing common problems and offering guidelines for interpreting differences in within- and between-level fit in group research. Keywords: group dependency, intraclass correlation coefficient, multilevel confirmatory factor analysis, partially saturated models Supplemental materials: Data collected from psychotherapy groups are, by design, nested. Throughout the therapy process, clients within a therapy group have the same therapist, interact with each other over time, and share common experiences during sessions. This natural grouping generally causes treatment responses of members within the same group to be more similar to one another when compared to members from a separate group who receive the same treatment. The effect of members nested within a group appears on process and outcome measures completed by Rebecca A. Janis, Department of Psychology, Pennsylvania State University; Gary M. Burlingame, Department of Psychology, Brigham Young University; Joseph A. Olsen, College of Family, Home, and Social Sciences, Brigham Young University. Correspondence concerning this article should be addressed to Rebecca A. Janis, Department of Psychology, Pennsylvania State University, State College, PA ruj134@psu.edu members in the same group, and this phenomenon is described as dependence among observations within groups. Observations of process and outcome are dependent if there is shared variance within groups, and in a nested design, this is to be expected (Baldwin et al., 2011; Zucker, 1990). This shared variance is often identified by an intraclass correlation coefficient (ICC), which measures how similar (or dissimilar) members within each group are to each other. Shared variance causes group data to violate the independence assumption of most statistical tests, which is the assumption that one observation does not tell you anything about another observation (Kenny & Judd, 1986). It has been acknowledged that violating this assumption in analysis of variance (ANOVA) and regression procedures has more serious consequences than violations of other assumptions (Kenny, Kashy, & Bolger, 1998). This produces spurious significant results and incorrect conclusions (Baldwin, Murray, & Shadish, 2005). 165

2 166 JANIS, BURLINGAME, AND OLSEN Even small levels of dependence, or small ICCs, can result in a large inflation of the Type I error rate if not analyzed properly. Simulation studies have found that Type I error rates can be inflated up to 58% (Burlingame, Kircher, & Honts, 1994) when group data are analyzed with the individual as the unit of analysis and group membership is ignored. For instance, when the ICC.2, Type I error rates ranged from 18% to 36% depending on the study design, an error rate that is 13% to 31% higher than the nominal level of 5%. Clearly, mishandling nested data has a huge impact on study results and their interpretation. Mishandling of dependent observations in group data has long been recognized as an issue in the field (Burlingame, Kircher, & Honts, 1994; Kenny & Judd, 1986; Kenny & la Voie, 1985; Wampold & Serlin, 2000). Two decades ago Burlingame, Kircher, and Taylor (1994) reviewed 192 group psychotherapy studies and found that 89% did not address dependence in the data. Ten years later, Baldwin, Murray, and Shadish (2005) evaluated 33 studies testing empirically supported group treatments, and none adequately accounted for the nested nature of the data. After applying corrections to the results using different magnitudes of dependencies, up to 59% of the studies no longer had significant results. Unfortunately, the problem affects the most recent clinical literature, with most studies ignoring dependence (Burlingame, Strauss, & Joyce, 2013) at a rate similar to preceding decades (Burlingame, MacKenzie, & Strauss, 2004), making interpretation of results difficult. This is particularly discouraging since the studies selected for these two reviews were the most rigorous of the published group treatment literature. The neglect of group dependency also affects the social psychology group literature. Hoyle, Georgesen, and Webster s (2001) 15-year review of group processes found an increased acknowledgment of nonindependence but little progress in adequately addressing it. It is impossible to know why the field continues to ignore the effect of dependence, although some have suggested the analytic complexity as well as a bias toward finding significant effects (Wampold & Serlin, 2000). Our goal in this paper is to address the former analytic complexity by describing and illustrating in simple language methods for assessing and addressing dependence in group data. Our hope is that future research will correctly analyze group data so that our field can move forward on this front. Methodological Recommendations for Group Dependency One of the most powerful techniques proposed to handle group dependency is multilevel modeling (Hoyle, Georgesen, & Webster, 2001; Kenny, Mannetti, Pierro, Livi, & Kashy, 2002). A multilevel model allows one to model information about differences between individual group members, how the group affects its members (shared within group variance), and any other fixed or random effects, such as the effect of a treatment (Tasca, Illing, Joyce, & Ogrodniczuk, 2009; Tasca, Illing, Ogrodniczuk, & Joyce, 2009). Beyond accounting for dependence when testing for treatment effects, it is equally important when validating the factor structure of measures used in group research. Measures are at the heart of conclusions drawn from group research, and Baldwin and colleagues (2011) have shown that dependency can vary widely from one measure to another. Unfortunately, the factor analyses conducted on measures used in the group therapy literature rarely take into account the grouped nature of the data (Burlingame, Kircher, & Taylor, 1994; Burlingame et al., 2004). They principally rely upon single level models that combine individual and group level variance. This procedure ignores group dependency, potentially creating artificially small standard errors and erroneously significant factor analytic results. Additionally, single level factor analysis obscures factor structures that may potentially differ between and within groups and assumes measurement and structural invariance that may not exist (Zyphur, Kaplan, & Christian, 2008). It is possible for items to relate differently to one another at the level of the individual member and the group, resulting in distinct latent factor structures (Kenny & la Voie, 1985). Multilevel factor analytic techniques allow researchers to discover theoretically important differences in relationships between observed variables and in latent constructs at each level of analysis.

3 FACTOR STRUCTURE OF GROUP MEASURES 167 Conceptual and Methodological Issues in Multilevel Modeling There are both conceptual and methodological issues to consider when one either searches for, or confirms latent factor structures (exploratory and confirmatory factor analyses, respectively) using multilevel modeling of group data. Conceptually one must first consider who is the most reliable source of data for each level in the model. Clearly, the individual member is the best source when modeling at the individuallevel (Burlingame, Fuhriman, & Drescher, 1984). However, there is a longstanding debate on who can provide reliable observations at the group level (Burlingame et al., 2004), which may explain the paucity of measures to assess group-level phenomenon (Burlingame, Fuhriman, & Johnson, 2002). Individual members have some, but not all of the group level information, and there is disagreement as to whether the sum of all member information truly captures group-as-a-whole phenomena (Burlingame et al., 1984, 2004). This disagreement is why behavioral rating systems completed by an independent rater have been offered as the most reliable source of group-level data (Beck & Lewis, 2000). In the absence of an independent rater, current convention is to use the individual source data and to assess the factor structures at the individual and group level (Huang & Cornell, 2016). Similarly, a primary methodological consideration in multilevel modeling of group data is establishing clarity on the desired inferences for the measure. Inferences regarding variables studied at the individual level are thought to be influenced by individual member properties making interpretation at this level crucial. For instance, past research has shown a reliable relationship between a client s perception of the therapeutic relationship and treatment outcome (Burlingame et al., 2004). More recent research has demonstrated that a client s attachment style an individual member property further interacts with member perception of the therapeutic relationship to predict outcome (Tasca, 2014). While there is some research linking a member s attachment style to group processes, the principal inference for measures of the therapeutic relationship in the group literature have remained at the individual level. Inferences at the group-level are typically associated with variables influenced by group properties. MacKenzie (1983) designed the Group Climate Questionnaire (GCQ) to tap universal group experiences of engagement, conflict, and avoidance, predicting that the levels of each would change as the group passed through four phases of development. He predicted that groups would begin with low levels of conflict that would increase as the group entered a storming phase of group development, which would then be followed by a reduction in conflict as the group moved into the working phase of group development. The GCQ is one of the most widely used measures in the clinical group literature and is a good example of a measure where inferences are made at the group level. However, we could not locate a single study that used multilevel modeling to test the three-factor structure separately at both the individual and group level with a longitudinal data set. This is, unfortunately, not a common occurrence in psychometric studies using clustered data. In summary, the statistical argument for accurately modeling clustered group data is well accepted as a method to avoid erroneous statistical findings. More recently, multilevel modeling of clustered data from school systems found different factor structures of school climate measures at the individual and group (class) level (Huang & Cornell, 2016). In short, the climate measures functioned differently depending upon level. Different factor structures are possible if theoretically supportable and levelappropriate inferences are made. However, it can create problems if inferences are made at the group level, but the factor structure is never tested for invariance at that level, a situation that describes most measures in the clinical group literature. For instance, it is psychometrically unknown whether the GCQ factors of engagement, avoidance and conflict are properties of the individual members who contributed the data or the group. In this paper, we provide an overview of ICCs and two methods for computing them: ANOVA and multilevel modeling. We then present multilevel factor analysis as a method for addressing group dependence and walk through interpreting results from a multilevel factor analysis. Finally, we present a method for determining model fit specifically at the be-

4 168 JANIS, BURLINGAME, AND OLSEN tween and within levels through partially saturated models. This work builds on prior research by reiterating the need and benefit of using multilevel techniques. It further augments the group literature by presenting a method for direct estimation of factor analytic models and evaluation of level specific fit in Mplus, newer techniques not fully utilized in the group psychotherapy literature. Group Example To illustrate group dependency and analytic models to detect and control for it, we selected a measure of the group relationship. Past research has suggested that members and groups differ widely on the experience of the therapeutic relationship (Burlingame et al., 2004), and cohesion is a reliable predictor of outcome for individuals within therapy groups (Burlingame, McClendon, & Alonso, 2011). The data are taken from three previously published studies using the Group Questionnaire (GQ; Krogel et al., 2013), an empirically derived measure of the therapeutic relationship. Data from these studies were collected from 18 university counseling centers in the United States and nonclinical groups held at the 2002 annual meeting of the American Group Psychotherapy Association (Chapman et al., 2012; Thayer & Burlingame, 2014; Johnson, Burlingame, Olsen, Davies, & Gleave, 2005). A previous study demonstrated factor invariance of the GQ in these two populations (Burlingame, Janis, Olsen, Thayer, & Berkeljon, 2015). The data from these three studies are pooled for the current paper yielding 1,058 group members treated in 195 groups with an average of 5.43 (SD 3.49) members per group. The GQ is a 30 item self-report measure that assesses the quality of the therapeutic relationship on a 7-point Likert scale from 1 (not true at all) to 7 (very true) using three subscales: positive bonding relationship (13 items), positive working relationship (eight items), and negative relationship (nine items). The GQ yields a score for each subscale with no total score. All three subscales have good reliability, with positive bond ranging from.79 to.92, positive work ranging from.85 to.90, and negative relationship ranging from.80 to.86 (Chapman et al., 2012; Krogel et al., 2013; Thayer & Burlingame, 2014). The GQ also assesses relationship structure using three dimensions: memberleader, member-member, and member-group, and it has shown acceptable criterion validity with the Working Alliance Inventory, Group Climate Questionnaire, Therapeutic Factors Inventory, and Empathy Scale (Thayer & Burlingame, 2014). The goal of the measure was to identify individual members who may be experiencing challenges in the therapeutic relationship that the group leader is unaware of (Chapman et al., 2012); the intended GQ inference is at the level of the individual member. Past research using the GQ has indicated significant dependence, although the ICC values have varied considerably between subscales and dimensions of the GQ. Past published ICC values for the three domains are as follows: positive bond.20, positive work.002, and negative relationship.08 (Chapman et al., 2012). Itemlevel ICCs ranged from.07 to.49, with average relationship dimension ICCs as follows: member-member,.15; member-leader,.11; and member-group,.27 (Thayer & Burlingame, 2014). Intraclass Correlation Coefficients ICCs ( ) provide a measure of how similar observations are within groups of clustered data (Baldwin et al., 2005), or the degree to which observations within groups are correlated (Kenny, Mannetti, Pierro, Livi, & Kashy, 2002). The ICC is most commonly defined as a ratio of the between group variance to total variance. Given that variance components are generally considered to be necessarily nonnegative, this produces a value ranging between 0 and 1. Calculating ICCs Traditionally, ICCs have been calculated through a one-way ANOVA with the following equation expressing the intraclass correlation in terms of the between groups ( b 2 ) and within groups ( w 2 ) variances. b 2 b 2 w 2 Using the estimates of these variance components from the expected mean squares for an

5 FACTOR STRUCTURE OF GROUP MEASURES 169 ANOVA, ICCs can be calculated from the ANOVA source table by MS b MS w MS w m 1 MS b where MS b and MS w are the mean squares between and within groups, respectively, and m is the size of the group (Kenny & Judd, 1986). This method is best suited for calculating ICCs from a single time point during treatment. For a discussion of methods of calculating ICCs using data from multiple time points within treatment, see Tasca, Illing, Ogrodniczuk, et al. (2009). Although this formula assumes equal group sizes, for groups of approximately similar size, an average group size can be used. For groups that drastically differ in size there are modifications that can be made to the formula (Donner & Koval, 1980; Kenny & la Voie, 1985). In our data, groups ranged from two to 20 members, with 90% of the groups between two and 10 members. In more recent literature, ICCs are more commonly estimated using multilevel modeling. Mplus (Muthén & Muthén, ) produces ICC estimates by calculating a ratio of within and between latent variable variance components using the maximum likelihood procedure; group is treated as a random effect. ICC estimates produced by Mplus will differ slightly from those calculated through the ANOVA source table. Mplus allows for the calculation of many ICCs at once, making it easy to calculate item level ICCs, a time consuming task when using the ANOVA table. Item level ICCs can vary dramatically within a single measure. For example, item level ICCs for the GQ in this data range from.06 to.50. One way to advance the field is by looking at these item level ICCs to ascertain which aspects of a construct are more or less sensitive to dependence. Interpretation The size of the ICC can vary depending on the method used to calculate it. Table 1 compares the ICCs derived from the ANOVA source table and from Mplus for each of the GQ subscales and relationship dimensions. The Mplus ICCs were consistently higher than those from the ANOVA source table. Table 1 Intraclass Correlation Coefficient Comparisons by Method Method ANOVA Mplus Positive bond Leader Member Group Positive work Leader Member Negative relationship Leader Member Group Note. ANOVA analysis of variance. The size of the ICC can also vary with the construct being tested, although most group therapy papers report small but significant values. For example, Tasca et al. (2009) reported positively skewed ICCs that ranged from.00 to.25, although the majority (56%) fell below.05. Baldwin et al. (2005) reported a representative range of ICC values (.00,.05,.15, and.30), and a reanalysis of two group studies found ICCs ranging from.02 to.12 (Baldwin, Stice, & Rohde, 2008). Although there are no agreed upon standards of interpreting ICCs, some have proposed ICCs.05 as indicative of moderate dependence (Stevens, 2002). When data are nested, researchers should calculate and report ICCs (and the method used to calculate it) to provide an index of the degree of dependence in the data. There is disagreement in the field on the magnitude of the ICC that necessitates statistical controls for group dependency. Some argue that nonsignificant ICCs eliminate the need for nested analysis and suggest that the group factor be removed from the statistical model to increase power (Crits-Christoph, Tu, & Gallop, 2003; Hoyle et al., 2001). Others suggest that ICCs less than.05 do not necessitate multilevel modeling and may even make model estimation difficult (Dyer, Hanges, & Hall, 2005). Significance tests of ICCs, however, are often underpowered, and statistical problems created by dependence depend not only on the magnitude of the ICC, but also on the number of individuals per group and number of groups (Baldwin et al., 2011). For groups of four or more, 25 30

6 170 JANIS, BURLINGAME, AND OLSEN groups are generally required in order to test whether an ICC is significantly different from zero (Kenny et al., 2002). Consequently, some have suggested that having nested data, regardless of the significance of the ICCs, warrants a statistical design that accounts for this nesting (Baldwin et al., 2011). We agree with this approach and suggest analyses for nested data regardless of size or significance of ICCs. Multilevel Confirmatory Factor Analysis Multilevel modeling enables one to test the factor structure of a measure while accounting for group dependence, which is particularly useful in group research. In order to test the factor structure, the data should include information about group membership, coding both the unique groups and which members participated in the same group together. This allows the researcher to properly account for the nesting by modeling factor structure both within and between groups. A multilevel factor analysis model has two sources of random variation: between groups (y B B B B B ) and within groups (y W W W W ). Each observed score (y ij ) is a function of an intercept ( ), regression coefficients or factor loadings ( ), the latent factor ( ), and residual errors ( ), (Hox, 2010). Combining the between and within components, the overall equation becomes (y ij B B B W W B W ). The between-groups variation comes from each group s mean score, while the within-group variation is comprised of each group member s deviation from their own group s mean score. Consequently, the analyses assume that the between-group and within-group random components are uncorrelated, and the full covariance structure of the data is represented by the sum of two orthogonal covariance matrices, cov(y ij ) cov(y B ) cov(y W ), or T B W. Analysis The first step in testing a multilevel confirmatory factor analysis is model specification. Model specification translates a theory or hypothesis about how the observed variables and latent variables relate to each other into a structural model; this is frequently depicted visually in the form of a path diagram. First, a number of latent factors (represented by ovals) are hypothesized to underlie the measure s observed items (represented by rectangles). Second, observed items are identified that theoretically load onto each of the latent factors. Items generally only load onto one factor, but it is possible to model items loading onto multiple factors. A pathway between an observed item and a latent factor (represented by a one way arrow) indicates an association between the two, while the lack of a pathway indicates no association. Third, if the specified model includes multiple latent factors, expected covariances among the latent factors can be directly specified using double-headed arrows, or a second order latent factor onto which the first-order latent factors are expected to load might be specified. Figures 1 and 2 present the path diagram for the current model, with parameters and fit statistics, which are discussed below. The hypothesized model can test a theoretically derived factor structure or a factor structure derived from previous principal components analysis or exploratory factor analysis (EFA) on a different dataset. If there is no a priori factor structure guided by theory, it is recommended to perform exploratory analyses on a different sample in order to inform the hypothesized factor structure tested in the CFA. Multilevel EFA estimation can be done directly in Mplus; however, a thorough treatment of this technique is outside the scope of the article and is discussed elsewhere (Huang & Cornell, 2016). In this example, we test a factor structure of the GQ that was derived both theoretically and through previous factor analyses. The proposed model (see Figures 1 and 2) is comprised of three second-order factors representing the GQ quality of relationship subscales (positive bond, positive work, negative relationship), and eight first-order factors representing the group therapy relationship dimensions within each of the GQ subscales (member-group bond, member-leader bond, member-member bond, member-leader work, member-member work, member-group negative relationship, member-leader negative relationship, and member-member negative relationship). In addition, not shown in the figures because of space constraints, the unexplained variance components of the three memberleader latent variables were allowed to covary

7 FACTOR STRUCTURE OF GROUP MEASURES 171 GQ2 GQ4 GQ6 GQ8 GQ18 GQ20 GQ22 GQ1 GQ3 GQ5 GQ Leader Bond Group Bond Member Bond Posi ve Bond Leader Work GQ26 GQ27 GQ28 GQ29 GQ Posi ve Work Member Neg Nega ve Rela onship Member Work GQ9 GQ11 GQ13 GQ15 GQ10 GQ12 GQ14 GQ with one another, and the measurement errors of several individual items were also allowed to covary. This indicates that there is shared variance between those items that is not captured by the latent variables. This same factor structure was tested at both the between- and withingroup levels; however, different factor structures can be tested at each level if guided by theory or prior exploratory analyses. It is not uncommon to have a simpler factor structure at the between level. In specifying the model, it is necessary to have at least two indicators per factor, although three to four is recommended. The following analyses are performed in Mplus; however, other software packages, such as Lisrel, EQS, gsem in Stata, or OpenMX in R, can also be used to perform multilevel confirmatory factor analyses. The current analyses use Mplus s Maximum Likelihood (ML) estimator, which maximizes the likelihood that the observed covariance matrix is drawn from a population in which the model implied covariance matrix is valid (Schermelleh-Engel, Moosbrugger, & Müller, 2003). ML assumes that the population is multivariate normal, that the variables are composed of two uncorrelated random components: between and within, and that each random component is independent and normally distributed (Ryu & West, 2009). ML allows for estimation even in cases of differing group sizes and missing data. Interpreting Results Leader Neg Group Neg GQ23 GQ24 GQ25 Figure 1. Within-level factor structure. This figure illustrates the within-level factor structure and parameters. Model fit for the within-level is as follows: 2 (380) , p.001; standardized root mean square residual.05; root mean square error of approximation.04; comparative fit index.97; Tucker-Lewis index GQ17 GQ19 GQ21 Multilevel CFA has two main goals: to assess the model s goodness of fit and to estimate the parameters of the model (Ryu & West, 2009). There are several different indices of goodness of fit, or how well the theorized model reproduces the covariance structure in the observed data. A good fitting model allows the parameter estimates to be interpreted. These fit indices are provided by the statistical software and include chi-square, Tucker-Lewis index (TLI), compar

8 172 JANIS, BURLINGAME, AND OLSEN GQ2 GQ4 GQ6 GQ8 GQ18 GQ20 GQ22 GQ1 GQ3 GQ5 GQ Leader Bond Group Bond Member Bond Posi ve Bond Leader Work ative fit index (CFI), the root mean square error of approximation (RMSEA), and the standardized root mean residual (SRMR). These fit indices vary slightly in how they assess fit, including whether they apply a penalty for model complexity and how sensitive they are to variations in sample size. Various cut offs have been suggested for each fit index, but Hu and Bentler (1999) offer one commonly used set of criteria, presented below. The chi-square test provides a test of exact model fit. The p value for the chi square test provides the probability that the observed and estimated matrices are the same. The chi-square statistic is substantially influenced by N, so with a large sample size (generally greater than 200), p is likely to be significant (Marsh, Balla, & McDonald, 1988). It is also affected by the size of the correlations, with larger correlations resulting in poorer fit. In evaluating the chi-square statistic, a value less than twice the model s degrees of freedom is often taken to indicate GQ26 GQ27 GQ28 GQ29 GQ Posi ve Work Member Neg Nega ve Rela onship Member Work GQ9 GQ11 GQ13 GQ15 GQ10 GQ12 GQ14 GQ Leader Neg Group Neg GQ23 GQ24 GQ25 Figure 2. Between-level factor structure. This figure illustrates the within-level factor structure and parameters. Model fit for the within-level is as follows: 2 (391) , p.001; standardized root mean square residual.05; root mean square error of approximation.14; comparative fit index.81; Tucker-Lewis index GQ17 GQ19 GQ21 acceptable model fit, although there is no agreed upon standard. Chi-square also provides the basis for the calculation of other indices of fit. Given the sensitivity of chi-squared to sample size, it is often augmented with other goodness-of-fit indices, which use chi-square and the associated degrees of freedom in their formulas. The TLI (also known as the nonnormed fit index, NNFI) and CFI are comparative measures of fit, with higher values indicating better fit. They compare the estimated model with a null model, which assumes no covariance among the variables included in the model. The CFI and TLI are highly correlated, and it is normally unnecessary to report both. Hu and Bentler (1999) suggest a cutoff of.95 for both. The RMSEA and SRMR are both absolute measures of fit, with lower values indicating better fit. The RMSEA provides the amount of error of approximation per model degree of freedom. It favors simpler models in the for

9 FACTOR STRUCTURE OF GROUP MEASURES 173 mula, which corrects for model complexity, and it takes sample size into account. Hu and Bentler (1999) suggested a cutoff of.06, whereas others have suggested.05 as a cutoff for good fit,.08 as a cutoff for acceptable fit, and.1 as an indication of poor fit. The RMSEA can be further evaluated through its confidence interval, which is informative about the precision of the estimate. A lower bound including or close to 0 and an upper bound less than.08 indicates good fit. The SRMR provides the standardized difference between the observed correlation matrix and predicted correlation matrix. It is biased for small Ns and models with small degrees of freedom, and it does not have a penalty for model complexity. Hu and Bentler (1999) suggest.08 as a cut off for good model fit. Given all the different indices for judging fit, it is typical to compute and report several. For our model, the RMSEA (.03) indicated good fit, whereas the CFI (.95), and TLI (.94) indicated borderline acceptable fit. However, 2 (798) 1, fell short of the criteria for acceptable model fit, although this is to be expected with a large sample size. These fit indices denote the fit for the model as a whole, taking into account both the between and within levels. Mplus provides SRMR values for both the between- and within-group levels. For our model, the within-level SRMR (.04) indicated acceptable fit, whereas between-level SRMR (.14) did not, indicating that the borderline fit evidenced in the global fit statistics reported above may be due to lack of fit at the between level. Methods for further assessing fit separately at the between and within levels are reported in the next section. Once establishing overall acceptable fit of the model, the model parameters can be interpreted with greater confidence. Parameters in a multilevel CFA are interpreted in the same way as those in a single level CFA, but they are interpreted at both the between and within levels. Parameters generally interpreted include factor loadings, correlations, and error variances for observed items. Factor loadings of observed items on latent factors represent the degree to which responses on that item are due to the underlying construct being assessed. The factor loadings can be interpreted like regression coefficients. Next, correlations between latent variables indicate the strength of the relationship between those variables. If the latent variables are theoretically closely related, we would expect the correlations to be high, reflecting that relationship. Finally, the error variance for an observed item represents the variance in the item that is not explained by the latent variable on which the item loads. This does not mean that the variance is not meaningful information, only that it is not explained by the latent variable. Most statistical software for estimating factor analytic models has the option of producing both unstandardized and standardized model parameters. In calculating unstandardized parameters, it is common to identify and provide a scale for the latent variables by fixing the loading of one of the indicators of each latent variable to one and freely estimating the other loadings and the variance of the latent variable. The unstandardized estimates retain the original metric of the selected scaling indicator and are most useful if the metric of this variable can be meaningfully interpreted. In producing standardized parameter estimates, the variances of the observed and latent variables are set to one and the factor loadings are estimated accordingly. Once standardized, all the parameters are in a common metric, which allows for comparisons across variables that originally may have been in different metrics. In our model, all items loaded significantly on their intended factors (p.001) at both the within (see Figure 1) and between levels (see Figure 2). The second-order subscales representing positive and negative aspects of the group relationship were correlated as expected at both the within and between levels, with stronger correlations at the between level. Positive bond and positive work were positively correlated with each other at the between, r.68, p.001, and within-groups level, r.64, p.001. Negative relationship was negatively correlated with both positive bond between, r.77, p.001, and within groups, r.60, p.001, and positive work between, r.72, p.001, and within groups, r.30, p.001. The first-order relationship dimensions all loaded significantly on their intended second order subscales. For each of the second-order latent variables, there was a particularly strong factor loading on one of the three first-order factors (member-member relationship).

10 174 JANIS, BURLINGAME, AND OLSEN Common Problems Many problems with multilevel SEM arise from model identification. For a model to be identified, it must have nonnegative degrees of freedom, that is, the number of known values (unique variances, covariances, and means) available to the model must be greater than or equal to the number of parameters being estimated. If the number of estimated parameters is greater than the number of known values, the model is underidentified and the model cannot usually be estimated. If the known values and number of parameters are equal, the model is just-identified, and the model can be estimated but model fit cannot be tested. We note that Mplus gives a (usually inconsequential) warning if the number of estimated parameters exceeds the number of clusters (in this case the number of groups). If the model is specified correctly and should theoretically be identified (i.e., it has positive degrees of freedom), it may still be empirical underidentified. Empirical underidentification occurs when a model should be identified based on its structure but is not due to the nature of the data. This could be caused by multicollinearity in observed variables or very low correlations between latent variables. One anomaly that can occur is a Heywood case, a negative error variance or other out-ofbounds parameter estimate. Heywood cases can arise for a variety of reasons: model misspecification, empirical underidentification, and outliers (Bollen, 1987). Heywood cases can also occur due to sampling fluctuation. If the confidence interval of a negative error term includes zero or a positive number, this is an indication that it may be due to sampling. One solution to Heywood cases is to initially consider them as model specification errors and verify that the model is specified as intended and that each latent variable has at least three indicators. In some cases, it may be possible to alter the model in such a way that Heywood cases do not occur. This is not always a feasible solution, however, as is the case when a specific a priori model is being tested. In that case, Heywood cases resulting from a small negative error variance estimates are sometimes fixed to 0 or constrained to be non-negative. Fixing or constraining parameters in this manner reduces the number of estimated parameters and increases the degrees of freedom for the lack-of-fit chisquare. Some have argued, however, that this may not be a good solution in that Heywood cases likely indicate an underlying problem with model specification (Chen, Bollen, Paxton, Curran, & Kirby, 2001). Our model produced four Heywood cases at the between level, so we ran models constraining the residual variances to zero and to be non-negative. Both additional models provided virtually identical results to the original model. Fit Between and Within The standard approach above has several limitations in evaluating model fit (Ryu & West, 2009; Yuan & Bentler, 2007). Model fit is based on the entire model, but the within level has a greater sample size, so it contributes more to the estimation of model fit. Consequently, the standard approach to evaluating model fit in multilevel structural equation models is generally less sensitive to fit at the between group level, and fit statistics can show acceptable fit for the overall model, even though the model actually has poor fit between groups. Further, poor overall fit does not indicate whether this is due to fit between, within, or both. To address this limitation, methods have been developed for evaluating fit at both the within and between levels (Ryu & West, 2009; Yuan & Bentler, 2007). In this article, we outline a procedure for assessing model fit both between and within by running partially saturated models models that are saturated at either the between or within level (Hox, 2002; Ryu & West, 2009). A saturated model provides fit for the observed data by perfectly modeling all of the variance of the variables and covariance between them. It is the best fitting model possible for the observed data and has zero degrees of freedom. The method of partially saturating models was originally introduced by Hox (2002). Because one level of the model is saturated and provides perfect fit, any lack of fit comes from the level not being saturated. In other words, when one level is saturated, the chi-square statistic can be taken as the fit for the level not being saturated. These level specific chi-square statistics are then used to calculate level specific TLI, CFI, and RMSEA statistics. For another method of evaluating level specific fit, see Yuan and Bentler (2007). Theory and past empirical findings regarding the measure or construct under study should

11 FACTOR STRUCTURE OF GROUP MEASURES 175 guide considerations regarding model fit at the between and within levels. The GQ assesses an individualistic quality of a group member s experience their perception of the therapeutic relationship and was designed to assist in identifying individual members who might be at risk for treatment failure. Thus, within-level model fit is of primary interest. On the other hand, if a measure were used to make decisions about group composition a characteristic of the group, not the individual between-level fit would be of particular importance. This relates to the consideration of clarity regarding inferences to be made by level. Analysis This example walks through obtaining model fit for the between level; however, the same procedure can be used for obtaining fit for the within level, and chi-square statistics are presented for both (see Table 2). The first step is to test a model with the hypothesized factor structure at both levels (Model 1), as was done above. The chi-square differs slightly from the chi-square reported above, as Heywood cases were not constrained for the purposes of testing level specific fit. The next step is to test a model in which the between level is as originally hypothesized, and the within level is saturated (Model 2). In Mplus, this is done by specifying covariances between all of the observed variables. The final step is to test an independence model (Model 3). An independence model is one in which all of the covariances are constrained to zero. For this method, the appropriate independence model indicates one in which the within level is saturated and the between level is an independence model. Table 2 Chi-Squares and Degrees of Freedom for Level-Specific Fit Calculations Model Between Within 2 df Model 1 Hypothesized Hypothesized Model 2 Hypothesized Saturated Model 3 Independent Saturated Model 4 Saturated Hypothesized Model 5 Saturated Independent Note. These models are based on 1,058 individuals in 195 groups. Once the partially saturated and independence models have been estimated, the chisquares and degrees of freedom from those models are used to calculate the fit statistics. The TLI between (Hox, 2010) is calculated by TLI 2 df indb,satw 2 hypb,satw df hypb,satw 2 df indb,satw 1 The elements in this equation reflect results from Models 2 and 3 in Table 1. For example, 2 indb,satw is the estimated chi-square value for a model combining an independence model at the between groups level and a saturated model at 2 the within groups level, whereas hypb,satw is the chi-square for a model combining the originally hypothesized model at the between groups level with a saturated model at the within groups level. The CFI between (Bentler, 1990; Ryu & West, 2009) is calculated by CFI 1 2 hypb,satw 2 indb,satw df hypb,satw df indb,satw In calculating the CFI, for both the numerator and denominator of the fractional part of the formula, take the maximum of the number or zero. That is, if either the numerator or denominator is negative, set it to zero. RMSEA between is calculated by 2 hypb,satw df hypb,satw RMSEA df hypb,satw *(n 1) where n is the sample size of the level being tested. The effective between sample size is equal to the number of groups (n J), whereas the effective within sample size is equal to the total number of individuals minus the number of groups (n N J, Ryu, 2014, p. 2). If the calculated RMSEA is negative, it is set to zero. Models 4 and 5 present the test statistics and degrees of freedom for the saturated and independence models used to calculate fit within. We provide syntax for level specific analyses,

12 176 JANIS, BURLINGAME, AND OLSEN sample data structure and a spreadsheet for calculating level specific fit statistics in the supplemental materials. Using the above equations and the values in Table 2, we can calculate the TLI, CFI, and RMSEA values separately for the between and within levels (see Figures 1 and 2). Levelspecific fit statistics are interpreted and reported in the same manner as overall fit statistics. For our model, the within-groups, 2 (380) , p.001, and between-groups, 2 (391) , p.001, chi-square values were both statistically significant. The adjusted withingroups (.04) and between-groups (.04) RMSEA values both indicated good fit, but the withingroups CFI (.97) and TLI (.97) showed much better fit than the between-groups CFI (.81) and TLI (.79). This indicates that the proposed model explains the factor structure for individual members of groups quite well but does not explain the factor structure for how groups differ from each other nearly as well. Taken together with the Heywood cases found at the between level, the poor between fit indicates a likely need to respecify the between groups model if inferences are to be made at this level. This would be accomplished by a further EFA to examine alternate factor structures at the between-level and respecifying the multilevel CFA to include a new between-level factor structure, which would then differ from the established within-level structure. It is likely that the variables are operating differently at each level, forming somewhat different underlying latent structures. For example, a school climate measure was found to have two latent constructs operating at the within level Personal Conviction and Concern for Others and one latent construct at the between level Positive Values (Huang & Cornell, 2016). Tay, Woo, and Vermunt (2014) offer several explanations for why a measure s factor structure may vary by level as well as what it means when this occurs. If the model had not been evaluated at both the between and within level, it would not have been apparent that the proposed model does not fit as well at the between level. This underscores the importance of evaluating fit at both levels and not taking overall acceptable model fit as an indication of acceptable model fit at both levels, especially between groups. Discussion Analyzing and interpreting group data without accounting for nesting has serious consequences. In two separate analyses, Burlingame, Kircher, and Honts (1994) and Baldwin and colleagues (2005) found Type I error rates of up to.58 and.59, respectively when grouping was not properly accounted for. Despite the clear consequences for ignoring grouping, Baldwin et al. (2005) also found that none of the studies reviewed properly accounted for nesting in the analyses. This problem has been raised in the literature for decades and is still routinely neglected (Burlingame, Kircher, & Honts, 1994; Burlingame et al., 2013; Hoyle, Georgesen, & Webster, 2001; Kenny & Judd, 1986; Kenny & la Voie, 1985; Wampold & Serlin, 2000). We encourage researchers to consider the seriousness of publishing studies that do not account for group dependency and to exercise caution in interpreting results in previously published works that did not account for nesting. In this article, we have presented methods for accounting for nesting that have existed for several decades but have not been well used. The goal of the article is to make these solutions more accessible and to encourage their use in group research. This article described and compared methods for calculating ICCs as an index of group dependency. It further offered multilevel confirmatory factor analysis as a technique for evaluating factor structures of measures used in grouped data. Finally, it presented a method for evaluating level specific fit in a multilevel confirmatory factor analysis through partially saturated models. The two methods for calculating ICCs (through the ANOVA table and Mplus) were largely consistent in their estimates of group dependency present in the GQ subscales, with the Mplus ICCs being slightly higher. The ICCs for the subscales and relationship dimensions (see Table 1) varied considerably, however, indicating that within a given measure, different constructs may be more or less sensitive to group dependency. We found the highest ICCs for the member-group relationship dimensions of each subscale. This is unsurprising, given that Kivlighan Jr. and colleagues (2015) found stronger ICCs for items worded with the group as the referent than for items referring to the

13 FACTOR STRUCTURE OF GROUP MEASURES 177 individual member. Further, the ICC values found in this study differed somewhat from those previously published for the GQ, showing that ICC values for the same constructs, and even the same measure, can vary between populations and settings. We encourage the creation of a repository of ICCs for measures used in group research. Baldwin and colleagues (2011) have advocated for an ICC database of psychotherapy outcomes from clients nested within therapists, and in conjunction with this Baldwin et al. (2008) report ICCs for six measures and note that researchers should publish their ICCs or make data available so that they can be computed. Population estimates of ICCs can then be established by combining ICCs from multiple studies using a common measure. This allows researchers to obtain and use previous ICC estimates for specific measures when designing new studies to ensure adequate power to detect an effect. This is especially important when comparing two treatments delivered in a group format, as power is often low in group studies (Baldwin et al., 2008). We echo Baldwin and colleagues call for a collection of ICCs in the group psychotherapy domain. In looking at the factor structure of the GQ, the hypothesized factor structure provides acceptable fit when looking at overall model fit. When the between and within levels are evaluated separately, we find differences, specifically poorer fit at the between level. Without looking at the level specific fit, this discrepancy was not apparent in past published research, highlighting the importance of evaluating fit at each level. We offer this as an admonition to the field to analyze level specific fit, as only looking at overall fit can mask poor fit at a specific level, especially between. The between and within levels each provide unique information about the constructs being measured. The between level provides information about how groups differ from each other, whereas the within level provides information on how individuals differ from each other after removing variance due to group membership. Consequently, if the main inference of interest is in how groups differ, the focus would primarily be on accounting for variation at the between level and fitting a model that provides good fit for the data at that level. If the main inference of interest lies with individuals within groups, the focus would be on accounting for the structure at that level. Often, researchers are interested in theoretical differences in a construct at a group level, but for an applied measure, individual differences may be more important. Measures are a foundational element in our knowledge of group treatment process and outcomes. Nested data need to be analyzed at the level of inference desired to insure that there is an adequate psychometric foundation for conclusions. Future Directions It is important to note that group psychotherapy data is often collected over time, resulting in a longitudinal dataset with at least three levels of nesting: between-groups, within-group, and within-individual. Similarly, clients are often nested within groups, which are further nested within therapists leading multiple psychotherapy groups. Although the current dataset does not lend itself to a three-level analysis, Mplus is capable of estimating three-level CFA models. In addition, the method presented here of partially saturating each level could be extended to three level models in order to determine fit at each level, although to our knowledge it has not been done in the literature. Future research should explore this technique with three-level data. Future group psychotherapy research should focus on psychometric exploration of key measures, particularly those known to be affected by group dependence, to determine if factor invariance exists across different levels of nested data. If factor invariance is not found, as evidenced herein, and inferences at both the between and within group are desired, different factor solutions may be necessary. Examples of these methods are just emerging in clustered data from school systems that can guide the interested group researcher (Huang & Cornell, 2016; Konold et al., 2014). Annotated Bibliography Hox, J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge. This book provides an introduction to multilevel regression modeling and multilevel structural equation modeling. The authors present these analytic techniques in a manner that is