MULTITRAIT-MULTIMETHOD ANALYSIS

Transcription

1 I III I I I I I III I II II I I I IIII I III II 2 MULTITRAIT-MULTIMETHOD ANALYSIS LEVENT DUMENCI Department of Psychiatry, University of Arkansas, Little Rock, Arkansas The multitrait-multimethod (MTMM; Campbell & Fiske, 1959) analysis procedure is a powerful measurement design in construct validation research. It offers an uncompromising perspective on the meaning of theoretical variables (e.g., intelligence and depression) that require indirect measurement. Originating in psychometrics, MTMM models now are applied over a broad range of disciplines including business (Conway, 1996; Kumar & Dillon, 1992), medical and physical education (Forsythe, McGaghie, & Friedman, 1986; Marsh, 1996), speech-language (Bachman & Palmer, 1981), political science (Sullivan & Feldman, 1979), nursing (Sidani & Jones, 1995), and medicine (Engstrom, Persson, Larsson, & Sullivan, 1998). The MTMM is a fully crossed measurement model whereby multiple procedures (methods) are used to measure multiple theoretical constructs (traits) in a large number of persons. In medical schools, for example, the standardized patient examinations are used to measure students' abilities to take patients' medical history (T1), to perform a physical examination (T2), to communicate effectively with the patient (T3), and to provide relevant feedback to the patient (T4). These four types of abilities are referred to as traits in the MTMM analysis. All four traits are measured Handbook of Applied Multivariate Statistics and Mathematical Modeling Copyright 9 2 by Academic Press. All rights of reproduction in any form reserved. 583

2 S84 LEVENT DUMENCl by three types of raters: trained persons playing the role of a patient and faculty in the examination room (M1 and M2, respectively), and trained raters observing students' performances live on a video monitor (M3). In this example, four traits measured by three types of performance ratings form an MTMM matrix, which is nothing more than a correlation matrix between 12 measures (4 traits x 3 methods). A measure (TiMj; i = 1,2,3,4 and j = 1,2,3) refers to an observed variable associated with each traitmethod pairing, (e.g., medical history-taking ability rated by the faculty) (i.e., T~M2). Other examples include the measurements of four competence traits (i.e., social, academic, English, and mathematics) measured by four rating methods (i.e., self, teacher, parent, and peer) (Byrne & Goffin, 1993) and the measurements of three types of attitudes (i.e., affective, behavioral, and cognitive) toward church assessed with four different methods of scale construction procedures (i.e., scalogram, equal-appearing intervals, selfrating, and summated ratings) (Ostrom, 1969). The objectives of MTMM analysis are (a) to determine whether the measurements of each trait derived by multiple methods are concordant (convergent validity); (b) to show that measurements of different traits obtained using the same method are discordant (discriminant validity); and (c) to estimate the influence of different methods on the measurement of traits (method effect). Traits are universal, and their existence should not depend on the choice of a method to measure them. Therefore, convergent validity requires empirical evidence that different methods can be used to measure a trait. Traits also are unobservable individual difference characteristics. Different labels are used to distinguish one trait from the other to facilitate verbal communications. Consequently, this gives rise to the possibility that different labels may refer to the same trait. The discriminant validity evidence is sought to demonstrate that different labels (e.g., depression and anxiety) indeed refer to two distinct traits. Finally, the measurement of traits unavoidably involves one or more methods. For example, an IQ score of 15 obtained using a performance measure may be an inflated or deflated measure of intelligence because of general or incidentspecific biases in the method of measuring intelligence. An IQ score different than 15 might have been obtained if teacher's ratings were used to measure intelligence. Thus, construct validity requires that the scores reflect the magnitude of the trait and be relatively free of method of measurement. All MTMM analysis procedures provide evidence pertaining to three types of construct validity: convergent validity, discriminant validity, and method effect. However, there are no universally accepted formal tests for construct validity. The analytic definitions of different validity types vary across different MTMM analytic procedures. Therefore, construct validity evidence should be interpreted under the framework of a particular MTMM procedure used in the analysis (Kumar & Dillon, 1992). The first analytic approach for the MTMM matrix, originally proposed

3 2. MULTiT~,T-MuLT, METHoD ANALYSIS $85 by Campbell and Fiske (1959), relied on the magnitudes, patterns, and averages of zero-order correlations. According to this approach, correlations among the measurements of the same trait by different methods (i.e., monotrait-heteromethod [MTHM] correlations or validity diagonals) should be statistically significant and large in magnitude. The validity diagonals should exceed the correlations among measures of different traits obtained from different methods (i.e., heterotrait-heteromethod) (HTHM) and the correlations between different traits measured by the same method (i.e., heterotrait-monomethod) (HTMM). Returning back to the example of standardized patient examination, a zero-correlation between the historytaking ability measured by the ratings of the faculty and the patient would imply that these two ratings are not measures of the same construct (i.e., history-taking ability). Also, the findings that the correlation between the physical examination and communication measured by the faculty ratings is of the same magnitude as the correlation between the faculty and the observer ratings of physical examination would suggest a substantial method bias in ratings. The method bias also is evident when the patterns of heterotrait correlations within monomethod and within hcteromethod blocks are different. The Campbell and Fiske (1959) procedure assumes that the operation of averaging correlations does not lead to any loss of information in the MTMM matrix (i.e., that all the correlation coefficients in an MTMM matrix were generated by only four correlations). This is a very restrictive assumption and is usually violated in practice. Applied researchers should verify (at least visually) that the range of correlations being averaged is very narrow before reaching any conclusions on construct validity. The Campbell and Fiske procedure of averaging correlations still is the most commonly used MTMM analytic procedure, but it lacks any formal statistical model, and the subjectivity involved in interpreting simple correlations has led researchers to develop new statistical models for analyzing MTMM matrices. Earlier approaches include exploratory factor analysis (Jackson, 1969), nonparametric ANOVA models (Hubert & Baker, 1978), partial correlation methods (Schriesheim, 1981), and smallest space analysis (Levin, Montag, & Comrey, 1983). With the exception of exploratory factor analysis, a distinguishing characteristic of these descriptive MTMM techniques is that they are relatively free from the problems of convergence and improper solutions. Unfortunately, the measurement structure underlying the MTMM matrix is either implicit or absent in these MTMM techniques. That means that the measurement model implied by these techniques cannot be subjected to the test of model disconfirmability, which is an essential element in scientific endeavor, prior to making any construct validity claims based on the parameter estimates. In sum, these techniques assume that the measurement model implied by the model is correct without offering any way of verifying it. The equal-level approach (Schweizer, 1991)

4 586 LEVENT DUMENCI and the constrained component analysis (Kiers, Takane, & ten Berge, 1995), relatively new statistical methods of MTMM analysis, also suffer from this limitation. The random-effect analysis of variance (ANOVA), confirmatory factor analysis (CFA), covariance component analysis (CCA), and composite direct product analysis (CDP), are statistical models commonly used for analyzing MTMM matrices; these are presented here under the framework of structural equation modeling (SEM). A good grasp of the SEM and CFA fundamentals (see DiLalla, chapter 15, and Hoyle, chapter 16, this volume) are required to implement these procedures with real data. Artificial MTMM matrices with 12 variables each (four traits measured by three methods) are used to illustrate the models. A random sample size of 1, is assumed throughout. Graphical representations of the models and parameter estimates also are provided so that readers can gain firsthand experience analyzing different MTMM models with a SEM software package (e.g., AMOS, EQS, LISREL, Mplus, or Mx) and verify their results. Indeed, readers are encouraged to replicate the examples presented here with a software of their own choice before analyzing a real MTMM matrix. All parameter estimates are significant unless otherwise indicated in the text. Different MTMM matrices are analyzed for different models because it is very unlikely, if not impossible, that all four models fit an MTMM matrix equally well in practice. A cross-sectional measurement design is assumed throughout this chapter. Some MTMM applications nonetheless adapt a longitudinal measurement design where occasions are treated as methods. An MTMM analysis may not be optimal for longitudinal designs. The issues of measurement of stability and change, measurement invariance, trait-state distinction, and interpretations of convergent validity, discriminant validity, and method effects should be addressed in longitudinal designs (Dumenci & Windle, 1996, 1998). A number of statistical models now are available to address such issues and should be considered before an MTMM analysis for such measurement designs is adopted. Readers should consult chapters on time-series analysis (see Mark, Reichardt, & Sanna, chapter 13, this volume) and modeling change over time (see WiUett & Keiley, chapter 23, in this volume). I. RANDOM ANALYSIS OF VARIANCE MODEL A full three-way random ANOVA model specification would allow for estimating the variance attributable to person, trait, and method main effects, three two-way interactions, and one three-way interaction (Guilford, 1954; Stanley, 1961). Four sources of variance are of special interest in analyzing MTMM data: (a) Person, (b) Trait Person, (c) Method Person, and (d) Trait Method Person (Kavanagh, MacKinney, &

5 2. MULTITRAIT-MULTINETHOD ANALYSIS S87 Wolins, 1971). The Person Trait Method interaction serves as the error term for testing the significance of the remaining three sources of variance. The ANOVA approach has early roots in generalizability theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; see Marcoulides, chapter 18, this volume) and parallels to the averaging correlations employed by Campbell and Fiske (1959). Following Kenny (1995), SEM representation of the random ANOVA model is depicted in Figure 2.1. Four unique parameters account for the entire MTMM matrix: trait variance (T), trait covariance (two directional arrows in Figure 2.1), method variance (M), and unique variance (e). Note that no subscript is used to distinguish different traits, different trait covariances, different methods, and different errors of measurement in Figure 2.1. For example, the correlation between historytaking ability and communication is the same as the correlations between history taking and the remaining two abilities (i.e., communication and feedback). Also, the error of measurement associated with communication measured by faculty rating is the same as the error of measurement associated with feedback measured by the rater at the monitor. This assumption is known as compound symmetry and applies to all four parameters. The ANOVA model also assumes that the correlation between a measure and a trait factor (all one-directional arrows from traits to measures in Figure 2.1) is equal to unity. A large Person x Trait effect would signify that the differences among individuals vary in magnitude as a function of the trait being measured (i.e., the rank ordering of participants is not the same across traits). That finding, along with low trait correlations, would provide evidence for discriminant validity. Convergent validity is established when the Person x Method effect is not significant (i.e., people are ranked the same way on a given trait, regardless of the method used to measure it). If the Person x Method effect is significant, it should be smaller than the Person x Trait effect, if convergent validity is to be demonstrated. The model depicted in Figure 2.1 implies that the traits are correlated (as represented by lines linking the T entries) but that the methods are uncorrelated (i.e., there are no lines linking the M entries). However, two additional SEM possibilities should be considered when the random ANOVA model is interpreted. First, it is possible that the traits are uncorrelated (i.e., the lines linking the T entries would be deleted from Figure 2.1) but that the methods are correlated (i.e., lines linking the M entries would be added). All other features of the model (i.e., the unit factor pattern coefficients, the number of free parameters, and the assumption of compound symmetry) would remain unchanged. Alternatively, three orthogonal (i.e., uncorrelated) common factors (i.e., a general, trait, and method factors) could underlie the MTMM matrix (see Figure 2.2). Any statistical program with the SEM capabilities can be used to analyze the MTMM matrix. I analyzed the artificial MTMM matrix in

6 588 LEVENT DUMENCI Methods Observed Scores (Measures) Traits Trait Correlations FIGURE 2. I Random analysis of variance model: Correlated traits.

7 2. MULTITP, AIT-MULTIMETHOD ANALYSIS $89 General Observed and Scores Methods (Measures) Traits FIGURE 2.2 Random analysis of variance model: General, trait, and method factors. Table 2.1 using EQS (Bentler, 1989) to illustrate all three ANOVA models. The unweighted least-square method was used to estimate the model parameters. All three ANOVA models fit the MTMM matrix equally well (~74) = 83.74; p >.1). The nonsignificant chi-square statistic provides evidence that each of the three ANOVA models is a plausible representation of the MTMM matrix, hence, the model assumptions (e.g., compound

8 ~ o~ m E X W o Z ~Z i" ~ o= ~ ~/~~~ / E o l L X e~ l,c 9 "~ 8 Ir--.,i. ~/<.,>,-- I q ~5/~ 1"41 9 I ~. ~ ~. L o U ~= I-- e~,=== I",l Ill,=i m I = Ir ~.)s IYP' EZ~~

9 2. MULTITRAIT-MULTIMETHOD ANALYSIS $9 i symmetry) are supported. Parameter estimates and average MTMM correlations appear in the upper half of Table 2.2. It is not coincidental that all three models have exactly the same fit index and that the parameter estimates from three ANOVA models are closely related. All three models are statistically equivalent, hence one cannot choose one or the other, because each model can be reexpressed by the parameters of the remaining two models, as well as by the average correlations. For example, the estimated method variance in the correlated-method ANOVA model (.4) is nothing more than the average correlation (see Table 2.1) between different traits measured by the same method (i.e., mean [cor(htmm)]). Note that the parameter estimate of.4 from the correlated-method ANOVA model can be reexpressed as the sum of the method variance and the trait correlation ( ) in the correlated-trait ANOVA model, and as the sum of the method variance and the general trait variance (G) in the general factor ANOVA model. As shown at the lower half of Table 2.2, all three ANOVA models are statistically equivalent, and the same set of parameters can be estimated by simply averaging the correlations (i.e., the Campbell and Fiske approach) in the MTMM matrix. The random ANOVA model is the most parsimonious approach to modeling the MTMM matrix because it requires the estimations of only four parameters. A meaningful interpretation of ANOVA parameters is difficult to reach, however, for when the assumption of compound symmetry is satisfied, the ANOVA model implies three conceptually different but statistically equivalent models, as demonstrated in Table 2.2. First, the HTHM triangle reflects the correlations between traits, as implied by Figure 2.1 (the correlated-trait ANOVA model); second, it reflects the correlations between methods in the correlated-method ANOVA model; and third, it reflects a common (general) latent variate uncorrelated with trait and method factors, as implied by Figure 2.2 (the general factor ANOVA model). The Campbell and Fiske procedure treats the HTHM correlations as "error" in that they establish a baseline against which to interpret the other correlations. Campbell and Fiske (1959) recommended that maximally distinct methods be chosen in the MTMM analysis so that any interpretation based on the correlated-method model is implausible, but even if that were done, there is no statistical justification for adopting one interpretation or the other because all three models are statistically equivalent. The only benefit of adopting the random ANOVA model is that it provides a statistical procedure for testing whether the MTMM matrix satisfies the assumption of compound symmetry, which underlies the Campbell and Fiske criteria. When a random ANOVA model does not fit, the interpretations of model parameters would be questionable, as in any other SEM model. For a well-fitting model, on the other hand, the ANOVA method does not provide any substantive information above and beyond

10 o o~ L o._~ H H H 888 E E ~ E "88~8 ggg g N E ~ I,,.1=.4.t s o 9. ) ) ) ~ I ~ I..4 (i.),.,1= o,,-i e:l s.,1= 1D I = 8.,1= E3 I,I m ul m o s.. I I o,.,~ I,,,,i s (1,) tl s tu 9 9 (~. EEriE 9 ~ p. p, s,,=i n_ ul _J o,.~ ~. ~"~ 8"~ ~ = i-i

11 2. MULTITRAIT-MULTIMETHOD ANALYSIS 593 what already is known from the averaging of correlations, except that the averaging operation is justified. That is, applied researchers do not need to have access to computers to obtain the unweighted least square parameter estimates for ANOVA models. This can be achieved by simply averaging the respective correlations in the MTMM matrix. In sum, interpretive difficulties associated with the problem of model equivalency seem the major obstacle to adopting the random ANOVA model in MTMM analysis. Therefore, despite its widespread use over the last three decades, the model is flawed and applied researchers should avoid its use. II. CONFIRMATORY FACTOR ANALYTIC MODEL For a given set of congeneric measures (Joreskog, 1971), the CFA model can be used to partition the variance of each measure in an MTMM matrix into a common trait and unique components. The unique components can be further decomposed into measure-specific (common method) and random error components. The MTMM can be conceptualized as a replication design in which the measurement of traits are replicated using multiple methods. An important feature of replication designs is that they permit the evaluation of the independent contribution of the measurespecific component to the unique variance. Therefore, the CFA can be used to evaluate trait and method components of measures (Werts & Linn, 197). The CFA has been one of the most widely used methods for analyzing MTMM matrices. Its attractiveness is due to the premise that, first, it allows for separating trait, method, and unique components from the measures. Second, it adopts hypothesis-testing strategies to evaluate convergent and discriminant validity and method effects. Third, it estimates within-trait and within-method correlations after the unreliability of measures is taken into account. The CFA model decomposes the MTMM matrix into three additive components: (a) t common trait factors, each explaining the correlations across different methods; (b) m common method factors, each explaining the correlations across different traits; and (c) tm unique variances (i.e., the variances in measures unexplained by the common factors). From the perspective of construct validity, the most desirable CFA model is the traitonly model, which includes t-uncorrelated trait factors and no method factor structure. A variety of CFA models can be specified with different trait, method, and factor correlation structures, ranging from the most parsimonious (trait-only) to the most complex (correlated-trait/correlated-method) model (see Figure 2.3). According to this model, trait factors are uncorrelated with method factors, trait and method factors are uncorrelated with unique components, and unique components are uncorrelated. Note that

12 S94 LEVENT DUMENCl Observed Method Scores Correlations Methods (Measures) Traits Trait Correlations FIGURE 2.3 methods. Confirmatory factor analysis model: Correlated traits and correlated this model differs from the random ANOVA model in three respects. First, it allows for correlations within traits and within methods. The ANOVA model can have only one set of such correlations (i.e., either traits or methods are correlated). Second, it estimates correlations between factors and measures (i.e., factor loadings). All factor loadings are set to unity in

13 i i i 2. MULTITRAIT-MULTIMETHOD ANALYSIS the ANOVA models. Third, it does not assume compound symmetry, hence, it imposes no equality restrictions on variances and correlations in the model. For this reason, latent variates are identified with trait or method number (e.g., T1 and M3, in Figure 2.3). The correlated-trait-correlated-method (CTCM) model is illustrated with the hypothetical standardized patient examination using an artificial MTMM matrix presented in Table 2.3. The CTCM model in Figure 2.3 fits the matrix quite well (~33) = 5.63; p >.1), providing evidence that Figure 2.3 is a reasonable measurement structure for the MTMM matrix in Table 2.3. Parameter estimates from the maximum likelihood solution appear in Table 2.4. All parameter estimates are significant, except for the method loading of T2M2 (.11) (i.e., the faculty rating of the physical examination is relatively free from method bias) and the method factor correlation between M2 and M3 (-.17) (i.e., the faculty and the observer ratings). Large and significant trait loadings in Table 2.4 indicate convergent validity. Providing further evidence for convergent validity is the fact that the overall factor patterns indicate larger trait loadings than the method loadings in Table 2.4. For example, the measure of history-taking ability by the observer (T4M3) accounts for approximately twice as much variance as the rater type (.86 versus.47). Visual inspection reveals that the trait factor correlations range from.37 to.9 (see Table 2.4) indicating that discriminant validity is mediocre. The discriminant validity diminishes to TABLE 2.3 Hypothetical Multitrait-Multimethod Correlation Matrix for the Confirmatory Factor Analysis Example Method 1 Method 2 Method 3 Measure TI T~ T3 T4 Tl T2 T3 T4 TI T2 T3 T4 Method 1 T1 1. Tz T T4 Method T T T T4 Method T T T T i i i i,i i ,,

14 596 LEVENT DUMENCI TABLE 2.4 Parameter Estimates from the Confirmatory Factor Analysis Model Trait factors Method factors Measure T1 Tz 1"3 T4 Ml Mz Ma Unique variance Method 1 T T T T Method 2 T1.79 T2.88 T3.65 T4.89 Method 3 T1.8 T2.8 T3.66 T4.86 o Factor correlations Traits T1 1 T T T Methods Mt M2 M the extent that the trait correlations approach to unity. Evidence for convergent and discriminant validity should be interpreted in the light of measure reliability estimates (i.e., one minus the unique variance in Table 2.4). For instance, the measurement of communication ability rated by trained individuals playing the role of a patient (i.e., T3M1) has the lowest reliability of.36 (1 -.64), which can also be obtained by the sum of squares of trait and method factor loadings: that is, (.58) 2 + (.12) 2 =.36. In other words, the unit variance of T3M1 is partitioned into trait, method, and unique components (i.e.,.34,.2, and.64, respectively). Finally, the magnitude of method factor loadings point to the presence of method effect. When specified a priori, nested model comparison strategies can also be used to test hypotheses regarding construct validity (Widaman, 1985). Hence, it is possible to express convergent validity numerically by reestimating the model fit without trait factors and comparing the fit statistics with

15 2. HULTITRAIT-HULTIMETHOD ANALYSIS those of the CTCM model. A substantive difference between two statistics indicates convergent validity. From the nested comparison perspective, discriminant validity can be evaluated by comparing the fit of the CTCM model with four trait factors to the fit with one trait factor. The statistical testing of method effect involves a comparison of model fit between the CTCM model with and without method factors. The method effect is absent if both models fit equally well. The conceptual elegance of the CFA-MTMM model has been challenged in a number of studies. Indeed, separation of trait and method effects is nothing but wishful thinking unless all measures are either tauequivalent or parallel (Kumar & Dillon, 1992). The CFA model does not make the assumptions of compound symmetry and unit factor loading as discussed for the ANOVA model, so it requires a great number of parameter estimates relative to the number of nonredundant correlation coefficients in the MTMM matrix. This lack of parsimony opens the door to a variety of problems. The empirical and statistical identification is a common problem, which implies the lack of a unique parameter set that can be estimated from the CFA model (see DiLalla, chapter 15, this volume, for model identification). Another common problem is the lack of convergence. That is, the minimizing function (usually maximum likelihood) fails to reach its minimum so no CFA solution can be obtained from the model. When convergence is reached, however, the solution often contains out-of-range parameter estimates (also referred to as Heywood cases), such as negative variance estimates and correlations greater than unity. Finally, difficulties exist in finding a well-fitting CFA model even if a unique solution is obtained. Despite its shortcomings, there have been successful applications of the CFA model (e.g., Bollen, 1989; Dumenci, 1995). III. COVARIANCE COMPONENT ANALYSIS Covariance component analysis is a multivariate extension of the random ANOVA design for completely crossed measurements (Bork & Bargmann, 1966) that offers a viable alternative to conventional models for analyzing MTMM matrices (Wothke, 1984,1987). The conceptual foundation of CCA follows closely from the generalizability theory of Cronbach et al. (1972). The model assumes that three sets of latent variables account for correlations in the MTMM matrix: (a) a general component factor, which accounts for correlations among all tm measures; (b) t -:- 1 trait contrast factors, which account for differences in traits; and (c) m - 1 method contrasts factors, which account for differences in methods. Interpretation of variance component estimates is facilitated by integrating both scaling differences and unreliability of measures into the model. Despite the fact that all MTMM models address the construct validity

16 i 598 LEVENT DUMENCl issues (i.e., convergent validity, discriminant validity, and relative influence of method variance), and that the meaning of the general factor in CCA framework is somewhat analogous to that in factor analysis, interpretations of trait and method factors differ in these two modeling approaches. The trait contrast factors in CCA capture differences between all possible (t - 1) sets of traits (e.g., set 1: history taking versus set 2: physical exam, communication, and feedback), whereas each trait factor in the CFA captures variability among different ratings of the same trait measures. Similarly, method contrast factors capture differences between two sets of methods in the CCA, method factors capture variability among different traits rated by the same rater type as in the CFA. Additionally, factor pattern coefficients specify the contrasts in the CCA, whereas they are the estimates of the strength of relations between measures and common factors in the CFA. Furthermore, differences in scaling among measures and the characteristics shared by all measures are taken into account in the CCA, but they are not a part of the CFA model. CCA is less susceptible to identification and convergence problems and improper solutions than the CFA models for the MTMM data. The artificial MTMM matrix in Table 2.5 is used to illustrate a scalefree CCA model with unknown scaling factor and diagonal component factor correlations. Alternative model specifications also are possible. For example, the CCA without the scaling factor provides a parsimonious model when all measures share the common metric. Alternative factor TABLE 2.5 Hypothetical Multitralt-Multimethod Correlation Matrix for the Covariance Component Analysis Example Method 1 Method 2 Method 3 Measure TI 2'2 T3 T4 Tl "1['2 T3 T4 Tl T2 T3 T4 Method 1 T1 T2 T3 T4 Method 2 T1 T2 T3 T4 Method 3 T1 T2 T3 T ii

17 2. HULTITRAIT-MULTIHETHOD ANALYSIS ~99 correlation structures also may be tested. For the block-diagonal CCA model, within-trait and within-method correlations are estimated. All variance components (i.e., general, trait, and method) are correlated in the full CCA model. A diagonal CCA model is depicted in Figure 2.4. There are twelve scaling factors corresponding to twelve measures. The scaling coefficients (the coefficients relating the scaling factors to the measures) are free parameters of the model. Disattenuated correlations among the scaling factors are accounted for by three sets of second-order factors: (a) one general factor (F_G); (b) three (t - 1) trait contrast factors (traits 1_123, traits 2_34, and traits 3_4); and (c) two (m - 1) method contrast factors (methods 1_23 and methods 2_3). The contrast matrix contains coefficients that signify the unidirectional relations between the scaling factors and the contrast factors. The contrast matrix entries are predetermined (i.e., fixed parameters), such that the sums of squares for each column are equal to 1, and the sum of product terms is equal to for all possible column pairs. Such matrices are labeled as columnwise orthonormal (the contrast matrix for four traits measured by three variables appears in Table 2.6; see also Wothke, 1996, for three traits measured by three methods). The model in Figure 2.4 fits the MTMM matrix well ()d(n9) = 57.59; p >.1); hence, the CCA model is a plausible underlying structure of the MTMM matrix in Table 2.5. Maximum likelihood parameter estimates appear in Table 2.6. The scaling variable facilitates interpretation of variance components, especially when a correlation matrix, instead of a covariance matrix, is used in the analysis. A visual inspection of Table 2.6 reveals a relatively low variability in scaling parameter estimates (i.e., ranging from.86 to.1.17), which indicates that measures have a similar metric. The reliability of measures (one minus the unique variance) ranges from.51 to.85. The variance of the general factor is fixed to unity, for identification; the variances of the trait and method contrast factors are free parameters. The size of both trait and method variance components are evaluated relative to unit variance of the general variate (F_G in Figure 2.4). The variance component estimates appear at the lower half of Table 2.6. Variance components for the three trait and two method contrast factors are statistically significant in this hypothetical illustration. For example, the variance component estimate for the trait contrast factor 1 versus 2, 3, and 4 (traits in Figure 2.4) is 1.44 and significant; hence, there is empirical evidence that history-taking ability (T1) is discriminated from remaining abilities (i.e., physical exam [T2], communications [T3], and feedback [T4]). The estimate of 1.6 for the method contrast factor 2 versus 3 (methods 2_3 in Figure 2.4) also is significant, meaning that faculty ratings (M2) are discriminated from the observer ratings (M3). Convergent validity is somewhat in question because the average size of trait components is similar to that of method variance components (from Table 2.6, 1.47 and

18 6 LEVENT DUMENCl Observed Scores Scaling Orthonormal Contrast (Measures) Factors Contrasts Factors FIGURE 2.4 Covariance component model.

19 2. MULTITRAIT-MULTIMETHOD ANALYSIS 61 TABLE 2.6 Parameter Estimates from the Covariance Component Analysis Model, i, i i i i Fixed orthonormal contrast matrix Trait contrasts Method contrasts Measure Scaling F_G!234 2_34 3_4 1_23 2_3 Unique Method 1 T l/2v~ +1/2 T /2V~ -1/6 T3.95 +l/2v~ -1/6 T /2X/3-1/6 Method 2 T /2V~ + 1/2 T /2~/3-1/6 T /2V3-1/6 T4.98 +l/2v~ -1/6 Method 3 T /2X/3 + 1/2 T /2X/3-1/6 T /2V3-1/6 T /2X/3-1/6 Variance components F_G 1 Traits 1_ _34 3_4 Methods 1_23 2_3 +1/X/ /3V~ + 1/V~.29-1/3V~ +l/v~ +l/v~.33-1/3v'2-1/v~ +I/V~.33-1/2X/'6 + 1/2X/ /3X/2-1/2V~ +1/2X/2.43-1/3V~ +I/V~ -1/2V~ +l/2v~.49-1/3v~ +I/V~ -1/2V~ +1/2V~.38-1/2X/6-1/2X/ /3V'2-1/2~/'6-1/2V~.49-1/3~c2 +l/x./6-1/2v6-1/2v~.37-1/3v~ +I/V~ +l/2v~ -1/2V~ , respectively). A strong convergent validity claim requires much larger trait components than the method components. Any substantive interpretation of the general variate lies outside the CCA and requires further research. IV. COMPOSITE DIRECT PRODUCT MODEL An additive trait-method relation is assumed in the ANOVA, CFA, and CCA modeling approaches to the MTMM matrix, but Campbell and O'Connell (1967, 1982) pointed out that trait and method effects may be combined in a multiplicative fashion. Browne (1984; see also Cudeck, 1988) introduced the composite direct product (CDP) model for MTMM data as a means of formally modeling the multiplicative trait-method effect. The

20 62 LEVENT DUNENCI restricted second-order factor model specification of the CDP model adopted from Wothke and Browne (199) is depicted in Figure 2.5. The free parameters include 12 scaling factors (first-order factor loadings), 12 unique trait components (Dij), and three sets of trait variance-covariance matrices that are set equal across methods (the second-order variance/ covariance matrix of F_Ti). The second-order factor loadings are related to the multiplicative method effect, as will be explained below. The CDP model is illustrated with an artificial MTMM matrix (see Table 2.7) involving four clinical abilities (traits) measured by three types of ratings (methods). The CDP model in Figure 2.5 provides an excellent fit to the MTMM matrix in Table 2.7 (~5) = 43.4; p >.1), suggesting that the CDP is a plausible model for the MTMM matrix. The maximum likelihood parameter estimates appear in Tables 2.8, 2.9, and 2.1. A visual inspection of parameter estimates suggests that all twelve variables are measured on a comparable scale, given the narrow range of scaling parameters from.69 to.72 (see Table 2.8). Correspondingly, the unique trait components are relatively similar in size (i.e., from.98 to 1.1). The size of unique component associated with the faculty ratings (M2), however, is twice as large as the ratings at the monitor (M3) (i.e., 1.24 and.68, respectively). Convergent validity and discriminant validity are assessed using trait and method multiplicative correlation components. Note that these components are estimated from the CDP model, which are adjusted for measurespecific and random error components (see Figure 2.5). Parameter estimates (see Tables 2.9 and 2.1) and some basic algebra are needed to derive the trait and method multiplicative correlation components, IIv_T and 1-IF_M, respectively. The multiplicative trait correlation components are obtained directly from Table 2.1. The pattern of multiplicative trait components (1-IF_T) is assumed to be the same across three methods but only those trait components associated with the first method (i.e., patient ratings) are given in boldfaces in Table 2.1 for clarity of presentation. The following steps are needed to obtain the disattenuated multiplicative method components. The elements of C(3x3) matrix are given in boldface in Table 2.9. The pattern of C matrix is set equal across all four traits but only those elements associated with history taking (T1) are given in boldface in Table 2.9. The sum of square of the C matrix (the C matrix multiplied by its transpose, i.e., the C') yields = C C' = CC'

21 2. MULTITRAIT-MULTIMETHOD ANALYSIS 63 Observed Scores (Measures) First-Order Factors Second-Order Factors Second-Order Factor Covariances._ i -'~ I T4MI ~' i T4M 2 t I~, I TM i T2M3 I T4M3 " 1 FIGURE 2.S Composite direct product model.

22 64 LEVENT DUMEhlCI TABLE 2.7 Hypothetical Multitralt-Multimethod Correlation Matrix for the Composite Direct Product Example =,,,, Method 1 Method 2 Method 3 Measure TI T2 T3 T4 T1 T2 T3 T4 T1 T2 Ta T4 Method 1 T1 1. T T T Method 2 T T T T Method 3 T T T T TABLE 2.8 Parameter Estimates from the Composite Direct Product Model Measure Scaling matrix Unique methods Unique traits Method 1 T1.7 T2.69 T3.71 T4.7 Method 2 T1.7 T2.69 T3.72 T4.69 Method 3 T1.7 T2.7 T3.71 T

23 x.s s E A. s_ o u_ k.? " C 4) 4) U) o Z k ~~ E u E & 1 E ul L E 1 N ILl _1

24 ~D o om.,m o u (',1 L_ M,,.m oo G~ T. t~ E a. t~ L~ on t~ G~ L4r~ ~ n_ E o E E ohm Ul u,i L_ o.m. c; r4 uj [-'l~l~l~l~l i I i i{"j

25 2. MULTITRAIT-MULTIMETHOD ANALYSIS 67 The CC' matrix is rescaled into a correlation matrix by dividing each covariance term into the product of two respective standard deviations, that is,.61/[(v~)(x/-.-.-.~)] =.91,.17/[(V~)( 1V'L~.54)] =.14, and.25[(v~-5)( 1V'~.54)] =.3, which yields the following multiplicative method correlation components: I-IF_M -" The off-diagonal elements of the rift and I'IF_M matrices are related directly to the validity diagonals and to the monomethod correlations of the MTMM matrix, respectively. Convergent validity is achieved when the off-diagonal elements of the 1-IF_M matrix approach unity. The average correlation components of the IIF_M matrix is equal to.45, indicating poor convergent validity in this example. For the discriminant validity, the trait correlation components should be close to zero and should also be less than the method correlation components. From the IIF_T matrix, the average trait correlation components are equal to.43, which not only is much larger than zero but also is slightly smaller than the average method correlation components. Therefore, discriminant validity for this hypothetical MTMM matrix is not supported in this hypothetical example. V. CONCLUSION Definitions of convergent validity, discriminant validity, and the method effect differ to some extent from one analytic technique to another. Extreme caution is warranted when interpreting the results of an MTMM analysis, especially when the analysis does not produce any fit statistic. When a fit statistic is provided, the investigator must first determine if the model fits the data. If not, no claims for convergent or discriminant validity can be made. The finding of a well-fitting MTMM model signifies that modelbased interpretations are justified, but not necessarily that support for construct validity was obtained. A well-fitting MTMM model may fail to support construct validity, as illustrated hypothetically in this chapter. When the fit is plausible, then the next step is to interpret the parameter estimates obtained from a particular MTMM model. Conclusions drawn from an MTMM analysis, therefore, should be based on the model of choice. The method effect plays an important role in the analysis and interpre-

26 68 LEVENT DUMENCI tation of MTMM matrices. Methods long have been considered trait contaminants, but both traits and methods have their own places in the nomological network (Cronbach, 1995; Dumenci & Windle, 1998). The attitude that "traits are good and methods are bad," therefore, should be abandoned. Independent replication is the essential element in any scientific inquiry. Thus, all information necessary to replicate the MTMM analysis should be reported so that the analysis can be verified independently by others. That includes the correlation-covariance matrix (MTMM) and distributional characteristics of measures. The adoption of the specific MTMM analytic procedure, on which the substantive interpretation is based, should be justified on theoretical, empirical, and statistical grounds. When two or more analytic techniques are used, all analyses should be reported, along with the problems encountered in fitting each MTMM model. Such information not only gives readers the opportunity to draw their own conclusions, but also informs the performance of different analytic models under certain conditions. A. Choosing an Analytic Technique An in-depth understanding of measurement process and an extensive knowledge of substantive theory provide the ultimate guidance in model selection. Familiarity with various MTMM techniques facilitates the choice of analysis. Applied researchers should start thinking about the possible MTMM analyses during the initial stages of research. The statistical model used should provide a sound representation of the measurement process. Research design plays an important role in choosing an MTMM analysis. For example, two traits measured by two methods satisfy the minimum requirements for an MTMM matrix, but a minimum of nine measures (i.e., three traits assessed by three methods) is required to use the CFA, CCA, or CDP models. Statistical tests also should be used to verify model assumptions. The expected method effect plays an important role in the selection of an analysis. For example, when the methods consist of three self-report batteries supposedly measuring the same set of traits, the choice of analysis should be able to test for the presence of a general method effect, related methods, unrelated methods, and no method effect at all. The presence of method correlations is much less likely when rather different procedures such as self-report, expert rating, and a physiological variable (e.g., reaction time) are used to measure a trait such as intelligence (Wolins, 1982). Such complexities should be taken into account in selecting the model. Applied researchers typically will face alternative strategies for choosing a statistical model for MTMM data. I recommend choosing the statistical model that best represents the measurement process and the substantive

27 2. MULTITRAIT-MULTIHETHOD ANALYSIS 69 theory. Strong construct validity claims can be made when the model provides a good fit to the observed MTMM matrix. When the model fit is poor, however, conclusions about convergent and discriminant validity cannot be drawn. Alternatively, two or more analytic procedures can be used to analyze an MTMM matrix. The use of multiple analytic techniques increases the possibility of finding a well-fitting model for the MTMM matrix. The major obstacle of choosing a best fitting model in such a post hoc fashion is that it has a strong likelihood of capitalizing on chance, hence, the findings may not be generalized across different samples. Furthermore, different analytic procedures may yield contradictory construct validity evidence when two or more MTMM models fit the observed MTMM matrix equally well. Therefore, applied researchers who adopt this strategy must be willing to settle for a weak construct validity claim. A third strategy offers a compromise between the first two strategies. Many analytic techniques (e.g., the CFA, the CCA, and the CDP) offer alternative parameterizations of trait, method, and error effects. For example, alternative specifications of the CDP model allow for testing of a range of error structures. The CDP model without the error parameters in Figure 2.5 corresponds directly to the model proposed by Swain (1975). Nested model comparison strategies for the CDP model also are available to test convergent validity and discriminant validity, as well as the method effect (Bagozzi & Yi, 199). Thus, applied researchers can specify alternative models a priori, without committing to a single model, thereby avoiding the problem of capitalizing on chance. Applied researchers should realize, however, that this strategy offers a weaker construct validity claim than the first strategy, and the probability of obtaining a well-fitting model for the MTMM matrix is less than with the second. In conclusion, there is no one simple answer to the question of what is the best quantitative technique to analyze an MTMM matrix. Instead, researchers must consider to what extent conclusions drawn from the analysis can be argued successfully against alternative explanations. ACKNOWLEDGMENTS I thank Werner Wothke for his comments on an earlier version of this chapter. I also am thankful to Anne Richards for editorial assistance. REFERENCES Bachman, L. F., & Palmer, A. S. (1981). The construct validation of the FSI Oral Interview. Language Learning, 31, Bagozzi, R. P., & Yi, Y. (199). Assessing method variance in multitrait-multimethod matrices:

28 61 LEVENT DUMENCI The case of self reported affect and perceptions at work. Journal of Applied Psychology, 75, Bentler, P. M. (1989). EQS structural equations program manual Los Angeles: BMDP Statistical Software, Inc. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley. Bork, J. K., & Bargmann, R. E. (1966). Analysis of covariance structures. Psychometrika, 31, Browne, M. W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, Byrne, B. M., & Goffin, R. D. (1993). Modeling MTMM data from additive and multiplicative covariance structures: An audit of construct validity concordance. Multivariate Behavioral Research, 28, Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, Campbell, D. T., & O'Connell, E. J. (1967). Method factors in multitrait-multimethod matrices: Multiplicative rather than additive? Multivariate Behavioral Research, 2, Campbell, D. T., & O'Connell, E. J. (1982). Methods as diluting trait relationships rather than adding irrelevant systematic variance. In D. Brinberg & L. H. Kidder (Eds.). Forms of validity in research. New directions for methodology of social and behavioral science (Vol. 12, pp ). San Francisco: Jossey-Bass. Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, " Cronbach, L., (1995). Giving method variance its due. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp ). Hillside, NJ: Lawrence Erlbaum Associates. Cronbach, L., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York: Wiley. Cudeck, R. (1988). Multiplicative models and MTMM matrices. Journal of Educational Statistics, 123, Dumenci, L. (1995). Construct validity of the Self-Directed Search using hierarchically nested structural models. Journal of Vocational Behavior, 47, Dumenci, L., & Windle, M. (1996). A latent trait-state model of adolescent using the Center for Epidemiologic Studies-Depression scale. Multivariate Behavioral Research, 31, Dumenci, L., & Windle, M. (1998). A multitrait-multioccasion generalization of latent traitstate model: Description and application. Structural Equation Modeling: A Multidisciplinary Journal 5, Engstrom, C. P. Persson, L. O., Larsson, S., & Sullivan, M. (1998). Reliability and validity of a Swedish version of the St Georges Respiratory Questionnaire. European Respiratory Journal, 11, Forsythe, G. B., McGaghie, W. C., & Friedman, C. P. (1986). Construct validity of clinical competence measures: A multitrait-multimethod matrix study using confirmatory factor analysis. American Educational Research Journal, 23, Guilford, J. P. (1954). Psychometric methods. New York: Wiley. Hubert, L. J., & Baker, F. B. (1978). Analyzing the multitrait-multimethod matrix. Multivariate Behavioral Research, 13, Jackson, D. N. (1969). Multimethod factor analysis in the evaluation of convergent and discriminant validity. Psychological Bulletin, 72, Joreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 57, Kavanagh, M. J., MacKinney, A. C., & Wolins, L. (1971). Issues in managerial performance: Multitrait-multimethod analysis of ratings. Psychological Bulletin, 75, Kenny, D. A. (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual

29 2. MULTITRAIT-MULTIMETHOD ANALYSIS 6 I I issues. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A #stschrift honoring Donald W. Fiske (pp ). Hillside, NJ: Lawrence Erlbaum Associates. Kiers, H. A. L., Takane, Y., & ten Berge, J. M. F. (1995). The analysis of multitrait-multimethod matrices via constrained component analysis. Psychometrika, 61, Kumar, A., & Dillon, W. R. (1992). An integrative look at the use of additive and multiplicative covariance structure models in the analysis of MTMM data. Journal of Marketing Research, 29, Levin, J., Montag, I., & Comrey, A. L. (1983). Comparison of multitrait-multimethod, factor, and smallest space analysis on personality scale data. Psychological Reports, 53, Marsh, H. W. (1996). Physical Self Description questionnaire: Stability and discriminant validity. Research Quarterly for Exercise & Sport, 67, Ostrom, T. M. (1969). The relationship between the affective, behavioral, and cognitive components of attitude. Journal of Experimental Social Psychology, 5, Schriesheim, C. A. (1981). The effect of grouping or randomizing items on leniency response bias. Educational and Psychological Measurement, 41, Schweizer, K. (1991). An equal-level approach to the investigation of multitrait-multimethod matrices. Applied Psychological Measurement, 15, Sidani, S., & Jones, E. (1995). Use of the multitrait multimethod (MTMM) to analyze family relational data. Western Journal of Nursing Research, 17, Stanley, J. C. (1961). Analysis of unreplicated three-way classifications, with applications to rater bias and trait independence. Psychometrika, 26, Sullivan, J. L., & Feldman, S. (1979). Multiple indicators: An introduction. Beverly Hills, CA: Sage. Swain, A. J. (1975). Analysis of parameter structures for variance matrices. Unpublished doctoral dissertation, University of Adelaide. Werts, C. E., & Linn, R. L. (197). Path analysis: Psychological examples. Psychological Bulletin, 74, Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitraitmultimethod data. Applied Psychological Measurement, 9, Wolins, L. (1982). Research mistakes in the social and behavioral sciences. Ames, IA: Iowa State University Press. Wothke, W. (1984). The estimation of trait and method components in multitrait-multimethod measurement. Unpublished doctoral dissertation, University of Chicago, Department of Behavioral Science. Wothke, W. (1987). Multivariate linear models of the multitrait-multimethod matrix. Paper presented at the meeting of the American Educational Research Association, Washington, D.C. (Educational Resources Information Center, Document No. TM ). Wothke, W. (1996). Models for multitrait-multimethod analysis. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advance structural equation modeling: Issues and techniques (pp. 7-56). Hillside, NJ: Lawrence Erlbaum Associates. Wothke, W., & Browne, M. W. (199). The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika, 55,