EJBRM Volume 12 Issue 1, July Ann Brown

Transcription

1 EJBRM Volume 12 Issue 1, 2014 July 2014 Ann Brown This issue has six papers of which the first three address important issues encountered in developing quantitative models (Niemelä-Nyrhinen, Leskinen and Gupta), the fourth (Wright and Ogbuehi) assesses the strengths and weaknesses of the three main data collection methods used in obtaining the views of current adolescents (generation Z), the fifth offers a detailed description of a complex example of multiple case study analysis (Vohra) and the last (Løkke and Sørensen ) makes a strong case for the potential value of theory testing using case studies. Niemelä-Nyrhinen and Leskinen are concerned with the apparent weakness of some marketing models developed using Structural equation modelling methods (SEM), which in their view fail to consider the potential effects of multicollinearity on these models. They develop a way to mitigate the effects of multicollinearity on such models. Bjorn assesses the effects of using standard statistical tests on ordinal data created on a Likert scale. This is an important issue for many researchers, as Likert scales are used in a wide range of subjects and disciplines. The paper focuses on the potential effects of using these tests when one of the assumptions required for the tests (developed for interval data) are breached. The factor analysed is the effect of non equistant values (the distance between the Likert scale ranks proves to vary significantly over the range of the scale). Saurabh Gupta details approaches using Structural Equation Modelling (SEM) that can be used to analyse experimental research designs, and illustrates the technique by re-analysing data from a previous IS research study The fourth paper by Wright and Ogbuehi offers a valuable guide to any researcher with adolescents as their subjects. They compare the three main methods (paper survey, electronic questionnaire and face to face interviews) of eliciting information from adolescents of generation Z to find out which produces the most accurate and robust information. The fifth paper by Vohra demonstrates the application of a multiple case study design, combined with mixed methods, to the complex question of business leadership. It offers a valuable guide to generating and analyzing an extra-ordinary range of data - both qualitative and quantitative. The analysis step of case study research and especially the synthesis of the results for multiple cases is always a challenge and this paper gives an insight into one successful approach. The last paper by Løkke and Sørensen returns to the contentious subject of case study research methods. The authers differentiate sharply between theory testing and theory building and focus on the strengths of the case study approach for theory testing. 1

2 2

3 Multicollinearity in Marketing Models: Notes on the Application of Ridge Trace Estimation in Structural Equation Modelling Jenni Niemelä-Nyrhinen 1 and Esko Leskinen 2 1 Jyväskylä University School of Business and Economics, Finland 2 University of Jyväskylä, Department of Mathematics and Statistics, Finland jenni.niemela-nyrhinen@jyu.fi Abstract: Multicollinearity in Structural Equation Modelling (SEM) is often overlooked by marketing scholars. This is unfortunate as multicollinearity may lead to fallacious path coefficient estimates or even bring about statistical nonsignificance of the parameter estimates. Previous empirical illustrations on mitigating the effects of multicollinearity are virtually non-existent in the literature. The purpose of this paper is to empirically illustrate the problem of multicollinearity in marketing models and the use of ridge trace estimation in mitigating the effects of multicollinearity in SEM, using the LISREL program. Two slightly differing ridge estimation procedures are illustrated using real data with a multicollinearity problem: Method A, in which the ridge constant is added manually to all diagonal elements of the correlation matrix of the variables in the model, and Method B, in which the ridge constant is added manually only to the diagonal elements of the correlation matrix of the exogenous and explanatory endogenous variables in the model. In evaluating suitable values of the ridge constant, the ridge trace method is used. It is concluded that ridge trace estimation is an effective way of mitigating the effects of multicollinearity in SEM. With same ridge constant values, both methods produce same point estimates of path coefficients, but Method B produces smaller standard errors of parameter estimates and larger squared multiple correlations than Method A. Keywords: marketing modelling, multicollinearity; structural equation modelling; ridge trace estimation; LISREL 1. Introduction Structural equation modelling (SEM) is particularly useful for marketing research that often deals with models consisting of unobserved theoretical constructs (e.g. benefits, attitudes, value, customer satisfaction) that may only be measured through observable indicators (Steenkamp & Baumgartner, 2000). Indeed, the advantages of SEM, such as its ability to account for measurement error and manage multiple endogenous variables, have probably contributed to its spread among marketing scholars (Steenkamp & van Trijp, 1991; Steenkamp & Baumgartner, 2000). From the 1970s to the early 1990s the use of SEM increased fairly steadily among marketing scholars (Baumgartner & Homburg, 1996). Following a clear decrease in published papers at the end of the 20 th century, in the 21 st century the use of SEM in marketing modelling has achieved a phase of maturity (Martinez-López et al., 2013). SEM is a commonly used tool for theory testing in marketing research. In fact, articles in which SEM is used appear frequently in most major marketing and consumer behaviour journals (Baumgartner & Homburg, 1996; Steenkamp & Baumgartner, 2000; Martinez-López et al. 2013). Extensive use of SEM has naturally also led to the appearance of articles addressing problem areas and suggesting improvements for future modelling (e.g. Baumgartner & Homburg, 1996; Hulland et al., 1996; Grewal et al., 2004). Regardless of its maturity, application of SEM in marketing research has a need for improvement (Martinez-López et al., 2013). As Baumgartner and Homburg (1996) have stated, SEM may be a dangerous tool in the hands of inexperienced users. The quality of marketing knowledge generated based on marketing research using SEM is dependent on how well researchers apply this methodology (Martinez-López et al., 2013). One of the possible issues encountered in using SEM is multicollinearity, that is, high correlations between the latent exogenous variables. This is one of the potential problems that marketing science models in general suffer from (Leeflang, 2011). Overall, correlation between exogenous variables is a fact of life in survey research (Grapentine, 2000) and in marketing models these explanatory variables are often highly correlated with each other (Mahajan et al., 1977, see also Grewal et al., 2004). Unfortunately, multicollinearity is frequently overlooked by marketing scholars. To prove this point Grewal et al. (2004) reviewed 42 articles using either confirmatory factor analysis or SEM that had been published in the 1999 and 2000 issues of the Journal of Marketing, Journal of Marketing Research and Journal of the Academy of Marketing Science. They found that although potential multicollinearity problems could be assessed for 31 ISSN ACPIL Reference this paper as: Niemelä-Nyrhinen J and Leskinen E. Multicollinearity in Marketing Models: Notes on the Application of Ridge Trace Estimation in Structural Equation Modelling The Electronic Journal of Business Research Methods, Volume 12 Issue (pp 3-15), available online at

4 Electronic Journal of Business Research Methods Volume 12 Issue of the studies through the published correlation matrix, not a single article discussed how multicollinearity might have affected the results. Grewal et al. (2004) speculate on the reasons behind this unfortunate dismissal of multicollinearity by marketing researchers and suggest that one basic cause might be an assumption that multicollinearity does not pose a problem in SEM. However, this is a severe misconception. As in other econometric models, in SEM the reliability of one s results may suffer due to multicollinearity (Jagpal, 1982). Multicollinearity may lead to fallacious parameter estimates and even induce statistical non-significance of parameter estimates (Grewal et al., 2004), consequently leading to misguided interpretation or elimination of important predictors from the model. Another basic cause of the dismissal of multicollinearity problems among marketing scholars may simply be a lack of knowledge of how to deal with these problems in the context of SEM (Grewal et al., 2004). This paper proposes ridge estimation as one possible way of mitigating the effects of multicollinearity in SEM using the LISREL program (Jöreskog & Sörbom, 2005). Although ridge regression analysis has for long been one of the most popular approaches to addressing multicollinearity among marketing scholars (Subhash & William, 1981), to date ridge-type estimation is not commonly used in the SEM context and illustrations of its use seem virtually absent from the literature (for one rare exception see Jagpal, 1982). The main purpose of this paper is to empirically illustrate the problem of multicollinearity and the use of ridge estimation in mitigating the effects of multicollinearity in SEM. Two slightly differing ridge estimation procedures (Method A and B) are illustrated. For these illustrative purposes the Technology Acceptance Model (TAM), (Davis, 1989; Davis et al., 1992) is used. 2. An illustrative example In the example data an extremely high correlation (0.97) between two latent exogenous variables (see Table 1), namely perceived usefulness and enjoyment, was found. This finding was somewhat unexpected and alternative explanations that are related to the particular target group, novelty of the services and nature of the services studied may be advanced. In any case, the data provides a particularly good example that may be used to illustrate the options available to mitigate effects of multicollinearity in SEM. Further, TAM provides a current example of multicollinearity because a great deal of the current effort expended on marketing modelling is likely to be directed at the issues in the field of electronic business (Mahajan & Venkatesh, 2000). The example data, used in this paper, are drawn from a study that aimed to explain acceptance of mobile content services among Finnish baby boomers (Niemelä-Nyrhinen, 2009). Examples of mobile content services include mobile news, mobile banking, downloading ring tones or logos, mobile shopping, mobile games and mobile ticketing. 2.1 Technology Acceptance Model, TAM TAM is tailored to model user acceptance of information systems with the objective of both explaining and predicting user behaviour across a wide range of technologies and user populations (see Figure 1). TAM uses Fishbein and Ajzen s (1975) Theory of Reasoned Action (TRA) as a theoretical basis on which to explicate causal linkages between the variables in the model (Davis et al., 1989). Although TAM was originally created to explain user acceptance of information technology at work (Venkatesh & Davis, 2000), over the years it has been successfully applied to consumer acceptance as well (see e.g., Moon & Kim, 2001; Dabholkar & Bagozzi, 2002; van der Heijden, 2003; Pavlou, 2003; Curran & Meuter, 2005; Kim & Forsythe, 2009). Figure 1: Technology Acceptance Model, TAM 4 ISSN

5 Jenni Niemelä-Nyrhinen and Esko Leskinen The original TAM posited that technology acceptance can be explained by two beliefs, namely perceived usefulness and perceived ease of use. In addition to the belief-intention links, TAM states that ease of use has an effect on usefulness (Davis et al., 1989). This relation is very logical the easier a system is, the more useful it can be (Venkatesh & Davis, 2000). Later Davis et al. (1992) made an important addition to TAM, namely a belief called perceived enjoyment. Taking into account the fact that many technological applications now include entertaining elements, it is important that a model explaining technology acceptance includes, in addition to utilitarian aspects, hedonic aspects too. Further, just as perceived ease of use influences usefulness, it is likely to influence perceived enjoyment, because systems that are difficult to use are less likely to be perceived as enjoyable (Teo et al., 1999). 2.2 Construct measurement The survey instrument (see Appendix A) was developed using existing scales for the belief constructs and behavioural intention. Four of Davis s (1989) usefulness items were used. Two of the items in the original scale were left out since their wording heavily stressed work-related goals. Perceived ease of use was measured with four items adapted from Davis et al. (1989). The three items used to measure perceived enjoyment were adapted from Venkatesh (2000). For measurement of behavioural intention three items recommended by Ajzen (2002) were applied. In measuring these constructs, a seven-point Likert-scale was used with completely disagree and completely agree as anchors. For the scales used, high scale reliability was indicated by Cronbach alphas ranging from 0.82 to 0.87 (Niemelä-Nyrhinen, 2009). 2.3 Data collection and descriptive statistics The data used in this paper were gathered through a structured postal questionnaire in It is part of a wider quantitative study of Finnish baby boomers and their usage of new innovative technological services (Niemelä-Nyrhinen, 2009). Finnish people born between 1945 and 1955 (aged between 50 and 60 at the time of the survey) were included in the sample. The sample was provided by the Population Register Centre of Finland and it was drawn from all Finnish-speaking Finns through random sampling. Gathering data on such advanced mobile content services is possible in Finland since mobile technology for communication purposes has been widely adopted. A total of 1500 surveys were mailed and 620 usable responses were gathered (response rate 41.3 per cent). A high response rate was expected as in our experience 50+ consumers respond relatively conscientiously to mail surveys. More females (58.8 per cent) than males (41.2 per cent) returned the survey questionnaire. Ages ranged from 50 to 60 years, with an average age of 55. Following the instructions given by Jöreskog (2005) missing values were imputed to the data by matching on other variables, a procedure that is available in PRELIS. After imputation the data with no missing values, fit for use in the analyses, consist of 584 cases (Jöreskog & Sörbom, 2005). 2.4 Estimation results of TAM The measurement models and the structural model were estimated using the LISREL program (Jöreskog & Sörbom, 2005). After a separate estimation of each measurement model, the measurement model with all TAM E 2 (71) = , p = 0.000), but rather good fit is implied by the following: RMSEA = 0.051, 90 per cent confidence interval for RMSEA = (0.042 ; 0.060), p-value for test of close fit (RMSEA<0.05) = 0.42, CFI = 0.98 and NFI = The correlations of the latent variables are high, particularly the correlation between perceived enjoyment and perceived usefulness (see Table 1). However, these correlations support the use of TAM. Table 1: The estimated correlation matrix of the latent variables measurement model (standard errors in parenthesis), N= 584 intention, enjoyment, (.01512) usefulness, (.01461) (.01199) ease of use, (.02829) (.02428) (.02638) 5 ACPIL

6 Electronic Journal of Business Research Methods Volume 12 Issue A s 2 test was performed to assure the discriminant validity of usefulness and enjoyment. In other words, a check was performed to see that usefulness and enjoyment form not one but two separate factors. According to this test the factors should be 2 (1) = 4.73, p = ). Also from a theoretical point of view this seems more reasonable. The latent variable pairs of usefulness-intention and enjoymentintention also have correlations above 0.9 (see Table 1). To assure discriminant val 2 tests were T 2 (1) = 35.13, p < 2 (1) = 36.37, p < 0.001). Following the seque 2 tests, the structural relationships of TAM were estimated (see the estimation results in Figure 2). Figure 2: The estimation results for Technology Acceptance Model The goodness-of-fit results of the estimated TAM equated with the previous four-latent-variables measurement model because of the data-equivalence of the models. Hypothesized effects are found: ease of use has a strong positive effect on both usefulness (s 1 = 0.782, t = 21.76) and enjoyment 2 = 0.775, t = 22.29). Usefulness in turn seems to have a strong positive effect on intention 1 = 0.897, t = 2.95). Exceptional results are the effect of ease of use on intention (standardized 3 = , t = - 2 = 0.060, t = 0.985), which are found to be non-significant. The effect of ease of use on intention is mediated by usefulness. Its standardized indirect effect on intention is positive and strong (0.749, t = 13.02). Ease of use explains approximately 60 per cent of the variance in both usefulness (squared multiple correlation, R 2 = 0.61) and enjoyment (R 2 = 0.60). The absence of significant effects of enjoyment on intention and ease of use on intention seems dubious since the correlation between enjoyment and intention is very high (0.91) and the correlation between ease of use and intention is high (0.72) (see Table 1 above). 3. Ridge estimation strategy 3.1 Model and data We analyse the following recursive structural equations model (TAM) between the four factors: 1 1 1, (1) 2 2 2, (2) , (3) is also estimated. The corresponding graph is presented in Figure ISSN

7 Jenni Niemelä-Nyrhinen and Esko Leskinen Figure 3: The parameterization of Technology Acceptance Model Data used for ridge estimation consists of the correlation matrix given in Table 1. In order to reduce effects of multicollinearity between latent variables, we added the ridge constant to the correlation matrix given in Table 1 (see e.g., Grewal et al., 2004; Kuusinen & Leskinen, 1988). The ridge estimation procedures are constructed in two ways, termed Method A and Method B. In Method A the ridge constant k is added to all the diagonal elements of the correlation matrix of the variables in the model (see Gunst at al., 1976) as latent root regression, and the ridge option available in the LISREL program (Jöreskog & Sörbom, 2005; Yuan & Chan, 2008). In Method B the ridge constant k is added only to the diagonal elements of the correlation matrix of the explanatory endogenous and exogenous variables as the ordinary ridge regression method (see Hoerl & Kennard, 1970). Table 2 below displays the collected characteristics of correlation matrices to the ridge estimation Methods A and B (see also Ofir & Khuri, 1986). Table 2: Detecting and analysing multicollinearity between latent variables R λ min λ max λ min p 1 i= 1 λi VIF-range A R 4x B R 3x Note 1. R is the determinant of R and 's are eigenvalues of R. Note 2. R is the 4x4-correlation matrix of 3, 2, 1 and for Method A, and R is 3x3-correlation matrix of 2, 1 and for Method B, respectively. All these values indicate strong collinearities between variables. The smallest eigenvalues are close to zero, and these values also affect other values presented in Table 2. For example, a small value of determinant of 3x3 correlation matrix R, R = may be an effect of the estimation results of the model. Further, for example, collinearity interpretations of the results of the correlation matrix by using Method B are T 2 1 T of the ng collinearity between explanatory variables in Equation 3. The collinearity equation between explanatory variables can be evaluated as follows: The eigenvector corresponding to the smallest eigenvalue is (.7012, ,.0145) T, from which we derive the following collinearity relation between explanatory variables 2-1 B collinearity equation, and then it can be seen that the collinearity relation is approximately 7 ACPIL

8 Electronic Journal of Business Research Methods Volume 12 Issue The ridge estimation procedures A and B were carried out by manually adding the ridge constant to the correlation matrices and by using the LISREL program. The SIMPLIS input files for Ridge Method A and B, Ridge constant k =.10 are provided below: Testing TAM Ridge Method A Observed variables: inten enjoy useful ease Correlation matrix: Sample size: 584 Relationships: inten = useful enjoy ease enjoy = ease useful = ease Set the Covariances of useful and enjoy free LISREL Output: ME=GLS SC MI EF ND=5 Path Diagram End of Problem Testing TAM Ridge Method B Observed variables: inten enjoy useful ease Correlation matrix: Sample size: 584 Relationships: inten = useful enjoy ease enjoy = ease useful = ease Set the Covariances of useful and enjoy free LISREL Output: ME=GLS SC MI EF ND=5 Path Diagram End of Problem 3.2 Results of ridge trace estimation Results of ridge trace estimation for TAM In Figure 4, the ridge trace (k between ) for the TAM is presented. It should be noted that ridge point estimation results are same for both Methods A and B. 8 ISSN

9 Jenni Niemelä-Nyrhinen and Esko Leskinen 1,00 0,80 0, , ,20 0,00 0 0,02 0,04 0,06 0,08 0,1 0,12 0,14 0,16 0,18 0,2 0,22 0,24 0,26 0,28 0,3 0,32 0,34 0,36 0,38 0,4 0,42 0,44 0,46 0,48 0,5 3-0,20 k usefulness to intention 1, enjoyment to intention 2, ease of use to usefulness 1 ease of use to enjoyment 2, ease of use to intention 3 Figure 4: Equal ridge traces for Methods A and B Furthermore, Tables 3 and 4 below present the ridge estimation results with corresponding standard errors and t-values for Methods A and B, respectively. In addition, the squared multiple correlations R 2 for the three equations are given. 9 ACPIL

10 Electronic Journal of Business Research Methods Volume 12 Issue Table 3: Estimation results by using Ridge Method A Ridge constant k R 2 1 R 2 2 R (s.e.) (.062) (.061) (.026) (.026) (.024) t-value (s.e.) (.054) (.054) (.027) (.027) (.026) t-value (s.e.) (.051) (.050) (.027) (.028) (.027) t-value (s.e.) (.048) (.048) (.028) (.028) (.028) t-value (s.e.) (.047) (.046) (.029) (.029) (.029) t-value (s.e.) (.046) (.045) (.029) (.029) (.030) t-value (s.e.) (.045) (.044) (.030) (.030) (.031) t-value (s.e.) (.044) (.044) (.030) (.030) (.032) t-value (s.e.) (.044) (.043) (.031) (.031) (.032) t-value (s.e.) (.043) (.043) (.031) (.031) (.033) t-value (s.e.) (.043) (.043) (.031) (.032) (.033) t-value (s.e.) (.041) (.041) (.035) (.035) (.037) t-value ISSN

11 Jenni Niemelä-Nyrhinen and Esko Leskinen Table 4: Estimation results by using Ridge Method B Ridge constant k R 2 1 R 2 2 R (s.e.) (.062) (.061) (.026) (.026) (.024) t-value (s.e.) (.051) (.050) (.027) (.027) (.024) t-value (s.e.) (.045) (.044) (.027) (.028) (.024) t-value (s.e.) (.041) (.041) (.028) (.028) (.024) t-value (s.e.) (.038) (.038) (.029) (.029) (.024) t-value (s.e.) (.036) (.036) (.029) (.029) (.024) t-value (s.e.) (.035) (.034) (.030) (.030) (.024) t-value (s.e.) (.033) (.033) (.030) (.030) (.0240) t-value (s.e.) (.032) (.032) (.031) (.031) (.024) t-value (s.e.) (.031) (.031) (.031) (.031) (.024) t-value (s.e.) (.031) (.030) (.031) (.032) (.024) t-value (s.e.) (.025) (.025) (.035) (.035) (.022) t-value When k = 0, estimations of parameters produce statistically non- 2 = (t 3 = (t = ), as was noted in Section 2.4 above. In evaluating appropriate values for the ridge constant k, the value k =.10 is chosen, because the ridge trace values of estimates seem to become rather stable beyond that value. This ridge solution gives the following estimation results: 11 ACPIL

12 Electronic Journal of Business Research Methods Volume 12 Issue Method A, k =.10 1 R 2 =.51 (1) (.029) t = R 2 =.50 (2) (.029) t = R 2 =.75 (3) (.046) (.045) (.030) t = Method B, k =.10 1 R 2 =.51 (1) (.029) t = R 2 =.50 (2) (.029) t = R 2 =.83 (3) (.036) (.036) (.024) t = The results of point ridge estimations in Equations 1-3 are the same for both Methods A and B. However, the estimated standard errors in Equation 3 are larger for Method A than for Method B, which produce smaller t- values for Method A than Method B, respectively. Therefore, for example, the chosen ridge constant k =.10 E 3 is non-significant using Method A (t = 1.76) but significant using Method B (t = 2.21, at a 5 per cent significance level). The differences in these results are caused by adding in Method A the ridge constant also into the diagonal element of dependent 3 in the correlation matrix R. From this it also follows that the estimated squared correlation coefficients R 2 3 are smaller for Equation 3 in using Method A than when using Method B, see Tables 3 and Estimation results of indirect effects Finally, indirect effects between ease of use and intention are considered, see Table 5 (see also MacKinnon, 2008). The special indirect effects are estimated using the Mplus program, Version 6.0 (Muthén & Muthén, ). Table 5: Estimation results for direct and specific indirect effects Ridge constant k (s.e.) (.062) (.061) (.026) (.026) (.024) (.054) (.048) t-value Method A (s.e.) (.046) (.045) (.029) (.029) (.030) (.036) (.033) t-value Method B (s.e.) (.036) (.036) (.029) (.029) (.024) (.030) (.027) t-value ISSN

13 Jenni Niemelä-Nyrhinen and Esko Leskinen When k = 0, the indirect effect via usefulness is statistically significant (t = ) but the indirect effect via B 3 from ease of use to intention is not statistically significant (t = -1.11) usefulness can be interpreted as the perfect mediator between ease of use and intention (MacKinnon, 2008). When k =.10 both special indirect effects via usefulness and intention are statistically significant for both Methods A and B. In Method A both usefulness and enjoyment are the 3 is not statistically significant (t = 1.764). On the other hand, using Method B, usefulness and enjoyment are now partial media 3 is statistically significant (t = 2.214). 4. Discussion Multicollinearity may blur the interpretation of a structural equation model. The empirical example used in this paper demonstrated how the dismissal of multicollinearity problems may lead to fallacious parameter estimates and even erroneous non-significance of the parameter estimates. We proposed the ridge estimation method for the treatment of multicollinearity and provided a detailed illustration of it in SEM using LISREL. The ridge estimation procedure was applied by using two different methods: Method A, in which the ridge constant was added manually to all diagonal elements of the correlation matrix of the variables in the model, and Method B, in which the ridge constant was added manually only to the diagonal elements of the correlation matrix of the explanatory exogenous and endogenous variables in the model. When evaluating suitable values of the ridge constant, the ridge trace method was used. For both Methods A and B, ridge point estimates were equal for the same ridge constant. Method B produced smaller standard errors of parameter estimates and larger squared multiple correlations, R 2, than Method A. Estimating the special indirect effects between variables produced results of the same kind. Although no assessment of superiority of one method over another can be made based on this study, we may conclude that Method A, which corresponds to the ridge option installed in the LISREL program, is not necessarily preferable to Method B. Overall, it should be noted that ridge option procedure in LISREL is not recommended to be used automatically, because it may produce arbitrary ridge constant values. Since it is beyond our illustrative purpose to untangle which of the two methods is actually superior to another, this endeavour is left for future studies. However, both methods illustrated in this paper would be easy to apply also to path models that are more versatile than the models considered herein. In our example when multicollinearity was not taken into consideration, the effect of a theoretically important variable perceived enjoyment on intention was estimated to be non-significant. The real non-significance of the enjoyment-intention link would most likely lead to the elimination of this independent variable from the model. However, closer scrutiny revealed that the non-significance of the effect found was caused by multicollinearity, that is, the extremely high correlation between usefulness and enjoyment. As a short-cut solution to multicollinearity, the highly correlated variables of usefulness and enjoyment could have been combined into one aggregate variable. However, this procedure would have had some negative implications compared to the handling of multicollinearity with ridge estimation, as it would have led to the formation of a theoretically difficult variable that combines behavioural beliefs regarding both usefulness and enjoyment of the services in question. Forming this aggregated variable would have caused the loss of interesting information, as in interpreting the results it certainly makes a difference if intention to use mobile content services among mature consumers is affected by their beliefs about the usefulness of the service alone or by both usefulness and enjoyment of the service. Further, since intention to use was affected by both usefulness and enjoyment the extent of those effects adds to our understanding of technology acceptance among mature consumers. Finally, if usefulness and enjoyment were combined, comparing research results with previous research would be difficult, if not impossible. In addition to the erroneous non-significance of the enjoyment-intention link, the direct effect of perceived ease of use on intention was estimated as non-significant when multicollinearity was not taken into consideration. The non-significance of the effect of ease of use on intention would imply that the effect of ease of use on intention is wholly mediated by usefulness and enjoyment. However when multicollinearity was addressed with ridge estimation (Method B) a small but significant direct effect of ease of use on intention was found, and usefulness and enjoyment were partial mediators of ease of use. This suggests that although ease 13 ACPIL

14 Electronic Journal of Business Research Methods Volume 12 Issue of use also directly affects intention, alone it is not likely to induce intention to use mobile content services. However, it is an important variable in acceptance research since it has an effect on intention mediated by usefulness and enjoyment. Further, the effect of ease of use on usefulness and enjoyment is very logical as has been noted in previous research (see Venkatesh & Davis, 2000; Teo et al., 1999). Ease of use alone is not enough to make a service useful or enjoyable but services that are difficult to use are less likely to be perceived as useful or enjoyable. Unfortunately, multicollinearity is a common problem in marketing research that often deals with models including such latent variables as quality, satisfaction, beliefs, attitudes and values. Latent constructs may be seen as the building blocks of marketing theory (Gilliam & Voss, 2013). Dismissing multicollinearity simply means unreliable results when testing marketing models and jeopardises the development of marketing theory. It is hoped that the illustration presented in this paper adds to the understanding of multicollinearity problems in SEM and provides marketing scholars with guidance on handling such problems. References Ajzen, I. (2002) Constructing a TPB Questionnaire: Conceptual and Methodological Considerations, [Online], Available: [17 March 2005]. Baumgartner, H. & Homburg, C. (1996) Applications of structural equation modeling in marketing and consumer research: A review, International Journal of Research in Marketing, vol. 13, no. 2, pp Curran, J. & Meuter, M. (2005) Self-Service Technology Adoption: Comparing Three Technologies, Journal of Services Marketing, vol. 19, no. 2, pp Dabholkar, P. & Bagozzi, R. (2002) An Attitudinal Model of Technology-Based Self-Service: Moderating Effects of Consumer Traits and Situational Factors, Journal of the Academy of Marketing, vol. 30, no. 3, pp Davis, F. (1989) Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly, vol. 13, no. 3, pp Davis, F., Bagozzi, R. & Warshaw, P. (1989) User Acceptance of Computer Technology: A Comparison of Two Theoretical Models, Management Science, vol. 35, no. 8, pp Davis, F., Bagozzi, R. & Warshaw, P. (1992) Extrinsic and intrinsic motivation to use computers in the workplace, Journal of Applied Social Psychology, vol. 22, no. 14, pp Fishbein, M. & Ajzen, I. (1975) Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research, Reading: Addison-Wesley. Gilliam, D. & Voss, K., (2013) A proposed procedure for construct definition in marketing, European Journal of Marketing, vol. 47, no. 1, pp Grapentine, T. (2000) Path Analysis vs. Structural Equation Modeling, Marketing Research, vol. 12, no. 3, pp Grewal, R., Cote, J. & Baumgartner, H. (2004) Multicollinearity and Measurement Error in Structural Equation Models: Implications for Theory Testing, Marketing Science, vol. 23, no. 4, pp Gunst, R.F., Mason, R.L. & Webster, J.T. (1976) A comparison of least squares and latent root regression analysis, Technometrics, vol. 18, no. 1, pp Hoerl, A.E. & Kennard, R.W. (1970) Ridge regression: biased estimation for nonorthogonal problems, Technometrics, vol. 12, no. 1, pp Hulland, J., Chow, Y. & Lam, S. (1996) Use of causal models in marketing research: A review, International Journal of Research in Marketing, vol. 13, no. 2, pp Jagpal, H. (1982) Multicollinearity in Structural Equation Models With Unobservable Variables, Journal of Marketing Research, vol. 19, no. 4, pp Jöreskog, K. (2005) Structural Equation Modeling with Ordinal Variables using LISREL, [Online], Available: [7 November 2006]. Jöreskog, K.G. & Sörbom, D. (2005) LISREL, Version 8.72, Scientific Software International Inc., Lincolnwood, IL. Kim, J. & Forsythe, S. (2009) Adoption of sensory enabling technology for online apparel shopping, European Journal of Marketing, vol. 43, no. 9/10, pp Kuusinen, J. & Leskinen, E. (1988) Latent Structure Analysis of Longitudinal Data on Relations Between Intellectual Abilities and School Achievement, Multivariate Behavioral Research, vol. 23, no. 1, pp Leeflang, P. (2011) Paving the way for distinguished marketing, International Journal of Research in Marketing, vol. 28, no. 2, pp MacKinnon, D.P. (2008) Introduction to Statistical Mediation Analysis, New York & London: Lawrence Erlbaum Associates. Mahajan, V., Jain, A. & Bergier, M. (1977) Parameter Estimation in Marketing Models in the Presence of Multicollinearity: An Application of Ridge Regression, Journal of Marketing Research, vol. 14, no. 4, pp Mahajan, V. & Venkatesh, R. (2000) Marketing Modeling for e-business, International Journal of Research in Marketing, vol. 17, no. 2-3, pp Martinez-López, F., Gázquez-Abad J. & Sousa, C. (2013) Structural Equation Modelling in Marketing and Business Research: Critical Issues and Practical Recommendations, European Journal of Marketing, vol. 47, no. 1, pp ISSN

15 Jenni Niemelä-Nyrhinen and Esko Leskinen Moon, J.-W. & Kim, Y.-G. (2001) Extending the TAM for a World-Wide-Web Context, Information & Management, vol. 38, no. 4, pp Muthén, L.K. & Muthén, B.O. ( ) Mplus, Version 6.0, Muthen & Muthen, Los Angeles, CA. Niemelä-Nyrhinen, J. (2009) Factors Affecting Acceptance of Mobile Content Services among Mature Consumers. University of Jyväskylä. Jyväskylä Studies in Business and Economics 72. Doctoral dissertation. Ofir, C. & Khuri, A. (1986) Multicollinearity in marketing models: Diagnostics and remedial measures, International Journal of Research in Marketing, vol. 3, no. 3, pp Pavlou, P. (2003) Consumer Acceptance of Electronic Commerce: Integrating Trust and Risk with the Technology Acceptance Model, International Journal of Electronic Commerce, vol. 7, no. 3, pp Steenkamp, J.-B. & Baumgartner, H. (2000) On the use of structural equation models for marketing modelling, International Journal of Research in Marketing, vol. 17, no. 2, pp Steenkamp, J.-B. & van Trijp, H. (1991) The use of LISREL in validating marketing constructs, International Journal of Research in Marketing, vol. 8, no. 4, pp Subhash, S. & William, J. (1981) Latent root regression: An alternate procedure for estimating parameters in the presence of multicollinearity, Journal of Marketing Research, vol. 18, no. 2, pp Teo, S., Lim, V. & Lai, R. (1999) Intrinsic and Extrinsic Motivation in Internet Usage, Omega The International Journal of Management Science, vol. 27, no. 1, pp van der Heijden, H. (2003) Factors Influencing the Usage of Websites: the Case of a Generic Portal in The Netherlands, Information & Management, vol. 40, no. 6, pp Venkatesh, V. (2000) Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation and Emotion into Technology Acceptance Model, Information Systems Research, vol. 11, no. 4, pp Venkatesh, V. & Davis, F. (2000) A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies, Management Science, vol. 46, no. 2, pp Yuan, K-H. & Chan, W. (2008) Structural equation modeling with near singular covariance matrices, Computational Statistics and Data Analysis, vol. 52, no. 10, pp Appendix A Perceived usefulness PU1 Using mobile content services makes it easier for me to use certain services. PU2 I find mobile content services useful. PU3 Using mobile content services enables me to accomplish tasks more quickly. PU4 Using mobile content services improves my performance in certain tasks. Perceived ease of use PEOU1 Learning to use mobile content services is easy for me. PEOU2 It is easy for me to become skilled at using mobile content services. PEOU3 I find mobile content services easy to use. PEOU4 I find it easy to get a mobile content service to do what I want it to do. Perceived enjoyment PE1 I have fun using mobile content services. PE2 I find using mobile content services enjoyable. PE3 Using mobile content services is pleasant. Intention to use INTEN1 I intend to use mobile content services during the forthcoming year. INTEN2 I plan to use mobile content services during the forthcoming year. INTEN3 I will try to use mobile content services during the forthcoming year ACPIL

16 The Impact of non-equidistance on Anova and Alternative Methods B L C U T G S B L Abstract: The normality assumption behind ANOVA and other parametric methods implies not only mound shape, symmetry, and zero excess kurtosis, but also that data are equidistant. This paper uses a simulation approach to explore the impact of non-equidistance on the performance of statistical methods commonly used to compare locations across several groups. These include the one-way ANOVA and its robust alternatives, the Brown-Forsythe test, and the Welch test. We show that non-equidistance does affect these methods with respect to both significance level and power, but the impact differs between the methods. In general, the ANOVA is less sensitive to non-equidistance than the other two methods are and should therefore be the primary choice when analyzing potentially non-equidistant data. Keywords: Likert-type scale; equidistance; Monte Carlo simulation; ANOVA 1. Introduction Likert items (usually referred to as Likert-type scales) are often assumed by business researchers to produce data with interval properties. The reason, in most cases, seems to be that there is a wider range of statistical methods available for data analyses if the interval assumption is valid, that is, if data are equidistant. However, research shows that subjects seldom perceive Likert-type scales as intervals, neither in general nor with respect to specific constructs. For example, it has been shown that subjects perceptions of such scales are related to the way verbal anchors are used (e.g., Lantz, 2013a; Weiters & Baumgartner, 2012), and correspondence analysis has been used to calculate the perceived non-equidistant gaps between scale points in individual studies (e.g., Bendixen & Sandler, 1995; Lee & Soutar, 2010). Hence, the assumption that Likerttype scales have interval properties seems questionable, at best. Even though the debate on whether parametric statistics can be used to analyze Likert-type data without invalidating the results goes many years back (e.g., Bradley, 1978), this paper does not explicitly aim to take a stand in that debate. Instead, we explore the sensitivity to non-equidistance of common parametric methods that applied researchers often use in analyses of Likert-type data. The results can be used to choose the most suitable method for different situations. Basic parametric test statistics, such as the omnibus one-way ANOVA, rely on assumptions of normality and homoscedasticity. Alternative parametric test procedures that compare locations, such as the Brown-Forsythe test (Brown & Forsythe, 1974) and the Welch test (Welch, 1951), are considered robust against heteroscedasticity, especially in the case of unequal sample sizes. However, they have slightly less power than the ANOVA when the homoscedasticity assumption is not violated (Tomarken & Serlin, 1986). Alternative nonparametric methods for comparing locations, such as the Mann-Whithey test (Mann & Whithey, 1947) and the Kruskal-Wallis test (Kruskal & Wallis, 1952), are robust against the violation of the normality assumption, as they only assume that data can be ranked, but they also have less power than the ANOVA when normality actually holds. The ANOVA and its alternatives have, for many years, been subject to Monte Carlo-based analyses of their robustness against violations of normality and homoscedasticity (see e.g., Harwell, Rubinstein, Hayes, & Olds, 1992, for an extensive historical review). In short, the ANOVA seems relatively robust against minor heteroscedasticity when normality holds (e.g., Glass, Peckham, & Sanders, 1972), and the Brown-Forsythe test and Welch test appear to be superior under various types of major heteroscedasticity (e.g., Tomarken & Serlin, 1986). However, violation of the normality assumption seems to be a more complex issue, since normality can be violated in several different ways, which may have different impacts on the ANOVA and its alternatives (Harwell et al., 1992). The parent distributions may, for example, be positively skewed, negatively skewed, platykurtic, leptokurtic, multimodal, and/or discrete. In some of these cases, the violation of normality seems to be of minor importance, while the difference in performance becomes substantial in other cases (e.g., Lantz, 2013b; Schmider, Ziegler, Danay, Beyer, & Buhner, 2010; Tomarken & Serlin, 1986; Bevan, Denton, & Myers, 1973). Hence, the preferred choice of statistical test procedure when normality has been violated depends a great deal on how this has occurred. ISSN ACPIL Reference this paper as: L B. The Impact of non-equidistance on Anova and Alternative Methods The Electronic Journal of Business Research Methods Volume 12 Issue (pp 16-26), available online at

17 B L Non-equidistance constitutes yet another way to violate the assumption of normality. A violation of equidistance means the researcher assumes that the distance between adjacent scale points is constant along the scale, even though it is not perceived as such by subjects. Since Likert-type scales have, in many cases, been proven to be non-equidistant, the phenomenon is obviously a potential validity issue within parametric statistics. Based on Greenacre (1984), Bendixen and Sandler (1995) showed how to use correspondence analysis to rescale data from specific Likert-type scales to interval scales and discovered that all the scales they examined were non-equidistant. The same approach has been used to determine the location of ordinal scale steps in other studies (e.g., Abratt, Bendixen, & Drop, 1999; Abratt & Russel, 1999; Bendixen, Sandler, & Cohen, 1993; Bendixen, Sandler, & Seligman, 1994; Bick, Brown, & Abratt, 2004; Heimerl, 1994; Kennedy, Riquier, & Sharp, 1996; Lee and Soutar, 2010; Yates and Firer, 1997). In all cases, the results were characterized by distinct non-equidistance. Table 1 displays the perceived location of the points for some specific five-point Likert-type scales from these studies, and the lack of equidistance is apparent. Similar results can be found in other studies where the Bendixen and Sandler (1995) approach was used. Table 1: Some non-equidistant Likert-type scales Scale 1 Scale 2 Scale 3 Scale 4 Scale 5 Scale 6 Notional Bendixen and Bendixen et Heimerl Bendixen et Bendixen et Kennedy scale value Sandler (1995) al. (1993) (1994) al. (1994) al (1994) al. (1996) The sensitivity of parametric statistics to non-equidistance seems essentially unexplored in the literature. The only exception appears to be Lantz (2013a), who used simulations to show that the preferred choice of statistical method seems to depend on the current type of non-equidistance. However, Lantz (2013a) only tested two arbitrarily chosen types of non-equidistance. The purpose of this paper is to explore the impact of actual and empirically measured non-equidistance on parametric statistical methods commonly used in applied research to compare mean values across several groups, that is, the ANOVA, Brown-Forsythe test, and Welch test. It should be noted that non-parametric alternatives, such as the Kruskal-Wallis test, are not included since they are perfectly insensitive to non-equidistance by definition. The remainder of the paper is organized as follows. In the next section, the methodology of the study is described. The simulation results are then presented and discussed. Finally, the paper concludes with the implications of the results for statistical analysis. 2. Methodology The binomial distribution is probably the discrete standard distribution that most closely resembles the normal distribution in terms of shape. It is a mound-shaped discrete distribution that exists for any number of steps, and it can have any mean value between 0 and the number of steps minus 1. It also approaches the normal distribution when the number of steps becomes large (Bowerman, O Connell, & Murphree, 2009). Hence, it is a suitable basis for an experiment in which the mound shape is important, even though the data are on a discrete scale. An experimental design with three populations and four different combinations of small (defined as five observations) and large (defined as 25 observations) sample sizes was used in the current study. Six different non-equidistant scales were used in each case (see table 1), representing actual empirically measured violations of equidistance. For each combination, the number of significant ANOVA, Brown-Forsythe, and Welch tests was compared to the number of significant tests when data was actually equidistant. Four different effect sizes (see Cohen, 1992) were used for each combination of sample size, test procedure, and scale: no, small, medium, and large effect. Furthermore, each effect size was created in two different ways, firstly, with equally spaced means, and secondly, with one extreme and two equal means (Tomarken & Serlin, 1986). Finally, all combinations were tested with symmetric data (defined as the case in which the mean is equal to the third scale point on the five-point scale), as well as with skewed data (defined as the case in which the mean is equal to the second scale point on the five-point scale). Table 2 displays the mean values for the underlying binomial distributions required to achieve the different effect sizes. et 17 ACPIL

18 Electronic Journal of Business Research Methods Volume 12 Issue Table 2: Effect sizes and population means Panel A: Equally spaced means Shape f Mean 1 Mean 2 Mean 3 Symmetric Skewed Panel B: One extreme and two equal means Shape f Mean 1 Mean 2 Mean 3 Symmetric Skewed Note that the binomial distribution with five possible values goes from 0 to 4, while the regular Likert-type scale goes from 1 to 5. In other words, the mean value, 2.12, in table 2, for example, corresponds with a Likerttype scale mean value of Table 3 displays the means and standard deviations for all six non-equidistant scales under these circumstances, as well as for the corresponding equidistant scale. All six non-equidistant scales were measured with correspondence analysis (Greenacre, 1984) based on real likert-type data from different types of research. The basic idea behind the application of correspondence analysis in this context is to conduct a principal component analysis of the contingency table that the AxB (where A is the number of steps on the scale, and B is the number of items in the study) data points constitutes, in order to measure the strength of the participants responses (see Bendixen and Sandler, 1995, for a detailed technical description of the algorithm). Table 3: Means and standard deviations for the non-equidistant scales Equi S1 S2 S3 S4 S5 S6 Symmetric Mean S.D Skewed Mean S.D For each combination of sample size, test procedure, scale, skewness, and effect size, 50,000 hypothesis tests based on simulated random numbers were conducted, where the null hypothesis that corresponds to no difference between the locations of the populations was challenged at an alpha level of 0.05 in all cases. All simulations and analytic procedures were conducted using Microsoft Excel Results In this section, the results from the simulations are presented, subdivided according to the three statistical methods under scrutiny. 3.1 ANOVA The ANOVA results are presented in tables 4a 4d. The numbers in the tables represent the proportion of significant tests out of the 50,000 tests conducted under the different conditions. Note that a higher value on the mean absolute percentage error (MAPE) indicates a higher sensitivity to non-equidistance. The overall pattern is similar in all four cases. When the effect size is 0.00, the MAPE values are generally low when at 18 ISSN

19 B L least one sample size is large. In these cases, there are very few significant differences to the equidistant case, indicating that the actual significance level is not severely affected by non-equidistance. However, when all sample sizes are small, the MAPE is considerably higher. However, there is no obvious tendency in how the probability of a type I error is affected by non-equidistance in these cases. Table 4a: ANOVA, symmetric case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.057* % 5,5, % 5,25, % ,25, % 5,5, * 0.065* 0.064* % 5,5, * * % 5,25, * * % ,25, * 0.438* * % 5,5, * 0.118* 0.116* 0.103* % 5,5, * * 0.128* 0.198* 9.2% 5,25, * % ,25,25 0,850 0,846 0,833* 0,833* 0,844 0,828* 0, % 5,5,5 0,205 0,207 0,191* 0,213* 0,215* 0,194* 0, % 5,5,25 0,416 0,420 0,436* 0,420 0,431* 0,314* 0,426* 6.1% 5,25,25 0,496 0,495 0,496 0,482* 0,495 0,438* 0, % Note: * indicates a significant difference from the equidistant case Table 4b: ANOVA, symmetric case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.056* % 5,5, % 5,25, % ,25, % 5,5, * 0.063* 0.063* % 5,5, * * % 5,25, * % ,25, * 0.433* 0.440* % 5,5, * % 5,5, * 0.122* 0.123* 0.161* 0.128* 10.6% 5,25, * 0.368* * % ,25, * 0.835* 0.843* 0.850* % 5,5, * * % 5,5, * 0.231* 0.241* 0.247* 0.325* 0.263* 10.9% 5,25, * 0.756* 0.763* 0.755* % Note: * indicates a significant difference from the equidistant case 19 ACPIL

20 Electronic Journal of Business Research Methods Volume 12 Issue Table 4c: ANOVA, skewed case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.053* 0.039* * % 5,5, % 5,25, % ,25, * * % 5,5, * 0.060* 0.044* * 0.059* 9.6% 5,5, * 0.060* 0.030* 0.046* 0.039* 0.055* 19.5% 5,25, * 0.066* 0.047* * % ,25, * * % 5,5, * 0.101* 0.072* * 0.099* 9.0% 5,5, * 0.144* 0.048* 0.098* 0.063* 0.127* 29.5% 5,25, * 0.167* 0.107* 0.143* 0.120* 0.16* 13.2% ,25, * 0.735* 0.678* * % 5,5, * 0.176* 0.124* * 0.172* 9.4% 5,5, * 0.303* 0.121* 0.224* 0.143* 0.275* 25.3% 5,25, * 0.361* 0.251* 0.327* 0.280* 0.354* 10.4% Note: * indicates a significant difference from the equidistant case Table 4d: ANOVA, skewed case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.041* * % 5,5, * % 5,25, % ,25, % 5,5, * 0.047* * % 5,5, * 0.058* * % 5,25, * * % ,25, * 0.357* 0.356* * % 5,5, * * % 5,5, * 0.115* 0.153* * 0.130* 10.4% 5,25, * * % ,25, * 0.761* 0.761* * % 5,5, * 0.192* 0.171* % 5,5, * 0.236* 0.306* * 0.272* 8.8% 5,25, * 0.638* * % Note: * indicates a significant difference from the equidistant case Non-equidistance seems to affect power more than significance. Many results-based non-equidistant scales differ significantly from their equidistant analogues. When all sample sizes are large, power is often slightly, but significantly decreased by the presence of non-equidistance. However, when at least one sample size is small, power responds erratically to non-equidistance, especially when exactly one sample size is large. It should also be noted that the ANOVA seems more sensitive to non-equidistance when data are skewed and means are equally spaced than in the other three cases (the MAPE values are about twice as high) ISSN

21 B L 3.2 Brown-Forsythe test The results regarding the Brown-Forsythe test are presented in tables 5a 5d. When the effect size is 0.00, the MAPE values are generally low when all sample sizes are large (albeit higher than when ANOVA is used to analyze data), and generally high when all sample sizes are small. When sample sizes are unequal, the MAPE values are considerably higher under the Brown-Forsythe test than under the ANOVA test. In these cases, many results differ significantly from the equidistant case. This indicates that the actual significance level is rather sensitive to non-equidistance. The MAPE values are also generally higher when data are skewed, for all combinations of sample sizes. Table 5a: Brown-Forsythe test, symmetric case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.034* * % 5,5, * % 5,25, * 0.063* 0.059* % ,25, % 5,5, * 0.040* * % 5,5, * 0.044* * 0.060* 16.7% 5,25, * 0.082* * % ,25, * 0.437* * % 5,5, * 0.079* * % 5,5, * 0.104* 0.145* 0.145* 0.195* 0.141* 24.4% 5,25, * 0.167* * 0.206* 11.6% ,25, * 0.832* * % 5,5, * 0.154* * 0.180* 16.1% 5,5, * 0.241* 0.296* 0.299* 0.378* 0.302* 27.2% 5,25, * 0.372* 0.434* 0.435* 0.508* 0.437* 12.6% Note: * indicates a significant difference from the equidistant case Table 5b: Brown-Forsythe test, symmetric case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.034* * % 5,5, * 0.053* % 5,25, * 0.061* 0.059* % ,25, % 5,5, * 0.040* * % 5,5, * * % 5,25, * * % ,25, * 0.432* 0.439* % 5,5, * 0.075* 0.090* * % 5,5, * % 5,25, * 0.306* 0.313* 0.358* % ,25, * 0.835* 0.842* 0.849* % 5,5, * 0.147* 0.165* 0.167* * 16.9% 5,5, * * % 5,25, * 0.647* 0.639* 0.649* 0.707* 0.674* 14.1% Note: * indicates a significant difference from the equidistant case 21 ACPIL

22 Electronic Journal of Business Research Methods Volume 12 Issue Table 5c: Brown-Forsythe test, skewed case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.037* 0.029* % 5,5, * % 5,25, * * % ,25, * * % 5,5, * 0.041* 0.033* % 5,5, * 0.058* 0.089* 0.075* 0.084* 0.063* 41.5% 5,25, * 0.073* % ,25, * * % 5,5, * 0.073* 0.055* % 5,5, * 0.122* 0.178* 0.168* 0.197* 0.139* 41.0% 5,25, * 0.167* * * 25.3% ,25, * 0.735* 0.675* * % 5,5, * 0.071* 0.127* 0.097* % 5,5, * 0.246* 0.315* 0.320* 0.366* 0.279* 21.6% 5,25, * 0.351* * 0.417* 0.383* 15.0% Note: * indicates a significant difference from the equidistant case Table 5d: Brown-Forsythe test, skewed case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.039* 0.031* % 5,5, * % 5,25, * * % ,25, * % 5,5, * 0.043* 0.033* % 5,5, * 0.061* 0.049* * % 5,25, * 0.084* % ,25, * 0.357* 0.354* * % 5,5, * 0.077* 0.058* % 5,5, * 0.113* 0.068* 0.086* 0.063* 0.102* 36.1% 5,25, * 0.260* * 0.285* 6.1% ,25, * 0.761* 0.760* * % 5,5, * 0.154* 0.130* 0.158* 0.120* % 5,5, * 0.213* 0.146* 0.181* 0.125* 0.203* 38.5% 5,25, * 0.587* 0.635* * 0.630* 7.3% Note: * indicates a significant difference from the equidistant case Unlike the ANOVA test, non-equidistance does not seem to affect the power of the Brown-Forsythe test more than it does the significance, even though power often decreases. Similar to the ANOVA, the exception is when exactly one sample size is large, in which case, many results differ significantly from the equidistant case. This indicates high sensitivity to non-equidistance. While the ANOVA test displays an increased sensitivity to nonequidistance when data are skewed and the means are equally spaced, the Brown-Forsythe test exhibits a similar increased sensitivity when data are skewed and two means are equal ISSN

23 B L 3.3 Welch test The results of the Welch test are presented in tables 6a 6d. When the effect size is 0.00, the MAPE values are rather low when all sample sizes are large (albeit higher than when using the ANOVA or Brown-Forsythe tests). There are only occasional cases in which sample sizes are large and the results differ significantly from the equidistant case. This indicates relatively low sensitivity to non-equidistance. However, when at least one sample size is small, the actual significance is heavily affected by non-equidistance. This is particularly true when data are skewed. In this case, almost all results differ significantly from the equidistant case. Table 6a: Welch test, symmetric case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.027* 0.048* 0.049* 0.035* % 5,5, * * 0.060* 0.074* % 5,25, * 0.064* 0.060* 0.081* % ,25, % 5,5, * 0.032* 0.055* 0.055* 0.040* % 5,5, * 0.074* 0.073* 0.126* 0.059* 26.4% 5,25, * 0.081* 0.081* 0.135* % ,25, * 0.436* * % 5,5, * 0.063* 0.105* 0.105* 0.077* % 5,5, * * 0.274* 0.139* 31.6% 5,25, * * % ,25, * 0.830* * % 5,5, * 0.124* 0.193* 0.196* 0.149* % 5,5, * 0.292* * 0.289* 29.8% 5,25, * 0.411* * 0.415* 15.8% Note: * indicates a significant difference from the equidistant case Table 6b: Welch test, symmetric case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, % 5,5, * 0.026* 0.047* 0.047* 0.033* % 5,5, * 0.058* 0.073* % 5,25, * 0.064* 0.061* 0.081* % ,25, % 5,5, * 0.030* 0.056* 0.057* % 5,5, * 0.065* 0.083* 0.077* 0.068* % 5,25, * 0.091* 0.093* 0.130* % ,25, * 0.438* * % 5,5, * 0.102* 0.105* 0.082* % 5,5, * 0.128* 0.164* 0.157* 0.102* % 5,25, * 0.280* * % ,25, * 0.842* 0.850* 0.839* % 5,5, * 0.189* 0.194* 0.163* % 5,5, * * 0.306* 0.190* 0.252* 15.1% 5,25, * 0.635* 0.646* % Note: * indicates a significant difference from the equidistant case 23 ACPIL

24 Electronic Journal of Business Research Methods Volume 12 Issue Table 6c: Welch test, skewed case with equally spaced means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, * % 5,5, * 0.014* * % 5,5, * 0.054* 0.158* 0.052* 0.053* % 5,25, * 0.063* 0.185* 0.058* 0.061* % ,25, % 5,5, * 0.016* * 0.039* 41.2% 5,5, * 0.054* 0.236* 0.081* 0.088* 0.055* 87.2% 5,25, * 0.068* 0.247* 0.090* 0.099* 0.066* 77.2% ,25, * 0.323* * % 5,5, * 0.031* 0.059* 0.048* 0.069* 39.1% 5,5, * 0.099* 0.395* 0.186* 0.205* 0.115* 73.3% 5,25, * 0.134* 0.381* 0.212* 0.232* 0.147* 45.5% ,25, * 0.733* 0.684* * % 5,5, * 0.058* 0.105* 0.083* 0.120* 38.2% 5,5, * 0.192* 0.533* 0.345* 0.375* 0.232* 43.1% 5,25, * 0.282* 0.521* 0.401* 0.420* 0.311* 22.8% Note: * indicates a significant difference from the equidistant case Table 6d: Welch test, skewed case with one extreme and two equal means f n1,n2,n3 Eq V1 V2 V3 V4 V5 V6 MAPE ,25, * % 5,5, * 0.014* * % 5,5, * 0.056* 0.157* 0.052* 0.053* % 5,25, * 0.064* 0.182* 0.059* 0.062* % ,25, % 5,5, * 0.020* * % 5,5, * 0.070* 0.136* * 31.1% 5,25, * 0.075* 0.210* 0.091* 0.098* 0.071* 38.6% ,25, * * % 5,5, * 0.077* 0.047* * % 5,5, * 0.131* 0.144* 0.078* 0.065* 0.097* 26.9% 5,25, * 0.215* 0.307* 0.268* 0.266* 0.228* 17.5% ,25, * 0.751* 0.724* * % 5,5, * 0.146* 0.108* * % 5,5, * 0.241* 0.191* 0.149* 1.076* 0.183* 38.9% 5,25, * 0.536* 0.559* * 17.6% Note: * indicates a significant difference from the equidistant case Regarding power, the Welch test performs in a similar manner to the Brown-Forsythe test. That is, the MAPE values are not notably higher than those cases in which the effect size is However, the apparent pattern is that the Welch test loses power in more cases than the other two tests when data are non-equidistant, even for large sample sizes. The MAPE values are constantly and substantially higher for the Welch test than they are for the ANOVA test, and notably higher than they are for the Brown-Forsythe test ISSN

25 B L 4. Discussion The normality assumption behind parametric methods such as the ANOVA, the Brown-Forsythe test, and the Welch test requires data to be equidistant. Through rescaling, different kinds of Likert-type data have been shown to be non-equidistant to different degrees. Hence, the common practice of using parametric methods on such data without rescaling them is doubtful. In Lantz (2013a), the effects of two different arbitrarily chosen types of non-equidistance were explored. This study contributes further to the topic by exploring the impact of several actual and empirically measured types of non-equidistance on the ANOVA, Brown-Forsythe test, and Welch test. In line with previous research (e.g., Tomarken & Serlin, 1986), the Brown-Forsythe test and the Welch test generally exhibit somewhat lower actual power than the ANOVA test, which can be seen as an indication of the reliability of the results in this study. As regards both significance level and power, all three test procedures are relatively insensitive to non-equidistance when all sample sizes are large, even though the ANOVA test seems somewhat less sensitive than the other two methods. However, there are substantial differences between the methods when at least one sample size is small, which indicates that the overall sensitivity to non-equidistance differs between the methods. One main result of this study is that the ANOVA test seems substantially less sensitive to non-equidistance than the Brown-Forsythe test and the Welch test with regard to the actual significance level. For the ANOVA test, there were essentially no significant differences between the equidistant case and the non-equidistant cases, unless all sample sizes were small. For the Brown-Forsythe test and the Welch test, many such differences were found when sample sizes were unequal. For the Welch test, there were even a few cases of differences when all sample sizes were large. Hence, as regards the actual significance level, the Welch test seems more sensitive to non-equidistance than the Brown-Forsythe test, but the Brown-Forsythe test still seems more sensitive than the ANOVA test. Another general observation is that actual power often decreases for all three methods as a result of nonequidistance, even though all sample sizes are large. This is particularly true at the medium and large effect size. When at least one sample size is small, an erratic pattern for actual power appears for all three methods, already at the small effect size, even though it is less apparent for the ANOVA test than for the other two methods. The sensitivity to non-equidistance seems to increase for all three methods when data are skewed. This is not surprising, since skewness in itself constitutes a violation of the normality assumption that all three methods rely on. However, whether a certain power level depends on equally spaced means or on one extreme mean does not seem to matter much for any of the methods. 5. Conclusion In this study, we used a simulation-based approach to explore the impact of actual and empirically measured non-equidistance on parametric statistical methods commonly used in applied research to compare mean values across several groups, that is, the ANOVA test, Brown-Forsythe test, and Welch test. The results are rather unambiguous. In general, the ANOVA test is less sensitive to non-equidistance than the Brown-Forsythe test and Welch test. Hence, unless there are other reasons for not selecting it (e.g., heteroscedasticity), the ANOVA test should be the primary choice among these three methods when analyzing Likert-type or other potentially non-equidistant data, especially if not all sample sizes are large. When potentially non-equidistant data are characterized by heteroscedasticity and all sample sizes are large, the Brown-Forsythe test and Welch test can still only be seen as a possible but not natural alternative to the ANOVA test because of their larger sensitivity to non-equidistance. In other cases, non-parametric alternatives to the ANOVA test, such as the Kruskal-Wallis test, should probably be preferred because of their total insensitivity to non-equidistance, even though they have lower power, in general. Future research in this area should explore the effects of non-equidistance further by combining the violation of equidistance with, for example, heteroscedasticity. In general, the effects of concurrent violations can produce anomalous effects not observed in separate violations (see, e.g. Zimmerman, 1998). Other types of parametric methods should also be tested for their robustness against non-equidistance ACPIL

26 Electronic Journal of Business Research Methods Volume 12 Issue References Abratt, R., Bendixen, M., & Drop, K. (1999). Ethical perceptions of South African retailers: management and sales personnel. International Journal of Retail & Distribution Management, 27, Abratt, R., & Russell, J. (1999). Relationship marketing in private banking in South Africa. International Journal of Bank Marketing, 17, Bendixen, M. T., Sandler, M., & Cohen, D. W. (1993). Environmental Issues and the Use of PVC in the Packaging Industry. Management Dynamics: Contemporary Research, 2, Bendixen, M. T., Sandler, M., & Seligman, D. (1994). Consumer Perceptions of Environmentally Friendly Products. South African Journal of Business Management, 25, Bendixen, M. T., & Sandler, M. (1995). Converting verbal scales to interval scales using correspondence analysis. Management Dynamics: Contemporary Research, 4, Bevan, M. F., Denton, J. Q., & Myers, J. L. (1974). The robustness of the F test to violations of continuity and form of treatment population. British Journal of Mathematical and Statistical Psychology, 27, Bick, G., Brown, A, B., & Abratt, R. (2004). Customer perceptions of the value delivered by retail banks in South Africa. International Journal of Bank Marketing, 22, Bowerman, B. L., O Connell, R. T., & Murphree, E. S. (2009). Business Statistics in Practice. McGraw-Hill Irwin, New York, NY. Bradley, J. V. (1978). Robustness?. British Journal of Mathematical and Statistical Psychology, 31, Brown, M. B. & Forsythe, A. B. (1974). The ANOVA and multiple comparisons for data with heterogeneous variances. Biometrics, 30, Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112, Dawes, J. (2008). Do data characteristics change according to the number of scale points used?. International Journal of Market Research, 50, Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance. Review of Educational Research, 42, Greenacre, M. J. (1984). Theory and Application of Correspondence Analysis, Academic Press, London. Harwell, M. R., Rubinstein, E. N., Hayes, W. S., & Olds, C. C. (1992). Summarizing Monte Carlo Results in Methodological Research: The One- and Two-Factor Fixed Effects ANOVA Cases. Journal of Educational Statistics, 17, Heimerl, J. (1994). A Comparative Study of Brand Loyalty between Urban Blacks and Urban Whites, Unpublished MBA Research Report, University of the Witwatersrand, Johannesburg. Kennedy, R., Riquier, C., & Sharp, B. (1996). Practical Applications of Correspondence Analysis to Categorical Data in Market Research. Journal of Targeting, Measurement and Analysis for Marketing, 5, Kruskal, W. H., & Wallis, W. A. (1952). Use of Ranks in One-Criterion Variance Analysis. Journal of the American Statistical Association, 47, Lantz, B. (2013a). Equidistance of Likert-type scales and validation of inferential methods using experiments and simulations, Electronic Journal of Business Research Methods, 10, Lantz, B. (2013b). The impact of sample non-normality on ANOVA and alternative methods, British Journal of Mathematical and Statistical Psychology, 66, Lee, J. A., & Soutar, G. N. (2010). Is Schwartz s value survey an interval scale, and does it really matter? Journal of Cross- Cultural Psychology, 41, Mann, H. B. & Whitney, D. R. (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Annals of Mathematical Statistics, 18, Schmider, E., Ziegler, M., Danay, E., Beyer, L., & Buhner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption. Methodology: European Journal of ResearchMethods for the Behavioral and Social Sciences, 6, Tomarken, A. J., & Serlin, R. C. (1986). Comparison of ANOVA Alternatives Under Variance Heterogeneity and Specific Noncentrality Structures. Psychological Bulletin, 99, Weijters, B. & Baumgartner, H. (2012). Misresponse to Reversed and Negated Items in Surveys: A Review. Journal of Marketing Research, 49, Welch, B. L. (1951). On the Comparison of Several Mean Values: An Alternative Approach. Biometrika, 38, Yates, A., & Firer, C. (1997). The Determinants of the Risk Perceptions of Investors, Investment Analysts Journal, 44, ISSN

27 SEM for Experimental Designs: An Information Systems Example Saurabh Gupta Coggin College of Business, University of North Florida, USA Abstract: IS research has matured significantly over the last three decades, leading to increasingly complex research designs as well as complex analytical techniques to analyze the data collected. Similar advances have happened in the experimental and quasi-experimental designs. Some key characteristics of these advances are: 1) use of latent variables approaches to operationalize key variables; 2) the need to understand the causal relationship between elements of the study; 3) the need to study the effects of technology as an addition to existing methods of working; and, 4) recognition that some conditions create greater change in outcomes over time. In spite of these advances in data collection and design, researchers are still confirming data collected via experiments to use ANOVA for analysis. This paper outlines an analytical technique that moves Information Systems experimental research beyond ANOVA. By combining and extending three advances in Structural Equation Modeling techniques, namely Mean and Covariance Structure analysis, Stacked Group modeling and Latent Growth modeling, the paper outlines a robust analysis technique that accommodates the above-mentioned advances in experimental design. The technique provides for an in-depth test of all model assumptions, as well as the flexibility to accommodate an increasing variety of experimental designs. A detailed example is provided to illustrate the usage of the technique in an Information Systems context. The example shows now only the accommodations needed in an information systems context, but also how this technique can be used to extract results from existing research methods that was not possible with ANOVA. The arguments presented in the paper as well as the example on how to use should provide future researchers with a guideline on how to use these techniques as well as provide a platform for how they can extend these techniques to accommodate more research method advances. Keywords: SEM, experiments, stacked group modeling, latent growth modeling, invariance, Information Systems 1. Introduction The landscape of information systems research is filled with different methods, ranging from surveys, case studies to experiments (Jenkins, 1985). To a large extent, the choice of the research method depends on the ability of the research question to answer the research question. Each of these methods has its own strength and weakness, especially as it relates to the analytical abilities associated with each. While most research methods used in business research have graduated to advanced analytical techniques (especially with the advances in Structural Equation Modeling (SEM)), analytical techniques for experimental data have been stuck in the ANOVA mindset syndrome (MacCallum, 1998) (e.g. (Sutanto et al., 2013, Gregg and Walczak, 2008)). This is especially true for Information Systems research, which relies heavily on experimental studies. The advances in analytical capabilities in other methods have provided researchers with the ability to answer more questions from a given research method. Consequently, there is a decrease in the number of empirical papers using experiments as a research method. The use of experiments as a research method in information systems have a long history (Benbasat, 1989, Jarvenpaa et al., 1985). This research design makes intuitive sense for a lot of questions asked by IS researchers as they are generally comparing a context without information technology with a context with advanced information technology or comparing two different technology usages. Such questions have been asked throughout the history of technology use, from the original Minnesota experiments (Dickson et al., 1977), to the contemporary studies (Zhang et al., 2009). Over the last 35 years IS experimental designs have matured considerably. Some of the complexity is due to the adoption of latent variable approaches to operationalize key variables in the studies. Further, there is recognition that experimental conditions may not only result in mean differences, but also differences in the strength of association between the key variables (i.e., a regression coefficient may grow stronger or weaker in the presence of one condition vs. another). For example, a recent study comparing the effectiveness of two different end-user training methods implied that the underlying cognitive latent variables might influence outcomes (in this case, it implied that paired training might improve over time) (Davis and Yi, 2004, Gupta and Bostrom, 2013). Similar examples exist in other IS domains as well (e.g. (Kearns, 2012, Hosack et al., 2012)). In spite of these changes, researchers have continued to use ANOVA/MANOVA as their dominant analytical approach; fitting their data to conform to constraints and assumptions of this technique. These traditional ISSN ACPIL Reference this paper as: Gupta S. SEM for Experimental Designs: An Information Systems Example The Electronic Journal of Business Research Methods Volume 12 Issue (pp 27-40), available online at

28 Electronic Journal of Business Research Methods Volume 12 Issue analytical tools cannot simultaneously account for the above mentioned complexities and their application results in inferences that may lack empirical validity. Given the prevalence of MANOVA, we do not have a formal section reviewing MANOVA literature. However, appropriate studies are cited where necessary. The purpose of this paper is to introduce how SEM based analysis can be extended to address some of these design challenges -- latent variables as inputs, compare non-equivalent structural models, as well as special cases like longitudinal designs by simultaneously comparing modeling variance, covariance and mean differences. While SEM does not lend itself directly to experiment design (because of imbalanced groups), in this paper we discuss the variations that makes it suitable. The rest of the paper is laid out as follows. First we compare the traditional techniques with the proposed technique of using SEM. Next, an illustration is provided as these issues are discussed. The paper ends with a set of guidelines for users and issues that have not yet been addressed. 2. Analytical Approach Comparison MANOVA/ANOVA techniques are familiar to all experimental researchers and thus, serve as a well-established benchmark. As mentioned earlier, the proposed analytical technique outlined in this paper is based on SEM. To understand the issues, benefits and usages, the two techniques need to be discussed across their purpose, assumptions and modeling process including the power to predict / model validation. This is presented the next section. 2.1 Purpose of Analytical Methods Most IS experiments are designed as input-output studies i.e. comparing dependent variable means across two or more treatments (as outlined in the reviews of IS areas such as virtual teams (Pinsonneault and Caya, 2005), computer self-efficacy (Compeau et al., 2005), or end-user training (Gupta et al., 2010)). In such cases, i.e. where the interest is in simple means test, MANOVA is a good analytical technique and SEM based technique add nothing of importance or even of value. The most important reason for using SEM is in cases where predictors beyond the treatment conditions are of interest. Called regression for explanations, in such cases, researchers want to know not only how well the predictors explain the criterion variable, but also which specific predictors are most important. In experimental setting, such techniques allow researchers interested in prediction that go beyond variance accounted for (rsquared) to specific regression weights (Maruyama, 1997). This has led some researchers to use PLS based analysis in an experimental setting, where a proxy variable is used to represent latent variables (e.g. (Cooper and Haines, 2008)). In cases of balanced designs, such use of PLS or covariance based techniques is fine and has been discussed extensively in Qureshi and Compeau (2009). However, PLS/covariance based techniques do not allow for simultaneous comparison of mean differences between dependent variables a critical requirement in an experimental study. The technique outlined in this study borrows heavily on work generally known as Means and Covariance analysis (MACS) by Ployhart et al. (2004) which allows for such analysis. Secondly, in cases where a group with advanced information technology is compared with a group without the advanced information technology, the introduction of technology also introduces a new latent variable in the model such as technology usage; this results in an unbalanced SEM design. This is a big issue for IS research because of the nature of the underlying research questions. For example, in a recent study, Gupta and Bostrom (2013) investigated technology based learning vs. traditional learning. In such cases, SEM techniques are not directly applicable since SEM based technique require structural models in all groups being compared to be the same. Some researchers have run separate PLS analysis for each of the groups (e.g. (Yoo and Alavi, 2001)) to analyze regression paths. Their analysis generally creates a separate independent variable to represent groups. The structural models are similar in this case. However, this creates issues of interpretational confounding due to possible structural invariance i.e. structural model estimates across the groups are assumed to be same. In addition, this analysis assumes a balanced structural model across groups; which might not be always true in experimental studies. To overcome the constraints outlined above, an SEM technique called stacked group approach can be used. This technique was initially proposed by Hayduk (1996). This technique allows for the existence of a latent 28 ISSN

29 Saurabh Gupta variable in one group that is absent in the other group. Overall, the SEM technique proposed in this paper does the following. First, it extends Hayduk s work to latent variables by including item level information. Second, and more importantly, it combines Hayduk s work with MACS analysis technique to provide a comprehensive analysis method for experimental studies. Finally, it is also possible to extent this technique for longitudinal experiments. 2.2 Assumptions among Analytical Methods As mentioned earlier, the dominant method in analyzing data based on experiment designs is MANOVA. MANOVA has three critical assumptions that need to be examined in the context of social science research Independence among observations, measurement invariance and normality of observations. Figure 1: Conceptual Representation of Measurement Invariance for Stacked Group Model 2.3 Independence among observations In MANOVA, the most basic, yet most serious, violation of an assumption occurs when there is lack of independence among observations. There are three reasons for such a violation: time-ordered effects, gathering information in group settings, and extraneous and unmeasured effect. Most IS experiments are done in group settings with common subject demographics across treatments. While this does increase the internal validity of the result, it could result in a lack of independence among observation. In addition, most experimental techniques assume orthogonality between the predictor variables, i.e., the treatment conditions. However, in behavioral science research, given the psychological nature of the variables, predictors are not completely orthogonal. In addition, there are no tests that detect all forms of dependence with absolute certainty. The advantage of using SEM methods is that they allow for exogenous variables to correlate with each other alleviating some of the effects of dependence. In addition, SEM does not assume orthogonality when analyzing data (Maruyama, 1997). 2.4 Latent Variables and Measurement Invariance The Information systems discipline sits at the interception of individual behavior and information technology. Thus, researchers in this area increasingly employ latent variables as a part of the measurement approach (Boudreau et al., 2001). In such an approach, latent variables data is collected using various underlying items which when loaded onto the latent variable provide information regarding the same. MANOVA based approached, however, are not designed to accommodate item level information. In such case, researcher generally start out appropriately by first examining the properties of the latent variables through a confirmatory factor analysis (CFA). Then, however, they fall back to some aggregate form of the measure (e.g., its algebraic mean or average) to test the hypotheses usually in an ANOVA framework. Doing so is inappropriate because the measure is assumed to contain no error. In addition, IS researchers have assumed, in the vast majority of studies, that a variable will have the same conceptual meaning to one group receiving one 29 ACPIL

30 Electronic Journal of Business Research Methods Volume 12 Issue level of the treatment as it does to the other groups receiving other levels of the treatment; i.e., the measures are invariant across conditions (Biros et al., 2002, Yi and Davis, 2003, Davis and Yi, 2004, Piccoli and Ives, 2003, Yoo and Alavi, 2001). An SEM approach, in contrast, includes measurement error because variables (i.e., latent variables) are operationalized using item-level information rather than an average or summed value. The related assumption in MANOVA is the equivalence of covariance matrices across the groups (James et al., 1982). A violation of this assumption might lead to overestimation of estimates and thus, resulting in a Type 1 error (although more recently researchers have suggested that a violation of this assumption has minimal impact if the groups are of approximately equal size (Horn and McArdle, 1992)). In well-controlled IS experiments, the groups sizes are generally not that different and this is not a problem in case of discrete measurements. However, in case of latent variables, other elements of invariance need to be tested. When measures are not invariant, any conclusions based on the findings can lead to misleading or even false conclusions (Taylor and Todd, 1995, Williams et al., 2009, Vandenberg and Lance, 2000, Ployhart and Oswald, 2004). SEM methods allow for testing of each of these concerns. A review of the literature points to three critical tests that are relevant to the current scenario: configural invariance, metric invariance and factor variance invariance (Schmitt and Kuljanin, 2008). Configural invariance requires a demonstration that the same factors and pattern of factor loadings explains the variance-covariance matrices associated with the groups responses (see Figure 1). If configural invariance is not supported, then comparisons between groups on the measures is not possible because this literally means that there is no conceptual equivalence of the constructs (Horn and McArdle, 1992). This is generally not a problem in IS research as most researchers tend to use wellvalidated existing instruments. However, the concern is valid when using new measures. The second test is metric invariance, also called a test of strong factorial invariance (Horn et al. 1992). It tests a null hypothesis that factor loadings for like items are invariant across time/group, i.e., a test of tau equivalence across groups. It evaluates whether the groups are calibrating their responses to the underlying latent variable to the same degree (e.g., a response of 3 means neither agree-nor disagree in all groups). At least partial metric invariance must be established in order for subsequent tests to be meaningful. The assumption of homosedacity or parallel equivalence among constructs of different groups is usually not a problem, except in cases where the means of dependent variables are correlated with the variance across groups. This assumption is addressed in the factor variance invariance test. Metric and factor invariance analysis is done by looking at CHI T three tests are summarized in Figure 1 and are further explained in the illustrative example later. 2.5 Normality The last assumption for covariance based analysis relates to normality of variables. In the strictest sense, the assumption is that all variables are multivariate normal. Multivariate normality assumes that joint effect of two variables is normally distributed. Even though this assumption underlies both of the mentioned techniques, there is no direct test for multivariate normality. Violations of this assumption have little impact with larger sample size. For SEM based methods, a departure from normality (as shown in a recent study based on Monte Carlo simulations) does reduce the ability to detect differences between groups (Qureshi and Compeau, 2009). 2.6 Modeling Process The sequence of steps required to setup a MANOVA based analysis is well known. MANOVA based analysis relies on comparison of mean vectors across two or more groups. Thus, it requires equity of means matrix dimensions. Hayduck (1996) extended the basic multi-group model approach (the term used to convey the comparison between groups in SEM) to accommodate the experimental design imbalance. While Hayduck s technique did not focus on latent variables, this paper extends the approach to incorporate latent variables. This approach requires the following modifications of the covariance matrix, the means vector, beta matrix, phi mate, and theta-epsilon matrix. Covariance matrices for all groups should typically consist of an equal number of items. However, in the cases where a condition is not present (and thus, a latent variable does not exist), proxy variables need to be created with dummy codes. This helps equate the number of items across groups. The correlation of these dummy items with observed items should be fixed to 0, and the variance of the items should be fixed to 1. The error of 30 ISSN

31 Saurabh Gupta the item is fixed to 0. This prevents the dummy variables from impacting other variables in the model, but creates a proxy latent variable in the overall model to deal with the unbalanced conditions. The variance of the latent constructs in cases where they do not really exist should be fixed to 1. Their factor loadings (lamdas) should also be fixed to 1. This allows for model identification in an SEM analysis. Beta values of the causal path for the dummy latent variables are fixed to 0 in groups without these latent variables. This prevents the dummy variable from having any causal effect on the endogenous variables. In groups where the latent variables exist, they should be freely estimated. Latent means (alpha values in the case of the example) for the dummy variables should be fixed to 0 in groups without these variables. Latent means for the endogenous variables are freely estimated, if group mean differences are expected. Mean differences are usually the case in experimental designs. All other variables should be constrained to be equal across groups. Figure 2: Conceptual Representation of a Latent Variable Stacked Group Model Figure 2 summarizes these manipulations for a two-group, three latent variable analyses. The conceptual model could be extended for more groups/latent variables. It is important to note that it is not recommended restricting the values of the measurement model based on an invariance test. The model parameters discussed in the above sections can used as starting estimates to help with model identification. Considerable literature exists in this area to support this position of not identifying measurement model and structural model separately (Anderson and Gerbing, 1988). 2.7 Assessing Overall Fit In MANOVA, the focus is on seeing if there are mean differences between dependent variables across groups. Thus, the criteria assess the differences across dimensions of the dependent variables. There seems to be an agreement that either Pillai s criterion or Wilks lambda best meets the needs, although evidence suggests that Pillai s criterion is more robust and should be used if sample size decreases, unequal cell sizes appear, or homogeneity of covariance is violated (Hair, 1998). SEM based analysis, on the other hand, tend to focus on overall model fit. The most basic chi-square goodness of fit test, although valuable, is limited because it is a direct function to sample size. A lot of other fit indexes exist and can be generally divided across two categories: Absolute and Relative. Absolute indexes address the question: Is the residual or unexplained variance remaining after model fitting acceptable? These indexes include those that use the function that is minimized. Chi-squared fit index falls in this category. Other fit indexes that are popularly reported goodness of fit index and the root mean residual. Relative indexes address the question: How well does a particular model do in explaining a set of observed data compared to other possible models? Thus, the model is viewed as falling along a continuum between the worst possible fitting 31 ACPIL