Using Bifactor-Exploratory Structural Equation Modeling to Test for a Continuum. Structure of Motivation. Joshua L. Howard and Marylène Gagné

Transcription

1 Running head: BIFACTOR-ESEM AND MOTIVATION 1 Using Bifactor-Exploratory Structural Equation Modeling to Test for a Continuum Structure of Motivation Joshua L. Howard and Marylène Gagné University of Western Australia Alexandre J.S. Morin Institute for Positive Psychology and Education, Australian Catholic University Jacques Forest Université du Québec à Montréal This is a prepublication version of a manuscript accepted on 23 March 2016 for publication in the Journal of Management. Citation: Howard, J., Gagné, M., Morin, A.J.S., Wang, Z.N., & Forest, J. (Accepted, 23 March 2016). Using bifactor exploratory structural equation modeling to test for a continuum structure of motivation. Journal of Management. Acknowledgements: Preparation of this article was facilitated through funds from the Australian Postgraduate Award scholarship awarded to the first author, as well as grants from the Fonds Québécois de la Recherche sur la Société et la Culture and the Social Sciences and Humanities Research Council of Canada awarded to the second and fourth author, a grant from the Australian Research Council (LP ) awarded to the third author, and a grant from the Fondation de l Ordre des CRHA awarded to the fourth author. Corresponding author: Joshua L. Howard, School of Psychology, University of Western Australia, 35 Stirling Hwy, Crawley 6009 WA, Australia. joshua.howard@uwa.edu.au

2 BIFACTOR-ESEM AND MOTIVATION 2 Abstract This paper explores the nature of workplace motivation by testing the continuum structure of motivation proposed by self-determination theory (SDT) through the application of relatively new and advanced methodological techniques. Specifically we demonstrate the usefulness of the overarching bifactor exploratory structural equation modeling (B-ESEM) framework in organizational psychology and discuss implications of such models over more traditional confirmatory factor analyses. This framework is applied to responses obtained from 1124 Canadian employees who completed a multidimensional measure of workplace motivation. The results support a continuum of self-regulation but furthermore indicate the importance of accounting for both quality of motivation in addition to its global quantity. Indeed, the results showed that specific types of motivation explained variance in covariates over and above the variance already explained by the global quantity of self-determination. The current study further demonstrates the limitation of the commonly used relative autonomy index and offers alternate conceptualizations of human motivation. Keywords: work motivation, self-determination theory, continuum, bifactor exploratory structural equation modeling (B-ESEM)

3 BIFACTOR-ESEM AND MOTIVATION 3 USING BIFACTOR-EXPLORATORY STRUCTURAL EQUATION MODELING TO TEST FOR A CONTINUUM STRUCTURE OF MOTIVATION Modern psychological science is progressing to a level of theoretical complexity which necessitates the use of equally sophisticated methodological and statistical tools. This observation calls for substantive-methodological synergy (Marsh & Hau, 2007). Substantivemethodological synergies are joint ventures in which methodological advances are applied to substantively important areas of research in order to help provide more precise, or refined, answers to complex research questions. The current paper is anchored into such a substantive-methodological synergy framework and aims to: (1) Test the continuum structure of motivation proposed by self-determination theory (SDT; Deci & Ryan, 1985), and (2) demonstrate the usefulness of the overarching bifactor exploratory structural equation modeling (B-ESEM) framework in organizational psychology. We start this paper by reviewing key substantive issues related to the SDT continuum hypothesis of motivation, and then present in greater details the bifactor-esem psychometric framework. SUBSTANTIVE ISSUES: THE CONTINUUM STRUCTURE OF MOTIVATION Self-Determination Theory of Human Motivation SDT (Deci & Ryan, 1985, 2000; Ryan & Deci, 2000) proposes that individuals experience autonomous motivation when their reasons for engaging in behaviors are volitional, and experience controlled motivation when their reasons for engaging in the behaviors are pressured either internally or externally. Each of these two forms of motivation can be characterized by different types of motivation, expected to form a continuum. At one extreme, the most autonomous form of motivation is intrinsic motivation. Intrinsic motivation occurs when individuals derive a sense of enjoyment and satisfaction from the enactment of the behavior itself. At the other extreme, the most controlled form of motivation is external regulation, which occurs when individuals engage in an activity for purely instrumental

4 BIFACTOR-ESEM AND MOTIVATION 4 reasons, such as to obtain rewards or avoid punishment. Between these extremes, introjected regulation happens when a person engages in behavior to reduce negative self-related feelings (e.g., shame, guilt), or to experience positive self-related feelings (e.g., pride), and identified regulation occurs when the outcome of the behavior is personally meaningful. Introjected regulation can be exemplified by employees who work late to maximize their performance and feel better about themselves, whereas identified regulation would be exemplified by employees who stay late to finish work that they perceive to be important to the organization. In the work domain, external regulation can be further subdivided according to whether the source of the external pressure to enact the target behavior is material or social (Gagné et al., 2015). External material regulation stems from tangible rewards and punishments, such as monetary benefits and job security. In contrast, external social regulation is related to social rewards and punishments, such as approval and criticism from others. Although not initially covered in SDT, others have noted that it is also important to assess amotivation (Pelletier, Fortier, Vallerand, Tuson, Brière, & Blais, 1995; Vallerand, Pelletier, Blais, Brière, Senécal, & Vallières, 1993), referring to an absence of willingness to exert effort, in order to cover scenarios where people have no reason or willingness to put any effort into an activity. Indeed, early SDT-based motivation instruments were limited in that they implicitly assumed that everyone would have some reason to embark on a targeted course of action, failing to explicitly assess lack of motivation. In the current study, we rely on the Multidimensional Work Motivation Scale (MWMS; Gagné et al., 2015), a newly developed scale designed to measure these distinct types of motivation in the work domain. Previous CFA has supported the multidimensional structure of the MWMS in seven different languages (Gagné et al., 2015). Here, we use a subsample from Gagné et al. (2015) and use bifactor-esem to test SDT s hypothesis that motivation types follow an underlying continuum. Motivation Types on a Continuum

5 BIFACTOR-ESEM AND MOTIVATION 5 A key aspect of SDT s conceptualization of motivation is that motivation types are ordered along a continuum depicting the degree of relative autonomy, or self-determination (Ryan & Deci, 2000). SDT suggests that the qualitatively different motivation types also differ quantitatively along a single continuum of self-determination. The continuum hypothesis has typically been examined through an inspection of the correlations between the motivation types (Ryan & Connell, 1989; see Guay, Morin, Litalien, Valois, & Vallerand, 2015 for a review in the education domain). For instance, Ryan and Connell (1989) tested whether the correlations between motivation types followed a simplex structure. The simplex structure refers to a correlation pattern showing that adjacent motivation types correlate more strongly and positively with one another than more distal motivation types (which should correlate negatively). For example, intrinsic motivation should correlate more strongly with identified regulation than with introjected or external regulations. Although amotivation has not traditionally been taken into account in tests of the SDT continuum, a case has been made for this factor to fall at the lowest point of the continuum (Cox, Ullrich-French, Madonia, &Witty, 2011; Guay, Ratelle, Roy, & Litalien, 2010; Stevenson & Lochbaum, 2008). Some past research has supported the continuum hypothesis (e.g., Li, 1999; Li & Harmer, 1996; Ryan & Connell, 1989), while other research has not, especially when using more advanced statistical techniques (e.g., Chemolli & Gagné, 2014; Guay et al., 2015; Wininger, 2007). For example, Chemolli and Gagné (2014) argued that if the continuum hypothesis truly represented the structure of motivation, CFA should support the adequacy of a single factor model, and loadings on this single factor would range from negative for the least self-determined forms of motivation to positive for the most self-determined forms of motivations. They used Rasch analysis, a stringent statistical test specifically developed to evaluate continuum structures (Rasch, 1960), to test whether there was a continuum structure underlying the items of the MWMS and the Academic Motivation Scale (Vallerand et al.,

6 BIFACTOR-ESEM AND MOTIVATION ), and found no support for it. These results concur with past research that has consistently supported multidimensional models over unidimensional ones (e.g., Gagné et al., 2010, 2015; Li, 1999; Mallett, Kawabata, Newcombe, Otero-Forero, & Jackson, 2007). A different test of this hypothesis was conducted by Guay et al. (2015), who investigated the continuum assumption of SDT using exploratory structural equation modeling (ESEM). Guay et al. (2015) further noted that in ESEM, the SDT continuum could be expressed in two distinct and complementary manners. In line with previous studies, the continuum hypothesis would be supported by the observation of the expected simplex pattern at the level of factor correlations. Because ESEM tends to result in more exact estimates of these factor correlations (see the methodological section below), the simplex pattern could be expected to be clearer using ESEM than CFA. Moreover, support for the continuum hypothesis could also come from the observation of larger cross-loadings between adjacent subscales than between more theoretically distal subscales. Testing these propositions with the Academic Motivation Scale (Vallerand et al., 1992), Guay et al. (2015) found that the data fit the ESEM representation better than the CFA model, and that factor correlations were more in line with the expected SDT continuum with ESEM than with CFA. However, even though the simplex pattern was cleaner with ESEM, the results still showed many digressions, and did not fully replicate across samples. Results also revealed cross-loadings somewhat in line with the SDT continuum (i.e., stronger between adjacent factors), though they remained generally small. Overall, these results partially supported the continuum hypothesis (for a similar ESEM representation of doctoral students academic motivation, see Litalien, Guay, & Morin, 2015). In the present study, we extend these previous studies by combining into a single bifactor-esem framework the tests conducted separately by Chemolli and Gagné (2014) and Guay et al. (2015) to a new measure of work motivation (Gagné et al., 2015). METHODOLOGICAL ISSUES: INTRODUCTION TO BIFACTOR-ESEM

7 BIFACTOR-ESEM AND MOTIVATION 7 CFA has become the ubiquitous test of factor structures in psychological measurement. Measures which fail to meet designated goodness-of-fit standards are deemed of little worth in the eyes of researchers and reviewers alike. Despite this, psychological scales which consistently meet these rather arbitrary benchmarks are few. This has caused many to question the necessity of the Independent Cluster Model (ICM) constraints inherent in CFA, in which cross-loadings between items and non-target factors are assumed to be exactly zero. Undoubtedly, CFA has had a positive influence on psychological measurement by encouraging researchers to develop more a priori, precise, and parsimonious models (e.g., Morin, Marsh, & Nagengast, 2013). Through its integration into the Structural Equation Modeling (SEM) framework, CFA has provided ways to test how the data fits with a priori expectations, to systematically investigate the degree to which a measurement or predictive model is invariant across meaningful subgroups of participants, and to assess relations between constructs corrected for measurement errors. These developments have been so substantial that CFA has completely superseded traditional Exploratory Factor Analyses (EFA) for all but the most preliminary tests of factor structure. Over and above the intuitive appeal of clearly defined concepts, measured by a small number of items perfectly designed to assess a single construct, has come a recent recognition that the ideals pursued through a CFA approach are often impossible to achieve in applied research (e.g., Marsh et al., 2009, 2010). Nowadays, many researchers recognize that the ICM constraints inherent in CFA are oftentimes not appropriate given the nature of the data (for a review, see Marsh, Morin, Parker, & Kaur, 2014). Specifically, the fallible and imperfect nature of typical psychometric indicators which typically can be expected to tap into more than one source of true score variance calls into question the usefulness of CFA (e.g., Morin, Arens, & Marsh, 2016a). As noted by Morin et al. (2016a), indicators are rarely, if ever, perfectly and uniquely related to a single construct, and will almost always display

8 BIFACTOR-ESEM AND MOTIVATION 8 some degree of construct-relevant association with non-target factors assessing conceptuallyrelated (such as interrelated motivation types, see Guay et al., 2015; Litalien et al., 2015) or hierarchically-ordered constructs (such as when motivation types are expected to assess an overarching motivation continuum, see Chemolli & Gagné, 2014). CFA versus ESEM Relying on an EFA measurement model allowing for the estimation of cross-loadings is typically required as a test of construct-relevant multidimensionality related to the assessment of conceptually related constructs (Morin et al., 2016a; Morin, Arens, Tran, & Caci, 2016b). However, EFA has often been criticized for being data driven and exploratory in nature (e.g., Kahn, 2006; Preacher & MacCallum, 2003). This implies an approach in which multiple models are compared and the model producing the best correspondence to the data (based on a variety of criteria) is retained for further use. In contrast, CFA is generally assumed to be theory-driven and models are assessed in and of themselves using a variety of goodness-of-fit indices. This view has led to the erroneous assumption that EFA is a datadriven procedure unsuited to confirmatory studies. According to Morin et al. (2013: 396): This perception is reinforced by the erroneous semantically-based assumption that EFA is strictly an exploratory method that should only be used when the researcher has no a priori assumption regarding factor structure and that confirmatory methods are better in studies based on a priori hypotheses regarding factor structure. This assumption still serves to camouflage the fact that the critical difference between EFA and CFA is that all crossloadings are freely estimated in EFA. Due to this free estimation of all cross-loadings, EFA is clearly more naturally suited to exploration than CFA. However, statistically, nothing precludes the use of EFA for confirmatory purposes, except perhaps the fact that most of the advances associated with CFA/SEM were not, until recently, available with EFA. The recent development of ESEM (Asparouhov & Muthén, 2009; Marsh et al., 2014;

9 BIFACTOR-ESEM AND MOTIVATION 9 Morin et al., 2013) provides a promising way to circumvent restrictive ICM assumptions. ESEM provides an overarching framework allowing for the combination of CFA, EFA, and SEM into a single model. ESEM thus incorporates the benefits from each technique into a single analytic framework where factors defined according to ICM assumptions can cohabitate with EFA factors incorporating cross-loadings. A frequent misunderstanding about EFA/ESEM is that the inclusion of cross-loadings is likely to change, or taint, the meaning of the latent factors that are estimated. This flawed criticism neglects the fact that EFA/ESEM corresponds to a reflective measurement model where the factors are assumed to influence the items, rather than the opposite. A perhaps more critical issue is whether the factor itself is adequately captured, from a psychometric perspective, as being primarily reflected in its a priori indicators. Indeed, whenever results show large and hard to explain cross-loadings suggesting that some specific factors are mainly reflected in unexpected items rather than in their a priori items, then alternatives models should be explored. As noted by Morin et al. (2016a: ): Small cross-loadings should be seen as reflecting the influence of the factor on the construct-relevant part of the indicators, rather than the indicators having an impact on the nature of the factor itself. It should be kept in mind that this interpretation applies to relatively small cross-loadings that are in line with theoretical expectations, whereas any model showing large and unexplainable cross-loadings or cross-loadings larger than target loadings should be re-examined. Forcing cross-loadings to be exactly zero involves ignoring some potentially true influence of a factor (such as stress) on indicators presenting some residual association with these factors (such as insomnia) over and above their association with their main a priori factor (such as burnout). In fact, even when large cross-loadings suggest a problem in the model, forcing them to be zero simply hides sources of misspecification that will in turn be

10 BIFACTOR-ESEM AND MOTIVATION 10 expressed as model misfit leading to an examination of model modification indices to locate the source of misfit. An advantage of EFA/ESEM is that it allows for the simultaneous consideration of all cross-loadings in a single step, whereas modification indices are calculated based on the inclusion of a single cross-loading at a time (Morin & Maïano, 2011). It could be argued that ignoring these associations between items and non-target constructs simply results in reduced goodness-of-fit indices and that typical interpretation guidelines for goodness-of-fit indices are just too stringent for complex measurement models (e.g., Marsh, Hau, & Wen, 2004). However, a clear demonstration that cross-loadings do not taint the meaning of the latent factors comes from simulation studies showing that EFA/ESEM tends to provide more exact estimates of true population values for factor correlations when cross-loadings (even small ones) are present in the population model, and to remain unbiased when the population model corresponds to ICM-CFA (Asparouhov & Muthén, 2009; Marsh, Lüdtke, Nagengast, Morin, & Von Davier, 2013; Morin et al., 2016a; Sass & Schmitt, 2010; Schmitt & Sass, 2011; for a more extensive discussion, see Asparouhov, Muthén, & Morin, 2015). In turn, biased CFA estimates of factor correlations likely affect the discriminant validity of the factors by creating artificial multicollinearity in subsequent analyses where these factors are used in prediction. Another legitimate concern about EFA/ESEM is the issue of rotational indeterminacy, meaning that the parameter estimates from any EFA/ESEM model will vary as a function of the rotation procedure that is retained (e.g., Morin & Maïano, 2011; Sass & Schmitt, 2010; Schmitt & Sass, 2011). More precisely, EFA/ESEM models based on different rotations procedures designed to reduce either cross-loadings or factor correlations to various degrees will converge on alternative solutions that have equivalent covariance implications (i.e., data fitting different models equivalently). In practice, alternative forms of oblique rotations tend to provide nearly identical, or at least equivalent in terms of substantive interpretations,

11 BIFACTOR-ESEM AND MOTIVATION 11 solutions as demonstrated in simulation studies by Sass and Schmitt (2010); and Schmitt and Sass (2011), and real data examples by Morin and Maïano (2011) and Morin et al. (2013). However, users should remain aware of this issue and, whenever they decide to use empirical (atheoretical) rotation procedures, should always conduct at least a preliminary exploration of alternative rotations to verify the stability of results. Recent recommendations suggest that this issue can be solved in confirmatory applications of EFA/ESEM relying on target rotation whereby the rotation is guided by a priori expectations regarding the expected factor structure (Marsh et al., 2014; Morin et al., 2016a). The development of target rotation (in which all cross-loadings are freely estimated but targeted to be as close to zero as possible) makes it possible to use a fully confirmatory approach to the specification of EFA/ESEM factors (Asparouhov & Muthén, 2009; Browne, 2001). Indeed, the most common use of ESEM so far has been the testing of theoretically established models in which the number and content of specified latent factors was a priori defined (Marsh et al., 2014). Bifactor-ESEM As noted above, a second source of construct relevant multidimensionality is related to the assessment of hierarchically-ordered constructs (such as an overarching motivation continuum). This possibility has typically been investigated using higher-order factor models, which directly test the hypothesis that the various factors can combine into one or many higher-order factors. However, higher-order factor models rely on highly restrictive implicit assumptions that may not hold in practice and may explain why they often fail to meet minimal requirements of adequate fit (Gignac, 2008; Morin et al., 2016a; Reise, 2012). More precisely, higher-order models assume that the association between items and the higherorder factor is fully mediated by the first-order factors (McAbee, Oswald, & Connelly, 2014), so that the higher-order factor does not in itself explain any unique variance over and above that already explained by the first-order factors. For this reason, the first-order factors in a

12 BIFACTOR-ESEM AND MOTIVATION 12 higher-order model reflect a combination of the variance explained by the higher-order factor and of the variance uniquely attributable to each first-order factor (Morin et al., 2016b). More importantly, because the relation between the higher-order factor and the item is mediated through the first-order factor, this relation is captured by the product of the loading of the item on a first-order factor, and the loading of this first-order factor on the higher-order factor: A constant for all items associated with a single first-order factor. Similarly, the relations between the items and the disturbances of the first-order factors (reflecting the variance uniquely attributable to the first-order factor) is also indirect, and reflected by the product of the loadings of the items on their first-order factor with a constant for all items associated with a single first-order factor. Because of this characteristic, the ratio of variance attributed to the global factor versus uniquely attributed to the first-order factor is constant for all items associated with a single first-order factor (Gignac, 2008; Morin et al., 2016b; Reise, 2012). An alternative, and far more flexible, way to examine whether the presence of a single global SDT factor underlying answers to motivation questionnaires involves the use of a bifactor representation, in which all items are used to define their respective motivation subscales while also being used to directly define a global SDT motivation factor that represents the continuum (Reise, 2012). Bifactor models have existed for decades (Holzinger & Swineford, 1937), and are well known in research on intelligence (Gignac, 2008) or personality (McAbee, Oswald, & Connelly, 2014). In comparison to higher-order models, bifactor models present none of these redundancies or restrictions, and provide a way to explicitly separate the variance attributable to specific factors from the variance attributable to the global general factor, while allowing for the estimation of direct relations between the items and both the specific and global factors. More precisely, bifactor models assume that the covariance among a set of n items can be explained by a set of f orthogonal factors including one Global (G) factor and f-1 Specific (S) factors. In bifactor-cfa models, each

13 BIFACTOR-ESEM AND MOTIVATION 13 item is used to define the G-factor and one of the S-factors. Bifactor models thus partition covariance into a G-factor underlying all items, and f-1 S-factors corresponding to the covariance not explained by the G-factor. This clean partitioning is made possible by the orthogonality of the factors, which forces all of the variance shared among all items to be absorbed into the G-factor, and the S-factors to represent what is shared among a specific subset of items but not the others. Interestingly, higher-order models form restricted nested versions of bifactor models. While the above mentioned proportionality constraints implicit in higher-order models introduce some parsimony to the model, they are unlikely to hold in most research settings (Reise, 2012; Yung, Thissen, & McLeod, 1999) or to make sense theoretically (Gignac, 2016), thus positioning bifactor models as the more robust modeling procedure. Jennrich and Bentler (2011) showed that while bifactor models were able to properly recover true higherorder factor structures, higher-order factor models could not always properly recover true bifactor structures. Bifactor models should thus be preferred over higher-order models unless strong theoretical reasons are present to support the need to model the relations between the indicators and the global factors as indirect, and the presence of the implicit proportionality constraints (for a more extensive discussion of these issues, see Gignac, 2016). Whenever a single instrument is expected to incorporate both conceptually-adjacent constructs and hierarchically-ordered constructs, it becomes important to rely on a model that allows for the incorporation of both cross-loadings (i.e., EFA) and global factors (i.e. bifactor). Indeed, research has shown that unmodeled cross-loadings tend to result in inflated estimates of the global factor in bifactor-cfa, and that an unmodeled global factor tends to result in inflated cross-loadings in EFA (e.g., Morin et al., 2016a; Murray & Johnson, 2013). Recent development of bifactor target rotation for EFA makes it possible to incorporate bifactor modeling to the ESEM framework (Reise, 2012; Reise, Moore, & Maydeu-Olivares,

14 BIFACTOR-ESEM AND MOTIVATION ). The resulting bifactor-esem method offers the most detailed and flexible models possible, more so than either EFA or CFA/SEM alone, and can now be implemented while relying on a confirmatory bifactor target rotation approach (Morin et al., 2016a, 2016b). The Present Study In the present study, we conducted an integrated test of SDT s continuum hypothesis of motivation combining the perspectives of: (1) Guay et al. (2015; also see Litalien et al., 2015), which showed that taking into account sources of construct-relevant psychometric multidimensionality related to the assessment of conceptually-related constructs was necessary to obtain a clearer representation of the continuum structure of motivation; and (2) Chemolli and Gagné (2014), who argued that the strongest evidence in favor of the SDT continuum hypothesis should come from the demonstration that all motivation items contribute to the assessment of a single overarching self-determination factor (i.e., from the observation of another source of construct-relevant psychometric multidimensionality related to the assessment of hierarchically-ordered constructs). To do so, we rely on the bifactor- ESEM framework proposed by Morin et al. (2016a, 2016b), which allows for the simultaneous consideration of these two perspectives into a single model. The bifactor component of this framework directly tests whether the items measuring the different types of motivation load onto a single factor with loadings ranging from negative to positive according to the expected position of the items along the SDT continuum (aligned with Chemolli and Gagné s perspective), while allowing the estimation of specific factors for each motivation type. Essentially, should this hypothesis be supported, this global factor is expected to provide a global estimate of the overall quantity of self-determined motivation characterizing individual employees, whereas the resulting specific factors would reflect the more specific quality, or flavor, of employees motivational profiles. More precisely, because of the inherent orthogonality of bifactor models, employees overall

15 BIFACTOR-ESEM AND MOTIVATION 15 amount (quantity) of self-determined motivation will be reflected in the G-factor, whereas the specific features (quality) of employees motivational profiles left unexplained by this global amount of self-determination will be reflected in the S-factors (e.g., pleasure, guilt, pressure). The ESEM component of this framework allows us to incorporate the presence of a second layer of continuity in motivation ratings expressed through the estimation of cross-loadings between motivation factors (e.g., Morin et al., 2016a), as advocated by Guay et al. (2015). Thus, based on current theory suggesting the existence of an overarching continuum of motivation (Chemolli & Gagné, 2014) and research suggesting the importance of controlling for cross-loadings between motivation factors in order to obtain a proper depiction of the underlying structure of motivation measures (Guay et al., 2015; Litalien et al., 2015), we expect the bifactor-esem model to provide the most adequate representation of employees answers to the MWMS (Gagné et al., 2015). However, following Morin et al. (2016a, 2016b) recommendations regarding the application of the bifactor-esem framework for the identification of the sources of construct-relevant multidimensionality present in complex psychometric measures, as well as basic principles of model testing (e.g., Bollen, 1989), we contrast this a priori bifactor-esem representation with more parsimonious alternative models including either none (CFA) or only-one (ESEM, Bifactor-CFA) of these likely sources of multidimensionality. These four alternative models are presented in Figure Insert Figure 1 about here To establish the criterion-related validity of the resulting global (G) and specific (S) motivation factors, we also test the extent to which they are related to a series of covariates that occupy a core position in SDT theorization both generically (i.e., the satisfaction of the needs for autonomy, competence and relatedness) and in the work context (i.e., employees levels of affective and continuance commitment to the organization). SDT proposes that the

16 BIFACTOR-ESEM AND MOTIVATION 16 satisfaction of basic psychological needs for autonomy (a sense of volition), competence (the experience of mastery) and relatedness (feeling connected to others) should promote autonomous over controlled types of motivation (Deci & Ryan, 2000; Gagné & Deci, 2005). Research has found strong support for this proposition both in general and organizational psychology (Deci & Ryan, 2000, 2008; Gagné et al., 2015). Past research has also found rather robust relations between different types of motivation and distinct commitment mindsets (Gagné, Chemollli, Forest, & Koestner, 2008; Gagné et al., 2015). In particular, affective organizational commitment (emotional attachment to the organization; Meyer, Allen, & Smith, 1997) has been positively related to autonomous (identified and intrinsic) motivation (Gagné et al., 2008), while continuance commitment (staying in the organization because of the perceived cost of leaving and lack of alternatives; Meyer et al., 1997) has been positively related to external regulation and negatively related to more internalized regulations (Battistelli, Galletta, Portoghese, & Vandenberghe, 2013; Gagné, Chemolli, Forest, & Koestner, 2008; Vandenberghe & Panaccio, 2012). In order to more precisely assess the criterion-related validity of the motivational factors, we also systematically contrast models in which only the G-factor (i.e., reflecting the overall quantity of self-determined motivation) is allowed to predict the covariates, with models in which the S-factors (i.e., reflecting the specific quality of motivation) are also allowed to predict the covariates. These comparisons systematically test the added value (in terms of percentage of explained variance in the covariates) that is afforded by the simultaneous consideration of both motivation quantity and quality. METHOD Participants and Procedure This study used archival data collected between 2008 and 2012 that has been previously used to validate the MWMS (Gagné et al., 2015). The current sample includes 1124 full time

17 BIFACTOR-ESEM AND MOTIVATION 17 Canadian employees from a range of organizations and industries. Content of the surveys varied within data sets in terms of covariates and demographics, but all participants completed the same 19 items forming the MWMS. Employees completed confidential surveys voluntarily on an online platform or in paper format on their work premises. Additional details are provided in Gagné et al. (2015). Measures The MWMS (Gagné et al., 2015) includes 19 items assessing six distinct motivation types. Each item is a response to the stem, Why do you or would you put efforts into your current job? along a 1 (not at all) to 7 (completely) point Likert scale. Example items include, I don t know why I m doing this job, it s pointless work (Amotivation; α =.78 in the current study), To get others approval (e.g., supervisor, colleagues, family, clients...) (External regulation social; α =.77), Because others will reward me financially only if I put enough effort in my job (e.g., employer, supervisor...) (External regulation material; α =.63), Because otherwise I will feel ashamed of myself (Introjected regulation; α =.71), Because putting efforts in this job aligns with my personal values (Identified regulation; α =.80), and Because the work I do is interesting (Intrinsic motivation; α =.90). Validation evidence for the MWMS based on the current data set has already demonstrated adequate fit for a sixfactor ICM-CFA structure (invariant across French and English languages), acceptable scale score reliability (Cronbach s α ranged from for all subscales), and supported the convergent and discriminant validity of the scales (Gagné et al., 2015). Need satisfaction was measured using an early version of the work-related Basic Needs Satisfaction Scale (Van den Broeck, Vansteenkiste, Witte, Soenens, & Lens, 2010). This instrument assesses the satisfaction of the three basic needs for: (a) autonomy (3 items, α =.81, e.g., I feel like I can be myself at my job ); (b) competence (3 items, α =.85, e.g., I

18 BIFACTOR-ESEM AND MOTIVATION 18 really master my tasks at my job ), and (c) relatedness (4 items, α =.82,e.g., At work I feel part of a group ) on a 1 (totally disagree) to 5 (totally agree) Likert scale. Affective commitment to the organization was measured using Meyer, Allen, and Smith s (1993) organizational commitment measure (6 items, α =.84, e.g., This organization has a great deal of personal meaning to me ). Continuance commitment to the organization was measured by Stinglhamber, Bentein, and Vandenberghe s (2002) French adaptation of Meyer et al. s (1993) measure to ensure a complete coverage of both highsacrifice (i.e., cost of leaving) and low-alternative (i.e., Lack of alternatives) facets (6 items, α =.70, e.g., I consider my job opportunities as too limited to consider leaving the organization ). All items were rated on a 1 (totally disagree) to 7 (totally agree) Likert scale. Estimation and Specification All models were estimated using Mplus 7.3 (Muthén & Muthén, 2014) robust Maximum Likelihood (MLR) estimator. CFA models were specified according to ICM assumptions, with items allowed to load onto their a priori motivation factor, and all crossloadings constrained to be exactly zero. ESEM was specified using target rotation: Item loadings on their a priori motivation factors were freely estimated, and all cross-loadings were also freely estimated but targeted to be as close to 0 as possible. Bifactor-CFA (B- CFA) models were specified as orthogonal, with each item specified as loading on the SDT G-factor as well as on their a priori S-factors corresponding to the six distinct motivation types. Finally, bifactor-esem (B-ESEM) was estimated using bifactor target rotation: All items were used to define the SDT G-factor, while the 6 S-factors were defined using the same pattern of target and non-target loadings and cross-loadings as in the ESEM solution. The current models correspond to typical bifactor specifications where all items are used to define the G-factor, and one S-factor in line with theoretical expectations that all items reflect motivation types organized according to the expected continuum structure of motivation

19 BIFACTOR-ESEM AND MOTIVATION 19 reflected in the G-factor. 1 We note however that hybrid models, such as models including more than one G-factor (e.g., Caci, Morin, & Tran, 2015), or models where only a subset of items are used to define the G-factor (e.g., Brunner, Lüdtke, & Trautwein, 2008), are also possible when theoretical expectations suggest that these might be more appropriate. Covariates were then integrated to the final retained measurement model, allowing estimation of relations between the motivation factors and the covariates. In a first model, only the G-factor was allowed to covary using the ESEM-within-CFA method described by Morin et al. (2013, 2016a) which allows for the estimation of relations between only a subset of B-ESEM factors (i.e., here only the G-factor) and covariates. This model simulates the common approach used in SDT of using a single motivation score (quantity; i.e., the relative autonomy index; Fernet, Gagné, & Austin, 2010; Grolnick & Ryan, 1987; Markland & Ingledew, 2007; Pelletier, Seguin-Levesque, & Legault, 2002), which ignores the relative impact of different types (or qualities) of motivation. In a second model (relying on a regular B-ESEM representation), both the G-factor and the S-factors were allowed to predict scores on all covariates. These two models were contrasted to one another on the basis of goodness of fit information, but also based on standardized regression coefficients and model-based estimates of the percentage of explained variance (R 2 ) in the covariates afforded by the model. Model Comparisons Because of the known oversensitivity of the chi-square test of exact fit to sample size and minor model misspecifications (e.g., Marsh, Hau, & Grayson, 2005), model fit was assessed using commonly used goodness-of-fit indices and information criteria: the comparative fit index (CFI), the Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA) with its confidence interval, the Akaike Information Criteria (AIC), the Constant AIC (CAIC), the Bayesian Information Criteria (BIC), and the sample-size adjusted BIC (ABIC). According to typical interpretation guidelines (e.g., Hu & Bentler, 1999; Marsh, Hau,

20 BIFACTOR-ESEM AND MOTIVATION 20 & Wen, 2004; Marsh et al., 2005), values greater than.90 and.95 for the CFI and TLI respectively support adequate and excellent fit of the data to the model while values smaller than.08 or.06 for the RMSEA support acceptable and excellent fit. When comparing models, changes in RMSEA, CFI and TLI greater than.01 were deemed significant as established by Cheung and Rensyold (2002), and Chen (2007). Although they cannot be used to assess the global fit of a single model, the information criteria (AIC, CAIC, BIC, ABIC) are particularly useful in the comparison of alternative models, with lower values supporting a better fitting model. These guidelines have so far been established for CFA, and have also been used in previous applications of ESEM (e.g., Marsh et al., 2009, 2014; Morin et al., 2013, 2016a). However, because ESEM includes many more parameters than ICM-CFA, due to the free estimation of cross-loadings, it has been suggested that indicators including a correction for parsimony (i.e., TLI, RMSEA, AIC, CAIC, BIC, ABIC) will be critical to the assessment of model fit in an ESEM context (Marsh et al., 2009, 2010, 2014; Morin et al., 2013, 2016a). It is important to keep in mind that these remain rough guidelines for descriptive model evaluation, which also needs to take into account the even more important information coming from parameters estimates, statistical conformity and theoretical meaningfulness (Marsh et al., 2004, 2005). Indeed, each of these models is able to absorb unmodelled sources of construct-relevant multidimensionality (e.g., Asparouhov et al., 2015; Morin et al., 2016a; Murray, & Johnson, 2013). For this reason, a close examination of parameter estimates and theoretical conformity is necessary to select the best alternative among a series of models as simple goodness-of-fit-assessment is often insufficient to differentiate among models that often provide similar levels of fit to the data (Marsh et al., 2011; Morin et al., 2016a). Morin et al. (2016a, 2016b) suggest to start with a comparison of CFA and ESEM solutions. In this comparison, as long as the factors remain well-defined by strong target factor loadings, the key issue is related to the factor correlations. Statistical evidence that ESEM tends to provide

21 BIFACTOR-ESEM AND MOTIVATION 21 more exact estimates of true factor correlations (Asparouhov et al., 2015) suggests that ESEM should be retained whenever the results show a discrepant pattern of factor correlations. Otherwise, the CFA model should be preferred based on parsimony. Then, the second comparison involves contrasting the retained model with its bifactor counterpart (B- CFA or B-ESEM). Here, the key elements favoring a bifactor representation are the observation of a G-factor that is well-defined by strong factor loadings, and the observation of reduced cross-loadings in B-ESEM compared to ESEM. RESULTS Measurement Models Table 1 presents the goodness-of-fit indices and information criteria associated with each of the estimated models. The ICM-CFA demonstrated marginally adequate fit, whereas the B-CFA did not. In contrast, the ESEM solution provided an excellent representation to the data according to all indices, and provided a better representation than the ICM-CFA solution based on lower scores on the information criteria and substantial improvement on the goodness-of-fit indices (ΔCFI = +.05; ΔTLI = +.06; ΔRMSEA = -.02). Furthermore, the 90% confidence intervals for the RMSEA showed no overlap between the CFA and ESEM solutions, indicating a high degree of differentiation between competing models. Finally, although the B-ESEM solution provided an excellent representation of the data, it displayed only marginal improvement relative to the ESEM solution according to goodness-of-fit indices (ΔCFI <.01; ΔTLI = +.01; ΔRMSEA = -.01), overlapping confidence intervals for the RMSEA, almost identical values for the information criteria, but resulted in a nonsignificant chi square value suggesting that it is the only model providing exact fit to the data. However, no analysis should be conducted in disconnection from theory, expectations, and a detailed examination of parameter estimates (Marsh et al., 2014; Morin et al., 2016a). Insert Table 1 about here

22 BIFACTOR-ESEM AND MOTIVATION 22 Because the data did not fit the B-CFA model to even a minimally acceptable standard, and following Morin et al. (2016a, 2016b) recommendations, we first turn to a comparison of ICM-CFA and ESEM, before moving on to the B-ESEM solution. Before doing so however, it is worth nothing that on the strict basis of goodness-of-fit assessment showing the superiority of the ESEM and B-ESEM solutions when compared to the ICM-CFA and B- CFA solutions, cross-loadings are clearly to be expected in the solution. Because of this, factor correlations are expected to be higher in the ICM-CFA model compared to the ESEM models as this is the only way through which these cross-loadings can be expressed. Because the B-CFA model is orthogonal however, the only way for these omitted cross-loadings to be expressed is through an inflated estimate of the factor loadings of items on the G-factor (Morin et al., 2016a), which is unlikely to be a sufficient to compensate for this source of misfit should the cross-loadings reflect another source of multidimensionality than the presence of an underlying global construct. This likely explains why the fit of the B-CFA model was lower and less adequate than that of the ICM-CFA. Parameter estimates for ICM-CFA and ESEM are reported in Tables 2 (correlations) and 3 (factor loadings, cross-loadings and uniquenesses). Looking first at the loadings and cross-loadings, the overall size of the factor loadings of the items on their target factors remained similar in the ICM-CFA (λ =.50 to.90; M =.69) and ESEM (λ =.37 to.93; M =.65) solutions, showing well-defined factors corresponding to a priori expectations. In the ESEM solution, target factor loadings systematically remained higher than cross-loadings, which generally remained very small ( λ = 0 to.37 M =.02). In fact, only two cross-loadings were higher than.30: Item 1 of identified regulation ( Because I personally consider it important to put effort into this job ) cross-loaded on the introjected regulation factor at.37, and introjected item 2 ( Because it makes me feel proud of myself ) on the identified factor at.31. Closer inspection suggested no pattern of larger cross-loadings between adjacent

23 BIFACTOR-ESEM AND MOTIVATION 23 factors and smaller or more negative cross-loadings between more distant factors, providing weak support for the continuum hypothesis at the cross-loadings level. Insert Tables 2 and 3 about here As expected, factor correlations proved to be slightly lower in ESEM (r = -.47 to.77, M =.31) than ICM-CFA (r = -.48 to.81, M =.37). 2 The overall pattern of those correlations, however, was not changed by the decision to rely on an ICM-CFA or ESEM solution. A closer examination of these correlations reveals that they match the continuum hypothesis relatively well, being stronger between conceptually closer factors than between conceptually distant factors. Additionally, the amotivation factor appears to represent one end of the hypothesized continuum, showing generally negative correlations with more autonomous forms of motivations (intrinsic and identified), a smaller correlation with introjected regulation, and positive correlations with the social and material forms of external regulation (with a slightly larger correlation for the material factor than for the social factor). Looking at the B-ESEM solution, we already noted that it represented the data quite well, and provided an exact fit to the data. It is interesting to note that typical (i.e. orthogonal) representations of bifactor models attempt to synthetize the covariance (i.e., correlations) among factors through the estimation of a single G-factor, and to keep in mind that the ESEM correlations generally supported the continuum hypothesis. As such, a key advantage of the bifactor-esem model in comparison to the ESEM model, in addition to its exact fit to the data, is that it provides a single directly interpretable self-determination G-factor. Interestingly, results from the bifactor-esem solution (see Table 4) revealed a well-defined G-factor representing general self-determination. This G-factor follows the idea of a continuum underlying motivation: The loadings on the G-factor were high and positive for the items associated with the autonomous motivation S-factors (λ =.71 to.75 for intrinsic motivation, and.56 to.79 for identified regulation), moderate for the items associated with

24 BIFACTOR-ESEM AND MOTIVATION 24 introjected regulation (λ =.26 to.61), lower for the items associated with external regulationsocial (λ =.02 to.21), small or negative for the items associated with the external regulationmaterial S-factor (λ = -.07 to.25), and negative for the items associated with amotivation (λ = -.35 to -.30). Insert Table 4 about here Further examination of this solution reveals reasonably low cross-loadings, remaining lower than target loadings ( λ <.01 to.33; M =.08), and reasonably well-defined S-factors (λ =.26 to.78; M =.50), with very few noteworthy exceptions. Importantly, the cross loadings tended to be smaller in the bifactor-esem solution than in the ESEM solution, suggesting that part of the ESEM cross-loadings reflected the presence of the unmodelled G-factor. It is also interesting to note that the S-factors located at the ends of the continuum (amotivation, external-social regulation, external-material regulation, and intrinsic motivation; λ =.41 to.78; M =.57) retained substantial specificity once the covariance attributed to the selfdetermination G-factor was taken into account. Conversely, the S-factors located toward the midpoint of the continuum (identified and introjected regulation; λ =.26 to.55; M =.38) retained less specificity once the general self-determination factor is taken into account. Based on the evidence presented thus far, both in terms of exact fit to the data but most importantly theoretical conformity of the parameter estimates, the final retained model is the bifactor-esem model. In practical terms, this model also provides a way to simultaneously take into account all motivation factors (motivation quality), together with a global estimate of the quantity of self-determined motivation into a single predictive model, and to do so without having to rely on a psychometrically suboptimal RAI. Predictive Models From the final retained bifactor-esem solution, SEM analyses were used to assess the criterion-related validity of the various motivation factors. More precisely, these models were