Testing the multidimensionality of the Inventory of School Motivation in a Dutch

Transcription

1 Testing the multidimensionality of the Inventory of School Motivation in a Dutch student sample Hanke Korpershoek 1 (Groningen Institute for Educational Research, University of Groningen, the Netherlands) Jacob Kun Xu (The Hong Kong Institute of Education, Hong Kong) Magdalena Mo Ching Mok (The Hong Kong Institute of Education, Hong Kong) Dennis M. McInerney (The Hong Kong Institute of Education, Hong Kong) Greetje van der Werf (Groningen Institute for Educational Research, University of Groningen, the Netherlands) 1 Corresponding author: Hanke Korpershoek, University of Groningen, GION - Institute for Educational Research, Grote Rozenstraat 3, 9712 TG Groningen, The Netherlands. h.korpershoek@rug.nl. Telephone:

2 Abstract A factor analytic and a Rasch measurement approach were applied to evaluate the multidimensional nature of the school motivation construct among more than 7,000 Dutch secondary school students. The Inventory of School Motivation (McInerney and Ali, 2006) was used, which intends to measure four motivation dimensions (mastery, performance, social, and extrinsic motivation), each comprising of two first-order factors. One unidimensional model and three multidimensional models (4-factor, 8-factor, higher order) were fit to the data. Results of both approaches showed that the multidimensional models validly represented the school motivation among Dutch secondary school pupils, whereas model fit of the unidimensional model was poor. The differences in model fit between the three multidimensional models were small, although a different model was favoured by the two approaches. The need for improvement of some of the items and the need to increase measurement precision of several first-order factors are discussed. Keywords: school motivation; confirmatory factor analysis, multidimensional Rasch analysis, MRCML models, secondary education 2

3 Introduction This paper demonstrates the construct validity of a measure of Dutch secondary school students school motivation, using confirmatory factor analysis and multidimensional Rasch analysis. The cross-culturally validated Inventory of School Motivation (McInerney and Ali, 2006) was administered to more than 7,000 students across the Netherlands for this purpose. The Inventory of School Motivation aims at capturing the multidimensional nature of the motivation construct among students in an educational setting. Scholars in the field of educational psychology have paid considerable attention to motivation theories such as the achievement goal theory of motivation (e.g. Elliot and McGregor, 2001). According to this theory, achievement goals direct students behaviour in the classroom. Furthermore, the theory proposes that students pursue different achievement goals in particular learning situations. The theory and its related research originally concentrated on two types of goal orientations, namely mastery orientation (or learning or task orientation) and performance orientation (or ego orientation). Whereas mastery-oriented students attempt to gain knowledge and improve their skills, performance-oriented students are particularly focussed on demonstrating their ability. Both mastery and performance motivation have been found to be positively related to education outcomes, although this relation seems to be more consistent for mastery than for performance motivation (see Huang, 2012 and Hulleman, Schrager, Bodmann, and Harackiewicz, 2010 for recent meta-analyses). In addition to mastery and performance goals, it has been suggested that both social goals and extrinsic goals should be included when investigating students motivation. These suggestions follow from Maehr s theory of Personal Investment (1984). Urdan and Maehr (1995) have defined social goals as the perceived social purposes for academic achievement. Ali and McInerney (2004) refer to these goals as social-grounded reasons for studying, resulting from 3

4 social affiliation and social concern (see also King and McInerney, 2012). Extrinsic motivation has been defined as one s desire for external rewards (e.g. Ryan and Deci, 2000). Social and extrinsic motivation, along with mastery and performance motivation, are assumed to guide students behaviour in their learning tasks (see for example Horst, Finney, and Barron, 2007; King, McInerney, and Watkins, 2010; McInerney and Ali, 2006). For social goals, positive associations with educational outcomes have been reported (e.g. King and McInerney, 2012; King et al., 2010). For extrinsic goals, both positive as well as negative relationships have been found in prior studies (e.g. McInerney, 2008; McInerney, Roche, McInerney, and Marsh, 1997; Wolters, Yu, and Pintrich, 1996). In this paper, the four types of motivation (mastery, performance, social, and extrinsic) are the core focus. These dimensions are drawn from the measurement framework validated by McInerney and Ali (2006; see also McInerney and Sinclair, 1991, 1992). Based on Maehrs work (1984; see also Maehr and McInerney, 2004; Urdan and Maehr, 1995), McInerney and Ali validated, across a range of cultural groups, the Inventory of School Motivation (ISM) to capture the four motivation dimensions. Each of the dimensions comprises two first-order factors. Mastery motivation includes items about task (task involvement) and effort goals, performance motivation about competitiveness and social power goals, social motivation about social affiliation and social concern goals, and extrinsic motivation about desiring praise and tokens for achievement. Following this measurement framework, the purpose of the present paper is to validate the dimensionality of the ISM in a Dutch student sample using confirmatory factor analyses and to investigate the measurement properties of the scales (such as reliability and model-data fit) using multidimensional Rasch analyses. In recent years, multidimensional Rasch analysis has found its way into the broader field of educational psychology, although studies that use of multidimensional Rasch models are still scarce (Mok, McInerney, and Cheng, 2011). Multidimensional Rasch analysis is 4

5 believed to increase measurement precision of multidimensional constructs (Wang, Chen, and Cheng, 2004), such as school motivation (e.g. Mok, McInerney, Cheng, and Lai, 2011). Hence, the main research question is: Does the Inventory of School Motivation validly represent school motivation among Dutch secondary school pupils? Although the ISM has been used in various cultural groups in the past two decades (e.g. Ganotice, Bernardo, and King, 2012; King, Ganotice, and Watkins, 2012; King and Watkins, 2013; McInerney, 2012; McInerney et al., 1997; McInerney and Ali, 2006; McInerney, Marsh, and Yeung, 2003), it has only recently been introduced in the Netherlands. To our knowledge, the COOL 5-18 project (see method section) is the first to administer the ISM in Dutch secondary education, using 33 translated items on a 5-point Likert scale. Hence, the validity of the multidimensional school motivation scale among Dutch secondary school students is unknown. In other cultural settings, confirmatory factor analyses have demonstrated the construct validity and reliability (i.e. internal consistency) of the ISM scales (e.g. Ganotice et al., 2012; King et al. 2012; McInerney and Ali, 2006; McInerney and Sinclair, 1991, 1992; McInerney et al., 1997, 2003, 2005). These studies revealed that the factor structure of the ISM is generally invariant across cultures. In a technical report of the Dutch COOL 5-18 project, it was demonstrated that the four suggested motivation dimensions could be replicated in the Dutch sample as well (Cronbach s alpha was.77 for mastery,.84 for performance,.74 for social, and.86 for extrinsic motivation; Zijsling, Keuning, Kuyper, Van Batenburg, and Hemker, 2009). However, the multidimensional structure of the data has not been investigated any further, particularly in comparison with a unidimensional model (i.e. one overall school motivation scale). Three plausible multidimensional models are suggested in the literature, namely a 4-factor model (including mastery, performance, social, and extrinsic motivation), a 8-factor model (including the eight first-order factors), and a higher order model (8 first-order factors nested in 4 factors) (Mok, McInerney, Cheng et al., 5

6 2011). The study of Mok, McInerney, Cheng et al. (2011) demonstrated that the 8-factor model was superior to the 4-factor model in describing the latent structure of motivation dimensions, and that the higher order model did not improve model fit as compared to the 8- factor model. Whether these results can be generalized to the Dutch student population will be examined in the present paper. By demonstrating the importance of assessing multidimensional validity for complex constructs, we believe that our paper contributes to the field in two major ways. First, the paper can serve as an example of how multidimensional Rasch analysis can be applied to various multidimensional constructs in the international field of educational psychology (see also Ackerman, Gierl, and Walker, 2003; Lee, Zhang, and Yin, 2010; Liu, Minsky, Ling, and Kyllonen, 2009; Mok, McInerney, Cheng et al., 2011, 2013; Mok and Xu, 2013, Yan and Mok, 2012), in addition to some excellent examples using Rasch analysis for unidimensional scales (e.g. Smith, Wakely, De Kruif, and Swartz, 2003). Rasch analysis assesses the measurement properties of Likert-scale items in a specific sample, which enhances our understanding of the validity of our measure in the Dutch data. Another advantage of applying a Rasch model is that Rasch analysis can confirm that the items used evoke and define the variable as intended (Wright and Masters, 1982). That is, the workings of some items might be context-specific or may not apply to Dutch students as compared with other samples. Hence, secondly, the analysis gives useful insights in Dutch students school motivation in secondary education. The Rasch measurement model for rating scales The purpose of a measurement model is to extract from suitable data a useful definition of an intended variable and then to measure persons on this variable (Wright and Masters, 1982, p. 6

7 90). The Rasch measurement model allows the item difficulty of each item to be based on the way in which a group of subjects actually responded to that item in practice. It treats rating scale data (e.g. Likert-type items) as ordered data instead of as interval data which is common practice in, for example, factor analytic procedures. Ordered data indicates that the value of each category is higher than that of the previous one, but that the size of the steps is not assumed to be spread equidistantly. The Rasch model is a stochastic model that applies the logarithmic transformation to estimate log-odds of each item and each person (Bond and Fox, 2001; Wright and Masters, 1982). For rating scale data, the scales have a difficulty estimate as well as a series of thresholds. This is the level at which the likelihood of failure at a given response category (below the threshold) turns to the likelihood of success at that category (above the threshold; Bond and Fox, 2001, pp ). Thus, for rating scales, success can be interpreted as agreement with or endorsement of a particular response category, whereas failure can be interpreted as failure to agree with or failure to endorse a particular response category. Similarly, for the items, statements with high scores are statements with which people were inclined to agree, whereas statements with low scores were statements with which people were inclined to disagree. Adams, Wilson, and Wang (1997) put forth the mathematical formula for the multidimensional Rasch model for polytomous (e.g. Likert-type) scales. They refer to these models as multidimensional random coefficient multinomial logit (MRCML) models. The MRCML model assumes that a set of latent traits affects subjects responses to items designed to measure the traits. Recently, Harrell-Williams and Wolfe (2013) advise practitioners working with highly correlated multidimensional data to use MRCML models (or other multidimensional IRT models) to minimize data-to-model- misfit. A multidimensional model can simultaneously calibrate all first-order factors and increase measurement precision by taking into account the correlations between the first-order factors (Wu and Adams, 2006). 7

8 Method Participants The participants in this study were 7,763 secondary school students from the Netherlands. The data were collected as part of a large-scale longitudinal study in primary and secondary education, which is the so-called COOL 5-18 project. The students in our sample were all in the 9 th grade of secondary education and were, on average, 16 years old. In total, 3,886 boys (50.1%) and 3,857 girls (49.7%) were included, and 20 students did not indicate their gender. The sample is fairly representative of the Dutch student population, however, students pursuing the highest educational tracks (senior general secondary education and preuniversity education) are slightly overrepresented. For more information on the aims of the COOL 5-18 project we refer to Zijsling et al. (2009). The Inventory of School Motivation We used an adapted version of the Inventory of School Motivation (ISM) of McInerney and Ali (2006). The ISM consists of eight first-order factors (task, effort, competition, social power, affiliation, social concern, praise, and token) that per pair belong to one of the four second-order factors, which are mastery, performance, social, and extrinsic motivation. The questionnaire used here included 33 translated items (into the Dutch language) using a 5-point Likert scale for the responses. In Dutch, the categories were labelled as follows: klopt helemaal niet (1), klopt een beetje (2), klopt matig (3), klopt vrij goed (4), klopt precies (5) (In English "does not at all apply to me" (1), applies to me to some extent (2), moderately 8

9 applies to me (3), generally applies to me (4), "strongly applies to me" (5)). The number of items used to measure each first-order factor varied from 3 to 5. Cronbach s alpha reliabilities were above.70 for 6 of the first-order factors and slightly lower for the first-order factors task (.59) and affiliation (.68). The second-order factors were measured with 7 to 9 items each. The items and the corresponding first- and second-order factors can be found in Appendix A. Originally, for adjacent item scores (e.g. students ticked 2 and 3 as a response on one item), the average score of the two responses was imputed. This was the case for 176 entries. To be able to run the MRCML analysis, however, decimal scores cannot be used (i.e. they are, in fact, additional thresholds, with too few entries per item for the model to converge). Therefore, they were recoded into missing. For 6,367 students (82.0%) all item scores were available, for 842 students (10.8%) only one item score was missing, for 195 students (2.5%) two item scores were missing, and for 356 (4.6%) more than two item scores were missing. Analyses First, descriptive results (means logits and standard deviations) were examined for the overall school motivation scale and for the four second-order factors and eight first-order factors suggested in the literature. Then, confirmatory factor analyses (CFA) using Mplus (version 7, Muthén and Muthén, ) were performed on the data to confirm the dimensionality. Four different factor models were fitted to the data (see also Mok, McInerney, Cheng et al., 2011), namely a unidimensional model, and three multidimensional models, that is, a 4-factor model including the mastery, performance, social, and extrinsic scales, a 8-factor model including the task, effort, competition, social power, social affiliation, social concern, praise, and token scales, and a higher order model with 8 first-order factors nested in 4 factors 9

10 (mastery, performance, social, and extrinsic). For each model, chi-squared statistics, Comparative Fit Indices (CFI), Tucker Lewis Indices (TLI), root mean square error of approximation (RMSEA), and weighted root mean square residuals (WRMR) are reported. For the multidimensional models the first factor loading within each subset of items was fixed to one. Weighted least squares with mean- and variance-adjusted chi-square (WLSMV) estimation was used in the models. Finally, MRCML models (Wang et al., 2004) were used to analyse the measurement properties of the scales. The analyses were performed with the ConQuest programme (version 2.0; Wu, Adams, Wilson, and Haldane, 2007). One unidimensional and two multidimensional rating scale models were applied. Similar to the CFA s, the first multidimensional rating scale model included the mastery, performance, social, and extrinsic motivation second-order factors. The second multidimensional rating scale model included the task, effort, competition, social power, affiliation, social concern, praise, and token first-order factors. By doing so, we took a confirmatory approach in the MRCML models, because we have prior knowledge on the presumed underlying latent traits (see also Harrell-Williams and Wolfe, 2013). Unfortunately, MRCML models cannot be used for higher order models (in our case the "8 first-order factors nested in 4 factors model). A number of fit statistics are available for analyzing polytomous data using the Rasch model, including mean square (MNSQ) fit statistics and their standardized version, namely the t-statistics (ZSTD). Mean square fit statistics were found to be relatively more robust, in comparison with t-statistics, to increase in sample size for the Rasch Rating Scale Model (Smith, Rush, Fallowfield, Velikova and Sharpe, 2008). Mean square fit statistics were therefore used in this study for scale quality assurance, that is, to determine how well our set of empirical data met the requirements of the three Rasch models. Two chi-square ratios are reported, namely the infit (weighted) and outfit (unweighted) mean square statistics, to 10

11 evaluate item fit. Item parameters (item difficulty parameter estimates) are reported and Wright maps are added to show the extent to which the items align with the respondents. The person separation reliabilities are reported for each scale. Finally, the unidimensional model and multidimensional models are compared with the Averaged Relative Efficiency (ARE) index (Wang et al., 2004) to evaluate the gain in efficiency for more complicated models. The ARE index is equivalent to the number of times a more complex model (i.e. more dimensions) is more precise than a simpler model in the estimation of parameters (Wang et al., 2004). Results Descriptive results Table 1 shows the mean logit scores and the standard deviations on the school motivation scales. [Insert Table 1 about here] Looking at the second-order factors, the highest logit mean is found for mastery motivation, followed by social motivation and extrinsic motivation. The lowest logit mean is found for performance motivation. From the final column presenting the first-order factors we see that particularly the logit mean for task motivation was high compared with the logit mean for effort, whereas they both belong to the same second-order factor of mastery motivation. In all other cases, the differences between the two scales per second-order factor were much smaller. 11

12 Results of the confirmatory factor analyses Tables 2 to 4 show the results of the CFA's. Table 2 presents the model fit indices for the unidimensional model and the three multidimensional models, that is, the 4-factor model, the 8-factor model, and the "8 first-order factors nested in 4 factors" model. [Insert Table 2 about here] The comparison shows that the higher order model performed the best in terms of having the smallest chi-square value and largest degrees of freedom, the higher CFI and TFI values, and the smallest RMSEA value (ideally <.05). The WRMR value of this model is smaller than the one of the 4-factor model, although slightly larger than the 8-factor model. Furthermore, the 8-factor model yielded slightly more satisfactory fit indices than the 4-factor model. The CFI and TLI values of the 8-factor model were higher than those of the 4-factor model and also the RMSEA and WRMR values were in favour of the 8-factor model. In all models, the 90% confidence intervals for the RMSEA's were narrow. The 1-factor model yielded unsatisfactory results, indicating limited model fit on the data. Table 3 presents the correlations among the latent factors in the multidimensional models. [Insert Table 3 about here] Correlations in the models varied from.06 to.83. The 4-factor model yielded three moderately high correlations (>.60) between three pairs of factors, namely between mastery and social motivation (.61), mastery and extrinsic motivation (.62), and between performance and extrinsic motivation (.65). In the higher order model, the correlations between the higher 12

13 order factors were slightly higher than in the 4-factor model. The 8-factor model yielded one high correlation (between praise and token:.83) and several moderately high correlations, namely between task and effort (.67), between effort and social concern (.61), between effort and praise (.62), between competition and social power (.65), between competition and praise (.67), and between competition and token (.67). Table 4 shows the parameter estimates for the models. [Insert Table 4 about here] Table 4 shows that the parameter estimates for the multidimensional models, overall, yielded moderately strong factor loadings. The parameter estimates for the unidimensional model were in all cases lower than those of the multidimensional models. Figure 1 graphically shows a comparison of the item difficulty parameter estimates of the 4-factor and 8-factor model. The results of the higher order model are not included in the graph, because the parameter estimates were similar to those of the 8-factor model (max difference). As shown in Table 4 and Figure 1, it can be seen that the standardized parameter estimates for the 4-factor model were almost equal to those for the 8-factor model and the higher order model. In the graph, almost all points lie on the x = y straight line. The exceptions were social affiliation items (represented by triangles in Figure 1), the task items (squares), and the social power items (circles). It seems that particularly the items intended to measure social affiliation had stronger factor loadings in the 8-factor model and the higher order model. However, in these models, we still found somewhat lower factor loadings (<.60) for 7 items (items 3, 5, 6, 11, 16, 25, and 26), of which three were intended to measure task motivation. The lowest estimate was found for item 11 (0.33). Item 11 was originally stated as "I care about other people at school". Inspection of the item in the Dutch questionnaire points towards ambiguous translation of this item ( Het kan me wat schelen hoe het met andere kinderen op school 13

14 gaat ; see also Zijsling et al., 2009). The final rows show another set of estimates for the higher order model, these are the inter-factor associations between the loadings of lower order factors (e.g. task) on higher order factors (e.g. mastery). Each lower order factor loaded strongly on its hypothesized higher order factor (varying between.66 to.98), except social affiliation which loading moderately on its higher order factor social motivation (.52). [Insert Figure 1 about here] In sum, our results show acceptable model fit for the multidimensional models, however, the fit indices and parameter estimates suggest that the 8-factor model and higher order model fitted the data slightly better than the 4-factor model. The Rasch analyses are presented in the next paragraph and will reveal further information on the multidimensionality of the ISM and fit indices for the items. Results of the uni- and multidimensional Rasch analyses Table 5 shows the MLE person separation reliabilities of the dimensions in the 1-factor, 4- factor, and 8-factor models, as well as the factor correlation matrices. Figures 2 to 4 show the Wright maps for the models. First, the model fit statistics are discussed here. [Insert Table 5 about here] [Insert Figures 2-4 about here] The results of the unidimensional model (1-factor model) show that the step calibrations increased monotonically from 1.040, 0.607, to logits, the distances between the 14

15 steps being 0.433, 0.617, and 1.647, respectively. This corresponds with the overall assumption that there is a linear trend in the Likert scales, however, the distance between the first two categories in particular is rather small (guideline for a 5-point Likert scale is a minimal distance of 1.0; Linacre, 2002). This result indicates that the first two categories of the items to a limited extent define a distinct position on the scale. Thus, generally, the items did not function well in the unidimensional model. Similar to the unidimensional model, the step calibrations in the 4-factor model increased monotonically from 1.376, 0.623, to logits, with the distances between the steps being 0.753, 0.740, and 1.765, respectively. In the 4-factor model, the values on the item level also increased monotonically, which is in line with the proposed theoretical model with Likert scales. The results show that the item MNSQ infit (weighted fit) varied between 0.68 and 1.37 and MNSQ outfit (unweighted fit) varied between 0.68 and Using a series of response strings and their mean square fit statistics, Smith (1996) highlighted that MNSQ values smaller than 1.0 indicate the fit of data to the model was better than expected and values above 1.0 indicate existence of unmodelled variance, or noise in the data (Smith, 1998). When there is too much noise in the data, information in the data will be masked, making it useless for measurement construction. Further simulation studies by Smith, Schumacker and Bush (1998) showed that distribution of rating scale fit statistics was not symmetrical. In addition, their study showed sample size to have differential effects on weighted and unweighted mean square fit values of items. Smith et al. (1998) recommended using critical values of 1.06 for MNSQ infit (weighted fit), and 1.19 for MNSQ outfit (unweighted fit), for studies with sample size of approximately 1,000. Although no simulation results were available for studies with sample sizes greater than 1,000 were available, Smith et al. s (1998) study indicated that more stringent critical values would be appropriate for such studies. This observation applies to the current study which involved a sample size of 7,763 15

16 cases. In the 4-factor model, we found 14 items (namely, items 1, 5, 6, 10, 11, 13, 14, 16, 17, 20, 22, 24, 28, 33) with MNSQ infit (weighted fit) greater than 1.06 and 5 items (namely, items 5, 6, 11, 16, 28) with MNSQ outfit (unweighted fit) greater than Using these two criteria of 1.06 and 1.19 respectively for the weighted and unweighted fit statistics, all items designed to measure praise in the extrinsic motivation scale (except item 24), all items designed to measure social concern in the social motivation scale (except item 11), and all items designed to measure effort in the mastery motivation scale fitted the Rasch model well. All items designed to measure social power in the performance scale had unweighted fit statistics within bound. Items designed to measure task in the mastery motivation scale, competition in the performance motivation scale, social affiliation in the social motivation scale, and token in the extrinsic motivation scale need to be further enhanced in order to better fit the 4-factor Rasch model. Table 5 shows that the correlations between the four dimensions correspond largely to the factor correlations in Table 3 which were based on the CFA's. Lastly, the 8-factor model had, similar to the 4-factor model, monotonically increasing step calibrations (from 1.589, 0.670, to logits, the distances between the steps were 0.919, 0.852, and 1.895) and monotonically increasing values on the item level. The item fit statistics (MNSQ infit) lay between 0.75 and 1.48 and between 0.75 and 1.51 (MNSQ outfit). We found 16 items (namely, items 2, 4, 5, 6, 7, 9, 10, 11, 14, 16, 17, 22, 24, 25, 28, 33) with MNSQ infit greater than 1.06 and 4 items (namely, items 5, 6, 11, 16) with MNSQ outfit greater than The item fit was therefore better in the 4-factor model than in the 8- factor model. Only the items designed to measure effort in the mastery motivation scale fitted the Rasch model well. All other scales need further refinement in order to better fit the 8- factor Rasch model. Person separation reliabilities were also clearly higher in the 4-factor model than in the 8-factor model. Particularly the person separation reliability of the dimension social power was unsatisfactory in the 8-factor model (.478). Again, the 16

17 correlations between the eight dimensions correspond largely to the CFA factor correlations presented in Table 3. Based on the Rasch analysis using the item fit statistics and person separation reliabilities, the 4-factor model fitted the data slightly better than the 8-factor model. This indicates that the CFA's and Rasch model results are similar but not equal. The Wright maps for the Rasch models are presented in Figures 2 to 4. The Wright maps give a visual impression of how well the item difficulty levels target well the students' endorsement levels. This is done by matching two distributions visually on the two sides of the common Rasch measurement scale. The Wright maps for the 4-factor and 8-factor models show a good match between item and person distributions; the Wright map for the 1-factor model shows a much poorer match. With the exception of the performance dimension (dimension 2) in the 4-factor model and of the competition and social power dimensions (dimensions 3 and 4) in the 8-factor model, there is good coverage of person ability (i.e. endorsement levels) by the distribution of items together with the item thresholds. Further, the Wright map of the 4-factor model (Figure 3) shows that the mastery dimension (dimension 1 of the 4-factor model) model has a positively skewed person distribution while the performance dimension (dimension 2 of the 4-factor model) the person distribution is negatively skewed. The positively skewed distribution means that items in the mastery dimension are easily endorsed by the students. The negatively skewed distribution means that items in the performance dimension are very difficult to be endorsed by the students. Person distributions in the other two dimensions in the 4-factor model are relatively more symmetrical. It can be seen from the Wright map of the 8-factor model (Figure 4) that the person distribution in the task dimension (dimension 1 of the 8-factor model) is positively skewed while the person distribution in the effort dimension (dimension 2 of the 8-factor model) is 17

18 very symmetrical. This means that items in the task dimension tend to be very easy to endorse for the students. Students tend to agree that they need to know that they are getting somewhere with their schoolwork, that they try harder with interesting work, that they like to see that they are improving in schoolwork, and being given the chance to do something again to make it better. Person distribution in the effort dimension is more even and well covered by the items in this dimension. In comparison, the Wright map of the 8-factor model (Figure 4) shows that the person distributions of both the competition and social power dimensions (dimensions 3 and 4 of the 8-factor model) are both negatively skewed, meaning that students tend not to agree with items in these two dimensions. Easier items (thus easier to endorse) in these two dimensions need to be constructed in the future to enhance the coverage in order to capture students attitudes toward competition and social power. As can be seen from the above, the 4-factor and 8-factor models are not in contrast with one another. Rather, the 8-factor model gives more refined details about the measurement of school motivation of 9 th grade students in the Netherlands. Finally, ARE ratios are used to compare the measurement precision of the multidimensional models (i.e. the 4-factor and 8-factor models) with the unidimensional model. Table 6 shows that the ARE ratios of the scales ranged from to An ARE equal to 1.5 indicates that the unidimensional approach needs 1.5 times test length as the multidimensional approach does in order to achieve the same level of precision (see also Yan and Mok, 2012). With two exceptions (task and token), they are all greater than 1, which means that measurement precision of the scales can be enhanced using the multidimensional approach in comparison with the unidimensional approach. That is, a value of ARE equal to 1.0 means that there is no difference between the precisions estimated in the unidimensional and multidimensional models, implying that in our case, the multidimensional model reached higher levels of measurement precision than the unidimensional models in almost all scales. 18

19 That is, the multidimensional approach was, on average, more precise than the unidimensional approach, although the differences in reliabilities did not typically favour any of the models. For the task and token first-order factors in the 8-factor model, the current items need some improvement to increase measurement precision in this more complex model. [Insert Table 6 about here] Conclusions Our findings suggest that the Dutch version of the Inventory of School Motivation (McInerney and Ali, 2006) validly represents the school motivation among Dutch secondary school pupils. Model fit indices and parameter estimates demonstrated that the ISM captured the multidimensional nature of the school motivation construct (i.e. construct validity). In line with previous findings, the factor structure of the ISM is replicated in the Dutch sample, indicating that, as expected, the factor structure is generally invariant across cultures (e.g. Ganotice et al., 2012; King et al. 2012; McInerney and Ali, 2006; McInerney and Sinclair, 1991, 1992; McInerney et al., 1997, 2003, 2005). More precisely, the results of our study revealed that the CFA's and multidimensional Rasch models were similar but not equal. In the CFA's, the 8-factor model fitted the data slightly better than the 4-factor model, and much better than the unidimensional model, which showed poor model fit (see also Mok, McInerney, Cheng et al., 2011). Additionally, a higher order model (8 first-order factors nested in 4 factors) refined the 8-factor model to some extent. This in contrast with Mok, McInerney, Cheng et al. (2011) in which the higher order model did not improve model fit, but in both studies, model fit of these two models was more or less similar. Rasch analyses demonstrated a good match between item and person distributions in both the 4-factor and 8-19

20 factor models (higher order multidimensional Rasch models cannot be estimated in Conquest). Based on the Rasch modeling results such as the item fit statistics and person separation reliabilities, the 4-factor model fitted the data slightly better than the 8-factor model. The correlations we found among the four higher order factors (mastery, performance, social, and extrinsic motivation) are theoretically meaningful, for example, performance and extrinsic motivation share an interpersonal focus and are relatively highly correlated (.65 in the CFA 4-factor model). Another example is that the low correlation (.18 in the CFA 4-factor model) between performance and social motivation indicates that the desire to outperform others does not correspond with, for instance, the desire to help others. Moreover, the firstorder factors that theoretically belong to the same higher order factor were generally more highly correlated than other pairs of first-order factors. By using both a factor analytic approach and a Rasch modeling approach to demonstrate the multidimensionality of the ISM, multiple fit indices and item fit statistics could be used to evaluate overall model fit. The advantage of the factor analytic approach was that a higher order factor model could be fit to the data, which in our case, was suggested by the literature on school motivation. The benefits of using Rasch analysis were that it handled ordered data appropriately, that it provided additional information on item thresholds (e.g. did the thresholds increase monotonically for each item), and that it evaluated item and person distributions in much more detail. Notwithstanding the overall model fit of the multidimensional models, we found that several items and first-order factors need further refinement (at least in the Dutch context). The translation of item 11 (I care about other people at school), for example, needs improvement, because the factor loading on the social concern first-order factor was low (.33). Apparently, it measured something different from social concern as measured by the 20

21 other social concern items. For the first-order factor task motivation, the factor loadings (based on the 8-factor CFA model) of three of the four of the items were also relatively low (<.60). Moreover, the reliability of the first-order factor task motivation was low (.59). Similarly, the Rasch models revealed that the measurement precision of several items in various first-order scales needs to be enhanced. Furthermore, the multidimensional Rasch models showed that easier items are necessary for the competition and social power dimensions, because students tended to disagree with the items. Finally, a limitation to consider when interpreting and generalizing the present findings is that the generalization of the present data is restricted because students in the highest educational tracks were slightly overrepresented in our sample. Further analyses in which the different educational tracks are analysed separately could demonstrate further validity of the ISM scales across educational tracks. All in all, in spite of the noted suggestions for improvement, this study demonstrated that the ISM adequately captured the multidimensional nature of the school motivation construct among Dutch secondary school pupils. References Ackerman, T. A., Gierl, M. J., and Walker, C. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practices, 22, Adams, R. J., Wilson, M. R., and Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, Ali, J. and McInerney, D. M. (2004, July). Multidimensional assessment of school motivation. Paper presented at the 3 rd SELF Research Conference, Berlin, Germany. 21

22 Bond, T. G. and Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum. Elliot, A. J. and McGregor, H. A. (2001). A 2x2 achievement goal framework. Journal of Personality and Social Psychology, 80, Ganotice, F. A., Bernardo, A. B. I., and King, R. B. (2012). Testing the factorial invariance of the English and Filipino versions of the Inventory of School Motivation with bilingual students in the Philippines. Journal of Psychoeducational Assessment, 30, Harrell-Williams, L. M., and Wolfe, E. W. (2013). The influence of between-dimension correlation, misfit, and test length on multidimensional Rasch model information-based fit index accuracy. Educational and Psychological Measurement, 73, Horst, S. J., Finney, S. J., and Barron, K. E. (2007). Moving beyond academic achievement goal measures: A study of social achievement goals. Contemporary Educational Psychology, 32, Huang, C. (2012). Discriminant and criterion-related validity of achievement goals in predicting academic achievement: A meta-analysis. The Journal of Educational Psychology, 104, Hulleman, C. S., Schrager, S. M., Bodmann, S. M., and Harackiewicz, J. M. (2010). A metaanalytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? Psychological Bulletin, 136, King, R. B. and McInerney, D. M. (2012). Including social goals in achievement motivation research: Examples from the Philippines. Online Readings in Psychology and Culture, Unit 5. Retrieved from King, R. B. and Watkins, D. A. (2013). Validating the Chinese version of the Inventory of School Motivation. International Journal of Testing, 13,

23 King, R. B., Ganotice Jr., F. A., and Watkins, D. A. (2012). Cross-cultural validation of the Inventory of School Motivation (ISM) in the Asian setting: Hong Kong and the Philippines. Child Indicators Research, 5, King, R. B., McInerney, D. M., and Watkins, D. A. (2010). Can social goals enrich our understanding of students motivational goals? Journal of Psychology in Chinese Societies, 11, Lee, J. C.-K., Zhang, Z., and Yin, H. (2010). Using multidimensional Rasch analysis to validate the Chinese version of the Motivated Strategies for Learning Questionnaire (MSLQ-CV). European Journal of Psychology of Education, 25, Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, Liu, O. L., Minsky, J., Ling, G., and Kyllonen, P. (2009). Using the standardized letters of recommendation in selection: Results from a multidimensional Rasch model. Educational and Psychological Measurement, 69, Maehr, M. L. (1984). Meaning and motivation: Toward a theory of personal investment. In C. Ames and R. Ames (Eds.), Research on motivation in education, Vol. 1 (pp ). New York: Academic Press. Maehr, M. L. and McInerney, D.M. (2004). Motivation as Personal Investment. In D. M. McInerney and S. V. Etten (Eds.), Big theories revisited. (pp.61-90). Greenwich, CT: Information Age. McInerney, D. M. and Ali, J. (2006). Multidimensional and hierarchical assessment of school motivation: Cross-cultural validation. Educational Psychology, 26, McInerney, D. M. and Sinclair, K. E. (1991). Cross-cultural testing: Inventory of school motivation. Educational and Psychological Measurement, 51,

24 McInerney, D. M. and Sinclair, K. E. (1992). Dimensions of school motivation: A crosscultural validation study. Journal of Cross-Cultural Psychology, 23, McInerney, D. M., Marsh, H. W., and Yeung, A. S. (2003). Toward a hierarchical goal theory model of school motivation. Journal of Applied Measurement, 4, McInerney, D. M., Roche, L. A., McInerney, V., and Marsh, H. W. (1997). Cultural perspectives on school motivation: The relevance and application of goal theory. American Educational Research Journal, 34, McInerney, D. M. (2008). Personal investment, culture and learning: Insights into school achievement across Anglo, Aboriginal, Asian and Lebanese students in Australia. International Journal of Psychology, 43, McInerney, D. M. (2012). Conceptual and methodological challenges in multiple goal research among remote and very remote indigenous Australian students. Applied Psychology, 61, Mok, M. M. C., McInerney, D. M., and Cheng, R. W.-Y. (2011, August-September). Using multidimensional Rasch modelling to enhance measurement precision: The case of selfprocesses scales. Paper presented at the European Association for Research on Learning and Instruction 2011 Biannual meeting. Exeter, United Kingdom. Mok, M. M. C., McInerney, D. M., Cheng, R. W.-Y., and Lai, M. P. Y. (2011, April). Qualitative and quantitative differences in primary and secondary students learning motivation: Multivariate multilevel and multidimensional Rasch analysis. Paper presented at the American Educational Research Association 2011 Annual Meeting. New Orleans, LA. Mok, M. M. C. and Xu, K. (2013). Using multidimensional Rasch to enhance measurement precision: Initial results from simulation and empirical studies. Journal of Applied Measurement, 14,

25 Muthén, L. K. and Muthén, B. O. ( ). Mplus user's guide. Seventh edition. Los Angeles, CA: Muthén and Muthén. Ryan, R. M. and Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, Smith, A. B., Rush, R., Fallowfield, L. J., Velikova, G., and Sharpe, M. (2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Medical Research Methodology, 8(33). DOI: / Smith, E. V., Wakely, M. B., De Kruif, R. E. L., and Swartz, C. W. (2003). Optimizing rating scales for self-efficacy (and other) research. Educational and Psychological Measurement, 63, Smith, R. M. (1996). Polytomous mean-square fit statistics. Rasch Measurement Transactions, 10, Smith, R. M., Schumacker, R. E., and Bush, M. J. (1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2, Urdan, T. C. and Maehr, M. L. (1995). Beyond a two-goal theory of motivation and achievement: A case for social goals. Review of Educational Research, 65, Wolters, C. A., Yu, S. L., and Pintrich, P. R. (1996). The relation between goal orientation and students motivational beliefs and self-regulated learning. Learning and Individual Differences, 8, Yan, Z. and Mok, M. M. C. (2012). Validating the Coping Scale for Chinese Athletes using multidimensional Rasch analysis. Psychology of Sport and Exercise, 13, Wang, W.-C., Chen, P.-H., and Cheng, Y. Y. (2004). Improving measurement precision of test batteries using multidimensional item response models. Psychological Methods, 9, Wright, B. D. and Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press. 25

26 Wu, M. L. and Adams, R. J. (2006). Modelling mathematics problem solving item responses using a multidimensional IRT model. Mathematics Education Research Journal, 18, Wu, M. L., Adams, R. J., Wilson, M. R., and Haldane, S. A. (2007). ACER ConQuest, Version 2.0 [Computer software]. Camberwell, Victoria, Australia: ACER Press. Zijsling, D., Keuning, J., Kuyper, H., Batenburg, Th. van, and Hemker, B. (2009). Cohortonderzoek COOL5-18. Technisch rapport eerste meting in het derde leerjaar van het voortgezet onderwijs [Cohort study COOL5-18. Technical report of the first wave in the 9 th grade of secondary education]. Groningen/Arnhem, the Netherlands: GION/Cito. 26

27 Table 1 Logit means of the scales Descriptive results: Overall mean (SD) Scale Mean (SD) Scale Mean (SD) (0.63) Mastery 0.30 (0.85) Task 1.08 (1.22) Effort (1.08) Performance (1.47) Competition (1.62) Social Power (1.67) Social 0.02 (0.83) Social Concern (1.06) Social Affiliation 0.37 (1.27) Extrinsic (1.13) Praise (1.41) Token (1.36) Table 2 Model comparisons 1-factor 4-factor 8-factor Higher order # free parameters χ 2 -test of model fit * * * * degrees of freedom CFI TLI RMSEA % confidence interval RMSEA [ ] [ ] [ ] [ ] WRMR Note. * p <.001. CFI = Comparative Fit Index, TLI = Tucker Lewis Index, RMSEA = root mean square error of approximation, WRMR = weighted root mean square residual. Table 3 Factor correlation matrices for multidimensional models based on Confirmatory Factor Analysis a Factor correlation matrix (CFA): 4-factor model: Mastery Performance Social Extrinsic Mastery 1 Performance.44 1 Social Extrinsic Social Concern Social Affiliation 8-factor model: Task Effort Competition Social Power Praise Task 1 Effort.67 1 Competition Social Power Social Concern Social Affiliation Praise Token Higher order model: Mastery Performance Social Extrinsic Mastery 1 Performance.54 1 Social Extrinsic Note. a All correlations are significant at α =

28 Table 4 Parameter estimates for the models 1-factor model a 4-factor model a 8-factor model a Higher order model a Scale Mastery motivation: ism03 (task) ism05 (task) ism14 (task) ism16 (task) ism18 (effort) ism23 (effort) ism26 (effort) ism29 (effort) ism30 (effort) Performance motivation: ism04 (competition) ism10 (competition) ism22 (competition) ism32 (competition) ism13 (social power) ism17 (social power) ism20 (social power) Social motivation: ism08 (social concern) ism11 (social concern) ism15 (social concern) ism25 (social concern) ism27 (social concern) ism01 (social affiliation) ism19 (social affiliation) ism28 (social affiliation) Extrinsic motivation: ism09 (praise) ism12 (praise) ism21 (praise) ism24 (praise) ism31 (praise) ism02 (token) ism06 (token) ism07 (token) ism33 (token) For the higher order model: Mastery by task 0.74 Mastery by effort 0.90 Performance by competition 0.98 Performance by social power 0.66 Social by social concern 0.89 Social by social affiliation 0.52 Extrinsic by praise 0.95 Extrinsic by token 0.87 Notes. a All standard errors for the estimates are All loadings are significant at α =