Martin, A.J. (2009). Motivation and engagement across the academic lifespan: A developmental construct

Transcription

1 Martin, A.J. (2009). Motivation and engagement across the academic lifespan: A developmental construct 0 validity study of elementary school, high school, and university/college students. Educational and Psychological Measurement, 69, DOI: / This article may not exactly replicate the authoritative document published in the journal. It is not the copy of record. The exact copy of record can be accessed via the DOI: /

2 Motivation and Engagement Across the Academic Lifespan: 1 A Developmental Construct Validity Study of Elementary School, High School, and University/College Students Andrew J. Martin KEYWORDS: construct validity; developmental; measurement; motivation; engagement; students Author Note. This article was in part prepared while the first author was Visiting Senior Research Fellow in the Department of Education at the University of Oxford. Requests for further information about this investigation can be made to Associate Professor Andrew J. Martin, Faculty of Education and Social Work, A35 Education Building, University of Sydney, NSW 2006, AUSTRALIA. a.martin@edfac.usyd.edu.au.

3 Motivation and Engagement Across the Academic Lifespan: 2 A Developmental Construct Validity Study of Elementary School, High School, and University/College Students Abstract From a developmental construct validity perspective, the present study examines motivation and engagement across elementary school, high school, and university/college with particular focus on the Motivation and Engagement Scale (comprising adaptive, impeding/maladaptive, and maladaptive factors). Findings demonstrated developmental construct validity across the three distinct educational stages in terms of good-fitting first and higher order factors, invariance of factor structure across gender and age, and a pattern of correlations with cognate constructs (e.g., homework completion, academic buoyancy, class participation) consistent with predictions. Notwithstanding the predominantly parallel findings, there was also notable distinctiveness, primarily in terms of mean-level effects such that elementary school students were generally more motivated and engaged than university/college students who in turn were more motivated and engaged than high school students. Implications for motivation and engagement measurement and theory, research in the psycho-educational domain, and the subsequent potential for performance profiling across the academic lifespan are discussed.

4 Motivation and Engagement Across the Academic Lifespan: 1 A Developmental Construct Validity Study of Elementary School, High School, and University/College Students Students within elementary school, high school, and university/college share a great deal in common. In each context students are required to apply themselves over a sustained period of time to develop their academic skills, engage with key performance demands, negotiate the rigors of competition, deal with setback and adversity, cope with possible self-doubt and uncertainty, and develop psychological and behavioral skills to effectively manage the ups and downs in the ordinary course of academic life. Given these congruencies across distinct educational stages, it is feasible to propose that there will be core and common constructs relevant and meaningful across the academic lifespan. The present study seeks to assess this issue in the context of academic motivation and engagement and, more specifically, the validity of recently developed academic motivation and engagement instrumentation in the context of students from elementary school, high school, and university/college. Analyses conducted in the present investigation across these three distinct educational stages are proposed as a developmental construct validity study of academic motivation and engagement. Substantive Background: An Integrative Framework for Motivation and Engagement and Implications for Measurement The substantive background to the study centers on academic motivation and engagement and the need for more pragmatic and integrative approaches to their measurement and theorizing. In critical reviews of motivation and engagement research, it has been suggested that such research oftentimes yields limited practical implications and applications and that there is a need to devise research that advances scientific understanding but which also has applied utility. Hence, there have been calls to give greater attention to use-inspired basic research in education and psychology contexts (Stokes, 1997; see also Greeno, 1998; Pintrich, 2000, 2003). Critical reviews of motivation and engagement research also point to the fact that such research is diverse and fragmented. As a

5 result, there have also been calls for more integrative approaches to its research and theorizing 2 (Bong, 1996; Murphy & Alexander, 2000; Pintrich, 2003). It is in this context that the Motivation and Engagement Wheel (Martin, 2001, 2002, 2007a) was developed. The Wheel is presented in Figure 1. As Figure 1 shows, there are two levels at which the Wheel has been conceptualized: the integrative higher order level comprising four factors and the lower (or first) order level comprising eleven factors. As discussed fully in Martin (2007a, 2008a, 2008b), higher (and first) order factors are adaptive cognitions (self-efficacy, valuing, mastery orientation), adaptive behaviors (planning, task management, persistence), impeding/maladaptive cognitions (anxiety, failure avoidance, uncertain control), and maladaptive behaviors (self-handicapping, disengagement). Initially this Wheel was developed to better understand motivation and engagement amongst high school students; however, in the present study its application to elementary school and university students is assessed from a developmental construct validity perspective (described below). Higher Order Dimensions of Motivation and Engagement Martin (2007a, 2008a, 2008b) proposed that over the past four decades a number of psychological theories and models have been developed that explain the nature of human cognition and behavior. He demonstrated that there are significant commonalities across these theories and models and which provide direction as to fundamental (higher order) dimensions of motivation and engagement. These commonalities operate at three levels. The first delineates cognitive and behavioral elements, including work encompassing cognitive and behavioral orientations in learning strategies (Pintrich & DeGroot, 1990; Pintrich & Garcia, 1991), cognitive antecedents of behavioral strategies used to negotiate environmental demands (Buss & Cantor, 1989), cognitive-behavioral approaches to engagement and behavior change (Beck, 1995), and cognitive-affective and behavioral dimensions to academic engagement (Miller et al, 1996; Miserandino, 1996). The second demonstrates the differential empirical strength of distinct aspects of motivation and engagement for example, self-efficacy reflects highly adaptive motivation (Bandura, 1997; Pajares, 1996), anxiety impedes individuals engagement (Sarason &

6 Sarason, 1990; Spielberger, 1985), and behaviors such as self-handicapping reflect quite maladaptive 3 engagement (Martin, Marsh, & Debus, 2001a, 2001b, 2003; Martin, Marsh, Williamson, & Debus, 2003). The third informs the structure of motivation and engagement frameworks, such as those hypothesizing and empirically demonstrating hierarchical models of human cognition and behavior that encompass specific factors under more global characterizations (e.g., Elliot & Church, 1997; Marsh & Shavelson, 1985; Shavelson, Hubner, & Stanton, 1976). Taken together and in consideration of the joint issues of: motivational and behavioral orientations; cognitive-behavioral frameworks; differing empirical levels of adaptive, impeding, and maladaptive dimensions in applied settings; and, hierarchical models of cognition and behavior, Martin (2007a, 2008a, 2008b) proposed that motivation can be characterized in terms of four higher order dimensions: (a) adaptive cognition, (b) adaptive behavior, (c) impeding/maladaptive cognition, and (d) maladaptive behavior. These dimensions and their component first order factors have been synthesized under the Motivation and Engagement Wheel (Martin, 2001, 2003a, 2003c, 2007a, 2008b) presented in Figure 1. First Order Dimensions of Motivation and Engagement Pintrich (2003) identified core substantive questions for the development of a motivational science. Taken together, these questions underscore the importance of considering, conceptualizing, and articulating a model of motivation from salient and seminal theorizing related to: self-efficacy, control, valuing, goal orientation, need achievement, self-worth, and self-regulation. These, it is suggested, provide a useful heuristic for the identification of first order constructs for operationalizing the Motivation and Engagement Wheel. As discussed fully in Martin (2001, 2002, 2003c, 2007a), (a) self-efficacy theory (e.g., Bandura, 1997) is reflected in the self-efficacy dimension of the Wheel, (b) attributions and control are reflected in the uncertain control dimension (tapping the controllability element of attributions see Connell, 1985; Weiner, 1994), (c) valuing (e.g., Eccles, 1983; Wigfield & Tonks, 2002) is reflected in a valuing dimension, (d) self-determination (in terms of intrinsic motivation see Ryan & Deci, 2000) and motivation orientation (see Dweck, 1986; Martin & Debus, 1998;

7 Nicholls, 1989) are reflected in a mastery orientation dimension, (e) self-regulation (e.g., Martin, , 2002, 2003c, 2007a; Martin et al., 2001a, 2001b, 2003; Zimmerman, 2002) is reflected in planning, task management, and persistence dimensions, and (f) need achievement and self-worth (e.g., Atkinson 1957; Covington, 1992; Martin & Marsh, 203; McClelland, 1965) are reflected in failure avoidance, anxiety, self-handicapping, and disengagement dimensions, and Hence, the Wheel comprises eleven lower or first-order dimensions see Figure 1. Measurement and the Motivation and Engagement Scale Alongside the Motivation and Engagement Wheel is its accompanying instrumentation the Motivation and Engagement Scale (MES). Typically administered to high school students, the Motivation and Engagement Scale High School (MES-HS; Martin, 2001, 2003c, 2007a, 2007b, 2008a) demonstrates a strong factor structure that is invariant across gender and age (but there are mean-level differences such that females generally report higher levels of motivation than males and middle high school students report lower motivation than junior and senior high school students) and is reliable and normally-distributed. It has also been found to predict a variety of educational outcomes such as enjoyment of school, classroom participation, educational aspirations as well as achievement-related outcomes such as school grades. To extend this line of research, the present investigation assesses a parallel form of the MES using the Motivation and Engagement Scale Junior School (MES-JS) and Motivation and Engagement Scale University/College (MES-UC). Over the past few years, there has been growing research around the Motivation and Engagement Wheel and its accompanying instrumentation, the Motivation and Engagement Scale. The MES is robust in high school (Martin, 2007a), workplace (Martin, in press b; see also Martin 2005b, 2005c), music (Martin, 2008b), sport (Martin, 2008b), and physical activity domains (Martin, Tipler and colleagues, 2006). The Wheel and MES are useful as bases for educational intervention (Martin, 2005a, 2008b). The Wheel and MES are helpful foundations for assessing group-level (climate) effects (Martin & Marsh, 2005). Finally, the Wheel and MES are useful in

8 addressing more specific educational issues such as domain specificity (Green, Martin, & Marsh, ), teacher effects (Martin & Marsh, 2005), and the role of parents and teachers in the motivation and engagement process (Martin, 2003b, 2006). However, to date, there has been no thoroughgoing and detailed scoping of the Wheel and MES across the span of education that is, across elementary school, high school, and universities samples (but see Martin, in press b, for brief research in the context of sport, music, work, and daily life motivation and engagement). The present study does so from a proposed developmental construct validity perspective. Methodological Background: A Developmental Construct Validity Perspective Researchers in psychology and education have increasingly emphasized the need to develop and evaluate instruments within a construct validation framework (e.g., see Marsh, 2002; Marsh & Hau, 2007). Investigations that adopt a construct validation approach can be classified as withinnetwork or between-network studies. Moreover, it is proposed here that when construct validity is assessed across distinct educational stages it constitutes something of a developmental construct validity perspective. Specifically, it is proposed that a dual within- and between-network approach across elementary school, high school, and university represents a developmental construct validity approach to assessing the generality of motivation and engagement across the academic lifespan. Within-network Validity Beginning with a logical analysis of internal consistency of the construct definition, measurement instruments, and generation of predictions, within-network studies typically employ empirical techniques such as exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and reliability analysis. The present study conducts within-network analyses across the three samples using confirmatory factor analysis to test the multidimensional motivation and engagement framework and reliability analysis to test the internal consistency of scores. Consistent with previous studies of high school students (e.g., Green et al, 2007; Martin, 2001, 2003c, 2007a) and across diverse performance settings such as music and sport (Martin, 2008b), it is hypothesized that at each educational stage (elementary school, high school, and university), the motivation and

9 framework instrumentation (MES) will evince a sound first and higher order factor structure and 6 comprise reliable scores. Between-network Validity Between-network research explores relationships between a target central framework and a set of factors external to the framework. It typically does so through statistical procedures such as correlation, regression, or structural equation modeling (SEM) analyses to examine relationships between measures and instruments. The present study conducts between-network analyses across the three samples by assessing: (a) the invariance of factor structure across gender, age groups, and educational stages (elementary school, high school, university/college), (b) mean-level differences across educational stages, and (c) the empirical links between the hypothesized first and higher order factors and a set of cognate between-network measures (enjoyment of school/university, class participation, positive intentions, academic buoyancy, homework/assignment completion). Each of these between-network techniques is described in turn. Factorial invariance in the structure of motivation and engagement. As described in Martin (2007a, 2008b), insufficient attention is given to analyses of factor structure of motivation and engagement and the extent to which a given motivation and engagement instrument and its components are invariant across different groups. Such concerns about factor structure invariance are most appropriately evaluated using CFA to determine whether and how the structure of motivation and engagement vary according to key sub-populations (see Hattie, 1992; Marsh, 1993). Martin (2004, 2007a) has previously shown the MES factor structure (factor loadings, uniquenesses, correlations/variances) to be invariant across early-, mid-, and late-adolescent samples and also across gender. The present study is an opportunity to assess invariance across gender and age within elementary school and university. It is also an opportunity to assess invariance across elementary school, high school, and university samples. Consistent with previous studies of high school students (e.g., Green et al, 2007; Martin, 2001, 2003c, 2007a) and across diverse performance settings such as music and sport (Martin, 2008b), it is hypothesized that factor

10 structure (including loadings, correlations/variances, and uniquenesses) across gender, age, and 7 educational stage will evince relative invariance. Mean-level educational stage effects. Very little research has assessed mean levels of motivation and engagement across the academic lifespan: elementary school, high school, and university. The transition from elementary to middle school has been found to pose difficulties and challenges unique to that time (Anderman & Midgley, 1997; Roeser, Eccles, & Sameroff, 2000) and a decline in student motivation and engagement is typically found to emerge after this transition (see Martin, 2001, 2003c, 2004, 2007a; Wigfield & Tonks, 2002) including changes in subjective task value (Wigfield, Eccles, Mac Iver, Reuman, & Midgley, 1991). As students move on to university/college some research has found them to be more confident in the quantity and quality of their abilities whereas other research finds it a difficult transition with less support and structure and a major challenge in asserting one s identity amongst highly capable peers (Martin, Marsh, Williamson, & Debus, 2003). Increasingly, universities and colleges are recognizing the stresses and strains of undergraduate life and the difficulties in making a successful transition from high school (see Martin, Milne-Home, Barrett, & Spalding, 1997; Martin, Milne-Home, Barrett, Spalding, & Jones, 2000). Indeed, Martin and colleagues (Martin, Marsh, Williamson, & Debus, 2003) have found university to present distinct challenges that instill doubts and uncertainties that in some cases lead to self-handicapping, poorer academic performance, and eventual dropout. Taken together, then, it is hypothesized that elementary school students will evince relatively higher mean levels of motivation and engagement than high school and university samples, however, no predictions are made regarding the relative mean levels of the latter two groups. Motivation, engagement, and cognate correlates. Consistent with the construct validity approach, it is proposed that five between-network constructs provide a theoretically relevant basis for examining the external validity of the MES across the academic lifespan: positive intentions, class participation, enjoyment of school, academic buoyancy, and homework/assignment completion. In terms of positive intentions, several researchers have shown that students higher in motivation and engagement are more likely to take advanced or optional courses and also more

11 likely to report future course enrolment intentions (Meece, Wigfield, & Eccles, 1990). In addition to positive intentions, class participation is deemed a feasible between-network construct. Learning environments that foster student participation are found to enhance students commitment to learning (Richter & Tjosvold, 1980) while a lack of participation is found to lead to unsuccessful educational outcomes such as emotional withdrawal and poor identification with the school (Finn, 1989). Enjoyment of school is another feasible between-network construct. Elliot and Sheldon (1997), for example, included enjoyment as one of the five key variables in their study of goal pursuit. Even research in higher education finds that enjoyment is a key factor in students engagement at university (Lee, Sheldon & Turban, 2003). Martin and Marsh (2006, 2008a, 2008b) have shown academic buoyancy to be a factor relevant to students ability to deal with academic setback in the ordinary course of academic life and also shown a variety of motivation and engagement factors to be significantly associated with such buoyancy. It is also proposed that in addition to these four intra-psychic measures, there is a need for more behavioral measures (Green et al., 2007) that in the present study takes the form of homework/assignment completion. Consistent with previous studies of high school students (e.g., Green et al, 2007; Martin, 2001, 2003c, 2007a) and across diverse performance settings such as music and sport (Martin, 2008b, in press a, in press b), it is hypothesized that the adaptive dimensions will be positively (to a modest or strong degree) associated with these correlates, the impeding/maladaptive dimensions will be associated at near-zero or negatively (to a weak or modest degree), whilst maladaptive dimensions will be more markedly negatively (to a modest or strong degree) associated with these correlates. Aims of the Present Study The overarching aim of the present study is to examine the developmental construct validity of motivation and engagement across elementary school, high school, and university samples. More specifically, the present study assesses a recently developed integrative motivation and engagement instrumentation across the academic lifespan with a view to assessing: (a) within-network validity in terms of first and higher order factor structure and reliability and (b) between-network validity in 8

12 terms of invariance of factor structure across groups (gender, age, educational stage), mean-level 9 differences across educational stage, and associations with cognate correlates. Method Elementary School Sample and Procedure The elementary school sample comprised 624 upper-age elementary students in five schools. All schools were located in urban areas drawing from two capital cities in Australia. Students were aged 9 to 11.5 years (N=114, 56% females and 44% males) and 11.5 years to 13 years (N=510, 38% females and 62% males). The mean age of students was (SD=.69) years. Teachers read the Motivation and Engagement Scale Junior School (MES-JS; Martin, 2007b) items aloud to students during class or pastoral care/tutorial groups. The rating scale was first explained and sample items were presented. Students were then asked to complete the instrument as the teacher read out each item in turn and to return the completed form to the teacher at the end of class or pastoral care/tutorial group. Previous work has been conducted in a smaller urban/rural elementary school sample (Martin, Craven, & Munns, 2006), however, this work only comprised a factor analysis of the MES-JS with no invariance testing, mean-level analyses, analyses in the context of the academic lifespan, and external validity checks. The present study, then, is a significant progression on previous work. High School Archive Sample and Procedure The high school sample comprised data collected from 21,579 high school students from 58 Australian schools. Thirty-six schools were government and 22 schools were independent, from urban and regional areas across most states in Australia. Students were aged 12 years to 13 years (N=6,640, 49% females and 51% males), 14 years to 15 years (N=7,894, 43% females and 57% males), and 16 years to 18 years (N=7,045, 44% females and 56% males). The mean age of students was (SD=1.57) years. The high school sample is something of an archive sample that has been compiled over recent years across numerous research projects. Portions of the data have been reported on

13 elsewhere with a more substantial construct validity study by Martin (2007a) assessing the 10 Motivation and Engagement Scale High School (MES-HS) amongst 12,237 high school students, all of whom are included as part of the present archive sample of 21,579 students. The reader is urged to consult Martin (2007a; see also Martin, 2008b, in press a, in press b) for these academic motivation and engagement data in the context of other performance domains such as sport, music, and work) as the first substantial large-sample investigation into the MES-HS. The archive dataset represents the integration of data collected over the previous five years and so can be considered to be relatively current. Teachers administered the MES-HS (Martin, 2001, 2003c, 2007a, 2007b) to students during class or pastoral care/tutorial groups. The rating scale was first explained and sample items were presented. Students were then asked to complete the instrument on their own and to return the completed form to the teacher at the end of class or pastoral care. University Sample and Procedure University (college) respondents were 420 undergraduate students from two Australian universities. One university is well-established and one of the oldest in the country (68% of sample). The other is a more recently established institution (32%). Most respondents were female (80%), with 20% male. Most students were enrolled in education (66%), with other students enrolled in arts (18%), psychology/social science (8%), social work (3%), science (3%), and communications (2%). Most were full-time students (96%), with 4% part-time. Most were in their first year of study (65%), with 25% in second year, 7% in third year, and 3% in fourth or fifth year. The mean age of students was (SD=6.62) years, with 60% under 20 years of age and 40% 20 years and over. Students completed the instrument in lecture or tutorial time. Students were asked to complete the Motivation and Engagement Scale University/College (MES-UC; Martin, 2007b) on their own and return the completed instrument at the end of the lecture or tutorial they were attending at the time. Motivation and Engagement Scale Materials

14 General overview. The Motivation and Engagement Scale Junior School (MES-JS; Martin, 2007b), Motivation and Engagement Scale High School (MES-HS; Martin, 2001, 2003c, 2007a, 2007b), and Motivation and Engagement Scale University/College (MES-UC, Martin, 2007b) are instruments that measure elementary, high school, and university students motivation and engagement respectively. Adapted from the MES-HS, the MES-JS and MES-UC assess motivation and engagement through three adaptive cognitive dimensions (self-efficacy, valuing, mastery orientation), three adaptive behavioral dimensions (persistence, planning, task management), three impeding/maladaptive cognitive dimensions (anxiety, failure avoidance, uncertain control), and two maladaptive behavioral dimensions (self-handicapping, disengagement). Each of the eleven factors comprises four items hence the MES is a 44-item instrument. The MES-JS and MES-UC comprise the same number of items (44) and the same number of first order (11) and higher order (4) factors as the original high school instrument (MES-HS). As much as possible, item adaptation aimed to make simple and transparent word and terminology changes in order to remain very parallel to the high school form. In the Appendix a sample item from the MES- HS is presented along with its MES-JS and MES-UC adaptations (see Martin, 2007a for a full account of the origins of and rationale for the scale and item development). To simplify the survey for younger students the MES-JS asks students to rate themselves on a shorter scale of 1 ( Strongly Disagree ) to 5 ( Strongly Agree ) whereas for the MES-HS and MES-UC, students rate themselves on a scale of 1 ( Strongly Disagree ) to 7 ( Strongly Agree ). In most studies using the MES (e.g., Martin, 2007a, 2008a, 2008b, in press a), the 7-point rating scale is typically used. However, the elementary school sample posed a distinct challenge in that a simpler survey form was desirable: pilot work indicated students had difficulty teasing apart the finer-grained rating points on the 7- point scale. Adaptive cognitive and behavioral dimensions. Each adaptive dimension falls into one of two groups: cognitions and behaviors. Adaptive cognitions include self-efficacy, mastery orientation, and valuing. Adaptive behaviors include persistence, planning, and task management. Self-efficacy is students belief and confidence in their ability to understand or to do well in their 11

15 school/university work, to meet challenges they face, and to perform to the best of their ability. 12 Valuing of school/university is how much students believe what they do and learn at school/university is useful, important, and relevant to them. Mastery orientation entails being focused on understanding, learning, solving problems, and developing skills. Planning is how much students plan their work and how much they keep track of their progress as they are doing it. Task management refers to the way students use their time, organize their timetable, and choose and arrange where they prepare for school/university and school/university tasks. Persistence reflects students capacity to persist in situations that are challenging and at times when they find it difficult to do what is required. Impeding and maladaptive cognitive and behavioral dimensions. Impeding/maladaptive cognitive dimensions are anxiety, failure avoidance, and uncertain control. Anxiety has two parts: feeling nervous and worrying. Feeling nervous is the uneasy or sick feeling students get when they think about their school/university work or school/university tasks. Worrying is their fear of not doing very well in their school/university work. Failure avoidance occurs when the main reason students try at school/university is to avoid doing poorly or to avoid being seen to do poorly. Uncertain control assesses students uncertainty about how to do well or how to avoid doing poorly. Maladaptive behavioral dimensions are self-handicapping and disengagement. Selfhandicapping occurs when students reduce their chances of success at school/university. Examples are engaging in other activities while they are meant to be doing their school/university work or preparing for upcoming school/university work tasks. Disengagement occurs when students give up or are at risk of giving up at school/university or in particular school/university activities. Between-network Correlates Students were also administered items that explored their enjoyment of school/university (4 items; e.g., elementary school item: I like school, Cronbach s =.94; high school item: I like school, =.91; university item: I like university, =.91), class participation (4 items; e.g., elementary school item: I get involved in things we do in class, =.90; high school item: I get

16 13 involved in things we do in class, =.90; university item: I get involved in things we do in class, =.93), positive intentions (4 items; e.g., high school item: I intend to complete school, =.82; university item: I intend to complete university, =.72), and academic buoyancy (4 items; e.g., elementary school item: I think I m good at dealing with schoolwork pressures, =.78; high school item: I think I m good at dealing with schoolwork pressures, =.80; university item: I think I m good at dealing with university pressures, =.84). These measures were rated on a 1 ( Strongly Disagree ) to 7 ( Strongly Agree ) scale and were adapted directly from Martin (2007a, 2008b; see also Martin & Marsh, 2006, 2008a, 2008b) who has shown them to be reliable, a good fit to the data in confirmatory factor analysis, and significantly associated with motivation and engagement in other performance domains such as sport and music. Homework/assignment completion ( How often do you do and complete your assignments? ) was a single item assessed on a 1 ( Never ) to 5 ( Always ) rating scale. Confirmatory Factor Analysis and Structural Equation Modeling Confirmatory factor analysis (CFA) and structural equation modeling (SEM), performed with LISREL 8.80 (Jöreskog & Sörbom, 2006), were used to test the hypothesized models. In CFA and SEM, the researcher posits an a priori structure and tests the ability of a solution based on this structure to fit the data by demonstrating that: (a) the solution is well defined, (b) parameter estimates are consistent with theory and a priori predictions, and (c) the subjective indices of fit are reasonable (McDonald & Marsh, 1990). Maximum likelihood was the method of estimation used for the models. In evaluating goodness of fit of alternative models, the root mean square error of approximation (RMSEA) is emphasized as are the comparative fit index (CFI), the non-normed fit index (NNFI), and an evaluation of parameter estimates. For RMSEAs, values at or less than.05 and.08 are taken to reflect a close and reasonable fit respectively (see Jöreskog & Sörbom, 1993). The CFI and NNFI vary along a 0 to 1 continuum in which values at or greater than.90 and.95 are typically taken to reflect acceptable and excellent fits to the data respectively (McDonald & Marsh,

17 1990). The CFI contains no penalty for a lack of parsimony whereas the RMSEA contains penalties for a lack of parsimony. 14 Missing Data For large-scale studies, the inevitable missing data is a potentially important problem, particularly when the amount of missing data exceeds 5% (e.g., Graham & Hoffer, 2000). A growing body of research has emphasized potential problems with traditional pairwise, listwise, and mean substitution approaches to missing data (e.g., Graham & Hoffer, 2000), leading to the implementation of the Expectation Maximization Algorithm, the most widely recommended approach to imputation for missing data that are missing at random, as operationalized using missing value analysis in LISREL. In fact, less than 5% of the MES data were missing in each of the elementary school, high school, and university samples and so the EM Algorithm was implemented for all samples. Also explored were alternative approaches to this problem which showed that results based on the EM algorithm used here were very similar to those based on the traditional pairwise deletion methods for missing data as would be expected to be the case when there was so little missing data. Multi-group CFA and Tests of Invariance Two broad sets of invariance tests were conducted. The first assessed invariance within samples. The second assessed invariance between samples. For the within-sample invariance tests, for each of elementary school, high school, and university, multi-group CFAs were conducted to assess invariance across gender and age. For the between-sample invariance tests, three invariance analyses were conducted that between high school and university on the original 7-point rating scale, that between elementary school, high school, and university using a common 5-point rating scale (reliabilities for the transformed 5-point variables: high school range = ; university range =.66 to.86), and that between elementary school and university on a common 5-point rating scale (the common 5-point rating scale was derived by aggregating the first and last two points of the 7-point rating scale). Although the chi-square difference test is the most

18 straightforward means of assessing differences between nested models, problems associated with such tests exist (e.g., see McDonald & Marsh, 1990; Tabachnick & Fidell, 1996). Hence, in formally assessing differences in models, emphasis is given to differences in fit indices (Cheung & Rensvold, 2002). 15 Multiple-Indicator-Multiple-Cause (MIMIC) Models Notwithstanding the importance of testing for invariance in factor structure, there is also reason to investigate the mean-level developmental effects on the eleven facets of the MES-JS, MES-HS, and MES-UC. Kaplan (2000) suggested the multiple-indicator multiple-cause (MIMIC) approach, which is similar to a regression model in which latent variables (e.g., multiple dimensions of motivation and engagement) are caused by discrete grouping variables (e.g., educational stage) that are represented by single indicators. This MIMIC model assessed the role of educational stage (elementary school, high school, university) as a predictor of motivation and engagement. Being a multinomial predictor and using high school as the reference point, educational stage was represented by two dummy variables: high school (0) vs elementary school (1) and high school (0) vs university (1) hence, positive beta weights for both dummy variables indicate higher scores for elementary school and university students compared to high school students and negative beta weights for both dummy variables indicate lower scores for elementary school and university students compared with high school students. Results First and Higher Order Confirmatory Factor Analysis (CFA) In the first instance, an 11-factor model was examined using CFA. The CFA yielded a very good fit to the data for elementary school ( 2 = 1,881.10, df = 847, p <.001, CFI =.98, NNFI =.97, RMSEA =.04), high school ( 2 = 28,217.75, df = 847, p <.001, CFI =.98, NNFI =.98, RMSEA =.04), and university ( 2 = 1,697.75, df = 847, p <.001, CFI =.96, NNFI =.95, RMSEA =.05). Factor loading ranges and means are presented in Table 1. Taken together, for all three samples the loadings are acceptable. This is supported by the acceptable reliability coefficients (e.g., see

19 Henson, 2001) also presented in Table 1. Correlations for the sample are presented in Table Predictably, for the three samples all adaptive dimensions were strongly (significantly) positively correlated and correlated strongly (significantly) negatively with maladaptive dimensions and slightly (but significantly) negatively or at near-zero with impeding/maladaptive dimensions. Maladaptive dimensions were markedly (significantly) positively correlated as were impeding/maladaptive dimensions. For the three samples, all correlations indicate lower levels of shared variance between factor groupings than within factor groupings. In addition to the first order dimensions constituting the eleven facets of the Motivation and Engagement Wheel, there is also hypothesized a higher order structure delineated by adaptive cognitive dimensions, adaptive behavioral dimensions, impeding/maladaptive cognitive dimensions, and maladaptive behavioral dimensions. In higher order models, correlations between first order dimensions are constrained to be zero and relations among these first order dimensions are explained in terms of higher order dimensions. For each of elementary school, high school, and university samples, the higher order CFAs comprised the 44 items, the 11 first order dimensions, and the four higher order dimensions. The higher order elementary school structure fit the data very well ( 2 = 2,155.87, df = 886, p <.001, CFI =.97, NNFI =.97, RMSEA =.05), as did the higher model for high school students ( 2 = 36,732.07, df = 886, p <.001, CFI =.98, NNFI =.98, RMSEA =.04) and university students ( 2 = 1,968.82, df = 886, p <.001, CFI =.95, NNFI =.94, RMSEA =.05). Table 2 presents higher order correlations which broadly confirm cluster correlations in the first order model. Multi-group Confirmatory Factor Analysis and Invariance Tests Eight models were tested in each of the multi-group CFAs assessing invariance of factor structure across gender, age, and educational stage. The initial five models related to the first order factor structure. The first model allowed all factor loadings, uniquenesses, and correlations to be freely estimated; the second held first order factor loadings invariant across groups; the third held first order factor loadings and correlations/variances invariant; the fourth held first order factor

20 loadings and uniquenesses invariant, and the fifth held first order factor loadings, uniquenesses, and correlations/variances invariant. The final three models focused on invariance of higher order loadings and correlations/variances: the sixth freely estimated the higher order loadings and correlations/variances, the seventh held higher order loadings invariant, and the eighth held higher order loadings and correlations/variances invariant. 17 Within sample invariance tests. For elementary school, results in Table 3 indicate that when successive elements of the first and higher order factor structure are held invariant across groups, the fit indices are predominantly comparable across (Table 3 also indicates 2, df, and p values): (a) males and females (ranges: CFIs=.97 for first order and.96 for higher order solutions; NNFIs=.98 for first order and.97 for higher order solutions; RMSEAs=.05 for first and higher order solutions) and (b) younger ( years) and older ( years) students (ranges: CFIs=.97 for first order and.96 for higher order solutions; NNFIs=.96 for first and higher order solutions; RMSEAs=.05 for first and higher order solutions). For high school, the fit indices are predominantly comparable across: (a) males and females (ranges: CFIs=.98 for first order and.97 for higher order solution; NNFIs=.98 for first order and.97 for higher order solution; RMSEAs=.04 for first order and higher order solutions) and (b) early- (12-13 years), mid- (14-15 years), and late- (16-18 years) adolescence (ranges: CFIs=.98 for first order and.97 for higher order solution; NNFIs=.98 for first order and.97 for higher order solution; RMSEAs=.04 for first order and higher order solutions). For university, the fit indices are predominantly comparable across: (a) males and females (ranges: CFI=.93 to.94 for first order and.92 to.93 for higher order solution; NNFIs=.93 for first order and.92 for higher order solution; RMSEAs=.06 for first order and higher order solutions) and (b) younger (17-19 years) and older (20+ years) students (ranges: CFIs=.94 for first order and.92 for higher order solution; NNFIs=.93 for first order and.92 for higher order solution; RMSEA=.05 to.06 for first order and.06 for higher order solutions). For all three samples, the application of

21 recommended criteria for evidence of lack of invariance (i.e., a change of 0.01 in fit indices see Cheung & Rensvold, 2002) indicates that there is invariance across groups. Between sample invariance tests. The final set of invariance tests assessed first and higher order factor structure across elementary school, high school, and university samples. This is a direct assessment of the generalizability of the framework and measurement across diverse settings. Fit indices in Table 4 (Table 4 also indicates 2, df, and p values) show that when successive elements of the factor structure are held invariant across high school and university samples on the original 7- point rating scale (ranges: CFIs and NNFIs=.98 for first order and higher order solutions; RMSEAs=.04 for first order and higher order solutions), there is invariance across all first order and higher order parameters. In terms of elementary school, high school, and university samples on a common 5-point scale (the common 5-point rating scale was derived by aggregating the first and last two points of the 7-point rating scale), there is also invariance across the three samples (ranges: CFIs and NNFIs=.98 for first order and higher order solutions; RMSEAs=.04 for first order and higher order solutions). Finally, when assessing invariance between elementary school and university samples (thereby omitting the extremely large high school sample which could bias invariance findings), there is also evidence of invariance when aspects of factor structure (loadings, correlations/variances, uniquenesses) are systematically constrained to be equal (ranges: CFI =.96 to.97 for first order and.96 for higher order solution; NNFI =.96 to.97 for first order and.96 for higher order solution; RMSEAs =.05 for first order and higher order solutions). For each of these three sets of between-sample invariance tests, the application of recommended criteria for evidence of lack of invariance (i.e., a change of 0.01 in fit indices) indicates that there is invariance across elementary school, high school, and university domains. Multiple-Indicator Multiple-Cause (MIMIC) Modeling The previous analyses explored possible differences in factor structure as a function of educational stage. It was also of interest to explore possible mean-level differences in motivation and engagement as a function of educational stage (elementary school, high school, university). Multiple-indicator multiple-cause (MIMIC) modeling was the analytical method used to examine 18

22 this and involved structural equation models in which educational stage was used as a predictor of the first and higher order factors of the Wheel. The first order model yielded a good fit to the data ( 2 = 39,347.85, df = 914, p <.001, CFI =.95, NNFI =.94, RMSEA =.04) as did the higher order model ( 2 = 45,508.66, df = 966, p <.001, p <.001, CFI =.95, NNFI =.94, RMSEA =.05). Beta coefficients are presented in Table 1 along with the main effects for educational stage. Results show that there are significant stage differences on all motivation and engagement factors. Compared with high school students, elementary school and university students are significantly higher on all adaptive dimensions. Also, compared with high school students, elementary school and university students are significantly lower in uncertain control, self-handicapping, and disengagement. However, compared to high school students, elementary school and university students are significantly higher on anxiety and failure avoidance. As a general finding, there is a greater difference between elementary and high school students than between high school and university students. Again, however, to note is that the high school and university 1-7 rating continuum was transformed to a 1-5 rating continuum to place them on the same scale of measurement as elementary school hence, caution is advised when interpreting these findings. Due to the large high school sample, caution is also advised when interpreting the significance of the MIMIC results and this being the case, greater emphasis is given to findings in relation to self-efficacy, mastery orientation, valuing of school, planning, task management, persistence, uncertain control, and selfhandicapping that yielded standardized beta values greater than Motivation, Engagement, and Between-network Cognate Correlates As indicated earlier, consistent with the between-network construct validity approach, it was of interest to explore the nature of relationships between each facet of motivation and a set of key between-network correlates across the three educational stages. To this end, the three samples were also administered items that explored enjoyment of school/university (elementary school, high school, university), class participation (elementary school, high school, university), positive academic intentions (high school, university), academic buoyancy (elementary school, high school,

23 university), and homework completion (high school, university). For each of the three samples, first and higher order CFAs were conducted. The first order elementary school CFA yielded a very good fit to the data ( 2 = 2,915.33, df = 1393, p <.001, CFI =.98, NNFI =.98, RMSEA =.04) and showed that: (a) adaptive dimensions are significantly positively associated with these between-network constructs and (b) impeding/maladaptive and maladaptive dimensions (particularly uncertain control, selfhandicapping, and disengagement) are negatively correlated with these constructs. Table 5 presents findings. These first order findings were broadly supported in the high school sample ( 2 = 52,112, df = 1650, p <.001, CFI =.98, NNFI =.98, RMSEA =.04) and the university sample ( 2 = 3,251.39, df = 1650, p <.001, CFI =.96, NNFI =.96, RMSEA =.05). Interestingly and consistent with Martin (2007; see also Martin & Marsh, 2006, 2008a, 2008b) academic buoyancy is a notable exception in being more markedly correlated with impeding/maladaptive cognitions than maladaptive behaviors largely a function of its very high correlation with anxiety (discussed fully in Martin & Marsh, 2006, 2008a, 2008b). Again, however, due to the large high school sample caution is advised when interpreting the correlations emphasis is given to the size and direction of the correlation coefficients themselves rather than their significance levels. The higher order factor analysis for elementary school ( 2 = 3,361.64, df = 1453, p <.001, CFI =.97, NNFI =.97, RMSEA =.05) provides general support for the first order findings. Higher order correlations are also presented in Table 5 (again, due to the large samples involved, emphasis is given to the size and direction of the correlation coefficients themselves rather than their significance levels). Consistent with the elementary school findings, the higher order factor analysis for high school ( 2 = 67,868.55, df = 1724, p <.001, CFI =.98, NNFI =.98, RMSEA =.04) provides support for the first order findings as did the higher order model for the university sample ( 2 = 3,683.58, df = 1724, p <.001, CFI =.95, NNFI =.95, RMSEA =.05). Discussion 20

24 Through the integration of multivariate measurement and the hypothesized motivation and engagement framework, the study supports the developmental construct validity of motivation and engagement at elementary school, high school, and university/college levels. From this developmental construct validity perspective, perhaps the most significant yield of the present study is the predominantly comparable findings across three very distinct educational stages. The data confirm the hypothesized generality of the Wheel and its accompanying instrumentation amongst very young students in elementary school through to mature age students in university. In some ways the most revealing tests were the multi-group invariance analyses across the elementary school, high school, and university samples. These analyses directly addressed the question posed at the outset of the study regarding the generality of the proposed motivation and engagement framework in diverse educational settings. The invariance data suggested that there is generality and developmental validity of the framework across the academic lifespan. Notwithstanding the important consistencies across the three educational stages, findings also suggest issues distinct to each academic setting. For example, the data showed that elementary school students reflect higher levels of motivation and engagement and this is consistent with prior work showing declines between elementary and middle/high school (e.g., Anderman & Midgley, 1997; Roeser et al, 2000; Wigfield et al, 1991; Wigfield & Tonks, 2002). In terms of university students, there was some question as to their level of motivation relative to school students with some research recognizing the challenges they face in higher education and other research reporting on their confidence in their abilities (e.g., see Martin, Marsh, Williamson, & Debus, 2003; Pitts, 2005). The present data shed light on these competing views by showing that, notwithstanding equivalence in factor structure, university students reflect higher mean levels of motivation and engagement than their high school counterparts. In the case of all MIMIC analyses, however, due to the large samples involved emphasis is given to the size and direction of the standardized beta coefficients rather than the attained significance levels. 21 Because the constructs within the Wheel have a theoretical basis, researchers are able to draw on theory to provide direction for intervention aimed at addressing facets within the Wheel.