The Reliability and Validity of the Goal Orientation and Learning Strategies Survey (GOALS-S): A Filipino Investigation

Transcription

1 The Asia-Pacific Education Researcher 20:3 (2011), pp The Reliability and Validity of the Goal Orientation and Learning Strategies Survey (GOALS-S): A Filipino Investigation Ronnel B. King,* David A. Watkins The University of Hong Kong *ronnel.king@gmail.com The Goal Orientation and Learning Strategies Survey (GOALS-S; Dowson & McInerney, 2004) is an instrument designed to assess four sets of constructs: (1) academic goals, (2) social goals, (3) cognitive strategies, and (4) metacognitive strategies of high school students. This instrument was initially developed and validated among students in Australia. The applicability of this instrument to the Philippine setting was tested in a study involving 1,147 Filipino students from Metro Manila. Responses to this questionnaire are shown to have good internal consistency reliability. Support is provided for its within-network construct validity in terms of its factorial structure and evidence of its between-network construct validity is shown through its correlations with other valued educational outcomes. Different multigroup confirmatory factor analysis likewise indicated that the instrument is invariant across genders, across year levels, and across type of school. Suggestions for further research using the GOALS-S are provided. Keywords: GOALS-S, validation, confirmatory factor analyses, Filipino students Students academic goals (Covington, 2000; Elliot, 2005; Kaplan & Maehr, 2007; Maehr & Zusho, 2009), social goals (Urdan & Maehr, 1995; Wentzel, 2000), cognitive strategies (Fredricks, Blumenfeld, & Paris, 2004; Lau, Liem, & Nie, 2008; Marton, Dall Alba, & Kun, 1996; Pressly & Harris, 2006), and metacognitive strategies (Hacker, 1998; Schunk, 2001; Wright & Jacobs, 2003) have all been implicated as crucial for successful learning in school. Thus the accurate measurement of these constructs is important for both educational researchers and practitioners. However, there has been a dearth of instruments that measured all these constructs in one scale. Most of the existing scales only measured one or two of these constructs together (e.g., Elliot & McGregor, 2001; Midgley et al., 2000; Pintrich & DeGroot, 1990). As such, most educational researchers are forced to use different instruments to assess constructs relevant to their research. The use of different instruments, however, may cause some problems because the psychometric properties of these instruments might not be fully known before data collection. Thus Dowson and McInerney (2004) developed the Goal Orientation and Learning Strategies Survey (GOALS-S) to remedy this absence of a coherent set of measures that assesses all these sets of constructs in a single instrument. GOALS-S is an 84-item self-report questionnaire that assesses four sets of constructs that are further divided into different subscales: (1) academic goals (mastery goals, performance goals, and work avoidance goals), (2) social goals (social affiliation goals, social approval goals, social concern goals, social responsibility goals, and social status goals), (3) cognitive strategies (elaboration, organization, and rehearsal), and (4) metacognitive strategies (monitoring, planning, and regulating). An advantage of GOALS-S is that it measures a multiplicity of students goals. It incorporates three kinds of academic goals (mastery, performance, and work avoidance) and five kinds of social goals (social Copyright 2011 De La Salle University, Philippines

2 580 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 affiliation, social approval, social concern, social responsibility, and social status goals). This is one of the few instruments that measure a wide range of goals, both social and academic in a single instrument (see McInerney & Ali, 2006 for an exception). Most of the existing instruments only focused on academic goals usually mastery and performance goals (e.g., Midgley et al., 2000, Elliot & McGregor, 2001; Elliot & Murayama, 2008). This narrow focus has been criticized for being overly restrictive since it neglects social goals which have also been shown to be salient in motivating students (Ford, 1996; Martin & Dowson, 2009; Urdan & Maehr, 1995; Wentzel, 1996, 2000). In addition, the measurement of social goals becomes even more relevant when studying students from collectivist cultures where the social dimension of motivation is more salient (Kumar & Maehr, 2007; Salili, 1996; McInerney, 2008; Yu & Yang, 1994). Although there have been some previous instruments that did measure social goals, they usually included only one (e.g., Chang & Wong, 2008; Cheng, 2005; Tao, 2003; Yu & Yang, 1994) or two types of social goals (e.g., McInerney & Ali, 2006; Miller, Greene, Montalvo, Ravindran, & Nichols, 1996). This can be considered a disadvantage given that a lot of qualitative studies have indicated that students pursue a multiplicity of social goals (Dowson & McInerney, 2001, 2003; Lemos, 1996; Mansfield, 2009). In addition, these existing instruments usually fail to make distinctions between various types of social goals. For example, Chang and Wong s (2008) Social Achievement Goal Scale combines items such as I study in order to avoid being looked down upon by my peers and I study so my teachers praise me. Whereas the first item pertains more to peers or friends, the latter one focuses more on teachers. This can be considered a weakness of the instrument because social goals pertaining to peers may have differential effects from those involving teachers. As Urdan and Maehr (1995) cogently argued, There is a critical need to untangle the many constructs represented by the term social goals (p. 232). Therefore the capacity of GOALS-S to distinguish among five different types of social goals is a considerable advantage when compared to other existing instruments that only measure one or two kinds of goals or those that confound different types of goals into an omnibus measure. It is also important to note that the social goals measured by GOALS-S refer to the social reasons for trying to achieve academically (Urdan & Maehr, 1995). This conceptualization is distinct from other social goal measures which try to assess how often students try to achieve specific social outcomes. For example, Wentzel s (1991, 1993, 1994, 1996) Social Goals Scale is designed to measure how often students want to be prosocial in school or how often they try to be responsible in school. GOALS-S, on the other hand, measures the social reasons behind studying and not social outcomes that students want to achieve per se. GOALS-S also measures different kinds of cognitive and metacognitive strategies. The cognitive component comprises three subscales: elaboration, organization, and rehearsal. The metacognitive strategies section focuses on students use of strategies to monitor and regulate their cognition, and it includes three subscales: planning, monitoring, and regulating. The inclusion of various types of cognitive and metacognitive strategies enables researchers using GOALS-S to have a more nuanced view of the different strategies that students use in school. Other widely used instruments mostly focus on larger units of analysis which fail to take into account finegrained distinctions in terms of the strategies that students adopt in the school setting (e.g., Biggs, 1987; Biggs, Kember, & Leung, 2001; Pintrich & DeGroot, 1990). For example, a popular instrument to measure strategies is the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & DeGroot, 1990). The MSLQ aggregates all metacognitive strategies into one metacognitive self-regulation subscale. For example, items such as When I study for this class I set goals for myself in order to direct my activities for each study period and I ask myself questions to make sure I understand the material I have been studying in this class are aggregated into one subscale. The first item refers to planning activities which involves goalsetting while the latter pertains more to monitoring which involves self-questioning and self-checking. In contrast, GOALS-S posits distinctions among different types of metacognitive strategies. Thus GOALS-S offers a more detailed view of the various kinds of strategies which can be beneficial for teachers and practitioners who may want to know more about the type of strategies students are adept at using and the kinds of strategies they need more help with. Within-network and Between-network Approaches to Construct Validation Thus far, GOALS-S has only been validated in the Australian setting (Dowson & McInerney, 2004)

3 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A. 581 and has only been used in studies with Australian students (e.g., Dowson, McInerney, & Nelson, 2006). Therefore, it is important that we investigate whether GOALS-S is also applicable to other non-western cultures such as the Philippines. The present study adopts a construct validation approach to the empirical assessment of GOALS-S (Marsh, 1997). Studies that adopt this approach can be classified as withinnetwork or between-network studies. Within-network construct validation, also called internal construct validation refers to the examination of the factor structure and factor correlation matrix. It typically involves statistical techniques such as confirmatory factor analysis (CFA) and reliability analysis. On the other hand, between-network or external construct validation approach entails examining patterns of relationships between the scales and other theoretically related constructs utilizing statistical techniques such as correlational analysis (Marsh, 1997). Few studies adopt this dual approach to validity thus providing relatively limited input into understanding the constructs being investigated. In contrast, the present study uses both approaches. First, we conduct a within-network study using confirmatory factor analysis to test the psychometric properties of GOALS-S. Consistent with the construct validation approach, it is not only important to address validity within an instrument (within-network validity) but it is also imperative to explore the possible differential relationships between the four sets of constructs in GOALS-S and a set of theoretically relevant measures (between-network validity). In our study, we looked at how the four sets of constructs in GOALS-S (academic goals, social goals, cognitive strategies, and metacognitive strategies) are related to behavioral and emotional engagement. Behavioral engagement has traditionally been defined as involvement in learning tasks which includes behaviors such as effort, persistence, concentration, asking questions, and contributing to class discussions (Birch & Ladd, 1997; Finn, Pannozzo, & Voelkl, 1995). Emotional engagement has been defined as students feelings about peers, teachers, schoolwork, or their affective reactions to the classroom (Fredericks et al., 2004). The four sets of constructs (academic goals, social goals, cognitive strategies, and metacognitive strategies) in GOALS-S are posited to have different kinds of relationships with behavioral and emotional engagement. For the academic goals scale, mastery 1 and performance goals 2 are assumed to have a positive relationship with both behavioral and emotional engagement. Work avoidance goals which refer to preference for easy work and the avoidance of effort exertion in school would be negatively related to both behavioral and emotional engagement (Elliot & Harackiewicz, 1996; Nicholls, Cobb, Wood, Yackel, & Patashnick, 1990). The relationship between the various kinds of social goals to both behavioral and emotional engagement is not yet that well-established, hence we did not advance any hypotheses a priori with regard to the relationship between these two sets of constructs. The various kinds of cognitive and metacognitive strategies are also assumed to be positively correlated with both behavioral and emotional engagement. Invariance testing Aside from testing for construct validity, it is also useful to look at whether different kinds of students respond to GOALS-S in a similar manner. It is common practice in educational research to pool together data from different kinds of students (e.g., different genders, year level, and types of school). However, combining datasets together would only be warranted if invariance in terms of factor structure can be shown. Previous studies have given inadequate attention to the investigation of the equivalence of educational constructs to students of different genders, year levels, and school types thus we wanted to investigate whether GOALS-S has invariant factor structure across different kinds of students. The Present Study The aim of this research was to examine crosscultural applicability of GOALS-S in the Philippine setting. This study adopted both within-network and between-network approaches to construct validation. In testing for within-network construct validity, we hypothesize that the model will reflect a sound firstorder factor structure in regard to the four sets of constructs measured in GOALS-S. For the academic goals scale, the first-order structure is composed of three factors: mastery goals, performance goals, and work avoidance goals. For the social goals scale, the first-order structure consists of five factors: social affiliation goals, social approval goals, social concern goals, social responsibility goals, and social status goals. For the cognitive strategies scale, there are three factors: elaboration, organization, and rehearsal. For the metacognitive strategies scale, there are also

4 582 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 three factors: monitoring, planning, and regulating. In addition to doing separate CFAs, we also tested for the invariance of the GOALS-S across students of different genders, year levels, and school types. For between-network construct validity, we tested for the relationship of the various scales of GOALS-S to behavioral and emotional engagement. It is hypothesized that academic goals such as mastery and performance goals will be positively related to these two between-network constructs while work avoidance goals will be negatively related to them. It is also hypothesized that all the cognitive and metacognitive strategies will be positively related to behavioral and emotional engagement. However, no specific hypotheses are advanced with regard to the relationship between social goals and these two between-network constructs, because the nomological network of social goals is still under-researched. Method Participants The participants of this study were 1,147 high school students from five schools (four private schools and one public school) in Metro Manila, Philippines. Students from the four private schools can be classified as middle class and their SES is comparatively higher than the SES of the students from the public school. The medium of instruction in all these schools was English. The students were distributed from first year to fourth year high school. There were 622 males (54.2%) and 524 (45.7%) females, with one student failing to identify the gender. Six hundred ninety-six students were drawn from the four private schools and 451 were from the one public school. At the time of testing, 603 students were in first or second year high school, while 544 students were in third or fourth year high school. The mean age of the students was years old (SD = 1.39). All these five schools cater to secondary school students from first year to fourth year high school. Administration The first author administered the GOALS-S to participants in class groups with the assistance of teachers at each school. The teachers were instructed not to interpret any of the GOALS-S items for students but to tell students to interpret the items as best as they can or leave it blank if they do not understand it. An earlier pilot study conducted on a smaller scale has indicated that the level of difficulty was appropriate for Filipino high school students thus we decided to administer the English version of the questionnaire. Measures The main instrument was the 84-item GOALS-S which was designed to measure four sets of constructs: (1) three academic goals, (2) five social goals, (3) three cognitive strategies, and (4) three meta-cognitive strategies. Each of these factors was measured through six items. A six-point Likert-type scale was used for each item ranging from 1 (lowest) to 6 (highest) with 1 reflecting lowest level of endorsement and 6 reflecting the highest level of endorsement for a particular item. 3 Academic goals Factor 1. Mastery goals. This measures the degree to which students want to achieve in school to develop their competence relative to self-set standards (e.g., I work hard at school because I am interested in what I am learning. ) Factor 2. Performance goals. This measures the degree to which students want to achieve in school to outperform other students (e.g., I want to learn things so that I can come near the top of the class. ) Factor 3. Work avoidance goals. This measures the degree to which students want to achieve with as little effort as possible (e.g., I choose easy options in school so that I don t have to work too hard. ) Social goals Factor 4. Social affiliation goals. This measures the degree to which students want to achieve in school to enhance a sense of belonging to the group (e.g., I want to do well at school so that I can feel close to my group of friends. ) Factor 5. Social approval goals. This measures the degree to which students want to achieve to gain the approval of parents and teachers (e.g., I want to do well at school so that I can get praise from my teachers. ) Factor 6. Social concern goals. This measures the degree to which students want to achieve to be able to assist others in their academic work when they need it. (e.g., I try to do well at school so that I can I help my friends with their schoolwork. ) Factor 7. Social responsibility goals. This measures the degree to which students want to achieve

5 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A. 583 to meet social role obligations or meet interpersonal commitments (e.g., I do good schoolwork so that I don t have any trouble with my parents or teachers. ) Factor 8. Social status goals. This measures the degree to which students want to attain wealth or position in school and later life (e.g., I do good schoolwork so that I have a good future. ) Cognitive strategies Factor 9. Elaboration. This strategy refers to making connections between present and previously learned information which my involve strategies like paraphrasing, generating analogies, etc. (e.g., When learning things for school, I try to see how they fit together with other things. ) Factor 10. Organization. This strategy refers to selecting, sequencing, outlining or summarizing important information in class (e.g., I try to organize my school notes when I want to learn things for school. ) Factor 11. Rehearsal. This strategy refers to listing, memorizing, and reciting facts to be learned (e.g., I try to memorize things I want to learn for school ) Meta-cognitive strategies Factor 12. Monitoring. This strategy involves selfchecking for understanding and involves a systematic attempt to evaluate the assimilation and organization of material that has been learned in class (e.g., I often ask myself questions to see if I understand what I am learning. ) Factor 13. Planning. This strategy involves prioritizing, managing time, and setting realistic goals in order to complete the work needed (e.g., When I want to learn things for school I pick out the most important parts first. ) Factor 14. Regulating. This strategy involves rectifying deficits in learning that were identified by the student and includes attempts to seek explanations from teachers and identify mistakes in reasoning (e.g., prioritizing, managing time, and setting realistic goals in order to complete the work needed (e.g., If I am having trouble learning something at school, I ask for help. ) To assess behavioral engagement, we used the ongoing engagement subscale (e.g., I work very hard on my schoolwork. ) of the Student Engagement and Disaffection in School Student Report from the Rochester Assessment Package (RAPS) for Schools developed by Wellborn and Connell (1987). This subscale had four items and exhibited good psychometric properties in the current study (M = 4.41, SD =.91, Cronbach s alpha =.70). To measure emotional engagement, we used the affect to school subscale (e.g., I like working at school. ) of the Facilitating Conditions Questionnaire (FCQ; McInerney, Dowson, & Yeung, 2005), which also consists of four items (M = 4.27, SD =.86, Cronbach s alpha =.58). Both the RAPS and FCQ from which the behavioral and emotional engagement indices were drawn have been shown to have good psychometric properties in a lot of previous studies conducted in various cultural contexts including the Philippines (e.g., Ganotice, 2010; Gonida, Voulala, & Kiosseoglou, 2009; McInerney, 2008; McInerney et al., 2005; Skinner & Belmont, 1993; Wellborn & Connell, 1987). Statistical Analysis To examine the within-network validity of GOALS-S among Filipino high school students, we first computed for the descriptive statistics and reliability alpha coefficients. Four separate confirmatory factor analysis were then conducted to test the factor structure of the four sets of constructs in GOALS-S. As reported earlier, the GOALS-S has 84 items. Bagozzi and Heatherton (1994) argued that when there are more than four or five indicators per factor in a large sample, it is quite likely to lead to an unsatisfactory fit in the measurement model. To address this, we aggregated the items to form parcels as indicators in the CFA (see Little, Cunningham, Shara, & Widaman, 2002 for a review of issues surrounding parcelling). The parcels were formed by randomly combining two items in each factor. Since each factor was originally measured with six items, this resulted in each factor in GOALS-S being measured by three parcels. All the parcels were specified as indicators of only one factor, and the uniqueness of each parcel was constrained to be independent. For example, mastery goals were measured through three parcels and this was the case for all other latent factors. This resulted in a total of 42 parcels (14 latent factors each measured by three parcels) We conducted four separate CFAs on conceptually distinct sets of scales relating to students (1) academic goals, (2) social goals, (3) cognitive strategies, and (4) metacognitive strategies. We allowed the factor correlations in the four separate CFAs to be freely estimated. All analyses were conducted using Amos 12 (Arbunckle, 2007),

6 584 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 and all parameters were estimated using maximum likelihood procedure. To test for between-network construct validity, we computed for the correlations of the four sets of constructs in GOALS-S with measures of behavioral and emotional engagement. We also conducted multigroup confirmatory factor analysis to assess the factorial invariance of GOALS-S. Invariance analysis is done to provide information about the equivalence of the data across multiple groups (Marsh, 1993, 1994). In the present study, we did three invariance tests related to testing the equivalence of GOALS-S (1) across genders (male and female), (2) across year levels (junior and senior high school) and (3) across academic institutions (private and public schools). In testing for invariance across year levels, students who were in their first and second year were put into the junior high school group, while participants on their third and fourth year were put into the senior high school group. We then conducted invariance analysis on the junior and senior high school groups to check whether the students in their early years (1 st and 2 nd ) and students who were more advanced (3 rd and 4 th year) answered the questionnaire in the same way. In testing for invariance across the private and public schools, students from the four private schools were put into the private school category, while students from the public school were put into the public school category. We then conducted an invariance analysis for students from these two groups. Results Preliminary Analyses The means, standard deviations, and reliability alpha coefficients of GOALS-S are shown in Table 1. Table 1 Descriptive Statistics, Cronbach s Alphas, and CFA Loadings for the Different Subscales of GOALS-S Academic goals Subscales Mean SD Cronbach s alpha in the current study Cronbach s alpha in Dowson and McInerney s (2004) study CFA range for the different parcels CFA mean Mastery goals to Performance goals to Work avoidance goals to Social goals Social affiliation goals to Social approval goals to Social concern goals to Social responsibility goals to Social status goals to Cognitive strategies Elaboration to Organization to Rehearsal to Metacognitive strategies Planning to Monitoring to Regulating to

7 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A. 585 Table 2 Bivariate Correlations Among the Academic and Social Goals in GOALS-S Mastery.429 *** ***.254 ***.330 ***.479 ***.550 ***.414 *** 2. Performance ***.620 ***.344 ***.542 ***.326 *** 3. Work avoidance *** *** *** *** 4. Affiliation ***.392 ***.423 ***.114 *** 5. Approval ***.528 ***.306 *** 6. Concern ***.280 *** 7. Responsibility *** 8. Status --- Note: * p <.05, ** p <.01, *** p <.001. Table 3 Bivariate Correlations Among the Cognitive and Metacognitive Strategies in GOALS-S Elaboration.617 ***.573 ***.631 ***.809 ***.549 *** 2. Organization ***.816 ***.638 ***.593 *** 3. Rehearsal ***.616 ***.705 *** 4. Planning ***.593 *** 5. Monitoring *** 6. Regulating --- Note: * p <.05, ** p <.01, *** p <.001 Internal consistency of the GOALS-S was satisfactory with reliability alpha coefficients for each subscale ranging from.67 to.92. These Cronbach s alphas are comparable to results shown in the original validation study conducted by Dowson and McInerney (2004). While there is no statistical test to establish whether the means for different variables are significantly different from each other, it appears that, in terms of academic goals, mastery goals (M = 4.76) were the most highly endorsed, while work avoidance goals were the least endorsed (M = 3.00). With regard to social goals, social status (M = 5.21) and social responsibility goals (M = 4.59) were the two most highly endorsed. Social affiliation goals were the least endorsed (M = 3.74). The level of endorsement for the different strategies were not that different from each other ranging from M = 4.47 for Regulating and M = 4.60 for Monitoring. The correlations among the different goal orientations and among the different strategies are shown in Tables 2 and 3. In terms of academic goals, the correlations indicated that both mastery and performance goals are positively correlated with each other (r =.429, p <.001) while work avoidance goals were negatively related to mastery goals and not significantly related to performance goals. In terms of social goals, social affiliation, social approval, social concern, social responsibility, and social status goals were all positively correlated with each other. The different kinds of cognitive and metacognitive strategies were also all positively correlated with each other (See Table 3). Separate Confirmatory Factor Analyses The four separate CFAs for the academic goals section, social goals section, cognitive strategies section, and metacognitive strategies section all indicated a good fit to the data as evidenced by the values of greater than.90 for GFI, IFI, TLI, and CFI and values of less than.08 for the RMSEA. Only the chi square statistics was not satisfactory in all the four CFAs. A statistically non-significant chi square value indicates that the model is a reasonably satisfactory representation of the data. However, as noted by Anderson and Gerbing (1988), the value of the chisquare statistic is dependent on sample size. As such, data that involve a large sample size will likely have a chi-square statistic that is significant although there

8 586 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 Table 4 Goodness-of-fit Indices for the Different Sections of GOALS-S Model χ 2 df χ 2 /df P value RMSEA GFI IFI TLI CFI Academic goals p < Social goals p < Cognitive strategies p < Metacognitive strategies p < Note: df = degrees of freedom; RMSEA = root mean square error of approximation; GFI = goodness of fit index; IFI = incremental fit index, TLI = Tucker-Lewis index; CFI = comparative fit index. Table 5 Invariance Tests of GOALS-S Across Males and Females Model χ 2 df χ 2 /df p RMSEA IFI TLI CFI (no invariance imposed) Academic goals Change in CFI p < Invariant factor p < Invariant factor variances and covariances (no invariance imposed) p < Social goals p < Invariant factor p < Invariant factor variances and covariances (no invariance imposed) p < Cognitive strategies p < Invariant factor p < Invariant factor variances and covariances (no invariance imposed) p < Metacognitive strategies p < Invariant factor p < Invariant factor variances and covariances p < are only minor discrepancies between the model and the data. Because of this, we decided to focus on the other goodness of fit indices which all indicate a good fit. Given that these four separate CFAs yielded a good fit to the data, we then proceeded to do a series of invariance tests on these four sections of GOALS-S. Multigroup Tests of Invariance To test for the measurement invariance of the GOALS-S, we followed a stepwise procedure. For the first model, we tested for configural invariance which indicates whether the number of factors and pattern of indicator-factor are identical. The second model holds the factor invariant, the

9 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A. 587 Table 6 Invariance Tests of the GOALS-S Across Junior and Senior High School Students Model χ 2 df χ 2 /df p RMSEA IFI TLI CFI (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances Academic goals Change in CFI p < p < p < Social goals p < p < p < Cognitive strategies p < p < p < Metacognitive strategies p < p < p < third model holds both factor and factor variances and covariances invariant. We followed this stepwise procedure in testing for invariance across genders, year levels, and school types. We did separate multigroup tests of invariance for the four distinct sets of constructs in GOALS-S: (1) academic goals, (2) social goals, (3) cognitive strategies, and (4) metacognitive strategies. Cheung and Rensvold (2002) indicated that a decrease of 0.01 in the CFI is evidence for lack of invariance. Following this suggestion, the results show that there is relative invariance across all models (See Tables 5, 6, and 7). This shows that the factor structure, factor, and factor variances and covariances are parallel for males and females, for junior and senior high school students, and for students from private and public schools.

10 588 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 Table 7 Invariance Tests of the GOALS-S Across Private and Public School Students (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances (no invariance imposed) 2. Invariant factor 3. Invariant factor variances and covariances Model χ 2 df χ 2 /df p RMSEA IFI TLI CFI Academic goals p < p < p < Social goals p < p < p < Cognitive strategies p < p < p < Metacognitive strategies p < p < p < Change in CFI Correlations with Between-network Educational Constructs To test for between-network construct validity, we also explored the nature of relationships among the 14 factors in GOALS-S and a set of key correlates. To this end, we administered items that tapped behavioral and emotional engagement. Studies have shown that students who use adaptive cognitive and metacognitive strategies are also those who are more behaviorally and emotionally engaged in the school experience (Fredericks et al., 2004). Between-network construct validity will be supported if the behavioral and emotional engagement are positively correlated with both mastery and performance goals. Work avoidance goals should be negatively correlated with behavioral and emotional engagement. The zero-order correlations generally supported this assumption. Mastery and performance goals were both positively

11 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A. 589 Table 7 Correlations with Between-network Educational Constructs Academic goals Subscales Behavioral engagement Emotional engagement Mastery goals.460 ***.482 *** Performance goals.208 ***.317 *** Work avoidance goals *** *** Social goals Social affiliation goals ** Social approval goals.165 ***.228 *** Social concern goals.291 ***.374 *** Social responsibility goals.346 ***.386 *** Social status goals.311 ***.294 *** Cognitive strategies Elaboration.350 **.465 *** Organization.443 ***.473 *** Rehearsal.392 ***.468 *** Metacognitive strategies Planning.403 ***.471 *** Monitoring.391 ***.445 *** Regulating.319 ***.417 *** Note: * p <.05, ** p <.01, *** p <.001. correlated with behavioral (r =.208 to r =.460, p <.001) and emotional engagement (r =.317 to r =.482, p <.001). On the other hand, work avoidance goals were negatively correlated with both behavioral (r = -.466, p <.001) and emotional engagement (r = -.364, p <.001). Social goals were also all positively correlated with emotional engagement (r =.161 to r =.386, p <.001) and behavioral engagement (r =.165 to r =.346, p <.001 ). The exception was social affiliation goals which were not significantly correlated with behavioral engagement. With regard to the cognitive and meta-cognitive strategies, they were all positively correlated with behavioural (r =.319 to r =.443, p <.001) and emotional engagement (r =.445 to r =.473, p <.001. These results provide additional evidence for the between-network construct validity of the GOALS-S. Discussion The present study adopted a construct validation approach to examining GOALS-S through an assessment of within- and between-network validity. In addition, we also wanted to test for the factorial invariance of the instrument across students of different genders, different year levels, and different schools. In terms of within-network validity, the applicability of GOALS-S in the Philippine setting was supported. The internal consistency reliability of the instrument was good as shown through the Cronbach s alpha coefficients for the various subscales of the instrument. The Cronbach s alphas range from.67 to.92 for the various factors in GOALS-S. These were comparable to those originally found by Dowson and McInerney (2004) in their study where they looked at the psychometric properties of GOALS-S among Australian students. The four separate confirmatory factor analyses on the four sets of constructs (academic goals, social goals, cognitive strategies, and metacognitive strategies) in GOALS-S all showed good fit indices providing further support for the applicability of GOALS-S. All the CFA factor for the parcels were highly significant. The results provided psychometric evidence that different types of social goals are distinct from each other, thus an omnibus measure of social goals does not seem to be appropriate. Previous research has sometimes aggregated different types of social

12 590 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 goals into one unitary measure (e.g., Chang & Wong, 2008; Cheng, 2005; Yang & Yu, 1988) or sometimes included social goal items as a part of scales measuring performance goals (e.g., McInerney & Ali, 2006; Nicholls, Patashnick, & Nolen, 1985). The current study suggests that such methods might not be appropriate and may lead to potential confounding given that different kinds of social goals are both theoretically and statistically distinct from each other. This study also supports conceptual and statistical distinction between the different types of cognitive and metacognitive strategies. Studies that aggregated different types of strategies into an omnibus adaptive or maladaptive approach (e.g., Biggs, 1987; Biggs, Kember, & Leung, 2001) might not have enough sensitivity to detect differences in how students utilize different strategies in their learning. GOALS-S is a more sensitive measure that can take into account the variety of strategies that students use in their academic work. The multigroup tests of invariance indicated that males and females, junior and senior high school students, and private and public school students responded to the questionnaire in a similar manner. In other words, in terms of underlying constructs and the composition of and relationships among these constructs, students of different genders, year levels, and school types are not substantially different. This has implications for educational research. Most of the research in educational psychology routinely aggregates males and females and also aggregates students from different year levels and from different schools. An important aspect to take into account in determining whether students should be pooled or treated separately is the extent to which the factor structure underpinning the dataset is invariant across genders, year levels, and types of schools. The present study showed that the factor structure and other components of the test are in fact invariant, thus providing justification for the common practice of pooling the data from different kinds of students together. In terms of between-network validity, the correlations of the subscales of GOALS-S with behavioral and emotional engagement confirm our initial hypotheses providing further evidence of the applicability of GOALS-S in the Philippine setting. In regard to academic goals, mastery goals and performance goals were both positively correlated with behavioural and emotional engagement. Note that the mastery and performance goals we measured were both positively valenced because GOALS-S does not take into account the avoidance dimension of mastery and performance goals. Results might have been different if we looked at how performance avoidance and mastery avoidance goals relate to behavioral and emotional engagement (see Elliot, 2005 for a review). In contrast, work avoidance goals were negatively correlated with both behavioural and emotional engagement. This was also in line with our hypothesis. Students who want to take the easy way out in school (i.e., pursue work avoidance goals) are also less likely to be behaviorally or emotionally engaged in the school setting. The relationship between social goals and behavioral and emotional engagement was interesting because previous studies have failed to adequately explore this area. In general, the different kinds of social goals such as social approval, social concern, social responsibility, and social status goals were moderately and positively correlated with both behavioral and emotional engagement. This supports the argument of Urdan and Maehr (1995) that social goals are also important components of student motivation despite the continued neglect of social goal research by goal theorists. Only social affiliation goals did not correlate significantly with behavioral engagement. Clearly, more research is needed to investigate the effects of various kinds of social goals on educational outcomes. Both behavioural and emotional engagement were also positively correlated with the use of various cognitive (elaboration, organization, and rehearsal) and metacognitive strategies (planning, monitoring, and regulating). This suggests that students who are more likely to use these cognitive and metacognitive strategies are also more behaviorally and emotionally engaged in the classroom experience. This corroborates the findings of previous studies showing that students who use adaptive cognitive and metacognitive strategies are more likely to persist and work hard in school, i.e., behavioral engagement, and they are also more likely to like school and take a positive attitude towards it, i.e., emotional engagement (Fredericks et al., 2004 for a review). Research in the Asian setting has indicated that the academic motivation of Asians has a strong social orientation (e.g., Ho, 1986, Salili, 1996, Yang & Yu, 1988; Yu, 1996), which studies in the Philippines have also confirmed (e.g., Bernardo,

13 The Reliability and Validity of the Goal Orientation king, r.b., & WATKINS, D.A , 2008). Kumar and Maehr (2007) asserted that among Asians, the motivational goals promoting achievement behavior are extrinsic, utilitarian, and social. They are directed toward achieving upward mobility, recognizing parental sacrifices, fulfilling familial obligations of achieving success, and making the family proud particularly in social comparisons with other families. Achievement and the motives to achieve are, therefore, more social and less individual in collectivistic societies that endorse interdependent selves (p. 54). This makes it imperative that researchers in collectivist settings incorporate the measurement of social goals when they conduct their studies. The literature relating to students goals in school has consistently made the distinction between students academic and social goals (Blumenfeld, 1992; Chang & Wong, 2008; Dowson & McInerney, 2001, 2003; Urdan & Maehr, 1995). In addition, a conceptual distinction between cognitive and metacognitive strategies has also been made in the literature (Bergin, 1998; Biggs, 1987). The GOALS-S offers a viable instrument for measuring these different sets of constructs. Positing more fine-grained distinctions between academic and social goals and between cognitive and metacognitive strategies can help advance conceptual clarity and measurement specificity. Limitations and Directions for Future Research A limitation of this study is that only students in Metro Manila area were sampled. The extent to which this sample differs from the general Filipino high school population limits the generalizability of the results. Future research needs to explore the reliability and validity of GOALS-S in a more heterogenous group of Filipino students. In addition, it might be useful to translate the GOALS-S into the Filipino language especially when it will be given to students from more remote areas. Although most of the researchers who administered educational psychological tests to Filipino students retained the English version (e.g., Bernardo, 2008; Dela Rosa, 2010; Ganotice, 2010; Watkins, McInerney, & Boholst, 2003), this may not be appropriate for students in faraway provinces where the level of English fluency is not that high. Thus far, the GOALS-S has been found to be valid in both Australia and the Philippines. Future research could investigate whether the GOALS-S is also suitable for use in other cultural contexts. Conclusion Student motivation is a primary issue in education. Both educational researchers and practitioners are interested in assessing and improving the motivational quality of students in school. As such, the accurate measurement of different facets of students motivation like their goals for schooling and the strategies they employ for learning is imperative. However, most of the instruments that are widely circulated in the literature are based on Western research. Their psychometric properties have usually not been tested in non-western cultures such as those found in Asia. The current study provides initial evidence of the applicability of a Western-developed instrument in the Filipino setting. The results of the current study indicate that GOALS-S has good psychometric properties. Different invariance tests have also shown that responses of the students to the instrument were equivalent across genders, year levels, and school types. This shows that GOALS-S is a suitable instrument that can be used in the Filipino cultural context. notes 1,2 A limitation of GOALS-S is that it only takes into account the approach dimensions of both mastery and performance goals. Recent theorizing in achievement goal theory has bifurcated mastery and performance goals into approach and avoidance dimensions. The most recent modification of achievement goal theory has resulted in four types of goals: (1) mastery-approach goal, which refers to wanting to achieve to gain new knowledge and improve one s competence; (2) performance-approach goal, which refers to wanting to achieve to outperform other students and demonstrate competence before others; (3) mastery-avoidance goal which refers to wanting to avoid misunderstanding and the loss of one s skills; and (4) performance avoidance goals which refers to wanting to avoid showing incompetence relative to others. The mastery goal subscale and performance goal subscale of GOALS-S only measures positively valenced goals. Mastery avoidance and performance avoidance goals are not measured in GOALS-S. 3 A 6-point Likert scale was used in the current study instead of the original five-point Likert scale in Dowson and McInerney s (2004) initial study. This was done to eliminate the midpoint-bias if a five-point Likert scale was used (Garland, 1991). Research has shown that the validity of an instrument is not significantly influenced by the number of points (e.g., five point or six point) in the Likert scale (e.g., Chang, 1994; Chomeya, 2010). As noted by Matell and Jacoby (1971) validity was found to be independent of the number of scale points contained in the rating scale (p. 670).

14 592 THE ASIA-PACIFIC EDUCATION RESEARCHER VOL. 20 NO. 3 References Anderson, J.C., & Gerbing, D.W. (1988). Structural equation modelling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, Archer, J. (1994). Achievement goals as a measure of motivation in university students. Contemporary Educational Psychology, 19, Arbuckle, J.L. (2007). Amos 16 user s guide. Chicago: SPSS. Bagozzi, R. P., & Heatherton, T. F.(1994). A general approach to representing multifaceted personality constructs: Application to state self-esteem. Structural Equation Modelling, 1, Bergin, D. A. (1998, April). Patterns of motivation orientation, learning strategies, and achievement of high school students of colour. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Bernardo, A. B. I. (2005, April). Direct and mediated effects of parenting characteristics on students achievement goal orientations: The role of social and individual goal orientations. Paper presented at the Biennial Conference of the Asian Association of Social Psychology, Victoria University of Wellington, New Zealand. Bernardo, A. B. I. (2008). Individual and social dimensions of Filipino students achievement goals. International Journal of Psychology, 43, Biggs, J.B. (1987). The Study Process Questionnaire (SPQ) Manual. Melbourne, Australia: Australian Council for Educational Research. Biggs, J.B., Kember, D., & Leung, D.Y. (2001). The revised two-factor Study Process Questionnaire: R-SPQ-2F. British Journal of Educational Psychology, 71, Birch,S., & Ladd, G. (1997). The teacher-child relationship and children s early school adjustment. Journal of School Psychology, 35, Blumenfeld, P. (1992). Classroom learning and motivation: Clarifying and expanding goal theory. Journal of Educational Psychology, 84, Chang, L. (1994). A psychometric evaluation of 4-point and 6-point Likert scale types in relation to reliability and validity. Applied Psychological Measurement, 18, Chang, W.C., & Wong, K. (2008). Socially oriented achievement goals of Chinese university students in Singapore: Structure and relationships with achievement motives, goals, and affective outcomes. International Journal of Psychology, 43, Cheng, R. (2005). Effects of social goals on student achievement motivation: The role of self-construal (Unpublished doctoral dissertation). The University of Hong Kong, Hong Kong. Cheung, G.W., & Rensvold, R.B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, Chomeya, R. (2010). Quality of psychology test between Likert scale 5 and 6 points. Journal of Social Sciences, 6, Covington, M.V. (2000). Goal theory, motivation, and school achievement: An integrative review. Annual Review of Psychology, 51, Dela Rosa, E. (2010). The 2 x 2 achievement goal framework and intrinsic motivation among Filipino students: A validation study. The Educational Measurement and Evaluation Review, 1, Dowson, M., & McInerney, D.M. (2001). Psychological parameters of students social and work avoidance goals: A qualitative investigation. Journal of Educational Psychology, 93, Dowson, M., & McInerney, D.M. (2003). What do students say about their motivational goals? Towards a more complex and dynamic perspective on student motivation. Contemporary Educational Psychology, 28, Dowson, M., & McInerney, D.M. (2004). The development and validation of the Goal Orientation and Learning Strategies Survey (GOALS-S). Educational and Psychological Measurement, 64, Dowson, M., McInerney, D.M., & Nelson, G. (2006). An investigation of the effects of school context and sex differences on students motivational goals. Educational Psychology, 26, Elliot, A.J. (2005). A conceptual history of the achievement goal construct. In A.J. Elliot & C. Dweck (Eds.), Handbook of competence and motivation (pp ). New York, NY: Guilford Press. Elliot, A., & Harackiewicz, J. (1996). Approach and avoidance achievement goals and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 70, Elliot, A., & McGregor, H. (2001). A 2 x 2 achievement goal framework. Journal of Personality and Social Psychology, 72, Elliot, A. J., & Murayama, K. (2008). On the measurement of achievement goals: Critique, illustration, and application. Journal of Educational Psychology, 100, Finn, J. D., Pannozzo, G. M., & Voelkl, K. E. (1995). Disruptive and inattentive-withdrawn behavior and achievement among fourth graders. Elementary School Journal, 95, Ford, M.E. (1996). Motivational opportunities and obstacles associated with social responsibility and caring behavior in school contexts. In J. Juvonen & K.R. Wentzel (Eds.), Social motivation: Understanding children s school adjustment (pp ). New York: Cambridge University Press. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of