Analyzing Teacher Professional Standards as Latent Factors of Assessment Data: The Case of Teacher Test-English in Saudi Arabia

Transcription

1 Analyzing Teacher Professional Standards as Latent Factors of Assessment Data: The Case of Teacher Test-English in Saudi Arabia 1

2 Introduction The Teacher Test-English (TT-E) is administered by the NCA as an assessment tool for teacher certification in KSA. The 69 operational test items used in this study are grouped into five content domains: Domain 1: Language Pedagogy (29 items), Domain 2: Curriculum Design (15 items), Domain 3: Theoretical Knowledge (13 items), Domain 4. Theoretical Application (8 items), and Domain 5. Language Proficiency (4 items). These domains are further operationalized into 16 professional standards (PSs) Domain 1: Language Pedagogy (29 items), Domain 2: Curriculum Design (15 items), Domain 3: Theoretical Knowledge (13 items), Domain 4. Theoretical Application (8 2

3 3

4 4

5 Purpose of the study The purpose is to examine the performance of English language teacher candidates on 16 professional standards (PSs) targeted with the TT-E and to test for gender differences on the entire TT-E construct and separately by PSs. To perform analyses at true score (error-free) level, the TT-E construct and individual PSs are treated as latent factors in the framework of confirmatory factor analysis (CFA). Testing for preliminary assumptions required under this approach is also targeted with this study (e.g., testing for differential item functioning of TT-E items for gender). 5

6 Research Questions 6

7 Method Data: The binary scores (1 = correct,0 = incorrect) of 19,167 English language teacher candidates on 69 items of the TT-E; (59.8% females and 40.2% males). Statistical Analysis: Latent variable modeling (LVM) in CFA framework using the computer program Mplus (Muthén & Muthén, 2010). The testing for differential item functioning (DIF) was performed via IRT analysis using the computer program Xcalibre 4.2 (Guyer & Thompson, 2012). 7

8 Results The results related to RQ1 and RQ2, obtained within CFA, show that the TT-E data are essentially unidimensional; that is, there is one dominant ability measured by TT-E. Also, the one-factor CFA model fit the data separately for males and females; i.e., there is configural invariance of the TT-E structure across gender. The score reliability, estimated via a latent variable modeling approach, was found to be for the entire sample, for the sample of males, and for the sample of females. 8

9 Testing for Data Fit of the One-Factor CFA Model With TT-E Data: Total Sample and by Gender 9

10 10

11 RQ4 results: Correlations Among Professional Standards 11

12 12

13 Results on RQ5 (b): CFA-based MIMIC model for gender differences on 16 professional standards (PS1,, PS16) 13

14 Females outperform males on all PSs, except in the case of no gender difference on one professional standard, PS11: Teachers have a general knowledge of the language as a system. 14

15 Gender differences on domains of professional standards 15

16 16

17 17

18 Conclusion 1. The TT-E data are essentially unidimensional; i.e., there is one dominant ability that underlies TT-E data, referred to as General Teacher Ability on Professional Standards (GTAPS). 2. The unidimensional factor structure of the TT-E holds also for males and females separately. 3. There is no differential item functioning (DIF) of TT-E items against males or females; i.e., examinees with the same ability (same GTAPS) have equal chances to answer any item correctly regardless of their gender. 18

19 4. Females outperform males on the general TT-E ability (GTAPS scores), with a difference of medium effect size. 5. There is no gender difference on PS11: Teachers have a general knowledge of the language as a system but females do better than males on all other 15 PSs, with a difference of small effect size on three PSs (PS3, PS6, and PS13) and medium effect size on 12 PSs (PS1, PS2, PS4, PS5, PS7, PS8, PS9, PS10, PS12, PS14, PS15, and PS16). 6. The pattern of differences among PSs (individually or grouped into domains) is similar for males and females, but the PS performance of females is higher compared to males. 19

20 Thank You! SHUKRAN 20