Gender Differential Item Functioning (GDIF) in an Online Intelligence Test

Size: px

Start display at page:

Download "Gender Differential Item Functioning (GDIF) in an Online Intelligence Test"

Norma Greer
6 years ago
Views:

1 Gender Differential Item Functioning (GDIF) in an Online Intelligence Test ARIFFIN SITI RAHAYAH, SYAKIMA ILYANA IBRAHIM, NURUL HUDA MOHD ABD MALEK, SHARIDA HANIM SARIF, SITI FATIMAH MOHD YASSIN Faculty of Education Universiti Kebangsaan Malaysia Bangi, Selangor MALAYSIA Abstract: - This study explores the differential performances between genders among Malaysian public university students. The instrument used was Malaysian intelligence test known as UKM2. The study aims at measuring gender bias in the UKM2 on-line test by investigating differential item functioning (DIF) across genders. There were 124 male and 373 female students who were randomly selected from one of the public universities in Malaysia as samples in the study. The Malaysian intelligence test consists of four indices (a) Verbal Comprehension Index (VCI), (b) Perceptual Reasoning Index (PRI), (c) Working Memory Index (WMI) and (d) Processing Speed Index (PSI). The overall 370 items were made up of dichotomous and partial credits from 13 constructs. Data gathered were analyzed using Winsteps version , a Rasch-based item analysis program. Study shows that gender-related DIF (GDIF) exists in the test administered. GDIF contrast which favored the males and females are displayed in 77 items and 76 items respectively. However, only 29 items are exhibiting significant GDIF, based on the size ranging from 0.61 to 5.61 (logits). Males excelled in two indices namely VCI and WMI, while females in PRI and PSI indices. The authors discuss possible explanation for differential clustering tendencies of items displaying GDIF and some practical and theoretical implications of GDIF in the development and interpretation of the online test. Key-Words: - Malaysian intelligence test, Verbal Comprehension Index (VCI), Perceptual Reasoning Index (PRI), Working Memory Index (WMI), Processing Speed Index (PSI), Gender Differential Item Functioning (GDIF), Online test. 1 Introduction The utilization and manipulation of computer network and information technologies, is becoming popular with such connections in all related field, especially in the field of education. The online test has newly become an ideal medium in assessing the students. This development provides a new paradigm, especially in terms of evaluation. The development of evaluation aspect using computer literacy can be a source of performance evaluation based on criteria of each student compared to the previous method that is based on norms. The online psychosocial test is able to evaluate each student comprehensively [4]. Online test allows for administration of assessment with lesser cost and time and has greater flexibility for maintenance. Virtual test can be administered using computer system and information technologies effectively, however the usability of the test should be considered. A test web page must not only have an easy to use interface, but it should also serve the purpose [3]The online Malaysian intelligence test known as UKM2 is developed as in effort to identify the gifted and talented students from all over Malaysia. Therefore, this instrument should be of high validity for its effectiveness in meeting the purpose or function of its construction. Fairness in the item is an important factor that is closely linked with the legitimacy to determine the effectiveness of a test. Every test that is set to measure students achievement especially those from multiple background and characteristics must be fair and equal. Issues related to fairness and equality in the construction of test items are often emphasized in assessing the effectiveness of psychosocial testing instruments [2],[6][13]. Measurement of the level of intelligence of Malaysian students using online instruments should be fair to feature the diversity of students in Malaysia. The variation of answers given for every item can lead to the different range of individual s score of the test taken. The analysis of differential item functioning (DIF) is used to detect the biased item, as an effort to ensure that the instruments are built equally [13] ISSN: ISBN:

2 2 Problem Formulation Assessment of students' achievement by gender is often studied in the field of educational research. Study by [16]suggested that male and female students possess the same attitude towards digital related activities but female students show higher performance in using e- learning materials. Many previous studies showed the differences in achievement between male and female students [5],[6]. In general, studies have shown that males perform better in mathematics particularly in numbers and geometry [1]. It was found that males have more ability in answering questions pertaining perception and visual spatial, compared to females who were more able in areas relating to verbal and social relations [15]. Studies conducted in the Malaysian context showed that females have the advantage in solving questions related to memorizing facts and order problems, whereas the males found it easier to answer open ended questions which were more practical and realistic compared to female students [18]. Achievement of students in public examinations and among university graduates in Malaysia shows a trend dominated by the females. Study by [6] had proposed for the development of a test which emphasizes on genderrelated item functioning (GDIF) as to ensure the fairness in Malaysian public test. Thus, this study is aimed at evaluating potential measurement of biasness across genders in the Malaysian online intelligence test and investigating the potential biased items by conducting DIF analysis. DIF may be investigated whenever the conditional probabilities of correct response differ between the two groups of the same ability. Analysis of DIF will result in a pattern of failed invariance clearly, however the causes and potential remedies of DIF item might not be obvious. By searching out possible sources or patterns to the occurrence of DIF over items within groups of gender, hypotheses as to the sources of DIF can be detected confidently because of the presence of more items for analysis. Information of DIF in students achievement responding to the online intelligence test will show their cognitive tendency by gender. This study focuses on the GDIF in the online Malaysian intelligence test through four clusters of indexes namely Verbal Comprehension Index (VCI), Perceptual Reasoning Index (PRI), Working Memory Index (WMI) and Processing Speed Index (PSI). The WMI and VCI indices and the constructs are highly correlated in academic area (word reading, comprehension, math and written expression), thus this test can predict academic achievement scores [9]. The PRI on the other hand is closely associated with spatial abilities. However, previous studies showed that PRI is more relevant with fluid intelligence than spatial abilities [5]. As for PSI, there is only symbol search index associated with reasoning by the conservation and working memory space for higher order fluid tasks [7]. A study by [5]revealed that for Germany WISC-IV, the males achieved higher than females in the VCI and PRI. Working memory index (WMI) contains three subtests which are much influenced by mathematical skills. Males have higher potential in the related fields of mathematics and look more at the geometric clues [1]. Study by [5] had discovered that there is no difference in scores between male and female students in the WMI index whereas used to measure working memory and short-term memory. There is a need to study on the cognitive achievement of students in Malaysia by gender in order to aid the development of test item. The test measures the achievement of students cognitive intelligence provides very useful information in perceiving the actual cognitive performance by gender. Through the information of students clustering tendencies in answering this test, the construction of items in online tests can be improved to give fairness to students of different genders. 2.1 Research Objectives i. To identify the existence and intensity of GDIF in online Malaysian intelligence test. ii. To analyse any patterns or trends detected in the GDIF items 2.2 Methodology This study is conducted by using quantitative approach in one of public university in Malaysia. The students were selected randomly from four faculties namely Faculty of Medicine, Faculty of Dentistry, Faculty of Allied Health Sciences and Faculty of Pharmacy. The survey was administered via online to a total of 497 students comprising of 124 males and 373 females. A Malaysian intelligence instrument known as UKM2 is used to measure students cognitive performances. A total number of 370 items have been administrated to test various 13 aspects of cognitive intelligence which comprises of (1) similarities (23 items), (2) vocabulary (30items), (3) comprehension (19 items), (4) information (24 items), (5) word reasoning ( 20 items), (6) block design (13 items), (7) picture concept, (28 items), (8) matrix reasoning (38 items), (9) picture completion (38 items). (10) digit span (16 items), (11) letter-number sequencing (30 items), (12) arithmetic (29 items) and (13) symbol search (62 items). The item reliability in the instrument range between 0.89 to Data of all the constructs are in the form of ISSN: ISBN:

3 dichotomous or partial credit. Test results are classified into four categories of Verbal Comprehension Index (VCI), Perceptual Reasoning Index (PRI), Working Memory Index (WMI) and Processing Speed Index (PSI). The Rasch model is an effective tool that generates quality information for meaningful use to construct a valid and reliable instrument [2]. In this study, the Rasch model analysis using the Winsteps software version was employed to identify the existence of GDIF intensity in the four indices of UKM2 instruments. GDIF index with a magnitude of ±0.5 is considered important and meaningful [5]. A two-tailed t-test was carried out to analyze the DIF significance of the differences between two difficulty indices. At 95% confidence level, the critical t value used to determine the level of significance ± 2.0. Therefore, the t-value of ± 2.0 is accepted as significant GDIF. genders in answering the test, the direction of biased items is not consistent for every construct. Out of 153 items showing existence of GDIF, 77 items (50.3%) was easier for males, while 76 items (49.7%) were easier for females. There is fairly substantial percentage of items favoring both genders in the test. Table 2 Clustering of significant GDIF items according to four indexes 3 Problem Solution The total number of respondents is 467. There were 373 students (75.1%) who were females and 124 of them were males (24.9%). However, the ratio of samples is only based on the population studied. There are many educational research studies which highlighted upon diversity of student s characteristics especially on gender, but fewer studies examined the DIF between genders in explaining the differential performance among them. GDIF analysis is performed to determine the existence of biased items in the UKM2 intelligence test. Table 1 shows the analysis GDIF items in each construct of the UKM2. Table 1 Analysis of GDIF items in the UKM2 The findings reveal that 153 items (41.35%) displayed the contrast between genders that is greater than 0.5 logit. Despite showing differences between ISSN: ISBN:

4 The differentials across genders are very modest and the direction of biasness is unsystematically in every construct. The moderate bias items are not problematic, especially when the direction of bias is unsystematically. Study indicated that item-bias does not weaken the measurement quality and predictive validity of the overall test [11]. These should have fairly substantial percentage of items in both groups ability that exhibit DIF moderately and tend not to affect the test quality as there is no advantage to remove these items. For that matter, this study suggests to extend the analysis to identify the extreme level of GDIF that could exist in the items. We can predict which items are likely to exhibit biasness and remove the most significant items exhibiting DIF for better test fairness by using the Rasch Model. Table 2 shows the clustering of significant GDIF based on four indices. There are 29 items being identified as significant of GDIF with the critical t-value set at ±2.0 and the confidence level at 95%. The negative t-value and GDIF size in this study shows that the items were easily answered by male than female students. In contrast, a positive t-value indicates that the items were answered easily by female students. Out of 29 items, 16 items (55.2%) were in favor of the males while 13 other items (44.8%) were in favor of the females. This study reveals that there is no significant difference between both genders in answering the five constructs (vc,cp,if,at and bd ). All the results were classified into four indexes (VCI, PRI, WMI and PSI) according to the themes. These four indexes are then interpreted as they are closed related and relevant to students intelligence abilities across genders. This study reveals that the VCI and WMI are of an advantage to male students. This finding is astonishing as many literatures suggest that females are better off with verbal abilities. This might be attributed to the different characteristics of tasks given. Study by [8] found that males overcome females in general knowledge. It is suggested that VCI in online test is more relevant to general knowledge. Furthermore, the WMI which composed of digit span, letter-number sequencing and arithmetic offers more advantage to male students as suggested by [1]that male students have an advantage with regards to mathematics. The finding however is not consistent with [6]who found that female students are more dominant in the Malaysian Mathematic National Exam. A study showed that the WMI index is an essential component of fluid reasoning and other higher order cognitive process [17]. Therefore, males perform better as they can express ideas and thoughts in the online test without fear and can practice thinking critically, as it is one of the higher order cognitive process [4]. As for the PRI index which focuses on measuring perceptual organizing skills and motor skills, this study discovers that there are differences in item functioning that favor females. It is rather an unexpected finding because PRI is likely to have advantage on males especially on the motor skill. The constructs on mr, pc and pm show that most displayed significant DIF are favoring females but there is no significant DIF on bd construct. The finding of this study suggests that the three constructs which lay emphasis on characteristics associated with organizational skills, are related to appropriate behavior of the females [10], that they are more patient and persistence when performing online test assessment. Meanwhile, the PSI index is only showing a marginal difference in gender ability. There are five significant GDIF items of PSI that comprised only the symbol search construct. Three items that show significant GDIF are easier for the females while two other items are in favor of male students. The items in favor of female students are items that appeared in the initial set of the questions of this construct, while the items in favor of male students emerged at the end of this set of construct. This shows that gender differences in speed-accuracy trade off in a timed test such as in PSI, where males works faster but less accurate than females, as previously reported by [15]. Male students will score on more difficult items at the end and less likely to be successful on easier items at the beginning of the test. In this situation, the mean scores for both genders might be identical but males with less ability might obtain a higher proportion of low scores than females of otherwise similar ability, while those with higher ability might gain a higher proportion of high scores than females of otherwise similar ability. The magnitude of GDIF favoring males in VCI and WMI show that ability tendencies in verbal intelligence is strongly related to academic achievement [9]. Given this close association, educators must be careful to avoid general conceptions of males as under achievers. On the other hand, GDIF analysis shows that females are prone to have more non-verbal intelligence than males. These outcomes suggest that the perception organization and motor skills may be obtained with persistence and accuracy. 4 Conclusion DIF analysis is an important tool for test development. Test is likely to be more valid and less biased when items that exhibit extreme levels of DIF are identified and omitted. In UKM2 test, the content analysis through items that exhibited significant GDIF illustrated this point well. This result confirms that there is a gender differentiation in intellectual pursuits but at the same ISSN: ISBN:

5 time suggests a differential intensity of DIF between genders. The information on genders clustering tendency of item displaying GDIF carries implication for both test developers and educators. For test developers, the information of the genders clustering tendencies in answering the GDIF items can be used as guidelines to improve the construction of items in achievement test to give fairness to students of different genders. Meanwhile educators can use the information to identify the strengths and weaknesses of their students so that more meaningful teaching and learning activities can be planned. The use of online instrument can be extended to investigating other factors such as ethnic groups, socioeconomic status or age groups that may contribute to DIF. Further studies are needed to understand the different performance in DIF items between groups, especially when the usage of online medium test still need to be developed. References: [1] Ayalon, H., Women and Men Go to University: Mathematical Background and Gender Diffrences in Choice of Field in Higher Education. Sex Roles. Vol. 48, No.5/6, 2003, pp [2] Azrilah,A.A., Azlinah,M., NoorHabibah, A., Sohaimi,Z., Azami,Z., Hamza,A.G.,& Mohd,S.M., Application of Rasch Model in Validating the Construct of Measurement Instrument, International Journal of Education and Information Technologies, Vol.2, No.2, 2008, pp [3] Campanella,S., Dimauro,G., Ferrante,A., Impedevo, S., Lucchese,M.G., Modugno,R.,Pirlo,G., Sareinella,L., Stasolla,E. &Trullo,C.A, Engineering e-learning Surveys: A New Approach, International Journal of Education and Information Technologies, Vol.1, No.2, 2007, pp [4] Fang,F.J.,Yang,H.J.,TsaiH.L.,Lee,C.J.,Tsai,T.S.& Li,D.H, A Type of Technological Course with Using Information Technologies, International Journal of Education and Information Technologies, Vol.1, No.4, 2007, pp [5] Goldbeck L., Daseking M., Brida S.H., Waldmann H.C, & Petermann F.,. Sex Differences on the German Weshler Intelligence Test for Children (WISC-IV), Journal of Individual Differences,Vol.31,No.1, 2010,pp [6] Hanizah Hamzah, Siti Rahayah Ariffin & Ruhizan Mohd Yassin, Girls Overachievement in the Mathematics National Examination: Can Type of Items be the Cause? Journal of Science and Mathematics Education in Southeast Asia, Vol. 29, No.2, 2006,pp [7] Kail, R.,Speed of Information Processing: Developmental Change and Links to Intelligence, Journal of School Psychology, Vol.38,No.1,2000,pp [8] Lynn,R., Irwing,P., & Cammork, Sex Differences in General Knowledge, Intelligence, Vol.30, No.1, 2002, pp [9] Mayes S.D., and Calhoun S. L., Wechsler Intelligence Scale For Children-Third And -Fourth Edition Predictors Of Academic Achievement In Children With Attention-Deficit/Hyperactivity Disorder, School Psychology Quarterly Vol.22, No. 2, 2007,pp [10] Meriac J.P, Polling T.L, & Woehr D.J..Are There Gender Differences in Work Ethics? An Examination of the Measurement Equivalence of the Multidimensionalwork Ethic Profile, Personality and Individual Differences,Vol.47, 2009,pp [11] Sheppard,R., Kyunghee Han, Colarelli, S. M., Guangdong Dai & King, D. W.,Differential item functioning by sex and race in the Hogan Personality Inventory,Assessment,Vol.13, No.4, 2006,pp [12] Siti Rahayah Ariffin & Nor Azaheen Abdul Hamid, Critical Thinking Skills Profile between the Science and Non-science students, Malaysian Education Deans' Council Journal, 3, 2009, [13] Siti Rahayah Ariffin, Rodiah Idris & Noriah Ishak, Differential Item Functioning in Malaysia Generic Skills Instrument, Journal of Education,Vol. 32,No.1, 2010,pp [14] Siti Rahayah Ariffin, Roseni Ariffin dan Hafsa Mohamed Makki, Contribution Factors in Multiple Intelligences Among Adolescene Students, Journal of Education, 33, 2008, [15] Strand, S., Deary, I. J. & Smith P, Sex differences in cognitive abilities test scores: A UK national picture, British Journal of Educational Psychology,Vol.76, 2006,pp [16] Wang,L.C.& Chen,M.P, Enhancing ICT Skills Learning Through Peer Learning: Perspectives of Learning Style and Gender, International Journal of Education and Information Technologies, Vol.2, No.1, 2008, pp [17] Williams,P.E, Weiss,L.G., & Rolfhus,E.,Clinical Validity (WISC-IV Technical Report #1), Psychological Corp.,Texas, [18] Zalizan Mohd Jelas, Saemah Rahman, Roselan Baki & Jamil Ahmad,Prestasi Akademik Mengikut Gender, Journal of Education,Vol.30, 2005,pp ISSN: ISBN:

Validity and Reliability of the Malaysian Creativity and Innovation Instrument (MyCrIn) using the Rasch Measurement Model

Validity and Reliability of the sian Creativity and Innovation Instrument (MyCrIn) using the Rasch Measurement Model SITI RAHAYAH ARIFFIN, FARHANA AHMAD KATRAN, AYESHA ABDULLAH NAJIEB BADIB & NUR AIDAH