Validity and Reliability of the Malaysian Creativity and Innovation Instrument (MyCrIn) using the Rasch Measurement Model

Transcription

1 Validity and Reliability of the sian Creativity and Innovation Instrument (MyCrIn) using the Rasch Measurement Model SITI RAHAYAH ARIFFIN, FARHANA AHMAD KATRAN, AYESHA ABDULLAH NAJIEB BADIB & NUR AIDAH RASHID Universiti Kebangsaan sia UKM, Bangi, Selangor MALAYSIA Abstract: - The sian Creativity and Innovation Instrument (MyCrIn) is a test designed to identify creativity and innovation among University students in sia. A sample of 285 students from a local public university in sia were randomly selected for the study. This study aims to determine the reliability and validity of MyCrIn by using item analysis. MyCrIn comprises of 5 constructs which include High Order Thinking (HOT), Curiosity (CUR), Sensitivity (SEN), Visionary (VIS) and Adaptable to Change (ATC). Using the Rasch Measurement Model, analysis shows that eleven items in the form of a five-point Likert-scale were detected as misfits while no Dichotomous and Partial credit items were detected as misfits. Differential Item Functioning (DIF) based on gender was detected in three Likert-scale items, two Dichotomous items and three Partial Credit items. DIF based on race was found in ten Likert-scale items and one Partial Credit item. The results of the study can be used to determine more accurately the level of creativity and innovation among sian students. Key-Words: - sian Creativity and Innovation Instrument, Rasch Model, validity, reliability 1 Introduction Creativity and innovation is a potential trait that exists within every individual. Those who display their creativity and innovation are those who have tapped into their inner creativity. Every individual possesses a creative and innovative potential in their respective fields [1]. Creative and innovative individuals possess the intrinsic quality of curiosity to produce and achieve something. This allows the individual to think, feel, concentrate and manipulate the environment in order to identify and understand their surrounding information. Manipulation of information makes an individual brave enough to take risks and make mistakes. This study uses the sian Creativity and Innovation Instrument (MyCrIn) to examine the level of creativity and innovation among students of Higher Learning Institutions (HLIs) in sia. The study was conducted to determine the reliability and validity of MyCrIn. In using the Rasch Measurement Model, item analysis is determined based on the values of item and person reliability, separation, unidimensionality, difficulty and fit statistics [4], [7], [6]. Item analysis is a statistical and empirical method that provides detailed information about the item [2], [3]. The study of psychological testing is essential in assessing the quality of the instrument developed in terms of instrument reliability and validity. The Item Response Theory (IRT) is the modern test theory used for item analysis in the development of an instrument [4]. IRT is a statistical approach based on the Item Characteristics Curve (ICC), which explains individual ability to answer the item correctly [4], [7], [6], [10]. Item analysis using the Rasch Measurement Model to determine instrument reliability and validity will be used to scrutinise each and every item [7], [6]. The Rasch Measurement Model analyses the quality of an item in an instrument according to the reliability and separation between items and individuals. [2], [3], [7], [6], [10] have conducted research on item analyses using the Rasch Measurement Model to determine instrument reliability and validity in their respective fields of education, psychology, health and medicine. Instrument reliability refers to the ISBN:

2 consistency of test results over time. Validity refers to what is measured by the instrument or the purpose of the test being carried out. The objective of the study is to examine the psychometric characteristics of the sian Creativity and Innovation Instrument (MyCrIn) through several aspects, which include 1) validity and reliability of the instrument 2) item difficulty 3) the existence of Differential Item Functioning (DIF) based on gender and race. The findings of the study will be a guide for relevant authorities to identify creative and innovative students of the nation in the development of the country towards a better future. 2 Methodology MyCrIn consists of 290 items comprised of the five constructs of High Order Thinking (HOT), Curiousity (CUR), Sensitivity (SEN), Visionary (VIS) and Adaptable To Change (ATC). The study was conducted on 1321 randomly selected students from a local public university in sia. However, analysis for this study will be carried out on 285 students from the Faculty of Pharmacy, and the Faculty of Economics and Business. MyCrIn was used to examine the level of creativity and innovation among students in sia. This instrument contains 290 items in the form of 5-point Likert scale, Dichotomous scale and Partial Credit scale. Item analysis is analysed using the program Winsteps The study aims to determine the reliability and validity of MyCrIn using the Rasch Measurement Model. 3 Research Findings The respondents comprised of 285 randomly selected students from a local public university in sia. As seen in Table 1 below, 36.10% of the respondents were males and 63.90% were females. 208 respondents were ethnically and 77 were non-s. 40 (14.00%) were in their first year of university, while 114 (40.00%) were in their second year, 65(22.80%) were in their third year and 66 (23:20%) in their final year. An analysis of summary statistics was conducted to determine item-person reliability and item-person separation index of the Likert-scale items. Results show, as seen in Table 2 below, item reliability is 0.98 and person reliability at Item separation is at 6.64 and person separation at Table 1 - Respondent Demography N Demography Frequency Percentage (%) Gender 285 Male Race Non Univ. Year Table 2 Summary Statistics of Likert Scale TABLE 3.1 Likert ZOU453ws.txt Apr 29 11: INPUT: 285 persons, 180 items MEASURED: 285 persons, 180 items, 5 CATS 3.49 SUMMARY OF 285 MEASURED persons RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.10 ADJ.SD.48 SEPARATION 4.86 person RELIABILITY.96 MODEL RMSE.09 ADJ.SD.48 SEPARATION 5.41 person RELIABILITY.97 S.E. OF person MEAN =.03 VALID RESPONSES: 98.9% SUMMARY OF 180 MEASURED items RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.08 ADJ.SD.50 SEPARATION 6.64 item RELIABILITY.98 MODEL RMSE.07 ADJ.SD.50 SEPARATION 7.06 item RELIABILITY.98 S.E. OF item MEAN =.04 UMEAN=.000 USCALE=1.000 Figure 2 Item Person Mapping of Likert Scale TABLE 12.2 Likert ZOU856ws.txt Apr 29 12: INPUT: 285 persons, 180 items MEASURED: 285 persons, 180 items, 5 CATS persons MAP OF items <more><rare> # T Q004.#.##.### S Q050 1 ######## +T Q145.######### Q037 Q045 Q097 Q121 Q171.########## M Q012 Q024 Q031 Q034 Q035 Q036 Q046 Q051 Q054 Q096 Q100 Q104 Q105 Q106 Q107 ########## S Q026 Q032 Q063 Q065 Q073 Q074 Q101 Q108 Q114 Q122 Q124 Q167 Q172 Q174.######## Q005 Q007 Q008 Q010 Q014 Q033 Q044 Q066 Q069 Q075 Q083 Q084 Q099 Q102 Q110 Q111 Q162 Q173 Q176.###### S Q003 Q006 Q017 Q022 Q029 Q038 Q042 Q049 Q052 Q072 Q089 Q090 Q092 Q098 Q103 Q109 Q112 Q119 Q120 Q123 Q130 Q131 Q149 Q159 Q160 Q166 Q168 0 ##### +M Q002 Q027 Q043 Q047 Q057 Q058 Q071 Q076 Q077 Q082 Q091 Q095 Q118 Q129 Q133 Q138 Q153 Q155 Q164 Q165 Q179 # Q001 Q013 Q021 Q040 Q048 Q064 Q068 Q080 Q085 Q086 Q087 Q088 Q113 Q132 Q134 Q136 Q139 Q141 Q150 Q151 Q163 Q169 Q170 Q175. T Q015 Q016 Q025 Q028 Q067 Q078 Q079 Q126 Q128 Q152 Q156 Q158 Q161. S Q020 Q039 Q053 Q062 Q070 Q081 Q093 Q094 Q127 Q135 Q144 Q147 Q154 Q157 Q180 Q030 Q041 Q056 Q116 Q117 Q125 Q137 Q146 Q177 Q178 Q011 Q019 Q059 Q061 Q142 Q143 Q T Q023 Q055 Q115 Q009 Q018 Q140. Q <less><frequ> EACH '#' IS 4. ISBN:

3 An analysis of summary statistics, as seen in Table 3, was conducted to determine the reliability and separation index of the Dichotomous items with two option responses (1 = correct answer and 0 = wrong answer). Results found item reliability at 0.99 and person reliability at Item separation is at 9.09 and person separation at Table 3-Summary Statisctics of Dichotomous TABLE 3.1 Dichotomous ZOU837ws.txt Apr 29 11: INPUT: 285 persons, 71 items MEASURED: 285 persons, 71 items, 2 CATS 3.49 SUMMARY OF 283 MEASURED (NON-EXTREME) persons RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.33 ADJ.SD 1.01 SEPARATION 3.08 person RELIABILITY.90 MODEL RMSE.32 ADJ.SD 1.02 SEPARATION 3.23 person RELIABILITY.91 S.E. OF person MEAN =.06 MINIMUM EXTREME SCORE: 2 persons VALID RESPONSES: 97.5% SUMMARY OF 285 MEASURED(EXTREME AND NON-EXTREME) persons RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.36 ADJ.SD 1.13 SEPARATION 3.12 person RELIABILITY.91 MODEL RMSE.35 ADJ.SD 1.13 SEPARATION 3.23 person RELIABILITY.91 S.E. OF person MEAN =.07 SUMMARY OF 71 MEASURED (NON-EXTREME) items RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.18 ADJ.SD 1.64 SEPARATION 9.09 item RELIABILITY.99 MODEL RMSE.17 ADJ.SD 1.64 SEPARATION 9.54 item RELIABILITY.99 S.E. OF item MEAN =.20 UMEAN=.000 USCALE=1.000 Figure 3 Item-Person Mapping of Dichotomous TABLE 12.2 Dichotomous ZOU612ws.txt Apr 29 11: INPUT: 285 persons, 71 items MEASURED: 285 persons, 71 items, 2 CATS 3.49 persons MAP OF items <more><rare> 4 + Q227 T Q Q204VI. Q204IV Q223.. Q209 2 # + Q200A Q204III Q214 Q222 Q224. T Q200B Q207 Q211 Q221.### S Q206 Q215 #.### Q226 #### Q213 Q216B 1 ##### + Q210 Q220.## S Q191 Q194.###### Q205VIII Q208 Q228 ####### Q201 Q216A Q218 ######## Q199B.####### Q195 Q198 Q212 0 ##### +M Q199A Q204V Q204VII ########### ########### M Q204VIII Q205V Q225.###### Q192 Q205VII Q219 ######### Q197 Q202III Q205III.####### Q193 Q203I Q203IV -1.####### + Q196 ########## Q205VI.### S Q190 Q202IV Q204I.##### Q187 Q203III Q205IV.### S Q188 Q203II Q205II.## Q202I Q202II Q204II Q205I -2 # + # Q186 ## T Q181.# Q182 Q189 Q185-3 # +. Q184 T Q # + <less><frequ> EACH '#' IS 2. An analysis of summary statistics, as seen in Table 4, was conducted to determine the reliability and separation index of the partial credit items with three response options (2 = correct answer, 1 = most appropriate answer and 0 = wrong answer). Results found item reliability at 0.99 and person reliability at Item separation is at 8.22 and person separation at Table 4-Summary Statistics Partial Credit TABLE 3.1 Partial Credit ZOU925ws.txt Apr 29 11: INPUT: 285 persons, 24 items MEASURED: 285 persons, 24 items, 3 CATS 3.49 SUMMARY OF 230 MEASURED (NON-EXTREME) persons RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.41 ADJ.SD 1.15 SEPARATION 2.80 person RELIABILITY.89 MODEL RMSE.38 ADJ.SD 1.16 SEPARATION 3.04 person RELIABILITY.90 S.E. OF person MEAN =.08 MINIMUM EXTREME SCORE: 55 persons VALID RESPONSES: 99.9% SUMMARY OF 285 MEASURED(EXTREME AND NON-EXTREME) persons RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.87 ADJ.SD 1.77 SEPARATION 2.04 person RELIABILITY.81 MODEL RMSE.86 ADJ.SD 1.78 SEPARATION 2.07 person RELIABILITY.81 S.E. OF person MEAN =.12 SUMMARY OF 24 MEASURED (NON-EXTREME) items RAW MODEL INFIT OUTFIT SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD MEAN S.D MAX MIN REAL RMSE.11 ADJ.SD.93 SEPARATION 8.22 item RELIABILITY.99 MODEL RMSE.11 ADJ.SD.93 SEPARATION 8.59 item RELIABILITY.99 S.E. OF item MEAN =.19 UMEAN=.000 USCALE=1.000 Figure 4 Item-Person Mapping of Partial Credit TABLE 12.2 Likert ZOU925ws.txt Apr 29 11: INPUT: 285 persons, 24 items MEASURED: 285 persons, 24 items, 3 CATS 3.49 persons MAP OF items <more><rare> 3 + Q242 T T Q241 # Q ## + Q240 Q244.# SS.###. Q245..#### Q233 Q252.#.## 0. +M Q230 Q232 Q248.## Q247.## Q239.# M Q229 Q235 Q246. Q249.# Q236 Q250.##.# S Q234 Q237 Q Q238 # ISBN:

4 ..# S..# Q231 # T #.## T -3 +.# -4 ########### + <less><frequ> EACH '#' IS 5. Following Wright and Linacre (1992), the mean square (MNSQ) infit and outfit for each item and respondent must fall within the range 0.6 to 1.5, while Bond and Fox (2007) states that the MNSQ infit and outfit for every item and respondent should fall within the range of 0.6 to 1.4. However, if an item or person does not fall within the range, then it may be deleted. In this study, the range stated by Bond and Fox (2007) will be used. According to this range, eleven Likert-scale items (items Q005, Q010, Q012, Q017, Q031, Q045, Q063, Q084, Q119, Q120 and Q178) were identified as misfit items, as seen in Table 5, while there were no misfit items for the Dichotomous and Partial Credit items. Table 5 Misfit Item Analysis for Likert-Scale Item Raw No Measure Model Infit Outfit Score Error MN SQ Z STD MN SQ Z STD Q Q Q Q Q Q Q Q Q Q Q A t-test was used to analyse Differential Item Function (DIF) based on gender and race to determine any significant difference. According to Siti Rahayah et al. (2008), at 95% confidence level, the critical t value used to determine significance is 2.0. Therefore, if t ± 2.0, it is accepted as significant DIF. DIF index, or DIF contrast, with a magnitude ± 0.5 is considered as important and gives meaning to the significant DIF (Bond & Fox 2007). Table 6 shows that three Likert-scale items, two Dichotomous items and three Partial Credit items were detected to have DIF based on gender (GDIF). These items did not meet the DIF t value (t ± 2.0 logit) and DIF Contrast (p ± 0.5 logit). The three Likert-scale items of Q031, Q055 and Q175 were in favour of the female respondents, where it is easier for female respondents to answer the items correctly. Dichotomous item Q199B was detected in favour of male respondents, where it is easier for male respondents to answer correctly, while item Q221 favoured female respondents. Partial Credit items Q243 and Q250 favoured females while item Q252 favoured males. Table 6 Differential Item Functioning Analysis based on Gender Item Gender DIF Contrast t Bias Q031 Male Q055 Male Q175 Male Q199B Male Male Q221 Male Q243 Male Q250 Male Q252 Male Male As seen in Table 7 below, ten Likert-scale items and one Partial Credit item were detected to have DIF based on race. did not meet the DIF t value (t ± 2.0 logit) and DIF Contrast (p ± 0.5 logit). Six Likert- scale items of Q003, Q014, Q046, Q084, Q091 and Q119 favoured respondents who were not s, where it was easier for non-s to answer the item correctly, and four Likert-scale items of Q136, Q143, Q0156 and Q158 favoured s, where it is easier for s to answer the said items correctly. Partial Credit item Q236 DIF is in favour of races other than s. This indicates a significant difference in item difficulty between males and females. DIF analysis also shows item difficulty between different groups that have similar levels of abilities. A negative DIF index shows that the item is easily agreed upon by a certain group while a positive DIF index means that the item is more difficult to be agreed upon by groups who have similar abilities but with different levels of probability in answering the item correctly. ISBN:

5 Table 7 DIF based on Race Item Race DIF Contrast t Bias Q Other than Other than Q Other than Other than Q Other than Other than Q Other than Other than Q Other than Other than Q Other than Other than Q Other than Q Other than Q Other than Q Other than Q Other than Other than Discussion Rasch analysis was used to generate separation index of person and items. [17] defines person separation index as the separation estimate or individual group differences according to the level of ability in the measured variable. The person separation index indicates the separation index ability strata of individuals identified in the sample. As found in this study, there were 5 levels of abilities in the Likert sclae items, and 3 levels of ability in the Dichotomous and Partial Credit items. Item separation index on the other hand, indicates the separation index of item difficulty. The Likertscale items were found to have 7 levels of difficulty, while the Dichotomous items had 9 levels of item difficulty and 8 levels of item difficulty in the Partial Credit items. According to Linacre [5], a person and item separation index above 2 is considered good. A separation index value 2.0 grades the measurement system as only one or two observations between 1.5 and 2.0 is not productive for the development of measurement but does not lower the level. Values between 0.5 and 1.5 is productive, while a value <0.5 is less productive for measurement but does not lower the level. It allows removing confusion on high reliability as well as separation coefficient [5]. The person and item separation for all the items of MyCrIn are therefore considered good and has fair representation across the scale. DIF analysis was conducted in order ensure that the respondents undergoing the test are fairly represented without elements of biasness. The study detected three Likert scale items, two Dichotomous items and three Partial Credits items with DIF based on gender, and ten Likert scale items and one Partial Credit item to have DIF based on race. These items need to be rectified or deleted in order to have a fairly represented instrument. A high validity and reliability index also needs to be ensured in order to provide an accurate picture of what is to be measured. As seen from the results, MyCrIn has a high reliability and validity index, however, slight modifications need to be taken for future research in order to strengthen the quality and credibility of MyCrIn in identifying creative and innovative individuals. 5 Conclusion Results of the data analysis using Winsteps records a high level of person and item reliability index as well as a consistent flow of item difficulty indicates that MyCrIn has a high level of reliability and validity. These findings support previous studies that an individual should know their level of creativity based on their own strengths in order to face life s challenges. It is recommended that creativity should be inculcated in teaching and learning in schools and in institutions of higher learning. This will in turn develop skills that will contribute to a more holistic individual. Therefore, the results of this study hopes to pave way for relevant authorities to implement an intervention programme in order to help students cultivate their creative potential for the development of the nation in their respective fields. References: [1] Ainon Mohd & Abdullah Hassan Kepintaran Daya Cipta & Kemahiran Berfikir. Utusan Publications & Distributors Sdn Bhd. KL [2] Azrilah Abdul Aziz, Azlinah Mohamed, NoorHabibah Arshad, Sohaimi Zakaria & Mohd Saidfudin Masodi, Appraisal Of Course Learning Outcomes Using Rasch Measurement: A Case Study In Information Technology Education. International Journal of Systems Applications, Engineering & Development.4, 1, 2007, [3] Azrilah Abdul Aziz, Azlinah Mohamed, NoorHabibah Arshad, Sohaimi Zakaria, Azami Zaharim, Hamza Ahmad Ghulman & Mohd Saidfudin Masodi, Application of Rasch Model ISBN:

6 in Validating the Construct of Measurement Instrument, International Journal of Education and Information Technologies, 2, 2, 2008, [4] Bond, T. G. & Fox, C. M., Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 2 nd ed, New Jersey: Lawrence Erlbaum Associated, London, [5] Linacre, J.M WINSTEPS. Chicago: MESA Press. [6] Rodiah Idris, Siti Rahayah Ariffin & Noriah Ishak, Application Of Rasch Model In Validating The Construct Of Measurement For Generic Skills Instrument For Higher Education (GeSIHE), Conference of the Pacific Rim Objective Measurement Symposium PROMS July, Hong Kong, 2009c, [7] Siti Rahayah Ariffin, Noriah Mohd Ishak, Riza Atiq O.K Rahmad, Abdul Ghafur Ahmad, Rodiah Idris, Nur Ashiqin Najmuddin, Assessing Generic Skills Using Rasch Model Approach: A Method for Construct Validity and Reliability, International Conference on Education, [8] Siti Rahayah Ariffin, Roseni Ariffin dan Hafsa Mohamed Makki, Contribution Factors in Multiple Intelligences Among Adolescene Students, sian Journal of Education, 33, 2008, [9] Siti Rahayah Ariffin & Nor Azaheen Abdul Hamid, Critical Thinking Skills Profile between the Science and Non-science students, sian Education Deans' Council Journal, 3, 2009, 1-22 [10] Siti Rahayah Ariffin, Rodiah Idris & Noriah Mohd. Ishak, Differential Item Functioning in sian Generic Skills Instrument, sian Journal of Education, 35, 1, 2010, 1-10 [11] Siti Rahayah Ariffin Theory, Concept and Practice in Evaluation and Testing. Academic Development Centre, National University of sia. [12] Siti Rahayah Ariffin Innovation in Educational Evaluation and Measurement. Faculty of Education, National University of sia. [13] Siti Rahayah Ariffin, Rodiah Idris & Nur Ashiqin Najmuddin Inovation using Rasch Model Approach in Measuring Generic Skills. International Conference on Education, [14] Siti Rahayah Ariffin, Noriah Mohd Ishak, Roseni Ariffin, Abdul Ghafur Ahmad & Rodiah Idris Evaluation Approaches and Challenges using Structural Equation Model (SEM). Proceeding International Conference on the Education of Learner Diversity [15] Siti Rahayah Ariffin, Rodiah Idris & Noriah Mohd Ishak. 2008e. Profilig of Generic Skills of Students from Higher Learning Institutions: Research of National University of sia. National Seminar on Teacher Education Supervising Committee, [16] Siti Rahayah Ariffin, Rodiah Idris & Noriah Mohd Ishak Differential Item Functioning in sian Generik Skills Instrument. sian Journal of Education, 35 (1): In Press [17] Wright, B.D. & Masters, G.N Rating Scale Analysis. Rasch Measurement. Chicago: Mesa Press. ISBN: