Verification of multiple intelligences construct validity in an online instrument

Transcription

1 Available online at Procedia Social and Behavioral Sciences 9 (2010) WCLTA 2010 Verification of multiple intelligences construct validity in an online instrument Siti Rahayah Ariffin a, Ida Kamalawati Abu Bakar b*, Mohd Sahardi Che Harun c, Anita Isa d a Professor, Universiti Kebangsaan Malaysia, Malaysia bcd Universiti Kebangsaan Malaysia, Malaysia Abstract This research aims to verify the construct validity of an online multiple intelligences assessment instrument. It contains 81 perception items and is administered online by random sampling. The Person and Items Distribution Map (PIDM) indicates good reliability attributes. The analysis yields 69 items from which 12 items were dropped. To further ensure the accuracy and consistency of this finding, another study is conducted. The Confirmatory Factor Analysis (CFA) indicates an acceptable fit between the model and the data. The result of the study shows that the good psychometric properties can be used to obtain the multiple intelligences profiles of youths Published by Elsevier Ltd. Open access under CC BY-NC-ND license. Keywords: Multiple intelligences; construct validity; online; psychometric testing; Rasch; Structural Equation Modeling (SEM); validity; reliability; instrument. * Ida Kamalawati Abu Bakar. Tel.: ; fax: address: idakamalawati@gmail.com Published by Elsevier Ltd. Open access under CC BY-NC-ND license. doi: /j.sbspro

2 Siti Rahayah Ariffi n et al. / Procedia Social and Behavioral Sciences 9 (2010) Introduction There are various interpretations of intelligence that have been studied such as the ability to learn or understand from experiences, the ability to receive and store knowledge, mental ability to respond quickly and successfully in new situations and the mental capacity to analyze a given situation (Sternberg 1994: 890). Sarwono (1979) concluded that intelligence is the overall ability of the individual to think and act as directed, and to effectively process and control the environment. There are various theories about intelligence that are along the lines of the theory of multiple intelligences. The theory was first proposed by Gardner (1993) in his book Frames of Mind: The Theory of Multiple Intelligences. He found seven types of intelligences that sometimes work simultaneously. They are the verbal-linguistic intelligence, logical-mathematical intelligence, visual-spatial intelligence, kinaesthetic intelligence, musical intelligence, interpersonal intelligence, and intrapersonal intelligence; to which naturalistic intelligence and spiritual intelligence were subsequently added. In his recent work Gardner (2006, p.50) also considered intelligence as "raw, biological potentials, which can be seen in its pure form only in individuals who are, in the technical sense, freaks." 2. Multiple intelligences measurement Academic achievement is related to multiple intelligences (Wu and Alrabah, 2009). It is a bio-psychological potential to process certain types of information in a specific way to produce something meaningful in a community (Gardner, 1993). However, different individuals have different and distinctive multiple intelligences. Many researchers have begun to explore the relationship between multiple intelligences and the academic performance of students. Gardner s multiple intelligences theory (1993) emphasized the diversity of students. Hence, various instruments have been developed to identify and measure intelligences. Multiple intelligences measurement models have long been developed by researchers in other countries to evaluate the relationship between intelligence and achievement of students (Shearer and Luzzo, 2009). IQ tests have always been regarded as the best benchmark to categorize and place students into various fields. Studies have shown that IQ is one s intelligence in language, logical thinking, and calculations. Gardner (1983) removed the definition of the traditional intelligence and intelligence test from the public and education which have consisted of the language and mathematical intelligences only. He made society recognize the abilities of musicians, fashion designers, farmers, athletes, weavers and also religious leaders as intelligence. The reason being is that they meet the definition of intelligence as the ability to solve problems and produce valuable products in their sociocultural context. Moreover, their intelligences become the source of income that meet the needs and demands of their lives. Klein (1997) suggested that teachers should increase their knowledge and strategies to identify students intelligences in problem-solving and other tasks. This would enhance the effectiveness and appropriateness of implementing the multiple intelligences theory in the classroom. Therefore, schools would need a valid and reliable tool or instrument for assessing students multiple intelligences and to consolidate their preferences in choosing the appropriate tools (Siti Rahayah et al, 2006). Previous studies have shown that very few multiple intelligences measurement instruments were tested for validity. The constructs that make up this multiple intelligence measurement model need to be ensured that they contain items of high validity and reliability (Siti Rahayah et al, 2010). Construct validity according to Messick (1989) and Tran et al (2010) includes content relevance, representativeness, and criterion-relatedness. Therefore, this study is necessary to develop a multiple intelligences measurement as an online instrument that would thoroughly measure the multiple intelligences constructs and to help individuals identify their multiple intelligences ability. The instrument used in this study is the first online culture-fair instrument built to suit the local context in assessing the perception and the preferences of respondents in Malaysia against their own multiple intelligences (Siti Rahayah et al, 2008b). Data and findings of this study will provide a predictive representation of students multiple intelligences in general and particularly by their gender and field of study. 3. Method

3 1896 Siti Rahayah Ariffi n et al. / Procedia Social and Behavioral Sciences 9 (2010) The online version of the Malaysian Multiple Intelligences Checklist for Adults (e-mymica) contains 81 perception items of five-point Likert scale that represent nine different intelligence domains. It applies a quantitative approach which involves the collection of data using an electronic questionnaire that is run online. Advantages of computer-based assessment include automated and rater-free scoring, immediate feedback, and is easily accessible. Benefits associated with educational assessment include the ability to process detailed data and the potential to build tasks that assess skills that cannot be easily done by other means of assessment (Zoanetti, 2010). The remarkable advantage of internet is the effortless access to information that has led to a new, fast and handy range of tools and capabilities for innumerable fields of activity (Boboila & Boboila, 2007). The instrument is administered on a sample of 179 students aged 16 by random sampling at a public school in Selangor, Malaysia. Items are quantitatively analyzed using WINSTEPS that is based on the Rasch model to assess the suitability of items and the differential functions based on gender, race and field of study Analysis using Rasch model Data are analyzed using Rasch model with the application of WINSTEPS to test the validity and reliability of the instrument. The Rasch model also takes into account the ability of the candidate or the respondent who answered questionnaires, tests or instruments as well as the difficulty of each test item or items (Rasch, 1980). Smith (1992) suggests using item fit statistics to evaluate the extent to which items are tapped into the same construct and places test-takers in the same order to assess the items technical quality empirically in the Rasch analysis. He argues that test-takers should be ranked consistently by items measuring the same construct. Otherwise, the misfit items (items that measure a different construct compared to other items in the test) should be revised or eliminated. Table 1 shows the criteria used as benchmarks for determining the validity of the online instrument. According to Wright and Stone (1979), the conditions to produce a useful measurement are: 1. the use of valid items in the measurement process to determine the construct measurement, 2. the definition of concepts and constructs are clear and consistent with the supporting theory, 3. testing item on appropriate individuals provides results that are consistent with the purpose of measurement, and 4. the application of valid response patterns. Without a valid response pattern, the individuals cannot be defined precisely. Table 1. Summary of item validity and reliability in the multiple intelligences instrument using Rasch model Criteria Statistical Info Results Item Validity a. Item Polarity PTMEA CORR > 0 (positive value) Item 81 b. Item Fit Total mean square infit and outfit of Item Misfit = 12 c. Separation (SE) All items show 2.0 d. Person reliability Value close to 0.8 value (0.62 < x <0.78) e. Item reliability Value > 0.8

4 Siti Rahayah Ariffi n et al. / Procedia Social and Behavioral Sciences 9 (2010) The reliability of the respondents indicates that the interpretation is equivalent to Cronbach Alpha or KR20 (Wright and Masters, 1982). The respondents reliability index of 0.98 is a good value (Pallant, 2001) for the expected consistency on the logit scale for the answers on different sets of items that measure the same construct (Wright and Masters, 1982). Linacre (2007) stated that the reliability of respondents of 0.8 and respondents separation index of 2.0 as good indices. The statistics generated by Rasch analysis estimate the degree of items suitability that measure latent variables, assuring the item-fit of the instrument are within an acceptable range. There are 12 items removed because the mean square infit and outfit radius are outside the range of 0.6 to1.4 as proposed by Bond and Fox (2007) Analysis using Standard Equation Modeling (SEM) To ensure accuracy and consistency of this finding, another study is then conducted using the same instrument and data applying the Structural Equation Modelling (SEM) using AMOS. Items that have been tested on the validity and reliability form a model representing each construct. SEM is an approach to measure the quality of instruments whilst Confirmatory Factor Analysis (CFA) is an approach to measure the validity of construct items. The application of SEM over CFA helps to assess the validity of the main structural factors. A measurement model that is assessed by CFA would link factors in a model (Kline, 2005). Modification indices provided by AMOS suggests that improvement in model fit could be made by allowing several measurement errors to correlate (Byrne, 2001; Joreskog, 1993). Figure 1 shows the suitability index for both hypothesized and revised model for every nine constructs that have gone through the CFA application. The constructs are logical-mathematical intelligence (LoM), verbal-linguistic intelligence (VeL), visual-spatial intelligence (ViR), musical intelligence (MuZ), bodily-kinaesthetic intelligence (KiB), interpersonal intelligence (InE), intrapersonal intelligence (InA), naturalistic intelligence (NaR), and spiritual intelligence (KeR). The overall results of analysis proof an overall fit and are deemed acceptable according to Arbuckle and Wothke (1999). The CFA shows that all 69 constructs of the online instrument fits the empirical data based on the Comparative Fit Index (CFI) of 0.9, and the Tucker Lewis Index (TLI) of 0.9. Meanwhile, the Root Mean Square Error of Approximation (RMSEA) of 0.08 is at an acceptable index, and the Chi-Square value of <2.0 or <3.0 indicates an acceptable fit between the model and the data. Result of the study shows that the good psychometric properties can be used to obtain the multiple intelligences profiles of youths. Hypothesized model Revised model Figure 1. Hypothesized model and revised model of multiple intelligences constructs using SEM

5 1898 Siti Rahayah Ariffi n et al. / Procedia Social and Behavioral Sciences 9 (2010) Discussion The validity and reliability of each item in the instrument are essential. Data is also important and the accuracy of data entry needs to be ensured as it contributes to the validity and reliability of the results. Thus, the e-mymica system is programmed to automatically extract the data online directly into the Microsoft Excel format. When the validity and reliability of the instrument is proven high, it can be construed that the instrument is valid and reliable. Although the pen-and-paper version of MyMICA had been tested on the validity and reliability by previous researchers, the instrument is yet retested since inferences obtained earlier were only suitable for a certain purpose and sample especially when the analysis were done using the classical test theory (CTT). The item response theory (IRT) with the application of the Rasch model using WINSTEPS should indicate high item reliability index and person reliability index. The second analysis using the SEM approach is in line with Stapleton (1997) who explained that the CFA method is suitable for testing a model developed based on theories, where the researcher starts with a hypothesized model before the analysis is conducted. The hypothesized model determines which indicator or variable are associated with latent variables. Once this model is predicted, a set of measurements are done to assess the goodness of fit. The e- MyMICA indicates that the items contribute to the measurement of each construct, while all the nine constructs show high construct validity. Thus the structure of nine constructs containing 69 items produce a valid measurement model and is deemed effective in assessing the level of multiple intelligences. The findings of an assessment will inform the relevant parties in an educational institution on the possibilities of implementing interventions that can enhance students competencies and skills described by the multiple intelligences. Students should also be made aware of their performance and tendencies, and the opportunity to improve themselves to a level of excellence before they embark on future undertakings. References Arbuckle, J.L & Wothke,W. ( 1999 ). Amos user s guide version 4.0, Chicago, IL : Small water. Boboila, C. & Boboila, M.S. (2007). Online Evaluation with Trainee-Adaptive Tests. Proceedings of the 4th WSEAS/IASME International Conference on Engineering Education. Greece. Byrne, B. M. (2001). Structural equation modeling with AMOS. Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum. Gardner, H.(1993). Frames of mind: a theory of multiple intelligences. New York: Basic Books. Gardner, H. (2006). The development and education of the mind:the selected works of Howard Gardner (pp.50). New York: Routledge. Joreskog, K.G. (1993). Testing structural equation models.in K.A. Bollen & J.S. Long (Eds.), Testing structural equation models (pp ). Newbury Park. CA: Sage. Klein, P.D. (1997). Multiplying the problems of intelligence by eight: A critique of Gardner s theory. Canadian Journal of Education,22: Kline, R.B. (2005) Principles and practice of structural equation modeling(2 nd ed.). New York: Guilford Press. Linacre, J.M. (2007). A user s guide to WINSTEPS Rasch-model computer programs. Chicago: MESA Press. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3 rd ed., pp ). New York: American Council on Education/Macmillan. Pallant, J. (2001). SPSS survival manual: a step by step guide to data analysis using SPSS for windows. (Ver.10). Illinois: Allen & Unwin. Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: The University of Chicago Press. Sarwono, S.W. (1979). Inteligensi, apakah itu? Jakarta: Fakultas Psikologi, Universitas Indonesia. Shearer C.B. & Luzzo, D.A. (2009) Exploring the application of multiple intelligences theory to career counseling. The Career Development Quarterly. Vol. 58: Siti Rahayah, A., Rosseni, D. & Hafsa, M.M. (2008a) Contributing factors to the multiple intelligences of youths. Jurnal Pendidikan, Universiti Kebangsaan Malaysia, Siti Rahayah, A., Noriah, I. & Saemah, R. (2006) Profiling of multiple intelligences test among youth in Malaysia. Proceeding for Teaching and Learning Conference, Australia. Siti Rahayah, A., Rodiah, I. & Noriah, I. (2010). Differential item functioning in Malaysian Generic Skills instrument (MyGSI). Jurnal Pendidikan Malaysia 35(1): 1-10 Siti Rahayah, A., Noriah, I, Riza Atiq, O.K.R, Abdul Ghafur, A., Rodiah, I., Nur Ashiqin, N. (2008b). Assessing generic skills using Rasch model approach: A method for construct validity and reliability. International Conference on Education. Siti Rahayah, A., Noriah, I., Rosseni, D., Abdul Ghafur, A. & Rodiah, I. (2008c). Evaluation approaches and challenges using structural equation model (SEM). Proceedings of the International Conference on the Education of Learner Diversity Siti Rahayah, A., Noriah, I., Abdul Ghafur, A., Rodiah, I. & Nur Ashiqin, N. (2008d). Communication, leadership, and teamwork skills as core competencies among higher education students. Proceedings of the ASAIHL International Conference. Bangkok,

6 Siti Rahayah Ariffi n et al. / Procedia Social and Behavioral Sciences 9 (2010) Smith, R. M. (1992). Applications of Rasch measurement. Chicago, IL: MESA Press. Sternberg, R.J. (1994). Encyclopedia of human intelligence. New York: Macmillan. Tran, H.P., Griffin, P. & Nguyen, C. (2010). Validating the university entrance English test to the Vietnam National University: A conceptual framework and methodology. Procedia Social and Behavioral Sciences Wright, B.D. & Masters, G.N. (1982). Rating scale analysis. Rasch measurement. Chicago: Mesa Press. Wright, B.D. & Stone, M.H. (1979). Best test design. Chicago: MESA Press. Wu, S. & Alrabah, S. (2009). A cross-cultural study of Taiwanese and Kuwaiti EFL students' learning styles and multiple intelligences. Innovations in Education and Teaching International, 46(4): Zoanetti, N. (2010). Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology 26(5),