The Content Validity and Items Analysis of Higher-Order Thinking Test in Natural Science Studies of Elementary School

Transcription

1 4 th ICRIEMS Proceedings Published by The Faculty Of Mathematics And Natural Sciences Yogyakarta State University, ISBN The Content Validity and Items Analysis of Higher-Order Thinking Test in Natural Science Studies of Elementary School Hafizhah Lukitasari a), Sri Yamtinah, and Peduk Rintayati Sebelas Maret University, Jl. Ir. Sutami No. 36A, Jebres, Surakarta, Central Java, Indonesia a) Corresponding author: hafizhahlukitasari@gmail.com Abstract. The main aim of this research is to measure the content validity and item analysis of higher-order thingking test in science studies of elementary school. This research is developing modified multiple-choice test used steps as follow: 1) determine the test specification; 2) writing the stems; 3) polarizes short-answer tests; 4) measure the content validity; 5) measure the item analysis; 6) improve and implement the test. The researcher use qualitative and quantitative technique to analyze the data obtained. The qualitative approach was used to analyze the results of expert s validation using Aiken s validity. The quantitative approach was used to analyze the results of item analysis using Iteman. Conclusion of this study as fol1ows: 1) The quality of the modified multiple-choice test has a valid criterion as an instrument, in terms of aspects of the construct, substance, and language. All these aspects meet a very good criterion and can be used with revisions. 2) The item analysis presents the reliability, the difficulty index, and the discriminating power of modified multiple choice test is good. It can be applied in elementary school assessment. INTRODUCTION Validity refers to the adequacy and appropriateness of the interpretations made from assessments, with regard to a particular use. Validity is required with proper interpretation (appropriateness) and adequacy in assessments [1]. Therefore, validity always relates to the particular use of the assessment results and interpretation validity which researchers pointed out. The meaning of the content validity is the extent to which elements of an instrument are really relevant and represents the contract in accordance with the purpose of measurement [2]. Content validity is the validity that checks the matches between test items which are made with indicator, learning material or learning goals predefined [3]. In most studies, researchers studied complex constructs for which valid and reliable instruments are needed. Reliability relates to the accuracy of the instrumen in measuring what measured [4]. Reliability as observation consistency obtained from recording both on an object and some objects [5]. Reliability is a coefficient that shows the consistency of measurement results of an instrument [6]. An instrument can be trusted if the test results unchanged or relatively the same if tested repeatedly. Such measuring instruments are called reliable. The reliability used in this analysis is internal consistency reliability. Internal consistency reliability is based on scores obtained from an instrument with one measurement on the testee [7]. The determination of the test item being answered correctly by the testee describes the reliability of the entire test. The instrument are otherwise reliable if the score of each items have relevance each mutually [8]. A good reliability coefficient is above Item analysis consists of reliability, difficulty index, discriminating power, and alternative (of stem) analysis. Item test difficulty level of learning outcomes can be known from rate which symbolizes the difficulty named difficulty index [9]. While the discriminating power is the index which shows the ability of the item distinguish between clever student and less-clever student [10]. Then, distractor otherwise good if it can lead the testee to choose at least 5% of the total testee [11]. Measuring content validity and item analysis are important aspects of a quantitative research inquiry. Content validity refers to the degree that the instrument covers the content that is supposed to measure. Item analysis is also valuable in improving items which will be used again in later tests. It can also be used to eliminate ambiguous or misleading items in a single test administration. Item analysis is valuable for increasing teachers skills in test construction, and identifying specific areas of course content which need greater emphasis SE-77

2 or clarity. The instrument is a vital analysis to consider as a good instrument. Most teaching evaluation questionnaires have not presented sufficient evidence of validity. If an instrument provides a measure of what it actually measures, validity is established. Content validity and item analysis used to examine higher-order thinking test in natural science studies. Higher-order thinking test developed as an instrument to measure students skills based on Bloom Taxonomy of Cognitive level. Higher-order thinking involves a cluster of elaborative mental activities requiring nuanced judgement and analysis of complex situations according to multiple criteria [12]. Higher order thinking is effortful and depends on self-regulation. Higher order thinking occurs when person takes new informations stored in memory and interrelates and/or rearranges and extends this information to achive a purpose or find possible answer in perplexing situation [13]. Higher order thinking includes critical, logical, reflective, metacognitive, and creative thinking. These skills are activated when students of any encounter unfamiliar problems, uncertainties, questions, or dilemmas. Successful applications of these skills result in explanatinos, decisions, performances, and products that are valid within the context of available knowledge experience, and promote continued growth in higher order thinking, as well as the intellectual skills [14]. But in fact, higherorder thinking skills learning and assessment still not developed optimally by the teachers, then it is important to began developing the assessment that can measure higher-order thinking learning outcome with various methods and examine the validity, reliability, and item analysis. Modified multiple-choice test is selected-response item that developed by researcher for measuring higherorder thinking skills. Multiple-choice test can effectively measure various types of knowledge and complex learning outcomes. It is easier to construct high-quality test items in multiple-choice form than in any of the other forms [15]. The multiple-choice format is widely used in achievement tests of all types, primarily to assess learning outcomes at the factual knowledge and comprehension levels. However, this format can also be used to assess higher-level thinking involving application, analysis, and synthesis (item 3 from the following examples is a multiple-choices item that assesses higher-level thinking) [16]. There are some previous study that discuss about the validity and reliability. Yaghmale develop a content valid scale for assessing experience with computer usage [17] and Ellen A. Drost explained about Validity and Reliability in Social Science Research [18]. Craig, Biderman, and Nguyen researched the relationship of reliability and validity of personality test [19], and Zohrabi studied about Mixed Method Research consists of instruments, validity, reliability, and reporting findings [20]. Olfos and Zulantay also studied about reliability and validity of authentic assessment in a web based course [21], as well as Kinyua and Okunya who has been researching about validity and reliability of teacher-made tests [22], and Mohamad, et.al study the same issues in science studies [23]. Based on the relevant research, higher-order thinking test still not fully analyzed in terms on validity and reliability. So that, the main aim of this research is to measure the content validity and item analysis of higherorder thinking test in science studies of elementary school including reliability, difficulty index, determinating power, and alternative statistics. METHODOLOGY This methodological study is part of a larger study carried out through the research and development design the instrument measuring higher-order thinking skills in science studies of 5 th grade students in Elementary Schools. This study is quantitative research involves two steps as below: Content validity by experts jugdement Researcher use content validity developed by Aiken. Aiken has formulated how to calculate the content validity coefficient using Aiken s Formula based expert judgment as many n experts about an item build upon the extent to which the aitem represents the measured construct. The formula is as follows: V =!! (!!!) ; with s = r lo Testing for item analysis Difficulty Index Modified multiple-choice test tested to 100 students who selected randomly from 5 Elementary School in Purbalingga Regency, Central Java, Indonesia. Afterwards, the test results analyze using Iteman 3.0 for to assess the reliability, difficulty index, discriminating power, and alternative statistics. SE-78

3 This analysis use internal consistency reliability use Alpha Formula in Iteman 3.0 application as follows:!!!!! = 3/2 1 (!!!! +!!!! +!!!!! )/!! Difficulty index value between 0,00 1,00. The lowest index is 0,00, its mean that the item is too difficult because no one testee could answer the item. The highest index is 1,00, its mean that the item is too easy because all of the testee could answer the item correctly. The formula used to measure the difficulty index as follows! =!!" The difficulty index can be obtained from proportional correct interpretation. If the test item is very easy it has a difficulty indexed of 0.91 and above, if the test item is easy it has a difficulty indexed of 0.76 to 0.90, if the test item is optimum difficulty it has a difficulty indexed of 0.26 to 0.75, if the test item is difficult it has a difficulty indexed of 0.11 to 0.25, if the test item is very difficult it has a difficulty indexed of 0.10 and below [20]. Determinating Power Discrimination index value between -1,00 to 1,00. In Iteman 3.0, the discrimination index calculated using point-biserial correlation formula (r pbi ) and biserial correlation formula (r pbis ) as follows:!!"# =!!!!!!! (1!) or!!"# =!!!(1!)!!! The item determinating power can be obtained from point-biserial correlation interpretation. If the point biserial value is 0.40 and up so the item is very good items, if the point biserial value is 0.30 to 0.39 so the item is reasonably good, but possibly, if the point biserial value is 0.20 to 0.29 so the item is marginal items, usually needing and being s, if the point biserial value is below 0.19 so the item is poor items, to be rejected or improved by revision. Furthermore, the alternative answers consists of key and distractor. The distractor is effective when it selected by at least 5% for all testee [22]. Distractor index can be formulated as follows:!" =! (!!)(! 1) 100% RESULTS AND DISCUSSION The Development of Higher-Order Thinking Test Measuring higher-order thinking in Elementary Schools has not been fully implemented by teachers because they thought higher-order thinking measurement is harder than normally classroom assessment. They do not have enough ways to measure the skills. Based on existing data sourced from teacher s item bank, we get an analysis of the lack of measurement in higher-order thinking such as analytical thinking, critical thinking, and creative thinking. This is a concern for researchers to develop an alternative assessment that can be measure the higher-order thinking skills. After considering reference analysis and prior research, researcher developed modified multiple-choice test for measuring higher-order thinking with the following stages: a) determine the purpose of measurement; d) determining the type of the test; c) identification of curriculum and learning materials; d) test specification; e) considering the distribution of item level-cognitive (analysis, evaluating, and creating); f) make essay test; g) polarization the essay test answers; h) make modified multiple-choice test; i) expert jugdement; and j) item analysis use Iteman. The modified multiple-choice specification form is two-tier alternative. Modified multiple-choice with twotier choices is the test that can describe about student competency in answering and the reason about their choice. This test can avoid inconsequential choice, which is a weakness about the usual multiple-choice. The developed test provides 4 choices answer and 4 choices reason. SE-79

4 Content Validity The instrument of higher-order thinking test was validated by 8 experts, they are the lecturer of Sebelas Maret University Postgraduate Program. By considering the coefficient tabel of Aiken Validity, test items was decided valid if its validity coefficient 0,75. The results of experts judgement to the modified multiple-choice test which developed in first validation steps as follows: Items TABLE 1. Expert Judgment s Result (1 st Validation) Validation Score CV Decision ,67 invalid ,79 valid ,71 invalid ,92 valid ,88 valid ,71 invalid ,75 valid ,92 valid ,67 invalid ,88 valid ,83 valid ,88 valid ,79 valid ,75 valid ,88 valid ,88 valid ,75 valid ,67 invalid ,63 invalid ,67 invalid ,83 valid ,88 valid ,79 valid ,92 valid ,67 invalid According the Table 1, there are 8 items which must be improved and revised. It has to revise again until the experts decision is valid (achieve validation coefficient 0.75). During the first validation process, there are qualitative analysis from some experts which is used as a guidance in revising the invalid items. According to the validator s note, the item number 1 s indicator has to changed to one dimension and the stem improvements on conditions, correct option of answers and reasons, and also improve the sentences or language. Note for item number 2 is the illustration is not working yet. Note for item number 5, it has to improve the concept of science material. Note for number 6 is simplify the stem and make the option become homogeny. Note for number 12, 23, and 25, it has to improve the option or alternative answers. Note for number 17, 18, and 19, it has to improve the stem, reason, and picture or illustration. After revision based on first validation and qualitative analysis by experts, researchers revised the test and re-validate to the experts until the test is valid as follows: SE-80

5 Items TABLE 2. Revision Results in Second Validation Before Revision After Revision CV Decision CV Decision 1 0,67 Invalid 0,75 Valid 3 0,71 Invalid 0,79 Valid 6 0,71 Invalid 0,75 Valid 9 0,67 Invalid 0,79 Valid 18 0,67 Invalid 0,79 Valid 19 0,63 Invalid 0,75 Valid 20 0,67 Invalid 0,79 Valid 25 0,67 Invalid 0,75 Valid After experts validation done twice, modified multiple-choice test is valid because all of the item have coefficient validity more than It is mean that modified multiple-choice test which developed has been accordance with the item indicator, material, and learning outcomes predefined, mesure higher-order thinking skills. The validation based on Aikens formula of content validity not just once or twice, it may be done several times until all items are valid in content. It is because the meaning of the content validity is the extent to which elements of an instrument are really relevant and represents the contract in accordance with the purpose of measurement [2]. All items have been prepared in accordance with each indicator to be measured, so all items must be valid for all indicators to be measurable. Reliability The reliability used in this analysis is internal consistency reliability. Internal consistency reliability is based on scores obtained from an instrument with one measurement on the testee. The instrument realiability has been analyzed use Iteman 3.0. There are 25 item that analyzed by Iteman 3.0 and 100 students which has done the test in this analysis. The average of students score is with is the maximum score and 1.00 is the minimum score. The test is reliable when every items has related with another items. A good reliability coefficient is above 0.75 [8]. Therefore, the modified multiple-choice test has a high reliability because the reliability from alpha rulon s formula is An instrument can be trusted because the test results unchanged or relatively the same if tested repeatedly [7]. Difficulty Index and Discriminating Power The analysis per item can be observed in Table 3. Difficulty index from the test can be analyzed based on Mean P value in Iteman 3.0 which is on average. It is mean that the modified multiple-choice test is optimum difficulty. While, the discriminating power of two-tier multiple choice test can be analyzed based on Mean Item-Tot in Iteman 3.0. The difficulty index is It is mean that the modified multiple-choice test has a very good items on average. Overall, it can be concluded that modified multiple-choice test which developed to measure the higher-order thinking skills has a high quality based on difficulty index and discriminating power. It is mean that, the test has optimum difficulty which is suitable for measuring students ability in analyze, evaluate, and create. The test is not to easy and also no too difficul to be implemented. The test has the potentiality to differentiate students with good higher-order thinking skills and students with poor higher-order thinking skills. SE-81

6 Item TABLE 3. Difficulty Index and Discriminating Power Difficulty Index Discriminating Power (Prop. Correct) (Point Biser.) Coefficient Decision Coefficient Decision Conclusion Difficult Very good items Optimum Difficulty Reasonably good, but possibly Difficult Very good items Optimum Difficulty Reasonably good, but possibly Optimum Difficulty Very good items Good Difficult Very good items Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Optimum Difficulty Reasonably good, but possibly Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Optimum Difficulty Reasonably good, but possibly Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Difficult Very good items Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Optimum Difficulty Very good items Good Difficult Reasonably good, but possibly Optimum Difficulty Reasonably good, but possibly Optimum Difficulty Marginal items, usually needing and being subject to improvements Difficult Very good items Difficult Very good items Alternative Statistics Iteman 3.0 also analyzed the alternative statistics. Some item need improvement as a follow up. The items number 2, 4, 5, 7, 8, 9, 10, 14, 18, 19, 20, and 22 has a good alternative without revision which all of the answers key was selected by most testee and all of the distractor works. After revising the items based on Table 4, then it can be concluded that modified multiple-choice test for measuring higher-order thinking skills feasible applied with some improvements. SE-82

7 TABLE 4. Alternative Statistics Analysis Item Alternative (Prop. Endorsing) Key Most selected distractor Distractor didn t work 1 A (0.190) B (0.160) - 3 C (0.210) B (0.340) - 6 B (0.210) C (0.630) - 11 D (0.390) A (0.440) - 12 B B A 13 C (0.430) A (0.470) D 15 A A C 16 D D B 17 C (0.170) D (0.550) - 21 B (0.200) C (0.490) D 23 D (0.320) C (0.500) - 24 B (0.230) A (0.440) - 25 B (0.190) A (0.480) - CONCLUSION Conclusion of this study as follows: 1. The quality of the modified multiple-choice test has a valid criterion as an instrument, in terms of aspects of the construct, substance, and language. Modified multiple-choice test is valid because all of the item have coefficient validity more than 0,75. It is mean that modified multiple-choice test which developed has been accordance with the item indicator, material, and learning outcomes predefined, mesure higher-order thinking skills. 2. The item analysis presents the reliability, the difficulty index, and the discriminating power of modified multiple-choice test is good. The reliability is 0.85 (high reliability), the average of difficulty index is 0,475 (optimum difficulty), the discrimination index is 0,471 (very good), and the alternative includes key and the distractor worked by improvement. It is mean that the test can be applied in elementary school assessment. This is a part of a research and development that developed the higher order thinking skills test. If teachers can implement test development steps and examine it use validity formula and Iteman 3.0 analysis, then difficulties in measuring student ability can be reduced. Easy steps for developing the test can be adapted with the following stages: a) determine the purpose of measurement; d) determining the type of the test; c) identification of curriculum and learning materials; d) test specification; e) considering the distribution of item level-cognitive (example: analysis, evaluating, and creating); f) make essay test; g) polarization the essay test answers; h) make modified multiple-choice test; i) expert jugdement; and j) item analysis use Iteman 3.0 or Anbuso. The researcher hope higher-order thingking assessment should be developed not only in science studies, but also in all subjects especially in 4 th to 6 th grade for preparing them to the next-secondary level education. REFERENCES 1. E. Grondlund and R. Linn, Measurement and Evaluation in Teaching (Macmillan Publishing Company, Macmillan, 1990), p S. Azwar, Reliability and Validity (Pustaka Pelajar, Yogyakarta, 2015), p B. Susetyo, Procedure of Preparation and Test Analysis for Assessment of Learning Outcomes of Cognitive Fields, (Refika Aditama, Bandung, 2015), p L.R. Thorndike and E.P. Hagen, Measurement and Evaluation in Psychology and Education (John Wiley Publication, Etobicoke, 1997) 5. C. Hopkins and R.L. Antes, Classroom Testing: Construction, (F.E. Peacock Publishers, Adelaide, 1989) SE-83

8 6. D. Mardapi, Measurement, Assessment and Evaluation in Education (Nuha Media, Yogyakarta, 2012), p B. Susetyo, Procedure of Preparation and Test Analysis for Assessment of Learning Outcomes of Cognitive Fields, (Refika Aditama, Bandung, 2015), p B. Susetyo, Procedure of Preparation and Test Analysis for Assessment of Learning Outcomes of Cognitive Fields, (Refika Aditama, Bandung, 2015), p H.C. Witherington, Educational Psycology (Aksara Baru, Jakarta, 1978), p E.P. Widoyoko, Learning Program Evaluation: A Handbook of for Educator and Prospective Educator, (Pustaka Pelajar, Yogyakarta, 2014), p B. Susetyo, Procedure of Preparation and Test Analysis for Assessment of Learning Outcomes of Cognitive Fields, (Refika Aditama, Bandung, 2015), p A. Lewis and D. Smith, Theory into Practice, 32 (3) Summer, , (1993). 13. F.J. King, L.Goodson and M.S.F. Rohani, Higher-Order Thinking Skills: Definition, Teaching Strategies, Assessment (Center for Advancement of Learning Assessment, Florida State University, 2010), pp E. Grondlund and R. Linn, Measurement and Evaluation in Teaching (Macmillan Publishing Company, 1990), p P. Airasian and H. Miranda, Theory Into Practice, 41 (4) Autumn, , (2002). 16. S. Azwar, Reliability and Validity (Pustaka Pelajar, Yogyakarta, 2015), p Yaghmale F., Journal of Medical Education, 3 (1) Spring, 25-27, (2003). 18. E.A. Drost, Education Research and Perspective, 38 (1), , (2012). 19. C.M. Reddock, M.D. Biderman and N.T. Nguyen, International Journal of Selection and Assessment, 19 (2) June, , (2011). 20. M. Zohrabi, Theory and Practive in Language Studies, 3 (2), , (2013). 21. R. Olfos and H. Zulantay, Educational Technology and Society, 10 (3), , (2007). 22. Kinyua and Okunya, African Educational Research Journal, 2 (2), 61-71, (2014). 23. M.M. Mohamad, et.al, Procedia Social and Behavioral Sciences, 204, , (2015). SE-84