Session 401. Creating Assessments that Effectively Measure Knowledge, Skill, and Ability

Transcription

1 Practical and Effective Assessments in e-learning Session 401 Creating Assessments that Effectively Measure Knowledge, Skill, and Ability Howard Eisenberg, Questionmark

2 Effectively Measuring Knowledge, Skill and Ability with Well-crafted Assessments Beyond Recall About the Presenter Page 1

3 Agenda A little theory Lots of examples Some practical techniques Citation This presentation uses ideas and examples from the work of Thomas Haladyna. Haladyna, Thomas (2004). Developing and Validating Multiple-Choice Test Items. Third Edition. Hillsdale, NJ: Lawrence Earlbaum Associates. It was inspired by my experience in the field, where customers and colleagues often want to know how to write better tests Page 2

4 How do we avoid this? How do we go beyond recall? How do we develop tests of different cognitive demand? Test more than recall? Tests should be part: Content Easy to test for recall Cognitive process Difficult to illicit appropriate cognitive process in the test Context performance environment Page 3

5 What is cognition? Cognition Knowledge Skill Ability Do our tests illicit the right level of cognitive demand? Bloom s Cognitive Level Taxonomy and Item Types Matching True/False Multiple Choice Fill-in-the-blank Short Answer Essay Page 4

6 Knowledge Cognitive Content Process Recall Fact Concept Principle Procedure Understanding Fact Concept Principle Procedure Recall = reproduce or recognize content as presented in instruction Understanding = paraphrasing of content or provision of examples or non-examples not encountered in instruction Models for Classifying Objectives According to Cognitive Level 1. Bloom s Taxonomy 2. Gagne s Learned Capabilities 3. Merrill s Component Design Theory 4. Framework for Instructional Objectives (Hannah and Michaels) 5. Tiemann-Markle Matrix Common Traits Hierarchical Make distinctions between Knowing - Knowledge Using -Skill Creating - Ability Objective-Item Congruence Items are written to match the cognitive level of the objective Page 5

7 Knowledge Example Testing for recall of a fact or concept Testing for understanding of a concept, principle or procedure Skill Performance of a cognitive task that requires certain knowledge Implies use of performance items or constructed response items For certain domains, can be effectively measured with selected-response items Page 6

8 Skill Example Ability Performance of a complex task that requires the individual to use knowledge and skills in a unique way to accomplish a complex outcome Pediatrician medical ability CPA accounting ability Architect building design ability Page 7

9 Ability Examples Reading Compare and contrast historical and cultural perspectives of literary selections Writing Write a report that conveys a point of view and develops a topic Mathematics Draw inferences from charts and tables that summarize data from real-world situations Ability Example Page 8

10 Fundamental Abilities + Professional Domain (Knowledge and Skill) = Professional Abilities Creative Thinking Creative Thinking Creative Thinking Critical Thinking Critical Thinking Critical Thinking Problem-Solving Problem-Solving Problem-Solving Which type of question? Format Knowledge Skill Ability Conventional MC X X Alternate choice X X Matching X X Extended X X X Matching T-F X X Complex MC X X Multiple T-F X X Pictorial Item Set X X X Problem-solving X Item Set Scenario Item Set X Interlinear X Page 9

11 Knowledge Matrix Alternate Choice Multiple True-False Multiple True-False Page 10

12 Matching Extended Matching Pull-down List Extended Matching Page 11

13 Pictorial Item Set Problem-Solving Item Set Page 12

14 Problem-Solving Item Set Page 13

15 Scenario Item Set Page 14

16 Item Shell Hollow item containing syntactic structure useful for writing sets of similar items Derived from existing items that are known to perform well Good for facts, concepts, principles, and procedures Which is an example of (any concept)? A. Example B. Plausible nonexample D. Plausible nonexample C. Plausible nonexample Item Shell Which is the definition of? Recall Which is synonymous with? Comprehension What would happen if? Application Page 15

17 Item Modeling Technique for developing scenario or vignette Models tend to be domain-specific Clinical (medical) professions Mathematics Tend to work best in well-defined problem domains Item Model: Medical A scenario is presented about a patient problem. How should the patient be treated? The item shell provides information about a patient disease or injury. The completed item will require the test taker to Make a correct diagnosis Identify the recommended treatment protocol Page 16

18 Item Model Process 1. Identify the stem of a successfully performing item. 2. Underline the key words or phrases representing the content of the item A 6-year-old child is brought to the hospital with contusions over the abdomen and chest as a result of an automobile accident. What should the initial treatment be? Item Model Process 3. Identify variations for each key word or phrase Age of patient: infant, child, adolescent, young adult, middle age, elderly Trauma and complications: Cuts, contusions, fractures, internal injuries Type of accident: Automobile, home, industrial, recreational 4. Select a combination of factors (age, trauma, accident) 5. Write the stem 6. Write the correct answer 7. Write the distractors Page 17

19 Key Features Technique Focus is on problem solving in professional domain Key feature either a difficult step in the thought process or the step where an error is most likely to occur Object is to identify those features that are most likely to discriminate among candidates with varying degrees of ability Used a lot in medical / clinical professions Key Features Process 1. Define the domain of problems a. For pediatricians: i. Near drowning, dehydration, adolescent diabetes 2. Provide examination blueprint a. Which problems fall within the specs of the blueprint? i. Medical specialty, body systems, clinical setting 3. Present clinical situations a. Patient complaints, typical or atypical problem, multiple-system problem, life-threatening, preventative care Page 18

20 Key Features 4. Select key features for problem a. A critical step that will produce a variety of actions/choices 5. Select case and write scenario a. Refer to clinical situations identified in step 3 6. Develop scoring for the results 7. Conduct pilot testing 8. Set standards Key Features Problem Example For an adult complaining of a painful, swollen leg, the physician should: Include deep venous thrombosis in the differential diagnosis Elicit risk factors for deep venous thrombosis through the patient s history Order a venogram as a definitive test for deep venous thrombosis Page 19

21 Key Features Example Questions Constructed Response or Selected Response? A complex performance item is more desirable to measure an ability but there are good reasons for developing a selected response item that mimics the complexity of the performance CR item can take longer time to administer CR item has to be scored by judges which is expensive Empirical evidence exists that supports the use of SR items in terms of validity and reliability Scoring CR items is subject to rater effects such as severity and inconsistency which diminish reliability Sacrifice a little fidelity for greater efficiency Page 20

22 Turning Constructed Response into Selected Response Question: Review the following screen shot. Please describe 2 strengths of the design and 2 opportunities for improvement as it relates to usability Answer: Generally looking for well constructed and thought out examples based on UE principles and concepts for both strengths and weaknesses. Focus should NOT be on functionality or technical performance. Turning Constructed Response into Selection Response Provide an example of a well-designed user interface Provide an example of a poorly-designed user interface Ask the participant to identify the principles met and the principles or the principles violated Callout areas of an interface Ask the participant to judge whether it s a strength or a weakness Page 21

23 Questions and Comments Howard Eisenberg Manager, Training and Consulting Questionmark Page 22