Goal for Today. Making a Better Test: Using New Multiple Choice Question Varieties with Confidence

Transcription

1 Making a Better Test: Using New Multiple Choice Question Varieties with Confidence Catherine Davis, CGFNS Int l David Foster, Kryterion, Inc. Goal for Today Create an appreciation for, and an optimism toward, correct and innovative uses of the multiple choice item format Single Correct Multiple Choice (98 Years Old and going strong) Which of the following drug-dose combinations is recommended as the initial medication to give a patient in asystole? A. Epinephrine 3 mg IV B. Atropine 3 mg IV C. Epinephrine 1 mg IV D. Atropine 0.5 mg IV 1

2 New Features (probably a partial list) Add graphics, video, audio, more text, etc. Add more correct options Prompt or don t prompt Increase number of options Present options differently Use in testing complex performance Cognitive Behavioral Use new scoring methods Partial Alternative Require speed in responding Facts about Multiple Choice Represents nearly 100% of items used in today s tests Incorrect Criticisms Multiple Choice can only be used to measure memorization skills Multiple Choice is simple Valid Criticism Measures significant constructirrelevant test taking skills Multiple Correct MC: Example Which are 3 mammals? a. Horse b. Bear c. Cat d. Lizard Which is NOT a mammal? a. Horse b. Bear c. Cat d. Lizard Paper Test Version: Confusing and doesn t measure the skill Technology Version: Direct and easy-tochoose correct answers Which are 3 mammals? a. Horse b. Bear c. Cat d. Lizard Which group has 3 mammals? I. Horse II. Bear III. Cat IV. Lizard a. I, II and IV b. I, III and IV c. II, III and IV d. I, II and III Paper Test Version: Very confusing and results in errors 2

3 Prompt or No Prompt Prompt: Choose X, + software requires X responses Not Prompted: Choose all that apply Single Correct option is, by definition and tradition, prompted Research: International nursing certification CGFNS Qualifying Exam 150 core and 15 pretest items Administered 4 times a year Purpose Visa screening Prerequisite for licensure Internet-based since 2011 CGFNS Qualifying Exam Predominantly multiple choice items Single correct answer Alternate item types Charts and graphs Multiple correct answer Four options No partial credit 3

4 Early Multiple Response Item A 6 month old is at the office for a well-child exam and immunizations. The child has a mild upper respiratory infection and temperature of (F). The nurse would A. B. C. D. give the immunization, since having a temperature less than is not a contraindication for immunization. hold the immunization, since a temperature greater than is a contraindication for immunization. hold the immunization, since having any illness/infection is a contraindication for immunization. give the immunization since having a mild acute illness is not a contraindication for immunization. Non-prompted Item A patient has a history of frequent asthma exacerbations. Which instructions should the nurse give to the patient about how to decrease the risk of asthma attacks? (Choose ALL that apply) A. Remove carpeting from bedroom. B. Use bronchodilator inhaler 30 minutes after large meals. C. Dress warmly in cold temperatures. D. Use inhaler 15 minutes before exercise. Prompted Item A patient has a history of frequent asthma exacerbations. Which instructions should the nurse give to the patient about how to decrease the risk of asthma attacks? (Choose TWO that apply) A. Remove carpeting from bedroom. B. Use bronchodilator inhaler 30 minutes after large meals. C. Dress warmly in cold temperatures. D. Use inhaler 15 minutes before exercise. 4

5 Proportion Answered Correctly Research Design 12 items, each prompted and nonprompted Split in balanced way across test forms, allowing for direct comparison Approximately 1200 examinees Major Finding: Difficulty Difference Difference in Difficulty Item Pairs Non-Prompted Prompted Construct-Irrelevant Variance Does the prompting add CIV easiness or does non-prompting add CIV difficulty? Not an easy question to answer A couple of other analyses Number of choices selected Timing 5

6 Mean # of Seconds Mean # of Options Selected Number of Responses Selected 2.60 Mean Number of Options for Incorrect Non-Prompted Items by Quintile Required Selected 2.00 Q1 Q2 Q3 Q4 Q5 Score Quintiles (Q5 Has Highest Scores) Timing: Prompt versus Non-Prompt P All NP All Q1 Q2 Q3 Q4 Q5 Score Quintiles (Q5 has the highest scores) Discrete Option Multiple Choice: Example Traditional Multiple Choice DOMC Demonstration: 6

7 Discrete Option Multiple Choice (DOMC) Options delivered randomly one-at-a-time Options continue until question is answered correctly or incorrectly No limit to pool of correct or incorrect options Potential Benefits of DOMC Better security Fairness Better measurement Motivates deeper learning Reduced development costs Research Design Form Items 1-10 Items Items Items A1 DOMC set 1 MC item set 2 DOMC set 3 MC item set 4 A2 DOMC set 1 MC item set 2 MC item set 4 DOMC set 3 B1 DOMC set 1 MC item set 2 DOMC set 4 MC item set 3 B2 DOMC set 1 MC item set 2 MC item set 3 DOMC set 4 Testlet Statistics Set 3 - Raw Frequencies (Groups are not random) Count Cum. Percent Score DOMC MC DOMC MC % 0.0% % 0.7% % 1.5% % 3.7% % 9.9% % 16.9% % 26.1% % 50.0% % 76.8% % 93.0% n Mean SD

8 MC Testlet Statistics Generalizability: Internal Consistency Cronbach Alpha Item Item Set Type DOMC MC Classical Item Statistics - Difficulty 1.00 Item P-Values Mean DOMC =.61 Mean MC = DOMC 8

9 MC Classical Item Statistics - Biserial Biserial Correlation DOMC Mean DOMC =.45 Mean MC =.44 Using MC for Performance Testing Any high-level cognitive skill can be measured with multiple choice Item provides necessary components and requires the cognitive effort MC is only used for convenience in collecting a response Example: Service and Support Exam Skill: analyze data, determine what is needed, and find information in a technical library Research Design Certification candidates rated themselves on their Service and Support competence They were also rated on the same competencies by their work supervisors Each then took two versions of the Service and Support test: 701: Previous knowledge-based exam 801: Measured ability to analyze data and use reference tools Both tests used multiple choice response framework 9

10 701/801 Comparison Research Results 701 score with Supervisor Rating: r = score with Supervisor Rating: r = score with Self Rating: r = score with Self Rating: r =.59 Research Conclusions Multiple choice items can be simplistic or complex It is possible to measure higherlevel cognitive and behavioral skills using multiple choice format Expectations for the Future of Multiple Choice We are just getting started It took almost 100 years + technology to find new complex and effective varieties There is still much we can learn 10

11 Thank You! 11