Postcards from the Edge Experiences from Early Adopters of Automatic Item Generation. Matt Burke. American Board of Internal Medicine:

Transcription

1 Postcards from the Edge Experiences from Early Adopters of Automatic Item Generation Greg Sadesky With Yardstick since 2009 as Psychometrician, EVP Psychometrics, CSO. Inspired by innovation, educating clients, and working with colleagues both within and outside of YAS. Matt Burke American Board of Internal Medicine: Director of Advances in Content Development Janel Swain Janel is an Advanced Care Paramedic with Emergency Health Services in Nova Scotia. Chair of the COPR Exam Working Group since Ground Services Clinical Supervisor for EHS Nova Scotia. and Regulation 2018 Annual Educational Conference 1

2 Goals of Session OVERALL: To showcase how AIG is being used in credentialing programs. 1.Generating and Selecting Test Questions Background on AIG Range and Selection 2.American Board of Internal Medicine (ABIM): Reverse Engineering of Templates 3.Canadian Organization of Paramedic Regulators (COPR): model-based item creation Principles 'Basic' examples Greg: Intro to AIG and Regulation 2018 Annual Educational Conference 2

3 What is AIG? Automatic Item Generation (AIG) is a methodology of item creation that uses variables and combinations to produce both more and a greater range of test content as compared with traditional one-at-a-time methods. The variable and combinations can represent task and item models which represent knowledge of the domain and enable the creation of a large (complete?) set of valid content. Combinations Combinations are the source of variation for questions produced by AIG. Two sources of variation: Variation in STEMS Variation in OPTIONS and Regulation 2018 Annual Educational Conference 3

4 Variation in stems: example Which of the following advantages of AIG is most significant for <STAKEHOLDER>? 1. Psychometricians 2. Credentialing bodies 3. Authors 4. Educators 5. Accountants With a list of length 5, there are 5 unique stems. Which of the following 1.Option A 2.Option B 3.Option C 4.Option D 5.Option E 6.Option F Variation in option sets How many unique combinations of 4 alternatives can be made from: 4 options? A: 1 5 options? A: 5 6 options? A: 15 and Regulation 2018 Annual Educational Conference 4

5 Combinations - continued If you had a template with: 5 Unique Stems, and 6 Options, and you were creating 4-option MCQs... How many unique questions could be generated? 5 Stems x 15 Unique option sets = 75 Questions! Right??? Almost. Because the 75 questions you created don t all contain the KEY in the option sets. And, maybe you have some combinations of alternatives that you don t want together. So, for good questions to be created, you have to identify which combinations are VALID. and Regulation 2018 Annual Educational Conference 5

6 Actually, it s very EASY to create many questions using AIG. Beyond a certain point, more is NOT always better To effectively use Itematic / AIG, you should know how many questions you need (per competency, L.O.) and create GOOD ones. Many questions at once: 36, for the previous template and Regulation 2018 Annual Educational Conference 6

7 Not all questions are helpful. This is what I wanted!! and Regulation 2018 Annual Educational Conference 7

8 Real life Example TDG Context: Assessing principles in the safe transportation of dangerous goods Basic knowledge: UN numbers corresponding to hazardous substances TDG example debrief Simple template structure Many possible items: 140!! Range consists of number of substances that have which UN numbers and Regulation 2018 Annual Educational Conference 8

9 Example #2 Restricted activities Context: Nursing Jurisprudence Basic knowledge: Knowledge of restricted activities under scope of practice Simple template structure JP example debrief Range comprises 4 restricted activities Enemies ensure that two correct answers don t appear in any one question Question selection ensures each restricted activity appears at least once and Regulation 2018 Annual Educational Conference 9

10 Matt: ABIM Item-modeling: People and process Summary/Lessons learned/is modeling right for you? American Board of Internal Medicine Physician-led, non-profit, independent evaluation organization Founded in 1936 Certify physicians in Internal Medicine and 19 subspecialties ~1/4 of all US physicians certified by ABIM Over 200,000 diplomates and Regulation 2018 Annual Educational Conference 10

11 American Board of Internal Medicine Offering new Maintenance of Certification pathway Knowledge Check-In Taken every 2 years (instead of every 10 years) Optional Take-at-home format Item-modeling efforts Basics of ABIM item models Example model and items Production and review processes and Regulation 2018 Annual Educational Conference 11

12 Source item An existing item Variant Terminology A new version of a source item Item model Specifications for developing variants Rationale How the questions work Source item Rationale Item model Question Features Rationale: What information is each sentence conveying to the test taker? Testing point: Recognize the presentation of stent thrombosis What other ways might this feature present while keeping "LAD Stent Thrombosis" the correct answer? A 52-year-old man is brought to the emergency department because he had substernal pressure for 45 minutes Cardiac arrest occurs, and sinus rhythm is restored by a single shock Prompts Features Instructions For each sentence, how does the information provided influence your thinking about the most likely diagnosis for this patient? Where possible, tell us how the information helps you distinguish among the answer options (e.g., "This feature makes me think that option C is more likely because..."). Acute (read: bad) chest pressure lasting 45 minutes that presumably is constant, opening up a differential that is consistent with any of the answer options How does this information influence your thinking about the patient's presentation? This information helps inform my differential Dx by telling me that he is having a myocardial infarction in one of the two vessels How does this information influence your thinking about the answer options? Cardiac arrest being relieved by single shock is esp consistent with LAD stent thrombosis and acute transmural injury (options C and E) wouldn't expect aortic dissection, cardiac tamponade, or pulmonary embolism to respond to a single shock Age: Sex: M or F (M more common) Presenting symptoms: (some combination of option 1 OR 2 and any other can be added) 1) Substernal/chest pain/pressure; 2) "Tearing" pain/sensation (typically associated with aortic dissection, which would increase attractiveness of option ); 3) SOB; 4) Diaphoresis; 5) Nausea; 6) Lightheadedness; 7) Presyncope Cardiac arrest: (Choose one) 1) Cardiac arrest occurs, and sinus rhythm is restored by [1-3] shock(s); 2) DROP THIS SENTENCE; and Regulation 2018 Annual Educational Conference 12

13 Sample item model and variants C:\Users\mburke\Desktop\Samples\Durning_Aug2 018_LAD stent thrombosis - (FINAL)_with new answers.xlsx C:\Users\mburke\Desktop\Samples\44216_LAD stent thrombosis_multiple versions.rtf New test development process Item-writing Task Force (Produce content) New teams of SMEs (one team per exam) Develop models and pre-approve variants Exam Committees (Review content) Existing teams of SMEs Review and approve variants for use and Regulation 2018 Annual Educational Conference 13

14 Quarterly production process Assigns source items to IWTF members based on needs Task Force member Create models for assigned source items Draft variants for review on quarterly IWTF webinar Exam Developer Exam Editor Quarterly review process IWTF preapproves set of variants to send to EC IWTF Exam Committee EC small group will review items offline* *Unanimous approvals go directly to pre-testing Full EC will vote on disputed items via webinar Pre-testing and Regulation 2018 Annual Educational Conference 14

15 Summary Basics of ABIM item models Example model and items Production and review workflow Lessons learned Buy-in is critical (Staff and SMEs) Culture change is biggest challenge The more you think in terms of models, the more potential there is for efficiency and Regulation 2018 Annual Educational Conference 15

16 Is item modeling right for you? Do you administer a lot of forms? Do you offer exams in less secure environments? Is there a great deal of change in the content you are assessing? Do you want a better understanding of the way your content works? Are you looking to bolster your validity argument? and Regulation 2018 Annual Educational Conference 16

17 Janel: Task and Item Modeling Canadian Organization of Paramedic Regulators Task Model development Deriving Items models of different levels of complexity and Regulation 2018 Annual Educational Conference 17

18 Differentials A Elements B C Sub-elements D E F G Differentials A Elements B C Sub-elements D E F G Knowledge Which of the following [D/E/F/G sub-elements] is associated with [A]? and Regulation 2018 Annual Educational Conference 18

19 Differentials A Elements B C Sub-elements D E F G Application Which of the following [A] is most likely if the patient has [sub-element D] and [sub-element E]? Differentials A Elements B C Sub-elements D E F G Critical Thinking A patient is presenting with [sub-element D] and [sub-element F]. Which of the following [subelement E] will they also most likely present with? and Regulation 2018 Annual Educational Conference 19

20 Differentials A Elements B C Sub-elements D E F G Knowledge Which of the following signs is associated with a hemothorax [A]? a) Fever [sub-element D option] b) Peripheral edema [sub-element E option] c) Hypotension [sub-element F option] d) Distended neck veins [sub-element G option] and Regulation 2018 Annual Educational Conference 20

21 Differentials A Elements B C Sub-elements D E F G Application On paramedic arrival, a patient is anxiously moving around [D] and is clearly in respiratory distress. Assessment reveals that the patient has chest pain [F] and has localized wheezes on auscultation [E]. Which of the following differential diagnoses is most likely? a) Pulmonary embolism [A option 1] b) Pneumothorax [A option 2] c) Hemothorax [A option 3] d) Heart failure [A option 4] Differentials A Elements B C Sub-elements D E F G Critical Thinking A patient is presenting with a sudden onset [D] of respiratory distress. On assessment paramedics note that she is hypertensive [F] and has peripheral edema [G]. Which of the following signs will the patient also most likely present with? a) Wheezes on auscultation [sub-element E option 1] b) Crackles on auscultation [sub-element E option 2] c) Absent lung sounds [sub-element E option 3] d) Clear lung sounds [sub-element E option 4] and Regulation 2018 Annual Educational Conference 21

22 Item Templates On paramedic arrival, a patient is [BLANK] and is clearly in respiratory distress. Assessment reveals that the patient has [BLANK] and has [BLANK] on auscultation. Which of the following differential diagnoses is most likely? Thank you! Questions? and Regulation 2018 Annual Educational Conference 22