Test Blueprinting II: Creating a Test Blueprint NBME

Test Blueprinting II: Creating a Test Blueprint NBME Natinal Bard f Medical Examiners 3750 Market Street Philadelphia, PA 19104

Test Blueprinting II: Creating a Test Blueprint Cpyright 2019 Natinal Bard f Medical Examiners (NBME )

By the end f this lessn, yu shuld be able t: Test Blueprinting II: Creating a Test Blueprint Objectives Understand hw curse cntent and learning bjectives will guide blueprint develpment Develp a test blueprint t guide the develpment f yur assessments Intrductin Cnsider yurself in the fllwing scenari. Yu are preparing an end-f-curse exam fr yur immunlgy curse. Yu want t make sure yur students have mastered the imprtant material frm the curse, and yu have a bank f 200 multiple-chice questins (MCQ) n immunlgy tpics frm a cllabrative prject with ther immunlgy curse instructrs. Hw d yu determine which questins t use n yur exam t ensure yur test is a high-quality, reliable, and defensible indicatr f yur students knwledge? Creating a test blueprint will help yu plan which questins t include in yur exam and ensure that it adequately assesses the learning bjectives f the curse. A test blueprint defines the knwledge and skills yu want t assess and will enable yu t build purpse-driven, successful assessments. A blueprint may als help yu identify areas where yur questin pl may be lacking. What is a Test Blueprint? A test blueprint is a list f key cmpnents defining yur test, including: The purpse f the test: It might be smething simple, such as assessing knwledge prir t instructin t a get a baseline f what students knw befre taking a curse. Alternatively, the test purpse might be mre cmplex, such as assessing retentin f material learned acrss several rgan-system curses t determine eligibility fr advancement. The cntent framewrk: Start with the schemas r framewrks cmmnly used t rganize and cnslidate medical knwledge. Fr example, basic science (e.g., bichemistry, genetics) r clinical science (e.g., surgery, pediatrics) disciplines are cmmn schemas. Test Blueprinting II: Creating a Test Blueprint 2

The testing time: This includes amunt f testing time available and the need fr breaks, as well as ther lgistical issues related t the test administratin. The cntent weighting (aka, number f items per cntent area): The number f questins per tpic categry shuld reflect the imprtance f the tpic; that is, they shuld crrelate with the amunt f time spent n that tpic in the curse. Fr example, if there are 20 ne-hur lectures, there may be 10 questins frm each hur f lecture r assciated with each hur f expected study. The number f questins per categry can be adjusted up r dwn t better balance the verall test cntent and represent the imprtance f each lecture, as well as the ttal lecture time. The item frmats (e.g., MCQ, essay questin): The item frmats shuld always be apprpriate fr the purpse f the assessment. Benefits f Test Blueprints Test blueprints will help ensure that yur tests: Apprpriately assess the instructinal bjectives f the curse Apprpriately reflect key curse gals and bjectives the material t be learned Include the apprpriate item frmats fr the skills being assessed Test blueprints can be used fr additinal purpses besides test cnstructin: Demnstrate t students the tpics yu value, and serve as a study guide fr them Facilitate learning by prviding a framewrk r mental schema fr students Ensure cnsistent cverage f exam cntent frm year t year Cmmunicate curse expectatins t stakehlders (e.g., trainees, ther faculty, administratin) Cntent Framewrks An utline is ne cmmn way t rganize a syllabus r plan a lecture. Smetimes, utlines are infrmal and simply list a few tpics; ther times, utlines are meticulusly detailed with tpics and subtpics at multiple levels arranged in a hierarchical manner. A curse utline, alng with the fllwing list, can serve as starting pints fr identifying the cntent framewrks that will be part f a test blueprint: Textbk table f cntents Lecture tpics Test Blueprinting II: Creating a Test Blueprint 3

Presentatin slides Patient cases Labratry experiments Practice analysis Behaviral bjectives Medicine prvides several schema per cntent area, s there are ften different ways f rganizing the same material inside a given cntent area. Medical schl curricula are frequently rganized by discipline r by rgan system. Anther framewrk is physician skill r cmpetency, which, in cntrast t the discipline and rgan system framewrks, fcuses n what type f tasks a student shuld be able t perfrm. In the fllwing table, every cntent area within Basic Science als fits int at least ne schema. Table 1. Cntent Areas Within Schema. Basic Science Clinical Science Organ System Physician Cmpetency Bichemistry Internal Medicine Hematpietic & lymphreticular Interpersnal & cmmunicatin skills Genetics Surgery Nervus Medical knwledge Pharmaclgy Pediatrics Skin & cnnective tissue Patient care Pathlgy Psychiatry Musculskeletal Practice-based learning & imprvement Anatmy Obstetrics & Gyneclgy Respiratry Systems-based practice Physilgy Family Medicine Cardivascular Prfessinalism Micrbilgy Neurlgy Gastrintestinal The fllwing image shws a list f medical framewrks r schemas that culd be used in test blueprinting. These schema are arranged in prgressin frm framewrks based n tpics t thse based n practice skills. Test Blueprinting II: Creating a Test Blueprint 4

Figure 1. Cntinuum f Schemas By Knwing/Ding Knwing / Tpic riented Academic Disciplines and Subdisciplines Organ Systems Disease Classificatins Clinical Presentatin r Reasn fr Visit Medical Specialty Physician Cmpetency Physician Task r Skill Dmain Ding / Practice riented Cntent Weighting Once yu have identified a framewrk r schema t rganize the test tpics, yu shuld cnsider the imprtance f each tpic in rder t determine the best cntent weighting fr the test as a whle. The number f questins fr each cntent area shuld reflect the imprtance f that cntent area. The fllwing questins will help yu determine the weight f each tpic in the assessment: Hw much f yur lecture time d yu spend teaching this tpic in yur curse? Hw much time are examinees expected t spend studying each tpic area independently? Hw much f the curse textbk r lab time is devted t this tpic? Hw much f this tpic will yur students need t knw as they mve thrugh their training? What ther tpics shuld get abut the same amunt f test time as this tpic? The fllwing table shws ne example f a blueprint with cntent weighting. In this example, there is rm fr flexibility, that is, sme tpics have tw days f lecture and require five questins, while thers have tw days f lecture and require eight questins. This flexibility allws fr tpics f similar imprtance that may cntain cntent f greater cmplexity. Test Blueprinting II: Creating a Test Blueprint 5

Table 2. Cntent Blueprint fr Physilgy. Physilgy Subtpic Lecture Days # f Test Questins Chemical cmp f bdy 1 3 Cell structure & genetic cntrl 3 12 Cell respiratin & metablism 2 5 Neurns & membrane ptentials 3 12 The nervus system 3 12 Central and autnmic nerv sys 1 4 Muscle 4 16 Bld & bld cltting 2 8 Bld & bld cltting 1 4 Cardivascular system 4 16 Respiratry physilgy 2 8 Yu are nt limited t using nly a single framewrk r schema t plan yur test. It is pssible t cmbine framewrks t frm a blueprint matrix. A blueprint matrix allws yu t balance cntent acrss mre than ne framewrk. The fllwing table shws a blueprint matrix with cardivascular disease tpics and physician tasks. The numbers refer t the number f questins in the test that will address that cntent. Fr example, tw questins will assess diagnsis f ischemic heart disease. Table 3. Cntent Blueprint by Cntent and Task. Cardivascular Cntent Health and Health Maintenance Diagnsis Management Infectius and Immunlgic 2 1 1 Neplasm 2 2 1 Dysrhythmias 2 1 2 Heart failure 1 2 Ischemic heart disease 2 1 Diseases f mycardium 1 2 1 Vascular disrders 1 2 Test Blueprinting II: Creating a Test Blueprint 6

Blueprinting Perfrmance Assessments The principles described n the previus screens can als be applied t ther types f assessments, including Objective Structured Clinical Examinatins (OSCEs). Like a test blueprint, an OSCE blueprint will define the purpse f the test, and cnditins fr sampling, such as the skill being assessed and the rgan system prblem presented by the patient. The example OSCE blueprint shwn belw is rganized by a framewrk describing cmmunicatin challenges. Additinal case infrmatin is prvided t describe the patient s presenting scenari (the main issue). Table 4. Cntent Blueprint fr OSCE. Case Age Gender Cmmunicatin Frmat Scenari Challenge 1 10 mnths F Infrmed cnsent SP encunter Child need lumbar puncture 2 18 F Cunseling fr safe sex SP encunter Sexually active teen with herpes diagnsis 3 35 M Behavir change SP encunter Weight lss 4 45 M Nn-adherence SP encunter Nn-therapeutic drug levels 5 65 M Behavir change SP encunter Smking cessatin 6 75 F Delivering bad news 2 SPs (mther and daughter) Dementia diagnsis Classifying Test Questins T make creating a test even easier, as yu develp and cllect test questins r cases, classify them accrding t the same framewrks yu use fr yur blueprint. This will help yu find specific material frm yur pl mre efficiently and keep track f needs during the item r case-writing prcess. The fllwing table shws sme sample test items cded fr diagnsis and rgan system. Test Blueprinting II: Creating a Test Blueprint 7

Table 5. Items Cded by Diagnsis and Organ System. Diagnsis Organ System Questin Cancer, skin, squamus cell Skin & cnnective tissue A 72-year-ld man wh is a retired cnstructin wrk cmes t the physician because f a lesin n his face fr 3 mnths. Physical examinatin shws Cancer, cervical Reprductive A 42-year-ld wman, gravida 2, para 2, cmes fr a rutine examinatin. She has a histry f vulvar cndylmat acuminata Cancer, lung Respiratry A 74-year-ld man has had cnfusin fr 2 weeks. He has smked tw packs f cigarettes daily fr 50 years Here are sme additinal ways t classify test material. Age (e.g., pediatric, adult) Gender Site f care (e.g., clinic, emergency department) Disease Presenting cmplaint Examinee level (e.g., end f curse, end f year) Task Putting It All Tgether Blueprinting can help yu: Assess the instructinal bjectives f the curse r instructinal perid Avid ver- r under-representing a tpic in yur test Use apprpriate frmats fr the skills being assessed Shw students the tpics yu value Test Blueprinting II: Creating a Test Blueprint 8

Ensure similar exam cntent frm year t year Cmmunicate expectatins t ther stakehlders (e.g., ther faculty, administratin) Blueprinting can help students: Cnstruct meaning f the cntent by prviding a framewrk r mental schema Plan their studying See the tpics yu value mst Resurces ACGME Outcmes Prject & ABMS. (2000). Tlbx f Assessment Methds. Chicag, IL: Accreditatin Cuncil fr Graduate Medical Educatin and American Bard f Medical Specialties. Bartlett, F. C. (1932). Remembering: A Study in Experimental and Scial Psychlgy. Cambridge, England: Cambridge University Press. Bulet, J., & Raymnd, M. R. (2013). Blueprinting: Planning Yur Tests. FAIMER-Keele Master s in Health Prfessins Educatin: Accreditatin and Assessment. Lessn 1, Unit 2. Lndn, England: FAIMER Centre fr Distance Learning (CenMEDIC). Dwning, S. M., & Haladyna, T. M. (2006). Handbk f Test Develpment. New Yrk, NY: Rutledge. Dwning, S. M., & Yudkwsky, R. (2009). Assessment in Health Prfessins Educatin. New Yrk, NY: Rutledge. McLed, S. A. (2009). Jean Piaget s Thery f Cgnitive Develpment. Simply Psychlgy. Retrieved January 2019 frm http://www.simplypsychlgy.rg/piaget.html. Raymnd, M. R., & Neustel, S. N. (2006). Determining the cntent f credentialing examinatins. In S. M. Dwning & T. M. Haladyna (Eds.), Handbk f Test Develpment (pp. 181 223). Mahwah, NJ: Lawrence Erlbaum Assciates. Rider, E. A., Nawtniak, R. H., & Smith, G. (2007). A Practical Guide t Assessing the ACGME Cre Cmpetencies. Marblehead, MA: HCPr, Inc. Test Blueprinting II: Creating a Test Blueprint 9