MKSAP 15. Performance Interpretation Guidelines with Norm Tables. Medical Knowledge Self-Assessment Program

Transcription

1 Medical Knowledge Self-Assessment Program Performance Interpretation Guidelines with Norm Tables MKSAP 15 Copyright 2010 American College of Physicians

2 MKSAP 15 Performance Interpretation Guidelines with Norm Tables Section I. How Do You Compare to Your Peers?... 6 Section II. Participant Groups... 9 Section III. What You Need to Know About Your Scores... 9 A. Introduction... 9 B. Content Validity C. Reliability (Rel) D. Standard Error of Measurement (SEM) Section IV. Norm Tables Section V. Answer Key and Percentage Answering Correctly Part A Part B Cardiovascular Medicine Gastroenterology and Hepatology Hematology and Oncology Neurology Rheumatology Dermatology Endocrinology and Metabolism General Internal Medicine Infectious Disease Nephrology Pulmonary and Critical Care Medicine Section VI. Self-Scoring Instructions Page 1

3 Principal Staff Deputy Executive Vice President, Senior Vice President, Medical Education and Publishing Steven E. Weinberger, MD, FACP Vice President, Medical Education D. Theresa Kanya, MBA Director, Self-Assessment Programs Sean McKinney Managing Editor Margaret Wells Program Administrator and CME Coordinator Valerie Dangovetsky Senior Staff Editors Katie Idell Becky Krumm, ELS Ellen McDonald, PhD, ELS Staff Editor Karen Nolan, MA Production Administrator/Editor Randy Hendrickson, MS, MPP Assistant Editors John Haefele Megan Zborowski Developed by the American College of Physicians Acknowledgements Computer scoring and reporting are being done by ACT, Iowa City, Iowa. Continuing Medical Education The American College of Physicians is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians. Page 2

4 The American College of Physicians designates this educational activity for a maximum of 166 AMA PRA Category 1 Credits. Physicians should only claim credit commensurate with the extent of their participation in the activity. The American Medical Association has determined that physicians not licensed in the United States who participate in this CME activity are eligible for AMA PRA Category 1 Credit. Credit is available from July 31, 2009 to July 31, Learning Objectives The learning objectives of the Medical Knowledge Self-Assessment Program are to: Close gaps between actual care in your practice and preferred standards of care, based on best evidence Diagnose disease states that are less common and sometimes overlooked and confusing Improve management of comorbidities that can complicate patient care Determine when to refer patients for surgery or care by subspecialists Pass the ABIM certification examination Pass the ABIM maintenance of certification examination Target Audience General internists and primary care physicians Subspecialists who need to remain up-to-date in internal medicine Residents preparing for the certifying examination in internal medicine Physicians participating in maintenance of certification in internal medicine (recertification) Page 3

5 Educational Disclaimer The editors and publisher of Medical Knowledge Self-Assessment Program 15 recognize that the development of new material offers many opportunities for error. Despite our best efforts, some errors may persist in print. Drug dosage schedules are, we believe, accurate and in accordance with current standards. Readers are advised, however, to ensure that the recommended dosages in MKSAP 15 concur with the information provided in the product information material. This is especially important in cases of new, infrequently used, or highly toxic drugs. The primary purpose of MKSAP 15 is educational. Information presented, as well as publications, technologies, products, and/or services discussed, are intended to inform subscribers about the knowledge, techniques, and experiences of the contributors. A diversity of professional opinion exists, and the views of the contributors are their own and not those of the ACP. The ACP disclaims any and all liability for damages and claims, which may result from the use of information, publications, technologies, products, and/or services discussed in MKSAP 15. Publisher s Information Copyright 2010 American College of Physicians, Inc. All rights reserved. This publication is protected by copyright. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic or mechanical, including photocopy, without the express consent of the ACP. Important copyright information from the ACP: Unauthorized reproduction of this publication is unlawful. The ACP prohibits reproduction of this publication in its entirety in any form either for individual use or for distribution. Page 4

6 The ACP will consider granting an individual permission to reproduce only limited portions of this publication for his or her own exclusive use. Send requests in writing to MKSAP Permissions, American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA Printed in the United States of America. For order information in the US or Canada call , extension All other countries call Fax inquiries to , or to Page 5

7 SECTION I. HOW DO YOU COMPARE TO YOUR PEERS? Welcome to the new Medical Knowledge Self-Assessment Program (MKSAP) Performance Interpretation Guidelines with Norm Tables. The NORM TABLES were based on the results of pre-publication test-takers who did not have access to the syllabus, test answers, and critiques. The goal of this approach was to record test scores that realistically reflected physicians knowledge. This supplement will help you interpret your scores on the multiple-choice questions from the 15th edition of MKSAP, and, in conjunction with your performance report, can help you plan an individualized program of continuing education. MKSAP 15 can also help you identify strengths or weaknesses in specific subspecialty areas of internal medicine, enhance your general knowledge, keep you abreast of new developments, and prepare you for future examinations. MKSAP 15 was composed of three program components, referred to as Parts A, B, and C. The first two components, Parts A and B, included Syllabi, Bibliographies, and Multiple-Choice Questions for 11 internal medicine subspecialty books. Part A was composed of five subspecialty test sections, and Part B was composed of six subspecialty test sections. After publication of the program, Individual Performance Reports for each part are sent to purchasers of the program who submit answer sheets to ACT, the American College of Physicians (ACP) scoring vendor, for CME credit. Individual Performance Reports are cumulative, presenting the total percent correct score and number of CME credits earned for each new subspecialty test completed as well as all previously completed subspecialty tests. The test question numbers of all questions answered incorrectly in the most recent test are listed with the score in each Individual Performance Report. These reports can aid you in determining your particular strengths and weaknesses. Parts A and B also included the Answers, Critiques, Educational Objectives, and Bibliographies for Multiple-Choice Questions. Page 6

8 Part C is this supplement, Performance Interpretation Guidelines with Norm Tables. If you have not received individual performance reports, you may score your own tests by using the answer keys provided in the MKSAP 15 books and Section V of this supplement and the Self-Scoring Instructions provided in Section VI. Self-scoring instructions are also provided at the beginning of the self-assessment section of each subspecialty book. To achieve the maximum educational benefit from Parts A and B of MKSAP 15, you should rely on the materials listed below: 1. Syllabus and Multiple-Choice Questions 2. Answers, Critiques, and Bibliographies 3. Your Individual Performance Reports, after submitting tests for CME credit 4. This supplement, Performance Interpretation Guidelines with Norm Tables Page 7

9 This supplement contains five additional sections. SECTION II. PARTICIPANT GROUPS. This section describes participants who submitted their tests for the analysis of data and norm groups. SECTION III. WHAT YOU NEED TO KNOW ABOUT YOUR SCORES. This section describes the steps taken to ensure the validity of the test questions and the scores. It also describes how the accuracy of the scores was evaluated and why this information is important. SECTION IV. NORM TABLES. This section contains norm data for participants and instructions for using the norm tables. SECTION V. ANSWER KEY, AND AGE ANSWERING LY. This section contains an answer key for each subspecialty section and defines the percentage of participants answering each test question correctly. SECTION VI. SELF-SCORING INSTRUCTIONS. This information will help you compute your percent correct score. Page 8

10 SECTION II. PARTICIPANT GROUPS Before you interpret the statistics provided in Section III and use the NORM TABLES, consider which participants' scores were used to develop the data. Groups were defined on the basis of information provided by participants in online question modules. These pre-publication participants, recruited through s sent to ACP members and notices posted on the ACP Web site, were asked to use their existing knowledge to answer questions, without accessing clinical reference material. As part of a pre-test screening process, physicians were asked to identify their primary subspecialty and were excluded from accessing tests related to their primary area of expertise. SECTION III. WHAT YOU NEED TO KNOW ABOUT YOUR SCORES A. INTRODUCTION Participants in MKSAP 15 may note that their scores vary greatly from the NORM TABLES scores. There may be a variety of reasons for this discrepancy. One, of course, is that some participants are significantly more knowledgeable or less knowledgeable than other participants. Other factors that could contribute to score variance: Participants who read the syllabus and soon thereafter answer the questions will do better on the test than someone who answers the questions without having read the syllabus. Those who read the test questions just before studying the syllabus and then answer the questions may also score higher. Some participants take the test as a "closed book" exercise; others do not. Some participants seek consultation from content experts; others attend educational group meetings before completing the tests. Your own scores may be influenced by such factors. Page 9

11 B. CONTENT VALIDITY The validity of the scores relied on the expertise of distinguished clinicians and educators who selected the content presented in each syllabus. Syllabus content emphasized recent significant medical developments. Test questions reflected primarily the material presented in each syllabus. After the questions were written, they passed through a series of editorial procedures designed to ascertain clarity, conciseness, and freedom from technical flaws. Even after the tests were administered, questions with unusual statistical performance were reviewed again by content experts. Questions in these tests that were determined to be flawed, too ambiguous, or too subspecialized were excluded from scoring. These questions are not counted toward CME credit. The excluded questions include: Cardiovascular Medicine: questions 67, 93, 118 Gastroenterology & Hepatology: question 63 Neurology: questions 43, 50 Rheumatology: question 62 Dermatology: question 21 General Internal Medicine: questions 58, 91 Infectious Disease: questions 31, 60 Pulmonary and Critical Care Medicine: questions 23, 36 Page 10

12 C. RELIABILITY (REL) It is important that test scores provide reliable estimates of ability. The reliability (REL) coefficient was a measure of internal consistency that provided an estimate of the accuracy or stability of the scores. An examination was reliable to the extent that, if the ACP were to re-examine participants with a comparable form of the test, scores would be about the same on the second test as on the first. The reliability coefficient may range from 0.00 to When scores are used to provide educational feedback for the assessment of strengths and weaknesses, as they are for MKSAP 15, it is recommended that the reliability be.70 or higher. The table below shows the reliability for each subspecialty section. SUBSPECIALTY SECTION REL Cardiovascular Medicine.94 Gastroenterology and Hepatology.94 Hematology and Oncology.95 Neurology.93 Rheumatology.89 Dermatology.80 Endocrinology and Metabolism.95 General Internal Medicine.93 Infectious Disease.93 Nephrology.90 Pulmonary and Critical Care Medicine.94 Page 11

13 D. STANDARD ERROR OF MEASUREMENT (SEM) Ideally, the ACP would like to create tests that provide perfectly accurate measurements of participants' knowledge. In practice, however, scores are never completely free of error; they are estimates of what might be called "true ability." The standard error of measurement (SEM) is an index that indicates how much error there is in test scores. Using the SEM, it is possible to define a score interval that indicates how far a test score is likely to stray from a participant's true ability. The interval is found by adding the SEM to a score and subtracting it from that score. Observed scores are expected to fall within one SEM of the true ability of participants two times out of three, or approximately 67% of the time. A participant whose true ability was 70 would usually receive a score between 65 and 75 (70-5 and ) in a test. The table below shows the SEM for each subspecialty test. SUBSPECIALTY SECTION SEM Cardiovascular Medicine 4.62 Gastroenterology and Hepatology 4.43 Hematology and Oncology 4.17 Neurology 3.98 Rheumatology 3.71 Dermatology 2.68 Endocrinology and Metabolism 4.58 General Internal Medicine 4.75 Infectious Disease 4.56 Nephrology 3.68 Pulmonary and Critical Care Medicine 4.59 Page 12

14 SECTION IV. NORM TABLES The NORM TABLES are useful for determining your standing relative to that of participants who completed tests before the program was published. Norm data are provided for each group of participants. The first column contains SCORES. The remaining columns contain ILE RANKS. The percentile rank indicates the percentage of the group that scores lower than the corresponding percent correct score. To use the NORM TABLES, first choose a percent correct score from either your Individual Performance Report or your own hand score computations. Then select the table for the norm group with which you would like to compare yourself. In the first column, locate your SCORE. Then read across the table to find your ILE RANK. For example, suppose that your score was 83% or 84% correct in Cardiovascular Medicine. At the percent correct scores of on the NORM TABLE, read across the page to find your percentile rank of Your percentile rank means that 90.6 of the group scored lower than you in Cardiovascular Medicine. For each norm table, the N under the subspecialty section indicates the number of participants who completed tests for that subspecialty before the program was published. Page 13

15 Cardiovascular Medicine (N=160) NORM TABLES FOR MKSAP 15 Gastroenterology Hematology and and Hepatology Oncology (N=132) (N=115) Percent Correct Score Neurology (N=120) 99 and above Rheumatology (N=132) Below 20 Page 14

16 NORM TABLES FOR MKSAP 15 (continued) Endocrinology General and Internal Infectious Metabolism Medicine Disease (N=132) (N=150) (N=132) Pulmonary and Critical Care Medicine (N=148) Percent Correct Score Dermatology (N=135) Nephrology (N=144) 99 and above Below 20 Page 15

17 SECTION V. ANSWER KEY AND AGE ANSWERING LY This section contains an answer key for each subspecialty section and the percentage of pre-publication test-takers who answered each test question correctly. As noted previously, subspecialists were excluded from taking tests in their primary areas of expertise to ensure a more realistic measurement of the general knowledge of all participants. CARDIOVASCULAR MEDICINE ANSWER ANSWER KEY KEY 1 D B 49 2 A A 49 3 C B 87 4 D C 50 5 B C 72 6 D E 64 7 C EXCLUDED 8 D B 66 9 A A C C D C B C E B A B C C D A B B A B C D A B D D C B C C E B B A D B E A C D D B B D C A B D 59 (Columns continued on next page) Page 16 (Columns continued on next page)

18 CARDIOVASCULAR MEDICINE (continued) ANSWER ANSWER KEY KEY 33 B EXCLUDED 34 A A D B B B B D B B C E C B E B D D C A A D D D D B D A A D D A B B E C D A B D A D B B B B D B D EXCLUDED 59 B A C D 36 Page 17

19 GASTROENTEROLOGY AND HEPATOLOGY ANSWER ANSWER KEY KEY 1 B B 47 2 D D 69 3 D D 83 4 C B 89 5 C E 68 6 A E 49 7 A D 80 8 D EXCLUDED 9 B C B A A D D C B B C B A C D A A E B B D C E C E C A A B D C C D A B B C B D B B D A B C A D A C D B E B B D B C C D B D B D C B B B B B C A C 82 (Columns continued on next page) Page 18 (Columns continued on next page)

20 GASTROENTEROLOGY AND HEPATOLOGY (continued) ANSWER KEY ANSWER KEY 45 A D D E B B A B D D B B B C D B A B D C D B 59 Page 19

21 HEMATOLOGY AND ONCOLOGY ANSWER KEY ANSWER KEY 1 C B 63 2 A D 39 3 A A 76 4 C C 94 5 E A 62 6 D B 56 7 E D 49 8 A A 83 9 C D A A C C A B B C D A C E B B D E A B D D B A D C D D D A B E C D B D C A A D D D C B A B D B B C C B C B A C D C C E C C B C E A C D B C B B B C 87 (Columns continued on next page) Page 20 (Columns continued on next page)

22 HEMATOLOGY AND ONCOLOGY (continued) ANSWER ANSWER KEY KEY 46 B C A E D B C C A B 57 Page 21

23 NEUROLOGY ANSWER ANSWER KEY KEY 1 B D 60 2 A B 68 3 B C 83 4 C D 81 5 D EXCLUDED 6 B C 66 7 E C 65 8 C D 54 9 A A C A A C D A A D C B C C B C C A B D A D C C E B B D A A D C C A A C C D B C B C D B A A D C C B D A B C A B D C C C C B C D A A D C EXCLUDED 88 B C B A A 71 Page 22

24 RHEUMATOLOGY ANSWER ANSWER KEY KEY 1 D B 58 2 A D 65 3 A D 89 4 E D 73 5 D D 56 6 C A 40 7 A C 82 8 B A 58 9 D B D D A E A E B D A A A B A D D B B A D B A A B B B EXCLUDED 23 B E B A D C B D D A D D D B A D C A C A D A A D A C A C D B D A C A B 36 Page 23

25 DERMATOLOGY ANSWER KEY 1 A 64 2 B 76 3 B 93 4 A 83 5 D 53 6 C 67 7 C 51 8 C 36 9 A C D D C C C A C C A D EXCLUDED 22 D D E D E A A A B B C E D B A C A C C 93 Page 24

26 ENDOCRINOLOGY AND METABOLISM ANSWER ANSWER KEY KEY 1 D D 83 2 A C 75 3 C A 63 4 E D 75 5 D B 80 6 C C 60 7 D D 64 8 B D 74 9 A C C D D B D C C D C C D B A D B D D A A B C D B C C D D B B D E C A D D B C B B C B A C D B D D B A B C D A C D B D C B C B D A C C C C C A E D B C C 47 (Columns continued on next page) Page 25 (Columns continued on next page)

27 ENDOCRINOLOGY AND METABOLISM (continued) ANSWER KEY ANSWER KEY 47 D A B D C B E A C A D D A B C D C C D D D B B D 82 Page 26

28 GENERAL INTERNAL MEDICINE ANSWER ANSWER KEY KEY 1 A D 55 2 A D 67 3 D D 79 4 A E 36 5 A A 47 6 B B 42 7 B D 92 8 A D 86 9 A B E C B D D A B D E B C E C C B B C E E D B D D B D A A E E EXCLUDED 25 E A B D C D A B B D C B A B A A C C C D A D C D B B D E C C E E E D A E C B 66 (Columns continued on next page) Page 27 (Columns continued on next page)

29 GENERAL INTERNAL MEDICINE (continued) ANSWER ANSWER KEY KEY 44 C A D A B A D D C D A C C D C D C E A A C A C D E B C B EXCLUDED 125 D A A B A B D A D B A C D A B C E C 91 Page 28

30 INFECTIOUS DISEASE ANSWER ANSWER KEY KEY 1 C EXCLUDED 2 A C 77 3 D B 83 4 C A 73 5 C D 19 6 B D 24 7 D C 52 8 D B 71 9 A D A B C B C D B C B D C C E E D A D B E D A B D C D A C B B C E C E D D A B D A B E B EXCLUDED 90 A D C B A A D B B C E C A C C B A D D C B A C B A D C C B C A 64 (Columns continued on next page) Page 29 (Columns continued on next page)

31 INFECTIOUS DISEASE (continued) ANSWER ANSWER KEY KEY 47 D D B B B B A D A D D A A A B B E E C C C D A C D C 64 Page 30

32 NEPHROLOGY ANSWER ANSWER KEY KEY 1 B D 44 2 C A 41 3 B A 70 4 A C 49 5 D B 37 6 C B 40 7 B B 45 8 D C 57 9 A C B D D C A B C D B C D D B A D B C D A B A A B C A D B A C B B D C C C C D C D D C B C C D C B B C D D D D A 58 Page 31

33 PULMONARY AND CRITICAL CARE MEDICINE ANSWER ANSWER KEY KEY 1 C C 73 2 A E 77 3 A B 57 4 E C 85 5 C A 55 6 E E 35 7 D D 65 8 A C 82 9 E E D B A A A C B A D A B B C D A B A B A D A D D C A C EXCLUDED 81 E A B E A D D D E A A C D D C D B D C B D A B B E EXCLUDED 94 A C D B B D C A A A A B B C C A D C D C E 41 (Columns continued on next page) Page 32 (Columns continued on next page)

34 PULMONARY AND CRITICAL CARE MEDICINE (continued) ANSWER KEY ANSWER KEY 47 B B D C D E A C B C D B B C D C A B B C C B A A 84 Page 33

35 SECTION VI. SELF-SCORING INSTRUCTIONS Compute your percent correct score as follows: Step 1. Give yourself a point for each correct response to each question in the test in which you are working. Step 2. Divide the number of correct responses in Step 1 by the number of accepted questions for the test in which you are working. The number of accepted questions in each test is listed in the table at the bottom of this page. Step 3. Multiply the result by 100. The final result will be your total percent score, which you can compare to the Norm Tables under Section IV. Example #1 Cardiovascular Medicine Step 1: Record your points. 112 Step 2: Divide the result by 117, which is the total number of questions Step 3: Multiply result by x 100 Total percent correct score 95.70% Subspecialty Section Number of Published Questions Number of Excluded Questions Number of Accepted Questions Cardiovascular Medicine Gastroenterology and Hepatology Hematology and Oncology Neurology Rheumatology Dermatology Endocrinology and Metabolism General Internal Medicine Infectious Disease Nephrology Pulmonary and Critical Care Medicine Page 34