THE PSYCHOMETRICS OF PERFORMANCE ASSESSMENTS A PHYSICIAN S PERSPECTIVE

Transcription

1 THE PSYCHOMETRICS OF PERFORMANCE ASSESSMENTS A PHYSICIAN S PERSPECTIVE Jeanne M. Sandella, DO

2 Objectives Discuss the reliability of performance assessments why are they used, what are the threats to having a good assessment? Identify the impact of human raters on the psychometrics of performance assessments Define the examinee centered approach for setting standards for performance assessments

3 Why a physician s perspective? y = log e (x/m sa) r 2 yr 2 = log e (x/m sa) e yr2 = x/m sa me yr2 - x sma me rry = x - mas

4 What makes a good assessment? (Kane s framework) 1. Scoring 2. Generalization 3. Extrapolation 4. Interpretation/decision

5 1. Scoring Fair administration Individuals were evaluated accurately Scoring rules consistently applied

6 Examination administration Fair administration

7 Accurate evaluations Scoring Checklists measure an explicit process (H and P) Evidence based development by experts Rubrics measure implicit processes; Holistic Require significant expertise/training

8 Human Raters are they consistently applying rules? Training of raters (physician and SP) Score equating/calibration Quality assurance of raters

9 Human raters

10 2. Generalization Generalization Good sampling of observations Enough samples

11 Generalizaton factors contributing to errors. Heterogeneity of candidates

12 Generalization factors contributing to errors. Number of cases

13 Generalize results adequate sampling Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge Y Case 1 A B C A A A A A Case 2 B C Case 3 B C Case 4 B C Case 5 B C Case X B C

14 3. Extrapolation Observations are relevant to the construct of interest Scores are not unduly influenced by sources of variance not related to the construct being measured

15

16 Interpretation/decision Framework for score interpretation can be supported Categorization is supported

17 Standard setting What do the scores mean? E.g. Pass/Fail

18 Standard Setting Anatomy A Standard setting method B Defining a performance standard C Derive the Cut-point D Finalization of Cut score

19 Standard Setting Method Examinee-Centered Method Panelists make independent judgments of qualified or not qualified performance on the clinical skill of interest by reviewing actual or proxy performance on the examination.

20 Defining a performance standard Not qualified Qualified

21 Deriving a cut score P r o p = Q Score scale

22 Finalization of Cut score NBOME Executive Committee Stakeholder Surveys Triangulation Expert Panelists

23 Factors contributing to classification errors. The cut score Higher cutscore = more false negatives. Lower cutscore = more false positives.

24 Objectives Discuss the reliability of performance assessments why are they used, what are the threats to having a good assessment? Identify the impact of human raters on the psychometrics of performance assessments Define the examinee centered approach for setting standards for performance assessments

25 Selected references 1. Boulet JR, Gimpel JR, Errichetti AM, Meoli FG. Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians. The Journal of the American Osteopathic Association 2003;103: Boulet, J., & Swanson, D. (2004). Psychometric challenges of using simulations for high-stakes assessment. In Dunn, W.F. (ed.). Simulators in Critical Care and Beyond. Society of Critical Care Medicine, Des Plaines, IL. 3. Holmboe, E. (2015) Direct Observations of Students Clinical Skills. In Pangaro, LN and McGaghie W.C. (eds) Handbook on Medical Student Evaluation and Assessment. Gegensatz Press, North Syracuse, NY. 4. Boulet, J.R. & McKinley, D.W. (2013). Criteria for good assessment. (Chapter 2). In McGaghie, W.C. (ed.). International Best Practices for Evaluation in the Health Professions. Radcliffe Publishing, London.

26 THANK YOU.