Reliability of oral examinations: Radiation oncology certifying examination
|
|
- Julian Barrett
- 5 years ago
- Views:
Transcription
1 Practical Radiation Oncology (2013) 3, Special Article Reliability of oral examinations: Radiation oncology certifying examination June C. Yang PhD, Paul E. Wallner DO, Gary J. Becker MD, Jennifer L. Bosma PhD, Anthony M. Gerdeman PhD American Board of Radiology, Tucson, Arizona Received 28 July 2011; revised 24 October 2011; accepted 25 October 2011 Abstract Purpose: Oral examinations are used in certifying examinations by many medical specialty boards. They represent daily clinical practice situations more realistically than do written tests or computerbased tests. However, there are repeated concerns in the literature regarding objectivity, fairness, and extraneous factors from interpersonal interactions, item bias, reliability, and validity. In this study, the reliability of oral examination on the radiation oncology certifying examination, which was administered in May of 2010, was analyzed. Methods and Materials: One hundred fifty-two candidates rotated though 8 examination stations. Stations consisted of a hotel room equipped with a computer and software that exhibited images appropriate to the content areas. Each candidate had a minute face-to-face encounter with an oral examiner who was a content expert in one of the following areas: gastrointestinal, gynecology, genitourinary, lymphoma/leukemia/transplant/myeloma, head/neck/skin, breast, central nervous system/pediatrics, or lung/sarcoma. This type of design is typically referred to as a repeated measures design or a subject by treatment design, although the oral examination was a routine event without any experimental manipulation. Results: The reliability coefficient was obtained by applying Feldt and Charter's simple computational alternative to analysis of variance formulas that yielded KR-20, or Cronbach's coefficient alpha of Conclusions: An experimental design to develop a blueprint in order to improve the consistency of evaluation is suggested American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved. Introduction Oral examinations continue to be administered as certifying examinations by 14 of 24 medical specialty boards. Among the advantages of oral examinations is Conflicts of interest: None. Corresponding author. American Board of Radiology, 5441 E. Williams Blvd, Tucson, AZ address: jyang@theabr.org (J.C. Yang). that they represent daily clinical practice situations more realistically than written examinations do. An oral examination can test the limits of a candidate's knowledge on the given topic. They are considered particularly effective in assessing a candidate's clinical decision-making ability and interpersonal skills, as well as intrapersonal qualities such as confidence and selfawareness. 1,2 However, concerns about the reliability, validity, and fairness of oral examinations have been expressed repeatedly in the literature /$ see front matter 2013 American Society for Radiation Oncology. Published by Elsevier Inc. All rights reserved.
2 Practical Radiation Oncology: January-March 2013 The major concerns that have been raised pertain to the inherent nature of oral examinations; that is, the effects of the personal interactions. These include the following: examinees' communication or language skills; examinees' familiarity with oral examination form; inhibition of the performance due to stress; the examiner's judgments related to demographic information such as age, gender, race, or socioeconomic status; and other factors that are unrelated to actual capabilities (knowledge, skills, and judgment). Would the candidate receive the same score if a different examiner tested him or her? Reliability is a major concern because subjective judgments may affect the scores. Different examiners may have different degrees of harshness or leniency. Content or follow-up questions may vary from one examiner to another. Contrary to the above concerns, Lunz and Bashook 2 found that when scores on communication ability were compared with scores on oral examinations in a medical specialty board certification process, the correlation coefficient between the communication scores and the oral examination scores was A random sample of 90 candidates was observed. Nonmedical researchers measured the communication ability of the candidates using a 21-item communication survey instrument. The authors concluded that the candidates' oral examination scores were not influenced by their communication abilities. Recently, the American Board of Psychiatry and Neurology decided to eliminate oral examinations from the certification examination process, citing as concerns the candidates' stress and anxiety associated with the oral examinations, as well as financial stress. 6 The American Board of Radiology (ABR) has also decided to discontinue (beginning in 2014) the oral examination as the final certifying examination for the diagnostic radiology certification process. This decision was based on concerns about subjectivity and technically inadequate presentations. For example, in years past case presentations during the oral examination were similar to what could be observed in daily clinical situations. However, with the advancement of electronic and technical modalities, such as computed tomography and magnetic resonance (MR) imaging, MR angiography, and MR spectroscopy, it has become more difficult for the oral examination to simulate actual clinical practice. Medical physics and radiation oncology (RO) will continue the oral examinations in their certifying processes for those who have passed the initial written qualifying examinations. Methods and materials Reliability of oral examinations 75 The radiation oncology oral examination, a criterionreferenced test, is administered as the final certifying examination to those who have completed one internshiptransitional year plus 4 years of radiation oncology residency. This generally occurs 10 months after candidates obtain a passing score on the qualifying examination, which is taken at the end of the fourth year of residency. In this study, the oral examination scores were obtained from the regularly scheduled and administered RO oral examination in May The oral examination was not an experimental design and there was no modification or manipulation made in yielding the candidates' performance scores. No Institutional Review Board application was submitted as no human subject or identity was used for the study. Following a long-established procedure, all RO oral examiners received about 1 hour of general orientation followed by several hours of discussion within their specific category groups, before the beginning of the oral examination. Also, all new examiners were required to observe 2 oral examinations before giving their first examinations. No announcements or comments were made to the examiners or examinees regarding a study of the reliability of oral examination. That is, the reliability coefficient was calculated simply by using the existing data. In this routinely administered RO oral examination, 152 candidates rotated through 8 examination stations. Stations consisted of a hotel room equipped with a computer and software that exhibited images appropriate to the content areas. Each candidate had a face-to-face encounter with an oral examiner who was a content expert in one of the following areas: gastrointestinal, gynecology, genitourinary, lymphoma/leukemia/transplant/ myeloma, head/neck/skin, breast, central nervous system/ pediatrics, or lung/sarcoma. Each oral examination lasted minutes. The scoring rubrics have been developed over the years, the continuous score scale ranged from 68 to 72 with a 1-point interval among the performances, and 70 as the passing score on each category. During each face-toface oral examination, the examiners, independent from each other, filled out specifics about the candidate's performance on a blank form provided with case numbers. At the end of the oral examinations, the examiners of each category met to discuss in depth the accuracy and fairness of some candidates' failing grades. After the meeting, those with failing or borderline performances were presented to the oral examiner panel of all 8 clinical categories so they could be reconsidered and their final grades determined. The textbook Educational Measurement presents several methods for estimating reliability coefficients, depending on the type of test and the theoretical framework. 7 The design of this RO oral examination is referred to as one way repeated-measures design, subject by treatment design, or complete randomized block design, and the analysis of the results are derived from analysis of variance (ANOVA) and produce exactly the same results. Although the result will be identical, one may also apply a generalizability model, as presented by
3 76 J.C. Yang et al Practical Radiation Oncology: January-March 2013 Brennan in his GENOVA program. 8 The author chose to use a simple computational alternative to ANOVA, or GENOVA, which was published by Feldt and Charter in 2004 [9], as follows: n h io r k =1 SD 2 x SD2 j SD2 s = ðk 1ÞSD 2 s ; where r k is k-judge reliability, SD x is the standard deviation of all scores using kn as the divisor. N is the number of subjects and k is the number of judges. SD j is the standard deviation of judge means using k as the divisor, and SD s is the standard deviation of each subject's test performance. The best score for each subject is the mean of the judges' ratings having a standard deviation. Note that SD 2 is variance. The above formula is identical to Cronbach's coefficient alpha, which is the same as the Kuder-Richardson formula 20 (KR-20), which can be obtained by ANOVA results with the interaction term subtracted. That is, ρ = (MS s MS A S )/ (MS s ), where MS represents mean square, the subscript s represents subject, and the subscript A S represents interaction in ANOVA terms. 9 Results Applying Feldt and Charter's formula, 9 the simple calculation method yielded the inter-rater reliability coefficient KR-20 of 0.81 for the oral examination administered by 8 content expert examiners. This means that 81% of the examinees' scores would maintain the same rank orders if a comparable examination was administered. Table 1 exhibits the data entry format, which contains partial data on the oral examination scores of 152 candidates, the mean of each candidate for 8 competency areas, and the mean of each rater who is the content expert examiner (the term rater is used interchangeably with examiner in this paper). Correlation coefficients were obtained among 8 oral examination scores and 3 RO written examination scores, biology, physics, and clinical, acknowledging that the examinees may have taken different written examination forms at different times and that the reliability coefficients varied somewhat from test to test. During the past 5 years, RO biology tests' reliability coefficients varied from 0.92 to 0.96 (0.95 on average); the reliability coefficients of RO physics examinations ranged from 0.88 to 0.96 (0.94 on average) and RO clinical component examinations had reliability coefficients ranging from 0.93 to 0.95 (0.95 on average). The correlation coefficients were obtained among the oral examination scores and the RO written tests. Restriction in range was not corrected because 152 examinees have taken different versions of the written examinations over the years. All correlation coefficients were statistically significant at 2-tailed P =.05. Most of them were statistically significant at P =.0001, suggesting highly consistent relationships between the written test scores and oral examination scores. The clinical examination, a written test, was highly correlated with all 8 categories of the oral examinations, yielding correlation Table 1 Subject (Candidate) Data format of May 2010 radiation oncology oral examination scores Rater 1 (GI) Rater 2 (Gyn) Rater 3 (GU) Rater 4 (Lymph) Rater 5 (HNS) Rater 6 (Breast) Rater 7 (CNS/PEDS) Rater 8 (Lung) Mean (Subject) Mean Standard deviation CNS/PEDS, central nervous system/pediatrics; GI, gastrointestinal; GU, genitourinary; GYN, gynecology; HNS, head/neck/skin; Lymph, lymphoma/ leukemia/transplant/myeloma.
4 Practical Radiation Oncology: January-March 2013 coefficients ranging from to All were statistically significant at probabilities ranging from 2E -6 to 8E -17. These results mean that such high correlation could not have occurred merely by a chance. One can expect or predict that candidates who obtain high scores on the clinical written examination will perform well on oral examinations. The correlation coefficients between the physics written examination and the 8 oral examination categories were also statistically significant, with probabilities ranging from to 5E -24. The same inference can be drawn on the predictability of the physics written test on the oral examinations: that these relationships could not have been the result of chance occurrence and that candidates who perform well on the physics written test will perform well on the oral examinations. Similarly, the biology written examination was a strong predictor of performance on the oral examinations, with probabilities ranging from to 2E -6. Intercorrelation among the 8 oral examination categories was statistically significant also, with probabilities ranging from to 3E -11, which suggests that candidates who perform well on one oral examination category will show successful performances on the other oral examination categories. Table 2 exhibits the detailed correlation coefficients for reference. Discussion Reliability of oral examinations 77 The fact that many correlation coefficients ranged from to suggests that unique factors existed in each form of examination; that is, the oral examinations and the written examinations measured some aspects of knowledge, skill, or judgment that were not measured in the other form of examination. If a correlation coefficient between a written test and oral examination approaches 1.0, it means the tests are probably measuring the same aspects or attributes of the criteria. Although statistically significant at P =.05, the correlation coefficient between the breast category oral examination and the RO physics written test correlated less highly (P =.028), and the same was true for the correlation coefficient between the breast category oral examination and the lymphoma/leukemia/ transplant/myeloma category oral examination (P =.034). No plausible explanation is certain. However, one possible explanation may be that the reliability of the breast category is lower than the other categories, possibly because all the regular breast category examiners were first-time examiners. There were 2 relievers who were experienced but each examined fewer candidates. As mentioned above, there are concerns in the literature about the reliability and validity of the oral examination because of its potential bias, effects of interpersonal Table 2 Intercorrelation among radiation oncology oral examination categories and computerized ( written ) test scores (n = 152) Category Computerized test Oral examination Clinical Physics Biology Rater 1 GI Rater 2 Gyn Rater 3 GU Rater 4 Lymph Rater 5 HNS Rater 6 Breast Rater 7 CNS/Peds Rater 8 Lung Clinical Pearson r Probability Physics Pearson r Probability Biology Pearson r Probability GI Pearson r Probability Gyn Pearson r Probability GU Pearson r Probability Lymph Pearson r Probability HNS Pearson r Probability Breast Pearson r Probability CNS/PEDS Pearson r Probability Lung Pearson r 1 Probability The probability was for 2-tailed tests. CNS/PEDS, central nervous system/pediatrics; GI, gastrointestinal; GU, genitourinary; GYN, gynecology; HNS, head/neck/skin; Lymph, lymphoma/ leukemia/transplant/myeloma.
5 78 J.C. Yang et al Practical Radiation Oncology: January-March 2013 communication skills, and possible subjectivity. However, some research studies dispute some of the concerns. In this study, each candidate was examined by 8 content experts for a period of minutes each. The raters scored candidates' performances independently. The inter-rater reliability of the May 2010 ABR RO certifying oral examination was Considering the lack of extensive systematic training, this is remarkably high for an oral examination administered by 8 content expert examiners. The reliability coefficient is higher than what might be commonly perceived or expected as concerns expressed in the literature and mentioned in the introduction of this paper. 1-5 However, more systematic examiner training before the beginning of the oral examination may further increase the consistency in the rater evaluations of the candidates' performances. Training for standardizing cases and questions may also help increase the consistency in grading. In addition, a variability in scores greater than the range of 68 to 72 would probably increase the reliability coefficient. The reliability of each category can affect the inter-rater reliability. If the reliability of a content category is low it will not correlate highly with other content categories. Perhaps the reliability of the breast category may be suspect because the correlation coefficient between the breast category and lymphoma/leukemia/transplant/ myeloma category (P =.028), as well as the correlation coefficient between the breast and physics categories (P =.034), were not as high as correlation coefficients between the other content categories, including the written test scores. It was not possible to determine one category's reliability coefficient by itself in this study. Though arbitrary, perhaps, a scale with 5-point intervals could be designed. The scale could be developed with scoring guidelines that may be familiar to the rater and are easy to follow. The trustees could be involved in the decision-making process regarding the development of the scoring rubric, especially whether or not they would allow the rater to assign a score within a 5-point interval scale with the wider range of points; for example, from 65 to 85 or 70 to 100, etc, with specific guidelines for each interval. A systematic method to increase consistency among the raters could be implemented by providing systematic training to standardize cases and questions. Each minute oral examination could be audiotaped by the content category. The audiotapes of each content category could be transcribed and analyzed and then categorized by the questions asked by the examiners. This method could produce the blueprint for the oral examination in each category. During the process of developing the blueprint, a scale with 5-point intervals could be implemented instead of a 1-point interval scale (although this will have no effect on developing the oral examination blueprints ). Although the reliability coefficient of the ABR May 2010 oral examination was remarkably high, as with any organization administering high-stakes examinations, the ABR must continue to strive for high reliability, and thus validity, of their examinations to protect candidates and the public. Reliability is a necessary condition for validity, but not a sufficient condition for validity. Currently, the associate executive director of radiation oncology is working on a revision of the case formats to better rationalize oral examination content in order to improve the reliability and thus validity of the radiation oncology oral examinations. References 1. Memon AM, Joughin GR, Memon B. Oral assessment and postgraduate medical examinations: Establishing conditions for validity, reliability and fairness. Adv Health Sci Educ Theory Pract. 2010;15: Lunz ME, Bashook PG. Relationship between candidate communication ability and oral certification examination scores. Med Educ. 2008;42: U.S. Congress. The Court Interpreters Act of 1978, PL USC Washington, DC: United States Congress Steinberg PI. Oral examination anxiety in physicians, narcissism, and object relations. J Appl Psychoanal Stud. 2002;4: Stansfield CW, Hewitt WE. Examining the predictive validity of a screening test for court interpreters. Lang Test. 2005;22: Pascuzzi RM. Opinion/education: The ABPN is the neurology resident's best friend. Neurology. 2008;70:e16-e Brennan RL, ed. Educational Measurement, 4th ed. American Council on Education, Westport, CT: Praeger Publishers; 2006;65-110; Brennan RL. Generalizability Theory. 4th ed. New York, NY: Springer-Verlag; Feldt LS, Charter RA. A simple computational alternative to analysis of variance formulas for estimating the k-judge reliability. Psychol Rep. 2004;94:
Survey Question. What are appropriate methods to reaffirm the fairness, validity reliability and general performance of examinations?
Clause 9.3.5 Appropriate methodology and procedures (e.g. collecting and maintaining statistical data) shall be documented and implemented in order to affirm, at justified defined intervals, the fairness,
More informationCatching the Hawks and Doves: A Method for Identifying Extreme Examiners on Objective Structured Clinical Examinations
Catching the Hawks and Doves: A Method for Identifying Extreme Examiners on Objective Structured Clinical Examinations July 20, 2011 1 Abstract Performance-based assessments are powerful methods for assessing
More informationRegistered Radiologist Assistant (R.R.A. ) 2016 Examination Statistics
Registered Radiologist Assistant (R.R.A. ) Examination Statistics INTRODUCTION This report summarizes the results of the Registered Radiologist Assistant (R.R.A. ) examinations developed and administered
More informationAnalysis of Confidence Rating Pilot Data: Executive Summary for the UKCAT Board
Analysis of Confidence Rating Pilot Data: Executive Summary for the UKCAT Board Paul Tiffin & Lewis Paton University of York Background Self-confidence may be the best non-cognitive predictor of future
More informationCHAPTER III RESEARCH METHODOLOGY
CHAPTER III RESEARCH METHODOLOGY In this chapter, the researcher will elaborate the methodology of the measurements. This chapter emphasize about the research methodology, data source, population and sampling,
More informationUsing Analytical and Psychometric Tools in Medium- and High-Stakes Environments
Using Analytical and Psychometric Tools in Medium- and High-Stakes Environments Greg Pope, Analytics and Psychometrics Manager 2008 Users Conference San Antonio Introduction and purpose of this session
More informationExamining Factors Affecting Language Performance: A Comparison of Three Measurement Approaches
Pertanika J. Soc. Sci. & Hum. 21 (3): 1149-1162 (2013) SOCIAL SCIENCES & HUMANITIES Journal homepage: http://www.pertanika.upm.edu.my/ Examining Factors Affecting Language Performance: A Comparison of
More informationUNIVERSITY OF CALGARY. Reliability & Validity of the. Objective Structured Clinical Examination (OSCE): A Meta-Analysis. Ibrahim Al Ghaithi A THESIS
UNIVERSITY OF CALGARY Reliability & Validity of the Objective Structured Clinical Examination (OSCE): A Meta-Analysis by Ibrahim Al Ghaithi A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL
More informationThe Impact of Statistically Adjusting for Rater Effects on Conditional Standard Errors for Performance Ratings
0 The Impact of Statistically Adjusting for Rater Effects on Conditional Standard Errors for Performance Ratings Mark R. Raymond, Polina Harik and Brain E. Clauser National Board of Medical Examiners 1
More informationRESEARCH ARTICLES. Brian E. Clauser, Polina Harik, and Melissa J. Margolis National Board of Medical Examiners
APPLIED MEASUREMENT IN EDUCATION, 22: 1 21, 2009 Copyright Taylor & Francis Group, LLC ISSN: 0895-7347 print / 1532-4818 online DOI: 10.1080/08957340802558318 HAME 0895-7347 1532-4818 Applied Measurement
More informationDetection of Differential Test Functioning (DTF) and Differential Item Functioning (DIF) in MCCQE Part II Using Logistic Models
Detection of Differential Test Functioning (DTF) and Differential Item Functioning (DIF) in MCCQE Part II Using Logistic Models Jin Gong University of Iowa June, 2012 1 Background The Medical Council of
More informationAMERICAN BOARD OF SURGERY 2009 IN-TRAINING EXAMINATION EXPLANATION & INTERPRETATION OF SCORE REPORTS
AMERICAN BOARD OF SURGERY 2009 IN-TRAINING EXAMINATION EXPLANATION & INTERPRETATION OF SCORE REPORTS Attached are the performance reports and analyses for participants from your surgery program on the
More informationChapter 3. Psychometric Properties
Chapter 3 Psychometric Properties Reliability The reliability of an assessment tool like the DECA-C is defined as, the consistency of scores obtained by the same person when reexamined with the same test
More informationGlobal Perspective Inventory (GPI) Report
Global Perspective Inventory (GPI) 2012-2013 Report Executive Summary display higher levels of global competence than freshmen in all of the GPI scales except for the interpersonal social responsibility
More informationCOMPUTING READER AGREEMENT FOR THE GRE
RM-00-8 R E S E A R C H M E M O R A N D U M COMPUTING READER AGREEMENT FOR THE GRE WRITING ASSESSMENT Donald E. Powers Princeton, New Jersey 08541 October 2000 Computing Reader Agreement for the GRE Writing
More informationIntroduction to Reliability
Reliability Thought Questions: How does/will reliability affect what you do/will do in your future job? Which method of reliability analysis do you find most confusing? Introduction to Reliability What
More informationalternate-form reliability The degree to which two or more versions of the same test correlate with one another. In clinical studies in which a given function is going to be tested more than once over
More informationChapter 11. Experimental Design: One-Way Independent Samples Design
11-1 Chapter 11. Experimental Design: One-Way Independent Samples Design Advantages and Limitations Comparing Two Groups Comparing t Test to ANOVA Independent Samples t Test Independent Samples ANOVA Comparing
More informationGENERALIZABILITY AND RELIABILITY: APPROACHES FOR THROUGH-COURSE ASSESSMENTS
GENERALIZABILITY AND RELIABILITY: APPROACHES FOR THROUGH-COURSE ASSESSMENTS Michael J. Kolen The University of Iowa March 2011 Commissioned by the Center for K 12 Assessment & Performance Management at
More informationLecture 4: Research Approaches
Lecture 4: Research Approaches Lecture Objectives Theories in research Research design approaches ú Experimental vs. non-experimental ú Cross-sectional and longitudinal ú Descriptive approaches How to
More informationEducation, Training, and Certification of Radiation Oncologists in Indonesia
Education, Training, and Certification of Radiation Oncologists in Indonesia with acknowledgement to Prof HM Djakaria (Indonesian College of Radiation Oncology) Outline Historical Context Current Status
More informationComparing Vertical and Horizontal Scoring of Open-Ended Questionnaires
A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to
More informationTHE ANGOFF METHOD OF STANDARD SETTING
THE ANGOFF METHOD OF STANDARD SETTING May 2014 1400 Blair Place, Suite 210, Ottawa ON K1J 9B8 Tel: (613) 237-0241 Fax: (613) 237-6684 www.asinc.ca 1400, place Blair, bureau 210, Ottawa (Ont.) K1J 9B8 Tél.
More informationAssessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination
Educational Research Article Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination Sara Mortaz Hejri 1, Mohammad Jalili
More informationDriving Success: Setting Cut Scores for Examination Programs with Diverse Item Types. Beth Kalinowski, MBA, Prometric Manny Straehle, PhD, GIAC
Driving Success: Setting Cut Scores for Examination Programs with Diverse Item Types Beth Kalinowski, MBA, Prometric Manny Straehle, PhD, GIAC Objectives 1 2 3 4 Describe the process for implementing a
More informationInvestigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education.
The Reliability of PLATO Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO M. Ken Cor Stanford University School of Education April,
More informationValidity, Reliability, and Defensibility of Assessments in Veterinary Education
Instructional Methods Validity, Reliability, and Defensibility of Assessments in Veterinary Education Kent Hecker g Claudio Violato ABSTRACT In this article, we provide an introduction to and overview
More informationThe Effect of Review on Student Ability and Test Efficiency for Computerized Adaptive Tests
The Effect of Review on Student Ability and Test Efficiency for Computerized Adaptive Tests Mary E. Lunz and Betty A. Bergstrom, American Society of Clinical Pathologists Benjamin D. Wright, University
More informationGUIDELINES FOR POST PEDIATRICS PORTAL PROGRAM
GUIDELINES FOR POST PEDIATRICS PORTAL PROGRAM Psychiatry is a medical specialty that is focused on the prevention, diagnosis, and treatment of mental, addictive, and emotional disorders throughout the
More informationTHESE STANDARDS ARE DORMANT.
THESE STANDARDS ARE DORMANT. TO INQUIRE ABOUT AOBA PARTICIPATION IN THE CAQ FOR CRITICAL CARE MEDICINE FOR ANESTHESIOLOGY PLEASE CONTACT THE AOBA aoba@osteopathic.org. BASIC STANDARDS FOR SUBSPECIALTY
More informationLinking Assessments: Concept and History
Linking Assessments: Concept and History Michael J. Kolen, University of Iowa In this article, the history of linking is summarized, and current linking frameworks that have been proposed are considered.
More informationACRO Scope of Practice Document
ACRO Scope of Practice Document I. ORGANIZATIONAL AND LEGAL STATEMENT The American College of Radiation Oncology (ACRO) is a nonprofit professional organization whose primary purposes are to advance the
More informationPSYCHOMETRIC PROPERTIES OF CLINICAL PERFORMANCE RATINGS
PSYCHOMETRIC PROPERTIES OF CLINICAL PERFORMANCE RATINGS A total of 7931 ratings of 482 third- and fourth-year medical students were gathered over twelve four-week periods. Ratings were made by multiple
More informationMayo Clinic Gynecologic Oncology Fellowship (Minnesota) Competency-based goals
Mayo Clinic Gynecologic Oncology Fellowship (Minnesota) Competency-based goals 1. PATIENT CARE (includes surgical skills) To train gynecologic oncology fellows to competency in evaluation, treatment and
More informationItem Analysis Explanation
Item Analysis Explanation The item difficulty is the percentage of candidates who answered the question correctly. The recommended range for item difficulty set forth by CASTLE Worldwide, Inc., is between
More informationLANGUAGE TEST RELIABILITY On defining reliability Sources of unreliability Methods of estimating reliability Standard error of measurement Factors
LANGUAGE TEST RELIABILITY On defining reliability Sources of unreliability Methods of estimating reliability Standard error of measurement Factors affecting reliability ON DEFINING RELIABILITY Non-technical
More informationThe AAO- HNS s position statement on Point- of- Care Imaging in Otolaryngology states that the AAO- HNS,
AAO- HNS Statement on Diagnostic Imaging Reimbursement for Otolaryngologist Head and Neck Surgeons (September 2014) The American Academy of Otolaryngology Head and Neck Surgery (AAO- HNS), with approximately
More informationCHAPTER VI RESEARCH METHODOLOGY
CHAPTER VI RESEARCH METHODOLOGY 6.1 Research Design Research is an organized, systematic, data based, critical, objective, scientific inquiry or investigation into a specific problem, undertaken with the
More informationBasic Standards for Fellowship Training in Sleep Medicine
Basic Standards for Fellowship Training in Sleep Medicine American Osteopathic Association and American College of Osteopathic Neurologists and Psychiatrists and American College of Osteopathic Internists
More informationSpecific Standards of Accreditation for Residency Programs in Adult and Pediatric Neurology
Specific Standards of Accreditation for Residency Programs in Adult and Pediatric Neurology INTRODUCTION 2011 A university wishing to have an accredited program in adult Neurology must also sponsor an
More informationImportance of Good Measurement
Importance of Good Measurement Technical Adequacy of Assessments: Validity and Reliability Dr. K. A. Korb University of Jos The conclusions in a study are only as good as the data that is collected. The
More informationMeasurement and Descriptive Statistics. Katie Rommel-Esham Education 604
Measurement and Descriptive Statistics Katie Rommel-Esham Education 604 Frequency Distributions Frequency table # grad courses taken f 3 or fewer 5 4-6 3 7-9 2 10 or more 4 Pictorial Representations Frequency
More informationConstruct Reliability and Validity Update Report
Assessments 24x7 LLC DISC Assessment 2013 2014 Construct Reliability and Validity Update Report Executive Summary We provide this document as a tool for end-users of the Assessments 24x7 LLC (A24x7) Online
More informationAnalysis of the Reliability and Validity of an Edgenuity Algebra I Quiz
Analysis of the Reliability and Validity of an Edgenuity Algebra I Quiz This study presents the steps Edgenuity uses to evaluate the reliability and validity of its quizzes, topic tests, and cumulative
More informationTHE PSYCHOMETRICS OF PERFORMANCE ASSESSMENTS A PHYSICIAN S PERSPECTIVE
THE PSYCHOMETRICS OF PERFORMANCE ASSESSMENTS A PHYSICIAN S PERSPECTIVE Jeanne M. Sandella, DO Objectives Discuss the reliability of performance assessments why are they used, what are the threats to having
More informationEstablishing Interrater Agreement for Scoring Criterion-based Assessments: Application for the MEES
Establishing Interrater Agreement for Scoring Criterion-based Assessments: Application for the MEES Mark C. Hogrebe Washington University in St. Louis MACTE Fall 2018 Conference October 23, 2018 Camden
More informationCLINICAL RADIATION SCIENCES (CLRS)
Clinical Radiation Sciences (CLRS) 1 CLINICAL RADIATION SCIENCES (CLRS) CLRS 101. Introduction to Clinical Radiologic Sciences. 1 Hour. Semester course; 1 lecture hour. 1 credit. Presentation and discussion
More informationCHAPTER NINE DATA ANALYSIS / EVALUATING QUALITY (VALIDITY) OF BETWEEN GROUP EXPERIMENTS
CHAPTER NINE DATA ANALYSIS / EVALUATING QUALITY (VALIDITY) OF BETWEEN GROUP EXPERIMENTS Chapter Objectives: Understand Null Hypothesis Significance Testing (NHST) Understand statistical significance and
More informationDevelopment, administration, and validity evidence of a subspecialty preparatory test toward licensure: a pilot study
Johnson et al. BMC Medical Education (2018) 18:176 https://doi.org/10.1186/s12909-018-1294-z RESEARCH ARTICLE Open Access Development, administration, and validity evidence of a subspecialty preparatory
More informationCollecting & Making Sense of
Collecting & Making Sense of Quantitative Data Deborah Eldredge, PhD, RN Director, Quality, Research & Magnet Recognition i Oregon Health & Science University Margo A. Halm, RN, PhD, ACNS-BC, FAHA Director,
More informationBasic Standards for Residency Training in General Neurology
Basic Standards for Residency Training in General Neurology American Osteopathic Association and American College of Osteopathic Neurologists and Psychiatrists Revised 2/2003 Revised 7/2004 Revised 6/2006
More informationPsychometrics for Beginners. Lawrence J. Fabrey, PhD Applied Measurement Professionals
Psychometrics for Beginners Lawrence J. Fabrey, PhD Applied Measurement Professionals Learning Objectives Identify key NCCA Accreditation requirements Identify two underlying models of measurement Describe
More informationBasic Standards for Residency Training in Child and Adolescent Neurology
Basic Standards for Residency Training in Child and Adolescent Neurology American Osteopathic Association and American College of Osteopathic Neurologists and Psychiatrists BOT 3/1985 Revised 03/1992 Revised
More information02a: Test-Retest and Parallel Forms Reliability
1 02a: Test-Retest and Parallel Forms Reliability Quantitative Variables 1. Classic Test Theory (CTT) 2. Correlation for Test-retest (or Parallel Forms): Stability and Equivalence for Quantitative Measures
More informationInterpreting the Item Analysis Score Report Statistical Information
Interpreting the Item Analysis Score Report Statistical Information This guide will provide information that will help you interpret the statistical information relating to the Item Analysis Report generated
More informationMedical Students Judgments of Mind and Brain in the Etiology and Treatment of Psychiatric Disorders. A Pilot Study
Medical Students Judgments of Mind and Brain in the Etiology and Treatment of Psychiatric Disorders A Pilot Study Michael A. Brog, M.D. Karen A. Guskin, Ph.D. Given the importance of how medical students
More informationDetecting Suspect Examinees: An Application of Differential Person Functioning Analysis. Russell W. Smith Susan L. Davis-Becker
Detecting Suspect Examinees: An Application of Differential Person Functioning Analysis Russell W. Smith Susan L. Davis-Becker Alpine Testing Solutions Paper presented at the annual conference of the National
More informationValidity refers to the accuracy of a measure. A measurement is valid when it measures what it is suppose to measure and performs the functions that
Validity refers to the accuracy of a measure. A measurement is valid when it measures what it is suppose to measure and performs the functions that it purports to perform. Does an indicator accurately
More informationPRINCIPLES OF STATISTICS
PRINCIPLES OF STATISTICS STA-201-TE This TECEP is an introduction to descriptive and inferential statistics. Topics include: measures of central tendency, variability, correlation, regression, hypothesis
More informationAssociation Between Geographic Concentration Of Chiropractors In 2008 And Circulatory Disease Death Rates In 2009
ISPUB.COM The Internet Journal of Chiropractic Volume 4 Number 1 Association Between Geographic Concentration Of Chiropractors In 2008 And Circulatory Disease Death Rates In 2009 J Hart Citation J Hart.
More informationMCAS Equating Research Report: An Investigation of FCIP-1, FCIP-2, and Stocking and. Lord Equating Methods 1,2
MCAS Equating Research Report: An Investigation of FCIP-1, FCIP-2, and Stocking and Lord Equating Methods 1,2 Lisa A. Keller, Ronald K. Hambleton, Pauline Parker, Jenna Copella University of Massachusetts
More informationThe Short NART: Cross-validation, relationship to IQ and some practical considerations
British journal of Clinical Psychology (1991), 30, 223-229 Printed in Great Britain 2 2 3 1991 The British Psychological Society The Short NART: Cross-validation, relationship to IQ and some practical
More informationA National Study of the Certified Diabetes Educator: Report on a Job Analysis Conducted by the National Certification Board for Diabetes Educators
A National Study of the Certified Diabetes Educator: Report on a Job Analysis Conducted by the National Certification Board for Diabetes Educators John Zrebiec, MSW, CDE Abstract Purpose. The job analysis
More informationA Brief Guide to Writing
Writing Workshop WRITING WORKSHOP BRIEF GUIDE SERIES A Brief Guide to Writing Psychology Papers and Writing Psychology Papers Analyzing Psychology Studies Psychology papers can be tricky to write, simply
More informationSchool orientation and mobility specialists School psychologists School social workers Speech language pathologists
2013-14 Pilot Report Senate Bill 10-191, passed in 2010, restructured the way all licensed personnel in schools are supported and evaluated in Colorado. The ultimate goal is ensuring college and career
More informationRADIOLOGIC TECHNOLOGY (526)
RADIOLOGIC TECHNOLOGY (526) 526-133 DMS General Procedures 2 Radiologic Technology (526) 1 526-130 Introduction to Diagnostic Medical Sonography This course introduces the student to the history of ultrasound
More informationSupporting Information
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 Supporting Information Variances and biases of absolute distributions were larger in the 2-line
More informationThe annual promotion assessment consists of the following components: a. Written examination b. Continuous Assessment
SAUDI FELLOWSHIP TRAINING PROGRAM PAEDIATRIC INTENSIVE CARE Promotion Examination 2019 Objectives: The general objective of the annual promotion assessment is to evaluate that the trainee has satisfactorily
More informationValidation of an Analytic Rating Scale for Writing: A Rasch Modeling Approach
Tabaran Institute of Higher Education ISSN 2251-7324 Iranian Journal of Language Testing Vol. 3, No. 1, March 2013 Received: Feb14, 2013 Accepted: March 7, 2013 Validation of an Analytic Rating Scale for
More informationCraft Personality Questionnaire
Craft Personality Questionnaire Evidence of Reliability and Validity 888-298-6227 TalentLens.com Copyright 2008 Pearson Education, Inc., or its affiliates. All rights reserved. Copyright 2008 by Pearson
More informationPractice Analysis and Content Specifications. for Registered Radiologist Assistant
Practice Analysis and Content Specifications for Registered Radiologist Assistant Final Report For New Documents Implemented July 2018 The American Registry of Radiologic Technologists 1255 Northland Drive
More informationControlled Experiments
CHARM Choosing Human-Computer Interaction (HCI) Appropriate Research Methods Controlled Experiments Liz Atwater Department of Psychology Human Factors/Applied Cognition George Mason University lizatwater@hotmail.com
More informationThe Personal Profile System 2800 Series Research Report
The Personal Profile System 2800 Series Research Report The Personal Profile System 2800 Series Research Report Item Number: O-255 1996 by Inscape Publishing, Inc. All rights reserved. Copyright secured
More informationCLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION IN RADIATION THERAPY (SECOND MODALITY)
Clinical Radiation Sciences, Bachelor of Science (B.S.) with a concentration in radiation therapy (second modality) 1 CLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION IN RADIATION
More informationReliability Study of ACTFL OPIc in Spanish, English, and Arabic for the ACE Review
Reliability Study of ACTFL OPIc in Spanish, English, and Arabic for the ACE Review Prepared for: American Council on the Teaching of Foreign Languages (ACTFL) White Plains, NY Prepared by SWA Consulting
More informationRegions Hospital Delineation of Privileges Radiation Oncology
Regions Hospital Delineation of Privileges Radiation Oncology Applicant s Last First M. Instructions: Place a check-mark where indicated for each core group you are requesting. Review education and basic
More informationData and Statistics 101: Key Concepts in the Collection, Analysis, and Application of Child Welfare Data
TECHNICAL REPORT Data and Statistics 101: Key Concepts in the Collection, Analysis, and Application of Child Welfare Data CONTENTS Executive Summary...1 Introduction...2 Overview of Data Analysis Concepts...2
More informationDiscrimination Weighting on a Multiple Choice Exam
Proceedings of the Iowa Academy of Science Volume 75 Annual Issue Article 44 1968 Discrimination Weighting on a Multiple Choice Exam Timothy J. Gannon Loras College Thomas Sannito Loras College Copyright
More informationThe Milestones are designed only for use in evaluation
The Neurology Milestone Project The Milestones are designed only for use in evaluation of resident physicians in the context of their participation in Accreditation Council for Graduate Medical Education
More informationConformity. Jennifer L. Flint. The University of Southern Mississippi
The modified Asch task: The Relationship between Actual Conformity and Self-Reported Conformity Jennifer L. Flint The University of Southern Mississippi 2 The modified Asch task: The Relationship between
More informationGlobal Perspective Inventory (GPI) - Pilot Report
Global Perspective Inventory (GPI) - Pilot 2010-11 Report Introduction The Global Perspectives Inventory is a nationally recognized instrument designed to measure a student s global perspective. The GPI
More informationCLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION IN NUCLEAR MEDICINE TECHNOLOGY (SECOND MODALITY)
Clinical Radiation Sciences, Bachelor of Science (B.S.) with a concentration in nuclear medicine technology (second modality) CLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION
More informationEnglish 10 Writing Assessment Results and Analysis
Academic Assessment English 10 Writing Assessment Results and Analysis OVERVIEW This study is part of a multi-year effort undertaken by the Department of English to develop sustainable outcomes assessment
More informationCHAPTER III RESEARCH METHODOLOGY
CHAPTER III RESEARCH METHODOLOGY Research methodology explains the activity of research that pursuit, how it progress, estimate process and represents the success. The methodological decision covers the
More informationMcCloskey Executive Functions Scale (MEFS) Report
McCloskey Executive Functions Scale (MEFS) Report Student's Name: Amy Date of Birth: Gender: Female Date of Ratings: Grade: 7 Age: 3 years, month School: Middle Rater s Name: Teacher Executive Functions
More informationMeasures of Dispersion. Range. Variance. Standard deviation. Measures of Relationship. Range. Variance. Standard deviation.
Measures of Dispersion Range Variance Standard deviation Range The numerical difference between the highest and lowest scores in a distribution It describes the overall spread between the highest and lowest
More informationA framework for predicting item difficulty in reading tests
Australian Council for Educational Research ACEReSearch OECD Programme for International Student Assessment (PISA) National and International Surveys 4-2012 A framework for predicting item difficulty in
More informationFigure 1: Design and outcomes of an independent blind study with gold/reference standard comparison. Adapted from DCEB (1981b)
Page 1 of 1 Diagnostic test investigated indicates the patient has the Diagnostic test investigated indicates the patient does not have the Gold/reference standard indicates the patient has the True positive
More informationTHE AMERICAN BOARD OF ORTHODONTICS SCENARIO BASED ORAL CLINICAL EXAMINATION STUDY GUIDE
THE AMERICAN BOARD OF ORTHODONTICS SCENARIO BASED ORAL CLINICAL EXAMINATION STUDY GUIDE 2019 The American Board of Orthodontics 401 North Lindbergh Blvd., Suite 300 St. Louis, Missouri 63141 314 432 6130
More informationSurvey Research. We can learn a lot simply by asking people what we want to know... THE PREVALENCE OF SURVEYS IN COMMUNICATION RESEARCH
Survey Research From surveys we can learn how large groups of people think and act. To trust generalizations made on the basis of surveys, however, the sample must be representative, the response rate
More informationPTHP 7101 Research 1 Chapter Assignments
PTHP 7101 Research 1 Chapter Assignments INSTRUCTIONS: Go over the questions/pointers pertaining to the chapters and turn in a hard copy of your answers at the beginning of class (on the day that it is
More informationCollecting & Making Sense of
Collecting & Making Sense of Quantitative Data Deborah Eldredge, PhD, RN Director, Quality, Research & Magnet Recognition i Oregon Health & Science University Margo A. Halm, RN, PhD, ACNS-BC, FAHA Director,
More informationCLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION IN RADIOGRAPHY (DEGREE COMPLETION)
Clinical Radiation Sciences, Bachelor of Science (B.S.) with a concentration in radiography (degree completion) 1 CLINICAL RADIATION SCIENCES, BACHELOR OF SCIENCE (B.S.) WITH A CONCENTRATION IN RADIOGRAPHY
More informationA review of statistical methods in the analysis of data arising from observer reliability studies (Part 11) *
A review of statistical methods in the analysis of data arising from observer reliability studies (Part 11) * by J. RICHARD LANDIS** and GARY G. KOCH** 4 Methods proposed for nominal and ordinal data Many
More informationUsing Generalizability Theory to Investigate the Psychometric Property of an Assessment Center in Indonesia
Using Generalizability Theory to Investigate the Psychometric Property of an Assessment Center in Indonesia Urip Purwono received his Ph.D. (psychology) from the University of Massachusetts at Amherst,
More informationBasic Standards for Osteopathic Fellowship Training in Sleep Medicine
Basic Standards for Osteopathic Fellowship Training in Sleep Medicine American Osteopathic Association and the American College of Osteopathic Neurologists and Psychiatrists and the American College of
More informationUNIVERSITY OF SOUTH ALABAMA RADIOLOGIC SCIENCES
UNIVERSITY OF SOUTH ALABAMA RADIOLOGIC SCIENCES 1 Radiologic Sciences RAD 300 Clinical Education I 2 cr Hospital-based laboratory allowing the student additional clinical experience in RAD 312 and RAD
More informationVARIABLES AND MEASUREMENT
ARTHUR SYC 204 (EXERIMENTAL SYCHOLOGY) 16A LECTURE NOTES [01/29/16] VARIABLES AND MEASUREMENT AGE 1 Topic #3 VARIABLES AND MEASUREMENT VARIABLES Some definitions of variables include the following: 1.
More information