Communication Skills in Standardized-Patient Assessment of Final-Year Medical Students: A Psychometric Study
|
|
- Annabel Harrington
- 5 years ago
- Views:
Transcription
1 Advances in Health Sciences Education 9: , Kluwer Academic Publishers. Printed in the Netherlands. 179 Communication Skills in Standardized-Patient Assessment of Final-Year Medical Students: A Psychometric Study GRETCHEN GUITON 1, CAROL S. HODGSON 2, GINETTE DELANDSHERE 3 and LUANN WILKERSON 1 1 David Geffen School of Medicine at University of California, Los Angeles; 2 Department of Education, Indiana University; 3 University of California, San Francisco ( Author for correspondence, Center for Educational Development and Research, UCLA, Box , Los Angeles, CA , USA; gguiton@mednet.ucla.edu) Abstract. The purpose of this study is to investigate the content-specificity of communication skills. It investigates the reliability and dimensionality of standardized patient (SP) ratings of communication skills in an Objective Structured Clinical Examination (OSCE) for final year medical students. An OSCE consisting of seven standardized patient (SP) encounters was administered to final-year medical students at four medical schools that are members of the California Consortium for the Assessment of Clinical Competence (N = 567). For each case, SPs rated students communication skills on the same seven items. Internal consistency coefficients were calculated and a two-facet generalizability study was performed to investigate the reliability of the scores. An exploratory factor analysis was conducted to examine the dimensionality of the exam. Findings indicate that communication skills across the seven-case examination demonstrate a reliable generic component that supports relative decision making, but that a significant case-by-student interaction exists. The underlying structure further supports the case-specific nature of students ability to communicate with patients. From these findings, it is evident that individual s communication skills vary systematically with specific cases. Implications include the need to consider the range of communication skill demands made across the OSCE to support generalization of findings, the need for instruction to provide feedback on communication skills in multiple contexts, and the need for research to further examine the student, patient, and presenting problem as sources of variation in communication skills. Key words: communication skills, generalizability theory, medical students, performance assessment, psychometrics, standardized patients Introduction In a qualitative analysis of narrative comments written by standardized patients (SPs) during an objective structured clinical examination (OSCE), Wilkerson and Rose (2001) noted an unexpected variability in individual student s communication skills from patient to patient. At one station the SP would comment on the student s ability to put him at ease while at the next station, the SP would describe the student as inattentive and distracted. This pattern of inconsistency was demonstrated by one-third of the 40 students included in the study. A review of other communication
2 180 GRETCHEN GUITON ET AL. Table I. Characteristics of cases in the OSCE, 2001 Presenting complaint Age Sex Exam components Abdominal pain 22 F History, physical exam, communication Chronic sore throat 36 F History, physical exam, communication Chest pain 49 F History, physical exam, information sharing, communication Routine check up 50 M History, information sharing, communication Follow-up visit for 60 M History, physical exam, information sharing, communication diabetes Cough and fatigue 60 M History, physical exam, information sharing, communication Recent weight loss 76 F History, physical exam, information sharing, communication skills studies (Hodges, 1996; Cohen et al., 1997; Colliver et al., 1998; Colliver et al., 1999; Donnelly et al., 2000; Blue et al., 2000) revealed a recurrent theme in the discussion of the results. The authors raised the question of content-specificity in communication skills similar to that already known to exist in other problemsolving skills (Shulman et al., 1975). The recognition of content specificity in areas such as history taking and physical exam skills has altered our instruction and assessment; yet we treat communication as a unidimensional skill set that can be taught separately from content and assessed as such. If the mounting evidence refutes this unidimensionality, both our teaching and assessment methods will be called into question. The current study extends this line of investigation into the content-specificity of communication skills. We examine the psychometric characteristics of SP ratings of communicative skills, including reliability and dimensionality, in an OSCE for final year medical students. Method A multi-station clinical performance examination administered to 547 final-year medical students attending four of seven medical schools in the California Consortium for the Assessment of Clinical Competence provided data for the study. The multi-station examination consisted of seven standardized patient (SP) encounters (six of 15 minutes and one of 37 minutes) in which students were instructed to perform focused patient workups with attention to skills in physician-patient communication and information sharing (e.g., treatment plan). The seven cases, developed by Consortium faculty, represented a mix of acute, chronic, well-care, behavioral, and ill-defined problems. A description of each case is provided in Table I. Students at each campus completed the half-day exam over a two- to three-week period. Standardized patients across the Consortium received 16 hours of training in presentation of the case and use of the checklist for scoring. SPs entered their
3 A PSYCHOMETRIC STUDY 181 ratings immediately following the clinical encounter. Schools varied in the mode of entry, using on-line or op-scannable forms. Schools also varied in the number of SPs trained to play the role of the patient in each case. Across the four schools, a total of 73 SP s were involved in the seven cases. Three of the schools used two SPs for each case, while the fourth school used four to six SPs for each case. On average, each SP rated 43 students (actual numbers ranged from 7 to 86). No SP was involved in more than a single case or school, so that rater effects are confounded with student, case, and school. Students were assigned to SPs based on the day and time selected for administration, which occurred non-systematically. Specific physician-patient communication items, based on the work of Forrest Lang (Bayer-Fetzer, 2001), were used to evaluate students. For each case, SPs rated students communication skills on the same seven items ( appeared professionally competent, effectively gathered information, listened actively, established professional rapport, appropriately explored my perspective, addressed my feelings, met my needs ) using a 6-point Likert-type rating scale ranging from outstanding to unacceptable. Using the same scale, SPs provided a rating of their overall satisfaction with this student encounter. In addition, SPs completed specific case-related items assessing history taking, physical examination, information sharing, and clinical courtesy and provided narrative comments at the end of the checklist in response to an open-ended request to write positive comments for this student and constructive comments for this student. To assess reliability, we calculated Cronbach s alpha for the physician-patient communication ratings in a number of ways based on how data are reported by the schools. For instance, data are reported as the sum of all communication items (i.e., across cases) and students are identified normatively for remediation. For this reason, we investigated whether a general dimension of communication skills was evident and would support the use of a total score. We compared within case reliability and cross-case reliability on individual items to investigate the content specificity of communication skills as well. Finally we estimated the intra-class correlation coefficient as a measure of the reliability of a single item from an individual case. We performed a generalizability analysis with students as the object of measurement to investigate the sources of variation in communication scores both in case and total scores. We estimated six variance components using SPSS, GLM: students, cases, items, students by cases, students by items, and cases by items. The three-way interaction was confounded with the error term along with other factors not estimated in the model (e.g., SP and school effects). Using these variance components, we computed a generalizability coefficient to estimate the reliability of the total communications score for relative decisions. We conducted an exploratory factor analyses using SPSS to further examine the dimensionality of the communication skill items. As a technique, factor analysis permits investigation of the statistical relationship among variables (e.g., the items) as empirical evidence for or against the relationship of these variables and an
4 182 GRETCHEN GUITON ET AL. Table II. Means, standard deviations, and internal consistency estimates of reliability of communication skills items for each case in the OSCE, 2001 Case Number of Number of Mean SD Cronbach s students items alpha 36 year-old female with chronic sore throat year-old female with abdominal pain year-old male with cough and fatigue year-old female with chest pain year-old male with preventive visit year-old male with diabetes year-old female with weight loss abstract concept (e.g., the factor) (Maradi, 1981). We used principal axis factoring methods to provide information on the number of underlying factors. The nature of the items loading on each factor suggests the constructs underlying variation in student performance. The results are based on data from students taking the exam at one of the four universities in the summer of Students in the Consortium schools received the same cases, although one of the three schools included in the study administered only a subset of cases to all students. Consequently, the number of subjects per case varies from 420 to 532. Because list-wise deletion reduces the number of subjects to 276, the analyses were conducted with all individuals completing each case. Results RELIABILITY We first assessed the scale characteristics and reliability across cases since current reporting of the items by the Consortium involves summing the seven communication items across the seven cases (n i = 49). To assess this practice, we computed Cronbach s alpha and obtained a high internal consistency reliability coefficient (alpha = 0.91) for the 49-item scale (n = 421, M = 33.18, SD = 3.26). Next, we examined the scale characteristics and reliability of the set of seven communication items within each case. As shown in Table II, very reliable ratings (0.89 to 0.94) of student performance are obtained within cases suggesting that these seven items measure a single construct. To further examine the content specificity of individual communication items (e.g., appropriately explored my perspective, listened actively ), we computed Cronbach s alpha for each item individually across cases. Table III presents the results of the cross-case analyses. The dramatically lower reliabilities obtained
5 A PSYCHOMETRIC STUDY 183 Table III. Means, standard deviations, and internal consistency estimates of reliability of individual communication skills items for the OSCE, 2001 Item across seven stations Number of Number of Mean SD Cronbach s students items alpha Appeared professionally competent Effectively gathered information Listened actively Established personal rapport Appropriately explored my perspective Addressed my feelings Met my needs (0.35 to 0.60) indicate considerable variation in student performance on a given communication skill across cases. Although the lower reliabilities may result from the small number of ratings (seven) of each item, we note that the within case reliabilities, also based on seven items, have considerably higher reliabilities. In combination, these results support other research suggesting that student performance differences across cases reflect case-specific variance. Finally we calculated the intra-class correlation coefficient using one-way ANOVA. The obtained ICC of indicates the reliability of a single item, independent of case, rated by a randomly selected rater. This low value further underscores the lack of homogeneity across items in student performance, lending support to the content specificity of items. GENERALIZABILITY STUDY Variance in communication scores was attributed primarily to student and studentby-case interaction in the two-facet design. As shown in Table IV, students account for 33.77% of the variance indicating that the CPX moderately discriminates among students in their competency in interacting with patients. The student-bycase interaction accounts for the highest amount of variance in scores (50%) and indicates that students standing differs on the different cases. Clearly students may demonstrate strong communication skills in one situation and do less well in others. The amount of variance contributed by the cases (8.84%) is small relative to the other estimated components suggesting that only a small portion of the variance is accounted for by differences in cases when considered across all students. The error term, accounting for 6.74% of the variance, suggests a very modest students-bycase-by-item interaction or a source of error variability that has not been captured by the model or both. Finally, the obtained generalizability coefficient, computed for all seven cases and seven items, is indicating reasonably high reliability for making relative decisions about individuals communication skills on the
6 184 GRETCHEN GUITON ET AL. Table IV. ANOVA estimates of variance components for the communication skill items on the OSCE, 2001 Source of variation Sum of df Mean Estimated Percent of squares squares variance total components variance Students % Case % Item E % Students by case % Students by item % Case by item % Error % basis of the full exam. One limitation of this analysis is its inability to disentangle rater and case effects. Overall, the results of the g-study support the idea that students ability to communicate with patients depends substantially on the presenting situation, although a two-hour, seven-station exam permits reasonably reliable judgments about students relative abilities. INTERNAL STRUCTURE Exploratory factor analysis demonstrated a case structure underlying communication skills. The data met all of the assumptions for using the factor model. 1 A principal axis factoring of all 49 communcation items, and the resulting scree plot, 2 suggested seven factors underlying the data with these factors accounting for 60.8% of the variance. The first factor, with an eigenvalue of 9.41, accounts for 19.27% of the variance. This factor is almost twice the magnitude of the second factor (eigenvalue = 5.35, accounting for 10.77% of the variance) indicating a dominant factor, but failing to meet the criteria for unidimensionality (Reckase, 1979). Consequently, we estimated a seven-factor solution using orthogonal rotation (varimax) to obtain a simple structure. 3 This analysis resulted in a highly interpretable factor structure. Items loaded strongly on factors defined by the individual cases with no items loading 0.2 or higher on a second factor. Loadings ranged from to on the chronic sore throat case, from to on the abdominal pain case, from to on the cough and fatigue case, from to on the chest pain case, from to on the preventive case, from to on the diabetes case, and from to on the weight loss case. These loadings are large with 39 of the loadings exceeding 0.71 indicating a 50% overlap with the factor and all but one of the remaining loadings reaching 0.63, indicating a 40% overlap with the factor. 4 Each factor accounts for 8% to 10% of the variance in the set of variables and 13% to 16% of the covariance.
7 A PSYCHOMETRIC STUDY 185 The squared multiple correlations of factor scores obtained from the factor score covariance matrix range from to and indicate a stable solution with factors well defined by the variables. Conclusion The finding that communication skill items vary systematically with specific cases provides support to prior research suggesting that communication skills, at least among novices, are content dependent. Hodges et al. (1996) identified the contentspecific nature of communication skills in a study of the reliability of OSCE stations specifically designed to measure communication skills and concluded communication skills are highly bound to content and that increased difficulty and increased score variance alone are not enough to improve generalizability (p. 42). The concept of a case-specific dimension of communication is further supported by the results of a multi-institutional patient satisfaction study (Meredith and Wood, 1996) in which the more serious the patient s condition, the more dissatisfaction patients reported with physicians communication. Likewise Wilkerson and Rose (2001), examining narrative comments, found that narrative themes clustered within cases based on the specific challenges that each presented. Recently in a generalizability study of communication cases, Keen et al. (2003) found that communication skill levels were the major source of variation with raters having a negligible effect on subject scores (p. 11). They recommend increasing the number of cases rather than raters to increase reliability. Together these studies and our current findings suggest that communication skills possess a generic component, but also appear to be comprised of specific, context-related elements. From a measurement perspective, then, we conclude that reliable measurement of the generic component can be achieved with a series of items administered across cases. Moreover, our findings do not support the representation of communication skills as consisting of a set of separable dimensions, at least as rated by SPs. On the other hand, we agree with Hodges et al. (1996) that the communication skills checklist could be built around specific aspects of communication needed for each particular case rather than expecting all cases to address the same skills. Our results concur with those from the qualitative studies of SP s narrative comments (Rose and Wilkerson, 2001) that suggest there is a fair amount of within student variability in performance across cases. The variability in communications skills demonstrated by an individual student may be a product of qualities of the student, such as personal interests, possession of relevant knowledge, selfconfidence or prior experience. Or it may reflect characteristics of the patient (e.g., age, ethnicity, communication style, culture, socio-economic status), or characteristics of the problem (e.g., complexity, ill-defined nature). Results from other studies (e.g., Keen et al., 2003) along with the strength and consistency of our results from the g-study and factor analysis with data from diverse schools
8 186 GRETCHEN GUITON ET AL. involving multiple standardized patients give us confidence in our findings. The situational variation in communication skills, however, would benefit from future study able to estimate the rater effect. Additional study will be needed to identify the sources of case variability. Of particular importance is the investigation of the relationship between the domain of knowledge and the quality of communication. For instructional purposes, the content specificity found in these data implies that students need multiple, varied communication experiences with an opportunity to practice and reflect on performance. Furthermore, in reporting communication scores for a multi-station examination, it will be important to reflect the variation in a student s performance through the use of case based scores, narrative comments, and opportunities to review the video taped encounters. Acknowledgements The authors wish to thank those faculty members and staff at the University of Irvine School of Medicine and the University of Southern California who agreed to allow the data from their students in the OSCE to be used in the present study. Notes 1 Specifically normality and multicollinearity assumptions were evaluated. Item descriptive statistics indicated that all items demonstrated normality with the exception of 5 items which had kurtosis values between 1 and 1.5. These deviations are not serious given the size of the sample. Bivariate correlations among items rarely exceed 0.7 (only 26 of the 2756 correlations) and none reach 0.9 indicating that multicollinearity is not a problem. 2 3 Following Muthen, an oblique (promax) rotation was estimated. Results indicate that each factor represents a single case with the items loading strongly (0.69 or higher) on a single case and no double loadings reaching significance (i.e., 0.3). Moreover, the factor correlation matrix indicates only a slight relationship among factors with none of the 21 correlations reaching 0.32 the level where 10% of the correlation among factors indicates that an oblique rotation might be warranted. Consequently, only the results of the orthogonal rotation are reported in detail.
9 A PSYCHOMETRIC STUDY Comrey and Lee (1992) note that any loading of 0.71 or higher indicates an overlap of at least 50% with the factor. References Bayer-Fetzer Conference on Patient-Physician Interaction in Medical Education (2001). Essential elements of communication in medical encounters: the Kalamazoo Consensus Statement. Academic Medicine 76: Blue, A.V., Chessman, A.W., Gilbert, G.E. & Mainous III, A.G. (2000). Responding to patients emotions: important for standardized patient satisfaction. Family Medicine 32: Cohen, D.S., Colliver, J.A., Robbs, R.S. & Swartz, M.H. (1997). A large-scale study of the reliabilities of checklist scores and ratings of communication skills evaluated on a standardized-patient examination. Advances in Health Sciences Education 1: Colliver, J.A. & Swartz, M.H. (1997). Assessing clinical performance with standardized patients. JAMA 278: Colliver, J.A., Swartz, M.H., Robbs, R.S. & Cohen, D.S. (1999). Relationship between clinical competence and communication skills in standardized-patient assessment Academic Medicine 74: Colliver, J.A., Willis, M.S., Robbs, R.S., Cohen, D.S. & Swartz, M.H. (1998). Assessment of empathy in a standardized-patient examination. Teaching and Learning in Medicine 10: Donnelly, MB., Sloan, D., Pymale, M. & Schwartz, R. (2000). Assessment of residents interpersonal skills by faculty proctors and standardized patients: a psychometric analysis. Academic Medicine 75(October): S93 S95. Hodges, B., Turnbull, J., Cohen, R., Bienenstock, A. & Norman, A. (1996). Evaluating communication skills in the objective structured clinical examination format: reliability and generalizability. Medical Education 30: Holmboe, E.S. & Hawkins, R.E. (1998). Methods for evaluating the clinical competence of residents in internal medicine: a review. Annals of Internal Medicine 129: Keen, A.J.A., Klein, S. & Alexander, D.A. (2003). Assess the communication skills of doctors in training: reliability and sources of error. Advances in Health Sciences Education 8: Maradi, A. (1981). Factor analysis as an aid in the formation and refinement of empirically useful concepts. In D. Jackson & E.F. Borgatta (eds.), Factor Analysis and Measurement in Sociological Research. Thousand Oaks, CA: Sage. Meredith, P. & Wood, C. (1996). Aspects of patient satisfaction with communication in surgical care: confirming qualitative feedback through quantitative methods. International Journal for Quality in Health Care 8: Reckase, M.D. (1979). Unifactor latent trait models applied to multi-factor tests: results and implications. Journal of Educational Statistics 4: Rose, M. & Wilkerson, L. (2001). Widening the lens on standardized patient assessment: what the encounter can reveal about the development of clinical expertise. Academic Medicine 76: Sloan, D.A., Donnelly, M.B., Schwartz, R. & Strodel, W.E. (1995). The objective structured clinical examination: the new gold standard for evaluating postgraduate clinical performance. American Journal of Surgery 222: 735. Vu, N.V. & Barrows, H.S. (1994). Use of standardized patients in clinical assessments: recent developments and measurement findings. Educational Researcher 23: Wilkerson, L. & Rose, M. (2001). Learning from narrative comments of standardized patients during an objective structured clinical examination of final-year medical students. ERIC Clearinghouse on Assessment and Evaluation.
10
PSYCHOMETRIC PROPERTIES OF CLINICAL PERFORMANCE RATINGS
PSYCHOMETRIC PROPERTIES OF CLINICAL PERFORMANCE RATINGS A total of 7931 ratings of 482 third- and fourth-year medical students were gathered over twelve four-week periods. Ratings were made by multiple
More informationRESULTS. Chapter INTRODUCTION
8.1 Chapter 8 RESULTS 8.1 INTRODUCTION The previous chapter provided a theoretical discussion of the research and statistical methodology. This chapter focuses on the interpretation and discussion of the
More information2013 Supervisor Survey Reliability Analysis
2013 Supervisor Survey Reliability Analysis In preparation for the submission of the Reliability Analysis for the 2013 Supervisor Survey, we wanted to revisit the purpose of this analysis. This analysis
More informationBasic concepts and principles of classical test theory
Basic concepts and principles of classical test theory Jan-Eric Gustafsson What is measurement? Assignment of numbers to aspects of individuals according to some rule. The aspect which is measured must
More informationThe reliability of a hypothesis generation and testing task
Medicul Education 199, 24, 57-51 1 The reliability of a hypothesis generation and testing task M. B. DONNELLY, J. C. SISSONt & J.. WOOLLISCROFTt Department of Postgraduate Medicine and Health Professions
More informationMaking a psychometric. Dr Benjamin Cowan- Lecture 9
Making a psychometric Dr Benjamin Cowan- Lecture 9 What this lecture will cover What is a questionnaire? Development of questionnaires Item development Scale options Scale reliability & validity Factor
More informationAssessing the Validity and Reliability of the Teacher Keys Effectiveness. System (TKES) and the Leader Keys Effectiveness System (LKES)
Assessing the Validity and Reliability of the Teacher Keys Effectiveness System (TKES) and the Leader Keys Effectiveness System (LKES) of the Georgia Department of Education Submitted by The Georgia Center
More informationVerification of the Structural Model concerning Selfesteem, Social Support, and Quality of Life among Multicultural Immigrant Women
, pp.57-62 http://dx.doi.org/10.14257/astl.2015.91.12 Verification of the Structural Model concerning Selfesteem, Social Support, and Quality of Life among Multicultural Immigrant Women Rack In Choi 1
More informationDimensionality, internal consistency and interrater reliability of clinical performance ratings
Medical Education 1987, 21, 130-137 Dimensionality, internal consistency and interrater reliability of clinical performance ratings B. R. MAXIMt & T. E. DIELMANS tdepartment of Mathematics and Statistics,
More informationPediatrics Milestones and Meaningful Assessment Translating the Pediatrics Milestones into Assessment Items for use in the Clinical Setting
Pediatrics Milestones and Meaningful Assessment Translating the Pediatrics Milestones into Assessment Items for use in the Clinical Setting Ann Burke Susan Guralnick Patty Hicks Jeanine Ronan Dan Schumacher
More informationFactor Analysis of Gulf War Illness: What Does It Add to Our Understanding of Possible Health Effects of Deployment?
October 3, 2006 Factor Analysis Examples: Example 1: Factor Analysis of Gulf War Illness: What Does It Add to Our Understanding of Possible Health Effects of Deployment? 1 2 2 Susan E. Shapiro, Michael
More informationIDEA Technical Report No. 20. Updated Technical Manual for the IDEA Feedback System for Administrators. Stephen L. Benton Dan Li
IDEA Technical Report No. 20 Updated Technical Manual for the IDEA Feedback System for Administrators Stephen L. Benton Dan Li July 2018 2 Table of Contents Introduction... 5 Sample Description... 6 Response
More informationSubescala D CULTURA ORGANIZACIONAL. Factor Analysis
Subescala D CULTURA ORGANIZACIONAL Factor Analysis Descriptive Statistics Mean Std. Deviation Analysis N 1 3,44 1,244 224 2 3,43 1,258 224 3 4,50,989 224 4 4,38 1,118 224 5 4,30 1,151 224 6 4,27 1,205
More informationPsychometric properties of the Chinese quality of life instrument (HK version) in Chinese and Western medicine primary care settings
Qual Life Res (2012) 21:873 886 DOI 10.1007/s11136-011-9987-3 Psychometric properties of the Chinese quality of life instrument (HK version) in Chinese and Western medicine primary care settings Wendy
More informationFactor Analysis. MERMAID Series 12/11. Galen E. Switzer, PhD Rachel Hess, MD, MS
Factor Analysis MERMAID Series 2/ Galen E Switzer, PhD Rachel Hess, MD, MS Ways to Examine Groups of Things Groups of People Groups of Indicators Cluster Analysis Exploratory Factor Analysis Latent Class
More informationalternate-form reliability The degree to which two or more versions of the same test correlate with one another. In clinical studies in which a given function is going to be tested more than once over
More informationChapter 3. Psychometric Properties
Chapter 3 Psychometric Properties Reliability The reliability of an assessment tool like the DECA-C is defined as, the consistency of scores obtained by the same person when reexamined with the same test
More informationQuantitative Methods in Computing Education Research (A brief overview tips and techniques)
Quantitative Methods in Computing Education Research (A brief overview tips and techniques) Dr Judy Sheard Senior Lecturer Co-Director, Computing Education Research Group Monash University judy.sheard@monash.edu
More informationSurvey research (Lecture 1) Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2015 Creative Commons Attribution 4.
Summary & Conclusion Lecture 10 Survey Research & Design in Psychology James Neill, 2015 Creative Commons Attribution 4.0 Overview 1. Survey research 2. Survey design 3. Descriptives & graphing 4. Correlation
More informationSurvey research (Lecture 1)
Summary & Conclusion Lecture 10 Survey Research & Design in Psychology James Neill, 2015 Creative Commons Attribution 4.0 Overview 1. Survey research 2. Survey design 3. Descriptives & graphing 4. Correlation
More informationEverything DiSC 363 for Leaders. Research Report. by Inscape Publishing
Everything DiSC 363 for Leaders Research Report by Inscape Publishing Introduction Everything DiSC 363 for Leaders is a multi-rater assessment and profile that is designed to give participants feedback
More informationSummary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2016 Creative Commons Attribution 4.0
Summary & Conclusion Lecture 10 Survey Research & Design in Psychology James Neill, 2016 Creative Commons Attribution 4.0 Overview 1. Survey research and design 1. Survey research 2. Survey design 2. Univariate
More informationThe Youth Experience Survey 2.0: Instrument Revisions and Validity Testing* David M. Hansen 1 University of Illinois, Urbana-Champaign
The Youth Experience Survey 2.0: Instrument Revisions and Validity Testing* David M. Hansen 1 University of Illinois, Urbana-Champaign Reed Larson 2 University of Illinois, Urbana-Champaign February 28,
More informationGlobal Perspective Inventory (GPI) Report
Global Perspective Inventory (GPI) 2012-2013 Report Executive Summary display higher levels of global competence than freshmen in all of the GPI scales except for the interpersonal social responsibility
More informationisc ove ring i Statistics sing SPSS
isc ove ring i Statistics sing SPSS S E C O N D! E D I T I O N (and sex, drugs and rock V roll) A N D Y F I E L D Publications London o Thousand Oaks New Delhi CONTENTS Preface How To Use This Book Acknowledgements
More informationStatistics for Psychosocial Research Session 1: September 1 Bill
Statistics for Psychosocial Research Session 1: September 1 Bill Introduction to Staff Purpose of the Course Administration Introduction to Test Theory Statistics for Psychosocial Research Overview: a)
More informationPLACE ATTACHMENT AND WILLINGNESS TO PAY: HOW DO VISITORS VALUE STATE PARKS?
PLACE ATTACHMENT AND WILLINGNESS TO PAY: HOW DO VISITORS VALUE STATE PARKS? Jason W. Whiting Warnell School of Forestry and Natural Resources, RM 1-301C whitingj@uga.edu Lincoln R. Larson Dr. Gary T. Green
More informationTeachers Sense of Efficacy Scale: The Study of Validity and Reliability
EUROPEAN ACADEMIC RESEARCH Vol. II, Issue 12/ March 2015 ISSN 2286-4822 www.euacademic.org Impact Factor: 3.1 (UIF) DRJI Value: 5.9 (B+) Teachers Sense of Efficacy Scale: The Study of Validity and Dr.
More informationDISCPP (DISC Personality Profile) Psychometric Report
Psychometric Report Table of Contents Test Description... 5 Reference... 5 Vitals... 5 Question Type... 5 Test Development Procedures... 5 Test History... 8 Operational Definitions... 9 Test Research and
More informationSubescala B Compromisso com a organização escolar. Factor Analysis
Subescala B Compromisso com a organização escolar Factor Analysis Descriptive Statistics Mean Std. Deviation Analysis N 1 4,42 1,108 233 2 4,41 1,001 233 3 4,99 1,261 233 4 4,37 1,055 233 5 4,48 1,018
More informationSelf-Compassion, Perceived Academic Stress, Depression and Anxiety Symptomology Among Australian University Students
International Journal of Psychology and Cognitive Science 2018; 4(4): 130-143 http://www.aascit.org/journal/ijpcs ISSN: 2472-9450 (Print); ISSN: 2472-9469 (Online) Self-Compassion, Perceived Academic Stress,
More informationBy Hui Bian Office for Faculty Excellence
By Hui Bian Office for Faculty Excellence 1 Email: bianh@ecu.edu Phone: 328-5428 Location: 1001 Joyner Library, room 1006 Office hours: 8:00am-5:00pm, Monday-Friday 2 Educational tests and regular surveys
More informationSP CASE WRITING WORKING GROUP MATERIALS Agenda
SP CASE WRITING WORKING GROUP MATERIALS Agenda Meeting Date Time Location Attendees Case Development [Chief Complaint] Facilitator Item 1. 1:30 pm Introduction 2. 1:45 pm Discussion of possible case scenarios
More informationUsing Analytical and Psychometric Tools in Medium- and High-Stakes Environments
Using Analytical and Psychometric Tools in Medium- and High-Stakes Environments Greg Pope, Analytics and Psychometrics Manager 2008 Users Conference San Antonio Introduction and purpose of this session
More informationAPÊNDICE 6. Análise fatorial e análise de consistência interna
APÊNDICE 6 Análise fatorial e análise de consistência interna Subescala A Missão, a Visão e os Valores A ação do diretor Factor Analysis Descriptive Statistics Mean Std. Deviation Analysis N 1 4,46 1,056
More informationHPS301 Exam Notes- Contents
HPS301 Exam Notes- Contents Week 1 Research Design: What characterises different approaches 1 Experimental Design 1 Key Features 1 Criteria for establishing causality 2 Validity Internal Validity 2 Threats
More informationInternational Conference on Humanities and Social Science (HSS 2016)
International Conference on Humanities and Social Science (HSS 2016) The Chinese Version of WOrk-reLated Flow Inventory (WOLF): An Examination of Reliability and Validity Yi-yu CHEN1, a, Xiao-tong YU2,
More informationExamining the efficacy of the Theory of Planned Behavior (TPB) to understand pre-service teachers intention to use technology*
Examining the efficacy of the Theory of Planned Behavior (TPB) to understand pre-service teachers intention to use technology* Timothy Teo & Chwee Beng Lee Nanyang Technology University Singapore This
More informationRELIABILITY OF REAL TIME JUDGING SYSTEM
RELIABILITY OF REAL TIME JUDGING SYSTEM Maja Bučar Pajek 1, Warwick Forbes 2, Jernej Pajek 3, Bojan Leskošek 1, and Ivan Čuk 1 1 Faculty of Sport, University of Ljubljana, Slovenia 2 Australian Institute
More informationHow Do We Assess Students in the Interpreting Examinations?
How Do We Assess Students in the Interpreting Examinations? Fred S. Wu 1 Newcastle University, United Kingdom The field of assessment in interpreter training is under-researched, though trainers and researchers
More informationFrom Bivariate Through Multivariate Techniques
A p p l i e d S T A T I S T I C S From Bivariate Through Multivariate Techniques R e b e c c a M. W a r n e r University of New Hampshire DAI HOC THAI NGUYEN TRUNG TAM HOC LIEU *)SAGE Publications '55'
More informationAn International Study of the Reliability and Validity of Leadership/Impact (L/I)
An International Study of the Reliability and Validity of Leadership/Impact (L/I) Janet L. Szumal, Ph.D. Human Synergistics/Center for Applied Research, Inc. Contents Introduction...3 Overview of L/I...5
More informationAUTISM SPECTRUM RATING SCALES (ASRS )
AUTISM SPECTRUM RATING SCALES ( ) Scoring the for Individuals Who Do Not Speak or Speak Infrequently Sam Goldstein, Ph.D. & Jack A. Naglieri, Ph.D. Technical Report #1 This technical report was edited
More informationGlobal Perspective Inventory (GPI) - Pilot Report
Global Perspective Inventory (GPI) - Pilot 2010-11 Report Introduction The Global Perspectives Inventory is a nationally recognized instrument designed to measure a student s global perspective. The GPI
More informationDevelopment and Psychometric Properties of the Relational Mobility Scale for the Indonesian Population
Development and Psychometric Properties of the Relational Mobility Scale for the Indonesian Population Sukaesi Marianti Abstract This study aims to develop the Relational Mobility Scale for the Indonesian
More informationTHE DIMENSIONALITY OF THE AARHUS UNIVERSITY QUALITY IN THE PHD PROCESS SURVEY
THE DIMENSIONALITY OF THE AARHUS UNIVERSITY QUALITY IN THE PHD PROCESS SURVEY CENTRE FOR TEACHING AND LEARNING AARHUS UNIVERSITY SCHOOL OF BUSINESS AND SOCIAL SCIENCES FUGLESANGS ALLÉ 4 8210 AARHUS, DENMARK
More informationReliability and Validity of the Pediatric Quality of Life Inventory Generic Core Scales, Multidimensional Fatigue Scale, and Cancer Module
2090 The PedsQL in Pediatric Cancer Reliability and Validity of the Pediatric Quality of Life Inventory Generic Core Scales, Multidimensional Fatigue Scale, and Cancer Module James W. Varni, Ph.D. 1,2
More informationJournal of American Science 2010;6(10) Age and gender differences and construct of the children s emotional intelligence
Age and gender differences and construct of the children s emotional intelligence Mojgan Mirza, Ma rof Redzuan* Department of Social anddevelopment Science Faculty of Human Ecology, University Putra Malaysia
More informationComparing Vertical and Horizontal Scoring of Open-Ended Questionnaires
A peer-reviewed electronic journal. Copyright is retained by the first or sole author, who grants right of first publication to the Practical Assessment, Research & Evaluation. Permission is granted to
More informationImpact of Task-based Checklist Scoring and Two Domains Global Rating Scale in Objective Structured Clinical Examination of Pharmacy Students
Pharmaceutical Education Impact of Task-based Checklist Scoring and Two Domains Global Rating Scale in Objective Structured Clinical Examination of Pharmacy Students Sajesh Kalkandi Veettil * and Kingston
More informationmeasuring performance
measuring performance Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments Stephen J Lurie, Christopher J Mooney, Anne C Nofziger, Sean
More informationDesign and Preliminary Validation of an Instrument to Measure Medical Student Attitudes Toward the Homeless
Design and Preliminary Validation of an Instrument to Measure Medical Student Attitudes Toward the Homeless David S. Buck, MD, MPH F. Marconi Monteiro, EdD Suzanne Kneuper, MA Dana L. Clark, MD Allegra
More informationReliability. Internal Reliability
32 Reliability T he reliability of assessments like the DECA-I/T is defined as, the consistency of scores obtained by the same person when reexamined with the same test on different occasions, or with
More informationThe Meta on Meta-Analysis. Presented by Endia J. Lindo, Ph.D. University of North Texas
The Meta on Meta-Analysis Presented by Endia J. Lindo, Ph.D. University of North Texas Meta-Analysis What is it? Why use it? How to do it? Challenges and benefits? Current trends? What is meta-analysis?
More informationThe Asian Conference on Education & International Development 2015 Official Conference Proceedings. iafor
Constructing and Validating Behavioral Components Scale of Motivational Goals in Mathematics Nancy Castro, University of Santo Tomas, Philippines Michelle Cruz, University of Santo Tomas, Philippines Maria
More informationValidity and Reliability of Sport Satisfaction
International Research Journal of Applied and Basic Sciences 2014 Available online at www.irjabs.com ISSN 2251-838X / Vol, 8 (10): 1782-1786 Science Explorer Publications Validity and Reliability of Sport
More informationEnglish 10 Writing Assessment Results and Analysis
Academic Assessment English 10 Writing Assessment Results and Analysis OVERVIEW This study is part of a multi-year effort undertaken by the Department of English to develop sustainable outcomes assessment
More informationEXPLORING CASINO GAMBLING IMPACT PERCEPTIONS: A GENDERED SOCIAL EXCHANGE THEORY APPROACH
University of Massachusetts Amherst ScholarWorks@UMass Amherst Tourism Travel and Research Association: Advancing Tourism Research Globally 2007 ttra International Conference EXPLORING CASINO GAMBLING
More informationCHAPTER VI RESEARCH METHODOLOGY
CHAPTER VI RESEARCH METHODOLOGY 6.1 Research Design Research is an organized, systematic, data based, critical, objective, scientific inquiry or investigation into a specific problem, undertaken with the
More informationInternal structure evidence of validity
Internal structure evidence of validity Dr Wan Nor Arifin Lecturer, Unit of Biostatistics and Research Methodology, Universiti Sains Malaysia. E-mail: wnarifin@usm.my Wan Nor Arifin, 2017. Internal structure
More informationMeasuring Perceived Social Support in Mexican American Youth: Psychometric Properties of the Multidimensional Scale of Perceived Social Support
Marquette University e-publications@marquette College of Education Faculty Research and Publications Education, College of 5-1-2004 Measuring Perceived Social Support in Mexican American Youth: Psychometric
More informationAssessment of sexual function by DSFI among the Iranian married individuals
Basic Research Journal of Medicine and Clinical Sciences ISSN 2315-6864 Vol. 4(2) pp. 68-74 February 2015 Available online http//www.basicresearchjournals.org Copyright 2015 Basic Research Journal Full
More informationThe Personal Profile System 2800 Series Research Report
The Personal Profile System 2800 Series Research Report The Personal Profile System 2800 Series Research Report Item Number: O-255 1996 by Inscape Publishing, Inc. All rights reserved. Copyright secured
More informationInvestigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education.
The Reliability of PLATO Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO M. Ken Cor Stanford University School of Education April,
More informationBrent Duckor Ph.D. (SJSU) Kip Tellez, Ph.D. (UCSC) BEAR Seminar April 22, 2014
Brent Duckor Ph.D. (SJSU) Kip Tellez, Ph.D. (UCSC) BEAR Seminar April 22, 2014 Studies under review ELA event Mathematics event Duckor, B., Castellano, K., Téllez, K., & Wilson, M. (2013, April). Validating
More informationDeveloping Skills at Making Observations
Developing Skills at Making Observations Lessons from Faculty Development and Rater Cognition Research Eric S. Holmboe Jennifer R. Kogan Roadmap 1. Define workplace based assessment and the theories supporting
More informationMiller s Assessment Pyramid
Roadmap Developing Skills at Making Observations Lessons from Faculty Development and Rater Cognition Research 1. Define workplace based assessment and the theories supporting direct observation 2. Identify
More informationMeasuring mathematics anxiety: Paper 2 - Constructing and validating the measure. Rob Cavanagh Len Sparrow Curtin University
Measuring mathematics anxiety: Paper 2 - Constructing and validating the measure Rob Cavanagh Len Sparrow Curtin University R.Cavanagh@curtin.edu.au Abstract The study sought to measure mathematics anxiety
More informationStatistics is the science of collecting, organizing, presenting, analyzing, and interpreting data to assist in making effective decisions
Readings: OpenStax Textbook - Chapters 1 5 (online) Appendix D & E (online) Plous - Chapters 1, 5, 6, 13 (online) Introductory comments Describe how familiarity with statistical methods can - be associated
More informationScales and Component Items March 2017
www.gpi.hs.iastate.edu Scales and Component Items March 2017 Recommended Citation: Research Institute for Studies in Education (2017). Global Perspective Inventory: Scales and component items. Iowa State
More informationResearch Questions and Survey Development
Research Questions and Survey Development R. Eric Heidel, PhD Associate Professor of Biostatistics Department of Surgery University of Tennessee Graduate School of Medicine Research Questions 1 Research
More informationDevelopment of a Shortened Form of the Coping Responses Inventory-Youth with an Australian Sample
Development of a Shortened Form of the Coping Responses Inventory-Youth with an Australian Sample Deborah J. Eyles & Glen W. Bates Swinburne University of Technology, Australia The 48-item Coping Responses
More informationFactorial Validity and Reliability of 12 items General Health Questionnaire in a Bhutanese Population. Tshoki Zangmo *
Factorial Validity and Reliability of 12 items General Health Questionnaire in a Bhutanese Population Tshoki Zangmo * Abstract The aim of this study is to test the factorial structure and the internal
More informationTeacher stress: A comparison between casual and permanent primary school teachers with a special focus on coping
Teacher stress: A comparison between casual and permanent primary school teachers with a special focus on coping Amanda Palmer, Ken Sinclair and Michael Bailey University of Sydney Paper prepared for presentation
More informationANOVA in SPSS (Practical)
ANOVA in SPSS (Practical) Analysis of Variance practical In this practical we will investigate how we model the influence of a categorical predictor on a continuous response. Centre for Multilevel Modelling
More informationValidity, Reliability, and Defensibility of Assessments in Veterinary Education
Instructional Methods Validity, Reliability, and Defensibility of Assessments in Veterinary Education Kent Hecker g Claudio Violato ABSTRACT In this article, we provide an introduction to and overview
More informationSEM-Based Composite Reliability Estimates of the Crisis Acuity Rating Scale with Children and Adolescents
Archives of Assessment Psychology, Vol. 1, No. 1, (pp) Printed in U.S.A. All rights reserved 2011 American Board of Assessment Psychology SEM-Based Composite Reliability Estimates of the Crisis Acuity
More informationPaul Irwing, Manchester Business School
Paul Irwing, Manchester Business School Factor analysis has been the prime statistical technique for the development of structural theories in social science, such as the hierarchical factor model of human
More informationGENERALIZABILITY AND RELIABILITY: APPROACHES FOR THROUGH-COURSE ASSESSMENTS
GENERALIZABILITY AND RELIABILITY: APPROACHES FOR THROUGH-COURSE ASSESSMENTS Michael J. Kolen The University of Iowa March 2011 Commissioned by the Center for K 12 Assessment & Performance Management at
More informationThe SAGE Encyclopedia of Educational Research, Measurement, and Evaluation Multivariate Analysis of Variance
The SAGE Encyclopedia of Educational Research, Measurement, Multivariate Analysis of Variance Contributors: David W. Stockburger Edited by: Bruce B. Frey Book Title: Chapter Title: "Multivariate Analysis
More informationCEMO RESEARCH PROGRAM
1 CEMO RESEARCH PROGRAM Methodological Challenges in Educational Measurement CEMO s primary goal is to conduct basic and applied research seeking to generate new knowledge in the field of educational measurement.
More informationVictoria YY Xu PGY-2 Internal Medicine University of Toronto. Supervisor: Dr. Camilla Wong
Validity, Reliability, Feasibility, and Acceptability of Using the Consultation Letter Rating Scale to Assess Written Communication Competencies Among Geriatric Medicine Postgraduate Trainees Victoria
More informationWhat Works Clearinghouse
What Works Clearinghouse U.S. DEPARTMENT OF EDUCATION July 2012 WWC Review of the Report Randomized, Controlled Trial of the LEAP Model of Early Intervention for Young Children With Autism Spectrum Disorders
More informationUNIVERSITY OF CALGARY. Reliability & Validity of the. Objective Structured Clinical Examination (OSCE): A Meta-Analysis. Ibrahim Al Ghaithi A THESIS
UNIVERSITY OF CALGARY Reliability & Validity of the Objective Structured Clinical Examination (OSCE): A Meta-Analysis by Ibrahim Al Ghaithi A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL
More informationSOCIAL SUPPORT FOR MEDICATION ADHERENCE AMONG THAI PEOPLE WITH POST-ACUTE MYOCARDIAL INFARCTION: FACTOR ANALYSIS
Original Research Article 1 SOCIAL SUPPORT FOR MEDICATION ADHERENCE AMONG THAI PEOPLE WITH POST-ACUTE MYOCARDIAL INFARCTION: FACTOR ANALYSIS Rapin Polsook *, Yupin Aungsuroch, Sureeporn Thanasilp Faculty
More informationPackianathan Chelladurai Troy University, Troy, Alabama, USA.
DIMENSIONS OF ORGANIZATIONAL CAPACITY OF SPORT GOVERNING BODIES OF GHANA: DEVELOPMENT OF A SCALE Christopher Essilfie I.B.S Consulting Alliance, Accra, Ghana E-mail: chrisessilfie@yahoo.com Packianathan
More informationTHE USE OF CRONBACH ALPHA RELIABILITY ESTIMATE IN RESEARCH AMONG STUDENTS IN PUBLIC UNIVERSITIES IN GHANA.
Africa Journal of Teacher Education ISSN 1916-7822. A Journal of Spread Corporation Vol. 6 No. 1 2017 Pages 56-64 THE USE OF CRONBACH ALPHA RELIABILITY ESTIMATE IN RESEARCH AMONG STUDENTS IN PUBLIC UNIVERSITIES
More informationREPORT. Technical Report: Item Characteristics. Jessica Masters
August 2010 REPORT Diagnostic Geometry Assessment Project Technical Report: Item Characteristics Jessica Masters Technology and Assessment Study Collaborative Lynch School of Education Boston College Chestnut
More informationAssessing the quality of environmental design of nursing homes for people with dementia: development of a new tool
University of Wollongong Research Online Faculty of Science, Medicine and Health - Papers Faculty of Science, Medicine and Health 2015 Assessing the quality of environmental design of nursing homes for
More informationUnit outcomes. Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons Attribution 4.0.
Summary & Conclusion Image source: http://commons.wikimedia.org/wiki/file:pair_of_merops_apiaster_feeding_cropped.jpg Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons
More informationUnit outcomes. Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons Attribution 4.0.
Summary & Conclusion Image source: http://commons.wikimedia.org/wiki/file:pair_of_merops_apiaster_feeding_cropped.jpg Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons
More informationComputing composite scores of patients report of health professional behaviour Summary, Methods and Results Last updated 8 March 2011
Computing composite scores of patients report of health professional behaviour Summary, Methods and Results Last updated 8 March 2011 Summary: The patient questionnaire includes items which assess patients
More informationMeasurement and Descriptive Statistics. Katie Rommel-Esham Education 604
Measurement and Descriptive Statistics Katie Rommel-Esham Education 604 Frequency Distributions Frequency table # grad courses taken f 3 or fewer 5 4-6 3 7-9 2 10 or more 4 Pictorial Representations Frequency
More informationISSN X Journal of Educational and Social Research Vol. 2 (8) October 2012
The Investigation of Czech Lower Secondary School Pupils Toward Science Subjects Milan Kubiatko Doi:10.5901/jesr.2012.v2n8p11 Institute for Research in School Education, Masaryk University, Brno,Czech
More informationDiscussion. were best fit by models of determination that did not include genetic effects.
A Behavior Genetic Investigation of the Relationship Between Leadership and Personality Andrew M. Johnson 1, Philip A. Vernon 2, Julie Aitken Harris 3, and Kerry L. Jang 4 1 Faculty of Health Sciences,The
More informationPSYCHOMETRICS APPLIED TO HEALTHCARE PROFESSIONS EDUCATION
PSYCHOMETRICS APPLIED TO HEALTHCARE PROFESSIONS EDUCATION COURSE PROGRAMME Psychometric properties such as reliability and validity are essential components in the utility of assessment in medical education.
More informationCatching the Hawks and Doves: A Method for Identifying Extreme Examiners on Objective Structured Clinical Examinations
Catching the Hawks and Doves: A Method for Identifying Extreme Examiners on Objective Structured Clinical Examinations July 20, 2011 1 Abstract Performance-based assessments are powerful methods for assessing
More informationThree Subfactors of the Empathic Personality Kimberly A. Barchard, University of Nevada, Las Vegas
1 Three Subfactors of the Empathic Personality Kimberly A. Barchard, University of Nevada, Las Vegas Reference: Barchard, K.A. (2002, May). Three subfactors of the empathic personality. Poster presented
More informationHANDOUTS FOR BST 660 ARE AVAILABLE in ACROBAT PDF FORMAT AT:
Applied Multivariate Analysis BST 660 T. Mark Beasley, Ph.D. Associate Professor, Department of Biostatistics RPHB 309-E E-Mail: mbeasley@uab.edu Voice:(205) 975-4957 Website: http://www.soph.uab.edu/statgenetics/people/beasley/tmbindex.html
More informationExploratory Factor Analysis Student Anxiety Questionnaire on Statistics
Proceedings of Ahmad Dahlan International Conference on Mathematics and Mathematics Education Universitas Ahmad Dahlan, Yogyakarta, 13-14 October 2017 Exploratory Factor Analysis Student Anxiety Questionnaire
More information