measuring performance

Transcription

1 measuring performance Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments Stephen J Lurie, Christopher J Mooney, Anne C Nofziger, Sean C Meldrum & Ronald M Epstein CONTEXT Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondentsõ rating styles, which may lead to unreliable measurement of examineesõ communication skills. METHODS Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actorsõ personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actorsõ response styles contribute to overall scores, and improvements in reliability achieved by standardising actorsõ ratings. RESULTS There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a CronbachÕs alpha of 0.8; standardisation reduces this number to 18. Office of Educational Evaluation and Research, University of Rochester School of Medicine and Dentistry, Rochester, New York, USA Correspondence: Stephen J Lurie, 601 Elmwood Avenue, Box 601, Rochester, New York 14624, USA. Tel: ; Fax: ; stephen_lurie@urmc.rochester.edu CONCLUSIONS Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters. KEYWORDS *communication; *patient standardisation; physician)patient relations; New York; education, medical, undergraduate *methods; education, dental *methods; clinical competence *standards; patient satisfaction; humans. Medical Education 2008: 42: doi: /j x INTRODUCTION Nearly all medical schools now have formal curricula for teaching and assessing communication skills. Communication is a major component of the Medical School Outcome Project of the Association of American Medical Colleges, 1 and its assessment is now a requirement of the National Board of Medical Examiners. 2 A number of formal tools for assessing communication skills have appeared in the last decade. Most of these tools involve theory-based coding systems that require trained raters to assess videotaped encounters between students and standardised patients (SPs), 3 5 whereas others 6 describe specific communicationrelated behaviours and tasks that can be objectively measured. Although such systems appear to be reliable, they are also quite labour-intensive. Thus, students may not have the opportunity to be scored on more than a small number of encounters. This limitation may be especially problematic because communication skills may be case-specific: a student who appears to be a good communicator in one set of clinical circumstances may perform poorly in another. 7 Because assessment of overall 662 ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

2 Subjective communication rating scale Overview What is already known on this subject Patients have idiosyncratic reactions to their doctorsõ communication skills. This may also be true for standardised patients. What this study adds Raters are highly variable in their use of a standardised communication rating scale. Such differences between raters substantially decrease the reliability of measurement. Using rater-standardised scores can increase reliability without additional cost. Suggestions for further research It remains unknown whether raters are similarly variable in other aspects of measurement, or whether it would be beneficial to train raters to produce more standardised responses. communication skills appears to require a relatively large number of cases, its reliability may be limited by insufficient time or resources to provide a sufficient number of stations to assess these skills. This problem may be exacerbated by the use of checklists that ask SPs to describe their personal reactions to the interviewerõs communication skills, rather than checklists of behaviours. Such subjective judgements may represent a different domain than that assessed with behavioural checklists. 3,8 In practice, however, their use is complicated by the fact that different SPs have their own systematic individual biases. Thus, it is possible that an interviewer who received a relatively high score from one SP may receive a relatively low score from another for exactly the same set of communications behaviours. This is an important source of extraneous variability in scores and it may make the overall score less reliable. Traditionally, the only way to improve the reliability of such a test is to make the test longer, in the hope that such sources of variability will cancel themselves out over multiple questions. To address this question, Colliver et al. 9 compared average scores on a subjective checklist completed by different actors who portrayed the same case. Although they found many significant differences between actors in their rating styles, these authors concluded that such differences tended to even out over a sufficiently large number of cases. 9 Unfortunately, because students did not receive repeated scores from different actors on the same cases, Colliver et al. were unable to quantify the amount of variability that such differences may introduce into individual scores, or how many cases would be required to negate effects of rating styles. 9 Other studies have used a more powerful generalisability study (G study) framework to assess these concerns. Colliver et al. 10,11 found that variability resulting from individual actors was relatively small compared with that of students and student)case interactions. Nonetheless, these authors found that 18 stations on an objective structured clinical examination (OSCE) were required to achieve a reliability coefficient It is possible, however, that reliability could be increased by standardising scores within actors, assuming that each actor evaluates a sufficiently large number of examinees to generate an actor-specific distribution of scores. Each actorõs scores would then be normalised to the same mean and variance, and these recomputed scores would then be added to yield an overall composite score. In order to assess this possibility, we performed three separate analyses of data acquired using the Rochester Communication Rating Scale (RCRS), 12 which SPs complete immediately after an encounter with a student. The RCRS was developed to assess the patient-centred communication skills of trainees at all levels, as well as those of practising doctors. Components of patient-centred communication that are assessed include: eliciting the patientõs perspective; understanding the psychosocial context; developing a collaborative healing relationship, and actively involving the patient in decisions about his or her health. Items were derived from a number of existing scales, the results of which have been correlated with patient satisfaction, adherence and well-being The item pool was piloted using an ethnically and educationally diverse focus group of SPs and real patients in order to ascertain that the items would be easily understood. The scale was then iteratively revised. Items with low variability were eliminated, yielding the current 19-item version, which we have used for SP examinations since the year An SP requires an average of 2 3 minutes to complete ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

3 S J Lurie et al the instrument. Items on the RCRS are displayed in Table 1. Firstly, within seven different cases in which students were assessed by the RCRS, we compared ratersõ assigned scores against one another. In theory, if students are randomly assigned to raters, there should be no difference between raters in the scores Table 1 Rochester Communication Rating Scale 1 Interviewer attended to my physical comfort during interview and physical examination 2 InterviewerÕs body language and tone of voice communicated caring and concern 3 Interviewer did not seem distracted 4 Interviewer made an effort to understand my feelings and emotions 5 Interviewer made me feel I could tell him or her anything, even something personal 6 Interviewer took interest in even my smallest problems and concerns 7 Interviewer allowed me to tell my story in my own words 8 Interviewer asked about all of my concerns early in the interview 9 Interviewer first asked about my general concerns, then asked about specific details 10 Interviewer greeted me warmly 11 Interviewer let me explain my problem without interruption 12 Interviewer asked about issues that affect my health, like family, culture, finances, work environment, access to care, alternative medicine, or spiritual beliefs 13 Interviewer asked how the illness affects my life at home or at work 14 Interviewer asked me if I had any questions 15 Interviewer checked to see if I was willing and able to follow through with the treatment plan 16 Interviewer clearly explained my problem and its treatment using language I could understand 17 Interviewer encouraged me to participate in treatment decisions to the extent I wished 18 Interviewer summed up and made sure he or she understood what I said 19 Interviewer tried to understand how I see my illness or problem All questions were scored on a 6-point scale ranging from 1 = strongly disagree to 6 = strongly agree they assign to students. If significant differences between raters were to be found, however, this would suggest that raters introduce a significant source of bias into the scores that students receive. Secondly, we performed a G study to more accurately quantify the amount of variance that raters introduce into scores. This analysis led to a decision study (D study) to predict how many different raters would be required to achieve a suitably reliable overall score. These results allowed us to compare the reliability of the RCRS against a similar scale used by Colliver et al. 9 Finally, we standardised each raterõs assigned scores, so that all the ratersõ distributions of assigned scores had identical means and standard deviations (SDs). We then assessed the reliability of these new standardised scores. If a significant improvement in reliability were to be found, this would suggest that standardising scores can provide a cost-effective alternative to increasing the number of SP stations as a way of improving reliability in assessing communication skills. METHODS Subjects were 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students undergo a comprehensive assessment at the end of both Years 2 and 3 of medical school. As part of this comprehensive assessment, students are exposed to eight SP cases in Year 2 and four in Year 3. After the encounter, the SP rates the student on a checklist of observed behaviours, as well as on the RCRS. Most actors return to portray the same case in multiple years. Some of the cases are changed each year in order to maintain the integrity of the cases. For the purposes of the current analysis, we analysed data only for the five cases that were common to both Year 2 assessments and the two cases common to both Year 3 assessments. Thus, we examined a total of seven cases that were used in both years. We then performed three related analyses to assess the relationship between actor effects on reliability of the test. Comparison of actors within cases For each of the seven cases, we first produced box and whisker plots that depicted the range of scores assigned by each actor, in order to provide a visual representation of the degree of variability in actorsõ 664 ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

4 Subjective communication rating scale use of the RCRS. Within each case, differences between actors were assessed by one-way analysis of variance. Generalisability and decision analyses Shavelson and Webb 19 point out that standard procedures for estimating variance components cannot be used when the objects of measurement (e.g. students) are nested within a facet of measurement (e.g. actors). They recommend the procedure followed by Shavelson et al., 20 who reported a study in which examinees rotated through several stations at which they were evaluated by one of several possible pairs of examiners. To address this situation in which examinees were nested within raters, these authors performed a separate G study for each subset of examinees, who were assessed by the same pair of examiners. This procedure was then repeated across each of the possible pairs of examiners. Each of these analyses thus represents a completely factorial design, as all examinees within each replication of the analysis were rated by the same pair of raters. (Raters are viewed as a random factor, with levels that change across each of the replications.) The results of these individual studies are then combined to yield pooled estimates and SDs of the variance terms. This procedure yields only three variance terms: students, raters and students raters. Note that there are no nested terms. To identify all possible pairs of actors, we first produced two-way frequency distributions of all 21 possible two-way combinations of the seven cases. For instance, if case A used six actors and case B used four actors, there would have been 24 possible combinations of evaluators in these two cases. We then examined each of these pairs of actors within this table, and retained those who had evaluated at least eight of the same students. Across the pairwise combinations of cases, a total of 452 actor pairs met this criterion. For each of these, we computed variance components for the factorial student actor design, and then pooled the 452 resulting variance terms to yield final estimates for each of the three model parameters (students, actors and students actors error). Because actors are completely confounded with cases, it is not possible to determine a separate variance component reflecting the cases in this design. Similarly to a jack-knife test, this procedure also allowed us to derive empirical estimates of the variability of each of the variance components. Finally, we used these estimates to compute a generalisability coefficient for an 18-station test, which allowed us to compare our results with those of Colliver and colleagues. 10,11 Effect of standardising ratings We standardised each studentõs communication rating scores for actor effects. Unlike the previous analyses, we used data from the RCRS scores the students received across all 12 cases in their particular year. We did this by first computing each actorõs mean and SD across all the students he or she rated. Because we knew which actors generated each studentõs RCRS scores, we were able to generate actor-specific Z scores for each student. These Z scores essentially place all the scores on the same metric, thereby largely eliminating actor effects. We then used the Spearman Brown prophecy formula to estimate the improvement in internal consistency across an entire OSCE examination using raw versus actor-standardised scores. RESULTS Comparison of actors within cases Figure 1 displays actorsõ distributions of scores for one of the seven cases; very similar results were found for the other six cases. As the figure indicates, actors differ widely in both their mean scores and the range of scores they assign. In many cases, some actorsõ scores do not overlap at all; in these cases, one actorõs highest-rated student may receive a lower score than another actorõs lowest-rated student. There were significant differences between raters in all seven cases (Case 1, F 12,197 = 8.91; Case 2, F 6,191 = 19.77; Case 3, F 10,185 = 9.70; Case 4, F 4,190 = 45.20; Case 5, F 8,188 = 4.35; Case 6, F 7,191 = 9.40; Case 7, F 8,193 = 5.48; all significant at P < 0.001). Generalisability and decision analyses Over the 432 replications, we found mean (SD) variance components of 0.04 (0.10) for students, 0.12 (0.22) for actors within cases, and 0.15 (0.12) for students actors. A hypothetical 18-station SP examination would have a projected reliability of These estimates are in fact quite similar to those reported by Colliver and colleagues. 10,11 We note, however, that the SDs of these estimates are large relative to the means, suggesting that the distribution ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

5 S J Lurie et al Figure 1 Individual standardised patient rating styles for Case 1 of these estimates is skewed to the right. This is particularly true of the estimate of the variance component for students. Thus these pooled estimates, which are weighted averages, may tend to overstate the actual reliability of the scale over several replications. Effect of standardising ratings If the 12 raw RCRS scores are combined, the composite total would have a CronbachÕs alpha of The reliability of the 12-item composite increases to 0.73 if the scores are first standardised within actors. Applying the Spearman Brown formula, a total of 27 raw RCRS scores would be needed to achieve a CronbachÕs alpha of 0.8, but only 18 such cases would be required to achieve that degree of reliability if the scores were first standardised within actors. DISCUSSION The RCRS was designed as a survey that could be completed by either SPs or real patients without any prior training in its use. Ideally such a scale, when utilised by an SP, would approximate the subjective impressions of real patients because these impressions are known to correlate with important outcomes such as adherence to treatment, switching doctors and health-related quality of life. Unfortunately, SPs may substantially differ from one another in the average ratings they assign to trainees, as well as in the variability of the scores they assign. Within the same case, there may be actors whose scores do not overlap at all; the highest score assigned by one actor may be lower than the lowest score assigned by another. Furthermore, some actors essentially assign every examinee the same score, whereas others use nearly the entire range of the scale. Thus, when comparing students who were rated by different SPs, scores may be more reflective of different SPsÕ biases in using the scale, rather than the studentsõ ÔtrueÕ performances. Despite this effect, we found that when such scores are combined over a relatively large number of cases, studentsõ ÔtrueÕ scores appear to emerge in a more generalisable way. Even when accounting for the possible nesting of actors within cases, we found that an 18-station OSCE would be expected to yield generalisability coefficients of 0.8 for studentsõ subjectively measured communication skills. Nonetheless, 27 such stations would be required to achieve an acceptable CronbachÕs alpha. This may 666 ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

6 Subjective communication rating scale explain why brief OSCE examinations based on SP satisfaction tend not to have high correlations with one another. 8 By the simple measure of standardising such scores within actors, only 18 such stations would be required to achieve a similar level of internal consistency. Such standardisation corrects for irrelevant sources of bias in rating style, while maintaining SPsÕ relative judgements about the communication skills of various trainees. Although many institutions may lack the ability to provide a full 18 testing stations, our results underscore the fact that standardisation of ratersõ scores is a low-cost way to improve the reliability of any test of communication skills, irrespective of the length of the examination. Thus, we recommend that such standardisation be performed irrespective of the number of stations. By removing the contaminating effect of actorõs rating styles, students can be provided with a more accurate estimate of their communication skills. Indeed, such a procedure would be concordant with the position of Swanson et al., of the National Board of Medical Examiners, who state: ÔDespite the obvious importance of adjusting scores on alternate test forms to reduce measurement error and ensure equitable treatment of examinees, equating procedures are not typically employed.õ 21 (p 67) An alternative to such standardisation of scores would be to train SPs to a reasonable level of calibration, such that SPs can agree on studentsõ levels of communication skills. 22 Such calibration, however, may also make the SPs less genuine and immediate in their responses to studentsõ interpersonal skills, and may thus bias SPs in ways that could diminish the predictive validity of the scale. 23,24 Thus, the use of rater-specific standardised scores allows each rater to express his or her own subjective reactions while maintaining fairness in grading across large groups of examinees. Of course, this procedure would also represent a highly cost-effective method of reducing the amount of resources required to obtain reliable scores of examineesõ communication skills. Our study has a number of limitations. Firstly, this analysis was performed on one SP programme within a single institution. It is possible that SPs elsewhere may be less variable in their responses. Nonetheless, we point out that SPs at our programme undergo a lengthy and rigorous training programme in order to depict these cases in a highly standardised way. Furthermore, because actors within each case are chosen to represent a specific patient, they are similar in terms of age, sex and racial or ethnic background. Another limitation of our study relates to the fact that we examined only seven SP cases, which represented a range of patient presentations. It is possible that an examination comprising more similar cases would find a higher reliability across cases, and thus less effect of individual rating styles. Secondly, we did not ask faculty observers to rate these interactions and thus it is impossible to assess the degree to which faculty may be more or less reliable observers than SPs. Finally, we must point out that the generalisability of the RCRS appears to be at least as high than that of the scale used by Colliver et al. 10,11 The RCRS requires an average of 2 3 minutes to complete, and has now been used several thousand times by our SPs in assessing communication skills of medical students. Thus, the RCRS appears to be a feasible, reliable and low-cost way of assessing these important skills, particularly if scores are normalised within each assessor. Contributors: SJL contributed to study design and the drafting of the article. CM, SM and RME contributed to the analysis and interpretation of data, and the critical revision of the manuscript. ACN contributed to the collection, analysis and interpretation of data, and the critical revision of the manuscript. Acknowledgements: none. Funding: none. Conflicts of interest: none. Ethical approval: this study was approved by the University of Rochester Institutional Review Board. REFERENCES 1 Association of American Medical Colleges. Medical School Outcome Project. msop. [Accessed 16 April 2008.] 2 National Board of Medical Examiners. Step 2 Clinical Skills Examination. steps2/step2.html. [Accessed 17 May 2007.] 3 Krupat E, Frankel R, Stein T, Irish J. The Four Habits Coding Scheme: validation of an instrument to assess cliniciansõ communication behaviour. Patient Educ Couns 2006;62 (1): van Thiel J, Kraan HF, van der Vleuten CP. Reliability and feasibility of measuring medical interviewing skills: the revised Maastricht History-taking and Advice Checklist. Med Educ 1991;25 (3): Makoul G. The SEGUE Framework for teaching and assessing communication skills. Patient Educ Couns 2001;45 (1): ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42:

7 S J Lurie et al 6 Roter D, Larson S. The Roter Interaction Analysis System (RIAS): utility and flexibility for analysis of medical interactions. Patient Educ Couns 2002;46 (4): Guiton G, Hodgson CS, Delandshere G, Wilkerson L. Communication skills in standardised-patient assessment of final-year medical students: a psychometric study. Adv Health Sci Educ Theory Pract 2004;9 (3): Chessman AW, Blue AV, Gilbert GE, Carey M, Mainous AG III. Assessing studentsõ communication and interpersonal skills across evaluation settings. Fam Med 2003;35 (9): Colliver JA, Robbs RS, Vu NV. Effects of using two or more standardised patients to simulate the same case on case means and case failure rates. Acad Med 1991;66 (10): Colliver JA, Morrison LJ, Markwell SJ, Verhulst SJ, Steward DE, Dawson Saunders E, Barrows HS. Three studies of the effect of multiple standardised patients on intercase reliability of five standardised-patient examinations. Teach Learn Med 1990;2 (4): Colliver JA, Marcy ML, Vu NV, Steward DE, Robbs RS. Effect of using multiple standardised patients to rate interpersonal and communication skills on intercase reliability. Teach Learn Med 1994;6 (1): Epstein RM, Dannefer EF, Nofziger AC, Hansen JT, Schultz SH, Jospe N, Connard LW, Meldrum SC, Henson LC. Comprehensive assessment of professional competence: the Rochester experiment. Teach Learn Med 2004;16 (2): Willliams GC, Freedman Z, Deci EL. Promoting motivation for diabeticsõ self-regulation of HgbAlc. Diabetes 1997;45: Greenfield S, Kaplan SH, Ware JE Jr, Yano EM, Frank HJL. PatientsÕ participation in medical care: effects on blood sugar control and quality of life in diabetes. J Gen Intern Med 1995;3: Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physician patient interactions on the outcomes of chronic disease. Med Care 1989;27 (Suppl): Safran DG, Taira DA, Rogers WH, Kosinski M, Ware JE, Tarlov AR. Linking primary care performance to outcomes of care. J Fam Pract 1998;47: Safran DG, Kosinski M, Tarlov AR, Rogers WH, Taira DH, Lieberman N, Ware JE. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care 1998;36: Safran DG, Montgomery JE, Chang H, Murphy J, Rogers WH. Switching doctors: predictors of voluntary disenrolment from a primary physicianõs practice. J Fam Pract 2001;50: Shavelson RJ, Webb NM. Generalizability Theory. Newbury Park, CA: Sage Publications 1991; Shavelson RJ, Mayberry PW, Li W, Webb NM. Generalisability of job performance measurements: marine corps riflemen. Mil Psychol 1990;2 (3): Swanson DB, Clauser BE, Case SM. Clinical skills assessment with standardised patients in high-stakes tests: a framework for thinking about score precision, equating, and security. Adv Health Sci Educ 1999;4: Srinivasan M, Franks P, Meredith LS, Epstein RM, Kravitz RL. Connoisseurs of care? Unannounced standardised patientsõ ratings of physicians Med Care 2006;44 (12): Fiscella K, Franks P, Srinivasan M, Kravitz RL, Epstein R. Ratings of physician communication by real and standardised patients. Ann Fam Med 2007;5 (2): Filipetto FA, Weiss LB, Switala CA, Bertagnolli JF Jr. The effectiveness of a first-year clinical preceptorship on the data collection and communication skills of second-year medical students. Teach Learn Med 2006;18 (2): Received 4 June 2007; editorial comments to authors 19 October 2007, 20 December 2007; accepted for publication 3 January ª Blackwell Publishing Ltd MEDICAL EDUCATION 2008; 42: