Evaluating the effectiveness of rating instruments for a communication skills assessment of medical residents

Transcription

1 Adv in Health Sci Educ (2009) 14: DOI / ORIGINAL PAPER Evaluating the effectivene of rating intrument for a communication kill aement of medical reident Cherdak Iramaneerat Æ Carol M. Myford Æ Rachel Yudkowky Æ Tali Lowentein Received: 28 June 2008 / Accepted: 17 October 2008 / Publihed online: 5 November 2008 Ó Springer Science+Buine Media B.V Abtract The invetigator ued evidence baed on repone procee to evaluate and improve the validity of core on the Patient-Centered Communication and Interperonal Skill (CIS) Scale for the aement of reident communication competence. The invetigator retropectively analyzed the communication kill rating of 68 reident at the Univerity of Illinoi at Chicago (UIC). Each reident encountered ix tandardized patient (SP) portraying ix cae. SP rated the performance of each reident uing the CIS Scale an 18-item rating intrument aking for level of agreement on a 5-category cale. A many-faceted Rach meaurement model wa ued to determine how effectively each item and cale on the rating intrument performed. The analye revealed that item were too eay for the reident. The SP underutilized the lowet rating category, making the cale function a a 4-category rating cale. Some SP were inconitent when aigning rating in the middle categorie. The invetigator modified the rating intrument baed on the finding, creating the Revied UIC Communication and Interperonal Skill (RUCIS) Scale a 13-item rating intrument that employ a 4-category behaviorally anchored rating cale for each item. The invetigator implemented the RUCIS Scale in a ubequent communication kill OSCE for 85 reident. The analye revealed that the RUCIS Scale functioned more effectively than the CIS Scale in everal repect (e.g., a more uniform ditribution of rating acro categorie, and better fit of the item to the meaurement model). However, SP till rarely aigned rating in the lowet rating category of each cale. C. Iramaneerat (&) Department of Surgery, Faculty of Medicine, Siriraj Hopital, Mahidol Univerity, Siamindra Building 12th fl., 2 Prannok Rd. Bangkoknoi, Bangkok 10700, Thailand icir@mahidol.ac.th C. M. Myford Department of Educational Pychology, College of Education, Univerity of Illinoi at Chicago, Chicago, IL, USA R. Yudkowky T. Lowentein Department of Medical Education, College of Medicine, Univerity of Illinoi at Chicago, Chicago, IL, USA

2 576 C. Iramaneerat et al. Keyword Validity Rating cale Communication kill Many-faceted Rach meaurement OSCE Introduction Communication and interperonal kill are one of the ix core competencie for which reidency program have to demontrate training outcome (Accreditation Council for Graduate Medical Education 1999). An aement of reident communication kill that can provide valid inference about their ability to exchange information and ally with patient require an oberved interaction with patient. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialtie (ABMS) recommend uing an aement format that ak reident to interact with tandardized patient (SP) in an Objective Structured Clinical Examination (OSCE) a the mot deirable approach for communication kill aement (Bahook and Swing 2000). The rating intrument that a tandardized patient ue to record hi/her obervation of a reident performance during a communication kill OSCE play a critical role in providing valid inference from an aement. A rating intrument not only guide the obervation but alo dictate the coring of the performance of individual reident. Several rating intrument for the aement of medical communication kill by SP in OSCE etting have been developed and validated, including the Interperonal and Communication Skill Checklit (Cohen et al. 1996), the Interperonal Skill Rating Form (Schnabl et al. 1991), the Arizona Clinical Interview Rating Scale (Stillman et al. 1976, 1986), the Brown Univerity Interperonal Skill Evaluation (Burchard and Rowland-Morin 1990), the SEGUE Framework (Makoul 2001), the Liverpool Communication Skill Aement Scale (LCSAS) (Humphri 2002; Humphri and Kaney 2001), and the Patient-Centered Communication and Interperonal Skill (CIS) Scale (Yudkowky et al. 2004, 2006). Depite the many available rating intrument for communication kill aement in OSCE etting, chooing an appropriate intrument to core reident performance in a communication kill OSCE i not an eay tak. Validity evidence that upport the ue of core obtained from thee rating intrument i quite limited. Reearcher conducting validity tudie of thee intrument have focued mainly on reporting internal conitency reliability, inter-rater reliability, and correlation of core with meaure of other variable. According to the Standard for Educational and Pychological Teting (American Educational Reearch Aociation et al. 1999), uch validity reearch only provide evidence baed on internal tructure and relation to other variable, leaving out evidence baed on tet content, repone procee, and conequence. In thi tudy, we evaluated validity evidence related to the ue of one of the exiting communication kill OSCE rating intrument the Patient-Centered Communication and Interperonal Skill (CIS) Scale. We focued on evidence baed on repone procee, a ource of validity evidence that tet core uer often overlook. In the context of a communication kill OSCE, the validity evidence baed on repone procee refer to the evaluation of the extent to which the SP apply rating criteria to rate the reident performance in a manner that i conitent with the intended interpretation and ue of core (American Educational Reearch Aociation et al. 1999). There are everal approache that reearcher can ue to gather validity evidence baed on repone procee. Reearcher can collect ome piece of evidence before the OSCE

3 Rating intrument for communication kill aement 577 adminitration (e.g., documenting the rating criteria and the procee for electing, training, and qualifying SP). Reearcher can collect other piece of evidence at the time a SP rate the performance (e.g., engaging SP in verbal think-aloud during the rating proce, thu allowing reearcher to know what SP are thinking while deciding what rating they will aign (Heller et al. 1998)). The focu of thi tudy wa on gathering validity evidence related to repone procee after an OSCE adminitration (i.e., when all the rating were available to u). That i, we carried out a pychometric analyi of rating to invetigate to what extent the OSCE rating were conitent with the intended ue of the core. OSCE rating are the reult of the interaction between reident, cae, item (and their rating cale), and SP. A comprehenive validity tudy of repone procee for an OSCE would require cloe examination of repone related to all thee component of an OSCE. In thi tudy, we limited the cope of our analye to repone procee related to item and cale on the rating intrument. That i, we invetigated the extent to which SP ued the rating intrument to rate the reident performance in a way that wa conitent with the intended ue of the core. Thi tudy looked at the ue of the CIS Scale in the coring of internal medicine reident performance in communication kill OSCE carried out at the Univerity of Illinoi at Chicago (UIC). The purpoe of our tudy were (1) to evaluate the effectivene of the CIS Scale in coring the reident performance in the communication kill OSCE, (2) to ue the finding obtained from the analyi to determine whether the rating intrument needed to be revied to improve it effectivene, (3) to ue the reult from the analyi to guide the intrument reviion proce, and (4) to compare the original CIS Scale to the modified rating intrument to determine whether the modification helped improve the cale functioning, thu in effect enhancing the validity of the inference made from core on the communication kill OSCE. In the coure of evaluating the effectivene of thee two rating intrument, we demontrate how reearcher can analyze OSCE rating data to provide validity evidence related to repone procee. Method Reearch deign We carried out the tudy in two phae. The firt phae wa a retropective analyi of the communication kill OSCE rating for internal medicine reident obtained in 2003, in which SP ued the CIS cale to rate the performance of reident. We identified certain item and cale on that rating intrument that did not function effectively and revied the rating intrument to addre thoe weaknee. We piloted the revied intrument with a mall group of SP trainer and medical faculty member and then further revied the intrument baed on the comment obtained from the pilot tudy. Thi led to a development of a revied rating intrument for communication kill aement called the Revied UIC Communication and Interperonal Skill (RUCIS) cale. In the econd phae of the tudy, we implemented the RUCIS cale in the 2007 communication kill OSCE for internal medicine reident. We carried out an analyi to evaluate the effectivene of the revied rating intrument in order to determine whether the intrument modification helped improve the effectivene of the intrument. Both the 2003 and 2007 communication kill OSCE were mandatory formative aement conducted a part of the tandard curriculum of the reidency program.

4 578 C. Iramaneerat et al. Participant Participant in the 2003 communication kill OSCE included 68 internal medicine reident (51% PGY-2 and 49% PGY-3; 66% male and 34% female) and 8 SP (38% male and 62% female). Participant in the 2007 communication kill OSCE included 85 internal medicine reident (54% PGY-1 and 46% PGY-2; 47% male and 53% female) and 17 SP (29% male and 71% female). Rating intrument The CIS Scale, which SP ued to rate the performance of reident in the 2003 communication kill OSCE, i an 18-item rating intrument. Each item ak SP to provide an agreement rating uing a 5-category rating cale, in which 1 correpond to trongly diagree and 5 correpond to trongly agree. Since all item are tatement of deirable communication behavior, higher rating indicate higher level of communication competence (See Appendix A). The RUCIS Scale, which SP ued to rate the performance of reident in the 2007 communication kill OSCE, i a 13-item rating intrument. Each item contain a hort decription of the particular apect of communication under conideration and four behaviorally anchored rating categorie unique to each item. For each item, the lowet rating category alway decribe the leat appropriate behavior for that apect of communication, while the highet rating category alway decribe the mot appropriate behavior for that apect. In addition to the four rating categorie for each item, ix item alo have a not applicable option that SP could ue when they did not oberve the behavior related to that apect of communication (See Appendix B). SP training In the 2003 communication kill OSCE, all the SP took part in an intenive training program to learn how to portray the cae and how to rate reident performance before participating in the OSCE. The training program included a review and dicuion of the cae cript and repeatedly practicing the appropriate portrayal of the cae under the uperviion of a trainer. Training on the CIS cale included a review and dicuion of the cale and practice uing it to rate a videotaped or oberved performance. There wa no attempt to reach agreement between the SP and trainer in the rating they aigned, but divergent rating were noted and dicued. The trainer enured that each SP could portray the cae conitently and rate the performance of reident to the trainer atifaction before the SP wa allowed to participate in the communication kill OSCE. In the 2007 communication kill OSCE, all the SP alo took part in an intenive SP training program imilar to the training for the 2003 communication kill OSCE to enure an accurate portrayal of the cae before participating in the OSCE. However, thi time we employed a frame-of-reference (FOR) approach in training the SP to provide rating (Bernadin and Buckley 1981). Prior to training, a group of SP trainer reviewed elected videotaped OSCE eion and provided a conenu gold tandard rating for each item in each encounter. During the training eion SP rated the elected videotaped OSCE eion uing the RUCIS cale, compared their rating to the trainer gold tandard rating, and dicued the rationale for the gold tandard. By practicing and receiving feedback from everal videotaped OSCE eion, the SP developed a common rating tandard (i.e., frame) by which to evaluate reident performance.

5 Rating intrument for communication kill aement 579 OSCE adminitration Both OSCE employed the ame cae and the ame adminitration format. Six reident were aeed in each half-day eion. In each eion, each reident encountered ix different SP in ix different clinical cenario (cae). In each cae, reident pent 10 min in the encounter with the SP, 5 min reviewing tak-related educational material while the SP rated the performance, and another 5 min receiving verbal feedback from the SP. The tak-related educational material conited of printed document decribing effective way to interact with a patient in the ituation they jut encountered. The verbal feedback eion provided SP and reident with the opportunity to dicu effective and ineffective behavior oberved during the encounter, and to practice technique that the SP uggeted. The SP did not inform the reident of hi/her pecific rating. The ix communication tak that reident encountered were: (1) providing patient education, (2) obtaining informed conent, (3) dealing with a patient who refue treatment, (4) couneling an elderly patient who ha been abued, (5) giving bad new to a patient, and (6) conducting a phyical examination. We repeated the OSCE eion once or twice a week until all reident had the opportunity to participate in the OSCE, which took 2 and 4 month, for the 2003 and 2007 communication kill OSCE, repectively. Analye Becaue the OSCE i a multi-faceted aement method where the rating of a reident performance depend upon many factor, including the communication competence of the reident, the difficulty of the item on the rating intrument, the everity of the SP, and the difficulty of the cae, we ued a many-faceted Rach meaurement (i.e., Facet) model (Linacre 1989) to analyze the data. The Facet model ue a logarithmic function of the odd of receiving a rating in a given category a compared to the odd of receiving a rating in the next lower category to define the communication competence of reident, the difficulty of item, the everity of SP, and the difficulty of cae. All meaure of thee four facet are reported on the logit cale, which i a linear, equal interval cale. Higher logit meaure indicate more competent reident, more difficult item, more evere SP, and more difficult cae. Becaue there were multiple rating categorie for each item, the Facet model alo calculated a et of tep threhold for each item. (A tep threhold i the tranition point between two adjacent categorie, where the probabilitie of receiving a rating in the two categorie are equal.) We ued the Facet computer program (Linacre 2005) to conduct the analye. To enure that the analye to obtain validity evidence baed on repone procee would be baed on reliable data, we firt examined the degree of reproducibility of reident communication competence meaure validity evidence related to the internal tructure of tet core. We calculated a meaure of internal conitency reliability, the reident eparation reliability, which i an index analogou to KR-20 or Cronbach Alpha. Becaue rating of multiple item on the ame cae by the ame SP can be dependent on one another, which could lead to overetimation of reliability (Sireci et al. 1991; Thien et al. 1989), we ued cae (rather than item) a coring unit. That i, we averaged the rating a SP gave to all item in a given cae to produce a cae core, which we conidered a one rating in the Facet analyi. An effective rating intrument for an OSCE hould produce rating that atify two tet related to repone procee. The firt one involve determining whether each rating cale functioned appropriately (i.e., were the categorie on the cale that the SP ued

6 580 C. Iramaneerat et al. well-defined, mutually excluive, and exhautive). The econd one involve determining whether each item on the rating intrument functioned properly (i.e., when evaluating each reident performance, did SP aign rating for each item in a conitent fahion). We ued the following ix criteria (Linacre 2004) a guideline for determining whether each rating cale category for each item functioned effectively (i.e., to determine whether the rating categorie of each item were well-defined, mutually excluive, and exhautive): (1) There hould be at leat 10 rating in each rating category to allow accurate calibration of tep threhold. (2) The frequency ditribution of rating acro categorie hould have a uniform or unimodal pattern. If SP ue only a few of the rating categorie and rarely ue other rating categorie, the reulting irregular ditribution of rating indicate a poorly functioning cale that cannot effectively differentiate reident according to their level of communication competence. (3) The average meaure of reident communication competence hould increae a the rating categorie increae. In other word, reident who receive rating in higher categorie hould have higher overall communication competence meaure than thoe who receive rating in lower categorie. (4) The tep threhold hould increae a the rating categorie increae. Thi criterion mirror the third criterion. Failure of tep threhold to increae a the rating categorie increae i called tep diordering, which ugget that SP may have difficulty differentiating the performance of reident in thoe categorie. One or more of the rating categorie for a particular item may not be clearly defined. (5) The tep threhold hould advance at leat 1 logit, but not more than 5 logit. The finding that two tep threhold advance by le than 1 logit would ugget that thoe two rating categorie are practically ineparable. That i, SP may not be able to reliably differentiate between them. On the other hand, tep threhold that are too far apart are an indication of a poible dead zone on the cale where meaurement loe it preciion. (6) The outfit mean-quare value for each rating category hould be le than 2.0. An outfit mean-quare value i a tatitical index that indicate how well the rating in each category fit the meaurement model. It value can range from 0 to infinity, with an expected value of 1. A high outfit mean-quare value for a rating category i an indicator that ome SP ued that rating category in an unexpected or urpriing manner that wa inconitent with the way that other SP ued that category. In addition to evaluating the functioning of the cale categorie, we evaluated fit tatitic for each item on the intrument to determine whether SP provided aberrant rating on any item, which might indicate problematic repone procee. Thee fit tatitic are indice that indicate how well the rating data for each item fit the meaurement model. In thi tudy, we examined both outfit and infit mean-quare tatitic for each item. We calculated an outfit mean-quare value for each item by dividing the um of the quared tandardized reidual for the item by it degree of freedom. (A reidual i the difference between the rating a SP aigned a reident on an item and the rating the meaurement model predicted the SP would aign.) Thi calculation produce a value that can range from 0 to infinity, with an expectation of 1.0. Value larger than 1.0 indicate the preence of unmodeled noie in the rating for that item (i.e., unexpected rating that SP aigned when evaluating reident, given how SP aigned rating for other item). By contrat, value le than 1.0 indicate that there wa too little variation in the rating SP aigned for that item (Linacre and Wright 1994; Wright and Mater 1982). However, outfit

7 Rating intrument for communication kill aement 581 mean-quare value are very enitive to outlier rating. To reduce the influence of outlier rating, we weighted each quared tandardized reidual by it information function before we ummed them. (Thi involved applying differential weight to tandardized reidual. That i, reidual that reulted from SP rating of item and cae that were far too eay or too difficult for reident received le weight than thoe that reulted from SP rating of item and cae that were at the appropriate difficulty level for reident.) Thi calculation produced an infit mean-quare tatitic that ha the ame ditribution and interpretation a an outfit mean-quare tatitic, but i more immune to the influence of the rating for reident on item or cae that are far too eay or difficult for them. Wright and Linacre (1994) recommended that an appropriate mean-quare fit tatitic for judge-mediated rating hould be in the range of From the analyi of the 2003 communication kill OSCE rating, we identified the item and rating categorie on the CIS Scale that did not function effectively according to one or more of thee criteria. We then ued thee finding to guide the development of a modified rating intrument the RUCIS Scale. We implemented the RUCIS Scale in the 2007 communication kill OSCE and evaluated the effectivene of the revied intrument uing the ame criteria to determine whether the modification helped improve the effectivene of the intrument, thu in effect enhancing the validity of the core interpretation. Reult Evaluating the effectivene of the CIS cale The analyi of the 2003 communication kill OSCE revealed that thi group of reident wa highly competent relative to the item and cae on the CIS Scale (Table 1). The average reident communication competence meaure wa higher than the average item and cae difficulty meaure, and there were few item or cae appropriate for meauring the communication competence of reident who were in the upper range of the communication competency continuum (i.e., in the logit range). Thee finding ugget that thee item and cae were not very well uited to meauring the communication competence of thi group of reident (i.e., it wa a relatively eay aement for them). Uing cae a coring unit, our analyi yielded a reident eparation reliability of Table 1 Summary of meaure obtained from the analyi of the communication kill OSCE Meaurement facet Minimum (logit) Maximum (logit) Mean (logit) SD (logit) A Communication kill OSCE Reident competence Item difficulty Cae difficulty B Communication kill OSCE Reident competence Item difficulty Cae difficulty

8 582 C. Iramaneerat et al. The problematic alignment between the reident communication competence meaure and the item and cae difficulty meaure i clearly demontrated in Fig. 1, a implified contruct map we obtained from the analyi, howing the relationhip between three of the four facet in our analyi. The firt column diplay the equal interval logit meaure. The econd column how the reident communication meaure. More competent reident appear higher in the column, while le competent reident appear lower in the column. The third and fourth column how the cae and item ordered by difficulty. Difficult item and cae appear higher in the column (e.g., item 18, 16, and 9, and cae 3 and 6), while eay item and cae appear lower in the column (e.g., item 1, 12, and 2, and cae 4). Column 5 22 how how SP ued the 5-category rating cale for each item on the CIS Scale. A horizontal line acro a column indicate the point at which the probability of a reident receiving the next higher rating begin to exceed the probability of receiving the lower rating (i.e., a tep threhold). According to thi contruct map, the mot likely rating that SP aigned on all the CIS item wa a 4 or 5. Another intereting finding i SP did not ue the cale in an identical fahion when aigning rating on thee 18 item, a demontrated by having different range of communication competence meaure for each rating category acro item (e.g., the region of rating category 3 i not the ame acro all item). The analyi of the 2003 communication kill OSCE revealed that the CIS Scale did not meet everal of Linacre (2004) guideline for evaluating rating cale category effectivene. We ummarized thee reult in Table 2. Firt, only five item had more than 10 rating in all five categorie. The 5-category agreement rating cale actually functioned a a 4-category rating cale. SP rarely aigned rating of 1. The item on the CIS Scale Step threhold eparating rating categorie for item Logit +Reident -Cae -Item (5)+(5)+(5)+(5)+(5)+(5)+(5)+(5)+(5)+ (5)+ (5)+ (5)+ (5)+ (5)+ (5)+ (5)+ (5)+ (5)+ * * --- *** *** **** ***** **** ****** ***** --- ***** ******* * ******* ****** *** 4 4 **** ** * * 0 * * * * * * * * * * 3 * * * 3 * * * 3 * 3 * * * * * (1)+(1)+(1)+(1)+(1)+(1)+(1)+(1)+(2)+ (1)+ (1)+ (1)+ (1)+ (1)+ (1)+ (1)+ (1)+ (1) Logit *=1peron -Cae -Item Fig. 1 A contruct map howing the mialignment between the reident communication competence meaure and the item and cae difficulty meaure for the 18 item on the CIS Scale

9 Rating intrument for communication kill aement 583 Table 2 Comparing the functioning of the CIS cale (2003) and RUCIS cale (2007) uing Linacre (2004) guideline CIS cale 5-category cale 18 item RUCIS cale 4-category cale 13 item Reident eparation reliability Criteria for evaluating the functioning of the categorie on each rating cale At leat 10 rating in each category 5 item (28%) 6 item (46%) Uniform/unimodal ditribution of rating acro 1 item (6%) 12 item (92%) categorie Reident with higher category rating have higher 7 item (39%) 12 item (92%) overall communication competence meaure No tep diordering 9 item (50%) 11 item (85%) Step threhold advance by at leat 1 logit, but not 1 item (6%) 10 item (77%) more than 5 logit An outfit mean-quare value \2.0 for each rating category 11 item (61%) 13 item (100%) Criteria for evaluating the functioning of the item on the intrument Outfit mean-quare value \ item (78%) 12 item (92%) Infit mean-quare value \ item (89%) 13 item (100%) appeared to be relatively eay for thee reident, reulting in an unbalanced ditribution of rating acro the five rating categorie: about 70 80% of all rating were 4 or 5. The only item that exhibited an acceptable rating ditribution wa item 18, which howed a unimodal ditribution that peaked in the middle categorie. The analyi alo revealed that ome SP experienced difficulty in differentiating between the middle categorie of the 5-category agreement cale, a demontrated by the failure of the average meaure and tep threhold to increae properly along with the rating categorie. Only even item (item 4, 5, 6, 9, 11, 16, and 18) exhibited proper advancement of average reident communication competence meaure a the rating categorie increaed. Nine item (item 2, 4, 6, 10, 11, 13, 14, 15, and 17) howed diordered tep threhold. Seven item (item 7, 8, 9, 10, 13, 14, and 15) had one or more rating categorie with outfit mean-quare value equal to or greater than 2, reflecting inconitent ue of the categorie. Only one item (item 12) had all adjacent tep threhold eparated by at leat one logit. The other 17 item had one or more tep threhold that were too cloe to adjacent threhold, epecially for tep threhold in the middle of the cale. However, none of the 18 item had tep threhold that advanced by more than five logit, uggeting that there were no ignificant gap between the categorie. We ummarized item fit tatitic in Table 3. Four item (item 3, 5, 10, and 15) had outfit mean-quare value higher than 1.2, indicating that ome SP aigned rating for thoe item that were unexpectedly high or low, given the other rating that the SP aigned. Item 10 and 15 had infit mean-quare value higher than 1.2. A cloe examination of the unexpected rating for item 10 (I felt you encouraged me to ak quetion) and 15 (I felt you were careful to ue plain language) revealed that ix out of eight SP were inconitent in rating item 10, and even out of eight SP were inconitent in rating item 15. Apparently, the SP did not have a hared undertanding of what they were evaluating when uing thee two item. Thi finding uggeted that we needed to revie thee item to make them clearer to SP.

10 584 C. Iramaneerat et al. Table 3 Summary of item fit tatitic Item fit tatitic Minimum Maximum Mean SD A Communication kill OSCE Outfit mean-quare value Infit mean-quare value B Communication kill OSCE Outfit mean-quare value Infit mean-quare value Modifying the rating intrument The finding from our validity tudy revealed that there were everal apect of the CIS Scale that did not function well. Uing thee finding a our guide, we worked with medical faculty and SP to revie the CIS Scale in everal way. Intead of uing a ingle Likerttyle agreement rating cale that wa applicable to all item on the intrument, we devied a behaviorally anchored rating cale (BARS) (Bernardin and Smith 1981; Smith and Kendall 1963) that provided a detailed decription of the pecific communication behavior characteritic of each rating category for each item. Our expectation wa that the change in the cale format would make each rating cale more pecific to the context of a particular item and le open to SP idioyncratic interpretation. Becaue our analyi revealed that the lowet rating category on the CIS Scale wa a non-functioning category, we decided to change the cale format from 5-category cale to 4-category cale. To addre the problem of an unbalanced rating ditribution in which 70 80% of rating were poitive rating, while only 20 30% of rating were neutral or negative rating, we developed 4-category cale that were aturated on the poitive ide. In other word, we created a eparate rating cale for each item with only one category decribing inadequate performance and three categorie decribing atifactory communication behavior that exemplified progreively higher level of performance. We alo provided a not applicable option for ix item. Our goal wa to eliminate ome unexpected rating that SP aigned in the neutral category of the agreement cale when they found themelve unable to rate a certain apect of communication during the encounter becaue they did not oberve any evidence that the reident engaged in that apect. Although we did not change the content coverage of the rating intrument, we revied the item to eliminate redundancy and improve their clarity. We combined into one item the redundant item that addreed the ame apect of communication. Specifically, we combined item 1 and 2 into an item on friendly communication; combined item 7, 8, and 9 into an item on dicuion of option; combined item 10, 11, and 12 into an item on encouraging quetion; and combined item 13 and 14 into an item on providing a clear explanation. We created a new item on phyical examination to allow SP to eparate the act of providing an explanation of a phyical examination from the act of providing an explanation about medical condition. Finally, we attempted to make everal item more difficult by requiring that reident demontrate communication behavior that are more ophiticated and/or difficult to perform to qualify for a rating in the highet category. Thee modification led to the development of a revied rating intrument, called the RUCIS Scale (Appendix B), which we later ued in the coring of reident performance in the 2007 communication kill OSCE.

11 Rating intrument for communication kill aement 585 Evaluating the effectivene of the RUCIS cale The analyi of the 2007 communication kill OSCE revealed that thi et of item wa better targeted for meauring the communication competence of the reident (See Table 1 and column 2 4 of Fig. 2). The ditribution of reident communication competence meaure wa better aligned with the ditribution of item and cae difficulty meaure than wa the cae for the 2003 communication kill OSCE. Uing cae a coring unit, our analyi yielded a reident eparation reliability of Depite fewer number of item on the RUCIS Scale, the rating on thi revied intrument could achieve the ame level of internal conitency reliability a the level obtained from the CIS Scale. Table 2 provide a point-by-point comparion of the finding from our analye of the functioning of the CIS Scale and the RUCIS Scale. We found that even item on the revied intrument till had fewer than 10 rating aigned in the lowet category. Beyond thi, nearly all the item and rating cale appearing on the RUCIS Scale atified Linacre criteria. All item but one had a uniform ditribution of rating that peaked in the middle or at the high end. Item 5 (interet in me a a peron) wa the only item that had a rating ditribution that peaked in rating category 1. Item 2 (repectful treatment) wa the only item that did not how increaing average meaure a rating categorie increaed. The rating categorie for all item fit the meaurement model (i.e., all outfit mean-quare value for the rating categorie were le than 2). Item 7 and 12 were the only two item with diordered tep threhold. Some of the ditance between tep threhold for Item 5, 6, and 10 were too narrow (i.e., le than one logit apart). However, all the tep threhold for Step threhold eparating rating categorie for item Logit +Reident -Cae -Item (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) + (4) * * ** * *** ***** *** * ** ********* ******* 3 ****** 3 **** ********** 6 3 ***** 11 ****** 7 9 ***** 3 5 * 0 * **** * 1 2 * 6 * * * * * --- * * * * 3 * * * * * ******* ** * (1) + (1) + (1) + (1) + (1) + (1) + (1) + (1) + (2) + (1) + (1) + (1) + (1) Logit * = 1 peron -Cae -Item Fig. 2 A contruct map howing the better alignment between the reident communication competence meaure and the item and cae difficulty meaure for the 13 item on the RUCIS Scale

12 586 C. Iramaneerat et al. the other item were appropriately ordered and advanced by more than one logit but le than five logit. We ummarized item fit tatitic obtained from the analyi of the 2007 communication kill OSCE in Table 3. All item howed good fit to the meaurement model according to their infit mean-quare value. Item 5 (interet in me a a peron) wa the only item with an outfit mean-quare value higher than 1.2, indicating too much unexplained variance in the rating that SP aigned for thi item. Thu, it wa the only item that needed cloe examination to try to determine what made it difficult for SP to ue the item behaviorally anchored rating cale to aign conitent rating. Dicuion Thi tudy demontrated the proce of uing validity evidence obtained from a Facet analyi to help revie an aement intrument uch a an OSCE rating cale. Validation i a continuing proce of gathering and evaluating variou ource of evidence to determine whether that evidence upport (or refute) the propoed core interpretation. The two phae of thi tudy correpond to the two tage of validation that Kane (2006) decribed. In the firt phae of the tudy, we focued on finding way to build a meaurement intrument that poeed appropriate pychometric propertie that would upport the intended ue of OSCE core. Thi phae correpond to the development tage of validation. In the econd phae, we critically evaluated whether the newly developed rating intrument actually functioned a predicted. Thi phae correpond to the appraial tage of validation. In the firt phae of our tudy, validity evidence baed on repone procee helped u identify everal apect of the CIS Scale that did not function a intended. The validity evidence uggeted that the 5-category Likert-tyle agreement cale functioned a an unbalanced 4-category rating cale (i.e., mot of the rating were poitive rating, while only a few rating were neutral or negative). Thi finding indicated that the item on the CIS Scale were too eay for thi ample of reident. Reult from our analye alo uggeted that ome SP were unable to differentiate performance in the middle categorie of the cale. Additionally, we found that ome SP aigned a number of urpriing or unexpected rating for item 10 (I felt you encouraged me to ak quetion) and for item 15 (I felt you were careful to ue plain language), uggeting that thee SP were not able to conitently apply the rating criteria for thee two item to rate ome reident performance. All thee piece of validity evidence provided ueful information to guide the development of a revied rating intrument in our attempt to addre thee weaknee of the CIS Scale. In the econd phae of our validity tudy, we implemented the revied intrument in a later adminitration of the communication kill OSCE and carried out the ame type of analye that had revealed the inadequacie of the CIS Scale. We conidered thi a a tet of whether the revied intrument could withtand the ame validity challenge a it predeceor. We found that in many apect the RUCIS Scale helped improve core interpretability. The SP more conitently applied the rating criteria to rate reident performance. The item on the RUCIS Scale fit the meaurement model quite well. Providing a clear decription of communication behavior that wa appropriate for each rating category for the two mifitting item on the CIS Scale (item 10 and 15) helped eliminate confuion among SP in rating thee two apect of communication (a demontrated by good item fit tatitic for item 7 and 10 on the RUCIS Scale).

13 Rating intrument for communication kill aement 587 However, the modification we made to the rating intrument did not addre all the validity iue we identified in the CIS Scale. There wa one area in which the revied intrument did not how ignificant improvement over it predeceor. When uing the behaviorally anchored rating cale, SP till aigned only a few rating in the lowet rating category of many item. Thi could be due to a retricted range of communication competence among the particular ample of reident aeed. We developed the RUCIS Scale with a broad range of communication competence in mind from very incompetent phyician to very competent phyician. The ubject included in the 2007 communication kill OSCE were a ingle group of reident in one reidency program. Thi limited the range of obervable communication kill that SP were likely to ee. If we were to ae a broader range of ubject, ranging from medical tudent in their early year of training to experienced phyician practicing in variou pecialtie from geographically divere medical etting, the SP would be more likely to oberve a broader range of communication behavior and would be more likely to employ the full range of rating categorie appearing on each behaviorally anchored rating cale. Teting thi hypothei would require that reearcher conduct additional tudie to evaluate validity generalization (American Educational Reearch Aociation et al. 1999). That i, we are uggeting that reearcher carry out tudie to determine the extent to which variation in ituational facet (e.g., reident from different reidency program, different SP, etc.) may affect the aignment of rating. The tudie would help u determine how generalizable the reult we obtained are acro ubject that differ in education and experience, and acro SP. Another poible explanation for non-uniform ditribution of rating i that SP may have been uncomfortable aigning very low rating to reident. If thi were the cae, then SP trainer could addre thi iue during the training, helping SP undertand that it i appropriate (and expected) that they will aign low rating if they ee evidence of phyician behavior that warrant thoe rating. However, we would be a bit cautiou in following thi criterion too trictly. For a formative aement or in a ummative aement where reident had not been properly trained, a uniform ditribution of rating i to be expected. However, in a ummative aement where the majority of reident have practiced the kill o that they are well prepared for the communication tak, a kew ditribution of rating where only few reident would have rating in lower categorie can be obtained, which might not ugget a problem with the rating intrument. The evaluation of item fit tatitic for the RUCIS cale revealed that item 5 (interet in me a a peron) wa the only item with too much unexplained variance in it rating. Interetingly, two of the SP were reponible for 65% of the tatitically ignificantly unexpected rating (i.e., rating with an abolute value of their tandardized reidual larger than 2.0) for thi item. Thi finding ugget that the ource of error in the rating of item 5 might be due to the inconitency of only two SP, and that the fit of the item might be improved through additional training of thee two SP to clear up any confuion they might have experienced when rating thi item. Although we carried out the tudy in two phae that addreed both the development and appraial tage of validation (Kane 2006), thi tudy by no mean preent a complete validation effort. Validation i an ongoing proce of gathering variou ource of evidence to upport propoed core interpretation. One could conider the finding from the econd phae of thi tudy a input to further modify the rating intrument to craft an even more pychometrically ound aement, thu cycling back to the development tage of validation once again. For example, our reult ugget that item 5 on the RUCIS Scale i till problematic, ince it continue to how inadequate fit to the meaurement model. Additional modification on thi item i a potential area for further intrument improvement.

14 588 C. Iramaneerat et al. There are ome limitation related to the interpretation and application of the finding from thi tudy. The firt limitation i the intrument limited focu on patient-centered medical communication kill. The ACGME (1999) definition of communication kill emphaize the importance of the ability to communicate not only with patient but alo with other member of a healthcare team. The RUCIS Scale doe not addre the kill needed to communicate effectively with other member of a healthcare team. The pychometric propertie of the RUCIS Scale demontrated in thi tudy might only apply to it ue in an OSCE etting where SP are trained properly on how to ue the rating intrument. Another limitation of thi tudy i the homogeneity of the reident ample we examined. Since our participant were internal medicine reident from a ingle training program, they were relatively homogeneou in their medical communication experience. Communication behavior that were not oberved in thee reident might be evident when other group of phyician are aeed. A multi-center trial of the rating intrument that involve medical chool from variou geographical region could tudy how the RUCIS Scale function with a more heterogeneou group of phyician. We hope that the finding from our tudy will benefit the medical education community in everal way. Firt, the product of thi validation effort the RUCIS Scale, along with validity evidence that upport it ue in the communication kill OSCE, hould erve the need of many reidency program, epecially given the increaing interet in communication kill aement that the ACGME Outcome Project ha generated. Second, our tudy provide a concrete example of how to ue a many-faceted Rach meaurement approach to improve the quality of SP rating intrument and to provide validity evidence baed on repone procee a outlined in the 1999 Standard for Educational and Pychological Teting (American Educational Reearch Aociation et al. 1999). Finally, thi tudy generated many intereting idea for future reearch. Appendix A Item on the Patient-Centered Communication and Interperonal Skill (CIS) cale 1. I felt you greeted me warmly upon entering the room. 2. I felt you were friendly throughout the encounter. You were never crabby or rude to me. 3. I felt that you treated me like we were on the ame level. You never talked down to me or treated me like a child. 4. I felt you let me tell my tory and were careful to not interrupt me while I wa peaking. 5. I felt you were telling me everything; being truthful, up front and frank; not keeping thing from me. 6. I felt you howed interet in me a a peron. You never acted bored or ignored what I had to ay. 7. I felt that you dicued option with me. 8. I felt you made ure that I undertood thoe option. 9. I felt you aked my opinion, allowing me to make my own deciion. 10. I felt you encouraged me to ak quetion. 11. I felt you diplayed patience when I aked quetion. 12. I felt you anwered my quetion, never avoiding them.

15 Rating intrument for communication kill aement I felt you clearly explained what I needed to know about my problem; how and why it occurred. 14. I felt you clearly explained what I hould expect next. 15. I felt you were careful to ue plain language and not medical jargon when peaking to me. 16. I felt you approached enitive/difficult ubject matter, uch a religion, exual hitory, tobacco/drug/alcohol hitory, exual orientation, giving bad new, etc., with enitivity and without being judgmental. 17. I felt the reident diplayed a poitive attitude during the verbal feedback eion. 18. If given the choice in the future, I would chooe thi reident a my peronal phyician. Note: All item are rated on a 5-point Likert cale, ranging from 1 (trongly diagree) to 5 (trongly agree). Appendix B Revied UIC Communication and Interperonal Skill cale Intruction Pleae chooe the option that bet decribe how you feel toward the reident communication kill. Some item alo have a not applicable option. Select thi option when the context of the cae doe not allow you to oberve that apect of the reident performance. (1) Friendly communication You did not greet me, or greeted me perfunctorily, or communicated with me rudely during the encounter. Your greeting and/or behavior during the encounter wa generally polite but imperonal or ditant. You greeted me warmly and communicated with me in a friendly, peronal manner throughout the encounter. Your greeting and overall communication were friendly and compaionate. Your tone of voice wa appropriate for the ituation. Overall, you created an exceptionally warm and friendly environment that made me feel comfortable to tell you all of my problem. (2) Repectful treatment You howed an obviou ign of direpect during the encounter. You treated me a an inferior. You did not how direpect to me. However, I oberved ome ign of condecending behavior. Although I believe it wa unintentional, it made me feel that I wa not at the ame level with you. You gave everal indication of repecting me. If there wa a phyical exam, thi include draping me appropriately. You were exceptionally repectful throughout the encounter. Your verbal and non-verbal communication howed repect for my privacy, my opinion, my right, and my ocioeconomic tatu.

16 590 C. Iramaneerat et al. (3) Litening to my tory You rarely gave me any opportunity to tell my tory or frequently interrupted me while I wa talking, not allowing me to finih what I aid. Sometime I felt you were not paying attention (for example, you aked for information that I already provided). You let me tell my tory without interruption, or only interrupted appropriately and repectfully. You eemed to pay attention to my tory and reponded to what I aid appropriately. You allowed me to tell my tory without interruption, reponded appropriately to what I aid, and aked thoughtful quetion to encourage me to tell more of my tory. You were an exceptional litener. You encouraged me to tell my tory and checked your undertanding by retating important point. (4) Honet communication You did not eem truthful and frank. I felt that there might be omething that you were trying to hide from me. You did not eem to hide any critical information from me. You explained the fact of the ituation without trivializing negative information or poibilitie (e.g., ide effect, complication, failure rate). You were exceptionally frank and honet. You fully explained the poitive and negative apect of my condition. You openly acknowledged your own lack of knowledge or uncertainty, and thing you would have to conult with other. When appropriate, you alo uggeted I eek a econd opinion. Not applicable. There wa no information for the reident to provide. (5) Interet in me a a peron You never howed interet in me a a peron. You only focued on the dieae or medical iue. In addition to talking about my medical iue, you pent ome time getting to know me a a peron. You pent ome time exploring how my medical iue affect my peronal or ocial life. You were exceptionally intereted in me a a peron. You not only explored how my medical problem affect my peronal and ocial life, but alo howed your willingne to help me addre thoe challenge. (6) Dicuion of option/plan You did not explain any option or plan; you jut told me what you would do without aking for my opinion. You explained option to me, but did not involve me in deciion making. If you olicited my opinion, you jut ignored it. You made all the deciion for me baed on your medical opinion. You dicued option with me, made recommendation, olicited my opinion regarding the option/plan, and incorporated my opinion into your medical planning.