Training and Evaluating Tobacco-specific Standardized Patient Instructors

Transcription

1 28 January 2006 Family Medicine Training and Evaluating Tobacco-specific Standardized Patient Instructors Kristie Long Foley, PhD; Geeta George, MPH; Sonia J. Crandall, PhD, MS; Kathy H. Walker; Gail S. Marion, PA-C, PhD; John G. Spangler, MD, MPH Background and Objectives: A comprehensive training program to develop tobacco-specific standardized patient instructors (SPIs) was implemented and evaluated at Wake Forest University. Methods: Descriptive statistics were used to assess SPIs experience with the training program and medical students perceptions of the SPI-student interaction. Two standardized scales, used to assess student performance on counseling (Tobacco Intervention Risk Factor Interview Scale [TIRFIS]) and cultural competency (Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale [TBMS-TCCS]), were tested for internal and interrater reliability and sensitivity to varied student performance. Costs of the program were measured. Results: SPIs highly rated the content, organization, and presenters of the training program. Medical students positively evaluated their experience with the SPIs. The TIRFIS and TBMS-TCCS subscales demonstrated good internal reliability, and inconsistencies in ratings by different SPIs were minimal. In addition, a range of scores on both measures attest to the sensitivity of the instruments to assess variations in student performance. Significant start-up costs are associated with developing this training program, although costs decline when SPIs are retained long term. Conclusions: The SPI training program was effective in developing a cohort of knowledgeable and reliable SPIs to train medical students in ways to improve their tobacco intervention counseling skills. Retaining SPIs long term should be a primary goal of implementing a cost-effective, successful training program. (Fam Med 2006;38(1):28-37.) Tobacco use is the leading cause of preventable death in the United States. Despite the well-known health hazards of tobacco, about one fourth of the US population smokes cigarettes. 1,2 In addition, cigarette and smokeless tobacco use may be increasing among certain demographic groups, 3-7 and the national per capita consumption of smokeless tobacco has tripled in the past several decades. 8 Most smokers would like their health care provider to help them quit smoking, but clinicians often feel inadequately trained to provide the help that addicted tobacco users need In fact, research has documented that US medical schools are inadequately preparing graduates to deal with tobacco dependence. 9,10 From the Department of Public Health Sciences (Dr Foley) and the Department of Family and Community Medicine (Drs Foley, Crandall, Marion, and Spangler, and Ms Walker), Wake Forest University; and the Physician Assistant Program, George Washington University (Ms Goerge). Explicit national guidelines are available to train physicians in tobacco cessation. 13 Various methods, such as the traditional didactic approach, patient-centered methods, role playing, and the use of standardized patient instructors (SPIs), have also been developed and used with varying degrees of success in training medical students in tobacco cessation counseling. 10 Using SPIs is a method that has proven especially successful SPIs are lay individuals trained to act as patients to teach medical students, residents, and physicians history-taking skills and interviewing and counseling techniques. Few resources are available, however, to train these lay individuals to become tobacco-specific SPIs who can reliably and effectively integrate into medical education. As a result, medical educators rely on their own knowledge and skills to provide such training and evaluation of their SPI program(s). This article describes the training, use, and evaluation of tobacco-intervention SPIs in undergraduate medical education at Wake Forest University School of Medicine (WFUSM).

2 Medical Student Education Vol. 38, No Methods WFUSM Program WFUSM s Integrated Tobacco Dependence Curriculum (ITDC), described elsewhere, 19 began in 1993 and relies heavily on SPIs for training medical students on how to counsel patients to quit tobacco. The SPI encounter has been a central feature of the ITDC. In 2003, WFUSM received a grant from the National Cancer Institute to expand the ITDC and develop a tobacco cessation curriculum that could be disseminated to all medical schools nationally. An important goal of this grant is to expand on the existing SPI program by recruiting a cadre of ethnically diverse SPIs and training the SPIs to reliably evaluate student performance regarding smoking and smokeless tobacco counseling skills, with explicit emphasis on culturally competent counseling skills. This study describes the implementation and evaluation of the SPI training program during its first year of implementation. Implementation of the SPI Training Program New SPIs (n=6) were recruited and trained, and existing SPIs (n=6) were retrained to (1) portray smoking or smokeless tobacco users, (2) improve the SPI cultural competency, specifically as it relates to tobacco, and (3) provide consistent, constructive feedback to medical students on culturally competent tobacco counseling. Recruitment was primarily conducted by word of mouth through employees at Wake Forest Family Medicine. Additional recruitment was attempted through the medical school s coordinator for the SPI program and through a local agency that serves minority persons, but these efforts were less successful. The training was organized by a master s-trained project manager, and content was delivered by four faculty with expertise in family medicine, tobacco, patient-physician communication, counseling, and public health. The new and existing SPIs were trained in two groups by the same faculty. Training occurred in 3 half days for the new SPIs and 2 half days for the existing SPIs to minimize the participant burden. Table 1 provides an overview of the three-part training program, including the goals, objectives, and methods to achieve the objectives. Newly recruited SPIs received a 9.5 hour training program (Sessions 1, 2, and 3), and existing SPIs received an abbreviated 6-hour training program (Sessions 2 and 3). All SPIs were provided reading materials that aided them in preparing to teach medical students on culturally competent tobacco cessation counseling. SPIs were encouraged to develop their own unique scenarios for their roles. They were provided assistance, as needed, to develop their scenarios. Also, a significant component of the training program was devoted to ensuring that SPIs provided consistent feedback to students using standardized instruments for tobacco cessation counseling. To improve the consistency of SPI ratings of student performance (ie, interrater reliability), individual and group ratings of videotaped SPI-student interactions were conducted. On completion of the training program, the SPIs are prepared to interact with third-year medical students, playing the role of a tobacco-using patient. A 30-minute counseling session with the student involves the SPI, who starts out in the role of a precontemplator (a tobacco-using patient who has never considered quitting use), who moves through the Stages of Change to a contemplator (a tobacco-using patient who is considering quitting use), and then an active quitter (a tobaccousing patient who is ready to quit use). 20 The student in turn is expected to interview the patient, using counseling specific to the patient s stage of change. Instruments Following the counseling session, the SPI breaks the role of the patient and takes on the role of the instructor, giving the student specific, constructive feedback about his/her tobacco knowledge and counseling styles, using the Tobacco Intervention Risk Factor Interview Scale (TIRFIS). This reliable instrument, based on the University of Rochester Risk Factor Interview Scale, 14 includes feedback regarding both smoked and smokeless tobacco counseling. 19,21 This instrument was originally developed to assess smoking counseling skills, but all smoking references were replaced in fall 2003 with generic tobacco language (to be applicable to smoking and smokeless tobacco). The wording of the items remained otherwise unchanged. The scale has four subscales of three items each: relationship, positive focus, instigating behavioral change, and general interviewing techniques. Each item is scored from 1 5, with a total possible score of 60. A higher score indicates better counseling skills. In an analysis of the Smoking Cessation Risk Factor Interview Scale (the precursor to the TIRFIS, adapted in 2003 to be tobacco generic), the scale demonstrated that the total scale demonstrated very good reliability (α=.85), and subscale reliabilities ranged from.58 to The item-to-total correlations (ITC) ranged from 0.43 to 0.66, with one item having an ITC of In addition, the SPIs rate the student s cultural competency skills specific to tobacco cessation counseling using the Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale (TBMS-TCCS). This scale was adapted from the Disease Beliefs and Measurement Scale (DBMS) and Cultural Concerns Scale (CCS). 22 In its original validation, the DBMS identified the extent to which students explored the patients beliefs about diabetes, blood sugar control, and insulin therapy (Cronbach s alpha=0.61). The CCS assessed the extent to which students acknowledged and attempted to accommodate the patient s cultural beliefs about food preparation (Cronbach s alpha=0.83).

3 30 January 2006 Family Medicine Table 1 Goals, Objectives, and Methods Used in Sessions 1, 2, and 3 of the SPI Training Program Session 1 (3.5 hours) Goals: To provide an overview of the epidemiology health effects of tobacco use To discuss strategies for the treatment of tobacco use and dependence To explore the role of the SPI in training and evaluating medical students in tobacco cessation counseling Objective Methods to Meet Objective To briefly describe the epidemiology of tobacco use PowerPoint presentation by WFUSM faculty member (project director) To discuss health effects associated with smoked Have SPIs read, as homework, appropriate materials that were mailed to them and smokeless tobacco prior to training To describe the need for increased education of physicians PowerPoint presentation by WFUSM faculty member (project director) for tobacco intervention To have SPIs be able to articulate the Stages of Change Discussion and review of the internally developed algorithm on tobacco cessation counseling by WFUSM faculty member (project director) To identify teachable moments Discussion by WFUSM faculty member (project director) To describe nicotine addiction and pharmacotherapies used Handout provided and in-class discussion on addiction and the appropriateness to treat it of various pharmacotherapies for nicotine addiction led by WFUSM faculty member (project director) To describe what a standardized patient is Discussion by WFUSM faculty member (project director) on the role and importance of SPIs in medical student education on tobacco cessation counseling To discuss the outline of a simulated patient interview Discussion by WFUSM faculty member (project director) on the typical with feedback SPI-medical student encounter To identify a typical algorithm used in tobacco cessation Discussion by WFUSM faculty member (project director) with review of the counseling internally developed algorithm on tobacco cessation counseling To address common excuses used by tobacco users in order not to quit To discuss the major categories contained within the TIRFIS Brief introduction to the items on the TIRFIS by WFUSM faculty member (project director). Have participants review the TIRFIS at home to further become familiar with the instrument Explore the role of the SPI in training medical students in As homework, have SPIs start to think about their scenarios and put it into providing culturally appropriate tobacco cessation counseling writing. Have them build their scenarios as they go through the 3-day training Session 2 (3.0 hours) Goals: To explore the impact of culture on tobacco use To discuss a method of assessing medical student competency in culturally competent tobacco cessation counseling Objective Methods to Meet Objective To explore the participants cultural experience with Participant discussion on the impact of culture (including racial/ethnic, tobacco (eg, work, family, social relationships, religion) geographic, and family background) on tobacco use. Display of prompts (tobacco and how this experience informs his/her perceptions leaf, smokeless tobacco products) to facilitate discussion and values about tobacco use To discuss What is culture? PowerPoint presentation by a WFUSM faculty member describing the components culture To discuss how culture influences our behavior Discussion led by WFUSM faculty member on the influence of culture To create awareness that there are health disparities on behavior and health disparities associated with different cultural groups among different cultural groups To provide SPIs with the appropriate training to evaluate Introduction and discussion* to the items on the TBMS-TCCS led by WFUSM medical students performance on culturally sensitive faculty member. tobacco cessation counseling View videotapes of a culturally competent SPI-medical student tobacco cessation counseling encounter and a culturally incompetent SPI-medical student tobacco cessation counseling encounter, rate the skills of the student in the culturally competent SPI-medical student encounter by completing the TBMS-TCCS, and discuss, as a large group, the ratings.** Discussion led by WFUSM faculty member (continued on next page)

4 Medical Student Education Vol. 38, No Table 1 (continued) Session 3 (3 hours) Goals: To practice the SPI role To practice evaluating and providing feedback to the medical student Objective Methods to Meet Objective To train SPIs to provide specific and constructive Handout and group discussion on ways of providing constructive feedback. feedback to medical students. Discussion led by WFUSM faculty member To have SPIs focus feedback on behaviors that can be changed, not on personality traits. To have SPIs act his or her scenario in a confident manner The SPIs break up into pairs to practice their scenarios as staff members provide assistance as needed To train SPIs to critique another SPI constructively in The SPIs provide constructive feedback, based on principles of giving feedback his or her role they were taught earlier in the session To train SPIs to evaluate a student with the TIRFIS View videotape of an SPI-medical student tobacco cessation counseling encounter, To elicit SPI justification for his or her rating of the student rate student skills by completing the TIRFIS, and discuss, as a large group, the on the TIRFIS ratings To help SPIs recognize realistic expectations and limitations Closing discussion led by project director summarizing expectations regarding of medical students the SPI-medical student encounter and providing feedback To repeat to the group basic points to remember regarding giving feedback SPI standardized patient instructor TIRFIS Tobacco Intervention Risk Factor Interview Scale WFUSM Wake Forest University School of Medicine TBMS-TCCS Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale * The instrument was modified based on some of the feedback from the SPIs. ** The culturally competent videotape was scripted and filmed by project staff members and a WFUSM faculty member, who played the role of the minority patient. The culturally incompetent video was of an interaction between an SPI-medical student that was clinically sound but did not address cultural issues in situations where appropriate. Because no current scales existed to assess cultural competency related to tobacco use, the DBMS-CCS was specifically reworded and reviewed by experts of different ethnic groups to ensure that it captured beliefs and cultural concerns of tobacco users. Data reported in this report are the first to assess the scale s reliability after the rewording of the instrument to be tobacco specific. Both instruments are available from any of the authors. For the first two to three counseling sessions, an experienced SPI was paired with a new SPI to review their ratings on the TIRFIS and TBMS-TCCS as a means of improving interrater reliability. Data for this study were compiled over a 7-month period (November 2003 to May 2004). This study was approved by the Institutional Review Board of Wake Forest University School of Medicine. Analyses SPI Evaluation of Training Program. SPIs evaluated each session of the SPI training program. They completed anonymous surveys that assessed content and organization of the training, presenter qualities, and influence of training on knowledge and ability to become a tobacco-specific SPI. To analyze the data, frequencies for each item for Sessions 1, 2, and 3 were calculated. Analysis of variance was used to assess differences in SPI ratings across the sessions both to evaluate instructor performance and to assess quality assurance of the overall training curriculum. Student Feedback. Students evaluated all SPI encounters following the SPI training sessions. Immediately following the SPI interaction and feedback session, medical students were requested to complete a survey to assess the effectiveness of encounter and feedback in improving their tobacco cessation counseling techniques. To analyze the data, frequencies and mean scores were calculated for each item. Utility of Measures to Assess Student Performance. The TIRFIS and TBMS-TCCS were assessed for their ability to reliably assess counseling skills and to determine the SPIs willingness to utilize the full scales to evaluate student performance. Descriptive statistics, internal reliability (Cronbach s alpha), interrater reliability (via percent agreement), Cohen s kappa statistic (a measure of concordance), and a generalizability analysis were computed. 23,24 The purpose of the generalizability analysis was to decompose the measurement variance, taking into account potential sources of variation, including the student, the instrument item, and the SPIs. This methodology was important for considering the effect of using many SPIs (who may have had variations in the way they scored students)

5 32 January 2006 Family Medicine on establishing interrater reliability. The underlying statistical methodology of a generalizability analysis is random effects models and variance component analysis. DeVellis provides a succinct description of generalizability theory in his text on scale development. 24 Descriptive statistics and internal reliability was based on 55 SPI-student encounters during the academic year. The interrater reliability was assessed using 40 videotaped SPI-student encounters. The 40 videotapes were viewed by two independent SPIs (in private sessions) and scored using the TIRFIS and TBMS-TCCS. Cost of Implementing the SPI Training Program. The cost of implementing the SPI training program was also calculated. Cost measures include recruitment of SPIs, materials development, training costs (eg, refreshments), and faculty and staff time spent preparing for and delivering the training program. Results SPI Evaluation of Training Program Twelve SPIs provided feedback regarding their experience of the SPI training program. They were 51.6 years of age, on average (SD =20.5, range 24 83), with the majority being female (n=8) and white (n=7). Persons who self-identified as black (n=2) or Hispanic (n=3) comprised the remainder of the SPIs. The SPIs highly rated the content, organization, and presenter characteristics of the training program (Table 2). The SPIs stated that they especially appreciated the opportunities for practicing their scenarios Item 1 CONTENT Reading materials helped me understand the importance of culturally competent tobacco cessation counseling. and the use of the TIRFIS and TBMS. They also enjoyed the group discussion about culture and its influence on tobacco use. At the end of the first session, a few SPIs mentioned that they felt overwhelmed about their role. At the conclusion of the training, however, Overall Score 1 3 (n=6) SESSION (n=11) 3 (n=9) Handouts were useful Slides/overheads were useful Practicing after viewing the videos was useful. 5.0 NA The small group exercise(s) were useful. 5.0 NA NA 5.0 ORGANIZATION Topics were covered in a logical sequence Amount of information covered in today s training was reasonable PRESENTER(S) Presenter(s) were well organized Material was presented in a clear and concise manner Presenter(s) were enthusiastic/stimulating/engaging Presenter(s) were well-informed on the topics Presenter(s) actively involved participants Presenter(s) encouraged questions on the topic Presenter(s) established rapport Presenter(s) set clear/reasonable expectations I am confident that I can provide feedback to students on their tobacco cessation counseling skills Scoring: 1 (not at all) to 5 (very much) 2 There were no statistically significant differences across sessions using analysis of variance (ANOVA). 3 Session 1 only included the newly recruited SPIs since the original SPIs had attended this session previously. One SPI did not attend Training Session #2, and three SPIs did not attend Training Session #3. SPI standardized patient instructor Table 2 SPI Rating of Sessions 1, 2, and 3 of the SPI Training Program

6 Medical Student Education Vol. 38, No Table 3 SPI Rating of Confidence in Knowledge and Ability to Be an SPI at the End of Session 3 of the Training Program (n=9)* Compared to before you came to this 3-day training, how confident do you feel now about the following: Average Score My knowledge regarding the adverse influence tobacco use has on health 4.9 My knowledge regarding the importance of teaching medical students to appropriately counsel their patients on tobacco cessation 4.8 My knowledge on tobacco dependence treatment 4.4 My knowledge on the health disparities between racial/ethnic and demographic groups 4.1 My knowledge on the tobacco use disparities between racial/ethnic and demographic groups 4.2 My knowledge about the role culture plays in tobacco use 4.3 My ability to play the part of a tobacco-using standardized patient 4.4 My ability to use the Tobacco Intervention Risk Factor Interview Scale (TIRFIS) to evaluate medical students on their 4.3 counseling skills My ability to use the Tobacco Beliefs and Management Scale and Tobacco Cultural Concerns Scale (TBMS-TCCS) to evaluate 4.2 medical students on their counseling skills My ability to provide good feedback to the medical student 4.3 * n=9 Scoring: 1 (not confident at all) to 5 (very confident) SPI standardized patient instructor the SPIs rated that they were highly confident about their knowledge and skill gained through the training (Table 3). Student Feedback Fifty-five students provided feedback regarding the SPI-student interaction. The average student was 26.4 years of age (SD 3.0, range 23 36), with the majority being male (66%) and white (48%). Black (10%), Asian or Pacific Islander (26%), Hispanic (6%), and other (10%) racial/ethnic groups were also well represented among students providing feedback. In general, medical students highly rated the SPI encounter using a structured evaluation form (Table 4). Twenty-one students also responded to an open-ended question that elicited additional comments regarding their interview experience; 18 of the responses were positive. Specific comments included: Although somewhat contrived, the experience is still a beneficial teaching tool. In receiving feedback I was able to better assess my own strengths and weaknesses, [I] very much enjoyed this experience [it] gave me great confidence to do this in the future, and A good format, a good learning experience helpful feedback. Of the three students who gave negative feedback, two were specifically directed at the encounter. I felt I did a fairly good job of explaining reasons and counseling my patient, but after my feedback was given I feel the patient mistook a lot of what I said and held it against me, quoting things I said out of the correct context. Another student stated One must remember Table 4 Medical Student Feedback About the SPI Encounter* Item** To what extent did the patient you interviewed behave like a real patient you might see in a genuine clinical setting? How realistic was this general scenario involving a doctor-patient interaction (ie, a setting involving a relatively healthy patient being seen by his/her doctor under similar circumstances?) (n=54) By the conclusion of the interview, were you able to forget you were acting out a contrived role and get into the role of this patient s primary care doctor? How helpful do you feel it is to receive feedback about your interview from the patient immediately following the interview? Were the feedback and discussion following the interview informative, instructional, and constructive? (n=54) In future smoking cessation counseling, are you likely to use the skills or techniques that were reviewed after today s interview? How effective do you feel this exercise is in teaching patient counseling skills? How much did you learn today as a result of this exercise? * n=55 ** Scoring: 0 (not at all), 1 (a little), 2 (somewhat), 3 (much), 4 (very much) SPI standardized patient instructor Average Score

7 34 January 2006 Family Medicine that actual interviews are a completely different experience, with real patients being much more resistant to stopping smoking. In real life, much of what we were taught simply does not work. same SPI-student encounter. Table 6 shows the percent of agreement between two raters on each item of both instruments and the Cohen s Kappa statistic (a measure of concordance). Utility of Measure to Assess Student Performance The TIRFIS and TBMS-TCCS subscales demonstrated good internal reliability (Table 5). There was also a wide range of scores attesting to the sensitivity of the instruments to assess variations in student performance. This suggests that the sessions were effective in appropriately training SPIs to use the instruments to assess students and pick up variations in student performance. Analysis of interrater reliability (n=40 videotape reviews by two independent SPIs) also suggests that the SPIs provide comparable performance evaluations on the TIRFIS and the TBMS-TCCS when viewing the TIRFIS 1 Subscale 2 Mean (SD) Range Relationship (2.4) Positive focus (2.6) Instigating behavior change (2.3) General interview techniques (2.1) Scale total (7.7) TBMS 2 Item Mean (SD) Range Discussed why I should quit using tobacco 3.64 (1.0) 2 5 Explored my fears about quitting 2.96 (1.2) 1 5 Explored my beliefs about different treatments such as 3.49 (1.3) 1 5 nicotine gum, patch, or Zyban to help me quit. Scale total (3.29) TCCS 3 Item Mean (SD) Range Explored my beliefs about the importance of tobacco use in my culture 2.05 (1.3) 1 5 Acknowledged the importance of my cultural preferences 2.31 (1.2) 1 5 Worked out a quit plan that is sensitive to my cultural beliefs 2.36 (1.2) 1 5 Scale total 6.73 (3.29) * n=55 Table 5 Medical Student Performance on the TIRFIS, the TBMS, and the TCCS* TIRFIS Tobacco Intervention Risk Factor Interview Scale TBMS Tobacco Beliefs Management Scale TCCS Tobacco Cultural Competency Scale 1 Scoring on TIRFIS items from 5 to 1, with a higher score reflecting better performance 2 Items of the TIRFIS Subscales: relationship confidence, joining, empathy; positive focus reinforce effort, highlight positive consequences, reframing failure; instigating behavior change stages of change, nicotine dependence, behavior change technique; general interview techniques organization, rapport/eye contact/body language, closure, and follow-up. 3 Scoring from 5 to 1, with a higher score reflecting better performance TIRFIS Perfect agreement ranged from 17.5% to 47.5 % on the individual items. Yet, an average of 78% (range 61% 93%) of paired SPI ratings were within one point of agreement across the total TIRFIS scale. Kappa values were considered slight to fair using Landis and Koch s guide on measurement agreement. 23 The low kappa values were at least partially due to the fact that the majority of SPI ratings fell within categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement resulted in a large drop in concordance. The generalizability analysis, a method to decompose the measurement variance taking into account student, item, and SPI, demonstrated that the measure is dependable and that SPI variability is relatively mild compared to overall variability of the scores (overall G coefficient=0.51) 25 (Table 7). TBMS-TCCS Perfect agreement ranged from 30% 35% on the TBMS and 25% 30% on the TCCS. Similar to the TIRFIS, the majority of paired ratings fell within 1 point of agreement. The generalizability coefficient was quite low (G=0.07) on the TBMS. The effect is mainly due to a low variance in student scores after factoring out other sources of variation (primarily SPI and item). Additionally, student-spi interaction is high compared to either student or SPI variation alone. This suggests that the pairing of the SPI and student may, in some way, influence the ratings on the

8 Medical Student Education Vol. 38, No Table 6 Table 7 Interrater Reliability of the TIRFIS and TBMS- TCCS: Paired Ratings of SPI-Student Encounters* Perfectly Agree Paired Ratings Differ by 1 Point Kappa** TIRFIS % % Item Item Item Item Item Item Item Item Item Item Item Item Overall TBMS Item Item Item Overall TBMS TCCS Item Item Item Overall TCCS * n=40 ** The kappa values are considered slight to fair using Landis and Koch s 23 recommended review criteria. The low kappa values are partially due to the fact that the majority of observations fall into categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement leads to a large drop in concordance. TIRFIS Tobacco Intervention Risk Factor Interview Scale TBMS Tobacco Beliefs Management Scale TCCS Tobacco Cultural Competency Scale TBMS. The TCCS, however, was not similarly affected. The generalizability coefficient (G= 0.55) suggests that the measure is dependable and that the SPI variance component was lower than the student variability or other factors. Cost of Implementing the SPI Training Program Costs associated with the training session included SPI payment ($1,400), refreshments ($90), producing printed and bound material handouts ($90), mailing Decomposition of the Measuring Variance of the TIRFIS and TBMS-TCCS, Taking Into Account the Variability in Students, SPIs, and Items (Generalizability Analysis)* Variance Component (Degrees Source of Variation of Freedom) TIRFIS TBMS TCCS Student 0.16 (39) 0.03 (39) 0.46 (39) SPI 0.08 (1) 0.08 (1) 0.39 (1) Item 0.08 (11) 0.06 (2) 0.00 (2) Student-SPI Interaction 0.14 (39) 0.42 (39) 0.20 (39) Item-SPI Interaction 0.10 (11) 0.14 (2) 0.01 (2) Student-Item Interaction 0.11 (429) 0.30 (78) 0.00 (78) Residual 0.52 (429) 0.43 (78) 0.44 (78) Generalizability coefficient (for absolute decision) * The purpose of this type of analysis is to partition the total variance observed on the TIRFIS, TBMS, and TCCS scores to variances among students, SPIs, and items on the survey. The underlying statistical methodology of a generalizability analysis is random effects models and variance component analysis. TIRFIS Tobacco Intervention Risk Factor Interview Scale TBMS Tobacco Beliefs Management Scale TCCS Tobacco Cultural Competency Scale SPI standardized patient instructor ($60), and copying and printing ($30). Faculty and staff time to prepare for and conduct the training were estimated at 130 hours. Salary cost for time spent by faculty and staff was approximately $3,730 ($2,560 for faculty and $1,170 for staff). Following completion of the training, each SPI was paid $20 per SPI-medical student encounter, resulting in a cost of $1,100 to date (55 SPI-medical student encounters times $20 per SPImedical student encounter). The total cost to implement the SPI training program was estimated to be $5,400 for 12 SPIs, resulting in a cost of $450 per SPI for recruitment and training ($5,400 total training cost/12 SPIs) and $98 per student encounter to date ($5,400 total training cost + $1,100 SPI-medical student encounter costs to date)/55 SPI-medical student encounters). Additional costs over time can be incurred should SPIs leave the program (requiring additional training) and/or retraining needs are identified. Discussion The SPI-medical student tobacco cessation counseling encounter and SPI feedback to students on their tobacco cessation counseling skills have been successful, both from the point of view of the students 19 and

9 36 January 2006 Family Medicine the SPIs. This paper provides initial evidence of the success of the program in training SPIs to be effective standardized patients as well as instructors who can provide constructive and reliable feedback to students. Essential to developing a cadre of tobacco-specific SPIs is a good training program that trains lay individuals to play the role of a tobacco-using patient as well as an instructor who can critique student tobacco cessation counseling techniques. Further, as SPIs become more integral teachers of medical education, it is essential that their ratings of student performance be consistent. Students will certainly insist on fair grading practices. Medical education administrators also have a responsibility to their students to ensure that they are evaluated in an equitable manner. There are limitations to this study, however, that deserve attention. The relatively low interrater reliability scores on the two instruments indicate cause for concern. The low kappa values are partially due to the fact that the majority of observations fall into categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement leads to a large drop in concordance. This could be improved by several mechanisms. First, the sample size could be increased, thereby minimizing the drop in kappa values when disagreement occurs among only a small number of SPIs. Second, more training can be done to enhance the distinctions between scores 4 and 5, thus improving perfect agreement between raters. Finally, scores 4 and 5 could be combined if the difference between 4 and 5 is not clinically meaningful. If the ultimate goal of instruments is to provide feedback to the student, the SPIs must be able to articulate specific improvements that could raise a student s score from a 4 to a 5; otherwise, the items should be combined without jeopardizing the utility of the instruments. A second limitation to this study is that the TBMS- TCCS has not been previously validated since the wording was revised from diabetes to tobacco. Although our study provides initial evidence of the reliability of the TBMS-TCCS, the limited sample size weakens any conclusions drawn about its measurement utility. Of some concern is the relatively low generalizability coefficient observed on the TBMS. This appears to be primarily related to low variance in student scores, which may be improved once students in this curriculum are trained on cultural issues related to tobacco. Further, there are no published validated scales that assess overall tobacco cultural competency. This represents a deficit in the published literature, especially for persons interested in understanding the cultural nuances of tobacco use and teaching culturally competent counseling skills specific to tobacco. Despite these limitations, we believe that the SPI training program described in this paper can be incorporated into a medical education curriculum to develop a cohort of knowledgeable, effective SPIs who can articulate ways to improve the students tobacco intervention counseling skills. We also feel that the basic training may be adapted to other health behaviors or medical concerns. In fact, family medicine faculty and staff at WFUSM have recently proposed to create a SPI-obesity curriculum that builds on the work presented in this paper. The challenges in adapting the program include identifying counseling instruments that are appropriate to other health behaviors, the time-consuming task of practicing scenarios and ensuring interrater reliability, and the initial costs. Yet, the groundwork for establishing such a curriculum could save administrators time in the long run by limiting (or eliminating) retraining of SPIs (a time-intensive and costly endeavor) and managing student complaints that result from perceived unfair grading practices of lay individuals. Costs for implementing our training program were initially high. As the program progresses and SPIs are engaged in more encounters, the average cost per encounter will subsequently decline. Since additional costs may be incurred as SPI are retrained or leave the program, it is imperative that medical schools integrate a retention program into their SPI-led medical education program to ensure continuity and cost-efficiency. With the emphasis on both smoked and smokeless tobacco, integration within the existing medical school curriculum, and explicit evaluation of cultural competency, we offer this model as a useful resource to medical educators who wish to institute a program using SPIs to train medical students in tobacco intervention. Acknowledgments: This project was supported by a grant from the National Cancer Institute R25CA Corresponding Author: Address correspondence to Dr Foley, Wake Forest University, School of Medicine, Department of Public Health Sciences, Medical Center Boulevard, Winston-Salem, NC Fax: kfoley@wfubmc.edu. REFERENCES 1. Annual smoking-attributable mortality, years of potential life lost, and economic costs United States, MMWR Morb Mortal Wkly Rep 2002;51(14): Cigarette smoking among adults United States, MMWR Morb Mortal Wkly Rep 2002;51(29): Tobacco use among high school students United States, MMWR Morb Mortal Wkly Rep 1998;47(12): Smith SS, Fiore MC. The epidemiology of tobacco use, dependence, and cessation in the United States. Prim Care 1999;26(3): Tobacco use among US racial/ethnic minority groups African Americans, American Indians and Alaska Natives, Asian Americans and Pacific Islanders, Hispanics. A Report of the Surgeon General. Executive summary. MMWR Recomm Rep 1998;47(RR-18):v-xv, Welty TK, Lee ET, Yeh J, et al. Cardiovascular disease risk factors among American Indians. The Strong Heart Study. Am J Epidemiol 1995;142(3): Use of smokeless tobacco among adults United States, MMWR Morb Mortal Wkly Rep 1993;42(14): Spit tobacco and youth. Washington, DC: Office of Evaluations and Inspections, US Department of Health and Human Services, 1992.

10 Medical Student Education Vol. 38, No Ferry LH. Overcoming barriers to nicotine dependence treatment. Prim Care 1999;26(3): Spangler JG, George G, Foley KL, Crandall SJ. Tobacco intervention training: current efforts and gaps in US medical schools. JAMA 2002;288(9): Orleans CT. Understanding and promoting smoking cessation: overview and guidelines for physician intervention. Annu Rev Med 1985;36: Gilpin E, Pierce J, Goodman J, Giovino G, Berry C, Burns D. Trends in physicians giving advice to stop smoking, United States, Tob Control 1992;1: Fiore MC, Bailey WC, Cohen SJ, et al. Treating tobacco use and dependence. Washington, DC: US Department of Health and Human Services, Public Health Service, Levenkron JC, Greenland P. Validation of the University of Rochester Risk Factor Interview Scale. Am J Prev Med 1987;3(3): Levenkron JC, Greenland P, Bowley N. Using patient instructors to teach behavioral counseling skills. J Med Educ 1987;62(8): Levenkron JC, Greenland P, Bowley N. Teaching risk factor counseling skills: a comparison of two instructional methods. Am J Prev Med 1990;6(2 suppl):s29-s Eyler AE, Dicken LL, Fitzgerald JT, Oh MS, Wolf FM, Zweifler AJ. Teaching smoking cessation counseling to medical students using simulated patients. Am J Prev Med 1997;13(3): Bland E, Oppenheimer L, Brisson-Carroll G, Morel C, Holmes P, Gruslin A. Influence of an educational program on medical students attitudes to substance use disorders in pregnancy. Am J Drug Alcohol Abuse 2001;27(3): Spangler JG, Enarson C, Eldridge C. An integrated approach to a tobacco-dependence curriculum. Acad Med 2001;76(5): Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: toward an integrative model of change. Consult Clin Psychol 1983;51: Foley KL, Crandall SJ, George G, Roman M, Spangler JG. Reliability of a Smoking Cessation Risk Factor Interview Scale (SCRFIS) for use with standardized patient instructors. J Cancer Educ 2003;18(3): Robins LS, White CB, Alexander GL, Gruppen LD, Grum CM. Assessing medical students awareness of and sensitivity to diverse health beliefs using a standardized patient station. Acad Med 2001;76(1): Landis JR, Koch GG. Agreement measures for categorical data. Biometrics 1977;33: DeVellis RF. Scale development. Theory and applications. Applied Social Research Methods Series Vol. 26. Newbury Park, Calif: Sage Publications, 2003: Shavelson RJ, Rowley GL, Webb NM. Generalizability theory. Am Psychol 1989;44: