The job of residency programs

Transcription

1 ORIGINAL ARTICLES Does Weekly Direct Observation and Formal Feedback Improve Intern Patient Care Skills Development? A Randomized Controlled Trial Gretchen Shelesky, MD, MS; Frank D Amico, PhD; Ruta Marfatia, MD, MPH; Anu Munshi, MD, MS; Stephen A. Wilson, MD, MPH BACKGROUND AND OBJECTIVES: Direct observation (DO) is considered to be an effective way to evaluate patient care. This study assesses if weekly direct observation with formal feedback (DO-FF) increases (1) clinical skills and (2) comfort with patient care skills (CWPCS) during the first 12 weeks of internship. METHODS: A single-blinded, stratified, randomized controlled trial (RCT) with allocation concealment incorporating interns at a community hospital family medicine residency program was performed. Interns (n=14) were stratified by the predicted number of calls in a 2:1 ratio of intervention: control group. The intervention group received DO-FF four times/month on inpatient history and physicals (H&Ps) by a family medicine senior resident or fellow. To assess skills, all interns were videotaped doing H&Ps at the beginning, middle, and end of the study. These were scored by two independent, blinded physicians using a validated tool. For selfassessment, all interns took a patient care comfort survey (PCCS) at baseline, 6 weeks, and 12 weeks. Analysis was done via intention to treat. RESULTS: At 6 weeks, interns who received weekly DO-FF had greater CWPCS on 32 of 35 questions. Eight were significantly different: (1) discussing end of life issues, (2) chest X ray interpretation, management of (3) congestive heart failure, (4) chronic obstructive pulmonary disease, (5) diabetes mellitus, (6) diabetic ketoacidosis, (7) stroke, and (8) venous thromboembolism/pulmonary embolism. In all but three questions, differences resolved by 12 weeks. Video data were not significantly different between groups. CONCLUSIONS: In certain aspects of patient care, DO-FF improved intern CWPCS at 6 weeks. There was no difference in the video assessments of clinical skills between the treatment groups. These results need to be further explored. However, this study showed that a well-designed RCT is feasible for educational research questions. (Fam Med 2012;44(7): ) The job of residency programs is to train residents to care effectively and efficiently for patients. Though medical knowledge is formally assessed annually with a standardized, in-training examination of family medicine residents, there is no standard for evaluating residents patient care skills. Literature review showed that direct observation (DO), though resource intensive, is considered effective to evaluate patient care and is part of the Accreditation Council for Graduate Medical Education (ACGME) toolbox for evaluation. 1 DO can identify otherwise unrecognized deficiencies, which can allow for remediation. 2,3 Similarly, DO has been used to aid needs assessment before implementing a resident-as-teacher (of medical students) curriculum. 4 Standardized patient and faculty evaluations via DO correlate and are reliable for formative feedback in surgery residents while self-assessment was not reliable. 5 Medical students who get more DO are more confident than their peers. 6 DO increased the frequency of feedback about resident-patient From Medical Student Education (Dr Shelesky) and Family Medicine Residency (Dr D Amico and Dr Wilson), UPMC St Margaret; and Department of Family Medicine (Dr Marfatia) and Department of Internal Medicine (Dr Munshi), University of Pittsburgh. 486 JULY-AUGUST 2012 VOL. 44, NO. 7 FAMILY MEDICINE

2 communication 7 and increased communication skills. 8 The purpose of this study was to determine the effect of direct observation and formal feedback (DO-FF) as a formative tool in the training of family medicine interns. This was examined by answering the question: Do family medicine interns in their first 3 months of residency who receive weekly DO-FF compared to those who receive routine practice demonstrate an increase in their clinical skills and their comfort with patient care skills (CWPCS)? Our literature review yielded no study using a randomized controlled trial to address this question. Methods This was an IRB-approved, stratified, single-blinded, randomized controlled trial that used allocation concealment and intention-totreat principles. We measured and compared two groups, routine practice versus augmented practice, to see what impact a more structured and intensified approach (ie, DO-FF) would have. Evaluation of effect was done by assessing repeated measures of (1) intern self-assessment of CWPCS and (2) videotapes of inpatient patient encounters (IPE) for each of the two groups. Site Description Fourteen interns at a communitybased, university affiliated family medicine residency program (FMRP) were studied. Intern year has 13 4-week medical rotations in various disciplines. Practice Prior to the Study Period DO of interns with or without feedback occurred randomly and nonuniformly by senior (second or third year) family medicine residents (SFMR) or faculty physicians. Any feedback forms completed were placed in interns academic folders. Videotaping was intermittently and infrequently used to review interns IPE skills. There was no formal instruction to teach upcoming SFMR how to give feedback. Study Design and Description Randomization and Allocation Concealment. Prior to having the names of the intern class (ie, those who would be recruited for the study), the order of the rotations was independently determined by the chief resident for Scheduling and the residency program director. Using the prior year s schedule, we estimated the number of calls expected during each rotation. (Call is a time outside of daytime working hours when interns are in the hospital taking care of patients.) The rotations were stratified into three groups by an estimate of the expected number of calls: six or less calls (low group), calls (medium group), or >30 calls (high group) (Appendix A available from corresponding author on request). Before the interns were assigned to their rotational schedule, all 14 incoming intern slots were stratified and then randomized to one of two groups (ie, DO-FF or routine) using a random number generator. Allocation concealment was maintained throughout the study period by interns never knowing to what group they were assigned. To obtain a treatment effect that was as strong as possible given the sample size restrictions, randomization was performed in a 2 to 1 ratio of DO- FF to routine practice. No a priori sample size estimation calculation was performed because the number of incoming interns was fixed. There were three rotations during which interns were unable to receive the DO-FF: Intensive Care Unit, Obstetrics Labor and Delivery, and Inpatient Surgery. Interns in the low group rotated through two of them during the study period. Those in the high group had a night float rotation for one of their rotations during the study period. This rotation gives a call-like experience five nights per week for 4 weeks. Feedback Workshop. Prior to the arrival of the interns, all rising SFMR participated in a How to Give Effective and Useful Feedback workshop administered by the primary investigator. This consisted of a 10-minute lecture followed by three 10-minute role-plays. Intern Consent. Verbal informed consent was obtained from the incoming intern class by reading a script during orientation week. Since this was an educational study that used tools already in practice, no written consent was needed. Interns were informed that they would be part of an educational study but not to which group they were assigned. They were given the option to opt out. Patient Consent. Consent for videotaping patients was obtained as per the University of Pittsburgh Medical Center (UPMC) policy with their consent form. No identifying information was collected. Videotaping was done by the primary investigator. Intervention. The intervention group received DO-FF four times/ month on their inpatient history and physicals (H&Ps) for the first 3 months of their internship by a SFMR or faculty development fellow who was on call with them. The content and length of the feedback were not controlled in order to simulate the realities and time pressures of being on-call. Our study attempted to see whether weekly, scheduled DO with intentionally given, semi-structured FF, without regard to quality of the feedback, had an effect on clinical skills and CWPCS. Feedback Mechanism. FF was written and verbal, facilitated by a feedback form internally developed (Appendix B vailable from corresponding author on request). The feedback form is based on the six ACGME competencies. Feedback Gathering, Encouragement, and Dissemination. The primary investigator coordinated and ensured that the interns in the intervention group received DO- FF four times monthly. She ed FAMILY MEDICINE VOL. 44, NO. 7 JULY-AUGUST

3 reminders to SFMRs who were responsible for the DO-FF and collected the feedback forms as a proxy measurement of the feedback completion. To encourage participation, incentives were provided. At the completion of the study, all feedback forms were put in each intern s academic file. Assessment of Clinical Skills Video. Videotaping occurred for all interns while on-call at approximately the beginning, middle, and end of the study period. Videos were numbered via random number generator to mask the time period from which they came. They were reviewed only after all were collected. To assess the effectiveness of this intervention, two blinded, independent reviewers evaluated videotaped inpatient H&Ps. To decrease potential bias from past experiences with interns, video reviewers had no prior contact with the interns in the videos. The video reviewers were blinded to each other, the treatment group, and date of the video. One reviewer was family medicine faculty at a medical school and had completed an academic fellowship. The other was an internal medicine academic fellow. Videotapes were evaluated using the validated Internal Medicine Resident Evaluation Form (IMREF). Based on the six ACGME competencies, 9 it consists of a total of seven questions, one addressing each of the following: patient care (PC), medical knowledge (MK), practice-based learning and improvement (PBLI), interpersonal and communication skills (ICS), professionalism (P), system-based practice (SBP), and overall clinical competence. Each question was scored on a scale ranging from 1 (unsatisfactory) to 9 (superior). For statistical analysis, the scores from both reviewers were averaged. comfort level in the areas of history and physical (15 questions), testing (four questions), and diagnoses (16 questions) (Appendix C available from corresponding author on request). Each of these questions was scored on a Likert scale ranging from 1 (limited experience) to 6 (very comfortable). This survey is based on two ACGME competencies: PC and MK. Data Collection. Figure 1 shows the flowchart of randomization as well as the collection of PCCS data and IMREF data at times 1, 2, and 3. Statistics Descriptive statistics were used to examine all variables collected throughout the study. Depending on the level of measurement for each variable, either parametric or nonparametric statistical tests were used to compare the randomization with respect to demographic characteristics as well as baseline PCCS scores and IMREF scores. The outcome variables (PCCS and IMREF data) were tested using a general linear mixed models analysis. In this model, terms entered consisted of (1) group (treatment or control) as a fixed effect, (2) intern, nested within group, as a random component, (3) time (baseline, 6 weeks, 12 weeks) as a fixed effect, and (4) the interaction between group and time as a fixed effect. The differences between the group mean scores at each time point were further compared using a Bonferroni multiple comparisons procedure (P value set at.05). PCCS and IMREF scores were analyzed individually. All results were analyzed using the intention to treat principle. Statistical analyses were performed using SPSS v18 and SAS v9 software. Results The incoming class totaled 14 interns. Nine were males (six in the treatment group) and five females (three in the treatment group). None were international medical graduates. Eight were graduates of allopathic medical schools and six graduates of osteopathic medical schools. The median age was 27.5 years (range 26 53). Three were Figure 1: Flowchart of Data Collection Over Time Assessment of CWPCS Survey. All interns completed a patient care comfort survey (PCCS) at baseline, 6 weeks, and 12 weeks. This survey was an internally developed tool in place for nearly a decade consisting of 35 questions about the resident s 488 JULY-AUGUST 2012 VOL. 44, NO. 7 FAMILY MEDICINE

4 married, and three had children. The number of calls ranged from four to 35 during the study period. One member of the medium group only had call four times instead of the predicted 10 because there was an injury in the intern class requiring a rotation change. Except for this single case, the predicted and the actual number of calls were similar. Randomization was effective as there was no statistical difference between treatment groups or strata with respect to demographic characteristics, initial PCCS scores (Table 1), or initial IMREF scores. Videotape Data The IMREF results showed that for the questions examining PBLI and SBP, the reviewers most often left their responses blank. Hence, these questions were not further analyzed. For the remaining five questions, the results showed no significant differences in clinical skills between the groups at any time point. All questions showed averages approximately in the middle of the 9-point scale, ranging from 5.1 to 6.6. PCCS Data The PCCS completion rate of return was 100% (14/14) at all three time points. There were no significant differences at baseline, demonstrating effective randomization (Table 1). All questions showed averages between 1.6 and 5.6. At 6 weeks, interns who received weekly DO-FF had greater PCCS scores in 32 out of 35 questions (data not shown); eight were significantly higher at 6 weeks (adjusted P value), and three remained significantly higher at 12 weeks (adjusted P value) (Figure 2). At 12 weeks, the PCCS scores were similar between groups and to baseline values. In certain patient care areas, this V-shaped distribution in the control group demonstrates that DO-FF prevented the loss of CWPCS early in internship. Summarizing these results, interns who received weekly DO-FF became comfortable more quickly in (1) discussing end of life issues (EOLI), (2) interpreting chest X rays (CXR), and management of (3) congestive heart failure (CHF), (4) chronic obstructive pulmonary disease (COPD), (5) diabetes mellitus (DM), (6) diabetic ketoacidosis (DKA), (7) stroke, and (8) venous thromboembolism/pulmonary embolism (VTE/PE) (Figure 2a-h). Discussion This study showed that, for certain areas, CWPCS increased in the treatment group compared to the control group. There was a loss of comfort felt at 6 weeks by the control group in each of the eight significant questions on the PCCS. In all but one (Figure 2a), this loss of comfort was not felt by the treatment group. However, in all but three questions (Figure 2a, 2d, 2g), this difference resolved by 12 weeks. This suggests that intentionally giving scheduled, semi-structured feedback early in internship can remove the loss of CWPCS in some areas and potentially diminish anxiety. This may be a consequence of being supported through the transition from being unconsciously incompetent to being consciously competent. It would be interesting to explore whether the removal of this loss of comfort translates into increased confidence and an increased sense of well-being that may allow interns to learn more in a shorter amount of time. The results show internal validity since the eight questions that were found to be significantly different are very common at the hospital where the study was conducted. The ability to discuss EOLI is used with every admission when assessing code status. CXR interpretation is one of the most common tests ordered on admissions. CHF, COPD, DM/DKA, stroke, and VTE/PE are among the most common diagnoses. Similarly, comfort scores for less encountered diagnoses, eg, sickle cell disease, were not affected by the scheduled, semi-structured feedback, thus supporting the hypothesis that repeated feedback impacts comfort. The results also show external validity by demonstrating an increased comfort with communication skills, namely discussing EOLI (Figure 2a), which supports the work of Perera et al. 8 There have been calls for more vigorous educational research. 10 This study answers that call with its rigorous design and methods that are are often challenging to implement in educational research. Withholding an intervention (ie, having a control group) in educational research can be difficult. This study took an established practice and made it more frequent and systematic to see if there was a treatment effect. This was possible due to the large size of the residency program in which the study was conducted. This single-blind, stratified, randomized controlled trial has several strengths. (1) There was allocation concealment as described in the methods. (2) The intervention was carried out by SFMR or fellows oncall with interns rather than faculty physicians, making this replicable and feasible. (3) The SFMR in the study were trained to give useful and effective feedback in a 50-minute workshop. (4) Interns in the treatment group got the amount of intervention predicted at the start of the study. (5) The assessment tools were based on the ACGME competencies. (6) All comfort surveys were completed at baseline, 6 weeks, and 12 weeks. (7) All but three interns completed all three videos; each intern completed at least two videos. (8) The blinded videotape reviewers were both trained in academic fellowships with training specifically for videotape review. (9) The repeated measures type analyses were carried out using intention to treat principles. (10) Despite having a small number of participants, significant differences in CWPCS were still found. This study also showed that clinical skills as measured via IMREF on videotapes of inpatient H&Ps were not different between treatment and control groups. There are several factors that may have impacted our results: (1) There was a low number of participants in the study (n=14) as dictated by the total number in the intern class; this may have FAMILY MEDICINE VOL. 44, NO. 7 JULY-AUGUST

5 Table 1: Baseline Mean Comfort Scores (Likert scale 1 6) for Treatment and Control Group Question Treatment Control H&P 1: Obtain a complete history H&P 2: HEENT exam H&P 3: Ophthalmoscope H&P 4: Otoscope H&P 5: Lung exam H&P 6: Cardiovascular exam including murmur recognition H&P 7: Abdominal exam H&P 8: Prostate exam H&P 9: Rectal exam H&P 10: Pelvic exam H&P 11: Neurological and mental status exam H&P 12: Medical decision making ability H&P 13: Ability to discuss end of life decisions H&P 14: Ability to communicate effectively with culturally diverse patients H&P 15: Comfort with patient-centered interviewing Test 1: Perform ABG Test 2: ABG interpretation Test 3: Chest X ray interpretation Test 4: EKG interpretation Diagnosis 1: Management of anemia Diagnosis 2: Management of CHF Diagnosis 3: Management of COPD Diagnosis 4: Management of diabetes mellitus Diagnosis 5: Management of diabetic ketoacidosis Diagnosis 6: Evaluation of acute abdomen Diagnosis 7: Management of HTN Diagnosis 8: Management of myocardial infarction/cad risks Diagnosis 9: Management of pelvic inflammatory disease Diagnosis 10: Management of renal failure/kidney disease Diagnosis 11: Management of sickle cell disease Diagnosis 12: Management of sexually transmitted diseases Diagnosis 13: Management of stroke/stroke management Diagnosis 14: Management of venous thromboembolism or pulmonary embolism Diagnosis 15: Management of obesity Diagnosis 16: Management of chronic pain Bold font indicates questions that showed a significant difference between groups at 6 weeks. HEENT head, ears, eyes, nose, and throat H&P history & physical ABG arterial blood gases CHF congestive heart failure COPD chronic obstructive pulmonary disease HTN hypertension CAD coronary artery disease 490 JULY-AUGUST 2012 VOL. 44, NO. 7 FAMILY MEDICINE

6 Figure 2 (a-h): Mean Comfort Scores Over Time in Treatment and Control Groups FAMILY MEDICINE VOL. 44, NO. 7 JULY-AUGUST

7 impacted the ability to detect some differences, especially smaller ones. (2) The difficulty of the patient encounter was not controlled, neither in the complexity of the medical decision making nor in the cooperation of the patient. (3) The video reviewers, though both academic fellowship trained and familiar with scoring videos, were not specifically trained to work with the IMREF. They were asked to use typical practice in watching and scoring the videos. This may have been too real world. (4) The study period was 3 months with some variation of time between videotapes as dictated by schedules. However, since there were no differences at all time points, this variation of time between videos did not affect the results in any meaningful way. (5) The tool used to score the videotapes was the IMREF, a validated and reliable tool especially for global assessment. However, it is a difficult tool to use on one patient encounter such as the videotaped H&Ps because it has general categories (rather than observable behaviors) based on the six ACGME competencies with anchors only at the two extremes (rather than with each number). (6) Some people in the control group received DO-FF. How often this may have happened is unknown. Throughout the study, it is possible that the culture of the program changed. SFMR reported that directly observing interns allowed them to identify deficiencies they felt they would have otherwise missed, which supports Cydulka s and Li s work. 2,3 As stated above, some people in the control group got DO-FF because it had become part of the on-call culture. Because the investigators could only add to the existing curriculum, this cross-over was allowed to continue without discouragement. Though only one DO-FF form was received from the control group, resident comments and feedback suggest that more of this behavior occurred because SFMRs found DO-FF to have value in itself. Despite, or possibly because of, the vigorous methods, the videotape data were not significantly different, and only approximately 25% of the PCCS questions were significantly different. Because DO/FF is a resource-intensive intervention in an environment with limited resources, DO/FF will need to be studied more before implementation into residency programs curricula. Some ways to continue the exploration and improve this study would include: (1) choosing a different evaluation tool like the mini-cex, which is better designed for single patient encounters with observable behaviors and that has anchors with each number; (2) increasing the number of participants in the study by including other residency programs; (3) training the reviewers to achieve better consistency and uniformity in videotape data; (4) attempting to control for patient variability when videotaping encounters; and (5) scrutinizing the study period since the learning curve for interns is so steep. In conclusion, this randomized controlled trial utilizing DO-FF between senior residents and interns showed a removal of the loss of comfort with some patient care skills at 6 weeks of internship. This argues that scheduled, semi-structured feedback with direct observation early in internship can improve intern comfort with some patient care skills, which may translate into increased confidence and the ability to learn more. However, more research must occur before recommending that this resource-intensive intervention should be routinely adopted in residency programs curricula. For now, less frequent, basic, faculty-led direct observation remains the accepted and encouraged formative and summative assessment tool. 1,11 ACKNOWLEDGMENTS: This research has been presented at the 2009 Society of Teachers of Family Medicine (STFM) Annual Spring Conference in Denver, the 2009 Family Medicine Education Consortium, the 2010 STFM Annual Spring Conference in Vancouver, the 2010 North American Primary Care Research Group Annual Meeting in Seattle, and the 2011 Pennsylvania Academy of Family Physicians meeting. The authors acknowledge the following who made this project possible: the residency director, Dr Ted Schaffer, as well as the faculty, fellows, residents, and interns at UPMC St Margaret Hospital. CORRESPONDING AUTHOR: Address correspondence to Dr Shelesky, UPMC St Margaret, Medical Student Education, th Street, Suite C, New Kensington, PA Fax: sheleskyg@upmc.edu. References 1. ACGME Outcome Project. Toolbox of Assessment Methods. toolbox.asp. Accessed April 16, Cydulka RK, Emerman CL, Jouriles NJ. Evaluation of resident performance and intensive bedside teaching during direct observation. Acad Emerg Med 1996;3: Li JT. Assessment of basic physical examination skills of internal medicine residents. Acad Med 1994;69: Katz NT, McCarty-Gillespie L, Magrane DM. Direct observation as a tool for needs assessment of resident teaching skills in the ambulatory setting. Am J Obstet Gynecol 2003;189(3): Brewster LP, Risucci DA, Joehl RJ, et al. Comparison of resident self-assessment with trained faculty and standardized patient assessments of clinical and technical skills in a structured educational module. Am J Surgery 2008;195: Chen W, Liao S, Tsai C, Huang C, Lin C, Tsai C. Clinical skills in final-year medical students: the relationship between self-reported confidence and direct observation by faculty of residents. Ann Acad Med Singapore 2008;37: Dattner L, Lopreiato JO. Introduction of a direct observation program into a pediatric resident continuity clinic: feasibility, acceptability, and effect on resident feedback. Teach Learn Med 2010;22(4): Perera J, Mohamadou G, Kaur S. The use of objective structured self-assessment and peerfeedback (OSSP) for learning communication skills: evaluation using a controlled trial. Adv Health Sci Educ Theory Pract 2010;15(2): Holmboe ES, Hawkins RE. Practical guide to the evaluation of clinical competence. Mosby Elsevier, 2008: Mainous AG III. In praise of conventional research methods for medical education research. Fam Med 2011;43(2): Wilson SA, Shelesky G. Direct Observation: what it is and how to effectively perform it. Education Column. STFM Messenger 2011;August JULY-AUGUST 2012 VOL. 44, NO. 7 FAMILY MEDICINE