Evaluation of a pharmacotherapy context-learning programme for preclinical medical students

British Journal of Clinical Pharmacology DOI:10.1111/j.1365-2125.2006.02742.x Evaluation of a pharmacotherapy context-learning programme for preclinical medical students J. A. Vollebregt, 1,4 J. van Oldenrijk, 1 D. Kox, 1 S. R. van Galen, 1 B. Sturm, 1 J. C. M. Metz, 2 M. C. Richir, 1 M. de Haan, 3 J. G. Hugtenburg 1 & T. P. G. M. de Vries 1 1 VU University Medical Centre, Department of Clinical Pharmacology and Pharmacy/Section Pharmacotherapy, Amsterdam, 2 Clinical Training Centre, Faculty of Medical Sciences, University of Nijmegen, Nijmegen, 3 VU University Medical Centre, Department of General Practice and 4 Jan van Breemen Instituut, Amsterdam, the Netherlands What is already known about this subject Context-learning in medicine is mainly based on theories explaining how medical expertise is achieved, particularly with regard to diagnostic problem solving. By frequently solving diagnostic clinical problems doctors generate so-called networks of organized knowledge in their memory. Preclinical medical students are well able to learn to choose and prescribe drugs. What this study adds Preclinical pharmacotherapy context learning for medical students has a modest but positive effect on learning cognitive pharmacotherapeutic skills, i.e. choosing a drug treatment and determining patient information. The effect can be obtained with role-play sessions, a suboptimal form of context learning, with a minimal study load and a high appreciation by students. Correspondence J. G. Hugtenburg PhD, VU University Medical Centre, Department of Clinical Pharmacology and Pharmacy/ Section Pharmacotherapy, Van der Boechorststraat 7, 1081 BT Amsterdam, the Netherlands. Tel: + 31 2 0444 8090 Fax: + 31 2 0444 8100 E-mail: jg.hugtenburg@vumc.nl Keywords clinical pharmacology and therapeutics, cognitive skills, context learning, medical undergraduate education, skills training Received 11 May 2006 Accepted 9 June 2006 Published OnlineEarly 12 September 2006 Aim To evaluate a context-learning pharmacotherapy programme for approximately 750 2nd, 3rd and 4th year preclinical medical students with respect to mastering cognitive pharmacotherapeutic skills, i.e. choosing a (drug) treatment and determining patient information. Methods The context-learning pharmacotherapy programme consists of weekly organized role play sessions in the form of consulting hours. Fourth year students sit for a therapeutic Objective Structured Clinical Examination (OSCE) in the form of consulting hours at the outpatient clinic. Sixty-one 2nd, 74 3rd and 49 4th year medical students who attended the role play sessions and the OSCE were randomly selected. Their performances were assessed by clinical examiners and clinical experts and compared with a reference group of 6th year graduated students. Additionally, the scores of a questionnaire on study load and appreciation were collected. Results The level of the pharmacotherapeutic skills of the 4th year students who followed the pharmacotherapy context-learning programme was not far below that of 6th year graduates who had finished their clinical clerkships, but had not followed the pharmacotherapy programme. The time spent on the programme was about 1% of the total study load per year. The students appreciated the role play sessions and OSCE by around 80% and 99% of the maximum possible scores. Conclusions Preclinical pharmacotherapy context learning has a modest but positive effect on learning cognitive pharmacotherapeutic skills, i.e. choosing a drug treatment and determining patient information. This effect has been obtained with role play sessions, a suboptimal form of context learning, with a minimal study load and a high appreciation by students. Br J Clin Pharmacol 62:6 666 672 666 2006 The Authors Journal compilation 2006 Blackwell Publishing Ltd

A pharmacotherapy context-learning programme Introduction In a previous study we have demonstrated that preclinical medical students are well able to learn to choose and prescribe drugs [1]. As the result, a pharmacotherapy curriculum was implemented at the VU University Medical Centre in Amsterdam for 2nd, 3rd and 4th year preclinical medical students. For this curriculum a method based on context learning was selected: students learn by performing role play sessions in the form of consulting hours. In context learning, students learn in a setting that is similar to their future profession. For medical students this is the primary healthcare or clinical setting [2]. Context learning in medicine is mainly based on theories explaining how medical expertise is achieved, particularly with regard to diagnostic problem solving [3]. By frequently solving diagnostic clinical problems doctors gradually generate so-called networks of organized knowledge in their memory. Increased exposure to patients increases experience and the networks condense into an easy accessible library of diagnostic scripts. Each script contains clinically relevant information about a certain disease, its consequences, the context in which it develops, personal circumstances and experiences of the doctor with patients. Due to recognition, more experienced doctors are then able to choose the right script for solving similar sets of diagnostic problems in an efficient manner, particularly in routine cases [4]. Less is known about this expertise with regard to therapeutic problem solving, i.e. choosing and prescribing a (drug) treatment. When they have to decide on a treatment for a patient with a frequently occurring disease, experienced doctors confine themselves to information that is already stored in their memory [5, 6]. Typically, a choice is made between two to five drug and nondrug treatments that are part of their personal standard treatment guideline. The actual choice is usually heuristic or according to rule of thumb [5]. In addition to the requirement that the setting resembles that of the future profession, three other requirements of context learning have been acknowledged: repetition, feedback and responsibility [2]. Within the relevant setting students should repeat the process of problem solving in several different patient cases. Students thereby add new knowledge and experiences to their existing knowledge networks, from which easily accessible scripts gradually evolve. The third requirement is that students gain feedback about their performance, preferably immediately after each patient case. This greatly contributes to the generation of knowledge networks and a proper condensation of these networks into scripts. Finally, students should be responsible for their own learning as much as possible. In this way there is ample opportunity to repair deficiencies in knowledge and skills as they become apparent during the clinical work and feedback. There are many forms of context learning [2]. With respect to the medical curriculum the most extreme form is learning by doing: (daily) clinical work by which medical knowledge is gained and skills are trained simultaneously. Other forms relate to a different concreteness of the setting, which ranges from actual clinical practice with patients through practising with standardized patients or observing patient demonstrations to solving written patient problems or reading case descriptions in a clinical textbook. In the present paper a simulation-based context-learning pharmacotherapy programme for undergraduate medical students is described. The methods and results of an evaluation study are also presented. The aim of this study was to evaluate the effect of the programme, in particular with respect to the mastering of cognitive pharmacotherapeutic skills, i.e. choosing a (drug) treatment and determining patient information. Pharmacotherapy programme The context-learning pharmacotherapy programme is made up of weekly organized role play sessions for approximately 100 2nd, 3rd and 4th year medical students and an Objective Structured Clinical Examination (OSCE) in the outpatient clinic for 4th year medical students. The basic unit of the role play sessions and the OSCE consists of three consultation rooms, a waiting room and nine students (three doctors, three patients and three assessors ) (see Box 1). Each weekly organized role play session consists of 12 basic units allowing 108 (12 9) students to practise three consultations. A student has to attend 15 role play sessions in the 2nd to 4th year (five times doctor, patient and assessor ). Finally, each 4th year student has to sit for the therapeutic OSCE. Students can enrol themselves for the role play sessions and the OSCE through the internet. For the role play sessions and the OSCE, 54 written patient cases were developed, three patient cases with a different level of complexity for each of 18 core diseases. In collaboration with clinicians of the relevant clinical departments the 18 core diseases were selected from 68 core diseases which in a national survey had been identified as final year learning objectives [7]. The levels of complexity are: (A) a first consultation of a Br J Clin Pharmacol 62:6 667

J. A. Vollebregt et al. Box 1 Basic unit of the role play sessions and Objective Structured Clinical Examination (OSCE) D D D P A P A P A WAITING ROOM Three consultation rooms and a waiting room for three doctors, patients and assessors. D = doctor, P = patient and A = assessor. The role play sessions consist of three phases including: consultation, argumentation and feedback. First, the three student doctors have to perform three therapeutic consultations of 10 min each. Before the start of a consultation, the doctors are allowed to enter the consultation rooms where three written patient cases have been placed. In these written cases, all patient information is given, including the (differential) diagnosis. Outside each consultation room, one patient is paired with one peer assessor. Each couple is assigned to one of the three patient cases, including instructions for playing the role and a precoded sheet for assessing the performance of the doctors. After 5 min of preparation, the couples enter the consultation rooms. Then the doctors have 10 min to choose the (drug) treatment interactively with the patient, write it down on a treatment/prescription form and give the necessary patient information. All doctors are allowed to use any source of information. The peer assessors observe the consultation and record several aspects of the treatment choice and patient education on the assessment form by using a four-point scale. After 10 min the couples leave the consultation room and move to the waiting room. The doctors then have 5 min of preparation for the next patient, while the couples have time for completing the assessment of the performance of the doctor they visited. After 5 min, each couple enters the next consultation room. This procedure is repeated until the doctors have treated the three patients, and the patient assessor couples have seen all three doctors. The second phase (argumentation) starts immediately after the three consultations. The patients and peer assessors visit the doctors in the same way again. In 3 5 min, they ask the three doctors about the argumentation for the chosen (drug) treatment and record it on the assessment form. For the third phase (feedback), all nine students sit together and discuss the various (drug) treatments selected and performances. A clinician and clinical pharmacologist are present as facilitators. The form of the OSCE is similar to the role play sessions. The main differences are that the students have to treat three standardized patients in the outpatient clinic and are being assessed by clinical specialists. middle-aged patient without complicating clinical features; (B) a second consultation concerning insufficiently effective treatment without unwanted effects or the occurrence of unwanted effects in combination with an adequate therapeutic response; and (C) a first or second consultation of a patient with comorbidity and/or comedication, or other complicating clinical features. All cases were of a standard design with general patient information (e.g. age, gender, occupation and pregnancy), a summary of previous and current diseases and treatments (comorbidity and comedication) and an extensive description of the patient s medical record, physical examination data, results of physical and laboratory examination and the diagnosis. During each consultation the doctor, interactively with the patient, had to choose the right (drug) treatment and to inform the patient adequately. The selected treatment and a summary of the patient information had to be written down on a treatment/prescription form. Doctors were allowed to use any source of information or communication. The peer assessors (role play) and clinical examiners (OSCE) observed the consultation and recorded several aspects of the treatment choice and interactive patient information process on a structured assessment form by using a four-point scale (0, insufficient; 1, moderate; 2, sufficient; 3, excellent). Evaluation One year after the full implementation of the programme (2002), a random selection was made of 135 2nd and 3rd year students who had enrolled as a doctor during the role play sessions. Also, 49 4th year students who attended the OSCE were randomly selected. This was on average 25% of the whole year classes. Neither the students nor the clinical examiners were informed in advance about the fact that sessions were evaluated. As a reference group, the OSCE scores of 66 6th year graduates from a previous study who had not followed the pharmacotherapy programme were included [8]. For the role play sessions and the OSCE the students were handed a selection of three out of 18 patient cases in such a way that these cases differed with regard to the disease and the level of complexity (A, B, C). In addition to the regular observation and assessment by peer assessors (role play) and clinical examiners (OSCE), the sessions were audiotaped. For each patient 668 62:6 Br J Clin Pharmacol

A pharmacotherapy context-learning programme case the various treatments as recorded on the treatment/ prescription forms and all oral information given to the patients as present on audiotape were blindly summarized on scoring sheets by the investigators. These sheets were presented to six independent clinical experts, one for each disease. They scored the different treatments and patient information by using a four-point scale (0, insufficient; 1, moderate; 2, sufficient; 3, excellent). Additionally, the scores of the questionnaires on study load and appreciation were collected from the students who participated. All data were collected in SPSS 9.0 (SPSS Inc., Chicago, IL, USA) and descriptive statistics were calculated regarding the cognitive skills, study load and appreciation. Differences were analysed by means of Student s t-test. Significance was accepted at a two-tailed P < 0.05. Results The results are presented in Table 1 as a percentage of the required level for graduation. An (ideal) maximum level of 100% has been taken for the core learning objectives [7] used. The mean score of choosing a (drug) treatment of the 4th year students who had followed the contextual learning course was 63.9% of the maximum level for graduation. This score was significantly lower than the score of the 6th year graduates (72.6%) who had not followed the contextual learning course but had completed their clinical internships. The mean score of the clinical examiners (63.9%) who had observed the performance of the students during the OSCE was significantly higher than that of the clinical experts (50.2%) who scored the treatment plans blindly. The difference was caused mainly by the different scores of the C-level cases (69.8% vs. 39%). On the basis of the scores of the clinical experts, it appeared that there was no significant difference between the therapeutic skill choosing a (drug) treatment of the 2nd, 3rd and 4th year students (43.3%, 45% and 50.2%, respectively). The mean score of determining information of the 4th year students who had followed the contextual learning course was 69% of the maximum required level for graduation. This score was significantly higher than that of the 6th year graduates (54.8%). The mean score of the clinical examiners (69%) who had observed the performance of the students during the OSCE was significantly higher than those of the clinical experts (50.7%) who scored the patient information independently of the therapeutic consultation. This was caused mainly by the different scores of the B-level cases (74.6% vs. 49.2%). The scores of the clinical experts indicated that there was no significant difference between the therapeutic skill determining patient information of the 2nd, 3rd and 4th year students (47.3%, 47.2% and 50.7%, respectively). The average study load of the role play sessions for the 2nd and 3rd year students and for the 4th year students of the role play and OSCE was, respectively, 22, 22 and 33.5 h year 1. This represents approximately 1% of the total study load per year. The students appreciation of the role play sessions for the 2nd and 3rd year students was, respectively, 78% and 82% of the maximum value. The appreciation of the OSCE by the 4th year students was 99% of the maximum value. Discussion The aim of this study was to evaluate the effect of a context-learning pharmacotherapy programme, in particular with respect to the mastering of the cognitive pharmacotherapy skills, i.e. choosing a (drug) treatment and determining patient information. The five main findings of this study are: 1 The level of the pharmacotherapeutic skills of the 4th year preclinical students who followed the programme was not far below the level of the 6th year graduates who had finished their clinical internships but had not followed the preclinical context-learning programme. With regard to the skill choosing the (drug) treatment, the level was lower, whereas for the skill determining patient information the level was higher [8]. 2 Preclinical students reached their skill levels after 15 therapeutic role play sessions with a relatively low study load (1% of the total study load per year). 3 The students greatly appreciated the context-learning programme, in particular the OSCE at the outpatient clinic. 4 Observing the therapeutic consultation sessions leads to a higher assessment of the pharmacotherapeutic skill levels than the assessment of the written treatment plans alone. 5 According to the assessment of the written treatment plans alone, the 2nd to 4th year preclinical students showed no increase in the levels of cognitive pharmacotherapeutic skills. Limitations Because the context-learning programme is only a small part of the whole curriculum (1% of the study load), the measured impact may not be ascribed to this programme alone. However, the higher levels of the pharmacotherapeutic skills of the students after the context-learning programme is in accordance with our previous study [1]. Br J Clin Pharmacol 62:6 669

J. A. Vollebregt et al. Table 1 Evaluation of the assessment of (I) therapeutic skills (a) choosing a (drug) treatment and (b) determining patient information, as a percentage of the maximum attainable score, (II) study load in h year 1 and (III) appreciation as a percentage of the maximum of 2nd, 3rd, 4th who followed, and 6th year graduates who did not follow the context-learning pharmacotherapy programme Role-play sessions 2nd year (n = 61) 3rd year (n = 74) OSCE 4th year (n = 49) Reference* 6th year (n = 66) I. Therapeutic skills a. Choosing (drug) treatment (% of max.) All cases: Clinical examiners 63.9 (58.1, 69.8) 72.6 (71.4, 73.8) Clinical experts 43.3 (38.5, 48.1) 45.0 (40.9, 49.1) 50.2 (45.2, 55.1) Level A: Clinical examiners 60.4 (51.5, 69.3) Clinical experts 45.2 (37.4, 53.0) 45.4 (38.1, 52.7) 55.6 (45.0, 66.1) Level B: Clinical examiners 59.2 (49.0, 69.4) Clinical experts 46.4 (33.9, 58.9) 51.7 (44.6, 58.9) 55.3 (49.7, 60.9) Level C: Clinical examiners 69.8 (61.3, 78.4) Clinical experts 45.0 (38.8, 51.1) 41.6 (36.8, 51.1) 39.0 (30.1, 47.7) b. Determining patient information (% of max.) All cases: Clinical examiners 69.0 (62.2, 72.2) 54.8 (53.3, 56.1) Clinical experts 47.3 (44.5, 50.1) 47.2 (43.8, 50.6) 50.7 (47.3, 54.3) Level A: Clinical examiners 62.1 (52.0, 72.2) Clinical experts 46.6 (42.6, 50.7) 52.4 (47.9, 56.8) 49.2 (42.0, 56.6) Level B: Clinical examiners 74.6 (65.1, 84.6) Clinical experts 44.4 (38.6, 50.3) 45.9 (42.6, 49.4) 49.2 (39.9, 52.5) Level C: Clinical examiners 63.2 (53.3, 73.0) Clinical experts 54.7 (49.8, 59.6) 51.2 (45.5, 56.9) 54.5 (47.7, 61.3) II. Study load (h year 1 ) 22 22 33.5 III. Appreciation (% of max.) 78 82 99 Data of the therapeutic skills are presented as mean scores as percentage of the maximum required level for graduation (100%) and (95% CI). Significant difference between clinical examiners and clinical experts (P < 0.05). Significant difference with the 6th year graduates (P < 0.05). *Objective Structured Clinical Examination (OSCE) scores of a reference group of 6th year graduates, who did not follow the context-learning pharmacotherapy programme, but who finished their clinical internships [8]. Furthermore, other education in therapeutic problem solving during the 2nd to 4th study years is scarce. Since this study included only students from the VU University Medical Centre, it is not known to what extent the results can be generalized to students at other universities. However, since most curricula are more or less of the sequential type, it is very likely that at other faculties the course described in the present study would also increase students therapeutic skills and be similarly appreciated. Unfortunately, it was not possible to include a control group who did not undergo the context-learning programme. The most important reason was the longitudinal character of the programme: a mix of 2nd, 3rd and 4th year students trained across the 2nd to 4th academic years. Instead, we included a reference group of 6th year graduates who had not followed the context-learning programme. To improve the reliability and validity of the OSCE assessment, the OSCE took place in a similar setting to that of the role-played consultations and the same structured assessment form was used. In addition, all clinical examiners were instructed on how to use the structured scoring list. Furthermore, each student was assessed by three different clinical examiners; in most cases the scores and experiences of the clinical examiners did not differ significantly. The scoring of the treatments and patient information by the clinical experts was also blinded to the name of the student, study year and whether it concerned a role play or an OSCE. This particularly improved the reliability of the scoring. 670 62:6 Br J Clin Pharmacol

A pharmacotherapy context-learning programme Interpretation of results The suboptimal pharmacotherapeutic skill levels (main findings 1, 2 and 3) should be discussed while taking into account the four required learning conditions of context learning as mentioned in the Introduction. For several reasons the programme could only partly fulfil these requirements. The setting of the role play sessions that was used in the programme can be placed in the middle of the spectrum of concreteness of the future profession, which runs from real practice with real patients to reading patient cases in textbooks. Role play sessions were the maximum attainable setting for this relatively large number of students. Although all students favoured the role of doctor, the opportunity to observe, assess and discuss the performance of three other doctors was also experienced positively. The ideal number of repetitions needed to obtain sufficient experience is not known. In view of the available space (consultation rooms), equipment, personnel and teaching time, for about 750 students 15 role play sessions and a therapeutic OSCE for each student was the maximum attainable number of repetitions. The third principle concerns the quality of feedback that students gain about their performance. Ideally, feedback is given immediately after the performance by experienced clinicians who are trained in giving feedback [9]. The feedback during the role play sessions was performed by peers who lacked clinical experience and had not been trained. The last principle for efficient context learning is the students responsibility for their own learning. Ideally, students should have made good the lack of knowledge and skills that appeared during the role play sessions and the feedback. The present study has not examined this effect but the relatively low study load indicates that it was suboptimal. The differences between the OSCE scores of the clinical examiners and the clinical experts (main findings 4 and 5) can also be explained by taking into account the differences in the A, B and C-level cases. In contrast to the clinical experts, the clinical examiners observed the students in their conversations and discussions with patients. In addition, they discussed the treatment choice with the student for 5 min immediately after the consultation. This has almost certainly contributed to the higher scores of the clinical examiners, particularly with respect to the C and B-level cases, since these cases were more complex than the A-level cases: C-level cases because of complicating patient characteristics such as comorbidity and comedication, B-level cases because no or less effect, or unwanted effects of the treatment occurred. In the C-level cases the argumentation is of great importance for assessing the treatment, and for the B-level cases the skill of communicating these effects to the patient. The clinical experts were deprived of this information, which may explain their lower scores. Conclusions and recommendations The preclinical pharmacotherapy context-learning programme presented here has a small but positive effect on the mastering of cognitive pharmacotherapeutic skills, i.e. choosing a drug treatment and providing the patient with adequate information. This effect has been obtained with a suboptimal form of context learning and a minimal study load. Role play sessions, however, are highly appreciated by students. With regard to the context of the programme, it is recommended that students are offered an environment with the highest level of concreteness [2], allowing students to gain personal experience, preferably by seeing and treating real patients in the clinical setting. In the case that this cannot be achieved, a maximally realistic situation should be created in which students are confronted with the entire process of consultation so that they can train both their diagnostic and therapeutic skills [10, 11]. The effect of a context learning-based programme on both the level of knowledge and the clinical skills of future doctors must be evaluated on a continuous basis. With respect to context learning in the form of role play sessions as presented here, the optimum number of sessions should be determined. Performance and peer assessment can be improved by taking measures by which students prepare themselves better, e.g. by means of a knowledge test at entry and training in peer assessment [12]. The programme, its outcomes and all measures introduced to improve it should be subjected to continuous evaluation. When applied in this manner, context learning in pharmacotherapy is a useful method to familiarize larger numbers of preclinical medical students with the art of rational drug prescribing and patient education. Financial support from the Dutch Ministry of Public Health, Welfare, and Sports (VWS), the Dutch Health Insurance Board (CVZ), the Dutch Association of the Research-based Pharmaceutical Industry (Nefarma) and the VU University Medical Center is acknowledged. References 1 Vollebregt JA, Metz JC, de Haan M, Richir MC, Hugtenburg JG, de Vries TP. Curriculum development in pharmacotherapy: testing the ability of preclinical medical students to learn therapeutic problem solving in a randomized controlled trial. Br J Clin Pharmacol 2006; 61: 345 51. Br J Clin Pharmacol 62:6 671

J. A. Vollebregt et al. 2 Coles C. How students learn: the process of learning. In: Medical Education in the Millenium, eds Jolly B, Rees L. Oxford: Oxford University Press 1998: 63 82. 3 Schmidt HG, Norman GR, Boshuizen HP. A cognitive perspective on medical expertise: theory and implication. Acad Med 1990; 65: 611 21. 4 Charlin B, Tardif J, Boshuizen HP. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med 2000; 75: 182 90. 5 Denig P, Haaijer-Ruskamp FM. Therapeutic decision making of physicians. Pharm Weekbl Sci 1992; 14: 9 15. 6 de Vries TP. Presenting clinical pharmacology and therapeutics: a problem based approach for choosing and prescribing drugs. Br J Clin Pharmacol 1993; 35: 581 6. 7 Vollebregt JA, Metz JC, de Haan M, de Vries TP. Learning objectives for undergraduate pharmacotherapy knowledge, skills and attitudes in the Netherlands; a national survey. In: From Learning Objectives to Student s Competence; Transformation Into a Pharmacotherapy Context-Learning Programme. Dissertation VU University Amsterdam 2004. 8 Vollebregt JA, Metz JC, de Haan M, de Vries TP. The competence in pharmacotherapy of final year medical students in the Netherlands: a descriptive study. In: From Learning Objectives to Student s Competence; Transformation Into a Pharmacotherapy Context-Learning Programme. Dissertation VU University Amsterdam 2004. 9 Branch WT Jr, Paranjape A. Feedback and reflection: teaching methods for clinical settings. Acad Med 2002; 77: 1185 8. 10 Ashley EA. Medical education beyond tomorrow? The new doctor Asclepiad or Logiatros? Medical Education 2000; 34: 455 9. 11 Dacre JE, Fox RA. How should we be teaching our undergraduates? Ann Rheum Dis 2000; 59: 662 7. 12 Sluijsmans D. Peer assessment als complexe vaardigheid. Tijdschrift Voor Med Onderwijs 2002; 21: 167 72. 672 62:6 Br J Clin Pharmacol