Failure of Internet-based audit and feedback to improve quality of care delivered by primary care residents

International Journal for Quality in Health Care 2005; Volume 17, Number 5: pp. 427 431 Advance Access Publication: 14 April 2005 Failure of Internet-based audit and feedback to improve quality of care delivered by primary care residents 10.1093/intqhc/mzi044 STEVEN R. SIMON AND STEPHEN B. SOUMERAI Department of Ambulatory Care and Prevention, Harvard Medical School and Harvard Pilgrim Health Care, Boston, Massachusetts, USA Abstract Objective. To determine the effectiveness of Internet-based audit and feedback to physicians to improve care for diabetes and hypertension. Design. Time-series analysis of an intervention. Methods. The study setting was Harvard Vanguard Medical Associates, a 14-site multispecialty group in greater Boston. The study period was July 1997 June 1999. Participants were 12 primary care internal medicine residents who provided care to adult patients with diabetes (n = 76 pre-intervention and n = 88 post-intervention), hypertension (n = 329 pre-intervention and n = 338 post-intervention), or both (n = 62 pre-intervention and n = 71 post-intervention). We determined the proportion of each resident s patients whose care fulfilled national guidelines for quality (i.e. diabetes patients had hemoglobin testing in the previous 6 months or hypertension patients received a β-blocker or diuretic in the same time period). After meeting individually with each resident to obtain informed consent and to encourage participation, we sent each resident information for accessing his or her practice profile on a secure website. The main outcome measures were (i) the proportion of resident physicians who accessed their profiles and (ii) change following the intervention in the proportion of patients whose care followed national guidelines. Results. Over a 1-year period, only four of the 12 residents accessed their websites. One of the residents visited her site three times, while the other three residents visited their sites once each. In interrupted time-series analyses, the intervention had no discernible effect on adherence to practice guidelines for diabetes or hypertension. Conclusion. The lack of participation in this Internet-based intervention may have important implications for the development of future programs that require physicians to interact with technology to improve quality of care. Keywords: diabetes mellitus, graduate medical education, guideline adherence, hypertension, managed care programs, quality of health care The capacity of the Internet to serve as both a tool for efficient communication and a resource for information makes it an appealing medium for implementing physician behavior-change intervention. Physicians are already using the Internet for a variety of activities, such as the retrieval of information and the submission of prescriptions electronically [1,2]. However, no controlled studies have demonstrated the value of the Internet in improving quality of care. Audit and feedback have been shown to result in small-tomoderate improvements in physicians practice [3]. Delivering practice-based feedback to physicians via the Internet, with facilitated linkages to related educational information, would be a relatively inexpensive intervention that could be implemented in any practice setting with Internet access. Because important gaps exist in the quality of care for hypertension and diabetes [4], we studied the acceptability of delivering practice-based feedback to primary care physicians via the Internet to improve treatment of these common conditions. Methods Setting The study was conducted within Harvard Vanguard Medical Associates (HVMA), a multi-specialty group practice caring for approximately 250 000 members at 14 practice sites in Eastern Massachusetts. HVMA has an electronic health record system that captures data from each patient encounter, Address reprint requests to Steven R. Simon, Department of Ambulatory Care and Prevention, Harvard Medical School and Harvard Pilgrim Health Care, 133 Brookline Avenue, Sixth Floor, Boston, MA 02215, USA. E-mail: steven_simon@hms.harvard.edu International Journal for Quality in Health Care vol. 17 no. 5 The Author 2005. Published by Oxford University Press on behalf of International Society for Quality in Health Care; all rights reserved 427

S. R. Simon and S. B. Soumerai all diagnostic imaging and laboratory studies, all prescription medications, and a set of demographic, benefit, and membership data. A major teaching affiliate of Harvard Medical School, HVMA administers two primary care residency training programs [5]. Study period The overall study period was 24 months, August 1997 July 1999. The study intervention occurred on 1 August 1998, the midpoint of the observation period. Physician and patient participants All 12 primary care residents (post-graduate years 2 and 3) agreed to participate in the study. We identified two cohorts of patients with diabetes and/or hypertension: one preintervention cohort and one post-intervention cohort. Adult patients whose primary care physician was one of the physician participants or who had two or more face-to-face encounters with one of the physician participants during the pre-intervention or post-intervention period were included. Patients could be included in either or both cohorts. Eligible patients with hypertension and/or diabetes mellitus were identified using the automated medical record based on previously used criteria [6]. The multifaceted intervention was encouragement by the residents program directors to access the Internet to obtain audit and feedback on their practice, with accompanying pertinent educational material. In personalized letters to each of the residents, the residency program directors endorsed the study, described the project as an opportunity for education and quality improvement, and encouraged the residents to access their information via the Internet. Feedback was provided on compliance with evidence-based practice guidelines via the Internet. One of us (S.R.S.) met individually with each of the 12 resident physicians in July 1998 to obtain their consent to participate in the study. These one-on-one face-toface meetings introduced the residents to the study, encouraged them to participate by accessing their personalized, password-protected website to review an audit of their practice, and provided them with contact information for any questions or concerns. The meeting itself was not intended to change practice behavior and addressed the clinical goals of the intervention only when raised by the resident physician. We determined each resident s rates of adherence to the two guideline-recommended measures (glucose monitoring and hypertension prescribing) during a 6-month baseline period (1 January 30 June 1998) and prepared graphical representations of these data, customized for each resident physician (Figure 1). We then incorporated each resident s feedback into a password-protected website (accessible from any computer with Internet access at work, home, or elsewhere) that was easily reached directly via URL or through an educational web portal with which the residents were already familiar. Accompanying the graphical representation of quality data were brief explanations of the underlying evidence in support of the guideline-based recommendations, as well as hypertext links to resources for learning more about the recommendations, the guidelines, and the clinical conditions themselves. We also included on the site several links to medical information sources. We provided residents with passwords and directions for accessing their websites via printed letter and electronic mail during August 1998. Process measures We used extreme TRACKING (http://www.extreme-dm. com/tracking/) to monitor the frequency with which each resident accessed his or her website. Quality measures We ascertained from the automated medical record whether patients with diabetes mellitus had a measurement of longterm glucose control, i.e. glycosylated hemoglobin or hemoglobin (Hb) A 1c, during each month of the study. Similarly we determined whether each patient with hypertension filled a prescription for a first-line anti-hypertensive agent (a β-blocker or a diuretic) during each month of the study. Analysis We used interrupted time series, a strong quasi-experimental design [7 9], to determine whether the intervention was associated with a change in trend in practice patterns for diabetes test ordering and hypertension prescribing, controlling for pre-intervention trends. These models included a constant term, a linear time trend, and terms to estimate changes in the level and trend of each practice pattern after the intervention. Analyses were conducted using SAS Proc Autoreg [10] for time-series models. Results Baseline characteristics We identified 202 patients with diabetes or hypertension or both who were seen by one or more of the 12 residents during the baseline period. For 147 of these 202 patients, the electronic medical record identified the resident as the primary care provider. Among the 202 patients, mean age was 56 years, and 56% were women; 129 patients had hypertension, 36 had diabetes, and 37 had both. Among patients with hypertension, 60% had been dispensed a diuretic or β-blocker. Among patients with diabetes, 79% had undergone HbA 1c testing during the 6-month baseline period. The proportion of patients of each resident physician whose care at baseline met guideline recommendations is shown in Figure 1. 428

Internet audit and feedback A Percent of Hypertensive Patients Receiving Recommended First-Line Agents (Diuretics or Beta Blockers). Your data shown by the WHITE bar. Percent Receiving First- Line Agents B Percent Undergoing Testing 100 80 60 40 20 0 100 80 60 40 20 0 ALL A B C D E F G H I J K L Residents Percent of Diabetic Patients Undergoing Recommended HbA1c Testing in Prior 6 Months. Your data shown by the WHITE bar. ALL A B C D E F G H I J K L Residents Figure 1 Audit and feedback graphics for Resident F, indicating the percentage of the resident s patients with hypertension who had received a diuretic or β-blocker (a) and the percentage of patients with diabetes undergoing HbA 1c testing (b) in the 6- month period preceding the intervention, as well as peer comparisons of these rates. The figure was presented to residents in color but was modified to black and white for this publication. Residents use of the website Only four of the 12 residents (residents A, B, D, and K in Figure 1) accessed their websites. One of the residents visited her site three times, while the other three residents visited their sites once each. The proportion of patients with guideline-adherent care for hypertension at baseline was similar among resident physicians who did and did not access their websites (58%). Those residents who accessed their websites had a somewhat lower proportion of patients adherent to diabetes testing guidelines at baseline (71% versus 88%). Effects of intervention We identified a total of 467 patients pre-intervention and 497 patients post-intervention with diabetes, hypertension, or both, who met eligibility criteria. Figure 2a shows the percentage of patients with diabetes who underwent glycemic monitoring in each month of the study. There was no change in the level or slope of the trend line when comparing the time series before and after the intervention. Similarly, Figure 2b shows the proportion of continuously enrolled patients with hypertension who received a diuretic or β-blocker in each month of the study. Again, there was no change in the level or slope of the time-series trend line before or after the intervention. We measured no effect of the intervention on mean HbA 1c or on average blood pressures in the relevant cohorts (data not shown). We examined the time-series plots for diabetes and hypertension stratified by those providers who did view their feedback on the Internet (n = 4) and those who did not (n = 8). We observed no apparent effect of the intervention in either stratum (data not shown). 429

S. R. Simon and S. B. Soumerai A 50% 40% 30% 20% 10% B 0% 50% 40% 30% 20% 10% 0% Figure 2 Monthly percentage of patients with diabetes undergoing HbA 1c testing (a) and patients with hypertension receiving diuretic or beta-blocker (b), and best-fitting trend (solid line), based on the final time-series regression model described in the text. Discussion 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Month In this study to test the effectiveness of Internet-based audit and feedback to improve primary-care management of diabetes and hypertension, we found that only four of 12 resident physicians accessed their own personalized, password-protected websites, despite a face-to-face visit from the study investigator and a letter of endorsement from their training program director. Not surprisingly, we detected no impact of this multifaceted intervention on residents ordering of tests to monitor diabetes control or their prescribing of anti-hypertensive medication. The lack of effect we observed should not be construed as evidence 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Month that audit and feedback are ineffective. Instead, the lack of participation in this Internet-based intervention provides important lessons for the development of future programs that require physicians to interact with technology to improve quality of care. We included patients for analysis if the study physician was named as their primary care physician or if the study physician had two or more face-to-face visits with them. Although virtually all the patients had encounters with providers other than the study physicians, which could have diluted any intervention effect, the inclusion criteria ensured that the study physicians would have either felt responsibility for the patient (as primary care physician) or would have had 430

Internet audit and feedback at least two opportunities to address the quality of care issues targeted in the intervention. Why did only four of the 12 residents access their websites? Unfortunately, we did not formally contact the residents upon completion of the study to ask them. Thus, in implementing a program that requires physicians to take action to engage with technology, a key lesson learned is to assess physician engagement early in the process and to diagnose the underlying causes of non-engagement through interviews and other investigative methods. We can conjecture several likely explanations for the residents non-participation. Residents, like practicing physicians, act on issues that require their immediate action, e.g. the patient in front of them or the request from a nurse, colleague, or clinical assistant. Asking physicians to go to a website that is not a part of their daily routine, and requiring them to input a password that they may have misplaced or may not remember, may simply present too many barriers to participation. The fact that most of these resident physicians, who would be expected to value selfevaluation as a learning tool, did not access their data suggests that busy, practicing physicians cannot be expected to go out of their way to access technology for quality improvement efforts. Several other limitations deserve mention. This study included only 12 residents in two primary care residency programs housed within an integrated delivery system known for its information systems and coordination of care. As such, the results may not be generalizable to residents in other settings or, more generally, to physicians in practice. The finding that three of the four residents who did access their website did so only once could also suggest that they did not find the material interesting or useful. Surveying or interviewing the residents before launching the intervention might have led to more useful or interesting content or format; engaging physicians in program design is a well-established principle of physician behavior change [11]. Additionally, this Internet-based intervention was implemented in 1998, at a time when, compared with today, the use of the Internet was not so woven into the fabric of physicians daily personal and professional lives. It is possible that similar interventions implemented today would have a better chance of improving care, given physicians current familiarity and comfort with the Internet. Improving the primary care management of diabetes and hypertension remains an important objective of efforts to translate research into practice. The lack of enthusiastic participation in this study among resident physicians, whose program directors had encouraged them to participate, suggests that future interventions using the Internet and other information technology should take care to ensure physicians participation and engagement. Acknowledgements We thank Fang Zhang and Claire Canning for support in data analysis and Stephanie Chauvet for assistance with manuscript preparation. This work was supported by funding (1F32HS000128) from the Agency for Healthcare Research and Quality and the Harvard Pilgrim Health Care Foundation. Drs Simon and Soumerai are investigators in the HMO Research Network s Center for Education and Research on Therapeutics (CERT) funded by the Agency for Healthcare Research and Quality (U18HS10391). The funder had no role in the design of the study, analysis of the data, interpretation of the results, or the decision to publish. References 1. Baker L, Wagner TH, Singer S, Bundorf MK. Use of the Internet and e-mail for health care information: results from a national survey. JAMA 2003; 289: 2400 2406. 2. Mullner RM. The Internet and healthcare: opportunities and challenges. J Med Syst 2002; 26: 491 493. 3. Jamtvedt G, Young JM, Kristoffersen DT, Thomson O Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2003. 4. McGlynn EA, Asch SM, Adams J et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003; 348: 2635 2645. 5. Epstein A, Pullock D. The HMO and the academic center: A primary care residency. HMO Practice 1988; 2: 133 138. 6. Soumerai SB, Mah C, Zhang F et al. Effects of health maintenance organization coverage of self-monitoring devices on diabetes selfcare and glycemic control. Arch Intern Med 2004; 164: 645 652. 7. Gillings D, Makuc D, Siegel E. Analysis of interrupted time series mortality trends: an example to evaluate regionalized perinatal care. Am J Public Health 1981; 71: 38 46. 8. Soumerai SB, Avorn J, Ross-Degnan D, Gortmaker S. Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity. N Engl J Med 1987; 317: 550 556. 9. Ross-Degnan D, Soumerai SB, Fortess EE, Gurwitz JH. Examining product risk in context. Market withdrawal of zomepirac as a case study. JAMA 1993; 270: 1937 1942. 10. SAS Institute Inc. SAS/ETS User s Guide, Version 6. 2nd Edn. 1993. Cary, NC. 11. Greco, PJ, Eisenberg JM. Changing physicians practices. N Engl J Med 1993; 329: 1271 1273. Accepted for publication 8 March 2005 431