Eur Radiol (2009) 19: 31 36 DOI 10.1007/s00330-008-1098-3 COMPUTER APPLICATIONS Petter Hurlen Truls Østbye Arne Borthne Fredrik A. Dahl Pål Gulbrandsen Do clinicians read our reports? Integrating the radiology information system with the electronic patient record: experiences from the first 2 years Received: 14 March 2008 Accepted: 6 June 2008 Published online: 6 August 2008 # European Society of Radiology 2008 P. Hurlen (*). F. A. Dahl. P. Gulbrandsen Helse Sør-Øst Health Services Research Centre, Akershus University Hospital, Sykehusveien 27, 1478 Lørenskog, Norway e-mail: petter@hurlen.no Tel.: +47-6792-8647 Fax: +47-6792-8637 P. Hurlen. A. Borthne Centre for Diagnostic Imaging, Akershus University Hospital, Sykehusveien 27, 1478 Lørenskog, Norway T. Østbye Department of Community and Family Medicine, Duke University Medical Center, 318 Hanes House, DUMC box 2914, Durham, NC, 27710, USA T. Østbye Duke-NUS Graduate Medical School Singapore, 2 Jalan Bukit Merah, 169547 Singapore P. Gulbrandsen Faculty of Medicine, University of Oslo, Forskningsveien 3 A, 0316 Oslo, Norway P. Gulbrandsen Faculty Division Akershus University Hospital, University of Oslo, Sykehusveien 27, Lørenskog 1478, Norway Abstract This study aimed to determine how clinicians adapted to and utilized new routines for accessing radiology reports after the integration of an electronic patient record (EPR) with a radiology information system (RIS). Activity-related data describing the availability and receipt of radiology reports were collected from the EPR and the RIS over a period of 2 years. Twelve percent of the final radiology reports had not been opened 4 weeks after they had been entered into the EPR. For opened reports, the median time after a report was available in the EPR until it was first opened by a clinician was less than 1 h for preliminary reports and less than 4 h for final radiology reports. The use of radiology reports was stable during the second observation year. Some reports were not opened for professional as well as technical reasons. The integrated information systems offered a potential for improving routines related to the transmission of radiology reports. Clinicians did not fully take advantage of this potential in the 2 years after its introduction. Keywords Radiology reporting. Electronic patient record. Radiology information system. Health services research. Radiology informatics Introduction Diagnostic imaging is important to modern medicine, and the results of radiology examinations may have vital influence on clinical decisions [1, 2]. Clinicians, the referring physicians, can access the results of radiology examinations in different ways. Some clinicians interpret images themselves [3 5]. Others consult radiologists directly for a first interpretation. However, the final and most comprehensive conclusions of the examination, the radiologists interpretation and diagnostic considerations, are presented in radiology reports [6]. Consequently, to fully utilize the results of diagnostic imaging, clinicians should read the reports. The potential capability of radiology information systems (RIS) and picture archiving and communication systems (PACS) is well documented [7 15]. The integration of these systems with the electronic patient record (EPR) in addition facilitates the access to and utilization of images and radiology reports. As an added benefit, the automatic logging of information availability and use enables evaluation of the clinicians utilization of radiology information.
32 The objectives of this study were to investigate how soon and to what extent clinicians read radiology reports in the EPR, to assess how clinicians adapt to this new routine, and to identify factors that influenced the use of the radiology reports. Material and methods Approval for this study was obtained from the Norwegian Social Science Data Service (NSD) and the Regional Ethics Committee, and exempted from review by the Duke University Medical Center Institutional Review Board. In this retrospective study, data were retrieved from the RIS and the EPR at a university-affiliated county hospital in Norway. The data were chosen to reflect the different steps in the radiology work flow. Work flow Before PACS was introduced, images from radiology examinations were available to clinicians in the reading rooms at the radiology department, where radiologists reviewed the images and summarized their findings and interpretation in preliminary reports. In accordance with the tradition in northern Europe, radiologists presented the results of all radiology examinations from the last 24 h to clinicians during morning radiology rounds in the radiology department. These rounds were performed by specialists in radiology for each clinical department between 7:45 A.M. and 9:30 A.M. every day except Sundays. The clinicians took notes, and both clinicians and radiologists regarded the results presented during these rounds as the final conclusions from the radiology department. The radiologists reviewed the images and preliminary reports in preparation for the clinical demonstration, and made the necessary corrections to the preliminary reports after the demonstration. The revised final reports were carried to the wards and made available to clinicians shortly after 3 P.M. the day of the radiology round. Traditionally, a responsible clinician reviewed and signed the reports the same afternoon, before storing them in paper patient records. The radiology department had used a RIS since 1999, upgraded in 2004 (Siemens MagicSAS, Erlangen, Germany). The EPR (DIPS EPJ, Bodø, Norway) was introduced hospital wide in the spring of 2004. In May 2005, a PACS (Siemens MagicView, Erlangen, Germany) was implemented. This led to a work flow change in the radiology department that enabled the integration of PACS and RIS with the EPR. After the integration, clinicians could review images in the EPR as soon as they were acquired, and read the reports as soon as they were typed. The radiology department practiced double reading, as this is known to reduce the error rate [16, 17]. In accordance with common practice [18, 19], the preliminary radiology report from the first reading was sent to the EPR as soon as it was typed. The final report from the second reading replaced the preliminary version, but the EPR version handling mechanism also saved the preliminary report. The new work flow is illustrated in Fig. 1. In order to prevent reports from being overlooked, the EPR supplied a clinicians work-list function. The hospital defined a set of individual as well as ward-affiliated worklists. Each new entry of a radiology report into the EPR should be listed in one, and only one, work-list according to hospital-defined rules. These rules were based on criteria such as referring physician, patient status, and ward affiliation, and could be modified by the IT department. The EPR logged the entry time of each new radiology report, and the time this report was first opened by a clinician with legal access, i.e., a clinician responsible for the patient. This applied both to preliminary and final reports. In addition, the EPR logged when a report was entered into and checked out of a work-list. Data sets This study was initiated by the redefined work flow in May 2005. To monitor the clinicians adaptation to the work flow change, data sets representing one full week s examinations were retrieved every 4 months during the study period, until February 2007. The data sets consisted of time stamps from the radiology work (Fig. 1), as well as examination-related variables such as patient identifier, modality, referring physician, patient category, and degree of urgency. Modalities included computed radiography (CR), computed tomography (CT), magnetic resonance (MR), ultrasonography (US), and others. Data sets from RIS and EPR were merged based on the unique patient identifier and the image acquisition time. All patient identifiers were removed before the data were made available for statistical analysis. Since the upgraded RIS was introduced well before the EPR integration, we were able to retrieve a data set PACS Image acquisition RIS Image available Preliminary report Final report EPR Preliminary report available Preliminary report opened Final report available Final report opened Final report checked out Fig. 1 Schematic diagram of the radiology reporting work flow after the EPR integration
33 describing image acquisition time in a similar fashion representing the status 4 months before the EPR integration and the work flow change. During the study period, the radiology reports were organized in the EPR according to the initial request. If multiple examinations were requested simultaneously, the results of these examinations were merged into one report. Unfortunately, these reports were tagged with the image acquisition time for only the first examination. Thus, multiexamination reports were excluded from this study. Because of an expected Hawthorne effect [20] and an expected high dropout rate, we decided to estimate when paper reports were read by clinicians before the integration rather than attempt to observe it or request the clinicians to record it. The average preintegration radiology round time was set to 8:30 A.M. Monday to Saturday, and the average time the paper report was read was set to 3:30 P.M. the day of the radiology round. The preintegration median time from image acquisition to the radiology round and to paper report reading was then estimated from these time points and the image acquisition time. Statistical approach We had no initial hypothesis regarding how the new routines would be adopted. In order to be able to identify and statistically verify important changes and trends, the data set was randomly split into two parts. The first part (25%) was used to generate hypotheses, and then discarded. The remaining data (75%) was used for confirmatory statistical analyses, and these are reported below. The purpose of splitting data in this way was to ensure that our statistical tests were performed on data that were not used to generate the hypotheses. This procedure guards against (unintended) hypothesis fishing, and guarantees the integrity of the computed p values [21]. Differences were analyzed using the two-sided nonparametric Mann Whitney U test for ordinal and the T test for nominal data. Significance levels (predetermined at α< 0.05) are reported. SPSS (v. 15.0.1, SPSS Inc.) was used for data management and analysis. The median, rather than the mean value, is reported on time intervals, since the final reporting and the final viewing of a few examinations sometimes are delayed for days or weeks, i.e., their distribution is skewed with the long tail to the right. Results A total of 16,470 radiology reports were entered into the EPR during the 5 weeks of observation. Of these, 4,140 were randomly selected for establishing the initial hypothesis, and then discarded. Of the remaining 12,330 reports, 2,232 contained the results of multiple examinations, and were excluded from the study. Only single-examination reports, 10,098, were included. Of these, 4,555 were reports relating to outpatients, 4,258 to inpatient emergency cases and 2,185 to inpatient routine cases. Unopened reports Figure 2 illustrates the percentage of preliminary and final reports opened within 4 weeks, for each observation period. In total, only 42% of the preliminary and 88% of the final reports had been opened 4 weeks after they became available in the EPR. For final reports, routine inpatient cases had the highest score (92%). Significantly fewer of the emergency inpatient reports were opened (89%, p<0.03), and even less of the outpatient reports (86%, p < 0.01). Orthopaedic surgeons opened fewer of the final reports than the other physicians (85% vs. 90%, p < 0.01); there was no significant difference between the other clinical specialities. Significantly fewer CR reports were opened (87% vs. 91%, p < 0.01), and there was no significant difference between the other modalities. The use of preliminary reports had a significant impact on the use of final reports. If a preliminary version had been opened, 91% of the final versions were also opened, both for inpatient and outpatient cases. If no preliminary version had been opened, only 88% of the final inpatient and 85% of the final outpatient reports were opened (both p < 0.01). The EPR work-list function erroneously listed only 92% of the inpatient reports, i.e., 8% were missed. This had a direct influence on the use of final reports. Ninety-four percent of the listed final reports were opened, but only 29% of the unlisted reports (p<0.01). Initially, the work-list Percentage opened 100 % 80 % 60 % 40 % 20 % 0 % +4m +8m +12m +16m +20m Observation period Preliminary Final Fig. 2 Percentage of preliminary and final reports opened within 4 weeks after they were entered into the EPR, for all patient categories for each observation period
34 function was also used for preliminary reports. This was discontinued before the second post-integration observation period. Median time to report opening Figure 3 shows the median time from an inpatient report was available in the EPR until it was first opened, by type of report (preliminary or final). Preliminary reports were opened significantly earlier than final reports (p<0.01). Emergency case reports were not read significantly earlier than routine case reports, neither preliminary (54 min vs. 1 h 02 min, p=0.09) nor final reports (3 h 11 min vs. 3 h 19 min, p = 0.84). For final outpatient reports, the median time from availability to use was 23 h 28 min. Preliminary inpatient US (37 min) and CT reports (46 min) were viewed significantly sooner than preliminary CR reports (1 h 05 min, p < 0.01). The same applied for final US (2 h 28 min) reports compared with final CR reports (3 h 32 min, p < 0.01), but there was no significant difference between final CR and CT reports (3 h 20 min, p =0.2). Final outpatient reports were read significantly sooner if a previous preliminary version had been opened (19 h 12 min vs. 24 h 49 min, p < 0.01). Final inpatient reports were read later if a previous version had been opened (3 h 33 min vs. 2 h 50 min), but the difference was not statistically significant (p=0.06). Figure 4 shows the median time from a report was available until it was opened, for inpatients from the five major clinical departments: paediatrics, general surgery, neurology, internal medicine, and orthopaedic surgery. Orthopaedic surgeons opened both preliminary and final reports significantly later than other clinicians (p<0.01). Median time (hours) 8 6 4 2 0 +4m +8m +12m +16m +20m Observation period Preliminary Final Fig. 3 Median time from an inpatient radiology report was available in the EPR until it was opened, for preliminary and final reports for each observation period Median time (hours) 21 18 15 12 9 6 3 0 Ped. Surg. Med. Neur. Ortho. Department Preliminary Final Fig. 4 Median time from a preliminary and final report was available in the EPR until it was opened, for the five major clinical departments Adaptation to new routines Figure 5 shows the median time from image acquisition until final reports were available and opened, for each observation period. It also indicates how the new routines compared with the old routines, as the first two bars are estimates of the median time from image acquisition until the radiology rounds, and until the paper reports were read by clinicians. There was no major difference between the old and the new routines. It should be added that, due to extensive demands, many morning radiology rounds were Median time (hours) 30 24 18 12 6 0-4m Integration +4m +8m +12m +16m +20m Observation period Reports presented during radiology round (estimated) Reports available in EPR Paper reports viewed by clinicians (estimated) Reports in EPR opened by clinicians Fig. 5 Estimated median time from image acquisition until a final radiology report was presented during radiology rounds and read by clinicians before the EPR integration. Median time from image acquisition until a final radiology report was available and opened in the EPR, for each observation period, after the integration
35 continued even after the routine change. During these rounds, only the most important and interesting cases were presented. Discussion To obtain the final and comprehensive conclusions of a radiology examination, a clinician must read the final, signed, radiology report. One consequently would expect all reports to be read. This study showed that only 88% of the final reports were actually opened (Fig. 2). We believe that there were professional as well as technical reasons for this relatively low proportion. It was not an introductory problem, as the percentage of unread final reports remained relatively unchanged during the whole period. In some cases, clinicians probably rely on their own interpretation of the images, either because they feel qualified to make an adequate interpretation themselves, or because the reports are not available soon enough. Our results indicate that clinicians felt more competent to interpret CR images than images from the other modalities, as fewer of the CR reports were opened. Also, preliminary CT and US reports were read significantly sooner than CR reports. In particular, it is our impression that in many cases, orthopaedic surgeons interpret the CR images themselves, using the radiology reports mostly for quality assurance purposes. Orthopaedic surgeons read fewer reports and opened final reports much later than their colleagues in other disciplines. In addition, some of the reports may have arrived too late to be of clinical interest, in particular in emergency cases. This may explain why significantly less emergency case than routine case final reports were opened, and why the final inpatient emergency case reports that were opened, were not opened earlier than the routine cases. One would, however, expect that all final reports were read in the end, for quality assurance purposes. Our study indicates that the clinicians appreciated having access to the preliminary reports, as the median time from the reports were available until they were opened was less than 1 h, compared with almost 4 h for final reports. One might suspect that a clinician would be less likely to read a final report if he or she had read a preliminary report. Our results showed the opposite. Possibly clinicians read preliminary reports relating to the examinations they were most interested in and consequently also read the final reports to confirm or adjust the preliminary findings. The rapid opening of these preliminary reports may also be an indication of such interest. In the remaining cases, the clinicians probably waited for the final reports; only approximately 40% of preliminary reports were opened during the last observation year. It is possible that some of the unread reports represented examinations the clinicians in reality had little interested in, perhaps ordered out of routine or by inexperienced residents. However, reports may also be missed for technical reasons. For example, our study showed that the EPRbased work-list function had a direct influence on whether a final report was opened, as 94% of the listed reports were opened, compared with only 29% of the unlisted. If clinicians are to trust such a function, it must be completely reliable. This function was not, as 8% were missed over the study period. We have not explored this finding further in the current study, but both inadequate sorting algorithms and unused work-lists have been reported. The algorithm was improved during the study period. However, this had no major influence on the percentage of reports read. Initially, preliminary reports were also listed in the worklists. This explain why so many preliminary reports were opened 4 months after PACS, and why the median time to opening was so high preliminary reports had to be checked out of the list even if the final report had been opened. Based on clinical demand, the work-list function for preliminary reports was turned off. The clinicians took approximately 1 year to adapt to the new routines. Although not directly comparable, Fig. 5 indicates that, while there was an initial gain, there were no major differences between the old and the new routines at the end of the study period. The decline only applied to final reports, not to preliminary reports. In our opinion, the main reason for this tapering off effect was that, while the radiologist used the new technology to reorganize their work, the clinicians did not use the opportunity to improve their own routines. Even though both images and reports were available through the EPR, many clinicians insisted that the traditional radiology rounds should be continued as before. As a consequence, the radiologists could not fundamentally change their reporting routines. The second reading was increasingly combined with preparation for radiology rounds in the morning, similar to the old routines. This may have contributed to the increase in median time from image acquisition to final report availability illustrated in Fig. 5. On the other hand, delayed availability of final reports probably augmented the need for radiology rounds in order to have the final conclusion available before the clinical rounds in the morning. More comprehensive changes in the daily routines and logistics would require a closer collaboration between radiologists and clinicians. Conclusion To fully utilize the results of diagnostic imaging, clinicians must read the final radiology reports. Our study showed that, for professional as well as technical reasons, 12% of the final reports were not read. Information and communication technologies offer a significant potential for improving patient care. Our study indicates, however, that in addition to technology, more fundamental changes
36 in the daily routines and logistics are also required to improve the quality and reduce the time of information transmission between radiologists and clinicians. More comprehensive routine changes would require a close collaboration between radiologists and clinicians. Acknowledgement This work was funded by the Eastern Norway Regional Health Authority and the Directorate for Health and Social Affaires. References 1. Gunderman RB (2005) The medical community s changing vision of the patient: the importance of radiology. Radiology 234:339 342 2. Smith PC, Araya-Guerra R, Bublitz C et al (2005) Missing clinical information during primary care visits. JAMA 293:565 571 3. Levin DC (1994) Merrill C. Sosman Lecture. The practice of radiology by nonradiologists: cost, quality, and utilization issues. AJR Am J Roentgenol 162:513 518 4. Pilling JR (2003) Picture archiving and communication systems: the users view. Br J Radiol 76:519 524 5. Espinosa JA, Nolan TW (2000) Reducing errors made by emergency physicians in interpreting radiographs: longitudinal study. BMJ 320:737 740 6. Berlin L (1997) Radiology reports. AJR Am J Roentgenol 169:943 946 7. Foord K (1999) PACS: the second time around. Eur J Radiol 32:96 100 8. Kim SA, Park WS, Chun TJ et al (2002) Association of the implementation of PACS with hospital revenue. J Digit Imaging 15:247 253 9. Nitrosi A, Borasi G, Nicoli F et al (2007) A filmless radiology department in a full digital regional hospital: quantitative evaluation of the increased quality and efficiency. J Digit Imaging 20(2):140 148, Epub 2007 Feb 23 10. Siegel EL, Reiner BI (2003) Filmless radiology at the Baltimore VA Medical Center: a 9 year retrospective. Comput Med Imaging Graph 27:101 109 11. Bryan S, Weatherburn G, Buxton M et al (1999) Evaluation of a hospital picture archiving and communication system. J Health Serv Res Policy 4:204 209 12. Mariani C, Tronchi A, Oncini L et al (2006) Analysis of the X-ray work flow in two diagnostic imaging departments with and without a RIS/PACS system. J Digit Imaging 19(Suppl 1):18 28 13. Lindhardt FE (1996) Clinical experiences with computed radiography. Eur J Radiol 22:175 85 14. Reiner B, Siegel E, Protopapas Z et al (1999) Impact of filmless radiology on frequency of clinician consultations with radiologists. AJR Am J Roentgenol 173:1169 1172 15. Reiner BI, Siegel EL, Flagle C et al (2000) Effect of filmless imaging on the utilization of radiologic services. Radiology 215:163 167 16. Fitzgerald R (2001) Error in radiology. Clin Radiol 56:938 946 17. Goddard P, Leslie A, Jones A et al (2001) Error in radiology. Br J Radiol 74:949 951 18. Cohen MD (2008) Making preliminary radiographic reports available to referring clinicians: current status. Acad Radiol 15:127 131 19. Holman BL, Aliabadi P, Silverman SG et al (1994) Medical impact of unedited preliminary radiology reports. Radiology 191:519 521 20. Landsberger HA (1958) Hawthorne revisited. Cornell University Press, Ithaca, New York 21. Dahl FA, Grotle M, Saltyte BJ et al (2008) Data splitting as a countermeasure against hypothesis fishing: with a case study of predictors for low back pain. Eur J Epidemiol 23:237 242