Off-Site Smartphone Reading of CT Images for Patients With Inconclusive Diagnoses of Appendicitis From On-Call Radiologists

Transcription

1 Special Articles Original Research Seong et al. Off-Site Smartphone Reading of CT for Appendicitis Special Articles Original Research Nak Jong Seong 1 Bohyoung Kim 1 Sungmin Lee 2 Hee Sun Park 3 Hyuk Jung Kim 4 Hyunsik Woo 5 Heung-Sik Kang 6,7 Kyoung Ho Lee 6,7 Seong NJ, Kim B, Lee S, et al. Keywords: abdomen, appendicitis, CT, mobile, smartphone DOI: /AJR Received August 23, 2013; accepted after revision November 20, Supported in part by Midcareer Researcher Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (MEST) (NRF ); in part by Basic Science Research Program through the NRF funded by the MEST (NRF-2013R1A1A ); and in part by grant from the Seoul National University Bundang Hospital Research Fund. 1 Department of Radiology, Seoul National University Bundang Hospital, 82 Gumi-ro, 173 Beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do, , Korea. Address correspondence to B. Kim (bhkim@snubh.org). 2 Department of Electrical Engineering and Computer Science, Seoul National University, Seoul, Korea. 3 Department of Radiology, Konkuk University School of Medicine, Seoul, Korea. 4 Department of Radiology, Bundang Jesaeng Hospital, Gyeonggi-do, Korea. 5 Department of Radiology, SMG Seoul National University Boramae Medical Center, Seoul, Korea. 6 Department of Radiology, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Institute of Radiation Medicine, Gyeonggi-do, Korea. 7 Seoul National University Medical Research Center, Gyeonggi-do, Korea. AJR 2014; 203: X/14/ American Roentgen Ray Society Off-Site Smartphone Reading of CT Images for Patients With Inconclusive Diagnoses of Appendicitis From On-Call Radiologists OBJECTIVE. The purpose of this study was to simulate a mobile consultation in patients with inconclusive diagnosis of appendicitis made by on-call radiologists, as well as to measure the diagnostic confidence and performance of the mobile consultation. MATERIALS AND METHODS. Two off-site abdominal radiologists interpreted the CT images from 68 patients (including 29 patients with confirmed appendicitis) on a smartphone for whom the preliminary CT reports by 25 in-house on-call radiologists were inconclusive. The smartphone readings were compared with the preliminary reports by on-call radiologists and with the original final reports by in-house abdominal radiologists. Heat maps, kappa statistics, Wilcoxon signed-rank tests, and ROC curves were used for data analysis. RESULTS. The heat maps and kappa statistics showed that the smartphone readings were more similar to the final reports than to the preliminary reports. In diagnosing or ruling out appendicitis, the smartphone readings were more confident than the preliminary reports (p 0.01) and did not significantly differ in diagnostic confidence from the final reports (p 0.19). The AUCs of the smartphone readings (0.91 and 0.92) did not differ significantly from those of the preliminary (0.85) or final (0.97) reports (p 0.09). CONCLUSION. With the given study sample, the diagnostic performance of the offsite smartphone readings did not differ significantly from that of the in-house preliminary reports. However, the smartphone readings provided higher diagnostic confidence than the preliminary reports. I n the diagnosis of acute appendicitis, preoperative CT is accepted as the standard of care in many developed countries [1 3]. Timely and accurate CT interpretation is important in decreasing unnecessary appendectomies, preventing delays in patient disposition, and improving the overall cost-effectiveness of care [1, 2, 4]. However, many hospitals rely partly on less-experienced radiologists, including residents, for CT interpretations after hours, for night-time coverage or training purposes [5 8]. According to a survey published in 2007 [8], only 10% of academic medical centers in the United States and Canada had 24-hour coverage 7 days a week by attending radiologists for their practice. The preliminary CT reports made by on-call radiology residents can be supplemented later by more-experienced radiologists; however, the interval until the final report could vary considerably with the regional health care system, caregivers motivation, and hospital resources. In addition, it has been re- ported that there must be some learning curve in the CT interpretation, because appendicitis is one of the diseases of which diagnosis made by residents after hours is most frequently missed or corrected [5]. Teleradiology consultation systems utilizing mobile devices have the potential to eliminate the spatial gap between patients and expert radiologists. Recent studies have focused on the technical feasibility of mobile consultation systems such as personal digital assistants, tablet PCs, and smartphones in the diagnosis of appendicitis [9] and in other applications [10 12]. However, in their study designs, little attention was paid to reproducing clinical practice. Specifically, it is unclear in those studies whether the study samples truly represented patient subgroups requiring a mobile consultation system in a medical context. To this end, we conducted a retrospective exploratory study reproducing the practice in a teaching hospital. The purpose of our study was to simulate a mobile consultation for patients with an inconclusive diagnosis of appen- AJR:203, July

2 Seong et al. dicitis made by on-call radiologists, as well as to measure the diagnostic performance and confidence of the mobile consultation system. Materials and Methods This retrospective study was approved by our institutional review board, and the requirement for informed consent was waived. The study included 68 patients for whom the preliminary CT reports made by 25 on-call radiologists were not conclusive for the diagnosis of appendicitis. We simulated a mobile consultation by having two off-site abdominal radiologists interpret CT images on a smartphone. The diagnostic performance and confidence were compared with those of the original in-house CT reports. Patient Selection Our patient selection was to reproduce our own practice in terms of the need for secondary interpretation by an expert radiologist after preliminary reports by an on-call radiologist (Fig. 1). The study sample was extracted from a previous randomized controlled trial that had enrolled the patients with suspected acute appendicitis [13]. The patient enrollment had taken place in the emergency department (with 67,000 annual visits) of Seoul National University Bundang Hospital, a 900-bed tertiary general hospital serving a population of 1.8 million in a region of metropolitan Seoul in Korea. In that trial, in-house radiologists on service prospectively rated the presence of appendicitis on a 5-point Likert scale (Table 1) in the original CT reports of 891 adults randomized to undergo either low- or standard-dose CT for suspected appendicitis from September 2009 through January In the current study, we included only the standard-dose CT group because TABLE 2: CT Parameters the use of low-dose CT may not yet be considered the standard of care in many institutions [13 15]. From the 447 patients in the standard-dose CT group, there were six dropouts. We excluded the following patients in whom the secondary CT interpretation may have not been essential: 225 with initial CT reports made by expert radiologists and 147 with initial CT reports made by on-call radiologists but with reasonably high diagnostic confidence (grade 1 or 5). One additional patient was excluded because this patient s CT report did not conform to the predefined structured format using the 5-point Likert scale. Therefore, our study finally included the remaining 68 patients (15% of the 447 patients assigned to the standard-dose CT group) for whom the on-call radiologists preliminary reports were not conclusive (grades 2 4). They were deemed representative of a population having a potential need for a mobile consultation. The study sample included 32 male and 36 female patients years old (mean [SD] age, 28.1 ± 9.9 years [29.7 ± 10.6 years for male and 26.7 ± 9.2 years for female patients]). Appendicitis was confirmed in 29 patients, whereas the remaining 39 patients were considered not to have appendicitis on the basis of reference standards as follows. In patients undergoing abdominal surgery, a final diagnosis was made on the basis of surgical and pathologic findings. In patients not undergoing surgery, independent assessors determined the final diagnosis on the basis of medical records and telephone interviews 3 months after the patient s initial presentation [13]. CT Imaging Protocol IV contrast-enhanced abdomen and pelvis CT scans were obtained using 16- or higher-channel scanners (Brilliance, Philips Healthcare). The effective radiation dose was aimed at 8 msv in a theoretic hermaphrodite [16]. From each helical scan, two transverse image datasets were reconstructed with different section thicknesses: thick (5 mm) and thin (2 mm). From the thin sections, coronal images were reformatted with a 5-mm section thickness. The technical advantages of this two-tier (thick and thin) image reconstruction have been previously discussed [17]. Detailed CT parameters are presented in Table 2. Considerable efforts had been exercised in the original trial to keep imaging parameters consistent across the CT machines used, particularly in regard to the radiation dose and image quality [13]. Therefore, the effect of the difference between the CT machines was deemed to be unimportant. TABLE 1: 5-Point Likert Scale for the Presence of Appendicitis Grade CT Findings and Recommendation 1 Definitely absent; clinical observation is recommended 2 Probably absent; clinical observation is recommended 3 Indeterminate; clinical observation or surgical exploration is recommended 4 Probably present; surgical exploration is recommended 5 Definitely present; surgical exploration is recommended Parameter 16-Channel Scanner 64-Channel Scanner 256-Channel Scanner Range scanned Diaphragm to symphysis pubis Diaphragm to symphysis pubis Diaphragm to symphysis pubis Detector collimation (mm) , double sampling Rotation speed (s) Pitch Tube potential (kvp) Reference tube current time product (mas) Dose modulation program Patient-to-patient modulation, angular modulation Patient-to-patient modulation, z-axis modulation Patient-to-patient modulation, z-axis modulation Reconstruction details Reconstruction filter Standard soft tissue Standard soft tissue Standard soft tissue Thick transverse images Section thickness, 5 mm; interval, 4 mm Section thickness, 5 mm; interval, 4 mm Section thickness, 5 mm; interval, 4 mm Thin transverse images Section thickness, 2 mm; interval, 1 mm Section thickness, 2 mm; interval, 1 mm Section thickness, 2 mm; interval, 1 mm Coronal reformation Section thickness, 5 mm; interval, 4 mm Section thickness, 5 mm; interval, 4 mm Section thickness, 5 mm; interval, 4 mm 4 AJR:203, July 2014

3 Off-Site Smartphone Reading of CT for Appendicitis TABLE 3: Display System and Reading Conditions for Initial CT Reporting and Mobile Consultation Condition Initial CT Reporting Simulated Mobile Consultation 5-mm-thick images Available Available Viewing mode Standard stack view in axial and coronal planes Standard stack view in axial and coronal planes Display system Typically liquid crystal medical display (ME312L, Totoku Electronic) iphone 4 (Apple) Size (in) 32 (diagonal) 3.5 (diagonal) Resolution (pixels) Pixel pitch (mm pixel pitch) (326 ppi) Color 10-bit gray scale 32-bit RGB Viewing software DS3000, Impax version 4.5 (Agfa HealthCare) Mobile PACS (Infinitt) 2-mm-thick images Available Not available Viewing mode Multiplanar sliding slab averaging Not available Display system Liquid crystal color displays of general purpose Not available (various specifications) Viewing software AquarisNET (TeraRecon) Not available Network 100 Mbit/s Ethernet Secure wireless (Wi-Fi, g) Irreversible image compression None JPEG, quality factor of 100 Light condition Typically 30 lux Normal office light Clinical and laboratory findings Available Not available Consultation with the referring physicians Available Not available Note RGB = red-green-blue imaging. Original CT Reports As a part of our routine practice (and also as part of the previous trial) [13], the original CT reports were made as follows. On-call radiologists 14 residents and 11 board-certified nonabdominal radiologists made the preliminary reports for 29 and 39 patients, respectively. All preliminary reports were supplemented later by addenda from three inhouse abdominal radiologists with 9, 8, and 4 years of experience after board certification. In both the preliminary and addendum final reports, the presence of appendicitis was rated on the 5-point scale. Whenever possible, radiologists suggested alternative diagnosis to explain the abdominal symptoms. The on-call radiologists and in-house abdominal radiologists initially reviewed the 5-mm-thick axial and coronal images and optionally reviewed the 2-mm-thick images using the multiplanar sliding slab averaging technique, which is a real-time image postprocessing technique widely used for the efficient review of large thin-section CT datasets [13 15, 18 21] (Table 3). The results of the original CT reports have been published elsewhere [13], together with the results for other patients not included in the current study. Simulation of Mobile Consultation Two independent off-site abdominal radiologists with 7 and 10 years of clinical experience reviewed the CT images on a smartphone (Table 3) and rated the presence of appendicitis on the 5-point scale. They were also asked to suggest alternative diagnosis whenever possible. They were aware that the patients had suspected appendicitis and that the preliminary reports from the on-call Fig. 1 Flowchart shows study sample selection and study design. Dotted lines represent data exclusion. Bidirectional arrows represent comparisons. 891 Included in the previous trial 447 Underwent standard-dose CT 216 Initial CT reports made by on-call radiologists Preliminary reports by 25 on-call radiologists Final reports by 3 in-house abdominal radiologists radiologists were not conclusive. Neither the inhouse preliminary reports nor subsequent addenda were given to the off-site radiologists. We decided to limit this smartphone review to the 5-mm-thick axial and coronal images, because we 68 Included in the final study sample (Likert scale grades 2 4) 444 Underwent low-dose CT 6 Dropouts 225 Initial CT reports made by expert radiologists 147 Initial CT reports were confident (Likert scale grade 1 or 5) 1 CT report not mentioning 5-point Likert scale Smartphone consultation reports by 2 off-site abdominal radiologists AJR:203, July

4 Seong et al. were not aware of any commercially available mobile solution providing the sliding slab technique for the review of noisy 2-mm-thick images. Because the two radiologists had no prior experience with smartphone reading, they were given a brief introductory session with 10 example cases not included in the study sample before the formal analysis was initiated. All patient data were anonymized and not stored in the smartphone. After the review, the smartphone was reset to its initial configuration. Mobile Consultation System The mobile consultation system (Infinitt mobile viewer 1.2, Infinitt Healthcare) consisted of a mobile PACS server and a smartphone (iphone 4, Apple) running a client image viewing program. The mobile PACS server, equipped with a 3.0-GHz 24-core Xeon X5675 central processing unit (Intel) and 32 GB of main memory, ran an application of Infinitt Mobile PACS Server (Infinitt) on an operating system of Windows Server 2008 R2 x64 Edition (Microsoft) and bridged the client program and the enterprise PACS. The mobile PACS server delivered the request for images from the client program to the enterprise PACS located in the hospital. The images retrieved from the enterprise PACS were compressed to the quality factor of 100 by the mobile PACS server, wherein the quality factor ranges from 0 (lowest quality) to 100 (highest quality), using the irreversible JPEG algorithm. The compressed images were then transmitted to the client program in the mobile device. Of many commercially available mobile devices, we chose iphone 4 running an operation system of ios (Table 3). The client program on the iphone handled user input, communication with the mobile PACS server, and display of the transmitted images. Its image-viewing interface was similar to that of standard radiology workstations. Image magnification, window setting, zooming, panning, and size measurement were provided. The iphone communicated with the mobile PACS server through a wireless network (Wi-Fi g). The mobile PACS server also allowed access to the data under several layers of security using preenrolled terminal authentication and user authentication (i.e., secure identification and password combination), along with the Mobile Device Management solution (version , Mobile Iron). Statistical Analysis We used heat maps and dendrograms to present the complete data on diagnostic confidence [22] (Appendix 1). Interobserver agreement was measured using kappa statistics. The diagnostic confidence in diagnosing or ruling out appendicitis was compared using Wilcoxon signed-rank tests. The AUCs were compared. Diagnostic sensitivity and specificity were calculated, and a decision threshold of grade 3 or higher was considered as positive [23]. Comparisons were made between the on-call radiologists and the in-house abdominal radiologists, between the on-call radiologists and each smartphone reader, and between the in-house abdominal radiologists and each smartphone reader. We formed the residents and board-certified nonabdominal radiologists into a single group of on-call radiologists by assuming minor variation in their experience level in diagnosing acute appendicitis. In terms of alternative diagnosis, we were unable to perform a formal statistical analysis owing to the small number of patients included in each category and the incomplete reference standard that often had to be based on CT results. The results of the suggested alternative diagnoses are presented in Appendix 2. Statistical software was used (MedCalc). Clopper-Pearson 95% CIs were calculated. No adjustments were made for the multiple comparisons in this exploratory study. Results Performance of the Simulated Mobile Consultation The off-site mobile consultation simulation was successfully performed in all cases without any malfunction or delay regarding image query, transmission, display, or navigation. Because a series of images of a patient was transmitted to the mobile device in a background manner, the waiting time from the examination opening to image navigation was negligible. Fig. 2 Heat maps with dendograms represent confidence in diagnosing and ruling out appendicitis. A and B, Heat maps with dendrograms for 29 appendicitis cases (A) and 39 nonappendicitis cases (B). At each intersection in maps, color of grid element indicates grade assigned by corresponding reader (or reader groups) (column) for corresponding patient (row). Darker red corresponds to higher grade, and lighter red corresponds to lower grade. Dendrogram attached to top of each heat map represents similarity in response pattern between readers (or reader groups). Shorter branch indicates higher similarity between readers (or reader groups). OR = on-call radiologist. AR = in-house abdominal radiologist. SR1 = smartphone reader 1. SR2 = smartphone reader 2. TABLE 4: Kappa Statistics Between Combinations of Readers Readers κ OR/AR 0.59 OR/SR OR/SR AR/SR AR/SR SR1/SR Note OR = on-call radiologist, AR = in-house abdominal radiologist, SR1 = smartphone reader 1, SR2 = smartphone reader 2. Overall Interpretation Patterns The complete data of the diagnostic confidence were provided in heat maps (Appendix 1). The two smartphone readers tended to be more confident in diagnosing (overall darker red indicating higher grades in Fig. 2A) and ruling out (overall lighter red indicating lower grades in Fig. 2B) appendicitis than were the on-call radiologists. Overall, the two smartphone readers showed more similar color patterns to the in-house abdominal radiologists than to the on-call radiologists. The kappa statistics are shown in Table 4. Diagnostic Confidence In diagnosing appendicitis in the 29 appendicitis cases, the in-house abdominal radiologists (p = 0.01) and the two smartphone readers (p < 0.01 for both) were more confident (assigning higher grades) than the on-call ra- Grade 5 Grade 4 Grade 3 Grade 2 Grade 1 OR AR SR1 SR2 OR AR SR1 SR2 A B 6 AJR:203, July 2014

5 TABLE 5: Diagnostic Performance Off-Site Smartphone Reading of CT for Appendicitis AUC Difference Readers AUC Compared With OR diologists, and no significant difference was observed between the in-house abdominal radiologists and each smartphone reader (p = 0.46 and p = 0.19). In ruling out appendicitis in the 39 nonappendicitis cases, the in-house abdominal radiologists (p < 0.01) and the two smartphone readers (p < 0.01 and p = 0.01) were more confident (assigning lower grades) than were the on-call radiologists, and no significant differences were observed between the in-house abdominal radiologists and each smartphone reader (p = 0.61 and p = 0.20). Diagnostic Performance The AUC, sensitivity, and specificity in the diagnosis of appendicitis are given in Table 5. The AUC for the in-house abdominal Compared With AR Difference (95% CI) p Difference (95% CI) p radiologists was significantly greater than that for the on-call radiologists (0.97 vs 0.85; p = 0.02). The AUC for each smartphone reader (0.91 and 0.92 for smartphone readers 1 and 2, respectively) did not differ significantly from that for the on-call radiologists (p = 0.26 and p = 0.19, respectively) or the inhouse abdominal radiologists (p = 0.10 and p = 0.09, respectively). Sensitivity, % (95% CI), No. of Patients Specificity, % (95% CI), No. of Patients OR 0.85 NA NA 0.11 ( 0.21 to 0.02) (64.2, 94.2), 24/ ( ), 31/39 AR ( ) 0.02 NA NA 96.6 ( ), 28/ ( ), 33/39 SR ( 0.04 to 0.16) ( 0.12 to 0.01) ( ), 24/ ( ), 34/39 SR ( 0.03 to 0.16) ( 0.11 to 0.01) ( ), 26/ ( ), 30/39 Note OR = on-call radiologist, AR = in-house abdominal radiologist, SR1 = smartphone reader 1, SR2 = smartphone reader 2, NA = not applicable. TABLE 6: Results of Readers Grading Grade Appendicitis Cases (n = 29) Nonappendicitis Cases (n = 39) OR AR SR1 SR2 OR AR SR1 SR Note Data are numbers of cases. OR = on-call radiologist, AR = in-house abdominal radiologist, SR1 = smartphone reader 1, SR2 = smartphone reader 2. TABLE 7: Results of Suggested Alternative Diagnoses Alternative Diagnosis Established Diagnosis (n = 12) OR (n = 12) AR (n = 15) Right colonic diverticulitis 5 5 (0) 5 (0) 4 (0) 5 (1) Pelvic inflammatory disease 1 0 (0) 0 (0) 0 (1) 0 (2) Complicated adnexal cyst 3 2 (0) 2 (3) 2 (0) 2 (0) Mesenteric lymphadenopathy 0 0 (2) 0 (2) 0 (0) 0 (0) Crohn disease 1 1 (1) 1 (1) 1 (0) 1 (0) Epiploic appendagitis 1 1 (0) 1 (0) 1 (0) 0 (0) Acute pyelonephritis 1 0 (0) 0 (0) 0 (0) 0 (0) SR1 (n = 9) SR2 (n = 11) Note Data are numbers of cases for which the suggested diagnosis was consistent with the established diagnosis, with numbers of cases for which the suggested diagnosis was not consistent with the established diagnosis shown in parenthesis. The inconsistent cases may not represent overdiagnosis, because the reference standards were not complete [13]. OR = on-call radiologist, AR = in-house abdominal radiologist, SR1 = smartphone reader 1, SR2 = smartphone reader 2. Discussion We simulated a mobile consultation for 68 patients with inconclusive diagnoses of appendicitis based on the on-call radiologists prospective preliminary CT reports. The smartphone readers readings were more similar to the in-house abdominal radiologists final reports than to the on-call radiologists preliminary reports and more confident than the preliminary reports in diagnosing and ruling out appendicitis, although the diagnostic performance in terms of AUC did not differ significantly from that of the preliminary reports. Even though the increased diagnostic confidence that we observed would undoubtedly facilitate the clinical decision on surgical disposition, our study design was unable to prove that the increased diagnostic confidence can result in better clinical outcome. Compared with previous studies [10 12, 24] reporting the technical feasibility of mobile consultation systems, our study has several distinctive features in reproducing the AJR:203, July

6 Seong et al. clinical practice. First, we carefully chose the study sample to represent a population having a potential need for a mobile consultation in the medical context. The 68 patients forming our study sample accounted for 15% of the 447 patients assigned to the standarddose CT group in the original trial [13]. This percentage is similar to the percentage of inconclusive (i.e., equivocal or probable) CT reports (12%) in a recent large single-center cross-sectional study [13], although many factors regarding patients (e.g., different disease severity, individual variation in the appendix size) and radiologists (e.g., level of experience, different diagnostic criteria) would have affected these percentages. Second, we evaluated the diagnostic performance and confidence, in addition to the technical feasibility, of the smartphone readings in comparison with the in-house CT reports. We were unable to draw a firm conclusion regarding whether a mobile consultation can replace in-house abdominal radiologists reports in the process of clinical decision-making for the following reasons. First, although the diagnostic performance of the AUCs did not significantly differ with our limited sample size between the smartphone readings and the abdominal radiologists final reports, the 95% CIs for the AUC differences were centered toward claiming the superiority of the abdominal radiologists final reports to the smartphone readings. The results might have been affected not only by the display configuration but also by many other factors, the individual effects of which we could not analyze. These factors would include the availability of clinical and laboratory findings including their interval changes during the clinical observation, the availability of thin-section images along with the multiplanar sliding slab averaging technique, the expertise of individual radiologists, the use of irreversible image compression, and different reading environments including light condition. Second, in our retrospective simulation, we were unable to analyze the effect of the smartphone readings on patient outcome. Therefore, it would be prudent to conclude from our study results that a mobile consultation can be used as an adjunct to on-call radiologists preliminary reports and that in-house abdominal radiologists final reports are still needed. Our study has some limitations. First, the usefulness of smartphone reading would have been further validated if less-experienced radiologists performed another inhouse retrospective image review as a negative control. However, we abandoned such a study design because choosing representative average less-experienced radiologists would be inevitably subjective and arbitrary, and the study results would be complicated by their learning curves. Second, despite the use of thorough reference standards described elsewhere [13], the reported diagnostic performance may have been inflated because the pathologic verification of appendicitis was made selectively in cases with positive results in prospective CT reports [25]. Third, the two off-site radiologists performed the smartphone reading sequentially on 68 patients during the daytime; however, a mobile consultation in practice would be needed for one patient at a time, typically after hours. Fourth, our study involved a single mobile consultation system, a single hospital, a standard-dose CT protocol, and only two off-site radiologists, which may limit the generalizability of the study. In conclusion, with the given study sample, the diagnostic performance of the offsite smartphone readings did not differ significantly from that of the in-house preliminary reports. However, the smartphone readings provided higher diagnostic confidence than the preliminary reports. Our results suggest that a mobile consultation is potentially a useful adjunct to an on-call radiologist s preliminary report when the oncall radiologist s diagnosis of appendicitis is not conclusive. References 1. Rao PM, Rhea JT, Novelline RA, Mostafavi AA, McCabe CJ. Effect of computed tomography of the appendix on treatment of patients and use of hospital resources. N Engl J Med 1998; 338: Raman SS, Osuagwu FC, Kadell B, Cryer H, Sayre J, Lu DS. Effect of CT on false positive diagnosis of appendicitis and perforation. N Engl J Med 2008; 358: Pickhardt PJ, Lawrence EM, Pooler BD, Bruce RJ. Diagnostic performance of multidetector computed tomography for suspected acute appendicitis. Ann Intern Med 2011; 154: , W Raja AS, Wright C, Sodickson AD, et al. Negative appendectomy rate in the era of CT: an 18-year perspective. Radiology 2010; 256: Accreditation Counsel for Graduate Medical Education (ACGME). ACGME program requirements for graduate medical education in diagnostic radiology, section VI.B.2. acgmeweb/portals/0/pfassets/2013-pr-faq-pif/ 420_diagnostic_radiology_ pdf. Approved Revised 2009 and Accessed April 1, Cooper VF, Goodhartz LA, Nemcek AA Jr, Ryu RK. Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies. Acad Radiol 2008; 15: Ruchman RB, Jaeger J, Wiggins EF III, et al. Preliminary radiology resident interpretations versus final attending radiologist interpretations and the impact on patient care in a community hospital. AJR 2007; 189: Hunter TB, Taljanovic MS, Krupinski E, Ovitt T, Stubbs AY. Academic radiologists on-call and late-evening duties. J Am Coll Radiol 2007; 4: Choudhri AF, Carr TM 3rd, Ho CP, Stone JR, Gay SB, Lambert DL. Handheld device review of abdominal CT for the evaluation of acute appendicitis. J Digit Imaging 2012; 25: Johnson PT, Zimmerman SL, Heath D, et al. The ipad as a mobile device for CT display and interpretation: diagnostic accuracy for identification of pulmonary embolism. Emerg Radiol 2012; 19: Toomey RJ, Ryan JT, McEntee MF, et al. Diagnostic efficacy of handheld devices for emergency radiologic consultation. AJR 2010; 194: Choudhri AF, Radvany MG. Initial experience with a handheld device digital imaging and communications in medicine viewer: OsiriX mobile on the iphone. J Digit Imaging 2011; 24: Kim K, Kim YH, Kim SY, et al. Low-dose abdominal CT for evaluating suspected appendicitis. N Engl J Med 2012; 366: Kim SY, Lee KH, Kim K, et al. Acute appendicitis in young adults: low- versus standard-radiation-dose contrast-enhanced abdominal CT for diagnosis. Radiology 2011; 260: Joo SM, Lee KH, Kim YH, et al. Detection of the normal appendix with low-dose unenhanced CT: use of the sliding slab averaging technique. Radiology 2009; 251: Huda W, Mettler FA. Volume CT dose index and dose-length product displayed during CT: what good are they? Radiology 2011; 258: Lee KH, Lee HJ, Kim JH, et al. Managing the CT data explosion: initial experiences of archiving volumetric datasets in a mini-pacs. J Digit Imaging 2005; 18: Kim B, Lee KH, Kim KJ, Mantiuk R, Kim HR, Kim YH. Artifacts in slab average-intensity-projection images reformatted from JPEG 2000 compressed thin-section abdominal CT data sets. AJR 2008; 190[web]:W342 W Lee KH, Hong H, Hahn S, Kim B, Kim KJ, Kim YH. Summation or axial slab average intensity projection of abdominal thin-section CT datasets: 8 AJR:203, July 2014

7 Off-Site Smartphone Reading of CT for Appendicitis can they substitute for the primary reconstruction from raw projection data? J Digit Imaging 2008; 21: Seo H, Lee KH, Kim HJ, et al. Diagnosis of acute appendicitis with sliding slab ray-sum interpretation of low-dose unenhanced CT and standard-dose IV contrast-enhanced CT scans. AJR 2009; 193: Lee KH, Kim YH, Hahn S, et al. Computed tomography diagnosis of acute appendicitis: advantages of reviewing thin-section datasets using sliding slab average intensity projection technique. Invest Radiol 2006; 41: APPENDIX 1: Heat Map With Dendrogram 22. Lee H, Kim B, Kim KJ, et al. Introduction of heat map to fidelity assessment of compressed CT images. Med Phys 2011; 38: Daly CP, Cohan RH, Francis IR, Caoili EM, Ellis JH, Nan B. Incidence of acute appendicitis in patients with equivocal CT findings. AJR 2005; 184: Mitchell JR, Sharma P, Modi J, et al. A smartphone client-server teleradiology system for primary diagnosis of acute stroke. J Med Internet Res 2011; 13:e Terasawa T, Blackmore CC, Bent S, Kohlwes RJ. Systematic review: computed tomography and ultrasonography to detect acute appendicitis in adults and adolescents. Ann Intern Med 2004; 141: Jiang D, Tang C, Zhang A. Cluster analysis for gene expression data: a survey. IEEE Trans Knowl Data Eng 2004; 16: Cormack RM. A review of classification. J R Stat Soc [Ser A] 1971; 134: Freedman MT, Osicka T. Heat maps: an aid for data analysis and understanding of ROC CAD experiments. Acad Radiol 2008; 15: Here we briefly describe how we drew the heat maps with dendrograms. More details on using heat maps with dendrograms for presenting Likert-scale data are available elsewhere [22]. We had 116 (i.e., four readers [or reader groups] times 29 patients) and 156 (i.e., four readers [or reader groups] times 39 patients) data points in appendicitis and nonappendicitis groups, respectively (Table 6). In each heat map, we arranged the readers (or reader groups) from left to right in the order of the on-call radiologists, attending abdominal radiologists, smartphone reader 1, and smartphone reader 2; and the patients from bottom to top in increasing order of the mean confidence grade across the readers (or reader groups). At each intersection in the heat maps, the color of the grid element indicated the grade assigned by the corresponding reader (or reader group) for the corresponding patient. The variation among the readers (or reader groups) in the grades they assigned to a given patient and the variation among the patients in each reader s (or reader group s) grades were visualized as color variations across the rows and across the columns, respectively. The dendrogram attached to the top of each heat map represented the similarity in the response pattern among the readers (or reader groups). In grouping the readers (or reader groups), hierarchic agglomerative clustering algorithm [26, 27] was used as follows. In each heat map, each reader s (or reader group s) responses (each column) were considered as a single cluster. A similarity matrix was computed between all pairs of four clusters in terms of kappa coefficient. Of the four clusters, two with the highest similarity were merged into a new cluster, and the formation of the new cluster was represented as a fusion of two branches in the dendrogram. The height of the fused branch indicated the degree of the similarity (i.e., 1 κ) between the two clusters being merged; a shorter branch indicated higher similarity. The similarity matrix was then updated by computing the similarity of the new cluster to all remaining clusters by using the average linkage method [22, 26 28]. Because this process was repeated until only a single cluster remained, the dendrogram illustrated the hierarchic structure of nested clusters. APPENDIX 2: Suggestion of Alternative Diagnoses In 12 of 39 patients who finally received diagnosis of not having appendicitis, an alternative diagnosis other than nonspecific abdominal pain or gastroenterocolitis was established on the basis of reference standards described elsewhere [13]. On-call radiologists, in-house abdominal radiologists, smartphone reader 1, and smartphone reader 2 suggested 12, 15, 9, and 11 alternative diagnoses, respectively (Table 7). AJR:203, July