Acute Appendicitis on Abdominal MR Images: Training Readers to Improve Diagnostic Accuracy 1

Transcription

1 Note: This copy is for your personal non-commercial use only. To order presentation-ready copies for distribution to your colleagues or clients, contact us at Marjolein M. N. Leeuwenburgh, MD Bart M. Wiarda, MD Shandra Bipat, PhD C. Yung Nio, MD Thomas L. Bollen, MD J. Joost Kardux, MD Sebastiaan Jensch, MD, PhD Patrick M. M. Bossuyt, PhD Marja A. Boermeester, MD, PhD Jaap Stoker, MD, PhD Acute Appendicitis on Abdominal MR Images: Training Readers to Improve Diagnostic Accuracy 1 Purpose: Materials and Methods: To determine if training with direct feedback helps to improve the diagnostic performance of inexperienced readers in the detection of appendicitis on magnetic resonance (MR) images. The institutional review board approved this retrospective study and waived the requirement for informed consent. Nine radiologists and eight residents without experience in evaluating MR images for acute abdominal conditions evaluated a training set of images from 100 MR imaging examinations of patients suspected of having appendicitis and received direct feedback after each evaluation. An expert panel made a diagnosis of appendicitis in 45 patients and an alternative diagnosis in 55 patients on the basis of histopathologic examination and follow-up. Readers recorded two diagnoses: the first after viewing images from conventional MR sequences (half-fourier rapid acquisition with relaxation enhancement and true fast imaging with steady-state precession) and the second after viewing diffusion-weighted (DW) MR images. Reader sensitivity and specificity were calculated per set of 25 cases. Original Research n Gastrointestinal Imaging 1 From the Departments of Radiology (M.M.N.L., S.B., C.Y.N., J.S.), Surgery (M.M.N.L., M.A.B.), and Clinical Epidemiology, Biostatistics and Bioinformatics (P.M.M.B.), Academic Medical Center, University of Amsterdam, Meibergdreef 9, G1-229, 1105 AZ Amsterdam, the Netherlands; Department of Radiology, Alkmaar Medical Center, Alkmaar, the Netherlands (B.M.W.); Department of Radiology, Sint Antonius Hospital, Nieuwegein, the Netherlands (T.L.B.); Department of Radiology, Gelre Hospitals, Apeldoorn, the Netherlands (J.J.K.); and Department of Radiology, Sint Lucas Andreas Hospital, Amsterdam, the Netherlands (S.J.). Received September 6, 2011; revision requested September 11; revision received December 12; accepted February 3, 2012; final version accepted February 21. Supported by the Dutch Organization for Health Research and Development, Health Care Efficiency Research Program (ZonMw, grant number ) Address correspondence to M.M.N.L. ( m.m.leeuwenburgh@amc.uva.nl). q RSNA, 2012 Results: Conclusion: The average reader sensitivity for detecting appendicitis improved significantly after training (0.82 vs 0.92, P =.003); the average specificity improved nonsignificantly (0.82 vs 0.88, P =.10). Sensitivity for radiologists increased from 0.81 in the first set of 25 cases to 0.91 in the last set, and specificity improved from 0.82 to For residents, sensitivity increased from 0.82 to 0.94, and specificity increased from 0.82 to Sensitivity improved from 0.80 to 0.87 (P,.001) in all readings combined when DW images were read in addition to conventional MR images. Diagnostic accuracy of inexperienced readers in the evaluation of abdominal MR images for acute appendicitis improved after training with direct feedback, and the addition of DW images improved reader sensitivity. q RSNA, 2012 Supplemental material: /suppl/doi: /radiol /-/dc1 Radiology: Volume 264: Number 2 August 2012 n radiology.rsna.org 455

2 Use of magnetic resonance (MR) imaging in emergency departments is likely to increase in the near future. The versatility of the technique, increasingly shorter acquisition times of MR imaging, and the increasing awareness of radiation-related health risks associated with computed tomography (CT) (1) have resulted in more frequent use of MR imaging in the diagnostic workup for several acute abdominal conditions (2), including the most common cause of acute abdominal pain: acute appendicitis (3,4). The appropriateness criteria of the American College of Radiology include a recommendation for the use of MR imaging in pregnant women suspected of having appendicitis when ultrasonographic (US) examination results are negative or inconclusive (5). As MR imaging becomes increasingly available in emergency settings (6), its use for evaluating all patients with acute right lower quadrant pain will be further explored. Diffusion-weighted (DW) MR imaging, a relatively new technique in abdominal imaging, has been shown to help increase the detection of neoplastic and inflammatory lesions (7). Authors Advances in Knowledge Average reader sensitivity in evaluation of abdominal MR images for appendicitis improved from 0.82 to 0.92 (P =.003) and specificity increased from 0.82 to 0.88 (P =.10) from the first 25 to the last 25 cases of a training set of 100 MR imaging examinations. Addition of diffusion-weighted images to the evaluation significantly increased reader sensitivity for detecting acute appendicitis from 0.80 to 0.87 (P,.001). Radiologists and residents performed equally well in the evaluation of abdominal MR images for acute appendicitis. Reading time of radiologists and residents decreased significantly after training (both P,.001). of a recent study (8) reported that the use of DW imaging was beneficial in the identification of acute appendicitis, because the inflamed appendix is conspicuous on DW images. The inflammatory process of acute appendicitis alters the structural organization of the appendix and adjacent tissue, thereby affecting the diffusivity of water molecules, which results in increased signal intensity on DW images (9). Therefore, the addition of DW imaging to current MR imaging protocols may further improve reader sensitivity. Most radiologists have limited or no experience in the evaluation of MR and DW MR images for acute abdominal conditions and most likely will need training to achieve the diagnostic accuracy that has been reported in the literature; however, the extent of training needed is not yet clear. The purpose of this study was to estimate the effect of training with direct feedback on the diagnostic performance of inexperienced readers in the detection of acute appendicitis on abdominal MR images. Materials and Methods The institutional review board approved this retrospective study and waived the requirement of informed consent for the use of patient records and images. All readers gave written consent for the use of their radiologic evaluations for research purposes. Seventeen inexperienced readers evaluated images from 100 abdominal MR imaging examinations of patients who were suspected of having appendicitis. All readers received direct feedback after each evaluation. Implication for Patient Care Radiologists and residents with no experience in evaluating MR images of patients suspected of having appendicitis can be trained to reach a diagnostic accuracy level comparable to that of readers of computed tomographic abdominal images for appendicitis, as described in literature. Patient Selection The training cases were extracted from an existing database and included 100 consecutive MR imaging examinations of patients who were examined in the emergency department of the Alkmaar Medical Center, the Netherlands, between December 2007 and January 2010 and were suspected of having appendicitis. The MR imaging examinations were performed (instead of CT) after inconclusive or negative US results. The mean age of the patients was 29 years (range, years) and the group consisted of 68 females (mean age, 28 years; range, years), of whom three were pregnant (with gestational ages of 15, 30, and 31 weeks), and 32 males (mean age, 31 years; range, years). All patients were imaged with a 1.5T MR imaging unit (Magnetom Avanto; Siemens, Erlangen, Germany) in supine position without oral or intravenous contrast material enhancement. The MR imaging protocol included the following sequences: coronal and transverse breath-hold half-fourier rapid acquisition relaxation enhancement (RARE), coronal and transverse breath-hold balanced true fast imaging with steady-state precession (FISP) with fat saturation, and coronal and transverse free-breathing Published online before print /radiol Content codes: Radiology 2012; 264: Abbreviations: CI = confidence interval DW = diffusion weighted FISP = fast imaging with steady-state precession RARE = rapid acquisition and relaxation enhancement Author contributions: Guarantors of integrity of entire study, M.M.N.L., P.M.M.B., M.A.B., J.S.; study concepts/study design or data acquisition or data analysis/interpretation, M.M.N.L., B.M.W., P.M.M.B., M.A.B., J.S.; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, M.M.N.L.; clinical studies, B.M.W., C.Y.N., T.L.B., J.J.K., S.J., experimental studies, B.M.W., C.Y.N., T.L.B., J.J.K., S.J.; statistical analysis, M.M.N.L., S.B., P.M.M.B.; and manuscript editing, all authors. Potential conflicts of interest are listed at the end of this article. 456 radiology.rsna.org n Radiology: Volume 264: Number 2 August 2012

3 Table 1 MR Imaging Parameters Half-Fourier RARE True FISP DW Imaging* Parameters Coronal Transverse Coronal Transverse Coronal Transverse Repetition time (msec) Echo time (msec) Section thickness (mm) Spacing between sections (mm) Flip angle (degree) Matrix Field of view (mm) * b Values, 50, 400, 800 sec/mm 2 Table 2 Final Diagnoses Assigned by Expert Panel Diagnosis Acute appendicitis 45 Nonspecific abdominal pain 21 Gynecologic disorder: ovarian 12 torsion (n = 2), pelvic inflammatory disease (n = 4), bleeding/ruptured ovarian cyst (n = 3), uterine myoma (n = 1), tubo-ovarian abscess (n = 2) Small bowel or colon inflammation 8 Urinary tract disorder: urinary tract 5 infection (n = 4), pyelonephritis (n = 1) Acute diverticulitis 3 Other: acalculous cholecystitis 6 (n = 1), bowel obstruction due to adhesion (n = 1), epiploic appendagitis (n = 1), herniation (n = 1), pneumonia (n = 1), spondylodiscitis (n = 1) No. of Patients DW imaging. The imaging parameters are listed in Table 1. An experienced radiologist (B.M.W., who had evaluated images from more than 500 abdominal MR imaging examinations) initially read the images, which had been used for patient care. All data that could reveal patients identity were concealed before the start of this study. An expert panel consisting of two surgeons (with 16 and 17 years of experience) and one radiologist (B.M.W., Figure 1 Figure 1: Flowchart of training program for inexperienced readers. HASTE = half- Fourier RARE, DWI = DW imaging. with 15 years of experience) made the final diagnosis on the basis of clinical features, imaging findings, surgery, pathologic examination, and details of follow-up of the patients for at least 1 month. Acute appendicitis was the final diagnosis in 45 of the 100 patients; the other diagnoses are listed in Table 2. None of the pregnant women had acute appendicitis. The 45% prevalence of acute appendicitis in the MR imaging training cases was comparable to that seen in emergency departments among patients who are suspected of having appendicitis, as reported in three metaanalyses (10 12). The final diagnosis served as the reference standard in the training and in the evaluation of diagnostic accuracy. MR Imaging Training Program Radiologists and residents from two university hospitals and five teaching hospitals who had no experience evaluating MR images for acute abdominal conditions were invited to participate in this study. Nine radiologists and eight residents accepted the invitation and completed the training program between March 2010 and January The radiologists had an average of 15 years (range, 7 25 years) of experience in radiology, and all had evaluated images from more than 1000 abdominal CT examinations for acute conditions. All but one radiologist had evaluated images from more than 1000 MR imaging examinations in general, but none of the radiologists had experience with MR image evaluation for acute abdominal conditions. The average experience of residents was 3 years (range, 2 5 years), with fewer than 1000 abdominal CT evaluations and fewer than 1000 MR imaging evaluations (none for acute abdominal conditions). The MR imaging training program is summarized in Figure 1. Readers received an introduction to conventional MR and DW MR imaging without the use of oral or intravenous contrast media for evaluation of acute appendicitis. A slide show consisting of 53 slides with a short explanation of both conventional MR and MR DW imaging was Radiology: Volume 264: Number 2 August 2012 n radiology.rsna.org 457

4 presented. Images from 17 patients who were suspected of having appendicitis were presented, with written instructions on viewing techniques. Readers were asked to examine the consecutive series of images from 100 MR imaging examinations on a picture archiving and communication system workstation and to record their findings on an online case record form. Clinical information, including patient history and physical examination and laboratory findings, was available for all patients. First, readers examined images from the conventional sequences (half- Fourier RARE and true FISP with fat saturation) and made a positive, negative, or inconclusive diagnosis for acute appendicitis. When a negative or inconclusive diagnosis was chosen, readers could select an alternative diagnosis. After this, readers examined the additional DW images and assigned a diagnosis again. Readers were asked to record whether the appendiceal area had an area of hyperintense signal on DW images and whether the additional DW images helped them to arrive at their diagnosis. Readers were asked to assign a level of confidence on a scale of 1 to 10 for positive and negative diagnoses. After they evaluated each case, readers could verify their diagnosis by checking the reference standard, which was printed in a syllabus and was present on the workstation. The reference standard for each case contained the reference diagnosis and details of the clinical MR image with series and section numbers for illustration of expert findings. Reading time per case was measured in minutes and seconds with a digital clock and was recorded by the participants. Statistical Analysis We calculated the average sensitivity and specificity per set of 25 MR imaging examinations by comparing the readers diagnoses after viewing conventional MR and DW images with the final diagnosis assigned by the expert panel. Inconclusive results were treated as negative in these calculations. Readers evaluated images from the 100 MR imaging examinations independently; therefore, we regarded these measurements as independent repetitions. To correct for repeated measurements among readers we used a generalized estimating equation method in our statistical software (SPSS version 18.0; SPSS; Chicago, Ill). The link function was set at logit, and an independent working correction matrix was used. Average sensitivity and specificity values with corresponding confidence intervals (CIs) were obtained by means of antilogit transformation of the estimated logit sensitivity, logit specificity, and the standard errors of the generalized estimating equation models. The average reader sensitivity and specificity for the first set of 25 cases were compared with those for the last set of 25 cases and were compared separately for radiologists and residents. Statistical differences in diagnostic accuracy before and after training with direct feedback were tested by using the z test with the logit parameters. Accuracy of diagnoses made by participants after they read the conventional MR images were compared with those made after they read the DW images for all 100 cases. These data were calculated for all participants and for radiologists and residents separately. Differences in diagnostic accuracy before and after review of DW images were tested for statistical significance by using the paired z test on the logit parameters. In addition, we calculated the percentage of inconclusive diagnoses before and after participants evaluated additional DW images. The percentage of inconclusive diagnoses was analyzed and corrected for repeated measurements by using methods similar to those used to determine sensitivity and specificity. The generalized k coefficient (Fleiss k) was used to calculate overall agreement among readers (13). We interpreted k values according to methods used by Landis and Koch (14) (k < 0.20, poor agreement; k = , fair; k = , moderate; k = , good; and k = , excellent). We evaluated the percentage of agreement by comparing the positive and negative findings of the majority of radiologists (four or more readers) and the majority of residents (four or more readers).differences in the level of confidence participants assigned on the basis of conventional MR images and DW images were tested for statistical significance by using the unpaired t test. Mean reading times were calculated per case for both groups and then averaged per set of 25 consecutive cases. Differences in means were tested for statistical significance by using the t test. Reading times were calculated separately for radiologists and residents. P values of less than.05 were considered to indicate statistically significant differences. Results Effect of Training with Direct Feedback on Diagnostic Performance Accuracy in diagnosis of acute appendicitis increased for all participants of the training program. The average reader sensitivity for acute appendicitis increased significantly, from 0.82 (95% CI: 0.77, 0.86) in the first set of 25 cases to 0.92 (95% CI: 0.87, 0.95; P =.003) in the last set of 25 cases. The average specificity increased nonsignificantly from 0.82 (95% CI: 0.75, 0.87) to 0.88 (95% CI: 0.83, 0.92; P =.10). Figure 2 shows the average sensitivity and specificity with corresponding 95% CIs per set of 25 cases for radiologists and residents separately. For radiologists, average reader sensitivity increased from 0.81 (95% CI: 0.77, 0.85) in the first set of 25 cases to 0.91 (95% CI: 0.82, 0.96; P =.06) in the last set of 25 cases. Specificity was 0.82 (95% CI: 0.72, 0.89) and 0.85 (95% CI: 0.78, 0.90; P =.54), respectively. For residents, the average sensitivity improved significantly after training, from 0.82 (95% CI: 0.72, 0.90) to 0.94 (95% CI: 0.89, 0.97; P =.01). Reader specificity improved from 0.82 (95% CI: 0.70, 0.89) to 0.91 (95% CI: 0.85, 0.95; P =.07) after the last set. None of the differences between radiologists and residents were significant. All absolute data, including numerators and denominators, are shown in Table E1 (online). 458 radiology.rsna.org n Radiology: Volume 264: Number 2 August 2012

5 Figure 2 Figure 2: Graphs show average (a, c) sensitivity and (b, d) specificity of radiologists and residents with corresponding 95% CIs for reading conventional MR images with and without additional DW images (DWI) per sets of 25 cases. HASTE = half-fourier RARE. Effect of Additional DW Imaging on Diagnostic Performance Sensitivity improved significantly when DW images were added to conventional images, from 0.80 to 0.87 (P,.001). Specificity decreased from 0.88 to 0.84 (P =.004, Figs 3 5). For radiologists, the average sensitivity improved from 0.79 (95% CI: 0.74, 0.82) to 0.85 (95% CI: 0.82, 0.87; P,.001) after evaluation of additional DW images. The average sensitivity in residents was 0.82 (95% CI: 0.77, 0.86) before and 0.89 (95% CI: 0.86, 0.92; P,.001) after evaluation of DW images. Average specificity decreased significantly for radiologists, from 0.89 (95% CI: 0.83, 0.93) to 0.84 (95% CI: 0.78, 0.88; P =.006), and nonsignificantly for residents, from 0.87 (95% CI: 0.82, 0.91) to 0.85 (95% CI: 0.79, 0.90; P =.21). The percentage of inconclusive findings decreased from 26% (434 of 1680; 95% CI: 21%, 31%) to 12% (195 of 1680; 95% CI: 8%, 16%; P,.001) after participants read DW images; 51% (220 of 434) of instances of inconclusive results were reassigned to the correct diagnosis after evaluation of DW images. For radiologists, the percentage of inconclusive findings decreased from 28% (247 of 892; 95% CI: 22%, 35%) to 11% (96 of 892; 95% CI: 6%, 18%; P,.001) after evaluation of DW images; for residents, inconclusive findings decreased from 24% (187 of 788; 95% CI: 18%, 31%) to 13% (99 of 788; 95% CI: 8%, 19%; P,.001). No statistically significant differences were found between radiologists and residents in the percentage of inconclusive findings. In 81% (726 of 800) of cases that were positive for appendicitis, hyperintensity of the appendiceal area was recorded. Readers regarded the additional DW images as helpful in 53% (887 of 1680) of cases. The level of confidence increased after evaluation of DW images from examinations with a positive diagnosis, from 8.28 (interquartile range [IQR], 7 9) to 8.50 (IQR, 8 10) (P =.001). Readers confidence in a negative diagnosis increased from 7.74 (IQR, 7 8) to 8.04 (IQR 7 9) (P,.001). Radiology: Volume 264: Number 2 August 2012 n radiology.rsna.org 459

6 Agreement between Radiologists and Residents The generalized k coefficient among all readers for the presence of acute appendicitis was 0.55 (95% CI: 0.54, 0.57) after evaluation of the conventional images and 0.58 (95% CI: 0.56, 0.60) after evaluation of additional DW images. Agreement among radiologists after reading DW images was 0.57 (95% CI: 0.54, 0.60), and among residents it was 0.60 (95% CI: 0.57, 0.64); this is considered moderate agreement. Agreement increased as the readers evaluated more MR images; the generalized k coefficient for all readers per set of 25 MR examinations was 0.44 (95% CI: 0.40, 0.47), 0.63 (95% CI: 0.59, 0.66), 0.62 (95% CI: 0.58, 0.65) and 0.64 (95% CI: 0.60, 0.67). For 33 of 100 MR imaging examinations, all 17 readers agreed on the presence or absence of acute appendicitis. For 96 of 100 examinations (96%), the majority of the radiologists agreed with the majority of residents were in agreement. Most residents read the other four cases as positive for appendicitis, and most radiologists did not. Most radiologists and residents correctly identified appendicitis in 38 of the 45 patients for whom appendicitis was the final diagnosis. For three other patients with appendicitis, most of the residents and a few of the radiologists identified appendicitis; for four patients, a few radiologists and residents identified appendicitis correctly. Reading Time The mean reading time for radiologists decreased significantly from 5 minutes 8 seconds per image in the first set to 3 minutes 28 seconds in the last set of 25 cases (P,.001). For residents, reading times decreased from 7 minutes 50 seconds to 5 minutes 23 seconds per case (P,.001, Fig 6). Discussion The results of this study showed that training with direct feedback helped to improve the diagnostic performance of inexperienced readers in the detection Figure 3 Figure 3: MR images in 16-year-old boy suspected of having appendicitis whose white blood cell count was /L and C-reactive protein level was 2 mg/l (19.05 nmol/l). (a) Axial half-fourier RARE image shows retrocecal localized thickened appendix (arrow) with an appendicolith in the lumen and some fluid around the appendix. (b) Axial DW image (b = 800 sec/mm 2 ) shows restricted diffusion of appendiceal wall (arrow). Appendicitis was confirmed by the expert panel. All 17 readers confirmed diagnosis of acute appendicitis. Figure 4 Figure 4: MR images in 42-year-old man suspected of having appendicitis whose white blood cell count was /L. (a) Axial true FISP fat-saturated image shows thickened (8-mm) appendix on right side (arrow) without infiltration or fluid around appendix. (b) Axial DW image (b = 800 sec/mm 2 ) shows restricted diffusion of appendiceal wall (arrow). Seven of 17 readers changed diagnosis to acute appendicitis after reading DW images. of acute appendicitis with abdominal MR imaging. This is evidenced by the increase in average sensitivity and specificity for radiologists and residents from the first to the last set of 25 cases. The results also showed that the average reader sensitivity for detection of acute appendicitis was significantly higher after evaluation of the additional DW images than that after evaluation of the conventional images alone. In addition, the interobserver agreement increased during the training process from 0.44 in the first set to 0.64 in the last set of 25 cases, while the reading time decreased. 460 radiology.rsna.org n Radiology: Volume 264: Number 2 August 2012

7 Figure 5 Figure 5: MR images in 37-year-old woman suspected of having appendicitis whose white blood cell count was /L. (a) Coronal true FISP fat-saturated image shows subtle inhomogeneous kidney parenchyma of lower pole of right kidney (arrow). (b) DW image (b = 800 sec/mm 2 ) shows restricted diffusion in same part of right kidney (arrow), consistent with pyelonephritis. Urine culture results affirmed diagnosis; appendix (not visualized here) considered normal. Accurate diagnosis was chosen by 11 of 17 readers. Figure 6 To our knowledge, there is not much literature on the topic of practical training, except in manual skills such as interventional endoscopy. A beneficial effect of reader training for evaluation of CT colonographic images has been described previously (15 17); one of these studies even evaluated the number of required training cases (16). To our knowledge, the effects of the learning curve in the evaluation of acute abdominal symptoms with MR imaging have not been studied. Because our study was limited to 100 cases, we do not know whether participants could further improve sensitivity and specificity if provided with more training cases. The maximal sensitivity and specificity Figure 6: Mean reading times for radiologists and residents to read images from all sequences (half-fourier RARE, true FISP, and DW imaging) with corresponding standard deviations per sets of 25 cases. in our study were 0.91 and 0.85 for radiologists, and 0.94 and 0.91 for residents, respectively. These results are good, and they are comparable to those of readers of CT examinations for acute appendicitis (10). However, they do not reach the high accuracy level reported in a recent meta-analysis of MR imaging for acute appendicitis (18). The studies that were included in the meta-analysis were mainly performed in small groups of children and pregnant patients who had a low prevalence of appendicitis (19 25). These findings may not reflect the accuracy of MR imaging diagnosis of appendicitis in a general adult population. In our study, participants viewed MR images of patients from the general population who were suspected of having appendicitis and who had negative or inconclusive US results. This selection could have influenced the complexity of the case mix, and therefore, the accuracy of results. The trained radiologists and residents may have performed better in an unselected set of patients who were suspected of having appendicitis. The increase in sensitivity after evaluation of the additional DW images is consistent with findings of a previous study. Inci et al (8) reported that the appendix in 99% of patients with appendicitis appeared hyperintense on DW images and that this helps in the identification of the inflamed appendix. In our study, reading additional DW images helped readers to diagnose cases that they previously considered inconclusive and made readers more certain about their positive and negative diagnoses, although the difference was small. These findings suggest that DW imaging is a valuable technique for detecting acute appendicitis. Because of the anticipated increased use of MR imaging in the emergency department (6), many radiologists may need to start evaluating abdominal MR images for conditions such as acute appendicitis. Our results suggest that novice MR imaging readers can improve their performance with training. Both radiologists and residents must have or acquire the ability to scrutinize MR images in patients with acute abdominal conditions. During out-of-office hours, a radiology resident may be the only person available to evaluate an MR imaging examination. Our study results did not show significant differences in accuracy between radiologists and residents after training. This is concordant with data in other articles showing that the error rate of resident readings of abdominal CT images was low (26 28). Even though radiologists started with more experience in the evaluation of the acute abdomen with CT images, residents performed equally well and seemed to have a steeper learning curve. This might be the result of the teaching environment of residents or a reduced ability of radiologists to process Radiology: Volume 264: Number 2 August 2012 n radiology.rsna.org 461

8 6. Rankey D, Leach JL, Leach SD. Emergency MRI utilization trends at a tertiary care acfeedback. Overall, radiologists and residents reached reasonable agreement on the identification of acute appendicitis with abdominal MR imaging. A number of potential limitations of our study must be considered. Because readers had the option to diagnose cases as inconclusive, they could avoid making the decision on the presence or absence of appendicitis. We chose to consider inconclusive diagnoses as negative for appendicitis for the calculations of sensitivity and specificity. Readers may have been more likely to assign an inconclusive finding to their first diagnosis because they knew that they would have a second opportunity to decide on each case after reading DW images. This may have contributed to the observed increase in sensitivity after evaluation of DW images. In our opinion, DW imaging is used most effectively in addition to conventional sequences; therefore, we used a study design that could evaluate the incremental value of DW imaging with conventional sequences. Another potential limitation was that all readers read the cases in similar order. The first set of 25 cases could have been harder to interpret than the last set of cases, which may have created an artificial increase in diagnostic performance. In our view, this is not very likely because we simply used all available cases in consecutive order. In this way, the proportion of patients with acute appendicitis did not differ among the four sets of 25 cases. In summary, we conclude that training in evaluation of abdominal MR images for acute appendicitis with direct feedback helps to improve the diagnostic accuracy of inexperienced readers. Also, the addition of DW images to conventional MR images improved reader sensitivity. Both radiologists and residents benefited from the training program and were able to reach a sufficient level of accuracy. Similar training programs can be provided to all inexperienced readers before they evaluate MR images of patients suspected of having acute appendicitis. Acknowledgments: The authors thank the participants in the training (Janneke E. van den Bergh, MD; Diederik W. de Boo, MD, PhD; Fleur de Korte, MD; C. Yung Nio, MD; Saffire S. K. S. Phoa, MD, PhD; Adrienne van Randen, MD, PhD [Academic Medical Center]; Carine O. Martins, MD [Alkmaar Medical Center]; Thomas L. Bollen, MD; H. Wouter van Es, MD, PhD; Hein P. Stallmann, MD, PhD; Michiel H. van Werkum, MD [Sint Antonius Hospital Nieuwegein]; JanWillem C. Gratama, MD, PhD; Joost Kardux, MD [Gelre Hospital Apeldoorn]; Sebastiaan Jensch, MD, PhD; Martina N. Weimann, MD [Sint Lucas Andreas Hospital, Amsterdam]; Aart Spilt, MD, PhD [Kennemer Gasthuis Haarlem]; Carsten Arnoldussen, MD [Maastricht University Medical Center]); Mai E. Thieme, MD (Alkmaar Medical Center) for collecting data; the expert panel (Alexander P. J. Houdijk, MD, PhD; Hermien W. Schreurs, MD, PhD [Alkmaar Medical Center]); and the Departments of Radiology in the participating hospitals. Disclosures of Potential Conflicts of Interest: M.M.N.L. No potential conflicts of interest to disclose. B.M.W. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: Institution receives financial support for research from Siemens. Other relationships: none to disclose. S.B. No potential conflicts of interest to disclose. C.Y.N. No potential conflicts of interest to disclose. T.L.B. No potential conflicts of interest to disclose. J.J.K. No potential conflicts of interest to disclose. S.J. No potential conflicts of interest to disclose. P.M.M.B. No potential conflicts of interest to disclose. M.A.B. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: research grant pending from Baxter Pharmaceuticals. Other relationships: none to disclose. J.S. No potential conflicts of interest to disclose. References 1. Schenkman L. Radiology. Second thoughts about CT imaging. Science 2011;331 (6020): Tkacz JN, Anderson SA, Soto J. MR imaging in gastrointestinal emergencies. Radio- Graphics 2009;29(6): Cobben LPJ, Groot I, Kingma L, Coerkamp E, Puylaert J, Blickman J. A simple MRI protocol in patients with clinically suspected appendicitis: results in 138 patients and effect on outcome of appendectomy. Eur Radiol 2009;19(5): Chabanova E, Balslev I, Achiam M, et al. Unenhanced MR Imaging in adults with clinically suspected acute appendicitis. Eur J Radiol 2011;79(2): Rosen MP, Ding A, Blake MA, et al. ACR Appropriateness Criteria right lower quadrant pain suspected appendicitis. J Am Coll Radiol 2011;8(11): ademic medical center: baseline data. Acad Radiol 2008;15(4): Coutinho AC Jr, Krishnaraj A, Pires CE, Bittencourt LK, Guimarães AR. Pelvic applications of diffusion magnetic resonance images. Magn Reson Imaging Clin N Am 2011;19(1): Inci E, Kilickesmez O, Hocaoglu E, Aydin S, Bayramoglu S, Cimilli T. Utility of diffusionweighted imaging in the diagnosis of acute appendicitis. Eur Radiol 2011;21(4): Qayyum A. Diffusion-weighted imaging in the abdomen and pelvis: concepts and applications. RadioGraphics 2009;29(6): van Randen A, Bipat S, Zwinderman AH, Ubbink DT, Stoker J, Boermeester MA. Acute appendicitis: meta-analysis of diagnostic performance of CT and graded compression US related to prevalence of disease. Radiology 2008;249(1): Terasawa T, Blackmore CC, Bent S, Kohlwes RJ. Systematic review: computed tomography and ultrasonography to detect acute appendicitis in adults and adolescents. Ann Intern Med 2004;141(7): Doria AS, Moineddin R, Kellenberger CJ, et al. US or CT for diagnosis of appendicitis in children and adults? A Meta-Analysis. Radiology 2006;241(1): Fleiss JL, Nee JC, Landis JR. Large sample variance of kappa in the case of different sets of raters. Psychol Bull 1979;86(5): Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1): European Society of Gastrointestinal and Abdominal Radiology CT Colonography Group Investigators. Effect of directed training on reader performance for CT colonography: multicenter study. Radiology 2007;242(1): Liedenbaum MH, Bipat S, Bossuyt PM, et al. Evaluation of a standardized CT colonography training program for novice readers. Radiology 2011;258(2): Taylor SA, Burling D, Roddie M, et al. Computer-aided detection for CT colonography: incremental benefit of observer training. Br J Radiol 2008;81(963): Barger RL Jr, Nandalur KR. Diagnostic performance of magnetic resonance imaging in the detection of appendicitis in adults: a meta-analysis. Acad Radiol 2010;17(10): radiology.rsna.org n Radiology: Volume 264: Number 2 August 2012

9 19. Oto A, Ernst RD, Shah R, et al. Rightlower-quadrant pain and suspected appendicitis in pregnant women: evaluation with MR imaging initial experience. Radiology 2005;234(2): Cobben LP, Groot I, Haans L, Blickman JG, Puylaert J. MRI for clinically suspected appendicitis during pregnancy. AJR Am J Roentgenol 2004;183(3): Incesu L, Coskun A, Selcuk MB, Akan H, Sozubir S, Bernay F. Acute appendicitis: MR imaging and sonographic correlation. AJR Am J Roentgenol 1997;168(3): Birchard KR, Brown MA, Hyslop WB, Firat Z, Semelka RC. MRI of acute abdominal and pelvic pain in pregnant patients. AJR Am J Roentgenol 2005;184(2): Nitta N, Takahashi M, Furukawa A, Murata K, Mori M, Fukushima M. MR imaging of the normal appendix and acute appendicitis. J Magn Reson Imaging 2005;21(2): Pedrosa I, Levine D, Eyvazzadeh AD, Siewert B, Ngo L, Rofsky NM. MR imaging evaluation of acute appendicitis in pregnancy. Radiology 2006;238(3): Israel GM, Malguria N, McCarthy S, Copel J, Weinreb J. MRI vs. ultrasound for suspected appendicitis during pregnancy. J Magn Reson Imaging 2008;28(2): van Randen A, Laméris W, van Es HW, et al. A comparison of the accuracy of ultrasound and computed tomography in common diagnoses causing acute abdominal pain. Eur Radiol 2011;21(7): Ruchman RB, Jaeger J, Wiggins EF 3rd, et al. Preliminary radiology resident interpretations versus final attending radiologist interpretations and the impact on patient care in a community hospital. AJR Am J Roentgenol 2007;189(3): Carney E, Kempf J, DeCarvalho V, Yudd A, Nosher J. Preliminary interpretations of after-hours CT and sonography by radiology residents versus final interpretations by body imaging radiologists at a level 1 trauma center. AJR Am J Roentgenol 2003;181(2): Radiology: Volume 264: Number 2 August 2012 n radiology.rsna.org 463