Interobserver agreement among endosonographers for the diagnosis of neoplastic versus non-neoplastic pancreatic cystic lesions

Transcription

1 Página 1 de 9 Use of this content is subject to the Terms and Conditions of the MD Consult web site. Gastrointestinal Endoscopy Volume 58 Number 1 July 2003 Copyright 2003 American Society for Gastrointestinal Endoscopy Interobserver agreement among endosonographers for the diagnosis of neoplastic versus non-neoplastic pancreatic cystic lesions Nuzhat A. Ahmad MD Michael L. Kochman MD Colleen Brensinger MS William R. Brugge MD Douglas O. Faigel MD Frank G. Gress MD Michael B. Kimmey MD Nicholas J. Nickl MD Thomas J. Savides MD Michael B. Wallace MD, MPH Maurits J. Wiersema MD Gregory G. Ginsberg MD Received October 2, For revision January 24, Accepted March 5, Current affiliations: Gastroenterology Division, Department of Medicine, Hospital of University of Pennsylvania, Philadelphia, Pennsylvania, Hospital of the University of Pennsylvania, University of Pennsylvania School of Medicine and VAMC, Philadelphia, Pennsylvania, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia Pennsylvania, Massachusetts General Hospital, Boston, Massachusetts, Portland VAMC, Portland, Oregon, Winthrop-University Hospital, Long Island, New York, University of Washington, Seattle, Washington, University Hospital Chandler Medical Center, Lexington, Kentucky, UCSD Medical Center, San Diego, California, MUSC, Charleston, South Carolina, Mayo Clinic, Rochester, Minnesota. Reprint requests: Nuzhat A. Ahmad, MD, Gastroenterology Division, Department of Medicine, Hospital of University of Pennsylvania and Philadelphia VAMC, University of Pennsylvania School of Medicine, 3rd Floor Ravdin Building, 3400 Spruce St., Philadelphia, PA Copyright 2003 by the American Society for Gastrointestinal Endoscopy /2003/$ Philadelphia, Pennsylvania, Boston, Massachusetts, Portland, Oregon, Long Island, New York, Seattle, Washington, Lexington, Kentucky, San Diego, California, Charleston, South Carolina, Rochester, Minnesota Background: The aim of this study was to evaluate the degree of agreement among endosonographers for EUS diagnosis of neoplastic versus non-neoplastic pancreatic cystic

2 Página 2 de 9 lesions and the specific type of cystic lesion. Methods: Videotapes of EUS procedures from 31 consecutive cases of a range of histopathologically proven lesions, including mucinous cystic neoplasm, serous cystadenoma, neuroendocrine tumor, intraductal papillary mucinous tumor, and pseudocyst, were used to make a study videotape, which was reviewed by 8 experienced endosonographers. The reviewers, blinded to clinical and surgical histopathology results, reviewed each case for the presence or absence of the following features: abnormality of pancreatic duct and parenchyma, margins, solid component, debris, and septations. They were asked to identify each lesion as neoplastic or non-neoplastic and to give a specific diagnosis for each lesion. Results: There was fair agreement between endosonographers for diagnosis of neoplastic versus non-neoplastic lesions (κ = 0.24). Agreement for individual types of lesions was moderately good for serous cystadenomas (κ = 0.46) but fair for the remainder. Agreement was moderately good for presence or absence of solid component (κ = 0.43); fair for presence or absence of abnormal pancreatic duct (κ = 0.29), debris (κ = 0.21), and septations (κ = 0.30); and slight for presence or absence of margins (κ = 0.01) and abnormal pancreatic parenchyma (κ = 0.01). Accuracy rates of EUS for the diagnosis of neoplastic versus non-neoplastic lesions ranged from 40% to 93%. Conclusions: There is little more than chance interobserver agreement among experienced endosonographers for diagnosis of neoplastic versus non-neoplastic, specific type, and EUS features of pancreatic cystic lesions. (Gastrointest Endosc 2003;58:59-64.) Pancreatic cystic lesions present a unique diagnostic dilemma. With the widespread use of crosssectional imaging techniques, cystic lesions in the pancreas frequently are found in asymptomatic patients with no history of current or prior pancreatic disease. When a cystic lesion is identified in the pancreas, management is based primarily on the perceived risk of malignancy or malignant potential. Short of operative resection and histopathologic assessment, the discrimination between benign lesions, such as pseudocyst and serous cystadenoma, and lesions with malignant potential, such as mucinous cystadenoma, intraductal papillary mucinous tumor (IPMT), and neuroendocrine tumor with cystic degeneration, is often difficult. EUS, because of superior spatial resolution, is considered valuable for characterization of pancreatic cystic lesions. However, there are few studies that describe EUS features of pancreatic cystic lesions. [1] [2] Most of the nomenclature for describing the appearance of pancreatic cystic lesions has been extrapolated from radiology publications. The degree to which endosonographers agree on the features and characterization of neoplastic versus non-neoplastic pancreatic cystic lesions is unknown. The primary aim of this study was to evaluate the degree of agreement among endosonographers for the diagnosis of neoplastic versus non-neoplastic pancreatic cystic lesions and for differentiation of the specific type of cystic lesions by EUS examination. A secondary aim was to determine the variation in accuracy rates of endosonographers for diagnosis of neoplastic versus non-neoplastic pancreatic cystic lesions by using EUS. Patients and methods Videotapes of EUS procedures in 31 consecutive patients evaluated for pancreatic cystic lesions, confirmed histopathologically, were used to make a study videotape. All EUS procedures were performed with a radial-scanning echoendoscope (GFUM-20; Olympus America Corp., Melville, N.Y.) by using previously described and accepted techniques. [3] All EUS examinations were performed by two experienced endosonographers from the Division of Gastroenterology at the Hospital of the University of Pennsylvania. Videotapes The 31 videotaped procedures were from consecutive cases encountered between 1995 and 2000 of

3 Página 3 de 9 histopathologically proven mucinous cystic neoplasms, serous cystadenomas, neuroendocrine tumors, IPMT, and pseudocysts. Pseudocyst was defined as a non-neoplastic lesion; the remainder, as neoplastic lesions. The average length of the study videotapes was 2 minutes, and each videotape included complete imaging of the cystic lesion and surrounding pancreatic parenchyma. Patient identity and other features, such as arrows and labeling, were masked. No annotations except for caliper measurements for cyst size were visible on the study videotape. Reviewers A copy of the study videotape was sent to 8 experienced, nationally recognized endosonographers, all of whom were in practice at tertiary care referral centers. The endosonographers were identifiable only to the principal investigator and were anonymous for purposes of the study. The reviewers were asked to complete a survey regarding their training, experience, and beliefs concerning EUS for the diagnosis of pancreatic cystic lesions. Training variables included type of EUS training (i.e., selftaught, general fellowship, or third-tier fellowship). Measures of experience included overall annual EUS case volume, EUS of the pancreas, and EUS of pancreatic cystic lesions. The reviewers were asked to respond to the following statement by using a visual analog scale of 1 to 5, (1, strong disagreement; 5, strong agreement): EUS can accurately differentiate between malignant and benign cystic lesions of the pancreas, as well as between the specific types of cystic lesions. The reviewers, who were blinded to the clinical/surgical history and histopathologic results for each patient, reviewed each case solely for the presence or absence of the following previously defined features of cystic lesions of the pancreas: abnormality of the pancreatic duct and pancreatic parenchyma, presence or absence of margins, solid component, debris, and septations. The septations were further defined as thick or thin; the margins, as smooth or irregular. The reviewers were asked to identify each lesion as neoplastic or non-neoplastic and to give a specific diagnosis for each. They were permitted to view the videotape as many times as desired. The Institutional Review Board at the University of Pennsylvania approved the study. Statistical analysis All statistical analyses were performed with statistical software (SAS version 8.1; SAS Institute, SAS Campus Drive, Cary, N.C.; Stata 7.0; Stata Corp., College Station, Tex.). Interobserver agreement for each feature and the global diagnosis were computed by using the kappa statistic. Kappa statistic measures the degree of agreement over and above chance agreement. Kappa values range from 1 (perfect conflict) to +1 (perfect agreement). When observed agreement is less than chance agreement, the kappa statistic takes a negative value, the minimum of which is 0 and 1. For this study, kappa values greater than 0.81 were considered almost in perfect agreement; 0.61 to 0.80 were considered substantial; 0.41 to 0.60, moderate; 0.21 to 0.40, fair; and 0.00 to 0.20 were considered as slight agreement. [4] By using histopathology as the reference standard, accuracy was calculated as the total number of correct identifications on EUS, divided by the total number of completed reviews, with each of 31 cases being reviewed by 8 reviewers. Standard errors were adjusted for clustering by the reviewer by using the Huber-White sandwich estimator of variance. [5] [6] An observation was excluded for analysis if two or more reviewers rated it as indeterminate. Generalized estimating equations (GEE) logistic regression analyses were performed for determining whether reviewer features are predictive of accuracy on EUS ratings. [7] [8] The GEE modeling framework allows the specification of a working correlation structure to account for the fact that repeated observations for a given case are not independent between the reviewers. Odds ratios (OR) and 95% confidence intervals (CI) were calculated. A series of 2 2 tables were produced to examine the association between EUS features and whether

4 Página 4 de 9 a lesion was malignant or benign, based on histopathology. Chi-square tests with the Yates continuity correction or the Fisher exact test, when cell counts were sparse, were used to test for statistically significant associations. A feature was considered present if 5 or more of the 8 reviewers agreed that it was present. A p value of less than 0.05 was considered significant against a two-sided alternative, where relevant. Results All 8 endosonographers completed an evaluation of the videotaped EUS examinations from 31 patients. All were highly experienced and had performed a median of 250 (range ) pancreatic EUS examinations per year. Among these procedures, a median of 25 (range ) examinations per year were for cystic lesions of the pancreas. Five of the 8 endosonographers had completed thirdtier fellowships in EUS. The study videotape included examinations of histopathologically proven serous cystadenomas (9), mucinous cystadenomas (10), inflammatory pseudocysts (6), neuroendocrine tumors (4), and IPMT (2). The results of the kappa statistic for agreement between the 8 reviewers for diagnosis of neoplastic versus non-neoplastic lesions and for the specific type of cystic lesion are shown in Table 1. Table 1. Kappa statistics for agreement between 8 reviewers, excluding cases rated as indeterminate by two or more reviewers EUS diagnosis Kappa Neoplastic vs. non-neoplastic Pseudocyst Serous cystadenoma Mucinous cystadenoma/adenoca Neuroendocrine Simple cyst Intraductal papillary mucinous tumor Type combined EUS features Pancreatic duct Pancreatic parenchyma Solid component Debris Septations Loculation 0.08 Margins There were 4 cases that were deemed indeterminate by two or more endosonographers for diagnosis of neoplastic versus non-neoplastic cystic lesions, and two cases that were rated as indeterminate by two or more endosonographers for diagnosis of the specific type of lesion. These cases were, therefore, excluded from the analysis of agreement between reviewers. There was only fair agreement between the endosonographers for the global diagnosis of neoplastic versus nonneoplastic lesions (κ = 0.24). When agreement was assessed for the individual type of lesions, it was moderately good for serous cystadenomas (κ = 0.46) and fair for the remainder of the cystic lesions. The overall agreement for diagnosis of the specific type of lesion was also fair (κ = 0.31).

5 Página 5 de 9 The agreement between endosonographers for individual features of cystic lesions is shown in Table 1. The number of cases that were deemed indeterminate by two or more endosonographers, and thus excluded from analysis of each of the features, were as follows: abnormal pancreatic duct (19), abnormal pancreatic parenchyma (18), solid component (1), debris (0), septations (1), and margins (5). There was moderately good agreement among endosonographers for the presence or absence of solid component (κ = 0.43). There was fair interobserver agreement for the presence or absence of abnormal pancreatic duct, debris, and septations, and only slight agreement for the presence or absence of margins and abnormal pancreatic parenchyma. Among the 8 reviewers, individual accuracy rates for the EUS diagnosis of neoplastic versus nonneoplastic lesions varied from 40% to 93% (Table 2). Table 2. Accuracy of EUS for predicting neoplastic vs. non-neoplastic lesions Overall, adjusted for clustering by reviewer Excluding lesions rated as All lesions indeterminate % Accurate 95% CI % Accurate 95% CI 71.0 [57.3, 81.7] 77.0 [67.7, 84.3] Reviewer [70.2, 96.4] 87.1 [70.2, 96.4] Reviewer [22.7, 59.4] 52.2 [30.6, 73.2] Reviewer [54.1, 87.7] 73.3 [54.1, 87.7] Reviewer [77.9, 99.2] 93.3 [77.9, 99.2] Reviewer [58.9, 90.4] 82.8 [64.2, 94.2] Reviewer [33.1, 69.8] 66.7 [44.7, 84.4] Reviewer [45.4, 80.8] 71.4 [51.3, 86.8] Reviewer [62.5, 92.5] 80.6 [62.5, 92.5] When lesions that were rated as indeterminate were excluded, the accuracy rates varied from 52% to 93%. Overall accuracy of EUS for diagnosis of non-neoplastic and neoplastic lesions was, respectively, 50% and 75% (Table 3). Table 3. EUS diagnosis compared with histopathologic findings Reviewer's EUS diagnosis Neoplastic (n = 200) Histopathology Non-neoplastic (n = 48) Indeterminate Neoplastic 150 (75%) 21 (44%) 0 (0%) Non-neoplastic 31 (16%) 24 (50%) 0 (0%) Indeterminate 19 (86%) 3 (14%) 0 (0%) Reviewer's EUS diagnosis Serous cystadenoma (n = 72) Histopathology All other neoplastic lesions (n = 128) Non-neoplastic lesions (n = 48) Serous cystadenoma 34 (47%) 13 (10%) 0 (0%) All other neoplastic 28 (39%) 88 (69%) 23 (48%) lesions Non-neoplastic lesions 10 (14%) 27 (21%) 25 (52%)

6 Página 6 de 9 Of 200 observations of histopathologically proven neoplastic lesions, 150 (75%) were correctly identified as neoplastic, whereas 31 (16%) were incorrectly identified as non-neoplastic. Of the 48 observations of histopathologically proven non-neoplastic lesions, 24 (50%) were correctly identified as non-neoplastic, while 21 (44%) were incorrectly identified as neoplastic. There were 22/248 (9%) observations that were classified as indeterminate, of which the majority (86%) were neoplastic by final histopathologic assessment. Of a total of 72 observations of histopathologically proven serous adenomas, 34 (47%) were correctly identified and 28 (39%) were incorrectly identified as other neoplastic lesions. None of the EUS features were found to be predictive of neoplasia (Table 4). Table 4. EUS features as predictors of neoplastic lesions Lesion feature OR (95% CI) Chi-square p value Solid component 0.68 [0.17, 2.80] * Septations 2.00 [0.43, 9.26] Loculation 2.40 [0.47, 12.13] Irregular margins 2.00 [0.16, 24.66] Absence of pancreatic features * * Continuity adjusted p value. Fisher exact test p value. n = 2 benign lesions have pancreatic parenchyma, n = 0 malignant lesions have pancreatic parenchyma. Discussion This study demonstrates that when using EUS criteria, there is only fair interobserver agreement among experienced endosonographers for the diagnosis of neoplastic versus non-neoplastic cystic lesions, as well as for diagnosis of the specific type of cystic lesion of the pancreas. In addition, there is little more than chance agreement among experienced endosonographers on EUS features of pancreatic cystic lesions when using published criteria. Once a pancreatic cystic lesion is identified, the main clinical issue is the characterization of the lesion, with prognosis and eventual therapeutic approach strongly influenced by the nature of the lesion. The distinction between neoplastic and non-neoplastic lesions is particularly difficult in patients with no history to indicate a particular diagnosis. Despite the availability of sophisticated imaging modalities, such as magnetic resonance imaging and CT, discriminating benign cystic lesions from those that have malignant potential is often difficult and short of operative resection and histopathologic assessment, differentiation is unreliable. This difficulty has been demonstrated in several studies. [9] [12] In a series of 67 patients, Warshaw et al. [9] found it impossible to distinguish mucinous cystadenoma from serous cystadenoma by using US or CT, unless the tumor exhibited features typical of serous adenoma such as honeycomb appearance. A radiographic analysis also called into question the validity of CT characteristics for a definitive diagnosis. [13] Other studies have shown that up to 40% of mucinous cystadenomas and up to 33% of serous cystadenomas are misdiagnosed as pseudocysts and inappropriately treated as a result. [10] [12] Thus, radiographic features are not sufficiently specific for accurate diagnosis. The difficulty with the non-operative diagnosis, the high proportion of lesions with malignant potential, and the good outcome with surgical resection have led to the suggestion that all lesions of the pancreas suspected to be cystic tumors should be resected. [9] [14] [16]

7 Página 7 de 9 EUS imaging is high resolution and is considered valuable for characterization of pancreatic cystic lesions. However, few studies have described the EUS features of these lesions. Most often cited is that by Koito et al., [1] which suggested that EUS provides precise characterization of the internal architecture of pancreatic cystic tumors less than 2 cm in diameter and thus helps to distinguish neoplastic from non-neoplastic lesions. However, the patterns used to describe the internal architecture of the lesions in that study can be difficult to reproducibly discern by EUS. Thus, little is known about the exact criteria that endosonographers use to distinguish between the different cystic lesions. Questions also arise as to whether endosonographers are using uniform criteria to describe cystic lesions, whether there is consensus among them as to the various EUS features of cystic lesions, and whether EUS morphologic characteristics can accurately discriminate between the different types of cystic lesions of the pancreas. For the present study, simple EUS features were selected that, based on prior multiple regression analyses, were predictive of neoplastic cystic lesions of the pancreas. [17] Based on these criteria, there was little agreement among a group of experienced endosonographers whether pancreatic cystic lesions were neoplastic versus non-neoplastic. In addition, there was also little more than chance agreement among endosonographers for EUS features of cysts, as well as the specific type of cystic lesion. These results are of concern because there does not appear to be much uniformity among endosonographers for characterization of pancreatic cystic lesions. Agreement among observers was moderately good only for serous cystadenoma. This finding is similar to that of studies of CT and transabdominal US, which found that serous cystadenoma is the only type of pancreatic cystic lesion that can be characterized with some certainty. [16] [18] However, only 47% of the serous cystadenomas were correctly identified in the present study, and 39% were labeled as another neoplastic lesion. Similarly, there was also a lack of agreement among endosonographers for individual features, except for solid component, where there was moderately good agreement. These results differ slightly from those of a study of interobserver agreement between two endosonographers for features of cystic lesions of the pancreas. [19] That study, by Ahmad et al., [19] demonstrated moderately good agreement for solid component, margins, and septations, but the design was based on review of single EUS images instead of videotape or dynamic digital images. Although the kappa statistic is a measure of agreement, excellent or good agreement does not necessarily imply that the outcome agreed upon is actually correct. Therefore, accuracy rates were assessed, which demonstrated a wide variation among experienced endosonographers. The number of pancreatic EUS procedures performed per year and the belief of reviewers that EUS can differentiate cystic lesions by type were not significantly associated with accuracy rates (data not shown). What could explain the results of the present study? First, it was based on review of videotape of examinations that may not have completely reproduced the findings as compared with an actual realtime examination. Second, the reviewers were not given clinical histories, which has been shown to be relevant for differentiating pseudocysts from neoplastic cysts. The combination of clinical history and cross-sectional imaging, along with real-time EUS, may increase the contribution of EUS to the characterization of cystic lesions of the pancreas. Third, the patient population from which the study videotapes were derived consisted of consecutive patients who underwent surgery. Thus, there is the possibility that the EUS findings in these patients were somewhat atypical, which may have led to results that are not entirely generalizable. However, the patients are likely representative of the population seen in major pancreaticobiliary centers, and this may be the population most likely to benefit from imaging studies that enhance diagnostic accuracy. Patients with pancreatic cystic lesions that are well characterized by cross-sectional imaging studies need not undergo EUS. The cystic lesions included in the study, therefore, represent the subset where the distinction between malignant and benign lesions is critical. This study also found a great deal of subjectivity in describing EUS features. That which is interpreted as loculation by one reviewer may be interpreted as septation by another. This may

8 Página 8 de 9 reflect a lack of a uniform nomenclature for describing the EUS features of cystic lesions. If morphology is to be used to stratify cystic lesions, it is necessary to validate currently used EUS descriptors in well-designed studies and for the endosonographers to reach consensus on EUS criteria so that uniform descriptors can be developed. These steps do appear to be potentially useful, as the findings in the present study indicate that the predictive ability for some reviewers was high. In addition, the present study may not have evaluated all of the determinative criteria for describing pancreatic cystic lesions. In conclusion, EUS morphology alone may not be adequate for discrimination between the various pancreatic cystic lesions. Given the limitations of currently used morphologic criteria, better predictive features need to be developed and studied in prospective trials. In addition, studies assessing needle sampling, including the use of histologic and biochemical markers of neoplasia, are needed. References 1. Koito K, Namieno T, Nagakawa T, Shyonai T, Hirokawa N, Morita K. Solitary cystic tumor of the pancreas: EUSpathologic correlation. Gastrointest Endosc 1997;45: Full Text 2. Gress F, Gottleib K, Cummings O, Sherman S, Lehman G. Endoscopic ultrasound characteristics of mucinous cystic neoplasms of the pancreas. Am J Gastroenterol 2000;95: Abstract 3. Hawes RH, Zaidi S. Endoscopic ultrasonography of the pancreas. Gastrointest Endosc Clin N Am 1995;5: Abstract 4. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33: Abstract 5. Huber PJ. The behavior of maximum likelihood estimates under non-standard conditions. Proceedings of the 5th Berkeley Symposium in Mathematical Statistics and Probability; Berkeley: University of California Press p White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 1980;48: Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics 1986;42: Abstract 8. Diggle PJ, Liang KY, Zeger SL. Analysis of longitudinal data. Oxford: Oxford University Press; Warshaw A, Compton CC, Lewandrowski K, Cardenosa G, Meuller PR. Cystic tumors of the pancreas. New clinical, radiologic, and pathologic observations in 67 patients. Ann Surg 1990;212: Abstract 10. Mathieu D, Guigui B, Valette PJ, Dao T, Bruneton JN, Bruel JM, et al. Pancreatic cystic neoplasms. Radiol Clin North Am 1989;27: Abstract 11. Warshaw A, Rutledge P. Cystic tumors mistaken for pancreatic pseudocysts. Ann Surg 1987;205: Abstract 12. Le Borgne J, De Calan L, Partensky C. Cystadenomas and cystadenocarcinomas of the pancreas. A multiinstitutional retrospective study of 398 cases. French Surgical Association. Ann Surg 1999;230: Abstract 13. Curry CA, Eng J, Horton KM, Urban B, Siegelman S, Kuszyk BS, Fishman EK. CT of primary cystic pancreatic neoplasms. Can CT be used for patient triage and treatment? AJR Am J Roentgenol 2000;175: Abstract 14. Ooi LL, Ho GH, Chew SP, Low CH, Soo KC. Cystic tumors of the pancreas: a diagnostic dilemma. Aust N Z J Surg

9 Página 9 de ;68: Abstract 15. Martin I, Hammond P, Scott J, Redhead D, Carter DC, Garden OJ. Cystic tumors of the pancreas. Br J Surg 1998;85: Abstract 16. Procacci C, Biasuitti C, Carbognin G, Accordini S, Bicego E, Guarise A, et al. Characterization of cystic tumors of the pancreas: CT accuracy. J Comput Assist Tomogr 1999;23: Abstract 17. Faigel DO, Kochman ML, Smith DB, Ginsberg G. Endosonography of complex cystic masses of the pancreas with clinicopathologic correlation [abstract]. Gastrointest Endosc 1996;43: Johnson CD, Stephens DH, Charboneau JW, Carpenter HA, Welch TJ. Cystic pancreatic tumors: CT and sonographic assessment. AJR Am J Roentgenol 1988;151: Abstract 19. Ahmad NA, Kochman ML, Lewis JD, Ginsberg GG. Can EUS differentiate between malignant and benign cystic lesions of the pancreas? Am J Gastroenterol 2001;96: Abstract Copyright 2005 Elsevier Inc. All rights reserved. Bookmark URL: /das/journal/view/ /n/ ?ja=364859&page=1.html&anchor=top&source=mi