Agreement Between Cytotechnologists and Cytopathologists as a New Measure of Cytopathologist Performance in Gynecologic Cytology

Transcription

1 Agreement Between Cytotechnologists and Cytopathologists as a New Measure of Cytopathologist Performance in Gynecologic Cytology Andrew M. Quinn, MD 1 ; Abu T. Minhajuddin, PhD 2 ; Linda S. Hynan, PhD 2 ; Joan S. Reisch, PhD 2 ; and Edmund S. Cibas, MD 3 BACKGROUND: Although objective measures of cytotechnologist (CT) and cytopathologist (CP) performance exist, challenges remain. Two assumptions deserve examination: CPs interpretations are correct, and CTs and CPs render interpretations independently of each other. This study presents a CT-CP interpretation comparison and provides insight into these assumptions. METHODS: Every gynecologic cytology specimen examined by both a CT and a CP from December 2004 to March 2015 was extracted from the laboratory information system; glandular interpretations were excluded. Excel and SAS were used for CT-CP pair analysis. CT-CP pairs with fewer than 32 specimens (the lowest quartile) were excluded. For the remaining CT-CP pairs, 30 specimens or 10% of the specimens (whichever was higher) were randomly selected for comparison by a weighted j statistic. j values greater than 0.6 represented good agreement within CT-CP pairs. RESULTS: This study evaluated 7116 of 53,241 gynecologic cytology specimens (13.4%) that received CT and CP interpretations. This resulted in 155 pair-specific j values from 15 CTs and 16 CPs. In aggregate, the j values had a mean of 0.64, a standard deviation of 0.14, a median of 0.65, and a range of 0.27 to Nine CTs exhibited good agreement in the majority of their pair-specific j values with CPs (high-concordance CTs; 88 pair-specific j values). This allowed us to identify outlier CPs who did not demonstrate good agreement with high-concordance CTs (16 of 88 pair-specific j values [18.2%]). CONCLUSIONS: Laboratories can use this j to determine when CP levels of agreement with CTs depart from those of their peers. Adding this to established metrics can give a more nuanced impression of CP performance. Cancer Cytopathol 2017;125: VC 2017 American Cancer Society. KEY WORDS: agreement; cytopathologist; cytotechnologist; gynecologic; j; outlier; Papanicolaou test; performance; quality; SQL. INTRODUCTION Much attention has been paid to quality improvement, competency assessment, proficiency testing, and performance evaluation in gynecologic cytology, particularly since the mandates of the Clinical Laboratory Improvement Amendments of A number of metrics are already in use for assessing the performance of cytotechnologists (CTs) and cytopathologists (CPs). For CTs, false-negative rates, abnormal rates, andunsatisfactory rates are familiar measures. For CPs, metrics include the ratio of atypical squamous cells of undetermined significance (ASCUS) interpretations to squamous intraepithelial lesion interpretations (ASCUS to SIL ratio)and the percentage of ASCUS cases that are positive for high-risk human papillomavirus. 1 Thegoaloftheseeffortsistoprovideobjective measures of performance. Corresponding author: Andrew M. Quinn, MD, Department of Pathology, University of Texas Southwestern Medical Center, 5200 Harry Hines Boulevard, Dallas, TX 75235; andrew.m.quinn@gmail.com 1 Department of Pathology, University of Texas Southwestern Medical Center, Dallas, Texas; 2 Division of Biostatistics, Department of Clinical Sciences, University of Texas Southwestern Medical Center, Dallas, Texas; 3 Department of Pathology, Brigham and Women s Hospital and Harvard Medical School, Boston, Massachusetts Received: January 5, 2017; Revised: February 8, 2017; Accepted: February 10, 2017 Published online March 27, 2017 in Wiley Online Library (wileyonlinelibrary.com) DOI: /cncy.21856, wileyonlinelibrary.com 576 Cancer Cytopathology July 2017

2 Cytotechnologists and Cytopathologists/Quinn et al The level of a CT s agreement with CPs interpretations is also a common measure of a CT s performance and one previously reported by this laboratory, 2 butitgenerally relies on 2 assumptions: 1) CPs diagnoses (vs those of CTs) are correct, and 2) CTs and CPs render their interpretations independently of each other. A 2013 survey of gynecologic cytology practice patterns by the College of American Pathologists found that most laboratories monitor CP upgrades and downgrades of CT interpretations as measures of CT performance. 3 Specifically referring to upgrades from negative for intraepithelial lesion or malignancy (NILM) or ASCUS to a high-grade squamous intraepithelial lesion (HSIL) or worse among 375 respondents, the survey reported that 82.7% and 35.7% of laboratories monitor these rates in CTs but only 37.3% and 18.1% monitor these rates in CPs. 4 Further analysis of these survey data revealed that laboratories are more likely to audit cases reviewed by CTs and take action when a CT (vs a CP) demonstrates performance variation. 5 These disparities are noteworthy in the face of data suggesting that inter- and intra-observer variability among CPs in gynecologic cytology can be high and that rates of interpretation errors by CPs exceed rates of screening errors by CTs. 6 8 Furthermore, side-by-side comparisons of CT and CP performance in gynecologic cytology are rare. Occasional studies, however, have addressed CPs capacity for screening, a task generally assumed to be the purview of CTs, and CP performance has been shown to vary with theprevalenceofabnormalcells. 9,10 Likewise, although the ratio of ASCUS interpretations tosquamousintraepithelial lesion interpretations (ASCUS to SIL ratio) is typically a measure of CP performance, it also has value in assessing CTs. 11,12 More elusive are comparisons in which CTs serve as the standard for interpretation. In this study, we attempt to make such a comparison and provide insight into the aforementioned assumptions through the use of a weighted j statistic for the most common gynecologic cytology interpretations. MATERIALS AND METHODS The Brigham and Women s Hospital is a tertiary-care academic medical center that serves a mix of privately and publically insured, high- and low-risk gynecologic patients in eastern Massachusetts. The cytology laboratory uses the ThinPrep Papanicolaou test and the ThinPrep imaging system (Hologic, Inc, Marlborough, Mass) for virtually all TABLE 1. Values Ascribed to Squamous Interpretations Interpretation Category Classification Unsatisfactory 1 NILM/NILM-RCT 2 ASCUS 3 LSIL 4 ASC-H 5 HSIL1 6 Abbreviations: ASC-H, atypical squamous cells, cannot exclude a highgrade squamous intraepithelial lesion; ASCUS, atypical squamous cells of undetermined significance; HSIL1, high-grade squamous intraepithelial lesion or worse; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy; NILM-RCT, negative for intraepithelial lesion or malignancy, reactive/reparative cellular changes. The laboratory adhered to Bethesda terminology for all interpretations during the study period. For statistical purposes, cytotechnologist-cytopathologist interpretation agreement counted only when the respective interpretation categories were identical. gynecologic cytology specimens. Results of human papillomavirus testing are not available at the time of screening or finalinterpretationbycps.ctsandcpsentertheirinterpretations into the laboratory information system (LIS; PowerPath, Sunquest, Tucson, Ariz) in a dictionary/menudriven fashion; this means that diagnoses are captured as discrete values (eg, NILM) in fields in the LIS database. In this study, using the LIS SQL management tool, we extracted from the LIS the CT and CP interpretations of every gynecologic cytology specimen between December 2004 and March 2015 (n 5 405,740). The laboratory adhered to Bethesda terminology for all interpretations during the study period. Squamous interpretations were classified in an ordinal fashion with assignment to the following interpretation categories: 1) unsatisfactory; 2) NILM and negative for intraepithelial lesion or malignancy, reactive/ reparative cellular changes (NILM-RCT); 3) ASCUS; 4) low-grade squamous intraepithelial lesion (LSIL); 5) atypical squamous cells, cannot exclude a high-grade squamous intraepithelial lesion; and 6) HSIL or worse (Table 1). Ordinal classification assumes that a relationship exists in the ranking of categories but does not hold that the intervals between categories are equal (eg, ASCUS is between NILM/NILM-RCT and LSIL but not necessarily equally so). For simplicity, glandular interpretations were excluded from the evaluation. Data manipulations, including de-identification, interpretation category assignment, and scrubbing, were performed in Excel (Microsoft Corporation, Redmond, Wash). Statistical analyses, including j calculations, were Cancer Cytopathology July

3 performed via SAS (SAS Institute, Inc, Cary, NC). For individual specimens, agreement occurred only when the interpretation category ascribed by the CP was identical to that of the CT. For the j values, which applied to the aggregate of interpretations for a CT-CP pair, a value greater than 0.6 represented good agreement. Figures were created in the R language and environment for statistical computing (R Foundation for Statistical Computing, Vienna, Austria) and were enhanced in Adobe Photoshop CS5 (Adobe Systems, Inc, San Jose, Calif). The institutional review board at the hospital approved the study under expedited review. RESULTS The laboratory received 405,740 gynecologic cytology specimens during the roughly 10-year study period (approximately 39,265 per year). Of these specimens, 53,241 received interpretations by both a CT (n 5 15) and a CP (n 5 17). Because the number of specimens evaluatedbyeachct-cppair(n5 206) ranged from 1 to 3019, the lowest quartile of CT-CP pairs (fewer than 32 specimens) was excluded. This left 155 distinct CT-CP pairs, including 15 CTs and 16 CPs. We then randomly selected a 10% sample from each CT-CP pair with the additional condition that no pair could have fewer than 30 specimens in the analysis. Thus, 7116 of the 53,241 specimens (13.4%) interpreted by both a CT and a CP were ultimately included in the j analysis. Figure 1 demonstrates the distribution of the j values, which had a mean of 0.64, a standard deviation of 0.14, a median of 0.65, and a range of 0.27 to Figure 2 consists of a heat map style representation of each of the 155 pair-specific j values. Nine of 15 CTs (60.0%) exhibited good agreement in the majority of their pair-specific j values with CPs (88 pair-specific j values [56.8%]). We dubbed these high-concordance CTs. Among the high-concordance CTs, 72 of the 88 pair-specific j values (81.8%) represented good agreement. Therefore, 16 of these 88 pair-specific j values (18.2%) with high-concordance CTs did not represent good agreement. Two CPs accounted for 7 of these 16 j values showing less than good agreement (43.8%). DISCUSSION In the clinical practice of gynecologic cytology, the CP s interpretation is the de facto gold standard. In this study, we have demonstrated that a laboratory can identify CTs Figure 1. Distribution of weighted j statistics for cytotechnologist-cytopathologist pairs. In aggregate, the j values had a mean of 0.64, a standard deviation of 0.14, a median of 0.65, and a range of 0.27 to A j value greater than 0.6 was considered to indicate good agreement. with whom CPs generally agree. These high-concordance CTs can thus serve as references against which CPs performance can be measured. For example, our analysis revealed that 2 CPs accounted for nearly half of the instances of less than good agreement with our high-concordance CTs and, therefore, could be considered outlier CPs. Although the use of j values in this way should not serve asthesolemeasureofcpperformanceingynecologic cytology, especially because they do not account for absolute interpretation correctness, in the context of other traditional measures, they can provide a clearer picture of a CP s performance with respect to her or his peers. We envision leveraging these j values as an additional measure of performance in 2 ways: 1) providing resultsasfeedbacktocpsand2)monitoringofresultsby the laboratory director as part of both individual and laboratory performance assessments. An augmentation of these ideas might also include a review of the shared specimens of CT-CP pairs of interest. The random selection of a smaller cohort of specimens (n [13.4%]) allowed us to demonstrate the feasibility of these j values in laboratories of all sizes. For example, higher volume laboratories can adopt an approach similar to the selection process characterized previously. Smaller volume laboratories or those with fewer CTs and CPs, on the other hand, can perform such j calculations on an annual or biennial basis without limiting cohort sizes. Either way, 7116 data points are more than sufficient for a j analysis such as this one. 578 Cancer Cytopathology July 2017

4 Cytotechnologists and Cytopathologists/Quinn et al Figure 2. Weighted j statistics by CT-CP pairs. Each colored box represents a j value for a specific CT-CP pair. The colored boxes in the key at the top of the figure identify the j quintiles. A j value greater than 0.6 (ie, yellow and green boxes) was considered to indicate good agreement. The cytotechnologist code is a study-specific number for each CT; the cytopathologist code is a study-specific number for each CP. The codes are ordered by ascending percentages of j values greater than 0.6 (from left to right for CTs and from bottom to top for CPs). High-concordance CTs fall to the right of the black line. CP indicates cytopathologist; CT, cytotechnologist. Regarding the assumption that CTs and CPs render their interpretations independently of each other, our results suggest that CPs have tendencies to agree with some CTs and disagree with others, which against purely independent decision making. A prospective study design, however, in which CTs and CPs are blinded to both reviewer and interpretation will be needed to complement this initial evaluation. Our retrospective review also cannot account for the possibility that CTs might tailor their interpretations to the CP for whom they are screening on a given day. Similarly, although they are not included in this analysis (because of an absence of electronic data), at academic institutions, residents and fellows participate in and may have an influence on interpretations. Another area of future study would be an investigation into the effect of experience on agreement. In this context, experience could refer to the number of specimens that CTs or CPs have reviewed both individually and collectively as well as the durations of their careers and certification credentials. Such a study would require a determination of interpretation correctness. The ordinality of our classification of squamous interpretations allowed us to use a weighted j statistic (vs a simple one). There is no consensus as to the best such ordinal classification of gynecologic cytology specimens. That said, nothing prevents future investigations from using different classifications of squamous (and even glandular) interpretations. The key to any approach is consistency and repetition to ensure that results have meaning to the laboratories that use them. Central to the success of this and related future analyses is capturing interpretations asdiscretevaluesinanlis database. So-called structured data allow easy extraction and scrubbing of data as well as fast transformation of data into manipulable numbers (in our case, the ordinal classification scheme). Though doable, using natural language processing, unstructured text mining techniques, or even document review to identify interpretations would be challenging, especially for a data set with hundreds of thousands of records. It has been demonstrated that CPs can improve performance through feedback and education. 9 Although Cancer Cytopathology July

5 current methods of assessing CP performance in gynecologic cytology provide a multifaceted snapshot of individuals and laboratories, shortcomings such as the biases inherent to internal auditing (eg, for the 10% rescreen and 5-year look-back) remain. 13 Harnessing large data sets with robust software and a well-designed LIS can provide novel insights and enhance the performance evaluation program of a laboratory. FUNDING SUPPORT No specific funding was disclosed. CONFLICT OF INTEREST DISCLOSURES The authors made no disclosures. AUTHOR CONTRIBUTIONS Andrew M. Quinn: Conceptualization, methodology, validation, formal analysis, investigation, resources, data curation, writing original draft, writing review and editing, visualization, supervision, and project administration. Abu T. Minhajuddin: Methodology, software, validation, formal analysis, data curation, writing original draft, writing review and editing, and visualization. Linda S. Hynan: Methodology, writing review and editing, and visualization. Joan S. Reisch: Methodology, formal analysis, writing original draft, writing review and editing, and visualization. Edmund S. Cibas: Conceptualization, methodology, writing review and editing, and supervision. REFERENCES 1. Cibas E. Laboratory management. In: Cibas E, Ducatman B, eds. Cytology: Diagnostic Principles and Clinical Correlates. 4th ed. Philadelphia, PA: Elsevier; 2014: Cibas E, Dean B, Maffeo N, Allred E. Quality assurance in gynecologic cytology. The value of cytotechnologist-cytopathologist discrepancy logs. Am J Clin Pathol. 2001;115: Clary K, Davey D, Naryshkin S, et al. The role of monitoring interpretive rates, concordance between cytotechnologist and pathologist interpretations before sign-out, and turnaround time in gynecologic cytology quality assurance: findings from the College of American Pathologists Gynecologic Cytopathology Quality Consensus Conference Working Group 1. Arch Pathol Lab Med. 2013; 137: Brainard J, Birdsong G, Elsheikh T, et al. Prospective and retrospective review of gynecologic cytopathology: findings from the College of American Pathologists Gynecologic Cytopathology Quality Consensus Conference Working Group 2. Arch Pathol Lab Med. 2013;137: Tworek J, Nayar R, Savaloja L, et al. General quality practices in gynecologic cytopathology: findings from the College of American Pathologists Gynecologic Cytopathology Quality Consensus Conference Working Group 3. Arch Pathol Lab Med. 2013;137: Rohr L. Quality assurance in gynecologic cytology. What is practical? Am J Clin Pathol. 1990;94: Schiffman M, Solomon D. Findings to date from the ASCUS- LSIL Triage Study (ALTS). Arch Pathol Lab Med. 2003;127: Stoler M, Schiffman M; Atypical Squamous Cells of Undetermined Significance Low-Grade Squamous Intraepithelial Lesion Triage Study (ALTS) Group. Interobserver reproducibility of cervical cytologic and histologic interpretations: realistic estimates from the ASCUS-LSIL Triage Study. JAMA. 2001;285: Clarke J, Thurloe J, Bowditch R, Roberts J. Assuring the quality of quality assurance: seeding abnormal slides into the negative Papanicolaou smears that will be rapid rescreened. Cancer. 2008;114: Evans K, Tambouret R, Evered A, Wilbur D, Wolfe J. Prevalence of abnormalities influences cytologists error rates in screening for cervical cancer. Arch Pathol Lab Med. 2011;135: Renshaw A, Auger M, Birdsong G, et al. ASC/SIL ratio for cytotechnologists: a survey of its utility in clinical practice. Diagn Cytopathol. 2010;38: Renshaw A, Deschenes M, Auger M. ASC/SIL ratio for cytotechnologists: a surrogate marker of screening sensitivity. Am J Clin Pathol. 2009;131: Renshaw A. Rescreening in cervical cytology for quality control. When bad data is worse than no data or what works, what doesn t, and why. Clin Lab Med. 2003;23: Cancer Cytopathology July 2017