difficult and may not be practical. Nevertheless, the performance characteristics of rapid screening have not been completely characterized.

Transcription

1 Anatomic Pathology / PERFORMANCE CHARACTERISTICS OF RAPID PRESCREENING Performance Characteristics of Rapid (0-Second) Prescreening Implications for Calculating the False-Negative Rate and Comparison With Other Quality Assurance Techniques Key Words: Cytopathology; Cytology; Quality control; Papanicolaou smear; Statistics; False-negative rate; Errors; Screening; Accuracy; Uterine cervix Abstract Rapid (0-second) prescreening of cervicovaginal smears can be used to detect false-negative cases and determine the false-negative rate of primary screening, hut the performance characteristics have not been evaluated fully. A test set of cases including 80 originally false-negative cases were rapidly screened by different cytotechnologists on occasions. Intraobserver and interobsen'er reproducibility were good. specificity for each round of observations was 89% (range, 0%-96%). sensitivity for all true-positive cases was 78% (range, 6%-97%); for all false-negative cases it was 59% (range, 8%~89%). The relative sensitivity of rapid screening for truepositive and false-negative cases varied with the diagnosis. Rapid screening detected almost the same percentage of false-negative cases of atypical squamous cells of uncertain significance (ASCUS) as true-positive ASCUS cases (median ratio,.; range, ). The median ratio of false-negative to true-positive ASCUS cases was significantly different than the ratio for low-grade plus high-grade squamous intraepithelial lesions (0.68; range, ). Although performance varies between individuals, in this test population the reproducibility, specificity, and sensitivity were good. Because it detects more false-negative cases at a lower cost per case than routine rescreening, rapid prescreening should be considered as an alternative to current quality control measures. Rapid prescreening of cervicovaginal smears is a quality control measure that has been practiced predominantly in the United Kingdom. Unlike the rescreening of 0% of cervicovaginal smears that is required of laboratories in the United States, this method involves prescreening 00% of all smears, but for only 0 seconds each. The results of this prescreening are recorded and used to identify any falsenegative cases that result from the full screening that occurs subsequently. Although the sensitivity of rapidly prescreening a case is lower than that of traditional screening without a time limit, rapid prescreening allows more cases to be reviewed in the same amount of time, which results in the detection of more false-negative cases compared with 0% rescreening.'~5 For example, the sensitivity of rapid screening for a diagnosis of atypical squamous cells of undetermined significance (ASCUS) is only about 0% to 50% that of routine screening or rescreening. 5-6 But because rapid screening is approximately 5 to 0 times faster than routine rescreening, it detects more false-negative cases than does routine rescreening. In addition, rapid prescreening is a better method than routine rescreening for determining the false-negative rate (FNR) of primary screening. To accurately measure the FNR. it is necessary to determine the FNR of the method used to find false-negative cases, whether routine rescreening or rapid prescreening. 7 Since rapid prescreening, by design, includes a review of all specimens, both positive and negative, the positive specimens can be used as controls to calculate the FNR. By contrast, incorporating known positive controls in routine rescreening is difficult and may not be practical. Nevertheless, the performance characteristics of rapid screening have not been completely characterized. Although Am J Clin Pathol 999; : Andrew A. Renshaw, MD, Janet A. Cronin, CT(ASCP),l LoriJ. Minter, CT(ASCP), Dorothy Nappi, CT(ASCP), Terry Whitman, CT(ASCP), MichaelJiroutek, MS, and Edmund S. Cibas, MDJ

2 Renshaw et al / PERFORMANCE CHARACTERISTICS OF RAPID PRESCREENING Methods Papanicolaou smears for the test set were accessioned at the Brigham and Women's Hospital, Boston, MA, between January, 987, and December, 997. All cases were diagnosed according to the Bethesda System.0 For cases originally reviewed before the Bethesda System was implemented, the slides were reviewed, and a Bethesda diagnosis was made. A set of test specimens was constructed consisting of randomly selected true-negative cases, true-positive cases, and false-negative cases. False-negative cases were identified from quality control log books summarizing the results of routine 0% review of negative cases during the time period of the study. True-negative and true-positive cases were randomly selected from the same time period as false-negative cases by reviewing case records. For statistical purposes, at least 75 negative cases and between 0 and 0 abnormal cases from each diagnostic category were included, although the exact number of cases within these boundaries was chosen at random. All true-negative cases had been screened once, and each true-negative case that was thought to be abnormal by using rapid review by any reviewer was rescreened at length. No false-negative cases were identified. The final test set comprised cases with 97 true-negative cases, 65 true-positive cases (9 ASCUS, 0 low-grade squamous intraepithelial lesions [LSILs], 6 high-grade squamous intraepithelial lesions [HSILs]), and 80 false-negative cases ( ASCUS, 0 LSIL, and 9 HSIL). All marks, including labels, were removed from these slides, and they were arranged in a random order. Four cytotechnologists with no previous experience in rapid screening reviewed the set on separate occasions. Each observer was asked to determine whether the slides were 58 Am J Clin Pathol 999;:57-5 negative or positive; a positive result was defined as a diagnosis of ASCUS or above. Each observer was allowed only 0 seconds per slide. The cytotechnologists could perform the rapid screening in any manner they chose provided they adhered to the allotted time. While there was variation in the direction and order in which the slides were reviewed, no observer was able to view the entire slide in 0 seconds. Reproducibility was measured by using the K statistic. Sensitivity and specificity were determined routinely by using the defined test subpopulation. Comparison of medians was done using a nonparametric Kruskal-Wallis test. Results Intraobserver Reproducibility The results of intraobserver reproducibility for the entire test set of 5 slides are summarized in liable II. Intraobserver reproducibility was good, with K values ranging from 0. to Three of observers had K values of 0. 5 or higher. Interobserver Reproducibility Interobserver reproducibility was assessed by using pairwise comparisons for each observation with all 6 other observations. These results are summarized in ITable. K values ranged between 0. and 0.9. had the greatest degree of disagreement with the other observers. When the evaluations of observer were disregarded, the lowest interobserver K value (between observers,, and ) was 0.7. Specificity Specificity is summarized in ITable. The overall median specificity was 89%. The first round of observations by observer resulted in a much lower specificity than was seen in any other round, but results improved markedly in the second round. Overall Sensitivity Overall sensitivity is summarized in ITable and ITable 5. Not surprisingly, the observer with the lowest ITable II Intraobserver Reproducibility for the Entire Test Set of 5 Specimens K the sensitivity of the test has been evaluated, reproducibility and specificity have not been as thoroughly studied. In addition, rapid screening may preferentially detect certain types of errors, specifically those that may be seen rapidly. Because false-negative smears typically have fewer cells, making them more difficult to detect than true-positive cases,89 rapid screening may be less sensitive for false-negative cases than for true-positive cases. If this is so, then the number of cases detected by this method will be less than that using true-positive specimens as controls. An accurate calculation of the FNR would have to account for this. To address these issues, we measured the performance characteristics of rapid screening of cervicovaginal smears using a test set of known true-negative, true-positive, and false-negative cases to compare the efficacy of rapid screening for detecting false-negative cases compared with true-positive cases.

3 Anatomic Pathology / ORIGINAL ARTICLE Table Interobserver Reproducibility for the Entire Test Set of 5 Specimens Table 5 Overall Sensitivity for False-Negative Cases From 8 Sets of Observations First Observation (%) Range Second Observation (%) Table Overall Sensitivity for True-Positive Cases From 8 Sets of Observations First Observation (%) Table 6 Sensitivity by Diagnostic Category* False-Negative Cases (%) Table Specificity for 8 Sets of Observations First Observation (%) ASCUS LSIL HSIL ASCUS specificity (observer, first observation) is also the one with the highest sensitivity. For the false-negative cases, there was no significant difference between the sensitivities for ASCUS, LSIL, and HSIL. For the true-positive cases as a group, the sensitivity for ASCUS was significantly lower compared with LSIL and HSIL (P =.00). The median overall sensitivity for true-positive cases (78%) was higher than that for false-negative cases (59%). (8-8) (0-90) (-00) (6-96) (8-89) (9-90) 90(77-00) 95(76-00) 9(78-00) 78 (6-97) ASCUS = atypical squamous cells of undetermined significance; LSIL = low-grade squamous intraepithelial lesion; HSIL = high-grade squamous intraepithelial lesion. * Data are given as median (range). Table 7 Ratio of Sensitivity for False-Negative Cases/True-Positive Cases by Diagnostic Category Second Observation (%) True-Positive Cases (%) ASCUS LSIL HSIL ASCUS + Range.* ASCUS = atypical squamous cells of undetermined significance; LSIL = low-grade squamous intraepithelial lesion; HSIL = high-grade squamous intraepithelial lesion. * Significantly different from (P =.006). of 8 observations was compared. The median ratio for ASCUS cases was significantly different than the ratio for LSIL (P =.00), HSIL (P =.006), and LSIL plus HSIL (P =.006). No significant differences were found between the ratios for LSIL, HSIL, or LSIL plus HSIL. Sensitivity by Diagnostic Category The sensitivity by diagnostic category is summarized in Table 6. The median sensitivity for false-negative ASCUS cases was higher than the sensitivity for true-positive ASCUS cases. On the other hand, the sensitivity for LSIL and HSIL was consistently lower for false-negative cases. This relationship is further shown in (Table 7, in which the ratio of false-negative cases/true-positive cases for each set Discussion Rapid prescreening offers significant advantages over routine rescreening, namely, detection of more abnormal cases and a more accurate measurement of the FNR of primary screening. The reproducibility and specificity of this technique in the present study were good. Variability Am J Clin Pathol 999;: ' Pairwise comparison of each of observations with the results of all 6 other observations from the other observers. Second Observation (%)

4 Renshaw et al / PERFORMANCE CHARACTERISTICS OF RAPID PRESCREENING 50 AmJCIinPathol 999;:57-5 ASCUS case may simply overshadow any differences that exist between false-negative and true-positive ASCUS cases. Although there are differences between the sensitivity for true-positive and false-negative cases, the importance of these differences is unclear. Even if real-life rapid screening is not as effective at detecting abnormal cases as originally suggested, the difference is slight, and the number of cases detected still exceeds the number detected by routine 0% rescreening efforts. Note that because the specificity of rapid screening is only 90%, approximately 0% of true-negative slides are flagged as abnormal; such flagged cases will be reviewed in the routine fashion and reclassified as negative. In addition, whatever abnormal slides are detected by rapid screening also will be reviewed. Thus, rapid rescreening generates a unique pool of slides, including false-negative and true-negative cases, that are selected for rescreening. This selection of slides for rescreening resembles the AutoPap System (Neopath, Redmond, WA) method for detecting false-negative cases. The AutoPap System is approved to select a pool of slides that are enriched in potential errors for routine rescreening.-5 How do the methods compare? The AutoPap system provides to 5 times the rate of error detection compared with routine rescreening. The sensitivity of the device at a 0% threshold is 8% for truepositive ASCUS cases, 5% for true positive LSIL cases, and 8% for true-positive HSIL cases. The calculated sensitivity for all false-negative cases is 5% and for falsenegative LSIL and higher cases is 5%. 5 Rapid prescreening seems to be at least as sensitive if not more so than the AutoPap device in all of these categories. Some information on the PAPNET Testing System (Neuromedical Systems, Suffern, NY) is also available. Although study seems to contain enough data to make similar comparisons,"6 it did not include ASCUS as a diagnostic category. The sensitivity for false-negative slides (including only LSIL and HSIL) was %; the specificity was 8%. In other studies, specificity ranged between 75% and 8%.u-7'8 Since other studies have generated data about the sensitivity of the technique without measuring the specificity,9 and others have used slide populations that may not be comparable,0 meaningful comparison of these results is difficult. Of note, recent study directly compared the results of rapid rescreening with the PAPNET system.7 Unfortunately, only a small number of cases (0) were evaluated. In addition, the results do not reflect rapid rescreening in a real-life situation because the cases that were selected were especially difficult and, thus, represent a sample of only the most difficult false-negative cases. Clearly, rapid screening does not detect all false-negative cases, and those that are difficult to detect with routine methods will be even harder to detect with rapid screening. On the other hand, one would expect rapid prescreening to be particularly good at detecting cases between individuals exists, and some may be more skilled at this than others. Nevertheless, we believe that the reproducibility, specificity, and sensitivity of rapid screening as performed by most cytotechnologists, including those without previous experience in rapid screening, justify serious consideration of rapid prescreening for routine laboratory use. In addition, whatever variability exists in the performance of this test can be measured and accounted for. Routine rescreening of 0% of the negative cases, because it cannot readily include abnormal cases for use as controls, does not have this advantage. Nevertheless, the data also show that careful analysis of the results is important for rapid screening to be valuable. Other studies have suggested that the results obtained in trial situations, such as those used in the present study, and those obtained in real-life situations may not be comparable, '' specifically because the vigilance of the observer may be different. Nevertheless, in experienced hands, in an intended-use, real-life situation, the sensitivity of rapid screening ranges from 70% to 90% for HSIL to 0% to 50% for a diagnosis of ASCUS or LSIL.,5 The sensitivities for ASCUS (%), LSIL (90%), and HSIL (95%) in the present study are similar to the results reported from these intended-use studies. Interestingly, the sensitivity of rapid screening is not the same for false-negative cases as it is for true-positive cases, at least for SIL cases, in which the sensitivity for a falsenegative case of LSIL or HSIL was significantly less than that of a true-positive case. In contrast, the sensitivity of rapid screening for a false-negative diagnosis of ASCUS was at least the same as for a true-positive case, if not higher. How is this difference explained? The lower sensitivity for false-negative SIL cases is easy to understand. False-negative cases of SIL are more difficult to appreciate rapidly because they have fewer abnormal cells that are harder to see than true-positive cases of SIL. On the other hand, why false-negative ASCUS cases are detected at a rate similar to, if not greater than, the rate of true-positive ASCUS cases is less intuitive. Perhaps this reflects more on the nature of ASCUS cases in general rather than any differences between false-negative and true-positive ASCUS cases. After all, the real difference in sensitivity is not so much between falsenegative and true-positive ASCUS cases as it is between true-positive ASCUS and SIL cases. In the present study, more than 90% of true-positive SIL cases were detected in 0 seconds, compared with only 0% of true-positive ASCUS cases. This suggests that unlike SIL cases, in which the diagnosis is often immediately apparent, for a very large percentage of ASCUS cases, the diagnosis takes more time to reach. More than likely, this reflects a difference in the quantity and/or quality of cytologic changes present. The difficulty in and resulting error from rapidly detecting any

5 Anatomic Pathology / ORIGINAL ARTICLE True FNR = (Calculated FNR)/( - FNR Rescreening) True FNR = A/B In this formula, factor A is the number of errors determined by whatever method is used to detect them, and factor B is the correction for the error in the review method used. This error can only be determined by using true-positive cases. Clearly a third factor (factor C) is needed to account for the difference in sensitivity between true-positive cases and false-negative cases. What should that factor be? This depends on the level of accuracy that is desired. Since the majority of false-negative cases consist of ASCUS, and the sensitivity for false-negative and true-positive ASCUS cases is similar, one could argue that a factor C is unnecessary. Alternatively, one could calculate the error rate for ASCUS alone (not ASCUS + SIL but ASCUS alone) by defining factor C as l/(ratio of sensitivity for false-negative ASCUS cases/true-positive ASCUS cases), in this case /.. Similarly, one could calculate the FNR for SIL and use a factor of l/(ratio of sensitivity for falsenegative SIL/True-positive SIL), in this case /0.68. The sum of these provides the total number of errors. This more accurately reflects the true FNR of the laboratory, although calculation is more laborious. We have shown that rapid screening provides reproducible, specific, and sensitive detection of false-negative cases. Compared with routine rescreening of randomly selected cases, rapid prescreening detects more false-negative cases, and the cost per false-negative case detected is less. Rapid prescreening also seems to compare favorably with the AutoPap and PAPNET instruments. Finally, rapid prescreening is the only technique that allows the calculation of an FNR that is controlled for the errors in the method of detection of false-negative cases. For all these reasons, we believe that rapid prescreening of all slides should be considered as an alternative to current methods. From the 'Division of Cytology and the Department of Pathology, Brigham and Women's Hospital and Harvard Medical School, and the Department of Biostatistical Science, Dana Farber Cancer Institute, Boston, Massachusetts. Address reprint requests to Dr Renshaw: Department of Pathology, Brigham and Women's Hospital, 75 Francis St, Boston, MA 05. References. Baker A, Melcher DH. Rapid cervical cytology screening. Cytopathobgy. 99;: Faraker CA. Partial rescreening of all negative smears: an improved method of quality assurance in laboratories undertaking cervical screening. Cytopathology. 99;: Farrell DJ, Bilkhu S, Gibson LM, et al. Rapid screening of cervical smears as a method of internal quality control: for how long should we rescreen? Acta QytoL 997;: Dudding N. Rapid rescreening of cervical smears: an improved method of quality control. Cytopathobgy. 995;6: Johnson SJ, Hair T, Gibson L, et al. An assessment of partial rescreening as an internal quality control method for cervical smears. C)itof>at/io/og> 995;6: AmJCIinPathol 999;: in which the error is so obvious the case is very difficult to defend from a medicolegal standpoint. Nevertheless, in that study, the specificity of the PAPNET system was very low (approximately 75%) and was much lower than that of rapid screening. Finally, the results of rescreening routine negative cases, although performed, was not reported. One important drawback of the AutoPap and PAPNET instruments is that the FNR of the laboratory cannot be determined, at least in the modes in which the instruments are currently used. First, the AutoPap instrument generates rankings rather than diagnoses. Determining the FNR of a ranked slide is not possible. Nevertheless, it would be possible to set a certain rank as the threshold for an abnormal diagnosis. Once this threshold is set, the instrument could determine the number of false-negative cases in a group of slides. To determine the FNR of primary screening, the FNR of the instrument must be determined. As with every other system,7 the only way to determine this is by evaluating abnormal and normal slides, and the sensitivity of neither device in this mode of operation is known. This could be tested and determined. Evaluation of these instruments in a prescreening rather than rescreening mode should be considered to optimize the quality control data that the instruments can generate. What about cost? Assume that a laboratory screens,000 slides, 00 (5%) of which are abnormal with an FNR of 0% (0 missed cases, a total of,90 negative cases in the laboratory). Also assume that the cost of cytotechnologist time is $0/h, or 50 cents per minute. Routine rescreening, rapid prescreening, and the AutoPap device require approximately 0% of slides to be reviewed. If this takes 5 minutes per case, the cost is,90 x 0% x 5 minutes x $0.50/minute = $ Routine rescreening will detect case, so the cost per false-negative case is $ Rapid prescreening also will have the cost of prescreening all cases, or,000 x 0.5 minutes x $0.50/minute = $ Thus, the total cost of rapid prescreening will be $ $500 = $977.50; however, this method will detect approximately 5 false-negative cases, so the cost per case is $ Such a cost analysis has been done before, although without knowledge of the specificity of rapid screening, and similar results have been obtained. These results compare favorably with those obtained with the use of automated systems. Finally, how then should the FNR be calculated? From before,7 the formula for determining the FNR after correcting for the FNR of the rescreening process is:

6 Renshaw et al / PERFORMANCE CHARACTERISTICS OF RAPID PRESCREENING 6. Renshaw AA, DiNisco SA, Minter LJ, et al. A more accurate measure of the false negative rate of Pap smear screening is obtained by determining the false negative rate of the rescreening process. Cancer Cywpathol. 997;8: Renshaw AA. Analysis of error in calculating the false negative rate for interpretation of cervicovaginal smears: the need to review abnormal cases. Cancer Cytopathol. 997;8: Mitchell H. PAPNET differences between false negative and true positive smears [abstract]. Acta Cytol. 997;: O'Sullivan JP, A'Hern RP, Chapman PA, et al. A case-control study of true positive versus false negative smears in women withcin [abstract]. ActaQytoL 997;:5. 0. The Bethesda Committee. The Bethesda system for reporting cervical/vaginal cytologic diagnoses. Acta QytoL 99;7:5-.. Mitchell H, Medley G. Detection of laboratory false negative smears by the PAPNET cytologic screening system. Acta Cytol. 998;: Fowkes FGR. Diagnostic vigilance. Lancet. 986;:9-9-. Wilbur DC, Prey WU, Miller WM, et al. The AutoPap system for primary screening in cervical cytology. Acta C^toi. 998;:-0.. Patten SF, Lee JSC, Wilbur DC, et al. The AutoPap 00 QC system multicenter clinical trials for use in quality control rescreening of cervical smears, II: prospective and archival sensitivity studies. Cancer Cytopathol. 997;8: Patten SF, Lee JSJ, Wilbur DC, et al. The AutoPap 00 QC system multicenter clinical trials for use in quality control rescreening of cervical smears, I: a prospective intended use study. Cancer Cytopathol. 997;8: Mitchell H, Medley G. Detection of unsuspected abnormalities by PAPNET-assisted review. Acta QytoL 998;: Halford J A, Wright RG, Ditchmen EJ. Quality assurance in cervical cytology screening: comparison of rapid rescreening and the Papnet Testing System. Acta Cytol. 997;: Keyhani-Rofagha S, Palma T, O'Toole RV. Automated screening for quality control using PAPNET: a study of 68 negative Pap smears. Diagn Cytopathol. 996;: Jenny ], Isenegger I, Boon ME, et al. Consistency of a double PAPNET scan of cervical smears. Acta Cytol. 997;: Mango L, Valente FT. Neural network-assisted analysis and microscopic rescreening in presumed negative cervical cytologic smears: a comparison. Acta Cytol. 998;:7-.. Hutchinson ML. Assessing the costs and benefits of alternative rescreening strategies. Acta Cytol. 996;0:-8. 5 Am J Clin Pathol 999;:57-5