Performance Evaluation of the CellaVision DM96 System WBC Differentials by Automated Digital Image Analysis Supported by an Artificial Neural Network

Transcription

1 Hematopathology / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY Performance Evaluation of the CellaVision DM96 System WBC Differentials by Automated Digital Image Analysis Supported by an Artificial Neural Network Alexander Kratz, MD, PhD, MPH, 1,2 Hans-Inge Bengtsson, MSc, 3 Jeanne E. Casey, H(ASCP), 1* Joan M. Keefe, MT(ASCP), 1 Gail H. Beatrice, MLT(ASCP), 1 Debera Y. Grzybek, H(ASCP), 1 Kent B. Lewandrowski, MD, 1,2 and Elizabeth M. Van Cott, MD 1,2 Key Words: Image analysis; Differentials; Hematology analyzers; Laboratory automation Abstract We evaluated the CellaVision DM96 (CellaVision AB, Lund, Sweden), an automated digital cell morphology and informatics system for peripheral blood smears. Technologists agreed with 82% of the instrument s preclassifications. Correlation coefficients between final results released from the CellaVision and results obtained by direct microscopy were 0.96 (all neutrophils), 0.94 (lymphocytes), 0.88 (segmented neutrophils), 0.73 (eosinophils), 0.69 (bands), and 0.67 (monocytes). After correction for statistically and clinically insignificant variations, the CellaVision DM96 had 95% sensitivity and 88% specificity for immature myeloid cells. It was 100% sensitive and 94% specific for blasts, and 100% sensitive and 97% specific for unusual WBCs and nucleated RBCs. Advantages of the CellaVision DM96 over direct microscopy include the ability to review slides from a remote location, consultation and quality control on a cell-by-cell basis, and potential labor savings. The differential counting of leukocytes, introduced almost 115 years ago by Ehrlich, 1 has remained a clinically important and frequently ordered laboratory test. 2 Progress in the design of laboratory instrumentation has replaced the microscope with automated cell counters as the instrument of choice for the majority of differential counts (differentials) in most laboratories. 3 Modern automated cell counters can provide a reliable WBC differential count for samples that are normal or that exhibit only a quantitative abnormality. Qualitatively abnormal samples, such as those with immature or abnormal cells, still require the preparation of a slide and microscopic analysis. In our laboratory, approximately 28% of differentials are flagged by the automated cell counter and require the preparation of a blood smear for microscopic review. Difficult differentials with unusual or abnormal cells may be a relatively rare event in many clinical laboratories; nevertheless, the significant clinical impact of a wrong diagnosis necessitates the around-the-clock on-site presence of highly trained personnel for the microscopic review of these blood smears. This increases labor costs, a major issue in today s cost-sensitive health care environment. Other disadvantages of direct microscopic examination of a stained blood smear include interobserver and intraobserver variation and the fact that the individual cells used for the differential count are not easily retrievable for subsequent review. These issues lead to difficulties in quality control and maintaining interobserver and intraobserver consistency in interpretations. Automated image processing systems have been developed to address these difficulties. These systems generally obtain digital images of objects on the blood smear and then use sophisticated software to identify and preclassify cells; the images are stored for possible later retrieval. A number of such 770 Am J Clin Pathol 2005;124: Downloaded 770 from

2 Hematopathology / ORIGINAL ARTICLE devices have been developed (recently reviewed by Tatsumi and Pierre 2 ). Capitalizing on advances in digital photography and data processing and storage, these devices have made it possible to envision a virtual blood film that can be reviewed, shared with experts in distant locations, stored, retrieved, and rereviewed at low cost. 4,5 The latest such automated image analysis system to become commercially available is the CellaVision DM96 (CellaVision AB, Lund, Sweden; distributed in the United States by Sysmex America, Mundelein, IL). The successor model to the DiffMaster Octavia (CellaVision AB), 6 the CellaVision DM96 is an automated image analysis system for peripheral blood smears. Barcode-labeled, stained glass slides are placed into a magazine. The instrument can be loaded with up to 8 magazines, each containing up to 12 slides, and operates with a continuous feed that allows magazines to be added constantly. By scanning the glass slides at low power, the instrument identifies potential WBCs and then takes digital images at high magnification. The images are analyzed by an artificial neural network based on a large database of cells and preclassified according to WBC class. The WBCs and their suggested classification are presented to the user on a customizable computer display for confirmation or reclassification Figure 1. The system also provides functionality for the review of RBC and platelet morphologic features and for estimation of the platelet count. Advantages of this approach over slide review with a microscope include possible labor savings owing to the localization and preclassification of the WBCs by the instrument, more reproducible results, and the ability to review cases from a remote location or to rereview cells at a later date. Direct microscopic review of blood smears is a well-established procedure; clinicians and laboratory workers have extensive experience with this method, are comfortable with it, and have confidence in the results obtained. Introduction of an instrument for automated image analysis, such as the CellaVision DM96, into the clinical laboratory will necessitate a change in work habits, retraining of personnel, and increased spending on hardware and software. For automated image analysis to replace direct microscopic review, 2 conditions will have to be fulfilled: the automated method will have to be shown to be at least as reliable clinically as the direct microscopic review of slides, and there will have to be significant labor savings associated with the new method to justify the additional expenditure associated with the purchase of the equipment. In this study, we compare the clinical performance of the CellaVision DM96 with direct manual microscopy. We also show the results of workflow timing studies comparing direct microscopy with the CellaVision DM96. Materials and Methods Study Site and Personnel The study was performed in the Clinical Hematology Laboratory, Massachusetts General Hospital, Boston, a tertiary care academic medical center serving a large inpatient and outpatient population, including an active hematology-oncology service. All studies were performed by hospital laboratory employees who had been trained in the use of the instrument on-site by a CellaVision company representative. A reliability log for the instrument was kept for the duration of the study. Figure 1 Computer screen on which WBCs preclassified by the CellaVision DM96 are presented to a reviewer for approval or reclassification. The reviewer can enlarge single cells as needed and can leave individual cells in the categories suggested by the instrument (accept the preclassification) or move individual cells into other groups (reclassify the cells). Digital images and the actions taken by the reviewer are stored by the instrument and can be reviewed later for quality control, confirmation of diagnosis, consultation, or comparison with later findings. Sample Selection and Preparation Routine patient samples on which WBC differentials had been ordered by the clinician and on which a microscopic slide review was necessary were used in the study. In our laboratory, the determination that a microscopic slide review is necessary is made when a sample fails criteria for the direct release of results from an ADVIA 120 Hematology System (Bayer HealthCare, Diagnostics Division, Tarrytown, NY). These criteria have been set to conform with the manufacturer s recommendations and the clinical needs of the patient population seen at our institution. All samples that were included in the study had been screened by the ADVIA 120 Downloaded from Am J Clin Pathol 2005;124:

3 Kratz et al / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY Hematology System and flagged for microscopic review. The use of cases prescreened by an automated cell counter was intended to mimic the standard workflow in a hematology laboratory, where usually only cases flagged by the automated cell counter are subject to slide review. Wedge blood smears were prepared manually with the push-pull method with a spreader slide or with a semiautomated method, the Miniprep blood smearing instrument (Geometric Data, Wayne, PA), air dried, and stained with a Sakura Stainer RSG-61 (Sakura Fineteck USA, Torrance, CA) with a Wright-Giemsa stain. 7 Manual Differential Counts (Reference Method) Manual differentials using standard microscopic technique were performed following the laboratory s standard operating procedure, which is based on National Committee for Clinical Laboratory Standards guidelines. 8 Analysis With the CellaVision DM96 After analysis by direct microscopy, slides were labeled with a barcode and loaded into the CellaVision DM96. This instrument scans slides, identifies potential WBCs, takes digital images of them, and uses artificial neural network based software to analyze the cells. Digital images of preclassified cells are presented to the technologist on a computer display (Figure 1). The technologist is asked to review the images and to accept the preclassification provided by the software or to reclassify cells. For this study, technologists using the CellaVision DM96 were provided with the same information about the cases as was available to them during direct microscopic analysis. This included the results from the automated cell counter on the present sample and previous CBC counts, WBC differentials, and clinical results for the patient, if available. Technologists also were asked to evaluate whether the areas presented by the CellaVision DM96 for the evaluation of RBC and platelet morphologic features and platelet count were adequate. Studies Performed Three sets of studies were performed: a clinical performance evaluation, a manual differential timing study, and a standard workflow timing study. The clinical performance evaluation was intended to determine the performance of the CellaVision DM96 in difficult cases; for this study, cases that met certain criteria (eg, low WBC count, unusual WBCs) were obtained during a 2-month period and used for this study. For the manual differential timing study, 10 cases of average difficulty were selected. Last, for the standard workflow timing studies, a random sample of cases for which slides were prepared in the laboratory during the study period was used. Clinical Performance Evaluation We selected 120 slides with abnormal blood smear findings thought to be difficult cases for evaluation of the clinical performance of the CellaVision DM96. These cases included slides with increased numbers of immature WBCs and the presence of abnormal WBCs and some cases with very low WBC counts. All slides had required a full microscopic differential count. Three laboratory employees (2 certified laboratory technologists [J.E.C. and J.M.K.] and 1 certified medical laboratory technician [G.H.B.]) were asked to participate in this part of the study; they were selected for representing the range of expertise in the performance of manual differentials found in the laboratory. Expertise of employees was determined by their level of formal training (certified technologists vs certified technicians), the length of their work experience (between 1.5 and 35 years), and whether the employee was training new employees in the performance of WBC differentials and was a designated resource person in the laboratory for difficult WBC differential cases. Each employee was asked to perform manual differentials on 40 slides with the microscope and approximately 1 week later on the same 40 slides with the CellaVision DM96. Manual Differential Timing Studies Ten slides that had required full microscopic differential count were used for a manual differential timing study. Five technologists and a technician, representing the range of expertise in the performance of manual differentials found in the laboratory, were timed while performing manual differentials on these slides with the microscope and then approximately 1 week later while performing the same task with the CellaVision DM96. Cases were considered completed on finalization of the results in the laboratory information system or on conclusion of analysis with the CellaVision DM96. Standard Workflow Timing Studies We used 236 additional randomly selected cases on which a manual slide review was necessary for a workflow study. Approximately half of these slides required the performance of a full microscopic differential count; the other half required only microscopic review ( scan ) for confirmation of the results of the differential obtained from the automated cell counter. These cases first were analyzed by a medical laboratory technician (G.H.B.) with 1.5 years of experience in reading blood smears (n = 160 cases) or a very experienced technologist (J.E.C.) with 35 years of experience (n = 76 cases) with the direct microscopic method as part of their normal work assignments in the clinical laboratory. The employees then were asked to reanalyze the same samples with the CellaVision DM96. These employees also had participated in the other studies. Statistical Analysis Statistical analysis was performed using Microsoft Excel software (Microsoft, Redmond, WA). Two-tailed paired t tests were used to evaluate differences between the time needed for 772 Am J Clin Pathol 2005;124: Downloaded 772 from

4 Hematopathology / ORIGINAL ARTICLE fully manual differentials with the CellaVision DM96 and with direct microscopy. Clinical sensitivity and specificity of the CellaVision DM96 were defined as its ability to obtain positive and negative results concordant with the results obtained by direct microscopy. Results System Reliability The CellaVision DM96 was in use in our laboratory for 2 weeks. During this time, a critical oil error required 5 minutes for resolution by priming the immersion oil pump to remove air bubbles. There were 4 rack jams; each of these incidents required approximately 1 or 2 minutes to resolve; 2 of the jams were caused by trying to load slides not appropriate for the instrument. There were 3 cases of software failure that required the viewing program to be restarted, leading to the loss of approximately 1 minute per incident. In total, approximately 12 to 20 minutes were lost in these 8 downtime incidents during 2 weeks. Ability of the CellaVision DM96 to Present to the Reviewer WBCs That Are Qualitatively and Quantitatively Adequate for a Differential Count We used 120 cases that required a full microscopic differential to evaluate the clinical performance of the CellaVision DM96. A laboratory employee first performed a differential on these cases using the microscope; the same person then performed a differential on the same slides using the CellaVision DM96. The slide readers were asked to note when they thought that the images presented by the CellaVision DM96 were of insufficient quality to allow them to perform a differential. Such a notation was made in 11 (9.2%) of the cases. For 10 of these 11 cases, the same technologists had been able to do a full 100-cell differential with the microscope; in the remaining case, the microscopic differential also was inadequate (it was based on 5 WBCs). Of the 11 slides, 10 had been prepared with the manual push-pull method with a spreader slide; 1 had been prepared with a semiautomated method, the Miniprep blood smearing instrument Table 1. These 11 cases in which no differential was available from the CellaVision DM96 were excluded from subsequent correlation analysis. The precision of a WBC differential is affected negatively if only a few cells are counted. 9,10 We compared the total number of cells found by the technologists when performing WBC differentials using the microscope with the number of adequate WBCs presented for review for each case by the CellaVision DM96 Figure 2. The ability of both methods to find identifiable cells on a blood smear was similar, with a slightly higher number of cases with very few (< 20) WBCs encountered with Table 1 Adequacy of Images From the CellaVision DM96 for WBC and RBC Analysis According to Slide Preparation Method * Manual Method Semiautomated Images Inadequate for (n = 90) Method (n = 30) WBC analysis 10 (11) 1 (3) RBC analysis 18 (20) 10 (33) * The manual method refers to slides prepared with the push-pull method with a spreader slide; the semiautomated method refers to slides prepared with the Miniprep slide preparation instrument. A total of 120 slides were analyzed. Numbers represent the total number of slides. Data are given as number (percentage). the microscopic method (Figure 2). There were 11 cases (9.2%) with fewer than 20 cells counted when using the manual method; the CellaVision DM96 differential was based on fewer than 20 cells in 7 cases (5.8%). Among the 11 cases in which the manual differential was based on fewer than 20 cells, in 5 cases the CellaVision DM96 was able to find more than 20 cells per slide. There were 2 cases in which the CellaVision DM96 differential had to be based on fewer than 20 cells per slide, whereas the manual differential was based on more than 20 cells per slide. Because of the limited reproducibility of differential results based on such low counts, the 13 cases in which the manual differential, the CellaVision DM96 differential, or both were based on fewer than 20 cells per slide were excluded from subsequent analysis. Ability of the CellaVision DM96 to Provide Images Adequate for Evaluation of RBC and Platelet Morphologic Features and the Platelet Count The CellaVision DM96 allows the user to evaluate RBC and platelet morphologic features and to estimate the platelet No. of Samples < >100 Counted Cells Figure 2 Total number of cells used by technologists for WBC differential counts when using the microscope (white bars) and the CellaVision DM96 (black bars). Only objects classified as WBCs by the technologist were included in the analysis. Downloaded from Am J Clin Pathol 2005;124:

5 Kratz et al / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY count by producing an overview image consisting of 35 patched areas of the slide. We asked technologists to indicate whether they thought that these images were adequate for the evaluation of RBCs and platelets. In 28 cases (23%), the technologists thought the images did not allow them to adequately evaluate RBC and platelet morphologic features. The use of a semiautomated slide preparation method was associated with a higher percentage of inadequate slides (33%) than the manual push-pull method (20%) (Table 1). When reviewing the same slides with the microscope, the technologists thought in 100% of cases that they were able to find areas allowing them to adequately address RBC and platelet morphologic features and to estimate the platelet count. Correlation of the CellaVision DM96 Preliminary Classifications With Final Classifications by Medical Technologists and Technicians The CellaVision DM96 presents, on a computer display, WBCs preclassified according to cell type. The technologist has to review every cell and can move a cell to a different cell class (reclassify the cell) or can leave the cell in the category suggested by the instrument by taking no additional action regarding the cell. Table 2 shows the percentages of the various cell types preclassified correctly by the CellaVision DM96. Overall, the instrument preclassified 82% of all cells correctly (86% if band forms were not differentiated from segmented neutrophils). The percentages of correctly preclassified cells were highest for mature cells and lowest for immature and abnormal cells. Correlation of Direct Microscopic WBC Differential Results With Final Results From the CellaVision DM96 After review and, if necessary, reclassification of the cells presented by the CellaVision DM96, a reviewer releases the final differential results to the laboratory information system. Differential results obtained by direct microscopy were compared with the final results released by the same technologist for the same slide from the CellaVision DM96. The correlation graphs for neutrophils (including segmented neutrophils and band forms), lymphocytes, monocytes, and eosinophils are shown in Figure 3. The correlation coefficients were highest for total neutrophils (0.96), lymphocytes (0.94), and segmented neutrophils (0.88). For eosinophils, the correlation coefficient was The lowest correlations were observed for band forms (0.69) and monocytes (0.67). Ability of Digital Image Analysis by CellaVision DM96 to Identify Clinically Important Abnormalities In clinical practice, the ability to identify the presence of an abnormality on a blood smear, for example, the presence of blasts, is an important determinant of the reliability of a differential counting method. We compared the ability of the Table 2 Percentages of Cells Correctly Preclassified by the CellaVision DM96 * Correct All Verified Results Suggestions by Suggested Correctly Cell Class CellaVision (%) by CellaVision (%) Segmented neutrophils (n = 3,510) Band neutrophils (n = 868) Lymphocytes (n = 2,585) Monocytes (n = 763) Eosinophils (n = 231) Basophils (n = 50) Blasts (n = 395) Immature myeloid cells (n = 627) Nucleated RBCs (n = 165) * The percentage of correct suggestions is the percentage of the cells in the various groups, as preclassified by the CellaVision DM96, in which the technologist agreed with the instrument s preclassification (ie, the percentages of original suggestions that were correct); the percentage of all verified results suggested correctly is the percentage of cells in each group released by the technologist that were preclassified correctly by the CellaVision (ie, percentages of final results preclassified correctly by the CellaVision). A total of 9,194 cells were analyzed. CellaVision DM96 to identify the presence of several clinically important abnormalities with the results of direct microscopy. These abnormalities included the presence of immature and abnormal WBCs and of nucleated RBCs. As shown in Table 3, a significant number of cases showed an abnormality on review with the CellaVision DM96 that were not reported on direct microscopy; some cases did not show an abnormality on review with the CellaVision DM96 that were reported on direct microscopic review. When direct microscopy was used as the reference method ( gold standard ) against which the results of the CellaVision DM96 were compared and when all discrepant results were included in the analysis, the specificity of the CellaVision was 82% to 93%; sensitivity was 25% to 91% Table 4. To resolve the significant percentage of false-positive and false-negative findings of the CellaVision DM96 compared with direct microscopy, digital images from the CellaVision DM96 and microscopic differentials were rereviewed by the medical director of the laboratory (A.K.) for all cases with discrepant results. As shown in Table 5, Table 6, and Table 7, most of the discrepant cases were due to small variations within the 95% confidence interval of the cell count (eg, most cases with a discrepancy involving nucleated RBCs showed 1 nucleated RBC/100 WBCs with one method and none with the other method). Some other discrepancies were due to a slight difference in nomenclature without clinical relevance (eg, the same technologist would call cells blasts with one method and use the expression other cells; immature myeloid cells, query blasts with the other method). In 15 of the discrepant cases, rereview of the digital images showed that the discrepancy was due to 774 Am J Clin Pathol 2005;124: Downloaded 774 from

6 Hematopathology / ORIGINAL ARTICLE A B 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% C 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% D 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% E 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% F 70% 60% 50% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 25% 20% 15% 10% 5% 0% 0% 5% 10% 15% 20% 25% Figure 3 Correlation of neutrophil (A), segmented neutrophil (B), band form (C), lymphocyte (D), monocyte (E), and eosinophil (F) counts on direct microscopy and on the CellaVision DM96. A, y = x ; R 2 = B, y = x ; R 2 = C, y = 1.188x ; R 2 = D, y = x ; R 2 = E, y = x ; R 2 = F, y = x ; R 2 = misidentification of cells by the initial reviewer. There were only 3 cases in which the discrepancy could not be resolved by statistical variation, differences in nomenclature, or misclassification of digital images presented by the CellaVision DM96; in these cases, there were clearly cells consistent with immature myeloid cells (promyelocytes, metamyelocytes, and/or myelocytes) present on the CellaVision DM96 images that had not been reported on direct microscopic review. A repeated standard microscopic differential also did not show these immature myeloid cells on the slides. When a third direct microscopic differential based on up to 300 cells was performed, populations of immature myeloid cells (between 2.3% and 8%) were identified. As noted in the preceding paragraph, most of the discrepancies were insignificant statistically (within the 95% confidence Downloaded from Am J Clin Pathol 2005;124:

7 Kratz et al / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY interval of the cell count) or clinically (slightly different nomenclature without clinical relevance). We therefore reanalyzed the data and omitted these cases. The results of the reanalysis are given in Table 8. Sensitivities for the various abnormalities now were between 95% and 100%; specificities varied between 88% and 97%. Timing Studies on Fully Manual Differentials An important issue in the evaluation of a new technology is its cost-effectiveness. For an instrument like the CellaVision DM96, a major determinant of cost-effectiveness is the amount of time it takes for a technologist to perform a manual differential. To compare the time needed to perform fully manual differentials on the CellaVision DM96 with direct microscopy, 10 slides that required a full manual differential (as opposed to a review of the slide and release of the automated differential from the cell counter) were analyzed by 6 technologists with a microscope and with the CellaVision DM96. It took the technologists, on average, 1.3 minutes longer per manual differential when they were using the CellaVision DM96 than when using the standard microscopic method (P <.01) Table 9. Standard Workflow Timing Studies The timing studies using fully manual differentials do not adequately reflect the usual workload of the differential workstation of a clinical hematology laboratory. At our institution, only approximately 50% of specimens that are flagged by the automated cell counter and have a slide prepared require a full manual differential. The other 50% of the slides are reviewed (scanned) by the technologist, and the WBC differential results of the automated cell counter are released, without need for a full manual differential. To capture this aspect of the normal laboratory workflow, 2 technologists (J.E.C. and G.H.B.) were timed on randomly selected slides. These slides consisted of cases that required a full manual differential and of scanned slides that were reviewed only by the technologist. The 2 technologists selected for this study were a senior technologist with a special interest in manual differentials who is the designated trainer for the differential workstation for new employees and an employee who had joined the laboratory more recently and who had no special expertise in manual differentials. By this Table 3 Identification of the Presence of Abnormal Findings on a Blood Smear * time, these 2 reviewers had received more extensive training in the use of the CellaVision DM96 than the technologists participating in the timing study involving only manual differentials. All slides used for workflow timing were prepared with the Miniprep blood smearing instrument. As shown in Table 10, there was no significant difference in the average time per slide for the very experienced technologist. The technician with no special expertise in WBC differentials, in contrast, was 25% faster using the CellaVision DM96 than using the manual method. The average time per slide for the first 77 slides that the technician analyzed with the CellaVision DM96 was 3.1 minutes; for the last 83 slides, her average time per slide using the CellaVision DM96 was 2.8 minutes. The reviewers thought that in all 236 cases, the images presented by the CellaVision DM96 for the WBC differential were adequate. Discussion CellaVision DM96 Manual Microscopy Positive Negative >3% Promyelocytes, myelocytes, and/or metamyelocytes Positive 21 2 Negative Blasts Positive 12 2 Negative 7 75 Other unusual WBCs, eg, promonocytes, plasma cells, prolymphocytes Positive 1 3 Negative 6 86 Nucleated RBCs Positive 9 5 Negative * Based on the analysis of 96 blood smears. Data are given as the number of blood smears positive or negative by the respective method. The CellaVision DM96 is an automated digital image analysis system. Routinely prepared peripheral blood smears are scanned at low power, WBCs are identified, and digital Table 4 Clinical Performance of the CellaVision DM96 Compared With Direct Microscopy as the Reference Method, Including All Discrepant Cases Cells Identified on a Slide Specificity (%) Sensitivity (%) False-Positive (%) False-Negative (%) >3% Promyelocytes, myelocytes, and/or metamyelocytes Blasts Other unusual WBCs Nucleated RBCs Am J Clin Pathol 2005;124: Downloaded 776 from

8 Hematopathology / ORIGINAL ARTICLE Table 5 Rereview of Cases With Positive Findings by the CellaVision DM96 and Negative Findings by Direct Microscopy (False-Positives on the CellaVision DM96) * Case No. Resolution Category Presence of immature myeloid cells (>3% promyelocytes, myelocytes, and/or metamyelocytes) on CellaVision DM96 differential and not on manual differential 004 Misclassification of CellaVision images by technologist % immature myeloid cells present on CellaVision differential; 2% immature myeloid cells present on microscopic 1 differential 019 Original review and rereview of CellaVision images showed presence of 17% immature myeloid cells; original 4 microscopic review and first rereview showed no immature myeloid cells; third microscopic differential based on 200 cells showed 6.5% immature myeloid cells 039 Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist Original review and rereview of CellaVision images showed presence of 4% immature myeloid cells; original 1 microscopic review showed 0% immature myeloid cells; microscopic rereview of the slide showed 2% immature myeloid cells 098 Misclassification of CellaVision images by technologist Original review and rereview of CellaVision images showed presence of 13% immature myeloid cells; original 4 microscopic review and first rereview showed no immature myeloid cells; third microscopic differential showed 8% immature myeloid cells 118 Cells classified as promyelocytes by CellaVision classified as Others, immature myeloid cells on the microscopic 2 differential 119 Original review and rereview of CellaVision images showed presence of 4% immature myeloid cells; original 1 microscopic review and rereview showed no immature myeloid cells 120 Original review and rereview of CellaVision images showed presence of 8% immature myeloid cells; original 4 microscopic review and first rereview showed no immature myeloid cells; third microscopic differential based on 300 cells showed 2.3% immature myeloid cells Presence of blasts on CellaVision DM96 differential and not on manual differential 013 Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist Original review and rereview of CellaVision images showed presence of 1% blasts; microscopic review and 1 rereview showed no blasts 083 Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist Blasts had been classified as Other, immature myeloid cells on manual differential 2 Presence of unusual cells on CellaVision DM96 differential and not on manual differential 046 Misclassification of CellaVision images by technologist Original review and rereview of CellaVision DM96 images showed presence of 1% plasma cells; microscopic 1 review and rereview showed no plasma cells 094 Original review and rereview of CellaVision images showed 1% prolymphocytes; microscopic review and rereview 1 showed no prolymphocytes 101 Misclassification of CellaVision images by technologist Cells classified as Other, immature myeloid cells on CellaVision classified as blasts on microscopic review Misclassification of CellaVision images by technologist 3 Presence of nucleated RBCs (nrbcs) on CellaVision DM96 differential and not on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential Misclassification of CellaVision images by technologist Misclassification of CellaVision images by technologist nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbc/100 WBCs on CellaVision; no nrbcs on manual differential nrbcs/100 WBCs on CellaVision; no nrbcs on manual differential 1 * Category 1, the discrepancy could be explained by random variation (ie, results of both methods were within the 95% confidence interval of the number of cells counted); category 2, cases with essentially identical diagnoses (eg, the technologist used the expression blasts with one method and language such as other cells; immature myeloid cells, query blasts with the other); category 3, rereview of the digital images obtained by the CellaVision indicated a misclassification by the technologist of cells shown on the images; category 4, an abnormality identified by one method was not identified with the other and the discrepancy could not be explained by random variation within the 95% confidence interval, differences in nomenclature, or misclassification of cells on digital images. Analysis was performed twice for all cases listed (once as part of the original analysis and then as rereview of cases with discrepant findings between direct microscopy and the CellaVision). As described in the Results section, a third microscopic review then was performed on cases 019, 104, and 120 to resolve the remaining discrepancy. Downloaded from Am J Clin Pathol 2005;124:

9 Kratz et al / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY Table 6 Rereview of Cases With Negative Findings by the CellaVision DM96 and Positive Findings by Direct Microscopy (False-Negatives on the CellaVision DM96) * Case No. Resolution Category Presence of immature myeloid cells (>3% promyelocytes, myelocytes, and/or metamyelocytes) on microscopic differential and not on CellaVision DM96 differential 046 3% immature myeloid cells present on CellaVision; 4% immature myeloid cells present on microscopic 1 differential 052 Misclassification of CellaVision images by technologist 3 Presence of blasts on microscopic differential and not on CellaVision DM96 differential 052 No blasts seen on CellaVision; 2% blasts seen on microscopic differential Blasts classified as Others (immature myeloid cells) on microscopic differential 2 Presence of unusual cells on microscopic differential and not on CellaVision DM96 differential 089 Others (immature myeloid cells) called blasts on the CellaVision Review and rereview of CellaVision differential showed no unusual cells; 1% prolymphocytes seen on 1 microscopic differential 108 Others (immature myeloid cells) called blasts on the CellaVision 2 Presence of nucleated RBCs (nrbcs) on microscopic differential but not on CellaVision DM96 differential 094 No nrbcs seen on CellaVision, 1 nrbc/100 WBCs seen on microscopic differential No nrbcs seen on CellaVision, 1 nrbc/100 WBCs seen on microscopic differential No nrbcs seen on CellaVision, 2 nrbcs/100 WBCs seen on microscopic differential No nrbcs seen on CellaVision, 1 nrbc/100 WBCs seen on microscopic differential No nrbcs seen on CellaVision, 1 nrbc/100 WBCs seen on microscopic differential 1 * See Table 5 for an explanation of the categories. images of these cells are taken at high magnification. An artificial neural network analyzes the pictures and preclassifies them according to WBC class. Preclassified WBCs are presented to a reviewer for confirmation or reclassification (Figure 1). We compared the clinical performance of this system with the standard direct microscopic method. In 11 (9.2%) of 120 cases, the technologists thought the cells presented by the CellaVision DM96 were inadequate for a reliable WBC differential. Such a high rejection rate would seriously interfere with the ability of a clinical laboratory to report differentials in a timely manner; however, 10 of these 11 cases had been prepared with the manual push-pull method with a spreader slide; only 1 of the cases had been prepared with a semiautomated slide maker. In the standard workflow timing studies, the technologists were asked to use the CellaVision DM96 to review 236 slides prepared with the semiautomated method. They thought that for all cases the images presented for the WBC differential were adequate. This indicates that the use of an automated or semiautomated slide maker significantly Table 7 Summary of Rereview of Cases With Differing Results on Manual Microscopy vs CellaVision DM96 * Category CellaVision false-positive cases Immature myeloid cells Blasts Unusual cells Nucleated RBCs CellaVision false-negative cases Immature myeloid cells Blasts Unusual cells Nucleated RBCs * Data are given as number of cases. See Table 5 for an explanation of the categories. decreases the number of cases with inadequate digital images presented by the CellaVision DM96 and allows the performance of the overwhelming majority of manual differentials using the CellaVision DM96. Table 8 Clinical Performance of the CellaVision DM96 Compared With Direct Microscopy as the Reference Method, Excluding Discrepancies Due to Statistically Insignificant Variations (Within the 95% Confidence Interval of the Cell Count) and to Clinically Insignificant Variations in Nomenclature Cells Identified on a Slide Specificity (%) Sensitivity (%) False-Positive (%) False-Negative (%) >3% Promyelocytes, myelocytes, and/or metamyelocytes Blasts Other unusual WBCs Nucleated RBCs Am J Clin Pathol 2005;124: Downloaded 778 from

10 Hematopathology / ORIGINAL ARTICLE Table 9 Average Time per Slide for 6 Technologists Performing 10 Fully Manual Differential Counts With the Microscope or the CellaVision DM96 * Technologist Microscope CellaVision A B C D E F Average for all technologists * Times are given in minutes. It is possible that in laboratories that decide to maintain the manual push-pull method for slide preparation, feedback provided to technologists will improve the quality of smear preparation. The importance of the enforcement of rigorous quality standards for slide preparation was recently stressed by Sandhaus and coworkers. 11 Among cases with an appropriate smear quality, the ability of the CellaVision DM96 to find a sufficient number of WBCs suitable for a reliable differential was at least equivalent if not better than the direct microscopic method (Figure 2). The number of cases in which the instrument provided images inadequate for RBC and platelet analysis was high (23%) and did not improve with the use of a semiautomated slide maker. By using the microscope, the reviewers were able to find areas appropriate for the evaluation of RBC and platelet morphologic features on 100% of these slides. It is possible that the percentage of slides with inadequate RBC areas presented by the CellaVision DM96 can be decreased by better slide preparation, better training of technologists in the use of the CellaVision DM96 for this purpose, and/or adjustments in the CellaVision DM96. Otherwise, laboratories using the CellaVision DM96 may have to decide not to provide information on RBC and platelet morphologic features as part of their standard manual differentials, unless there is reason to suspect an RBC or a platelet abnormality (ie, an RBC- or a plateletspecific flag from the automated cell counter). Such a development would be in line with the fact that already, neither RBC nor platelet morphologic features are reviewed microscopically for differential samples that are reported without preparation of a slide directly from the automated cell counter. The CellaVision DM96 presents preclassified WBCs to a reviewer for approval or reclassification. We found that the instrument overall correctly preclassified 82% of all cells. Preclassification of WBCs was most accurate for segmented neutrophils (92.5% correctly classified), lymphocytes (96.4%), and monocytes (81.4%); it was less reproducible for basophils (80.0%) and for eosinophils (63.2%) (Table 2). A study based on images of WBCs sent to 2,400 laboratories in surveys of the College of American Pathologists and classified by human observers was published in ; 99% of segmented neutrophils, 96% of lymphocytes, 87% of monocytes, 95% of basophils, and 96% of eosinophils were identified correctly by the participants. This indicates that the ability of the CellaVision DM96 to preclassify segmented neutrophils, lymphocytes, and monocytes is similar to that of the human observer, whereas improvements to the neural network are needed for eosinophils and basophils. After the WBCs presented by the instrument are reclassified or confirmed by the user, final results are released. We compared the final results obtained with digital image analysis to differentials from direct microscopy. The correlation coefficients for differential counts for the WBC classes found in blood samples with a qualitatively normal differential were between 0.67 and 0.96 (Figure 3). Koepke and colleagues 13 evaluated the performance of differential WBC counting by 73 technologists and technicians in 5 different laboratories in a large medical center by comparing them with differentials performed by referees. Correlation coefficients were lowest for basophils (0.32) and monocytes (0.41) and highest for lymphocytes (0.73), eosinophils (0.83), and segmented neutrophils (0.87). The correlation coefficients for the cell counts from the CellaVision DM96 compare quite favorably with these results. Correlation for the various cell classes with the direct microscopic differential also was very similar to the findings reported by Swolin and colleagues 6 for the DiffMaster Octavia, the predecessor of the CellaVision DM96. The low correlation for band forms might be due to the known high variability in band counts. 14 There was a significant percentage of cases that were false-positive or false-negative for clinically significant abnormalities on the CellaVision DM96 compared with the results of direct microscopy. Similar findings were reported by Koepke and coworkers, 13 who found that the sensitivity of Table 10 Average Time per Slide Using the Standard Workload of the Differential Workstation (Approximately 50% Fully Manual Differential Counts and 50% Scanned Slides With Release of the Automated Differential Count) Staff Member/No. of Slides Microscope CellaVision DM96 Very experienced technologist with special expertise in WBC differentials (n = 76) Technician with no special expertise in WBC differentials (n = 160) Downloaded from Am J Clin Pathol 2005;124:

11 Kratz et al / DIGITAL IMAGE ANALYSIS IN HEMATOLOGY slide review by technologists ranged from 34% to 100%, depending on the abnormality. Review of the discrepant cases in our study showed that the majority of discrepant cases were due to variation that was not statistically significant (within the 95% confidence interval of the number of cells on which the differential was based) or to clinically insignificant variations in nomenclature (eg, blasts vs other cells, query blasts). When we corrected for these statistically or clinically insignificant discrepancies, the sensitivity of the CellaVision DM96 was between 95% and 100% and the specificity between 88% and 97%, depending on the abnormality. The remaining discrepancies were mostly (16/19 discrepant cases) due to misidentification of cells on digital images by the technologists. In the remaining 3 discrepant cases, cells that were clearly present on the digital images obtained with the CellaVision DM96 had not been reported on direct microscopy. A rereview of these slides by experienced reviewers at first did not allow the visualization of these cells with the microscope. When an additional microscopic differential based on up to 300 cells was performed on these slides, the immature myeloid cells seen with the CellaVision DM96 were seen with the microscope. Therefore, these discrepant cases were most likely due to random variation outside the 95% confidence interval of a 100-cell differential count. As described in the preceding paragraphs, when we found discrepancies between the findings with direct microscopy and with the CellaVision DM96, we rereviewed the digital images provided by the CellaVision DM96 and determined that in a number of cases, the initial reviewers had misidentified cells. This episode underlines a strength of digital image analysis compared with direct microscopy: we were able to do quality control on a cell-by-cell basis, identify errors, correct them, and perform corrective action by educating the technologists who had made the misidentification. Such interventions are much more difficult or impossible using direct microscopy. Even after correction for misclassified cells, there were more positive findings on the CellaVision DM96 than on direct microscopy. Most of these false-positive findings involved very small populations of cells (<5%). Since the presence of abnormal WBCs will only be recognized with greater than 95% certainty in a 100-cell differential if the percentage of the abnormal cell population is at least 5%, random variation is the most likely explanation for these false positives. 15 Another possibility is that some cells were misclassified or missed on direct microscopy. Finally, it is very plausible that some of the positive findings identified with the CellaVision DM96 and not reported on direct microscopy were due to the fact that it is not possible for the technologist to skip cells on the CellaVision DM96. Every cell has to be classified (and this classification remains subject to rereview), leading to a higher likelihood of reporting rare or unusual cells when using the CellaVision DM96. To evaluate the impact of the CellaVision DM96 on labor costs, we performed 2 studies. First, we timed 6 technologists performing 10 fully manual differentials with the microscope and with the CellaVision DM96. It took the technologists, on average, 5.1 minutes per slide to perform a manual differential; the time required with the CellaVision DM96 was longer, 6.4 minutes. The limitations of this study are that the technologists had years of experience in performing differentials with a microscope; in contrast, the CellaVision DM96 was new to them, and they lacked any prolonged experience on this instrument. It therefore is possible that with increased exposure to the CellaVision DM96, the average time per manual differential would have decreased. The second limitation of this study was that it addressed only fully manual differentials; the usual workflow at the differential workstation in our laboratory consists of approximately 50% manual differentials and 50% slides that only need to be reviewed for release of the differential results from the automated cell counter (scanned slides). It is possible that the major labor saving that can be derived from the CellaVision DM96 will be from these scanned slides, for which the technologist is conveniently presented with a large number of WBCs on a computer screen that can be reviewed quickly for the presence of any abnormality, with subsequent release of the differential results from the automated cell counter. To address these possibilities, we performed a second timing study, in which 2 employees were trained more extensively in the use of the CellaVision DM96. They were asked to result randomly selected cases with the microscope and to process the same cases with the CellaVision DM96. These cases consisted of both fully manual differentials and of slides that required only a scan. As shown in Table 10, it took a very experienced technologist approximately the same time per slide to result differentials from the CellaVision DM96 (1.5 minutes per slide) as from the microscope (1.4 minutes per slide). However, a less experienced employee was faster using the CellaVision DM96 (3.0 minutes per case) than using the microscope (4.1 minutes per case), indicating that after training on the CellaVision DM96 and with a caseload similar to the normal workflow in the laboratory, an average technologist is likely to be more productive using the CellaVision DM96 than using the microscope. The possibility of a learning curve was also indicated by the fact that the less experienced employee was 10% faster using the CellaVision DM96 when analyzing the second half of her assigned slides than when analyzing the first half. The cost-effectiveness of the CellaVision DM96 will be determined by a variety of factors. Our data indicate that the instrument may allow some labor savings by decreasing the time for technologists trained in the use of the instrument to release a differential. Increased efficiencies also could be achieved by centralizing the expertise for morphologic diagnosis and by making it less time-consuming to obtain confirmation of abnormal findings by referees. These cost savings 780 Am J Clin Pathol 2005;124: Downloaded 780 from