Accuracy and precision of computer-aided sperm analysis in multicenter studies*t

Transcription

1 FERTILITY AND STERILITY Copyright 1992 The American Fertility Society Printed on acid-free paper in U.S.A. Accuracy and precision of computer-aided sperm analysis in multicenter studies*t Russell O. Davis, Ph.D.:j: Susan A. Rothmann, Ph.D. II James W. Overstreet, M.D., Ph.D.:j: University of California-Davis, Davis California, and The Cleveland Clinic Foundation, Cleveland, Ohio Study Objectives: To develop methods of instrument calibration and standardization that enable the use of computer-aided sperm analysis (CASA) technology in multicenter studies of sperm motility. Setting: Clinical semen evaluation for the andrology laboratory and in vitro fertilization programs. Patients: Semen specimens were selected from the videotape archives of the University of California at Davis Andrology Laboratory and used to produce a videotape reference standard for CASA instrument testing and calibration. Four identical CASA instruments, two each at two sites, were used to analyze the videotape. Machine accuracy was verified by manual analysis of the videotape. Results: The coefficient of variation (CV) for repeated measures was between 1% and 8% for each variable on all CASA instruments. Mean values for CASA parameters varied as a function of the digitization threshold. A minimum CV was obtained at a particular threshold setting, suggesting an optimum for each machine. Because of high within-machine precision, slight differences in mean values between machines were statistically significant for some variables and some specimens; such differences are probably not biologically significant. Conclusion: Multicenter standardization of CASA instrumentation is possible, but threshold settings are one source of variation that must be controlled using CV s or another means of objective calibration. One method is to adjust the threshold of each instrument until the count obtained by CASA equals the count obtained by the manual method used to determine the dilution requirements for neat semen. Fertil Steril 1992;57: Key Words: Computer-aided sperm analysis, quality control, semen analysis Results from manual semen analysis are not often correlated with fertility (1). Reasons may include: subjective assessment of semen parameters, use of Received May 22, 1991; revised and accepted November 25, * Presented at the Annual Meeting of the American Society of Andrology, Montreal, Quebec, Canada, April 27 to 30, t Supported in part by grants ROI-HD25907 and R01-ES03614 from the National Institutes of Health, Bethesda, Maryland. :j: Department of Obstetrics and Gynecology, Division of Reproductive Biology and Medicine, School of Medicine, University of California -Davis. Reprint requests: Russell O. Davis, Ph.D., Division of Reproductive Biology and Medicine, University of California-Davis, Davis, California II Department of Laboratory Medicine, The Cleveland Clinic Foundation. inaccurate or inappropriate semen measures, and inadequate definition and control of study populations (2). Computer-aided sperm analysis (CASA) provides an objective and automated alternative to the traditional approach, but its results can be affected by system parameter settings (3), threshold settings (4), video frame rate (5), and other variables. Because of these difficulties, as well as the lack of standardization for CASA instruments between centers and across brands, it has been difficult to develop adequate quality-control programs for this important, new laboratory technology. With increasing interest in multicenter studies of fertility, it is essential that methods be developed for instrument standardization and calibration and that such methods be incorporated into ongoing laboratory quality-control programs. In this study, methods 648 Davis et ai. Use of CASA in multicenter studies Fertility and Sterility

2 were developed to determine the accuracy and precision of multiple CASA instruments at two sites: The Cleveland Clinic Foundation, Cleveland, Ohio, and The University of California -Davis, Davis, California. MATERIALS AND METHODS Four semen specimens from the University of California Davis Andrology Clinic were used to make a videotape reference standard for CASA instrument testing and calibration. An aliquot from each specimen was diluted 1:1 in a fixative solution and a 7-tIL drop from each aliquot was placed in a 20-tIm deep MicroCell counting chamber (Fertility Technologies' Inc., Natick, MA) and allowed to settle. The sperm concentration was determined using an Olympus BH2-S phase-contrast microscope (Olympus Corp, New York, NY) with a 20X positive phase S-plan objective lens and a 5 X 5, 250-tIm square eyepiece reticle. Based on this manual assessment, specimens were diluted to a concentration of 20 to 50 X 10 6 sperm/ml in warm Dulbecco's phosphatebuffered saline before videotaping. All specimens were videotaped using a lox positive phase S-plan objective lens, a 6.7X photo-ocular, an MTV-3 adaptor, a COHU 6,400 series video camera (Cohu Inc., Electronics Division, San Diego, CA), and a Panasonic AG-1960 Super VHS video recorder (Panasonic Communications and Systems Co., Secaucaus, NJ) operated in Super-VHS mode. An Olympus IF -550 neutral density interference contrast filter was placed over the field diaphragm of the microscope to produce the optimum spectrum for the camera (i.e., green). The microscope light intensity was adjusted for each specimen before recording using a video oscilloscope to obtain the optimum video pedestal (i.e., maximum picture gain without clipping the image). The Super VHS videotape was copied to a 2.54-cm (I-inch) studio master for editing. The studio master was computer edited to produce 10, 30-second-long microscope fields for each specimen. A I-second audio tone was inserted into each specimen video field to mark the location for analysis. An identical set of 10 video fields for each specimen was also produced that contained a 4 X 4 computer-generated counting grid (200 tim overall; 50 tim per small square). The completed studio master was copied to standard VHS videotape for analysis at the two sites. Each specimen on the videotape was analyzed manually at The Cleveland Clinic Foundation and The University of California at Davis for count and percent motility. Manual count was performed by pausing the videotape at the audio tone marker in each specimen video field, then counting the sperm within the superimposed video grid. The manual count for each specimen (in X 10 6 of sperm/ml) was computed as the average value for its 10 video fields. Manual percent motility was determined for each specimen by starting the tape at the beginning of each specimen video field and scanning the outer border ofthe superimposed video grid while counting the motile and immotile sperm lying beneath it as the eye passed their location. The manual percent motility for each specimen was also computed as the average value of its 10 video fields. A second estimate of manual percent motility was made by pausing the videotape at the audio tone in each video field, marking the location of all sperm on the video monitor, counting the marked sperm, advancing the videotape five frames, then counting the number of sperm that moved. Computer-aided sperm analysis of the videotape was performed using four CellTrak-S instruments (Motion Analysis Corp., Santa Rosa, CA). Two instruments were located at The University of California at Davis (machines no. 1 and no. 2),and two were located at The Cleveland Clinic Foundation (machines no. 3 and no. 4). All instruments were identically calibrated using a system parameter setup previously found to be optimum for human semen (Table 1). A video threshold was chosen for each specimen on each machine that produced the same concentration of sperm found using the manual count of the videotape (Table 2). Two additional thresholds were chosen that were 50 units above and below these optima to study the effects of threshold setting. Each specimen was analyzed five times on each machine at each threshold setting (i.e., 240 experiments). Each analysis was automatically triggered by the audio tone in each specimen video field. Data were collected for: the number of motile sperm detected by the instrument (no. MOT), percentage of motile sperm (MOT), sperm concentration (CON), straight-line velocity (VSL), curvilinear velocity (VCL), linearity (LIN), amplitude of lateral head displacement (ALH), and mean angle of deviation (MAD). The definition of these variables has been published previously (6). RESULTS Table 3 shows the data obtained by CASA for all four specimens on machine no. 1 at its optimum digitization threshold (see Table 2). Similar results Davis et al. Use of CASA in multicenter studies 649

3 Table 1 System Parameter Settings Used for the CeliTrak-S CASA Instrument* Software parameters VPllO settings Parameter Value Option Setting Frame rate (frames/s) Duration of data capture (frames) Minimum motile speed (/Lm/s) Maximum burst speed (/Lm/s) Distance scale factor (/Lg/pixei) Camera aspect ratio ALH path smoothing factor (frames) X search neighborhood (pixels) Y search neighborhood (pixels) Cell size minimum (pixels) Cell size maximum (pixels) Path maximum interpolation (frames) Path prediction percentage (%) Depth of sample (/Lm) Video processor model o 20 VPllO Edge select mode Threshold Filters Right vertical edge only Set accordingly None * Version 3.20 F-Beta of the CTS software was used. In this version, cell concentration (CON) and percent motility (MOT) are automatically analyzed for five video frames, regardless of the number of frames chosen for kinematic analysis. were obtained for machines no. 2 to no. 4 (data not shown). In general, machine precision was high for all specimen/instrument/variable combinations (coefficient of variation [CV] = 1 % to 8%). Some differences in mean values were found between machines for some variables and some specimens when operated at their optimum digitization thresholds. These differences were few and were <3 % of mean values. The differences between machines no. 1 and no. 2 are shown in Table 3. Similar results were found for all machine comparisons at their optimum digitization thresholds. A larger number of differences was found between machines when the instruments were operated at the digitization thresholds 50 units above or below their optimum settings (data not shown due to space limitations). Table 4 shows the average percentage difference from mean values when the instruments were op- Table 2 Optimum Digitization Thresholds for Four Semen Specimens on the Videotape Reference Standard Analyzed on Four CASA Instruments * Machine Specimen Specimen Specimen Specimen no ,125 1,125 1,125 1, ,160 1,140 1,155 1, ,222 1,205 1,185 1, ,210 1,210 1,150 1,180 * Optimal thresholds were determined by selecting the digitization threshold on each instrument that produced values for sperm concentration that were comparable with the values found using the manual technique. erated 50 threshold units below their optima (Low T) and 50 threshold units above their optima (High T). Significant differences from the values obtained at the optimum thresholds were found for no. MOT, MOT, and CON at both the Low-T and High-T settings. At the Low-T setting, the instruments tended to overestimate no. MOT by an average of 14%, MOT by 6%, and CON by 9% for all specimens. Significant between-specimen variation also occurred (range = 3% to 18% difference from the values obtained at the optimum thresholds). Conversely, at the High-T settings, the instruments tended to underestimate these variables by 17%, 5%, and 12%, respectively, and the range of variation between specimens was even greater (2% to 23%). Mean differences in kinematic variables from the values obtained at the optimum thresholds were much lower, ranging from 0% to 7%. Table 5 shows the manual results for CON and MOT compared with the CASA results for all machines. The minimum overall variation for a given specimen was obtained at its optimum threshold setting (Tables 2 and 3). At these settings, no significant differences were found between the manual counts and the CASA counts. However, at the threshold settings above and below these optima, significant differences between manual and CASA counts were found (data not shown). Percent motility (MOT) from a manual scan ofthe original semen specimen and a manual scan of the videotape produced values that were not different from each other but that were significantly higher than those 650 Davis et al. Use of CASA in multicenter studies Fertility and Sterility

4 Table 3 Measures of Five Kinematic Variables Analyzed by CASA * Specimen 1 Specimen 2 Specimen 3 Specimen 4 No. MOT MOT CON VSL VCL LIN ALH MAD 82.0 ± 2.8 (0.08H 31.8 ± 1.0 (0.07) 24.0 ± 0.0 (0.00):1= 30.4 ± 0.4 (0.03) 49.8 ± 0.4 (0.02):1= 61.2 ± 0.8 (0.03) 2.8 ± 0.04 (0.03) 13.5 ± 0.2 (0.03) ± 2.8 (0.03):1= 46.0 ± 0.3 (0.02) 43.4 ± 0.4 (0.02):1= 35.6 ± 0.2 (0.02) 60.0 ± 0.5 (0.02) 60.6 ± 0.2 (0.01) 3.5 ± 0.02 (0.02):1= 15.2 ± 0.1 (0.02) ± 2.7 (0.04) 36.8 ± 0.4 (0.02) 38.8 ± 0.2 (0.01) 35.4 ± 0.4 (0.03) 72.0 ± 0.7 (0.02) 52.0 ± 0.3 (0.01) 4.4 ± 0.05 (0.03) 16.1 ± 0.1 (0.01) ± 2.1 (0.04) 43.4 ± 0.7 (0.03) 22.6 ± 0.2 (0.02) 57.4 ± 0.5 (0.02) 90.4 ± 0.5 (0.01) 62.8 ± 0.6 (0.02) 4.7 ± 0.04 (0.02) 13.6 ± 0.2 (0.03) * The videotape was analyzed five times for each specimen-machine-threshold block of the experiment. Approximately 255 sperm were analyzed at the optimum digitization threshold for each repetition of specimen no. 1, 490 for specimen no. 2, 426 for specimen no. 3, and 248 for specimen no. 4. Data shown are for CASA instrument no. 1 operated at its optimum digitization threshold for all specimens (T = 1,125, Table 2). Comparable results were obtained for machines no. 2, no. 3, and no. 4 when operated at their optimum digitization thresholds (data not shown because of space limitations). Slight differences in mean values between machines were found for some variables. t Values are means ± SE with CV in parentheses. :1= Mean values for machine no. 2 were significantly different from mean value from machine no. 1 (paired t-test, P < 0.05) when both machines were operated at their optimum digitization thresholds for each specimen. All differences were <3%. Similar results were obtained for machines no. 3 and no. 4 when operated at their optimum digitization thresholds (data not shown). found when performing a manual frame-by-frame analysis of the videotape or when using the frameby-frame method employed by CASA for MOT. DISCUSSION In establishing a quality-control program for CASA instrumentation, both accuracy and precision are important. Because of the variability of manual results (7), it is difficult to obtain accurate control data for comparison with CASA results. As a minimum requirement, CASA instruments must be able to visualize and recognize the same number of sperm cells recognized by a careful, manual method. Manual count, when done correctly, is the least variable of the manual measures. Clearly, if the number of cells counted by CASA is different from manual controls, other CASA measures are also likely to differ. In this study, we adjusted the threshold of each instrument until it reported the same sperm concentration as determined by our manual method. Machine no. 1 was remarkably uniform across all specimens; the same threshold setting (i.e., 1,125) was optimum in each case (Table 2). However, different optimal thresholds were found for different specimens for the other three instruments, despite our careful control of the video pedestal during production of the videotape reference standard. Comparable differences within a machine/specimen combination resulted in some significant differences for CASA measures (Table 3). Such small differences are difficult to detect visually by a system operator who is attempting to set the digitization threshold of the instrument by monitoring the digitized image on the monitor. Hence, until the sensitivity of CAS A instruments can be normalized across a reasonable range of optical conditions commonly found in semen analysis, it is not possible to adjust a particular instrument to a given threshold, then use this setting for all specimens. Instead, the threshold must be adjusted for each specimen according to its manual control. The precision of these four instruments was considerably better than precisions reported for manual methods (7). The accuracy of the instruments, once their thresholds were optimized for each specimen, was within 1 % to 5% of the manual counts (Table 5). Although significant differences were found between some machines for some CASA variables, these differences were small and are not likely to be clinically or biologically significant. Such differences are probably caused by the extremely high precision of the instruments. The differences between percent motility determined by CASA and by manual methods (Table 5) may be explained by several factors. First, manual estimates of percent motility are based on a definition in which a cell is classified as motile when its flagellum is moving, even if it is not progressive. Because all CASA instruments require a cell to attain a minimum average progressive velocity (usually 10 JLM/s) over a minimum number of video frames (usually 2 to 5) to be classified as motile, CASA estimates of percent motility will always be lower than manual estimates. Second, manual estimates of per- Davis et al. Use of CASA in multicenter studies 651

5 Table 4 Average Percentage Difference from Optimum Mean Values* Variable Specimen 1 Specimen 2 Specimen 3 Specimen 4 Overall Low-T No. MOT MOT CON VSL VCL LIN ALH MAD High-T No. MOT MOT CON VSL VCL LIN ALH MAD * Average percentage difference in CASA measures from the values shown in Table 3 when the videotape reference standard was repeatedly analyzed (n = 5) at 50 thresholds units below the optimum setting for each specimen/machine combination (Low-T) and at 50 threshold units above the optimum setting for each specimen/machine combination (High -T). See Table 2 for the optimum digitization thresholds for each machine/specimen combination. In general, significant differences in CASA measures from the optimum threshold settings were found within specimens for no. MOT, MOT, and CON for Low-T and High-T. Similarly, significant differences were found across specimens at both the Low-T and the High-T settings for these variables. cent motility are based on a visual scan of the specimen or of the videotape. With this method, the number of motile cells counted is influenced by: the density of cells in the field of view, their swimming speeds, and the speed of the visual scan. The importance of this second set of factors is illustrated by the results we obtained when we manually recounted motile and immotile cells in each video field, starting at the tone, using a frame-by-frame method, rather than a real-time visual scan (Table 5, tape frame-by-frame analysis). Using this method, manual estimates for percent motility agreed well with the CASA measures. Because of the fundamental differences between traditional manual methods and CASA methods for estimating the percentage of motile cells, the comparison of manual results with CASA results for percent motility may be inappropriate until a manual method can be developed that is more comparable with the method used by CASA. Multicenter studies require careful instrument calibration, standardization of protocols, and quality control. Findings presented in this study and elsewhere (8) demonstrate that similar results can be obtained using different CASA instruments at different sites and that the accuracy and precision of such measurements can be high. They also reveal that CASA results are extremely sensitive to the optical qualities of the image, differences between specimens, and the digitization threshold setting. Such parameters are largely beyond the control of instrument manufacturers. It is essential that pro- Table 5 Manual and CASA Results for CON and MOT for All Machines CON Specimen Tape Tape no. scan CASA* (192)t 23.6 (256) (360) 44.9 (493) (296) 38.9 (426) (183) 22.5 (248) * Values from optimum center threshold setting. MOT Semen Tape Tape Tape scan scan CASA* FBF 50.0 (50) 54.8 (50) 31.7 (256) 41.2 (250) 43.0 (50) 58.9 (50) 45.8 (493) 46.6 (424) 50.0 (50) 52.9 (50) 37.1 (426) 41.3 (346) 71.0 (50) 67.5 (50) 42.8 (248) 46.7 (210) t Values are means with average number of sperm analyzed in parentheses. 652 Davis et al. Use of CASA in multicenter studies Fertility and Sterility

6 cedures that standardize the optical platforms used with CASA as well as the methods used to select the digitization threshold be developed within the laboratory. REFERENCES 1. Hinting A, Comhaire F, Vermeulen L, Dhont M, Vermeulen A, Vandekerckhove D. Value of sperm characteristics and the result of in vitro fertilization for predicting the outcome of assisted reproduction. Int J Androl 1989;13: Davis RO, Niswander PW, Katz DF. New measures of sperm motion I: adaptive smoothing and harmonic analysis. JAn drol. In press. 3. Ginsburg KA, Moghissi KS, Abel EL. Computer-assisted semen analysis: sampling errors and reproducibility. J Androl 1988;9: Mack SO, Wolf DP, Tash JS. Quantitation of specific parameters of motility in large numbers of human sperm by digital image processing. BioI Reprod 1988;38: Katz DF, Davis RO, Delandmeter BA, Overstreet JW. Realtime analysis of sperm motion using video image digitization. Comput Methods Programs Biomed 1985;21: Boyers SP, Davis RO, Katz DF. Automated semen analysis. Curr Probl Obstet Gynecol Fertil1989;12: Dunphy BC, Kay R, Barratt CLR, Cooke ID. Quality control during the conventional analysis of semen, an essential exercise. J Androl 1989;10: Davis RO, Katz DF. Standardization and comparability of CASA instruments. J Androl. In press. Davis et al. Use of CASA in multicenter studies 653