Reproducibility of computer-aided semen analysis: comparison of five different systems used in a practical workshop*

l FERTILITY AND STERILITY Copyright c 1994 The American Fertility Society Vol. 62, No. 6, December 1994 Printed on acid-free paper in U. S. A. Reproducibility of computer-aided semen analysis: comparison of five different systems used in a practical workshop William Holt, Ph.D.t:j: Paul Watson, Ph.D. Mark Curry, Ph.D. Clare Holt, B.A. t Institute of Zoology, Zoological Society of London, and the Royal Veterinary College, London, United Kingdom Study objectives: To assess a single (donor) sample by the use of five computer-assisted semen analysis (CASA) systems. Setting: British Andrology Society advanced course on computer-assisted semen analysis techniques, September 1992. Participants: Clinical, technical and scientific personnel with mixed experience of CASA techniques, but all interested in semen assessment technology. Results: 22 sets of data, comprising 6 commonly derived parameters, from the same sample were obtained using five CASA systems. The coefficients of variation for sperm concentration (29%) and proportion motile (24%) were comparable to those previously reported for manual assessments. For all parameters, "within system" variability was considerably greater than "between system," indicating that differences in sample handling and operator expertise were more significant sources of variation than the CASA systems themselves. Conclusions: Emphasis on operator training and standardization of sample handling techniques would enhance the reproducibility of CASA measurements more than improvements in the CASA systems themselves. There was evidence, however, that the CASA measurements were more consistent than equivalent manual measures, and also provide information about the quality of sperm motility which cannot be obtained by alternative techniques. Fertil Steril1994;62:1277-1282 Key Words: CASA, computer-aided sperm analysis, quality control, semen analysis The potential advantages of computer-aided semen analysis (CASA) over subjective assessment techniques include better objectivity, sensitivity, and reliability. The introduction of CASA techniques into clinical laboratories should therefore Received February 21, 1994; revised and accepted July 1, 1994. Data obtained at a British Andrology Society workshop on Computer-assisted Sperm Motility Analysis at the Royal Veterinary College, London, United Kingdom, September 24 to 25, 1992. t Institute of Zoology, Zoological Society of London. :j: Reprint requests: William Holt, Ph.D., Institute of Zoology, Regent's Park, London NW14RY, United Kingdom (FAX: 44-71-586-2870). Royal Veterinary College. Vol. 62, No.6, December 1994 help to improve laboratory standardization and the implementation of quality control procedures. In turn, these advantages will permit valid comparison of data from different laboratories and foster greater confidence in the diagnostic value of the data generated from semen analysis procedures. Current use of CASA systems falls short of this ideal situation (for commentary, see reference 1). They are subject to variations in operator expertise, and differences in setup conditions can affect both magnitude and reproducibility of the data obtained (2). Moreover, manufacturers have adopted different programming algorithms for deriving nominally identical motion parameters. This partly is due to differences in image capture and processing hardware between systems, although it is also a Holt et al. Communications-in-brief 1277

reflection of the mathematical approaches adopted by programming teams. Subtle differences of this nature usually are not obvious to users, although carefully designed calibration videotapes showing computer graphic-derived test objects moving with known parameters may be useful in overcoming this problem. Variability also is imposed by noncomputational issues such as specimen preparation and microscopic technique. Users themselves are responsible for many ofthese aspects, which probably represent an important source of variability, perhaps more significant in practice than the subtle differences between instruments. Assessment of this variability is difficult as few opportunities arise to compare results both between systems and between different operators; however, identification of such problems would allow them to be overcome by operator training and the use of standardized techniques. At a British Andrology Society Advanced Course in computerized semen analysis (September 1992), five different systems were available simultaneously, and the unique opportunity arose to compare their measurement output from a single semen sample. This comparison was not the primary purpose of the course, and the measurements therefore were not made under rigorously controlled conditions. However, the exercise included all the operator-dependent steps needed in making a semen analysis, such as loading the semen into a chamber and selecting fields for microscopy, and both within -system and between -system comparisons can be examined within the data. The levels of expertise and experience differed between participants, ranging from novices with no previous experience of CASA to those working with CASA systems on a daily basis. The data are therefore of interest because they represent the variability to be expected when the systems are used routinely in a clinical setting, where operator variation is probably more important than that introduced by electronic means. On the whole the results were unexpectedly encouraging. However, although most of the variability would probably be insignificant in biologic or diagnostic terms, this sample would have been assessed inconsistently if sperm concentration or proportion of motile sperm were the only parameters measured. MATERIALS AND METHODS This study was performed at the 1992 Advanced Course on Computer Assisted Semen Analysis orga- nized by the British Andrology Society in September 1992 and held at the Royal Veterinary College, London. Twenty-seven participants, working in groups, attended the practical session held on Friday, September 25th; the session commenced at 2:15P.M. and finished at 4:00P.M. Each group was asked to obtain a set of semen parameters from every type of available system, using the single semen sample provided. Specifically, the parameters requested were sperm concentration, proportion motile, mean curvilinear velocity (VCL), mean average path velocity (VAP), mean linearity, and mean amplitude of lateral head displacement (ALH). The exact number of cells used for deriving the mean values was not recorded, but these were based upon estimates from a minimum of 200 cells. Equipment Six CASA systems, representing five different models, were available. These were one CellTrak system (Launch Diagnostics Ltd, Dartford, United Kingdom); two Hamilton-Thorn HTM-2000 systems; one Hamilton-Thorn IVOS system (Microm UK Ltd, Letchworth, United Kingdom), one Hobson Sperm Tracker (Sense & Vision Electronic Systems Ltd, Sheffield, United Kingdom); and one Mika system (Mika Medical, Rosenheim, Germany). Apart from the two HTM-2000 systems, help in using the equipment was provided by technical representatives who had given earlier detailed demonstrations of their use. Preparation of Samples for Microscopy A single fresh sample of human semen from a regular donor was provided for use during the afternoon; it was maintained undiluted at room temperature (approximately 22 C) for use over approximately 1.5 hours. Statistical Analysis The CSS/Statistica computer programme (Statsoft UK, Letchworth, United Kingdom) was used for data analysis. Analyses of variance were used to examine between-system variability; where differences were observed, the data were further analyzed as nested designs to evaluate within-system variation. Specific contrasts between systems were examined using the Least Significant Difference (LSD) technique or Tukey's honest test for unequal group 1278 Holt et al. Communications-in-brief Fertility and Sterility

Table 1 Sets of Data Collected by Analysis of a Single Semen Sample by the use of Five Different CASA Systems Curvilinear Average Amplitude of lateral System Concentration Motility velocity path velocity Linearity head displacement XI(/' % p.mjs p.mjs p.m CellTrak 38 65 51 m 48 2.7 62 22 41 31 79 6.0 m m 68 115 55 1.2 44 65 53 m 49 2.2 Hobson Sperm Tracker 17 60 36 47 55 5.3 57 67 54 m 43 3.1 44 54 41 28 69 5.0 m m 27 18 51 3.7 HTM-2000 40 55 57 52 81 2.9 75 62 64 53 77 3.1 32 54 47 34 63 2.4 46 53 57 52 89 2.7 37 36 58 52 82 2.4 37 48 47 36 m 2.1 40 35 64 59 89 2.1 IVOS 64 51 63 50 69 3.4 52 47 63 52 m 2.5 40 54 53 49 82 3.0 Mika 36 57 51 m 44 2.5 52 61 49 35 63 2.5 43 46 45 35 71 1.9 35 30 48 37 70 2.1 Mean± SD 44.5 ± 12.93 51.1 ± 12.23 51.7 ± 10.05 46.4 ± 20.35 66.5 ± 14.95 2.94 ± 1.16 cv 29.05 23.93 19.43 43.85 22.48 39.45 Missing value. sizes (3), which weighted means shown in Table 1 to account for the number of times each system was tested. Correlation analyses also were performed. RESULTS Participants in the practical workshop provided 22 sets of semen analysis data from the single semen sample; the entire data set is shown in Table 1. Six data sets contained one missing value, and two data sets contained two missing values. A series of box-whisker plots illustrate the means, SDs, and SEMs obtained for each parameter (Fig. 1A to F). Sperm Concentration Mean values for sperm concentration showed remarkably close agreement (P = 0.788), being clustered in the range from 39 to 52 X 10 6 spermatozoa/ ml. Over the whole data set the coefficient of variation (CV) was 29%; this was comparable to CV values obtained for individual systems. The most extreme single measurements were 17 X 10 6 (Hobson Sperm Tracker) and 75 X 10 6 (HTM-2000). Sperm concentration was correlated weakly with VCL (r = 0.48; P = 0.03). Proportion Motile Mean values ranged from 49% to 60%, with an overall CV of 24%. This indicated close agreement (P = 0.759), although a greater degree of withinsystem variability was apparent than for the sperm concentration measurements. This effect could have been caused by differences in sample handling technique, because most of the variability was caused by individual, exceptionally low results. Velocity Measurements (VCL and V AP) Significant between-system variation was noted for VCL (P = 0.024) but not for VAP (P = 0.182). Mean VCL values for each model ranged from 39.5 to 59.7 J.Lm/s, but individual measurements varied widely from 27 to 68 J.Lm/s. Between-systems comparisons were not significant when the Tukey honest test was used, and the variation is therefore attributed to random errors in sample handling. This conclusion was further supported by examining the data as a nested design, where between-operator variation was an order of magnitude (P = 0.003) more significant than between systems (P = 0.02). Sixteen of 18 V AP measurements were lower than the corresponding VCL values; this is the expected relationship given that the path used in cal- Vol. 62, No.6, December 1994 Holt et al. Communications-in-brief 1279

60 r-~----~--~~---r----~-, A g 60 ~ 40 20 60 ~,-----r---~----,---~~ ~cp. g a ~ 60 :3 0 8 :. 40 i ~ 20 CT HTII HOBSON IVOS MIKA 60 ~~----~---,----~----~ c CT HTII HOBSON IVOS MIKA 7~~----r----r----~--~--, 6 5 ~ 4 3 2 g E 0 ~~----~--~----~----~ CT 120 i HTM HOBSON IVOS. IIIKA 100 60 : 60 40 20 'Q ~ D ol-...1...-----jl..----'-----...., CT HTM HOBSON IVOS IIIKA IOOr-~--~----,---~----,-~ ;: 80 40 ol-~----~--~----~--~~ CT HTII HOBSON IVOS MIKA CT HTII HOBSON IVOS IIIKA Figure 1 Box-whisker plots showing means ( ), SEM (boxes), and SD (bars) of the aggregated data for individual parameters and computer systems. The asterisks indicate individual data points that lie more than 1 SD away from the mean. culating V AP represents the shorter distance. Within this subset, VCL to V AP ratios varied between 1.1 and 1.5, with most values toward the lower end of the distribution (mean 1.256; median 1.272; mode 1.096). Two sets of data indicated VCL to V AP values below unity, i.e., 0.59 and 0. 76, from CellTrak and Hobson Sperm Tracker, respectively. As identical linearities (55%) were reported in each case, true VCL values should have been approximately 1.5- fold higher than V AP values. These discrepancies probably were caused by misreporting of the data; the errors originated within two separate groups of participants. Mean curvilinear velocity and V AP values were correlated (r = 0. 7 4; P = 0.0004) within the data set. Mean curvilinear velocity also was correlated negatively with ALH (r = -0.59; P = 0.008); this is considered further in a subsequent section. Linearity Significant variation was detected between systems (P = 0.018). Although the LSD technique indicated that several specific contrasts were responsible for this effect (i.e., Hobson Sperm Tracker versus CellTrak, HTM-2000, and IVOS, P = 0.029, 0.0044, and 0.0048, respectively), the Tukey test showed that only the Hobson Sperm Tracker versus HTM-2000 contrast was valid (P = 0.045). Fur- 1280 Holt et al. Communications-in-brief Fertility and Sterility

thermore, when the data were analyzed as a nested design, the between -operators variability was significant (P = 0.0128) whereas between-systems variablity was not (P = 0.08). Mean linearity values for different systems were clustered into two groups: 75% to 80% for the two Hamilton-Thorn systems, and 54% to 67% for the other three models. There was no obvious pattern, however, when the individual data was examined and this effect may have been coincidental. Linearity measurements were negatively correlated with percentage motility values (r = -0.59; P = 0.009). Amplitude of Lateral Head Displacement Measurements of ALH were consistent between systems (P = 0.147); mean values varied between 2.4 and 4.3 Mill. The lowest and highest individual results (1.2 and 6.0 Mill) were obtained with the same system, so the differences probably were caused by sampling and operator errors rather than by the equipment. Amplitude of lateral head displacement measurements were correlated negatively with VCL values (r = -0.59; P = 0.008) when the entire data set was considered. This effect was eliminated by the exclusion of a cluster of four outliers in the data and by the omission of all Hobson Sperm Tracker data sets. It is believed that the correlative effect was attributable either to differences in setup parameters between systems or to the different techniques used for both image and data analysis. DISCUSSION For all parameters, the values for within-system variability encountered in this study were high in comparison to those (1% to 8%) reported by Davis et al. (2). However, their study involved carefully performed CASA measurements on standardized, prerecorded, videotaped sequences. Although that approach is clearly the best for evaluating intersystem agreement, the present data includes major components representative of operator error in sample preparation, handling, and recording. Use of a single semen sample in this study was an advantage in highlighting these components. Although additional semen samples from normal or subfertile donors would, in principle, have provided more data, it is likely that in the workshop setting a less complete data set would have been obtained. It is probably the case, however, that the degree of variation observed with the normal donor would have been exacerbated if oligospermic samples had been used. In the present study, the sperm concentration results ought to be the least affected by delays in performing the assessments. The averaged values for different systems were, indeed, remarkably consistent. However, the overall CV within the data set was 29%, only a slight improvement upon values reported by Jequier and Ukombe (4) (44%) and Neuwinger et al. (5) (37.5%) for similar comparisons undertaken by manual counting methods. These considerations suggest strongly that operator and setup errors were responsible for the variability in the present data, and targets this area as being more important than system performance in generating standardized data. Despite the agreement between overall mean values for sperm concentration generated by the different systems, individual data points varied considerably. Even if the two most extreme values are omitted, almost 100% discrepancy between the next two outliers (35 and 62 X 10 6 /ml) is still evident. In this particular instance the differences would probably not, however, significantly affect decisions regarding infertility treatment. The highly significant negative correlation between percentage of motile spermatozoa and linearity may indicate a potential artefact. As more motile spermatozoa enter the measurement field the probability oftrack crossing increases, thus giving false tracks of artificially low linearity. This type of artefact was noted previously by Vantman et al. (6), who attributed an inverse relationship between straight line velocity and sperm concentration to the same effect. As some systems offer the opportunity to examine the fidelity of individual tracks, it would be possible to confirm the source of this artefact. In the present study there was no significant correlation between sperm concentration and linearity (P > 0.05), as might have been expected from the explanation given above. One possible reason would be that as all measurements were performed on the same sample, the actual sperm concentration was always the same despite the perceived differences between measurements. This would mean that the sperm crossing artefact could only be influenced by changes in the proportion of motile cells, because there was no possibility of real changes in sperm concentration. Had a range of genuinely differing concentrations been studied, there would have been a greater likelihood of detecting an effect upon linearity. The CV s for the sperm motion parameters fell Vol. 62, No.6, December 1994 Holt et al. Communications-in-brief 1281

into two groups, VCL and linearity (approximately 20%) and V AP and ALH (approximately 40% ). As the algorithms used to calculate ALH differ between manufacturers, it is doubtful that the reported ALH values truly represent the same parameter when derived from different systems (see manufacturers handbooks). Calculation of both V AP and ALH initially involves estimation of the smoothed track, using either rolling averages derived from a fixed or adaptively variable number of previous coordinates or using entirely different smoothing algorithms. Distances between the smoothed track and individual track points are then derived and presented as mean deviations. The mean deviations thus also are dependent upon the number of sperm flagellar beat cycles represented within each individual track. These findings clearly indicate that considerable caution is needed when V AP and ALH values derived from different systems or studies are compared. Mean sperm concentration and percentage motility value are of some interest because although most individual results fell above the minimum limits for fertility, others fell short. The analysis presented here would support the view that such variability is largely introduced by operator error. As the clinical evaluation of infertility would be influenced by such differing results, the case for improving quality control measures and staff training is demonstrated clearly. The data reported here provide some encouragement to those concerned with quality control and precision in the andrology laboratory. Each parameter revealed that errors arising through variations in operator expertise and sample handling had considerably more influence upon the data than the more subtle differences between computer systems. In a working environment the use of standardized sample handling procedures and highly trained staff must therefore represent a higher priority than the choice of CASA system to purchase. However, given the mixed abilities of those attending the course, the surprisingly good agreement between overall mean values for many of the parameters indicates that the use of CASA systems enhances reproducibility of assays compared with manual results ( 4) and gives access to other information regarding quality of motility, which would otherwise be unavailable. The results of this analysis should not be regarded as a performance comparison between different CASA systems, especially as it was found that operator error was the more significant source of variation. It is recognized that software and hardware designs are being updated and improved continuously, and also that critical choice of setup parameters for CASA systems has an important influence on the quality of data obtained. Clearly, improvements in the application of CASA systems will be gained through developments in all aspects of the technology. Acknowledgements. We thank the companies who both sponsored the workshop financially and provided their CASA equipment and expertise. We also thank the International Society of Andrology for financial support toward the course. REFERENCES 1. Davis RO, Katz DF. Computer-aided sperm analysis: technology at a crossroads. Fertil Steril 1993;59:953-5. 2. Davis RO, Rothman SA, Overstreet JW. Accuracy and precision of computer-aided sperm analysis in multicenter studies. Fertil Steril 1992;57:648-53. 3. Spjotvoll E, Stoline MR. An extension of the T -method of multiple comparison to include the cases with unequal sample sizes. J Am Stat Assoc 1973;68:976-8. 4. Jequier AM, Ukombe EB. Errors inherent in the performance of a routine semen analysis. Br J Urol 1983;55:434-63. 5. Neuwinger J, Behre HM, Nieschlag E. External quality control in the andrology laboratory: an experimental multicenter trial. Fertil Steril 1990;54:308-14. 6. Vantman D, Koukoulis G, Dennison L, Zinaman M, Sherins RJ. Computer-assisted semen analysis: evaluation of method and assessment of the influence of sperm concentration on linear velocity detemination. Fertil Steril 1988;49:510-5. 1282 Holt et al. Communications-in-brief Fertility and Sterility