Assessment of "Average of Normals" Quality Control Procedures and Guidelines for Implementation

Transcription

1 Assessment of "Average of Normals" Quality Control Procedures and Guidelines for Implementation GEORGE S. CEMBROWSKI, M.D., PH.D., ELLIOT P. CHANDLER, M.D., AND JAMES O. WESTGARD, PH.D. The capabilities of "Average of Normals" control procedures have been assessed by determining power functions, graphs of the probability of error detection versus the size of analytic error. The power functions indicate that the most important determinants of statistical power are the ratio of the standard deviation of the patient population (s p ) to the analytic standard deviation (s a ), s p /s ; the number of data points averaged (N); the control limits (probability for false rejection); truncation limits for selecting the population; and the magnitude of the population lying outside the truncation limits. General guidelines for the implementation of "Average of Normals" are provided, along with a nomogram for the selection of N as a function of s p /s and the probability of error detection. Optimal performance of these procedures may require simulation studies on a per analyte basis. (Key words: "Average of Normals"; Quality control; Power functions; Statistics) Am J Clin Pathol 1984; 81: HOFFMAN AND WAID described the "Average of Normals" (AON) method of quality control in In AON, an error condition is signalled in an analytic process whenever the average of selected consecutive patient data is beyond the control limits established for the average of the patient population. In the original description of AON, patient results were included in the average if they were within a "normal range" that was determined from patient data. 5 For control limits, Hoffman and Waid used the 95% confidence limits for the stable patient mean and indicated that significant analytic error could be detected by averaging as few as ten consecutive BUN or glucose values. After the initial interest in AON, there have been only scattered reports describing its application. 7 " 14 A recent Expert Panel of the International Federation of Clinical Chemistry has reported that "... uncertainty about the stability of the patient population and the time and effort of calculation have relegated these (AON) methods to auxilliary use." 3 With the advent of instrument-based minicomputers and microcomputers, the time and effort for calculating averages of patient data have been reduced enormously. Received June 29, 1983; received revised manuscript and accepted for publication September 6, Address reprint requests to Dr. Cembrowski: Department of Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, 3400 Spruce Street, Philadelphia, Pennsylvania Clinical Laboratories and Department of Pathology and Laboratory Medicine, University Hospital and Clinics, Madison, Wisconsin As examples, both the KDA (American Monitor, Indianapolis IN 46268) and the SMAC (Technicon, Tarrytown, NY 10591) analyzers can calculate the averages of selected patient data. Other factors, including the uncertainty about the stability of the patient population, have not been fully investigated, and therefore may be limiting the present application of AON procedures. One of the first evaluations of AON procedures was in 1968 by Amador and co-workers.' In comparisons to reference sample quality control, AON was judged insensitive to systematic error when the averages of ten consecutive normal glucose or SGOT values were outside the 95% limits for the accepted mean. Kilgariff and Owen 6 found that AON was insensitive to simulated systematic error in eight different analytes when ten consecutive data were averaged, selecting patients using truncation limits derived from healthy individuals. Owen and Campbell 9 found that averages of 25 and 50 patient values detected error more effectively, but they still concluded that the use of reference samples was superior to the daily means of even 50 patients. Reed 10 investigated the choice of truncation limits for selecting the patient results to be included in the average and found that the use of the end points of the estimated patient normal range was "not always optimal but... reasonably close." Lewis and Dixon 8 and Begtrup and associates 2 related the sensitivity of the AON technic to the degree of interindividual variation in the measured analyte. They showed that tests with a low interindividual variation relative to the analytic variation, e.g., sodium, required comparatively few (approximately ten) values to be averaged for a sensitive indicator of systematic error. Begtrup and associates developed a formula, a function of the ratio of the population standard deviation (Sp) to the analytic standard deviation (s a ), to calculate the minimum sample size to detect shifts of Sp with the same sensitivity as reference sample quality control. Despite the many references in the literature, descrip- 492

2 Vol. 8i-No. 4 AVERAGE OF NORMALS 493 tion of the performance characteristics* of AON procedures and a rational approach to optimizing their performance are still lacking. With the increasing emphasis on efficient laboratory operation, the capabilities of AON procedures must be assessed more fully so they can be applied when effective. In this article we describe the capabilities of AON in the form of power function graphs, where the probability of error detection is plotted versus the size of analytic error. 12 We have investigated the importance of several parameters that depend on the patient population (Sp, distribution of outliers), the analyte measured (s a ), and the choice of conditions for the AON control procedure (control limits, N or the number of observations averaged, and truncation limits). Frequency Histograms Materials and Methods Consecutive SMAC patient data were accumulated over four successive weekdays. Frequency histograms of each analyte were constructed, and the approximate bounds of the central portion estimated by visual inspection. The data within these bounds were averaged to obtain the population mean (x p ) and population standard deviation (Sp). The analytic standard deviation (s a ) for each analyte was derived from the long-term standard deviation of one of three different commercial control products, selecting the control product whose analyte concentration was closest to x p. Computer Simulation To study the performance characteristics of AON procedures, simulated patient data were generated by a computer program. A random number subroutine first generated patient data with a Gaussian distribution with a specified mean (x p ) and standard deviation (Sp). Varying amounts of systematic error were introduced into the patient results by adding multiples of s a from 0.5 to 5.0. The data at each error level then were subjected to statistical calculations and testing with the proportions of means exceeding the control limits tabulated. Five hundred simulations were done at each error level and each N. Power functions were generated by plotting the tabulated data against the size of the systematic error. The number of observations averaged was varied from * The performance of a control procedure can be characterized by its probability (p) for giving a rejection signal. The probability for false rejection (p rr ) is the probability for giving a rejection signal when there are no analytic errors present except for the inherent imprecision of the analytic method. The probability for error detection (p s) is the probability for giving a rejection signal when an analytic error is present. Table J. Population Mean (x p ), Population Standard Deviation (s p ), Analytic Standard Deviation, s a and Sp/s a Analyte Sodium (mmol/l) Potassium (mmol/l) Chloride (mmol/l) C0 2 (mmol/l) Glucose (mg/dl) Urea Nitrogen (mg/dl) Creatinine (mg/dl) Calcium (mg/dl) Phosphorus (mg/dl) Uric acid (mg/dl) Cholesterol (mg/dl) Total protein (g/dl) Albumin (g/dl) Total bilirubin (mg/dl) Gamma glutamyl transpeptidase (U/L) Alkaline phosphatase (U/L) Aspartate aminotransferase (U/L) Lactate dehydrogenase (U/L) Xp Sp Sa Sp/S to 100. The control limits were selected so that the probability of false rejection was 0.2%, 1%, and 5%. To study the effects of outliers, we chose to generate outliers with a uniform random number generator. This produced specified ranges of values that were evenly distributed. To study the worst case condition, the population of outliers was located adjacent to the 99% limits of the Gaussian distribution. The population thus was situated 2.58 Sp from x p and extended 5 s a (since 5 s a was the maximum systematic error simulated for the power functions). The fraction of outliers to total population was varied from 0% to 10%. While the outlier populations could have been modelled with skewed Gaussian distributions, their characterization and simulation would have been far more complex. Verification of Simulation Model In order to verify the computer simulation results, more rigorous and exact results were determined (for limited cases) by probability calculations. Gaussian curves corresponding to patients with specified means and standard deviations were shifted by amounts corresponding to the analytic error. Truncation limits then were applied to this shifted distribution and the distributions of mean values were calculated for the shifted, truncated population. The distributions of means then were compared with the control limits to determine the area or proportion exceeding the control limits (which corresponds to the probability for error detection). This approach was used to generate families of power functions for varying Sp/s a and N.

3 494 CEMBROWSKI, CHANDLER, AND WESTGARD A.J.C.P. April 1984 GLUCOSE Sp/Sa=7 PDTR55IUM Sp/Sa= _ l 1 _J OQ. 1 " i- 100^ /so / _ M _l -S M -1 m - 3 S SYSTEMATIC 3 4 ERROR ( S a B SYSTEMATIC ERROR (S a l CRERTININE Sp/Sa=4 SODIUM Sp/Sa=2.7 SYSTEMATIC ERROR ( S a l D SYSTEMATIC ERROR ( S a l FIG. 1. Power functions illustrating the effect of varying N and Sp/s a. Probability of false rejection (p fr ) = Results Table 1 summarizes Sp, s a, and Sp/s a for the SMAC chemistry tests. Glucose has an Sp/s a ratio of 7.0, potassium a ratio of 5.4, creatinine 4.0, and sodium 2.7. These four analytes were selected for detailed simulation studies because of their approximate Gaussian distributions and their wide range of Sp/s a values. The effect of varying the number of patient results that are averaged (N) by the AON procedure is shown by the

4 Vol.81 -No. 4 AVERAGE OF NORMALS 495 power functions of Figure 1. The probability of rejection is plotted against the size of systematic error. The bottom line in each power function graph shows the probability for error detection for N = 10, the second line is for N = 20, the third for N = 50, and the top line for N = 100. In Figure 1, control limits have been calculated to provide a probability of false rejection of 0.01, or 1%. Error detection increases as N increases. For glucose (Fig. 1,4), with N = 100 there is a probability of about 0.50 (or a 50% chance) of detecting a systematic shift equivalent to twice the size of the standard deviation of the analytic method. Figures IB, ICand ID show power functions for potassium, creatinine and sodium respectively. As Sp/ s a decreases, equivalent power can be provided with lower N. For example, an AON procedure for sodium with N of 20 can be expected to provide as much error detection as a glucose AON procedure having N = 100. The general effect of Sp/s a on the detection of a systematic shift equivalent to 2 s a is shown by the nomogram in Figure 2, which was derived from the probability calculations. Here the probability for rejection is plotted versus N, with the scale on the x-axis being logarithmic. Thisfigureillustrates more generally how error detection depends on the number of patient results and the ratio of the population and analytic standard deviations. Figure 3 shows the effect of varying the control limits for glucose and sodium. The bottom curve in each graph corresponds to control limits calculated as 3.09 times the EFFECT OF SP/SA (25 SHIFT! m o_ 0- y V s a / / 2 3 / / / / / / / / / / / / 4/^^^x^ i i i i J-» >, I^~ M '^*S / / / n / ////8 / / / / / / ///// i i FIG 2. Nomogram for selecting N based on Sp/s, and the probability for detecting a 2 s shift (p fr = 0.01). standard error (SE) of the mean, which provides a false rejection rate of 0.2%. The middle curve is for 2.58 SE control limits or 1.0% false rejections, and the top curves 9 GLUCOSE, N = 100 PFR 0.002,0.01,0.05 SODIUM, N = 20 JLF R 0.002, n D 9-8-?- 6- S- 4-5/ 7 1 / yo.2% ^_.9 z M _l.5 t 1 rn.1 rn o or < r *""~^--* m i 0.0 SYSTEMATIC ERROR (5J B SYSTEMATIC ERROR (S a l FIG. 3. Effect of p fr on power functions.

5 496 CEMBROWSKI, CHANDLER, AND WESTGARD A.J.C.P. April 1984 GLUCOSE, N 1 00 SODIUM, N 20 <C 03 O ac Q_ i ? i.2^ POPULATION GAUSSIAN TRUNCATION 3.09 SP 2.58 SP / 1.96 SP ft / / m o Q_ ? H P0PULATION GAUSSIAN TRUNCATION / S P // 2.58 SP If 1.96 S?// 1, /1S i i i i SYSTEMATIC ERROR (5 a l B 1 l 1 i i i SYSTEMATIC ERROR (SJ FlG. 4. Effect of truncation limits on power functions for populations without outliers (p fr = 0.01) are for 1.96 SE control limits or 5.0% false rejections. As control limits narrow and the rate of false rejections increases, the apparent error detection increases. The effect of outliers and the use of truncation limits to minimize the influence of outliers were studied by simulating two populations, one without outliers and the other with outliers. Figure 4 shows the power curves for patients without outlying glucose and sodium concentrations. The family of curves represents the power obtained when various truncation limits are used. The top curve in each power function graph is for no truncation limits, the next lower curve is for 3.09 Sp limits, the next for 2.58 Sp limits, and the bottom curve is for 1.96 Sp limits. In the case where there are no outliers, maximum error detection is achieved when truncation limits are widest. When an AON procedure is applied to hospitalized populations, there are usually a relatively large number of patient results outside the truncation limits. The frequency distributions of glucose, potassium, creatinine, and sodium data used to formulate the simulation model are shown in Figure 5. For glucose, about 8% of the population in our hospital was adjacent to the visually estimated Gaussian portion of the distribution. If a negative systematic shift occurred and thus caused the patient results to decrease, this part of the population would be shifted below the upper truncation limit and therefore could affect the observed average. Figure 6 shows how the power of the AON procedure would depend on the choice of truncation limits. The highest power curve is observed for 2.58 Sp truncation limits, followed by 3.09 Sp truncation limits, and finally by the power curve when no truncation limits were used. In this situation, the narrower truncation limits provide better error detection. As the percentage of the outlier population increases, the power for error detection decreases, as shown in Figure 7, for 0%, 5%, and 10% outliers and for 2.58 Sp truncation limits. The effect of increasing the outlier population, at a fixed truncation limit, is somewhat smaller for a sodium AON procedure than was observed for the glucose AON procedure. Discussion These studies indicate that the performance of AON procedures are affected by several factors: the ratio of population to analytic variation (Sp/s a ); the number of patient results averaged (N); the control limits chosen; the truncation limits chosen; and the percentage of the population lying adjacent to the truncation limits. Implementation of an AON procedure therefore requires careful choices of conditions. The ratio of the population standard deviation to the analytic standard deviation should be recognized as a particularly critical characteristic in developing efficient AON procedures. The number of patient results should

6 Vol. 81 'No. 4 AVERAGE OF NORMALS 497 PATIENT DATA PATIENT DATA 3 a GLUCOSE N 62 1 MEAN ? DATA i i r i so IlLiflflW 1MG/0LI SELECTEO -i i i r r (_) 60 Z LiJ S0- a 40 ae. 30 B ' M~ 3X " POTASSIUM N 678 MEAN 4.4 DATA SELECTED [L Jurfl rvm - IT-i ( MMOL'L -n 9.0 PATIENT DATA PATIENT DATA N 698 MEAN 1.6 DATA SELECTED O 60- Z Ul S0- N 685 MEAN DATA SELECTED r _ri 5% L^rf>wflHh(hlmJLw. i - -n- - -Unn m>n ? a 40- a: % - ^^^ r n i P-. -i ITIM-I i nn_ i i i i i i i i i ISO 1 60 CREATININE I MG^DL ) SODIUM 1MMDL/L I FIG. 5. Frequency histograms of patient data. The darkened areas correspond to the population outside the truncation limits which would be shifted in and averaged if the systematic error were 5 Sj. be chosen based on Sp/s a. As few as 20 results may be efficient for sodium, whereas 100 or so may be required for glucose. Uniform application of an AON procedure with a constant N is not an efficient design. The relationship between N and Sp/s a, as shown in Figure 2, may be used to select an approximate value of N. Control limits should be chosen as the mean ± 2.5 to 3.0 times the standard error of the mean. The probability of false rejection then should be 1% or less. Use of narrower limits will increase the false rejections. Use of wider limits will decrease false rejections and also error detection. Error detection increases with narrowing control limits (and high N), and shifts as small as 1 s a are detectable if desired. For many analytes, however, errors of this magnitude may not be clinically significant. The control limits and N thus should be chosen to provide appropriate sensitivity. The choice of truncation limits will depend on the population being tested. In populations without outliers, the best error detection is achieved with the widest limits. However, most populations contain some outliers that cause patient results to shift into the truncated range whenever an analytic shift occurs. Narrower limits then provide better performance. The selection of truncation limits therefore will depend on the population of the particular analyte to which the AON procedure is being

7 498 CEMBROWSK.I, CHANDLER, AND WESTGARD A.J.C.P. April 1984 GLUCOSE N 1 00 SODIUM, N SYSTEMATIC ERROR (S a ) B SYSTEMATIC ERROR r 5 I5 a ) FIG. 6. Effect of truncation limits for a population that has 5% outliers 2.58 Sp from x p. p fr is increased from Based on these studies and observations, we would recommend the following approach for establishing AON procedures: (1) Collect consecutive patient data over sev- eral weeks, and plot a frequency histogram of these data. applied. With a predominately healthy population, such as seen in health screening, outpatient clinics, etc., the truncation limits can be set to 3.0 Sp. With a hospitalized population, they should be set narrower, at 2.5 Sp or so. GLUCOSE N 1 00 SODIUM N I * POPULATION BELOW 2.58 SP.?.6- * POPULATION BEL0H 2.58 SP <C O C 0_.4-3H * 1 0* TRUNCATION m o Ql. t I * 10' TRUNCATION 2.58 SP SYSTEMATIC ERROR T" S (S a ) B S SYSTEMATIC ERROR (S a ) FIG. 7. Effect of percentage of outliers immediately outside the truncation limits (p rr = 0.01).

8 Vol. 81 -No. 4 AVERAGE OF NORMALS 499 When the distribution appears to be stable, estimate the bounds of the central area. (2) Characterize the patient population by using the data within the central region to calculate the mean (x p ) and standard deviation (Sp). (3) Determine the analytic imprecision by calculating Sa from a control product whose mean concentration is close to that of x p. (4) Determine the approximate number of patient values (N) to be included in the average. First calculate Sp/s a. Then use the nomogram in Figure 2 to estimate the minimum N required to detect a 2 s a shift with a probability of (Larger N will detect smaller errors at P = 0.50, or 2 s a shifts at P > 0.50.) (5) Select truncation limits after inspection of the frequency histogram of the data. Wide truncation limits (3.0 Sp) should be considered when there are few outliers. Narrower limits (2.5 Sp) should be used when there are many outliers. (6) If the number of patient results required, N, is greater than the number of specimens in a typical analytic run, the results from two or more runs must then be combined. If this is the case, the AON procedure will be useful only for retrospective QC. If the number of patient results is less than or equal to the number in a typical run, the AON procedure may be useful for monitoring analytic performance. (7) Select the control limits so that the probability of false rejection is 1%. If power functions are available, adjust N and the control limits so that clinically insignificant error is detected with low probability. Computerized implementation is necessary for practical applications of AON procedures. Moving averages or exponentially smoothed means 413 may be advantageous since they provide continually new estimates of the means. Caution should be exercised in applying AON information when patient populations are unstable. For example, it might not be possible to use AON on weekends in acute care hospitals when very few "normal" patients remain. Also, exclusion of certain units or wards may be desirable but would require implementation on a laboratory computer system. Proper application of AON procedures is seen to be a cumbersome task. Performance can be improved by following the general recommendations outlined here. Optimal performance probably requires detailed simulation studies for each analyte and method. Such studies are beyond the resources of most clinical laboratories, thus, optimized applications are not likely without further research and commercially available software, for both simulations and also for implementation. Given the difficulties in developing optimized AON procedures, the role of these procedures should be secondary, or supplementary, at least for analytic methods where stable control materials are available. These procedures should be useful in monitoring analyzers in which few controls are analyzed relative to the number of patient specimens. AON may be useful to extend the monitoring to prelaboratory variables. The use of such procedures may take on considerably more importance when stable control materials are lacking. References 1. Amador E, Hsi BP, Massod MF: An evaluation of the "average of normals" and related methods of quality control. Am J Clin Pathol 1968; 50: Begtrup H, Leroy S, Thyregod P, Walloe-Hansen P: "Average of normals" used as control of accuracy, and a comparison with other controls. Scand J Clin Lab Invest 1971; 27: Buttner J, Borth R, Broughton, PMG, Bowyer RC: Quality control in clinical chemistry. J Clin Chem Clin Biochem 1980; 18: Cembrowski GS, Westgard JO, Eggert AA, Toren EC: Trend detection in control data: optimization and interpretation of Trigg's technique for trend analysis. Clin Chem 1975; 21: Hoffmann RG, Waid ME: The "average of normals" method of quality control. Am J Clin Pathol 1965; 43: KJlgarifT M, Owen JA: An assessment of the "average of normals" quality control method. Clin Chim Acta 1968; 19: Leclercq R, Mascart P: Improved quality control of blood acid-base equilibrium using the daily mean of patients' results. Am J Clin Pathol 1977; 68: Lewis PW, Dixon K: Action limits for internal quality control. Clin Chim Acta 1971; 35: Owen JA, Campbell DG: Laboratory quality control using patients' results. Clin Chim Acta 1968; 20: Reed AH: Use of patient data for quality control of clinical laboratory tests. Clin Chem 1970; 16: Undrill PE: Computer assisted methods of organisation quality control and performance control in the clinical chemistry laboratory. Med Lab Sci 1980; 37: Westgard JO, Groth T: Power functions for statistical control rules. Clin Chem 1979; 25: Westgard JO, Groth T: Design and evaluation of statistical control procedures: applications of a computer "quality control simulator" program. Clin Chem 1981; 27: White JD: Use of patient data in the control of urea, creatinine and electrolyte estimations. Clin Chim Acta 1978; 84: