Analytical error of home glucose monitors: a comparison of 18 systems

Transcription

1 Original Article Ann Clin Biochem 1999; 36: Analytical error of home glucose monitors: a comparison of 18 systems Roger N Johnson and John R Baker From the Department of Clinical Biochemistry, Green Lane and the National Women's Hospitals, Auckland, New Zealand SUMMARY. In a quality review of 18 home blood glucose monitors, we measured the imprecision and incidence of significant error of 12 colorimetric and six amperometric systems by using capillary blood specimens from patients attending diabetes clinics. Imprecision at the mean glucose concentration found in the respective blood specimens (about 9 mmol/l) gave coefficients of variation (CV) ranging from 5 2 to 22 8%. Eight monitors including five of amperometric design had a CV of less than 10%. The incidence of significant error (defined as the proportion of specimens differing in value by more than 15% from a reference hexokinase assay of glucose in capillary blood) varied from 6 to 76%. Among the eight monitors identified as being most precise, the majority produced results that differed markedly from the reference assay, underlining the need for a common approach to calibration of home glucose monitors. Additional key phrases: blood glucose self-monitoring; blood glucose analysis In recognition of the importance of selfmonitoring of blood glucose (SMBG) to the management of diabetes mellitus, the American Diabetes Association and other expert groups (collectively abbreviated ADA) met in 1986 to reach a consensus view on its application and performance.' They proposed analytical goals for devices used for 5MBG, namely 'a variability (system plus user) of < 10% at glucose concentrations of 3{}-400mg/dL ( mmol/L) 100% of the time', and 'with current systems, 5MBG measurements should be within 15% of the results of the reference method'. Later, at a second conference in 1993, they recommended tightening the limits on 'analytic error' to within 5%.2 This more stringent guideline came at the same time as the clear benefits of tight control of blood glucose concentration were shown by the Diabetes Control and Complications Trial.' Correspondence: Dr R N Johnson, Department of Clinical Biochemistry, National Women's Hospital, Private Bag 92\89, Auckland 3, New Zealand. nwbiochem@ahsl.co.nz Whether monitors are able to match these recommended performance standards is uncertain. Earlier surveys such as those by Koschinsky et al. in and Havlin et al. in 199)5 did not find any monitors that reached the ADA goals of Since that time, manufacturers have made monitors easier to use and less dependent on the skill of the operator, but whether those currently available are better able to meet the guidelines remains in doubt. In a previous paper.f we described a wide variation in inaccuracy or calibration bias in 17 different monitoring products, and commented on the difficulties that this factor alone may cause in the interpretation of results and the consequent management of diabetes. In the present paper, we describe measurements of imprecision and the incidence of significant error in many of these same 17 systems. We compare our findings with the recommendations made by ADA,l and discuss where the state of the art stands in relation to these putative analytical goals of a decade ago. We comment on the difficulty of applying the ADA guidelines in practice and re-emphasize the need for a common basis for calibration and reporting, 72

2 Analytical error ofhome glucose monitors 73 either as a concentration of glucose in whole blood or in plasma. PATIENTS AND METHODS The study was approved as a clinical audit by the Ethics Committee of the Northern Regional Health Authority of New Zealand. As described in detail previously," patients attending two diabetes clinics were invited to participate. Upon their agreement, each had a single capillary blood specimen taken by one of four registered medical laboratory technologists. A portion of the blood was applied directly to the particular glucose strip under test, with the remainder collected in a fluoride micro-tube that was stored TABLE 1. Imprecision ofcolorimetric monitors Streck controls Manufacturer's controls Regression" Monitor Concentration! (n)l CV (%) Concentration! (n)l CV (%) Concentration! (n)l CV (%) One Touch 4 2 (6) (6) (51) 5 2 Profile 9 3 (6) (6) 7-6 SureStep 4 9 (8) (8) (57) (8) (8) 2-9 Accutrend 3 7 (13) (13) (76) 6 5 Alpha 9 0 (13) (13) (13) 3 5 Accutrend 3 9 (15) (15) (55) 10-4 Maxi 9 3 (15) (15) (15) 5 9 Reftolux S 4 3 (9) (9) (58) (9) (9) (9) 1.6 BM-I (10) (55) (10) (10) 14 0 Accutrend 3 9 (10) (10) (52) 12 5 Mini 9 1 (10) (10) (10) 3-8 Glucometer (10) (10) (62) (10) (10) (10) (10) 7 1 Glucometer (10) (10) (57) (10) (10) (10) (10) 5 6 Betachek 3 9 (11) (58) (11) (11) 17 2 Lynx 5 0 (10) (52) (10) (10) 16 9 Dia-Screen 4 1 (12) (52) (12) (12) 14 8 YSI ' (14) ' (16) (55) ' (14) ' (25) 1 3 CV from regression = SEE/mean [glucose] x 100%. tmean [glucose] observed (mmol/l), In = number in sample. 'Aqueous glucose solutions. YSI = the dedicated laboratory glucose analyser. Ann Clin Biochem 1999: 36

3 74 Johnson and Baker immediately at - 20 C. A rmmmum of 50 patients, representing about five clinic sessions irrespective of location, had glucose measurements made in this way with a particular monitor. The validity of the preservation technique was assessed by comparing results on the frozen capillary specimens with those from a dedicated laboratory glucose analyser (YSI 2300, Yellow Springs Instrument Co., Yellow Springs, OH, USA) used with fresh blood at one of the clinics as if it were a home glucose monitor. The monitors were also tested at each collection time with control solutions, both Sugar-Chex (Streck Laboratories, Omaha, NE, USA) from a single batch and, where available, manufacturer's controls. The monitors tested were those listed previously," with the following modifications: the Glucometer Elite (Bayer) has been superseded by the Esprit; Glucocard (Kyoto Daiichi Kagaku) by Glucocard II; and the Advantage (Boehringer-Mannheim) results reported here are solely for the newer strip based on glucose dehydrogenase (previously referred to as Universal Strip"), One Touch Profile and SureStep (both reflectance devices from Lifescan) have been made available to us since the completion of the previous report. Laboratory analysis was done after thawing the frozen blood specimens and deproteinizing them promptly with dilute perchloric acid (final concentration 0 3 mol/l), Glucose in the acid extracts was assayed with a reagent based on hexokinase and glucose-6-phosphate dehydrogenase (Boehringer-Mannheim, Mannheim, Germany) in a centrifugal analyser (Cobas Bio, Hoffman-La Roche, Basle, Switzerland). Comparison data were analysed statistically by least-squares linear regression. The results obtained by this technique did not differ from those of orthogonal regression when sample analyses were carried out with the least imprecise and most imprecise monitors. Frequency distributions were analysed by X 2 test and differences in paired data by Student's r-test, RESULTS Imprecision of measurements with glucose monitors Conventional imprecision statistics at particular concentrations of glucose were calculated after using Streck control solutions and, where provided, controls from the manufacturer of the monitor (Tables I and 2). The mean concentrations found for the Streck solutions TABLE 2. Imprecision of amperometric monitors Streck controls Manufacturer's controls Regression" Monitor Concentration" (n)1 CV (%) Concentrationt (n)1 CV (%) Concentrationt (n)1 CV (%) Glucocard II 2 6 (6) (6) (54) (6) (6) (6) 2 7 Advantage 3-4 (10) (10) (51) (10) (10) (10) 3 5 Esprit 1 8 (8) (8) (55) (8) (8) (8) (8) 1 8 Companion (14) (75) (14) (14) 5 4 Sensorex 3-8 (10) (10) (52) (10) (10) (10) 5 5 ExacTech 4 2 (10) (10) (51) (10) (10) 5 3 >25 (10) _tt *CV from regression = SEE/mean [glucose] x 100%. 'Mean [glucose] observed (mmol/l), In = number in sample. tt All measurements outside the measuring range. Ann cu«biochem 1999: 36

4 Analytical error ofhome glucose monitors 75 differed among the monitors, possibly reflecting in part disparities in accuracy' and in part a variable response to glucose presented in an unusual matrix. To the extent that comparison was possible, imprecision as judged from the Streck solutions seemed broadly similar to that obtained with the manufacturers' own controls. Imprecision of measurement of the patients' blood specimens was assessed from the dispersion of points about the line made by regressing the values from each monitor on the corresponding laboratory whole-blood glucose results. It was expressed numerically as the standard error of estimate (SEE, Sy.xy.' Coefficients of variation calculated as the SEE(mean monitor result (Sy.x(y) gave imprecision estimates that tended to be greater than those obtained with the control solutions (e.g., Streck mid-range control) at comparable mean values. Such an outcome might be anticipated because the blood specimens provided a wider range of glucose concentration than did any single control, and probably offered additional variability in such factors as viscosity, haematocrit, oxygen tension and interfering substances. In these respects the estimates of imprecision based on blood specimens more nearly represent day-to-day imprecision in actual use. Proportion of measurements with significant error The incidence of significant error in measurement was assessed after direct comparison of the result from the monitor with that from the laboratory on the corresponding capillary blood. The difference between the two values was expressed as a proportion by dividing by the laboratory result: error = (monitor result - hexokinase result)( hexokinase result The ADA Consensus Panel of 1986 recommended that results from monitors should fall within 15% of the target value set by the laboratory method: I 100 (monitor result - hexokinase result)( hexokinase result < 15 Table 3 lists the proportions of patient specimen results that exceeded this limit. It is evident that by this criterion all but a few monitors appeared to be in substantial error. Components of error in relation to the ADA guideline The relative contributions of imprecision and inaccuracy to the total error can be inferred by TABLE 3. Incidence ofsignificant error Rank Monitor Error (%). I YSI2300 t 0 2 Sensorex 6 3 SureStep 7 4 Accutrend Maxi 9 5 Esprit 15 6 Accutrend Alpha 16 7 Companion ExacTech 25 9 Glucometer Lynx Accutrend Mini Refiolux Betachek Advantage BM-I Glucometer One Touch Profile Dia-Screen Glucocard II 76 10" 13" DB Bias(%) 19't, 17 B FIGURE I. Bias and imprecision of monitors. The statistics on imprecision and on bias are from Tables 1, 2 and 4. The monitors are as ranked in Table 3: 1 = YSI 2300 (the dedicated laboratory glucose analyser); 2=Sensorex; 3=J&J SureStep; 4=BM Accutrend Maxi; 5 = Bayer Esprit; 6 = BM Accutrend Alpha; 7= Medisense Companion 2; 8= Medisense ExacTech; 9=Bayer Glucometer 2; 10=NDP Lynx; ll=bm Accutrend Mini; 12=BM Reflolux; 13=NDP Betachek; 14=BM Advantage; 15=BM-144; 16=Bayer Glucometer 3; 17=J&J One Touch Profile; 18=Dia Screen; 19 = KDK Glucocard II (see text for manufacturers' details). The circles indicate the upper bounds on bias and imprecision that are consistent with 95% of results falling within 15% ofthe target. Ann Clin Biochem 1999: 36

5 76 Johnson and Baker TABLE 4. Bias of monitors Value for Value for hexokinase monitor Difference Monitor (mmol/l)" (mmol/l)" (mmol/l) Error (%)1 pi SureStep < Esprit < Glucocard II < One Touch Profile < Sensorex! Maxi! YSI23()() < ExacTechi Betachek! Mini! Lynx! Reflolux! Glucometer 2! Companion 2! < Alpha' < BM-I < Advantage! < Glucometer < Dia-Screenl < *Mean of patients' specimen data. t[(value for monitor)-(value for hexokinase)]/(value for hexokinase) x 100%, sign ignored. IProbability that the Monitor and Hexokinase results are the same (by paired r-test), 'Results from Reference 6. TABLE 5. Predicted reading ofmonitors at defined glucose concentrations 95% confidence interval" At 4mmol/L At 17mmol/L Monitor Lower Upper Lower Upper YSI One Touch Profile H SureStep Accutrend Alpha Glucocard II Companion Esprit Sensorex Accutrend Maxi Advantage Reflolux Accutrend Mini Lynx BM-I Betachek ExacTech Glucometer Glucometer Dia-Screen *Expected deviations were calculated as bias± 2Sy.x at the selected concentration. "The YSI 2300 instrument was the dedicated laboratory glucose analyser. Ann Clin Biochem 1999: 36

6 Analytical error of home glucose monitors 77 plotting them as shown in Fig. 1. The imprecision for each monitor is the CV from regression (see Tables 1 and 2), and the bias is from Table 4. The monitors are numbered according to their ranking for significant error (see Table 3). Also shown are the allowable limits if results are to be within 15% of the target value on 95% of occasions, corresponding to 5% error in Table 3. Consistent with the information in Table 3, the YSI 2300 instrument met the 5% error limit and is included in the target area, whereas all others, apart from the SureStep, fell outside the target area. By applying a "i test to the data in Table 3, it was found that the Sensorex (6% error), SureStep (7%) and Accutrend Maxi (9%) did not differ significantly from the 5% error limit. It is evident from Fig. 1 that several of the monitors could have fallen within the allowable limits if calibrated differently, i.e., with less positive bias at the same degree of imprecision. Possible impact of total error on routine monitoring The combined effects of bias and imprecision on routine testing can be gauged from the predicted range of readings calculated as bias ±2Sy.x. The results shown in Table 5 are for a low and high concentration of glucose with the monitors ranked according to the extent of the greater (upper or lower) deviation at 4mmol/L. These values indicate that for a target concentration of 4 mrnol/l, the One Touch Profile can be expected to give a result (in about 95% of cases) of mmol/L. At the other extremity of performance, the Dia-Screen can be expected to give a result of mmol/l, While the extent of error at this low concentration may be overestimated by use of a constant estimate of error (standard deviation), difference plots" (not shown) did not suggest that a proportional estimate of error (CV) was more appropriate. DISCUSSION Meaning of the ADA guidelines Although the ADA guidelines'< ideally could be interpreted directly as analytical goals, the meaning of these recommendations in terms of the accepted analytical criteria of imprecision, inaccuracy and total error is unclear. The recommendation on variability being less than 10% might be interpreted as a CV of less than 10%. If this were the case, eight of 18 monitors in this study (Tables 1 and 2) would show acceptable performance. On the other hand, some authors have suggested that the recommendations imply an imprecision as a CV of less than 4% with no more than 5% bias,? targets approaching the performance of sophisticated laboratory analysers. Such performance is clearly beyond the capability of all the monitors evaluated in this study. Even these strict limits fall short when considering that they must embody the notion of '100% of the time'.' Similarly, while a majority of results might fall within 15% ofthe target value, it is unrealistic to expect every result to achieve this standard. We have proposed an alternative interpretation of the ADA guidelines by asking what degrees of imprecision and bias are consistent with 95% of values differing from the reference value by no more than 15%. This approach sets no explicit target for imprecision and bias in that the same total error can be achieved by various combinations of these parameters, with lesser precision being compensated by greater accuracy, and conversely (Fig. 1). However, upper limits are definable: with no imprecision, bias can be no higher than 15%, and with no bias, imprecision as CV can be no greater than 7 7%, very close to the CV of 7% recommended for these devices following interviews of patients and physicians. 10 Performance of the monitors against the ADA guideline While no monitor clearly fell within our interpretation of the guideline, three (Sensorex, SureStep and Accutrend Maxi) did not differ significantly from it and others (Esprit, Accutrend Alpha, Advantage, One Touch Profile and Glucocard II) were of suitably low imprecision, suffering only because of a high bias against our whole-blood glucose assay (see Fig. 1). We have discussed previously" how the adoption by manufacturers of a common basis for calibration and reporting whether as glucose in whole blood, as determined here, or as glucose in plasma would simplify both the interpretation of results and decisions on patient management. Such an approach is straightforward to implement and might resolve much of the variation in bias. Reliability of the results Imprecision in the comparison method does not affect this conclusion, as may be appreciated from the imprecision statistics obtained with the YSI 2300 instrument: the aqueous control solutions giving CVs of 2 5% (at 2-4mmol/L) Ann Clin Biochem 1999: 36

7 78 Johnson and Baker and 1 8% (at 23 7 mmol/l) with the YSI (see Table 1) had CVs of 2 2 and 0 9%, respectively, with the hexokinase laboratory method." Assuming that the two methods of measurement are equally imprecise, the apparent CV of 2 1% achieved with capillary blood specimens on the YSI (see Table 1)will include an equal contribution from the laboratory method, resulting in an estimated CV of 1 4% for each. Taking this estimated imprecision of the laboratory method into account for the most precise monitor, the One Touch Profile, results in its CV being lowered from 5 2 to 5 0%. For less precise monitors, the decrement is progressively smaller so that imprecision of the ExacTech and Dia-Screen (CV of 19 0 and 22-8%, respectively) is unaffected. Improved performance with amperometric devices As a group, amperometric devices offered a decisive advantage: three of the six had a CV of less than 7 7%, our derived upper limit for imprecision, and five of the six were below 10% (see Table 2). Of the colorimetric devices in our study, only three of the twelve offered comparable performance, while, as might be expected, most of the other non-wipe systems (Accutrends) tended to be more precise than the earlier systems that required wiping, the Dia Screen being an exception in this respect. This improvement in precision follows acceptance of the ADA recommendation that 'blood glucose monitors that are easier to use and less dependent on user skill should be developed';' The implication for those monitors with poor precision in this study is that they are insufficiently robust for routine use. The need for reconsideration of the ADA guideline It is clear that only the most modem meters can achieve the ADA recommendation of a decade ago and this only following a consensus among manufacturers on how meters should be calibrated and whether they should report whole blood or plasma glucose values. Whether the goal itself is unrealistic remains in doubt. Comparison with the acceptable performance criteria of the Royal College of Pathologists of Australasia-Australasian Association of Clinical Biochemists Quality Assurance Programme for glucose analysis in the laboratory is helpful in answering this question: this Programme accepts results within 1mmol/L of the target value up to a concentration of 10mmol/L and within 10% for target concentrations above 10mmol/L. Against the ADA recommendation of an error of less than 15% at all concentrations, the laboratory criterion is the more stringent for target concentrations of 6 7 mmol/l and above, whereas below 6 7mmol/L the situation is reversed. Although diabetic patients are understandably concerned about bypoglycaemia," it is clearly inappropriate to recommend a performance standard for home monitors that is more demanding than that for laboratory analysers. As noted above, the guideline also fails to allow for the natural dispersion of values in apparently stipulating that all monitor results should fall within 15% of the reference. We have suggested that 95% should reach this standard, but whether such a proportion satisfies the intention of the ADA requires clarification. This tolerance needs to be known in order to set the allowable limits on imprecision as illustrated in Fig. 1. CONCLUSION In conclusion, the present study shows that modem meters, and especially those ofampero metric design, are capable ofperforming close to the standard implied by the ADA's recommendations. However, there are two main reasons underlying their apparent failure here which may also affect evaluation of their performance in future: firstly, there is a lack of consensus among manufacturers on calibration and reporting of monitor results; and secondly, the ADA recommendation that results from meters should be within 15% of the laboratory result is without regard either to the concentration of glucose or to the imprecision of measurement. Both of these issues require urgent attention. Acknowledgements We wish to express our thanks to Pharmac for financial support, to Boehringer-Mannheim (NZ) for providing autolancets, to Medica Pacifica for making the YSI glucose analyser available to us; to the staff and patients of the Auckland Diabetes Centre and of the Diabetes Clinic of North Shore Hospital who made the study possible; and to Suzanne Boric, Suzanne Grapengiesser, Sarah Sargon and Barbara Smith for expert technical assistance. REFERENCES American Diabetes Association: Consensus statement on self-monitoring of blood glucose. Diabetes Care 1987; 10: 95-9 Ann cu«biochem 1999: 36

8 Analytical error ofhome glucose monitors 79 2 American Diabetes Association: Consensus statement on self-monitoring of blood glucose. Diabetes Care 1994; 17: The Diabetes Control and Complications Trial Research Group: The effect of intensive treatment of diabetes on the development and progression of long-term complications of insulin-dependent diabetes mellitus. N Engl J Med 1993; 329: Koschinsky T, Dannehl K, Gries FA. New approach to technical and clinical evaluation of devices for self-monitoring of blood glucose. Diabetes Care 1988; 11: Havlin CE, Parvin CA, Cryer PE. The accuracy of blood glucose monitoring devices. Clin Diabetes 1991; 9: Johnson RN, Baker JR. Accuracy of devices used for self-monitoring of blood glucose. Ann Clin Biochem 1998; 35: Westgard JO, Hunt MR. Use and interpretation of common statistical tests in method-comparison studies. Clin Chem 1973; 19: Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; i: Rainey PM, Jatlow P. Monitoring blood glucose meters. Am J cu«pathol 1995; 103: Weiss SL, Cembrowski GS, Mazze RS. Patient and physician analytic goals for self-monitoring blood glucose instruments. Am J Clin Pathol 1994;102:611-5 Accepted/or publication 23 June 1998 Ann cu«biochem 1999: 36