How Should Blood Glucose Meter System Analytical Performance Be Assessed?

Transcription

1 598599DSTXXX1.1177/ Journal of Diabetes Science and TechnologySions research-article215 Coentary How Should Blood Glucose Meter Syste Analytical Perforance Be Assessed? Journal of Diabetes Science and Technology 216, Vol. 1(1) Diabetes Technology Society Reprints and perissions: sagepub.co/journalsperissions.nav DOI: / dst.sagepub.co David A. Sions, MD 1 Abstract Blood glucose eter syste analytical perforance is assessed by coparing pairs of eter syste and reference instruent blood glucose easureents easured over tie and across a broad array of glucose values. Consequently, no single, coplete, and ideal paraeter can fully describe the difference between eter syste and reference results. Instead, a nuber of assessent tools, both graphical (eg, regression plots, odified Bland Altan plots, and error grid analysis) and tabular (eg, International Organization for Standardization guidelines, ean absolute difference, and ean absolute relative difference) have been developed to evaluate eter syste perforance. The strengths and weaknesses of these ethods of presenting eter syste perforance data, including a new ethod known as Radar Plots, are described here. Keywords accuracy, blood glucose onitoring syste, perforance, precision Blood glucose onitoring systes (BGMSs) are used by people with diabetes to onitor and anage their blood glucose levels, which ay prevent or delay coplications associated with poor glyceic control. 1,2 Correspondingly, a wide array of BGMSs are available to individuals with diabetes. The accuracy of results obtained fro BGMSs is iportant because they ay be used, for exaple, to calculate an appropriate insulin dose; inaccurate eter syste results can lead to insulin dosing errors, 3-5 and these errors have the potential to be clinically iportant, as shown by odeling analyses. 6 Thus, it is iportant to assess eter syste perforance and counicate these perforance results to the public. This coentary is fro the perspective of a diabetologist and soeone who works in the industry and ust counicate eter syste perforance inforation to regulatory agencies, health care payers, health care providers, and people with diabetes. What Is Meter Syste Perforance? This coentary focuses specifically on the analytical perforance of BGMSs, eaning the perforance attributable to the eter syste itself (as opposed to the user, the blood saple, or changing experiental conditions). To assess eter syste perforance, a laboratory reference instruent is used to easure the glucose concentration of a blood saple; this result is defined as the true glucose concentration. The BGMS is also used to easure the glucose concentration of the sae blood saple, and this result is then copared with that of the reference instruent. It is iportant to note that the assuption that the easureent taken with the laboratory reference instruent is the truth is inherently flawed, as error is also associated with this instruent (and the error inherent to the reference instruent becoes ore consequential as the BGMSs being tested becoe ore accurate and precise). 7 It is also iportant that the validity of the laboratory reference ethod used for coparison to the eter syste results should be established by onitoring the instruent s accuracy and precision using National Institute of Standards and Technology traceable seru controls 8 spanning the range of saples being tested. Multiple pairs of glucose easureents are ade over tie and across a broad array of glucose values. As a result, no single, coplete, and ideal paraeter can fully describe the differences between eter syste and reference results. Key easures of coparison include precision (the degree of dispersion of deviations fro the reference) and trueness (the average deviation of easureents fro the reference; Figure 1). 1 Bayer HealthCare LLC, Diabetes Care, Whippany, NJ, USA Corresponding Author: David A. Sions, MD, Bayer HealthCare LLC, Diabetes Care, Bayer Blvd, PO Box 915, Whippany, NJ 7981, USA. Eail: dr-david.sions@bayer.co

2 Sions 179 Increasing precision Low precision Low trueness High precision Low trueness Increasing trueness Low precision High trueness High precision High trueness Figure 1. Graphical representation of the concepts of precision and trueness. A circular target is used to illustrate precision (the degree of dispersion of deviations fro the center of the target) and trueness (the average deviation fro the center of the target). Multiple ethods have been developed to assess BGMS perforance data, each with its own strengths and weaknesses. These ethods and the inforation they convey are described here using anonyized data fro 3 eter systes to illustrate different perforance characteristics. Graphical Representations Regression Plots Perhaps the siplest ethod of presenting eter syste data is using Cartesian coordinates to plot the reference value on the x-axis and the BGMS value on the y-axis, for each pair of easureents, to generate a regression plot. 9 Ideally, the x- and y-values would be identical, and all data points would fall on the line y = x. In the exaples illustrated in Figure 2, the data points for eter A fall near the line at y = x; the data points for eter B fall within a broader area around the y = x line; and the data points for eter C, while in a relatively tight band, are not centered on the line at y = x. Using linear regression analysis, a best-fit line for a given dataset can be copared with the ideal scenario by generating a regression equation and R 2 value. A eter syste with higher accuracy, such as eter A, will have a slope closer to 1 and a y-intercept closer to copared with eter systes with lower accuracy, such as eters B and C. An R 2 value close to 1 indicates that the data are linear and precise (as seen in the plots for eters A and C); however, the R 2 value alone provides no inforation about accuracy. While both eters A and C have an R 2 value close to 1, eter A is ore accurate than eter C because it has uch higher trueness. It is also iportant to note that coparing R 2 values between different studies is difficult because the R 2 value is highly influenced by the range of glucose values. Accuracy guidelines fro the International Organization for Standardization (ISO) can also be incorporated in regression plots. For exaple, dashed lines can be plotted on the graph to represent ISO 15197:3 accuracy guidelines 1 (eter syste results within ±15 g/dl or ±2% of the ean reference result for saples with glucose concentrations <75 g/dl and 75 g/dl, respectively) or the updated ISO 15197:213 guidelines 11 (eter syste results within ±15 g/dl or ±15% of the ean reference result for saples with glucose concentrations < g/dl and g/dl, respectively). The ore rigorous ISO 15197:213 guidelines are represented in the regression plots as dashed lines that are tighter around the y = x line than those of ISO 15197:3. For both sets of ISO criteria, a greater nuber of data points for eter A fall within the lines copared with those of eters B and C. Using regression plots as a ethod of representing eter syste accuracy data has several advantages including enabling assessent of large aounts of data, quantification of the coparisons between eter syste and reference results, and visualization of extrinsic criteria (eg, ISO guidelines). However, a straightforward coparison of BGMS and reference results is ipractical due to the any paraeters generated using regression analysis (eg, slope, intercept, R 2 value). In addition, regression analysis does not allow for siple visual assessent of the intrinsic bias (ie, systeatic error) of a eter syste. Modified Bland Altan Plots Using odified Bland Altan plots, 12 the difference between eter syste and reference results is plotted on the y-axis, with reference results plotted on the x-axis. Ideally, all data points would fall on the line y = ; however, any intrinsic bias of a eter syste would be illustrated with data points trending either above or below the y = line (indicating positive or negative bias, respectively). For exaple, as illustrated in Figure 3, eters B and C both show a negative bias. Siilar to regression plots, ISO accuracy guidelines can be incorporated into odified Bland Altan plots, with the ore stringent ISO 15197:213 criteria represented by lines that are narrower in the vertical diension than those of ISO 15197:3. A strength of using odified Bland Altan plots to evaluate eter syste accuracy is that it allows for siple visual

3 18 Journal of Diabetes Science and Technology 1(1) y = 1.2x.16 R 2 =.997 y =.98x 9.9 R 2 =.973 y =.87x 4. R 2 =.991 ISO 15197:3 accuracy criteria ISO 15197:213 accuracy criteria Figure 2. Visualization of BGMS accuracy using regression plots. For each panel, the black diagonal line represents the ideal scenario (y = x); ISO criteria are plotted using dashed lines; the red line denotes the regression line; and the regression equation and R 2 value are shown in the top left corner of the plot area. BGMS, blood glucose onitoring syste; ISO, International Organization for Standardization; YSI, YSI glucose analyzer Difference fro Difference fro Difference fro ISO 15197:3 accuracy criteria ISO 15197:213 accuracy criteria Figure 3. BGMS accuracy visualized using odified Bland Altan plots. For each panel, the black diagonal line represents the ideal scenario (y = ) and ISO criteria are plotted using dashed lines. BGMS, blood glucose onitoring syste; ISO, International Organization for Standardization; YSI, YSI glucose analyzer. assessent of the degree to which eter syste results differ fro the reference; oreover, these plots allow for the detection of intrinsic eter syste bias (ie, systeatic error). However, odified Bland Altan plots theselves do not provide any nuerical easures to describe the data. Error Grid Analysis While regression plots and odified Bland Altan plots represent the analytical error associated with BGMSs, they do not convey the possible clinical significance of that error. The surveillance error grid (SEG) 13 is a tool used to assign a level of clinical risk associated with BGMS easureent inaccuracies. The SEG is a refineent of the Clarke error grid 14 and Parkes error grid 15,16 ; copared with the older grids, the SEG allows clinical risk to be quantified with greater precision, particularly for low levels of risk. Using error grid analysis, the data are plotted in the sae anner as in a regression plot, with reference results on the x-axis and eter syste results on the y-axis; however, the data overlay the error grid contained within the plot area (Figure 4). The SEG is color-coded to represent clinical risk, with the level of risk associated with each possible BGMS reference result pair having been epirically

4 Sions None Mild Moderate Risk level High Extree Figure 4. Surveillance error grid analysis to evaluate the clinical risk associated with BGMS inaccuracy. As shown in the legend, the level of clinical risk associated with each pair of BGMS and reference results is represented by a color on the grid. BGMS, blood glucose onitoring syste; YSI, YSI glucose analyzer. deterined using a survey of diabetes clinicians. 13 The grid encopasses 15 risk zones, but color-coding using a continuous spectru of color captures an even greater degree of nuance in SEG plots. The ability of error grid analysis to indicate the clinical significance of BGMS error is useful, as this is the ultiate consequence of eter syste error. However, error grid analysis specifically assesses acute clinical risk and not the risk associated with chronic eter syste iprecision. Because people with diabetes using intensive insulin therapy ay easure their blood glucose 6 to 1 ties or ore each day, 17 the potential consequences of even relatively sall eter syste errors on insulin dosing 3-6 could be copounded over tie. Tabular Representations International Organization for Standardization Guidelines As noted previously, ISO 15197:3 1 and ISO 15197: guidelines specify accuracy criteria for BGMSs (with eter systes required to have 95% of results within the liits defined above) and are used by regulatory agencies to assess eter syste accuracy. The nuber and percentage of BGMS results within defined error liits, including those specified by ISO, can be presented in tabular for. As shown in Table 1, 99.5% and 99.1% of results for eter A were within ISO 15197:3 and ISO 15197:213 error liits, respectively; thus, eter A surpassed both sets of ISO accuracy criteria. While ISO guidelines are a siple and effective easure of BGMS accuracy, the relatively arbitrary boundary between acceptable and unacceptable error likely oversiplifies eter syste perforance. Moreover, these guidelines do not specify any criteria for 5% of eter syste errors; thus, there is no liit to the error allowed in up to 5% of BGMS results. Mean Absolute Difference (MAD) and Mean Absolute Relative Difference (MARD) MAD and MARD 18 further siplify the tabular representation of eter syste error by quantifying an entire dataset with a single nueric value, either in ters of absolute error (MAD) or percentage error (MARD). As shown in Table 2, the MAD and MARD values for eter A were lower than those of eters B and C, indicating that eter A has a higher degree of accuracy than the other eter systes. However, although these analyses are useful for coparing ultiple eter systes in a single study, MARD is not a sufficient statistic and cannot by itself be used to describe eter syste quality. 19 Radar Plots Radar Plots are a new way to visualize BGMS analytical perforance. Using this ethod, differences between BGMS and reference instruent easureents are plotted using polar rather than Cartesian coordinates. While the sae inforation is plotted using a Radar Plot and a odified Bland Altan plot (ie, the difference between BGMS and reference results

5 182 Journal of Diabetes Science and Technology 1(1) Table 1. Assessent of BGMS Accuracy Using ISO Criteria (). Glucose concentration n Nuber of results within specified error liits <75 g/dl 5 ±5 g/dl ±1 g/dl ±15 g/dl ±2 g/dl 4 (8%) 5 (%) 5 (%) 5 (%) 75 g/dl 216 ±5% ±1% ±15% ±2% 168 (77.8%) 28 (96.3%) 214 (99.1%) 215 (99.5%) Total 221 ±5 g/dl or ±5% ±1 g/dl or ±1% ±15 g/dl or ±15% ±15 g/dl or ±2% a 172 (77.8%) 213 (96.4%) 219 (99.1%) 22 (99.5%) < g/dl 39 ±5 g/dl ±1 g/dl ±15 g/dl ±2 g/dl 3 (76.9%) 39 (%) 39 (%) 39 (%) g/dl 182 ±5% ±1% ±15% ±2% 143 (78.6%) 174 (95.6%) 18 (98.9%) 181 (99.5%) Total 221 ±5 g/dl or ±5% ±1 g/dl or ±1% ±15 g/dl or ±15% b ±2 g/dl or ±2% 173 (78.3%) 213 (96.4%) 219 (99.1%) 22 (99.5%) BGMS, blood glucose onitoring syste; ISO, International Organization for Standardization. The exaple data are representative of an ISO 15197:213 section 8 study and are shown only to illustrate the ISO accuracy criteria; coplete satisfaction of ISO would also require a section 6/7 study with saples in a prescribed distribution. 1,11 a ISO 15197:3 accuracy criteria. 1 b ISO 15197:213 accuracy criteria. 11 Table 2. MAD and MARD Analysis of BGMS Accuracy. Meter syste MAD (g/dl; N = 538) MARD (%; N = 538) BGMS, blood glucose onitoring syste; MAD, ean absolute difference; MARD, ean absolute relative difference. MAD and MARD are defined as follows: n 1 MAD = Meteri -Labi n i = 1 n 1 Meteri -Labi MARD = n Lab i = 1 i Where Meter = eter syste blood glucose easureent. Lab = laboratory reference blood glucose easureent. n = nuber of blood saples. MAD and MARD were coputed for all saples across the overall glucose range ( g/dl). vs reference results), the circular contours of constant error levels in Radar Plots add another diension to the visualization of the distribution of eter syste error. Figure 5 shows that the data points for eter A cluster around the center of the plot while those of eters B and C are generally further fro the center, indicating lower accuracy. Consistent with the previous visualizations of eter syste perforance, the data points for eter B are ore dispersed and thus less precise than those of eter C. As with regression plots and odified Bland Altan plots, ISO guidelines can be incorporated into Radar Plots (although as a circle as opposed to lines). Radar Plots siplify the presentation of eter syste perforance data by including ultiple easures of analytical perforance (accuracy [in ters of average error and based on ISO criteria] and precision) in a single graphic. While odified Bland Altan plots and regression plots provide soe of the sae inforation, Radar Plots do so in a ore intuitive fashion. Because of the Radar Plot s reseblance to a target, a tight clustering of data points in the center of the plot intuitively represents higher accuracy and precision. Moreover, Radar Plots can facilitate visual coparison of ultiple eter systes in a single study. However, as with any plot, the plot itself does not provide a nuerical easureent of eter syste perforance. Conclusions Although assessing BGMS perforance is iportant, it is not straightforward because pairs of blood glucose easureents fro a eter syste and reference instruent are copared over tie and using a variety of glucose concentrations. In addition, there are ultiple easures of eter syste perforance, including accuracy and precision. Given this coplexity, a variety of ethods have been developed to assess eter syste perforance, broadly divided into graphical and tabular representations. Each ethod has advantages and liitations that should be acknowledged when used to interpret data. Radar Plots are a new ethod of assessing BGMS perforance and are a valuable visual indicator to illustrate eter syste accuracy and precision, as well as to copare the analytical perforance of ultiple eter systes in a single study.

6 g g Sions YSI (g/dl) YSI (g/dl) YSI (g/dl) >35 g/dl 3 g/dl 25 g/dl 2 g/dl 15 g/dl 1 g/dl 5 g/dl 5% 1% 15% 2% 25% 3% >35% 15 3 g/dl 25 g/dl 2 g/dl 15 g/dl 1 g/dl 5 g/dl 5% >35 g/dl 1% % 15% 1 2 % % 25% 3% >35% 15 3 g/dl 25 g/dl 2 g/dl 15 g/dl 1 g/dl 5 g/dl 5% 15% 2 % % >35 g/dl 1% 25% 3% >35% g/dl or 1% Zone (15 g/dl or 15%) Figure 5. Visualization of BGMS accuracy using Radar Plots. Each data point represents the error of a eter syste result; error is given in units of g/dl for YSI results < g/dl (region of the plot within the agenta dashed lines) and in units of percentage for YSI results g/dl (region of the plot outside the agenta dashed lines). The outer green circle (bolder line) represents ISO 15197:213 accuracy criteria; data points within this circle (the Zone) eet these criteria. In a Radar Plot, the distance fro the center of the plot to a data point indicates the agnitude of the eter syste error. The direction of a line connecting the point to the center indicates the value of the laboratory reference easureent. If the point is above the horizontal idline, the error is positive (ie, the eter syste overestiated the laboratory reference blood glucose value). If the point is below the horizontal idline, the error is negative (ie, the eter syste underestiated the laboratory reference blood glucose value). If the laboratory reference blood glucose value is < g/dl (~5.55 ol/l), then the errors are expressed in g/dl (or ol/l). Otherwise, the errors are expressed as a percentage of the laboratory reference easureent. BGMS, blood glucose onitoring syste; YSI, YSI glucose analyzer. Abbreviations BGMS, blood glucose onitoring syste; ISO, International Organization for Standardization; MAD, ean absolute difference; MARD, ean absolute relative difference; SEG, surveillance error grid; YSI, YSI glucose analyzer. Acknowledgents I would like to thank Scott Pardo, PhD, PStat ; Bern Harrison, BA; Robert Morin, MD; and Jane F. Wallace, BA, CCRA; of Bayer HealthCare LLC, Diabetes Care, Whippany, NJ, for their review of the technical details in this anuscript. Medical writing assistance was provided by Courtney St. Aour, PhD, of MedErgy, Yardley, PA, and was supported by Bayer HealthCare LLC, Diabetes Care. Declaration of Conflicting Interests The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DAS is a full-tie eployee of Bayer HealthCare LLC, Diabetes Care. Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Bayer HealthCare LLC, Diabetes Care, Whippany, NJ. References 1. UK Prospective Diabetes Study Group. Intensive blood-glucose control with sulphonylureas or insulin copared with conventional treatent and risk of coplications in patients with type 2 diabetes (UKPDS 33). Lancet. 1998;352(9131): Diabetes Control and Coplications Trial Research Group. The effect of intensive treatent of diabetes on the developent and progression of long-ter coplications in insulin-dependent diabetes ellitus. N Engl J Med. 1993;329(14): Boyd JC, Bruns DE. Quality specifications for glucose eters: assessent by siulation odeling of errors in insulin dose. Clin Che. 1;47(2): Raine CH III, Schrock LE, Edelan SV, et al. Significant insulin dose errors ay occur if blood glucose results are obtained fro iscoded eters. J Diabetes Sci Technol. 7;1(2): Virdi NS, Mahoney JJ. Iportance of blood glucose eter and carbohydrate estiation accuracy. J Diabetes Sci Technol. 212;6(4): Raine CH III, Pardo S, Parkes JL. Predicted blood glucose fro insulin adinistration based on values fro iscoded glucose eters. J Diabetes Sci Technol. 8;2(4): Bailey T, Klaff LJ, Wallace JF, et al. Fundaental iportance of reference glucose analyzer accuracy for evaluating the perforance of blood glucose onitoring systes (BGMSs). Diabetes. 215;64(suppl 1):A Neese JW, Duncan P, Bayse D, et al. Developent and evaluation of a hexokinase/glucose-6-phosphate dehydrogenase

7 184 Journal of Diabetes Science and Technology 1(1) procedure for use as a national glucose reference ethod. HEW Publication No (CDC) HEW USPHS, Centers for Disease Control and Prevention; Stockl D, Dewitte K, Thienpont LM. Validity of linear regression in ethod coparison studies: is it liited by the statistical odel or the quality of the analytical input data? Clin Che. 1998;44(11): International Organization for Standardization. ISO 15197:3(E): In vitro diagnostic test systes requireents for blood-glucose onitoring systes for self-testing in anaging diabetes ellitus. Geneva, Switzerland: International Organization for Standardization; International Organization for Standardization. ISO 15197:213(E): In vitro diagnostic test systes requireents for blood-glucose onitoring systes for self-testing in anaging diabetes ellitus. Geneva, Switzerland: International Organization for Standardization; Bland JM, Altan DG. Statistical ethods for assessing agreeent between two ethods of clinical easureent. Lancet. 1986;1(8476): Klonoff DC, Lias C, Vigersky R, et al. The surveillance error grid. J Diabetes Sci Technol. 214;8(4): Clarke WL, Cox D, Gonder-Frederick LA, Carter W, Pohl SL. Evaluating clinical accuracy of systes for self-onitoring of blood glucose. Diabetes Care. 1987;1(5): Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the easureent of blood glucose. Diabetes Care. ;23(8): Pfutzner A, Klonoff DC, Pardo S, Parkes JL. Technical aspects of the Parkes error grid. J Diabetes Sci Technol. 213;7(5): Aerican Diabetes Association. Standards of edical care in diabetes 215. Diabetes Care. 215;38(suppl):S1-S Tack C, Pohleier H, Behnke T, et al. Accuracy evaluation of five blood glucose onitoring systes obtained fro the pharacy: a European ulticenter study with 453 subjects. Diabetes Technol Ther. 212;14(4): DeGroot MH. Probability and Statistics. Menlo Park, CA: Addison-Wesley; 1975.