Analytical and Clinical Performance of the epoc Blood Analysis System. Experience at a Large Tertiary Academic Medical Center

Transcription

1 AJCP / Original Article Analytical and Clinical Performance of the epoc Blood Analysis System Experience at a Large Tertiary Academic Medical Center Brie A. Stotler, MD, MPH, 1,2 and Alexander Kratz, MD, PhD 1,2 From the 1 Department of Pathology and Cell Biology, Columbia University College of Physicians and Surgeons, New York, NY, and 2 Clinical Laboratory Services, New York-Presbyterian Hospital, New York, NY. Key Words: Point-of-care testing; Blood gas analysis; Evaluation; epoc ABSTRACT Objectives: To describe validation and performance of epoc, a blood gas analysis point-of-care system, in a live clinical setting. Methods: Data were collected for 156 epoc systems over 12 months. Preimplementation precision and correlation studies and postimplementation quality assurance data were collected, including test card, reader, and personal data assistant (PDA) failure rates. Results: The coefficient of variation was clinically acceptable for all analytes. Correlation studies yielded an R 2 from (for sodium) to (for potassium) with the Nova analyzer and from (sodium) to (glucose) with the i-stat. Average test card failure rate was 13%. Of the PDA/reader units, 55% needed repair within 1 year. Conclusions: The analytical performance showed high precision and good correlation with the Nova and i-stat platforms. Test card and instrument failure rates were higher than that of the i-stat system. Blood gas analysis can be performed in a laboratory or at the point of care (POC). The advantages of central laboratory testing include higher accuracy and precision associated with the use of a few large, well-maintained instruments operated by highly trained laboratory professionals, as well as lower costs due to the cheaper reagents and more efficient use of labor. Transport delays, sample queuing, and the need to look up the results in the electronic medical record have led many hospitals to use POC devices for urgent samples. Although POC instruments provide much faster turnaround times than a central laboratory, the reliability of results can be compromised because of testing by clinical staff, whose primary expertise is not laboratory work. In addition, the cost per test is usually higher for POC assays due to higher reagent/cartridge costs. 1,2 Historically, i-stat instruments (Abbott Point of Care, Princeton, NJ) dominated the POC blood gas and electrolyte testing market in the United States. 3-5 The US Food and Drug Administration approved the epoc blood gas and electrolyte analysis system (Alere, Waltham, MA) in 2006 for testing many of the analytes available on the i-stat platform. The epoc instruments, which use similar technology to the i-stat, are being marketed as a costeffective alternative. In addition to cost, the advantages of epoc include built-in Wi-Fi connectivity and individually barcoded cards that can be stored at room temperature. 6 The analytical performance of the epoc blood analysis system was found to be comparable to the i-stat for ph, CO 2, O 2, sodium, potassium, ionized calcium, and hematocrit. 7 Our institution was among the first large academic medical centers to implement the epoc. This communication describes the performance of the epoc system in a large clinical Downloaded from Am J Clin Pathol 2013;140: Stotler_ indd 715

2 Stotler and Kratz / Analytical and Clinical Performance of epoc setting and its effects on the total use of blood gas analysis. We also extend the prior analytical evaluation to include glucose and lactate, two analytes that were not previously available on the epoc, and correlate the epoc results with central laboratory methods. Materials and Methods epoc Blood Analysis System The epoc system consists of test cards containing the sensors, a wireless card reader, and a personal data assistant (PDA) for data analysis. 7 The test cards use selective electrode potentiometry to measure ph, pco 2, sodium, potassium, and ionized calcium. Glucose, po 2, and lactate are measured amperometrically. Hematocrit is determined conductometrically. Bicarbonate, total carbon dioxide, base excess, oxygen saturation, and hemoglobin are calculated. 8 The epoc card reader is a battery-powered portable device with a barcode scanner. Insertion of a test card into the reader releases calibration fluid over the sensors. After completion of a 190-second calibration cycle, the user has seven minutes to apply a patient sample to the card. The reader then sends the sensor data via Bluetooth to the PDA. The PDA is a handheld computer that calculates analytical values from the data sent by the reader. The host displays the results and sends them via Wi-Fi to the hospital network. Evaluation Site Data were collected at the Columbia University Medical Center (CUMC) campus of New York-Presbyterian Hospital (New York, NY), a tertiary care academic medical center with more than 1,000 adult and pediatric inpatient beds. Active transplant, cardiac, and general surgical services; multiple intensive care units (ICUs); and two emergency departments require fast turnaround times for blood gas results. In 1995, the CUMC introduced i-stat POC devices for measuring blood gases, electrolytes, and hemoglobin in adult and pediatric ICUs, operating rooms, and the pediatric emergency department. Initially, two types of instruments were used: 140 blood analysis modules (BAMs) built into patient monitors in the adult ICUs and 52 portable i-stat clinical analyzers (PCAs) used in the pediatric emergency department, pediatric ICUs, and the operating rooms. Over time, the hospital discontinued using BAMs, and by 2008, only the portable PCAs were in use. Departments that did not use i-stat instrumentation sent samples requiring blood gas analysis to the centralized core laboratory, where they were analyzed with Nova CCX instruments (Nova Biomedical, Waltham, MA). In 2010, the hospital replaced the i-stat instrumentation with epoc blood analysis systems. In addition, epoc instruments were deployed in several locations that were not previously using i-stat devices for blood gas measurements. A total of 156 epoc systems were deployed over approximately three months. Each system consisted of a reader and a PDA. The vendor and the POC service staff trained more than 800 nurses, 300 physicians, 70 respiratory therapists, and 20 perfusionists. Training sessions lasted approximately one hour and included hands-on demonstrations of the instruments. Initial competency was assessed by a written quiz and direct observation of testing using quality control (QC) material. Competency was reassessed six months after initial training using direct observation of patient testing combined with an online review course and quiz. Evaluation Protocol Precision was assessed by analyzing three levels of aqueous QC material (Eurotrol BGEM control fluids [Eurotrol, Burlington, MA] and Mission Blood Gas and Electrolytes Control Calibration Verification Set [Diamond Diagnostics, Holliston, MA]) on 10 consecutive days. The precision data shown in this article are based on 37 readers. Three levels of QC material were run on each reader every day for 10 consecutive days, giving 370 measurements for each level of QC material. The means, standard deviations, and coefficients of variation were then calculated with Excel 2008 (Microsoft, Redmond, WA). Linearity studies were performed on all 156 instruments with five levels of commercially available linearity material (RNA Medical, Devens, MA), with three repetitions of each level. Acceptability criteria were a slope between 0.9 and 1.1 and a recovery between 90% and 110%. Correlation studies were performed on 15 randomly selected epoc systems, representing 10% of the analyzers purchased. Residual sample volume from patient samples collected for clinical analysis on Nova CCX instruments in the core laboratory was used for these studies. Samples collected in lithium heparin syringes were of arterial, mixed venous, and venous types. Approximately 40 patient samples were analyzed on each of the 15 epoc systems and on two i-stat instruments; results from the 15 epoc systems were compared with those from the i-stat analyzers and Nova CCX instruments. Quality Assurance Data After implementation of the epoc blood gas analyzer, the following quality assurance (QA) data were collected with the Electronic Data Manager (Alere) and analyzed in Excel 2008 (Microsoft): number of tests performed, number of tests reported in the electronic medical record, test card failure rate, and error codes associated with test card failures. 716 Am J Clin Pathol 2013;140: Downloaded 716 from Stotler_ indd 716

3 AJCP / Original Article The number of instruments replaced during the first year of use was tracked manually. Comparable QA data for i-stat had been collected previously with an i-stat Central Data Station. Data for the core laboratory were obtained with the laboratory information systems (Mysis, Tucson, AZ, and Cerner Millennium, North Kansas City, MO). Results Analytical Performance Precision studies were performed by analyzing three levels of QC material on 10 consecutive days on each epoc instrument. Summary data from 37 representative instruments are shown in Table 1. Coefficients of variation were generally close to the manufacturer s specifications and met clinical needs (0.1% for ph and up to 0.9% for sodium, 3.3% for pco 2, and 1.4% for potassium). Linearity studies showed recoveries between 90% and 110% on all 156 instruments (data not shown). Fifteen randomly selected instruments were correlated with the Nova CCX and i-stat instruments Table 2. The coefficient of determination for all analytes for both instrument comparisons was greater than The R 2 for all analytes ranged from (sodium) to (potassium) with the Nova instrument and from (sodium) to (glucose) with the i-stat instrument. The bias between the methods was assessed at two clinical decision levels for each analyte and met clinical needs. Test Card Failure Rate The epoc analyzer is an automated self-monitoring system. It monitors internal circuitry and reports problems on the display screen. Messages indicate the cause of the malfunction. Examples of user-generated errors include insufficient amount of sample, sample injected too early, or sample not introduced within seven minutes of calibration. Examples of instrument- or test card generated errors include lack of calibration fluid in the test card or failure of QC of the fluidics channels or sensors. Failure rates of the system were broken down into two main categories: (1) Card Failures, in which the entire test card failed and no assays were performed, and (2) Internal Quality Control (IQC) Errors, in which single or multiple analytes failed to pass the IQC and, thus, individual assays were not reported but other results were available. Table 3 shows the card failure rate of the epoc and i-stat cartridges over a 12-month period. The mean complete card failure rate was 13.0% for epoc and 4.7% for i-stat. IQC errors were less frequent than card failures for both the epoc and the i-stat systems. The mean IQC error rate for epoc was 3.1% during Table 1 Precision Data a Manufacturer s Analyte/Quality Specifications for Control Material CV, CV at Comparable Level Mean (SD) % Level of Analyte, % ph (ph units) CV (0.01) CV (0.01) 0.1 CV (0.01) pco 2, mm Hg CV (2.1) CV (0.9) 2.1 CV (0.7) po 2, mm Hg CV (6.1) CV (6.3) 5.7 CV (8.3) Sodium, meq/l CV (0.8) CV (1.2) 0.9 CV (1.4) Potassium, meq/l CV2 1.8 (0.0) CV3 4.2 (0.1) 1.3 CV4 6.2 (0.1) Ionized calcium, mg/dl CV2 5.6 (0.0) CV3 4.4 (0.0) 1.9 CV4 2.0 (0.0) Glucose, mg/dl CV (1.8) CV3 211 (8.0) 3.8 CV (15.7) Lactate, mg/dl CV2 4.5 (0.1) CV (0.1) 3.6 CV (0.4) Hematocrit, % H (0.4) H (0.5) H (0.6) 0.9 CV, coefficient of variation. a Summary data are presented for 37 epoc analyzers. Each level of quality control (QC) material was run on each analyzer on 10 consecutive days, resulting in 370 measurements per level of QC material. the 12-month study period; thus, for the 106,360 successful test cards used, approximately 3,780 cards had at least one IQC error for an analyte. The mean i-stat IQC error rate was 0.9% in the year before epoc implementation. Table 4 highlights the clinical units that had the highest and lowest overall failure rates using the epoc system. The average failure rate was highest in the adult emergency department (43% total failure rate) and in various ICUs. The lowest failure rates were in the children and adult hospital cardiovascular operating rooms (CVORs). The mean total failure rate in 2011 was 13% in the adult CVOR and 15% in the children s CVOR. In the CVOR locations, only perfusionists perform POC blood gas tests, whereas on all other units these tests are primarily performed by nurses, respiratory therapists, and/or anesthesiologists. The children s Downloaded from Am J Clin Pathol 2013;140: Stotler_ indd 717

4 Stotler and Kratz / Analytical and Clinical Performance of epoc Table 2 Correlation Studies of 15 epoc Systems With Nova CCX and i-stat Instruments No. of Decision Decision Instrument/Analyte Assays a Intercept (Slope) R 2 Level 1 Bias (95% CI) Level 2 Bias (95% CI) epoc vs Nova CCX ph (1.070) ( to 0.021) ( to 0.007) pco (1.073) mm Hg 7.8 (7.4 to 8.1) 50 mm Hg 8.9 (8.6 to 9.1) po (0.950) mm Hg 2.2 ( 3.0 to 1.3) 80 mm Hg 4.6 ( 5.2 to 4.1) Sodium (0.995) meq/l 0.0 ( 0.1 to 0.1) 150 meq/l 0.1 ( 0.3 to 0.2) Potassium (0.928) meq/l 0.14 ( 0.15 to 0.12) 5.8 meq/l 0.34 ( 0.35 to 0.33) Ionized calcium (1.106) mg/dl ( to 0.059) 1.4 mg/dl (.011 to 0.001) Glucose (0.863) mg/dl 1.5 (0.8 to 2.3) 180 mg/dl 17 ( 17.7 to 16.3) Lactate NA NA NA NA NA NA NA Hematocrit (1.289) % 1.86 ( 1.98 to 1.73) 55% 4.36 (4.04 to 4.68) epoc vs i-stat ph (0.949) ( to 0.018) ( to 0.028) pco (1.015) mm Hg 0.4 ( 0.7 to 0.2) 50 mm Hg 0.2 (.04 to 0.0) po (1.002) mm Hg 6.4 (5.7 to 7.1) 80 mm Hg 6.5 (6.0 to 6.9) Sodium (0.938) meq/l 1.4 (1.3 to 1.5) 150 meq/l 0.4 (0.3 to 0.6) Potassium (0.939) meq/l 0.03 (0.02 to 0.04) 5.8 meq/l 0.14 ( 0.15 to 0.13) Ionized calcium (1.015) mg/dl ( to 0.032) 1.4 mg/dl (.029 to 0.024) Glucose (0.959) mg/dl 2.6 (1.9 to 3.4) 180 mg/dl 2.9 ( 3.6 to 2.3) Lactate (0.949) mg/dl 0.2 (0.1 to 0.2) 3.4 mg/dl 0.0 (0.0 to 0.1) Hematocrit (1.087) % 1.41 (1.30 to 1.51) 55% 3.29 (3.02 to 3.55) CI, confidence interval; NA, not available (the Nova CCX does not perform lactate measurements). a Differences in the number of assays performed for each individual analyte are due to internal quality control errors, in which single or multiple analytes fail to pass the internal quality control and no results for the particular analyte is available. Table 3 Failure Rates of epoc vs i-stat Test Cards January- April- July- October- Test Card March % June % September % December % Annual % Rates of complete card failures (ie, no results reported from the card) i-stat epoc Rates of cards with at least one IQC failure (one or more analytes are not reported from a card) i-stat 1.0 NA epoc Sum of complete card failures and IQC result errors i-stat 5.4 NA epoc IQC, internal quality control; NA, not available. Table 4 Units With the Highest and Lowest Test Card Failure Total No. No. of No. of Tests With No. (%) of Combined of Test Successful at Least One IQC No. of Card Card Failures and Location Cards Used Tests Result Error a Failures b IQC Result Errors Adult ED (43) Neurology ICU 1,946 1, (23) Cardiac care unit 4,208 3, (22) Medical ICU 7,592 6, ,307 1,539 (20) Children s hospital respiratory therapists 1,496 1, (20) Adult hospital cardiovascular operating room 22,983 20, ,174 2,923 (13) Children s hospital ICU 21,100 18, ,368 2,993 (14) Children s hospital cardiovascular operating room 5,596 4, (15) ED, emergency department; ICU, intensive care unit; IQC, internal quality control. a IQC error can occur for single or multiple analyte(s) during test performance. No result for one or more individual parameters is given, but other analytes on the test card may be reported. b Card failure is failure of the entire test card in which no results for any analytes are reported. 718 Am J Clin Pathol 2013;140: Downloaded 718 from Stotler_ indd 718

5 AJCP / Original Article hospital ICU also had a low total rate of test card failure, averaging 14% for the year. The epoc card failure rate was significantly higher than the failure rate of the i-stat cartridges in the previous year. Because this was a new system, we instituted retraining programs to units with higher-than-average failure rates within 3 months of initial training; however, no appreciable decline in card failure rates was seen after secondary training. By the end of 2011, all units had used the epoc system for at least six months, yet the test card error rate remained high (only a 1.2% decrease over 12 months from 17% to 15.8%). Reader and Host Failure Rate A total of 156 PDAs and card readers were deployed throughout the institution. Each reader was paired with its own PDA; thus, if one of the two parts was malfunctioning, the entire unit would be returned to the biomedical engineering department. Over the course of 2011, 49 (33%) PDAs and 37 (24%) card readers needed to be repaired and/ or replaced for various reasons. Because our usage model considered the paired readers and PDAs as one unit, 55% of the original inventory of PDA/reader units needed to be repaired and/or replaced within one year of implementation. Common reasons for PDA failures included broken power connectors, damaged reset buttons, and host cradles breaking inside the reader. Common causes for reader malfunction included readers not powering on and failing to read barcodes. Investigation into the mechanical failures discovered multiple potential causes, such as corrosion inside the instruments, blood contamination inside the readers, improper host cradle assembly, and use of an inappropriate power supply leading to battery failure. Usage Data In 2010 before implementation of the epoc system, results of 87,532 blood gas panels were reported using the i-stat system. In 2011, 101,808 blood gas panels were reported with the epoc system, a 16.3% increase in POC blood gas testing. Interestingly, the number of blood gas panels reported from the core laboratory decreased from 103,098 panels in 2010 to 78,519 in 2011, a decrease of 23.8%. Overall, the total number of blood gas panels performed at the CUMC (both at the POC and in the core laboratory) decreased from 190,630 in 2010 to 180,327 in 2011, a decrease of 5.4%. Discussion The epoc blood analysis system showed high precision and good correlation with the i-stat platform for all nine directly measured analytes. This confirms previous findings showing good correlation of the epoc and i-stat systems for seven of these analytes (glucose and lactate were not previously available on the epoc platform). 7 We also found good correlation with the Nova CCX, a central laboratory blood gas instrument. However, there was a clear bias toward lower O 2 levels on the epoc and i-stat instruments relative to the Nova CCX results. Of 122,317 epoc test cards used at the CUMC during the first year after implementation, 15,957 (13%) failed. An additional 3,781 (3.1%) cards yielded only partial results because one or more analytes failed the IQC check (IQC errors). This compared unfavorably with a cartridge failure rate of 4.7% and an IQC error rate of 0.9% for the i-stat instruments during the previous year. In addition, 31% of the epoc PDAs and 24% of the card readers failed and had to be returned to the manufacturer for repair or replacement. The high card failure rate after implementation was expected because the epoc system was new to our institution and was also deployed at several sites within the hospital that did not previously perform POC blood gas testing. During the first year after implementation, the POC department retrained staff in units experiencing the highest card failure rates; however, retraining did not significantly decrease card failure rates. The highest card failure rates (20%-40%) continued to occur in areas that previously did not offer POC blood gas testing, confirming unsatisfactory user competency as a major contributing cause. Units that had a dedicated core group of testing personnel, such as the CVOR perfusionists, had the lowest failure rates (15%). Our usage model viewed each paired epoc PDA and reader as one unit and returned them together to the manufacturer for repair. Therefore, when 31% of our PDAs and 24% of our card readers failed during the first year, the failures affected 55% of our inventory of epoc systems. An alternate usage model could have been implemented whereby any PDA could be used with any reader. With this model, the overall number of damaged inventory at any given time would have been less since only the individual damaged component (ie, PDA or reader), and not both parts, would be returned for service. Had we instituted the latter model, the damaged equipment for the first year would have totaled 86 of 312 PDAs and readers, or 28% of the total inventory. Our hospital replaced 52 i-stat instruments with 156 epoc analyzers. This allowed the deployment of more analyzers to established sites of POC testing, as well as expansion of the POC program to new sites. This resulted in an increase in annual POC blood gas testing by 14,276 panels or 16.3%. The number of blood gas panels performed in the core laboratory decreased by 24,579 or 23.8%. This decrease can be directly attributed to the deployment of epoc analyzers to new POC sites that did not use the i-stat before epoc implementation. Downloaded from Am J Clin Pathol 2013;140: Stotler_ indd 719

6 Stotler and Kratz / Analytical and Clinical Performance of epoc Table 5 Factors That Can Influence the Outcome of a POC System Implementation Factor Clinical reliability of the method Cost of disposables Cost of instruments Availability of infrastructure required Previous experience of clinical staff with similar technology Appropriate training of staff Failure rate of test cards Failure rate of instrumentation Attitude of clinical staff toward the system Practical Examples Accuracy, precision, and correlation with other methods in use at the institution Test cards, reagents, calibrators, QC material Wi-Fi, data cables High test card failure rates lead to delays in diagnosis and care, additional costs for replacement cards, labor cost, and staff frustration High instrument failure rates can cause lack of availability of instrumentation and require additional efforts by POC and IT staff for validation and connection to network Willingness of clinical staff to assume responsibility for testing; willingness to trust the POC results IT, information technology; POC, point of care; QC, quality control. In summary, implementing and expanding a new POC blood gas testing program requires investment from all relevant hospital departments. Our experience provides examples of some factors that can determine the success or failure of such a project Table 5. Challenges faced during this implantation arose from a lack of user familiarity with the test system, a high card failure rate, and inventory supply problems due to the high PDA/reader mechanical failure rate. The POC department had to invest a significant amount of time retraining staff and orchestrating equipment repairs. Address reprint requests to Dr Kratz: New York-Presbyterian Hospital, Core Laboratory, 622 W 168th St, PH3-363, New York, NY 10032; ak2651@columbia.edu. Acknowledgments: We thank Steven Spitalnik, MD, for critical reading of the manuscript and Anthony Sireci, MD, for statistical advice. 2. Cox CJ. Acute care testing: blood gases and electrolytes at the point of care. Clin Lab Med. 2001;21: Schneider J, Dudziak R, Westphal K, et al. The i-stat analyzer: a new, hand-held device for the bedside determination of hematocrit, blood gases, and electrolytes [in German]. Anaesthesist. 1997;46: Dascombe BJ, Reaburn PR, Sirotic AC, et al. The reliability of the i-stat clinical portable analyser. J Sci Med Sport. 2007;10: Erickson KA, Wilding P. Evaluation of a novel point-of-care system, the i-stat portable clinical analyzer. Clin Chem. 1993;39: epoc Blood analysis Accessed November 23, Nichols JH, Rajadhayaska A, Rodriguez M. Evaluation of the enterprise point-of-care (EPOC) system for blood gas and electrolyte analysis. Point Care. 2008;7: Epocal. epoc System Manual. 6th ed. Ottawa, Ontario, Canada: Epocal; References 1. Holloway PAH. Point-of-care testing in intensive care. In: Kost GJ, ed. Principles and Practice of Point-of-Care Testing. Philadelphia, PA: Lippincott Williams & Wilkins; 2002: Am J Clin Pathol 2013;140: Downloaded 720 from Stotler_ indd 720