CHALLENGES IN SIX SIGMA IMPLEMENTATION JAKARTA, DECEMBER 7TH, 2017

Transcription

1 1 CHALLENGES IN SIX SIGMA IMPLEMENTATION JAKARTA, DECEMBER 7TH, 2017 STEN WESTGARD, MS WESTGARD QC, INC

2 3 WHY LISTEN TO THESE GUYS? Father knows best! Son knows better? The Westgard 40+ years at the University of Wisconsin Westgard Rules Method Validation Critical-Error graphs OPSpecs A Westgard 25 years at Westgard QC Publishing Web Blog course portal

3 WHY TAKE ADVANTAGE OF WESTGARD WEB? 4 Blog: >400 Short articles Q&A Website: >56,000 members >3 million views >600+ essays, lessons, QC case studies, reference, resources Course Portal: Training in QC, Method Validation, Risk Analysis, Quality Management

4 6 WHY WORRY ABOUT QC AND QUALITY? Manufacturer SD used for control limits All data within 2 SD. Too good to be true!

5 POOR QC = POOR PATIENT CARE 7 Clinical consequences of erroneous laboratory results that went unnoticed for 10 days Tse Ping Loh, Lennie Chua Lee, Sunil Kumar Sethi et al. J Clin Pathol March 2013, Vol 166, No test error 5 tests in error 63 results in error

6 THE RIGHT QC COULD HAVE CAUGHT THE ERRORS 8 49 patients Affected 4 procedures ordered in error (including CT Scan) 7 patients ordered for retesting 6 misdiagnoses 4 Control 1 Values 4 Control 2 Values

7 9 STAYING IN COMPLIANCE DIDN T KEEP THIS LAB (AND ITS PATIENTS) OUT OF TROUBLE CAP certified JCI certified 2004 Singapore Service Class award 2004 ISO certified Triple ISO certification ISO 9001 ISO ISO Awards and Awards and Awards We need Detailed QC HELP!

8 10 WE MUST WORRY NOT ONLY ABOUT BAD QC ALSO BAD METHODS For 2 YEARS, Mayo Clinic: about 5% of all IGF-1 tests were false positives. If the Mayo Clinic observations are generalized, a laboratory performing 1000 IGF-1 tests/month would be expected to generate around 50 falsepositive results each month. Some of these can be expected to lead to follow-up appointments or further testing and, ultimately, increased financial burden and anxiety for patients. UVA: 8-month period in 2011, 20 abnormally high IGF-1 results in 17 patients that did not agree with clinical findings. In 17 of the 20 samples, the IGF- 1 concentrations measured by a mass spectrometric method were within reference intervals. In 7 of the patients, expensive growth hormone suppression tests were done; the results were within reference intervals in 6, with the result in the seventh nondiagnostic.

9 2015: CLEVELAND CLINIC CATASTROPHE 16 $769,000 fine levied on Cleveland Clinic Marymount Hospital [P]atients in "immediate jeopardy," CMS says Cleveland Plain Dealer September 2015 Failing to run QC Using expired reagent PT/EQA cheating Lab closed for 4 months, 2 weeks; lab director fired CAP inspection in December 2014 missed these problems "Cobas 6000 Instrument Printouts: K- B2014: Sample K-06:. e) Date: 07/27/14 12:54:53 Handwritten notes stated: "COBAS 1 "RERUN" The ID number 6 and date and time were circled. The word rerun was underlined twice. Each numeric results listed below had a checkmark to left. Folate, Seru 1.50 ng/ml Ferritin L ng/ml Free T L ng/dl Quant Beta H H miu/ml TSH L uiu/ml Vitamin B L pg/ml T L ug/dl FT3 III 3.82 pg/ml"

10 WHY WORRY ABOUT A LACK OF TRUENESS IN ALKALINE PHOSPHATASE? 17 Evaluation of the trueness of serum alkaline phosphatase measurement in a group of Italian laboratories, CCLM 2016: aop Braga F, Frusciante E, Infusino I, Aloisio E, Guerra E, Ceriotti F, Panteghini M Instrument Low Bias Med Bias High Bias 1 Cobas Cobas AU Cobas Cobas 6000/ AU AU AU ADVIA Cobas Cobas Vista ARCHITECT c

11 WHY WORRY ABOUT A LACK OF TRUENESS IN 5 COMMON IMMUNOASSAYS? 18 Analytical Bias Exceeding Desirable Quality Goal in 4 out of 5 Common Immunoassays: Results of a Native Single Serum Sample External Quality Assessment Program for Cobalamin, Folate, Ferritin, Thyroid-Stimulating Hormone, and Free T4 Analyses Clin Chem 62: (2016) Kristensen GBB, Rustad P, Berg JP, Aakre KM Method Abbott ARCHITECT Beckman Coulter Unicel Roche Cobas Siemens ADVIA Centaur Cobalamin Meets Goal Fails Goal Meets Goal Meets Goal Folate Meets Goal Meets Goal Meets Goal Meets Goal Ferritin Meets Goal Fails Goal Fails Goal Fails Goal TSH Meets Goal Meets Goal Fails Goal Meets Goal Free T4 Fails Goal Fails Goal Fails Goal Meets Goal Free T4 (reference) Fails Goal Fails Goal Fails Goal Fails Goal

12 WHY ISN T THE FDA (OR SOMEONE) PROTECTING THE PATIENT FROM POOR QUALITY? 19 Conclusion 7-1. The 510(k) clearance process is not intended to evaluate the safety and effectiveness of medical devices with some exceptions. The 510(k) process cannot be transformed into a premarket evaluation of safety and effectiveness as long as the standard for clearance is substantial equivalance to any previously cleared device. Institute of Medicine 2011: Medical Devices and the Public s health: the FDA 510(k) Clearance Process at 35 years, prepublication copy

13 WHY IS THERE A GAP BETWEEN CERTIFICATIONS AND TRUE QUALITY ASSURANCE? 20 Systems without specifics Policies without performance data Aspirations without Achievement Assumptions that all tests are the same??? If we have a quality system without a quality method, without quality practices, are our patients really safe?

14 6 World Class Performance (3.4 DPM) WHERE DO WE START WITH SIX SIGMA? THE CONCEPTS Defects Per Million (DPM) Scale of 0 to 6 (Sigma short-term scale) Sigma is minimum for any business or manufacturing process (66,807 dpm) 5

15 24 Total Allowable Errors (TEa) PT/EQA groups CLIA RCPA Rilibak Biologic Variation Database Ricos Goals SIGMA VP PROGRAM HOW DO WE IDENTIFY THE PERFORMANCE SPECIFICATION?

16 25 HOW DO WE SELECT THE RIGHT PERFORMANCE SPECIFICATION? November 2014: Milan Meeting Draft Milan Consensus : Outcome-based, Clinical Use Goals (HbA1c NGSP guidance) 2: Biologic-based, Ricos Goals 3: State of the Art (CLIA, Rilibak, EQA/PT) Different analytes will get goals from Different models. Ricos Goals are not practical for all tests! In the future, many of these goals will become unreachable Supposedly not a hierarchy merely different models

17 A CONSENSUS ANSWER: SIGMA VP ESTABLISHES GLOBAL TEA GOALS 26 Analyte Recommended Total Allowable Error Goal (TEa %) TEa Source Recommended Critical Decision Level Albumin 10% CLIA Near 2.5 g/dl or 25 U/L Alk Phos 30% CLIA Near 150 U/L ALT 20% CLIA Near 95 U/L Amylase 30% CLIA Near 145 U/L AST 20% CLIA Near 40 U/L Bilirubin, Direct 44.5% Ricos Desirable Near 0.3 mg/dl or 5.13 umol/l Bilirubin, Total 0.4 mg/dl or 20% (greater) (6.84 umol/l or 20%) CLIA Near 3.0 mg/dl or 51.3 umol/l Calcium 1 mg/dl CLIA Near 13.0 mg/dl or 3.24 mmol/l (0.249 mmol/l) Chloride 5% CLIA Near 100 mmol/l Cholesterol 10% CLIA Near 180 mg/dl or 4.68 mmol/l CO2 25% CAP PT Near 25 mmol/l Creatinine Kinase (CK) 30% CLIA Near 275 U/L Creatinine 0.3 mg/dl or 15% (greater) CLIA Near 2.0 mg/dl or 176 umol/l (26.41 umol/l) GGT 22.11% Ricos Desirable Near 85 U/L Glucose 6 mg/dl or 10% (greater) CLIA Near 120 mg/dl or 6.66 mmol/l (0.33 mmol/l) HDL 30% CLIA Near 50 mg/dl or 1.23 mmol/l Iron 20% CLIA Near 75 mg/dl or 13.4 umol/l LDL 20% CLIA Near 100 mg/dl or 2.59 mmol/l LDH 20% CAP Near 170 U/L Lipase 37.88% Ricos Near 45 U/L Magnesium 25% CAP Near 2.5 mg/dl or 1.02 mmol/l Phosphorous 0.3 mg/dl or 10.7% (greater) CAP Near 4.0 mg/dl or 1.29 mmol/l (0.1 mmol/l) Potassium 0.5 mmol/l CLIA Near 2.5 mmol/l Total Protein 10% CLIA Near 5.7 g/dl or 57 mg/dl Sodium 4 mmol/l CLIA Near 115 mmol/l Triglycerides 25% CLIA Near 130 mg/dl or 1.46 mmol/l Urea Nitrogen 2.0 mg/dl or 9% (greater) CLIA Near 40 mg/dl or mmol/l (0.714 mmol) Uric Acid 17% CLIA Near 3.3 mg/dl or umol/l

18 27 HOW DO WE MEASURE (SIX) SIGMA PERFORMANCE? Measure Variation Use existing data Can we measure imprecision (CV)? Can we measure inaccuracy (bias)? Capture this data at critical medical decision levels

19 28 HOW DO YOU GET THE BEST CV AND BIAS? Imprecision 3 months or longer, third party controls, original manufacturer reagent Bias Your mean vs Peer group mean Your mean vs EQA/PT group mean Your mean vs the Package Insert/Target mean (Ideal but generally impractical: vs reference mean)

20 29 True Value HOW DO YOU ACTUALLY CALCULATE SIGMA METRIC? Sigma-metric = (TE a Bias)/CV - TEa + TEa Bias CV defects -6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

21 How do you select the appropriate level for Sigma? 3 levels of cholesterol method, Clin Chem July 2014 CLIA PT criterion for acceptability = 10% Total Precision (CV): 1.0% 0.9% 1.0% Bias : 3.0% 2.5% 2.3% Sigma = (10 3) / 1.0 = 7.0 / 1.0 = 7.0 Sigma = (10-2.5) / 0.9 = 7.5 / 0.9 = 8.3 Sigma = (10 2.3) / 1.0 Average Sigma = ( ) / 3 = 7.67 = 7.7 / 1.0 = 7.7

22 COMPARISON OF 6 COMPETITORS ON 8 CHEMISTRY ANALYTES 20 patient serum samples Comparison against reference methods or all-method-trimmed-mean Additionally, large laboratory effects were observed that caused interlaboratory differences >30%. There is a need for improvement even for simple clinical chemistry analytes. In particular, the interchangeability of results remains jeopardized by assay standardization issues and individual laboratory effects.

23 SIGMA EVALUATION OF RESULTS Test Abbott Beckman Ortho Roche Siemens Thermo Cholesterol Creatinine Glucose HDL LDL 5.41 n/a n/a Phosphate n/a Uric Acid Triglycerides Average Sigma-metric calculated of 3 levels measured Approximately 10 labs for each instrument CLIA goals used

24 STANDARDIZATION CONCLUSION Given conditions: achieving >6-Sigma performance or highest performance among competitors: Abbott: 6 of 8 analytes Beckman: 4 of 7 analytes Ortho: 3 of 7 analytes Roche: 3 of 8 analytes ThermoFisher: 3 of 7 analytes Siemens: 2 of 8 analytes

25 HOW DO YOU ASSESS SIGMA METRIC VISUALLY? 36

26 CS-6400 CHEMISTRY ANALYZER Evaluation of the CS-6400 Automated Chemistry Analyzer. Hyo-Jun Ahn, Hye-Ryun Kim, and Young- Kyu Sun, J Lab Med Qual Assur 2013;35: Assay TEa Level 1 CV% Bias% TEa Level 2 CV% Bias% Albumin 10% % 4.37% 10% % 10.26% Alk Phos 30% % % 30% % % AST 20% % 12.12% 20% % 10.13% ALT 20% % 18.65% 20% % 19.93% Amylase 30% % % 30% % % Bilirubin, Direct 44.5% % 12.93% 44.5% % 27.51% Bilirubin, Total 63.5% % 12.63% 20.0% % 5.04% Calcium 15.48% % 0.36% 10.54% % 6.17% Cholesterol 10% % 3.88% 10% % 4.63% Creatinine Kinase (CK) 30% % 13.07% 30% % 13.71% Chloride 5% % 0.05% 5% % 0.42% Creatinine 25.21% % 10.92% 15% % 0.29% GGT 22.11% % 32.92% 22.11% % 32.76% Glucose 10% % 6.53% 10% % 4.76% HDL 30% % 13.94% 30% % 15.66% Potassium 18.05% % 5.56% 11.42% % 2.55% Sodium 2.65% % 1.24% 2.84% % 1.64% Phosphate 10.7% % 2.29% 10.7% % 1.31% Total Protein 10% % 7.59% 10% % 8.26% Triglycerides 25% % 8.78% 25% % 18.98% Urea Nitrogen 9% % 19.85% 9% % 5.0% Uric Acid 17% % 8.14% 17% % 10.85% LDH 20% % % 20% % % Magnesium 25% % 9.70% 25% % 4.46% LDL 20% % 4.65% 20% % 6.31% Lipase 37.44% % 53.91% 37.44% % %

27 DISPLAY OF SIGMA-METRICS: NORMALIZED METHOD DECISION CHART (26 TESTS) 39

28 40 Importance of implementing an analytical quality control system in a core laboratory, Marques-Garcia F, Garcia-Codesal F, del Rosario Caro-Narros M, Contreras-San Feliciano T, Revista de Caldidad Asistencial 2015 Nov- Dec;30(6): methods

29 48 SMALL VOLUME CHEMISTRY ANALYZER 23 of 25 tests are >5 Sigma 92% of assays Kern Valley Hospital District (25 methods) >6 Sigma: Sodium Potassium Chloride Creatinine Total Protein Albumin Alk Phos ALT AST Total Bilirubin Direct Bilirubin Amylase Lipase CK Phosphorus Iron Uric Acid Cholesterol Triglycerides HDL

30 HOW DO YOU USE THE SIGMA METRIC TO IMPROVE YOUR LAB? THE OPSPECS CHART 49 Free download at

31 OPSPECS QC DESIGN FOR CS-6400 ANALYZER

32 COBAS 8000 QC DESIGN 51

33 53 QC DESIGN FOR KERN VALLEY

34 63 SIX SIGMA IS AN INCREASINGLY IMPORTANT TECHNIQUE FOR LABS AROUND THE WORLD IFCC committee on HbA1c standardization recommends Six Sigma as the way to judge assay quality. Clin Chem May;61(5): doi: /clinchem Epub 2015 Mar 3. Investigation of 2 models to set and evaluate quality targets for hb a1c: biological variation and sigma-metrics. Weykamp C 1, John G 2, Gillery P 3, English E 4, Ji L 5, Lenters-Westra E 6, Little RR 7, Roglic G 8, Sacks DB 9, Takei I 10 ; IFCC Task Force on Implementation of HbA1c Standardization. IFCC / EFLM committee on Pre-analytical and Post-analytical Quality Indicators recommends benchmarking on the Sigma scale Clin Chem Lab Med May;53(6): doi: /cclm Performance criteria and quality indicators for the pre-analytical phase. Plebani M, Sciacovelli L, Aita A, Pelloso M, Chiozza ML. Collective Opinion Paper 2013/2015 in Australian recommends adoption of Six Sigma metrics Collective Opinion Paper on a 2013 AACB Workshop of Experts seeking Harmonisation of Approaches to Setting a Laboratory Quality Control Policy Graham Jones John Calleja Douglas Chesher Curtis Parvin John Yundt-Pacheco Mark Mackay Tony Badrick Clin Biochem Rev 36 (3) 2015 Belgian EQA program offers Sigma-metrics to their participants Clin Chem Lab Med Feb 9. pii: /j/cclm.ahead-of-print/cclm /cclm xml. doi: /cclm [Epub ahead of print] Expressing analytical performance from multi-sample evaluation in laboratory EQA. Thelen MH, Jansen RT, Weykamp CW, Steigstra H, Meijer R, Cobbaert CM. Sigma-metrics calculated by Bio-Rad, Randox, MAS, and Technopath

35 HOW DOES IT IMPACT OUR DECISIONS? IFCC COMPARISON OF MAJOR HBA1C METHODS 64 Bias assessed using Accuracy-based materials across 8 levels of HbA1c Performance comparing Tosoh Bio-Rad Variant Turbo 2.0 Roche ARCHITECT enzymatic

36 65 HBA1C SIX SIGMA PERFORMANCE

37 HOW TO QC HBA1C 66

38 UNDERSTANDING THE IMPACT OF LABORATORY PERFORMANCE ON OUTCOMES IN A SCREENING POPULATION FOR CARDIOVASCULAR Wu L 1, Jülicher P 2, Liu L 3 DISEASE IN TAIWAN 1 ChiMei Medical Center, Tainan, Taiwan; 2 Abbott Diagnostics, Wiesbaden, Germany; 3 ChiMei Medical Center, Health Management Center, Tainan, Taiwan ISPOR 7 th Asia-Pacific Conference, 3-6 September 2016, Singapore Data were collected from 1,396 people (Age >=40 years) enrolled for CVD screening between January and April 2015 in Tainan (Table 1). A time-to-event microsimulation model was developed (Figure 1). Starting with screening, each individual was classified into risk categories based on observed values for LDL-, HDL-, total cholesterol, and a 10-years CVD risk score. Patients with observed values of LDL 190mg/L, 70<LDL<190 and a risk score 7.5%, and diabetic patients with LDL between 70 and 190 plus a risk score between 5 and 7.5% were referred for treatment. They received lipid lowering drugs thus reducing risk for a CVD event. Individuals not assigned to treatment remained in the No treatment - state until the next screening cycle after 1-4 years, the occurrence of a CVD event, or death. Minimum and optimum test specifications as suggested in literature were tested against a control scenario assuming perfect performances (Table 2). Samples were bootstrapped from the cohort with 100,000 iterations. Model followed a lifetime Table 2. Test horizon performance andof acomparing health strategies system perspective. Results were expressed in costs, quality adjusted life-years (QALY), Total and -C relative over- HDL-C or under-treatment. LDL-C Model was tested in 1- Strategy Bias, % CV, % Bias, % CV, % Bias, % CV, % Source way and probabilistic sensitivity analysis. Introduction Risk scores for cardiovascular disease (CVD) events based on laboratory values have been established in primary prevention programs [1]. The performance of laboratory test systems may lead to discordant treatment decisions in some cases. The goal of this study was to investigate the impact of laboratory diagnostic system performance on outcomes in a screening population for CVD in Taiwan. Mimimun (MIN) [16,17] Optimum (OPT) Control (CON) Assumption Results in terms of incremental values compared to the control scenario are summarized in Table 2. Analytical measurement uncertainty caused by CV and bias resulted in discordant management in some cases. The MIN and OPT strategy led to different decisions in 14.1% and 4.1%, respectively. Patients who had not received preventive treatment based on erroneous results had a higher risk for CVD events at an earlier time. The observed strategies showed a small but significantly increased number of CVD events and CVD related deaths compared to the control. MIN resulted in a loss of life years (LY) of 131 significant p. 1,000 for subjects MIN but not for OPT. Table 2. Incremental results from microsimulation. Figure 2. Incremental costs and QALY per strategy compared to the Control. Microsimulation with 100,000 samples. Mean, 95%CI of Δ Costs per patient, and Δ QALY per 1,000 subjects. Minimum strategy (MIN), Optimum strategy (OPT), Control strategy (CON) as defined in Table 2. *Per 1,000 individuals screened. MIN vs. CON OPT vs. CON MIN vs.opt Outcome value Mean (95%CI) Mean 95%CI Mean 95%CI Δ Costs per patient, NT$ 8753 (7516; 9990) 2075 (844; 3307) 6678 Δ QALY* The selection of high performance -56 diagnostic (-96; systems -17) plus -21 a strict quality (-60; 18) control management -36 in Methods Figure 1. Time-to-event microsimulation model structure Δ Total LY* -131 (-220; -43) -33 (-122; 55) -98 Results are limited to the information derived from the cohort. Risk scores may not accurately estimate the actual risk. Patient characteristics were sampled from distributions in order to reflect variability and uncertainty. The model assumed a stable lipid status over the time span of the simulation. Acknowledgement We thank Roger Low and Sten Westgard for motivation and fruitful discussions. Table 1. Characteristics of screening cohort (n=1,396) per 1,000 subjects (95%CI 43, 220), whereas a non-significant trend was observed for the OPT performance. Loss in QALY resulting from unnecessary treatment, earlier and increased risk for events, or increased mortality was found to be CVD related life-time costs were significantly higher compared to CON for MIN (+NT$ 8,753) and OPT (+NT$ 2,075). Conclusions Analytical measurement uncertainty may impose a higher risk for missing prevention opportunities. the laboratory conforming to the optimum specification is critical to consistently providing high and efficient quality of care. Limitations Variable Mean (SE) Sex (% women) 35.9 Age, years* 52.9 (40-85) BMI 24.2 (0.1) Pulse 72.9 (0.3) BPSys (mmhg) (0.5) BPDia (mmhg) 74.2 (0.3) GLUC_FAST (0.8) TCHOL (mg/dl) (1.0) HDL-C (mg/dl) 54.7 (0.4) LDL-C (mg/dl) (0.9) Results ( ; ) ; 3 ) ( ; ) ( Variable Value Distribution Source Setting, risk & events CVD risk (10 years) Risk based equations [1] In-hospital mortality from stroke 10.1 Beta [2] In-hospital mortality from AMI 6.5 Beta [3] Mortality from stroke at 1 year, % 12.0 Weibull [2] Mortality from MI at 1 year, % 6.0 Weibull [4] Mortality (non-cvd) Age-, sex-specific lifetable Weibull [5] Annual risk for recurrent CVD, % 6.8 Weibull [4] Risk reduction under treatment, % (95%CI) 65 (58, 73) Uniform [6] CVD event type (Stroke vs. MI), % 76 Beta [7] Smoking prevalence Male, % (95%CI) 31 (28.0, 35.2) Uniform [8] Smoking prevalence Female, % (95%CI) 3.4 (2.8, 4.2) Uniform [8] Laboratory results Mean from cohort Normal Assumption Screening cycle, years 1 to 4 Uniform Assumption Utility Baseline Beta [9] Lipid lowering treatment, Mean (SD) (0.001) Beta [10] MI (disutility), Mean (SD) (0.048) Beta [9] Stroke (disutility), Mean (SD) (0.039) Beta [9] Post MI (0.010) Beta [9] Post stroke (0.010) Beta [9] Costs, NT$ 1,2 Screening & visit 5,585 Uniform [11] Permanent lipid lowering treatment 15,639 Uniform [12] Stroke 74,832 Uniform [13] Post-stroke 45,630 Uniform [14] MI 189,497 Uniform [15] Post MI 82,897 Uniform [14] Annual discount rate for costs and utility, % 3.0 Table 3. Model input assumptions. (1) All costs adjusted for inflation with a 3% rate to 2015 New Taiwan dollar.(2) A range of ±25% was used to create upper and lower bounds. The Minimum and Optimum strategy led to higher costs compared to the Control for 32% and 27%, respectively (Fig. 3). As revealed from a sensitivity analysis, for each increase in percent point of CV, negative or positive bias 22, 43 or 7 individuals per 1,000 screened subjects would be either over- or under-treated compared to the control (Fig. 4, Tab. 3). Negative bias particularly increased the risk for denying preventive treatment, and would affect six times more patients than positive bias. Incremental costs per patients caused from discordant management decisions and related consequences would accrue Table 3. Impact of CV and bias on over- and under-treated individuals, and incremental to NT$786 (96%CI 734;838), NT$762 (612;912) and NT$699 costs. OT/ (668;729) UT: Over-/Under-treated; per percent Data increase based on linear in regression negative model. bias, Sensitivity positive analyses bias and assumed CV, respectively perfect %CV or (Tab. bias. 3). Variable Response Coeff. (95%CI) R-Sq, % (-) bias ΔUT, per 1,000 subjects 43.3 (40.6; 45.9) ΔCosts, NT$ 786 (734;838) (+) bias ΔOT, per 1,000 subjects 6.5 (5.6;7.3) References Figure 3. Distribution of incremental costs (IC) per strategy. Figure 4. Impact of increasing Bias and CV on discordant treatments. OT: Over-treated; UT: Under-treated. No significant deviation from the Control strategy was observed for both, the number of individuals over-treated with increasing negative bias, and the number of individuals under-treated with increasing positive bias 1. Stone NJ, Robinson JG, Lichtenstein AH, et al ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults. Circulation. 2014;129(25 Suppl 2):S Lee HC, Chang KC, Huang YC, et al. Readmission, mortality, and first-year medical costs after stroke. JCMA. 2013;76(12): Yin WH, Lu TH, ΔCosts, Chen NT$ KC, et al. The temporal trends of incidence, 762 treatment, (612; 912) and in-hospital mortality of acute myocardial infarction over 15years in a Taiwanese population. Int J Card. 2016;209: Chiang FT, Shyu KG, Wu CJ, et al. Predictors of 1-year outcomes in the Taiwan Acute Coronary Syndrome Full Spectrum Registry. J Formos Med Assoc. 2014;113(11): Ministry of Interior, Department of Statistics, Taiwan. Liftetable Accessed May 18, CV Taylor F, Huffman ΔUT, M, per Macedo 1,000 subjects A, et al. Statins for the primary 10.2 prevention (9.8;10.6) of cardiovascular disease. The Cochrane database of systematic reviews. 2013(1). 7. Cheng CL, Chien HC, Lee CH, Lin SJ, Yang YH. Validity of in-hospital mortality data among patients with acute myocardial infarction or stroke in National Health Insurance Research Database in Taiwan. Int J Card. 2015;201: Ng M, Freeman MK, Fleming TD, et al. Smoking prevalence and cigarette consumption in 187 countries, JAMA. 2014; 311(2): Kang EJ, Ko SK. ΔOT, A catalogue per 1,000 subjects of EQ-5D utility weights for 12.1 chronic diseases (11.8;12.5) among noninstitutionalized community residents in Korea. Value Health. 2009;12 Suppl 3:S Pandya A, Sy S, Cho S, Weinstein MC, Gaziano TA. Cost-effectiveness of 10-Year Risk Thresholds for Initiation of Statin Therapy for Primary Prevention of Cardiovascular Disease. JAMA. 2015;314(2): Lin YK, Chen CP, Tsai WC, Chiao YC, Lin BY. Cost-effectiveness of clinical pathway in coronary artery bypass surgery. J Med Sys. 2011;35(2): Cobiac LJ, Magnus ΔCosts, A, NT$ Barendregt JJ, Carter R, Vos T. Improving 699 the (668;729) cost-effectiveness of cardiovascular disease prevention in Australia: a modelling study. BMC Public Health. 2012;12(389). 13.Lien HM, Chou SY, Liu JT. Hospital ownership and performance: evidence from stroke and cardiac treatment in Taiwan. J Health Econ. 2008;27(5): Chung CW, Wang JD, Yu CF, Yang MC. Lifetime medical expenditure and life expectancy lost attributable to smoking through major smoking related diseases in Taiwan. Tob Control.

39 78 HEALTH ECONOMICS OUTCOMES OF OPTIMAL SIX SIGMA QUALITY A time-to-event microsimulation model was developed. Starting with screening, each individual was classified into risk categories based on observed values for LDL-, HDL-, total cholesterol, and a 10- years CVD risk score. Patients with observed values of LDL 190mg/L, 70<LDL<190 and a risk score 7.5%, and diabetic patients with LDL between 70 and 190 plus a risk score between 5 and 7.5% were referred for treatment. They received lipid lowering drugs thus reducing risk for a CVD event. Individuals not assigned to treatment remained in the No treatment - state until the next screening cycle after 1-4 years, the occurrence of a CVD event, or death. Minimum and optimum test specifications as suggested in literature were tested against a control scenario assuming perfect performances. Samples were bootstrapped from the cohort with 100,000 iterations. Model followed a lifetime horizon and a health system perspective. Results were expressed in costs, quality adjusted life-years (QALY), and relative over- or under-treatment. Model was tested in 1-way and probabilistic sensitivity

40 79 The Minimum and Optimum strategy led to higher costs compared to the Control for 32% and 27%, respectively (Fig. 3). As revealed from a sensitivity analysis, for each increase in percent point of CV, negative or positive bias 22, 43 or 7 individuals per 1,000 screened subjects would be either over- or under-treated compared to the control.negative bias particularly increased the risk for denying preventive treatment, and would affect six times more patients than positive bias. HOW SIX SIGMA METHODS IMPACT QC AND PATIENT OUTCOMES Incremental costs per patients caused from discordant management decisions and related consequences would accrue to NT$786 (96%CI 734;838), NT$762 (612;912) and NT$699 (668;729) per percent increase in negative bias, positive bias and CV, respectively

41 80 HOW SIX SIGMA METHODS IMPACT QC AND PATIENT OUTCOMES Results in terms of incremental values compared to the control scenario are summarized in Table 2. Analytical measurement uncertainty caused by CV and bias resulted in discordant management in some cases. The MIN and OPT strategy led to different decisions in 14.1% and 4.1%, respectively. Patients who had not received preventive treatment based on erroneous results had a higher risk for CVD events at an earlier time. The observed strategies showed a small but significantly increased number of CVD events and CVD related deaths compared to the control. MIN resulted in a loss of life years (LY) of 131 p. 1,000 subjects (95%CI 43, 220), whereas a non-significant trend was observed for the OPT performance. Loss in QALY resulting from unnecessary treatment, earlier and increased risk for events, or increased mortality was found to be significant for MIN but not for OPT. CVD related life-time costs were significantly higher compared to CON for MIN (+NT$ 8,753) and OPT (+NT$ 2,075). Figure 2. Incremental costs and QALY per strategy compared to the Control. Microsimulation with 100,000 samples. Mean, 95%CI of Δ Costs per patient, and Δ QALY per 1,000 subjects.

42 81 CLINICAL IMPACT SIMULATION MATCHES HEALTH ECONOMIC STUDY RESULTS Method differences result in risk SCORE misclassifications Method HDLc median Mmol/L CV% Bias Mmol/L Men Risk SCORE Wrong class SCORE>5 Abbott Beckman % Olympus Roche % Siemens % Laboratories supporting lipid clinics with a high proportion of specimens with atypical lipoproteins could observe discrepant results on certain specimens that might confound treatment decisions. It is vitally important for clinical laboratories to consider assay reliability and specificity when choosing methods, particularly in dyslipidemic samples.

43 100 CONCLUSION Six Sigma Implementation is not Simple, but it can be Simplified Use the Westgard Six Sigma tools to allow the laboratory to Identify the RIGHT method Select the RIGHT rules Run the RIGHT number of controls Run the controls at the RIGHT time and frequency next!

44 101 HOW DO WE MAKE THE SIGMA IMPLEMENTATION EASIER TO IMPLEMENT? SIGMA VP PROGRAM

45 102 THANK YOU FOR YOUR KIND ATTENTION! (QUESTIONS?)