Setting of quality standards

Transcription

1 Setting of quality standards Graham Jones Department of Chemical Pathology St Vincent s Hospital, Sydney AACB ASM Adelaide October 2014

2 Setting of Quality Standards The 2013 QC workshop revealed that most Australian laboratories are not consciously adopting performance goals and many are not ensuring that their QC algorithms have high detection of critical errors. This is worrying to say the least. Quality Limits are numerical limits for a test outside which we do not wish to release results (only one of us had them)

3 NPAAC G4 Laboratories should set routine performance goals based on the clinical use of the test results.

4 Approach to Quality Specifications IFCC, IUPAC, WHO Consensus Conference, Stockholm

5 Stockholm Consensus Conference on Quality Specifications in Laboratory Medicine 1. Studies on clinical outcomes 2. Clinical decisions in general, data from: biological variation clinicians opinions 3. Published professional recommendations 4. Performance goals set by regulatory bodies or organisers of External Quality Assessment Schemes. 5. Goals based on the current state of the art as demonstrated by data from EQA or from current

6 Approaches Set / Select Quality Limits - design system to meet limits 6 sigma Capability Assess performance against standards (UM) Optimal, Desirable, Minimal Improve where needed Reverse Engineering Know how good you are

7 Quality Limits Eg: AST result should be within 20% of correct result (CLIA Guidelines) Questions arising 1. What is correct result? 2. How do we achieve this?

8 Capability Sigma Metric σ = Change/SD Important Change

9 Capability Sigma Metric σ = 6, good σ = 4, OK σ = 3, poor 2 SD spread Required shift detection 4 SD Shift 2 SD Shift 1 SD Shift Important Change

10 Capability Sigma Metric σ = Change/SD Important Change

11 Capability Sigma Metric σ = (Change-Bias)/SD Important Change

12 Power Function Graph (n=2) 1 2s 1 2.5s MR 1 3.5s

13 Quality Limits Meets process needs Set goals based on clinical need in advance Establish QC protocols based on: Defined need Assay Precision Issues: What are the limits to use? How do I handle Bias?

14 Assess performance against standards (MU) Run assay over a period of time Obtain Precision data from QC Compare performance against Highest level of Stockholm criteria Typically within-subject Biological Variation (Cvi) Level 2a Compare assay precision (Cva) against CVi

15 Precision Goals 40% Increase in total CV (%) 35% 30% 25% 20% 15% 10% 5% 0% Minimum (0.75, 25%) Optimal (0.25, 3%) Desirable (0.5, 12%) CVa / CVwi

16 Assess performance against standards (UM) Stockholm Level 2a CVa < 0.25 CVi - Optimal CVa < 0.5 CVi - Desirable CVa < 0.75 CVi - Minimal

17

18 UM Approach UM is required by ISO and NPAAC Assess performance using real data But Typically several month s QC data For new assay use limited run-in data Analysis performed AFTER setting up assay (can use manufacturers estimate of CVa) QC data may have outliers excluded If fails higher standard (eg BV), use lower limit (eg state of the art) Assessment not planning

19 Reverse Engineering Recommended process: Set goals, use processes to meet goals Reverse Assess processes to understand performance How bad may a result from my lab be?

20 Understanding our assays For any assay, with the QC protocol in place, we should be able to say how much analytical error may occur. These rules have the power to cause a STOP 90% of the occasions when there is a shift in the assay of 2.8 x LSD and cause a PAUSE 90% of the occasions when there is a shift in the assay of 2.6 x LSD. SydPath Quality Control SOP

21 So far Quality Limits UM Reverse Engineering

22 Quality Limits What Limits?

23 04/ae2004.php?B1=Chemistry +A- C&find=&start=1&NOLINKS= m/products/epevaluator/allowable-totalerror-table

24

25 RCPAQAP(%) CLIA (%) Range:

26 RCPAQAP ALP

27 Limits/acceptability

28 Meaning of ALP Basis Total Error Can share reference interval Imprecision Can Monitor across labs Level Optimal no need to improve Desirable satisfactory Minimal just satisfactory

29 Revision of ALP ALP are applied to Total Error Used in interim reports Single results include bias and imprecision Will use categories of CV: 1,2,3,4,5,6,8,10,12,15,20,25,30% Round to nearest category Change between absolute and percentage based on precision profile

30 Revision of ALP Top category (Imprecision): Within-Subject Biological Variation (Opt, Des, Min) Monitoring Single Laboratory reaching standard: Can monitor a patient at lab Many Labs within standard: Can monitor a patient between labs Can share reference intervals

31 Revision of ALP Next Category: Total Error (Opt. Des, Min) Within and between subject BV combined Diagnosis Single Laboratory reaching standard: Satisfactory performance Bias and / or precision target for improvement Multiple Labs meeting standard Likely to be able to diagnose at different labs (share reference intervals)

32 Revision of ALP Final Categories State-of-the-art If unable to meet a higher category Expert Opinon If suitable data not available

33 Using QAP ALP as Quality Limits Vanessa Lo Sigma Metrics as Performance Indicator Contributes to Effective Cost and Man-hour Saving in Chemical Pathology Laboratory Department of Clinical Pathology Chemical Pathology Laboratory Hong Kong Sanatorium and Hospital Roche Oral Presentation Prize AACB ASM 2014

34 16* 7 4 >= 5 Sigma 17 analytes ALT Pancreatic Amylase Total Amylase AST Direct Bilirubin CK Glucose GGT LDH Magnesium Phosphate Triglyceride Uric Acid Bicarbonate HDL-C LDL-C CRPLX 4 Sigma 5 analytes ALP Calcium Cholesterol Iron Total Protein 3 Sigma 7 analytes Albumin Total Bilirubin Creatinine Urea Sodium Potassium Chloride Chemistry SydPath* Albumin Sodium Chloride Bicarbonate

35 >= 5 Sigma 12 analytes CKMB-STAT pro-bnp hs-tnt beta-hcg E2 FSH LH Progesterone Prolactin Thyroglobulin PTH-STAT Ferritin 4 Sigma 1 analyte B12 3 Sigma 2 analytes SFolate Vit. D Total Immuno assay

36 Sigma No. of QC daily Chemistry No. of Analytes Immunoassay >=

37 Vanessa Lo Reduction in QC performance Saving of 1.6% of reagent costs Uprising in the emotional status of the staff

38 USE OF ALP as Quality Limits >5 Sigma: Can have confidence that results will be within RCPAQAP ALP using simple rules 4-5 Sigma: need tighter rules to be sure results will meet RCPAQAP ALP <4 Sigma: Cannot be confident that Results will meet RCPAQAP ALP Will see poor results?

39 RCPA General Serum Chemistry CYCLE 90 1,2 3,4 5,6 7,8 9,10 11,12, 13,14 15,16 Working mean Working 80th Outliers No outliers* in entire cycle for any analyte! (35 analytes, 1120 Results) * Method Specific Targets (includes bias)

40 RCPA General Serum Chemistry Cycle Flagged Test 91 4 PO4(2),FT4,GGT hcg(2), Cl FT4(5),HDL(2), Bic,Lip,Na,K 87 2 Ca, FT Ca, FT4(2) 84 1 Ca 79 4 Ca,Ferr,TG,Bil,Ca 76 8 Ferr,Cl,Bic(2)gluc,Cbil 75 1 Ferr TOTAL % of total > 3 years: No albumin failures, 1 Na, 1 CO2, 2 Cl All Incapable (Sigma <4)

41 Roche Modular BCG Albumin (2014)

42 Albumin QC Level1: CVa = 31 g/l (ALP 6.4%), Sigma = 3.2 QC Level 2: Cva = 46 g/l (ALP 6.0%), Sigma = 3.2 Albumin CVi = 3.1% (both QC levels meet minimal Standard) An incapable assay how does it succeed?

43 Assay Characteristics Stable assays: Performance defined by mean and SD QC never fails Results always within +/- 2SD

44 Stable Assay QC -3SD -2SD Mean +2SD +3SD Mean = 20, SD = 1 95% of QC results between 18 and 22.

45 Assay Characteristics Unstable Assays Mean drifts over time (fluctuating bias) QC process used to detect drifts Variation in results due to scatter plus drift

46 Unstable Assay SD -2SD Mean +2SD +3SD QC Mean = 20, SD = 1, Plus fluctuating mean. Interpretation: Result of 20 has 95% confidence limit of 18 22, PLUS undetected bias at time of measurement.

47 Put all this together

48 Put all this together 1. Setting Quality Limits provides the most robust approach What Limits to use? RCPAQAP (where capable) Others (where not capable against RCPAQAP) ALP different for different analytes, What limits, Complexity

49 What I do Compare CVa with CVi If CVa small relative to CVi: few rules, wide limits n=2, 1 x approx 3SD If CVa not small relative to Cvi: few rules, tighter limits n=2, 1 x approx 2.2 SD Understand performance risks (Reverse Engineering) Compare with quality standards regularly (Cvi, RCPAQAP ALP, state of the art QAP, PI)

50 What can we do Define limits based on CVi Turn internal process into Logical steps Remember stat of the art

51 Bias: what I do Consider separately to precision Base on Fraser concepts: percent of patients wrongly classified Eg ~one 10 th of population reference interval (if Gaussian)

52 Results Change Protocol ANALYTE ALLOWABLE LIMITS Albumin +/- 4 up to 40 g/l, then +/- 10% ALT +/- 8 up to 60 U/L, then 15% (NCIRI) Amylase +/- 15% (NCIRI) AP +/- 15% (NCIRI) AST +/- 8 up to 60 U/L, then 15% (NCRI) Bicarbonate +/- 4 mmol/l (6 mmol/l IRI) Bilirubin +/- 8 up to 60 umol/l, then 10% (NCIRI) Calcium +/- 0.2 mmol/l (NCIRI) Chloride +/- 5 mmol/l (NCRI) Cholesterol +/- 0.5 up to 10 mmol/l, then 5% CK +/- 15 up to 100 U/L, then 15% CK-MB +/- 2 ug/l to 7 ug/l, then 15% (NTIRI) In the event of re-running an assay following the suspicion of an analytical error, significant changes must be changed in the computer and notified to the requesting/treating doctor. Changes equal to, or greater than those shown below may be considered significant (these values based on the RCPA-AACB Allowable limits of performance). If in doubt, consult the Pathologist or Senior Scientist.

53

54 Thank you