Confirmation - A Practical Approach

Transcription

1 Data Mining i for Reference Interval Confirmation - A Practical Approach Graham Jones Chemical Pathologist St Vincent s Hospital, Sydney AACB Webinar 5 th June 2013

2 Summary A little (more) background Current status in Australian adult Common Reference Intervals (CRI) Local Validation of CRI (data mining) Bhattacharya

3 Reference Intervals Fixed proportion of a reference population Usually central 95% Usually Health population Most common interpretive tool for numerical laboratory results Changes a result (eg X mmol/l) into information (eg grossly elevated ) Customers have great faith in them Customers have great faith in them its all right I interpret the results against the laboratory s own reference interval

4 Reference Intervals Alb Cr Ratio Upper Reference Limit Number 1.0 mg/mmol mg/mmol mg/mmol (m) / 3.5 (f) mg/mmol mg/mmol Data

5 Reference Intervals Alb Cr Ratio Upper Reference Limit Number 1.0 mg/mmol 3 Highest over 3 x lowest 2.0 mg/mmol mg/mmol 5 Assay variability ~20% 2.5 (m) / 3.5 (f) mg/mmol 1 Reference interval changes the information 3.5 mg/mmol Data

6 Statement of belief Any survey of reference intervals will show a wide scatter This scatter is not related to method differences We will all select and assess the data differently This causes our laboratories to give out different information G Jones 26 th July 2012 (revised 4 th June 2013)

7 Statements of Belief (2) Laboratories should act similarly where possible Pre-analytic analytic, analytic, post-analytic This can only happen if there is: (e)meeting of people Discussion Agreement Action

8 Common Reference Intervals (CRI) Clinical Decision Points Defined by expert groups Glucose, HbA1c (diabetes Dx & Mx), Lipids (risk and treatment) Methods with comparable results Unbiased from each other CRI possible for all labs Methods with non-comparable results Biased from each other Method-specific CRI possible

9 Clinical Decision Points HbA1c HbA1c acceptable performance in EQA, standardised to international reference values, MJA 2012;197 (4) 20 th August 2012

10

11 Sample RCPA QAP Whole Blood HbA1c 2012 NGSP 11 ALP High = Median / Target = 10.3 NGSP (% %) 10 ALP Low = What we can do: - Select good method -Make method run well RCPAQAP Chemical Pathology

12 Common Clinical decision points Source same references / guidelines Interpret them in the same way Avoid minor inconsistencies Eg >1.0 mmol/l >1.00 mmol/l (1.02 not flagged / flagged depending on decimal places) Guidelines should state explicitly what should be flagged or we should decide and implement

13 Population Reference Intervals

14 Population reference intervals (Approximately) Gaussian Symmetrical Simple statistics (central 95%) Bias criteria well established Data-mining i robust Skewed / assymetrical Statistics and criteria less well established May include common pathology (eg ALT) May need clinical overlay (eg exclude overweight / obese) WARNING!!!

15 CRI - National Activities AACB Harmonisation WG National Meeting May participants 19 chemical pathologists All 6 specialist paediatric chemical pathologists

16 Process Working groups with information supplied Meta data about test Reference interval data Reference NORIP Interval to share ARQAG / SIQAG (with thanks) UK harmonised reference intervals Data extracts t from Sonic labs (with thanks) Aussie Normals (with thanks) Method comparison data QAP data,,published data BIAS study (Gus Koerbin) Small enough method bias

17 Magnesium Magnesium 1.00 Obtain ned (mmol l/l) ADVIA 2400 Architect AU2700 DxC Integra Modular Vitros Average (mmol/l)

18 Magnesium Population RI Gender differences mmol/l No No expected methodological Fasting: differences No reason to have a fasting RI Analytically there are differences Inpatients vs outpatients. No Integra and Architect possibly different Age related intervals No Serum vs Plasma Interchangeable Proposed RI: Pre-analytical Preanalytics should not have an effect on Mg

19 Creatine Kinase CK ained (U/L Obt )ADVIA Architect AU2700 DxC Integra Modular 90.0 RxL Vitros Average (U/L)

20 Reference interval Data CK MALES NORIP y NORIP - 50 y+ ARQAG SIQAG Level of LRL URL evidence (U/L) (U/L) 2a) direct ) Professional recommendation UK HARMONY 4) Consensus SONIC <60 y SONIC <90 y SONIC 90 y+ 2a) indirect AUSSIE NORMALS 2a) direct (49) (291) (exclusions to follow) (N=434) Lo et al. Hong Kong 2a) direct ejifcc March 2012 (N=378) Schumann G et al. CCA 2003;327: a) direct

21 Creatine Kinase SONIC RI data are from largely outpatients, whereas Schumann data are from hospitalised inpatients and may reflect lower muscle mass. AUSSIE NORMALS data require exclusion of individuals with increased muscle mass, e.g. athletes. Testosterone-CK correlation less change in women than men with age. Further information about the impact of haemolysis is required. Further investigation of differences between AUSSIE NORMALS and SONIC data is required, particularly for females. Caution on ethnicity investigation of the indigenous population may be required.

22 Creatine Kinase Analytically there are no differences between methods. Possible to share a common RI but need to define population p by gender & age Provisional RIs: Male: 16 -<60 y U/L Female: 60 - <90 y U/L 90 y U/L U/L

23 Analyte Male Female Calcium mmol/l Phosphate Magnesium LDH [L to mmol/l mmol/l U/L P]IFCC Sodium mmol/l Potassium mmol/l Chloride Bicarbonate mmol/l mmol/l Creatinine umol/l umol/l ALP U/L AST <40 <35 ALT <40 <30 Total Protein g/l

24 Current Paradigm Based on recommendations from the NCCLS and the IFCC Repeated in Product Information from most reagent suppliers Encoded in the NATA summary of ISO/IEC guide laboratories may perform their own detailed d reference interval studies or may validate reference intervals published elsewhere for their own methods and populations

25 Adopting (Common) Reference Intervals Laboratory must determine that reference interval is suitable for use in their lab. Two components Analytical accuracy Is my method biased compared with the method used to set the reference interval? Population accuracy Is my population the same as that included in the external reference interval?

26 Adopting Common Reference Intervals Analytical Accuracy Show method has small enough bias from method used to define reference interval Eg: Value assigned QC (manufacturer) External QA Reference Materials Certification This is all that is possible for decision points Seek multiple sources of confirmation

27 Adopting Common Reference Intervals Analytical Accuracy Show method has small enough bias from method used to define reference interval Eg: Value assigned QC (manufacturer) Eg if adopting data from Aussie External QA Normals, and using Abbott assays, Reference Materials show your results match other Abbott Certification results This is all that is possible for decision points Seek multiple sources of confirmation

28 Reference Interval Size To contain 95% of results, reference Intervals include the following: Between-person variation (CVg) Within-person variation (only one sample, CVi) Pre-analytical l variation (only one collection, CVpa) Analytical variation (only one measurement, CVa) CVri = (CVg 2 + CVi 2 + CVpa 2 + CVa 2 )

29 Common Reference Interval Size CVri = (CVg 2 + CVi 2 + CVpa 2 + CVa 2 ) CVa All samples in one run: within-run CV Same machine over time: between-run CV Different machines, same lab: CVx Different machines, different labs: CVy Different suppliers, different : CVz CVz > CVy > Cvx > CVbr > CVwr MORE MACHINES BIGGER Cva WIDER REFERENCE INTERVALS

30 Common Reference intervals When CRI applied in one lab % outside RI < 2.5% Less sensitive for abnormalities (more specific) Allows for some Wriggle room Lot to lot variation calibrator variation lab Lot-to-lot variation, calibrator variation, labto-lab variation etc

31 Adopting (Common) Reference Intervals CLSI method: run 20 samples from healthy subjects: 18 in ranges, accept. Local reference interval study Within experimental uncertainties Database mining Midpoint Number of outliers All test method and population More data = better assessment

32 Analysing the Data

33 SydPath Validation Data extracts from routine database GP samples only Ages above 18 (some upper age limits) ~9000 subjects in most groups

34 Spreadsheet Application Available from

35 Instructions Caveat: Spreadsheet put together with care. But no guarantees of being error free User assumes all responsibility. Always view data, check usual statistics and assess for reasonableness

36 Data Entry Standardised data entry List of: result, sex, age, (date) 50,000 now 65,000 max in Excel ,048,576 max in Excel 2010 Limit to outpatient data LIS, Middleware, backup server

37 More data (Excel 2010) Extend calculations (50003 x?) Copy these 2 columns down Extend Lookuo (50003 x?)

38 Additional Data Input (metadata)

39 Numerical Output

40 Application Output graph Allows visual assessment Variables: Xaxis Bin size (only affects graph)

41 Graph X- axis

42 Bin Size for Graph

43 Application Averages compared over age and sex Over time

44 Data Recording

45 Data Reporting Sydpath data analysis for Common Reference Intervals Test Sodium Potassium Chloride Bicarbonate Creatinine F Creatinine M Harmonised Limits LRL URL Current Limits LRL URL Excluded data LexL UexL Age exclusions Age-L Age-H Sex exclusions Sex(M,F,B B B B B F M Harmonised Limits Midpoint Results Average Median %low 15% 1.5% 05% 0.5% 14% 1.4% 12% 1.2% 11% 1.1% 13% 1.3% %high 1.7% 2.6% 0.2% 1.5% 2.8% 6.4% 2.5th th n start date 2/5/12 2/5/12 2/5/12 2/5/12 2/5/12 2/5/12 end date 20/9/12 22/9/12 22/9/12 22/9/12 22/9/12 22/9/12

46 Data Reporting

47 Serum Bicarbonate

48 Interpretation - Bicarbonate

49 SydPath Evaluation

50 Serum Sodium

51 Serum Creatinine - Male 6.4%

52 Serum Calcium

53 Serum ALT - Male Median= 21 (mid=21) 12.8% Hitachi IFCC no P5P

54 SydPath Evaluation

55 CRI Validations Before agreeing to the CRI there should be multiple local validations. They will only be common RI if they are very widely used. Template available Produces common analysis and format Simple to use Available now (help provided)

56 What we can do Assess the AACB CRI (whatever method) Can I use them in my lab? Feedback on decision. If enough labs say yes they become CRI!

57 What else can we do Select traceable assays JCTLM-listed if possible Reference materials, methods, laboratories Select correct version of assay: GGT (IFCC not Szasz) LDH (P L v L P) AST (P5P or P5P) Albumin (BCG or BCP)

58 Non-Standardised Tests

59 Ferritin Abbott Group Tight Group Different from others Others not the same Endocrine Program Endo-of Cycle Report (2012) RCPA QAP data used with permission.

60 Ferritin Roche ECL Group Tight group Different from others Endocrine Program Endo of Cycle Report (2012) RCPA QAP data used with permission.

61 The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

62 Non-Standardised Tests CRI possible within manufacturer group (not across groups) Ideally standardise assays (global manufacturer issue) Next best would be Group CRI s such that different labs give same information Again co-operative work required

63 Bhattacharya Analysis A Data mining tool to assist with establishing or verifying reference intervals using local data.

64 Bhattacharya Bhattacharya, LG. Journal of the Biometric Society. 1967;23: Example data: Frequency Distribution of the forkal length of the Porgy caught by pair-trawl fishery in the East China Sea.

65 Porgy Forkal Length

66 Graphical Technique

67

68 Bhattacharaya Find a Gaussian Distribution in the midst of other data Outputs are midpoint and SD of Gaussian Distribution Needs The majority of results to be unaffected by disease Underlying Gaussian distribution ib ti Appropriate comparator (outpatients) Note: not affected by outliers

69 Bhattacharya Spreadsheet

70 Instructions Caveat: Spreadsheet put together with care. But no guarantees of being error free User assumes all responsibility. Always view data, check usual statistics and assess for reasonableness

71 Bhattacharya Spreadsheet Data Entry 3 columns Value, Sex, Age 65,000 (as is) More in Excel 2010

72 Bhattacharya Spreadsheet Analysis Inputs

73 Bhattacharya Spreadsheet Outputs (1) Basic Statistics

74 Bhatacharya Spreadsheet Bin Size (smaller more noise; larger fewer bins) - Operator dependent - Good data resistant to changes - More data is better - Aim for very straight line >3 data points

75 Error example Non-Gaussian Data Upward curve on Bhattacharya line

76 WARNING Log Transformation Can often make a good fit May be including disease state Log transform is only one of many transforms (eg Cox-Box) (has no effect on narrow distributions)

77 Assessment Criteria CLSI- number of outliers Data Mining Bias of midpoint (+/- 1/10th of RI) Number of outliers Raw data Data distribution Raw data Bhattacharya analysis Analytical Bias, QAP ALE (total error)

78 CRI The next Steps Adult Common RI Local validations Will I use them in my lab? Each lab (spreadsheet available) Agreement gee e t(some e/a all?) Together Implementation Each lab Other developments Bias 2 Study Gus Koerbin Liquid serum chemistry from RCPA QAP Method alignment (traceable assays) Next analytes?

79 Conclusions To provide the same information from different labs we need the same results and the same reference intervals (or RI that match the method) The current stated occurred through many decisions in many labs A change requires many more (good) decisions, based on good data Data mining (done well) provides a useful tool