Measuring data quality through a source data verification audit

Transcription

1 Measuring data quality through a source data verification audit Lauren Houston¹, Dr Yasmine Probst¹, Dr Allison Humphries¹ ¹School of Medicine, Faculty of Science, Medicine and Health, University of Wollongong lah993@uowmail.edu.au

2 Background Health data has long been scrutinised 1,2 A large proportion of errors are from transcribing data 3,4 No gold standard method exists to measure data quality error rates. 1. Y.W. Lee et al. (2006) 3. M. Mealer et al. (2013) 2. M.N. Zozus et al. (2014) 4. M.L. Nahm et al. (2008)

3 What is source data verification? Case report form Source data Electronic record

4 What are the gaps in the knowledge? ICH GCP guidelines are non specific to amount, timing and frequency of monitoring 5 Cost-effectiveness of SDV 6 No single definition to define data quality or universally accepted method to measure error rates 7,8 Audits may be published but not for public viewing 8 5. ICH GCP (1996) 7. R.Rostami et al. (2009) 6. C. Baigent et al. (2008) 8. R.V. Gómez-Rioja et al. (2013)

5 Aim To monitor data quality by developing and conducting source data verification audits to achieve quality assurance.

6 Study Background UOW and IHMRI Healthy-lifestyle blinded-rct 12 months 5 clinical Accredited Practising Dietitians To limit bias the audit was blinded

7 Method 100% SDV on the 10% random sample Quality assurance rule developed whereby if, >5% of data variables were incorrect a second 10% random sample was extracted Manual verification checks conducted Calculation of error rate

8 Procedure of data audit Participants n=210 10% random sample Random sample n=21 Total of 820 data points from 21 participants If >41 (5%) data points are incorrect 100% SDV Second 10% random sample of original participants Participants n=210 Random sample n=21 Total of 685 data points from 21 participants If >34 (5%) data points are incorrect 100% SDV of all data points

9 Audit 1 Source data verification Source documents Anthropometric Physiological Medications Electronic spreadsheet record Coded electronic spreadsheet record

10 Audit 2 Source data verification Source documents Anthropometric Medications Electronic spreadsheet record Coded electronic spreadsheet record

11 Coded electronic spreadsheet

12 Statistical Analysis Total error = (code 2+3+4) / (code ) Data not entered (code 5) excluded Chi square, p<0.05 Post-hoc; adjusted standardised residuals and z test of column proportions

13 Results % Minor error 3 Audit 1 Audit % Major error Audit 1 Audit 2 0 Anthropometric Physiological Medications 0 Anthropometric Physiological Medication

14 % 60 Results Not recorded data Audit 1 Audit 2 % 30 Not entered data Audit 1 Audit Anthropometric Physiological Medication 0 Anthropometric Physiological Medication

15 Correct and Incorrect % Correct Incorrect Anthropometric audit 1 Anthropometric audit 2 Medication audit 1 Medication audit 2 Total audit 1 Total audit 2

16 Results Chi square had a significant difference; χ2 (4, 1293) = , p = Adjusted standardised residuals determined audit sections were significantly different From the z test of column proportions anthropometric audit 1 and 2; medications audit 1 and 2 do not differ. All other sections differed from each other.

17 Summary Audit 1 physiological section <5% error Average total error anthropometric (9%), medications (76%) and overall (34.5%) Proportion of error trended upward as length of study increased Not recorded (code 4) data had the greatest contribution to total error

18 Discussion Importance of identifying errors and determining solutions If >10% of a clinical dataset is erroneous the data may be considered unreliable Developed a 5% quality assurance rule Data quality variations

19 Limitations Source documents considered the gold standard Source document-to-electronic spreadsheet Audits cannot guarantee 100% free from error Clinical research setting and trial design Did not determine the impact of audit findings

20 Recommendations Examine the 10% snapshot model with a 5% quality assurance error rate Standardise a SDV audit process Assess the frequency and cost-effectiveness Overcome barriers and increase awareness

21 How many errors are too many?

22 Acknowledgements Dr. Yasmine Probst Dr. Allison Humphries Sr/Prof. Linda Tapsell A/Prof. Marijka Batterham Illawarra Health and Medical Research Institute Smart Foods Centre All participants and staff involved