QUALITY CONTROL for MOLECULAR DIAGNOSTICS Quality issues highlighted through international external quality assessment

Transcription

1 QUALITY CONTROL for MOLECULAR DIAGNOSTICS Quality issues highlighted through international external quality assessment William G MacKay PhD Neutral Office Coordinator QCMD The Altum Building, Todd Campus West of Scotland Science Park Glasgow G20 0XP williammackay@qcmd.org

2 Quality Control for Molecular Diagnostics Provider of EQA programmes to the molecular diagnostics community worldwide An independent and international organisation Endorsed by the major scientific societies (ESCV & ESCMID)

3 QCMD: state of the art EQA panels Help labs to determine their performance Designed by experts Consist of 8-12 samples Include different serotypes / genotypes at various concentrations Reporting time 4-6 week Accompanied by questionnaire on technical details

4 QCMD: panels designed by experts Extended panels: dilution series, duplicates, negative samples, specificity samples Sam ple Sam ple Sam ple Sam ple conc. Stock Sam ple content matrix Copies/ml dilution type HSVDNA10-07 HSV-1 VTM 9, x10-5 Core HSVDNA10-02 HSV-1 VTM x10-6 Core HSVDNA10-06 HSV-1 VTM x10-7 HSVDNA10-10 HSV-1 VTM x10-7 HSVDNA10-01 HSV-2 VTM 4, x10-4 Core HSVDNA10-05 HSV-2 VTM 1, x10-5 Core HSVDNA10-09 HSV-2 VTM x10-6 HSVDNA10-03 HSV-2 VTM x10-7 HSVDNA10-04 VZV VTM Core HSVDNA10-08 Negative VTM Core

5 QCMD: comprehensive feedback to participants State of the art scoring systems Covering qualitative and quantitative data, and genotyping / sequencing where applicable Detailed final reports with expert feedback Including region / country specific reports Individualised reports for each participant Supported through the QCMD Neutral Office

6 Diagnostics in laboratory medicine Serology Cell culture, antigen tests Direct IF Nucleic acid tests

7 NATS the upside Faster and easier Reliable and robust Scalable and cost effective Better sensitivity and specificity Ever widening range of target pathogens Greater contribution to patient care

8 Contamination NATS the downside Sensitivity and specificity Inter-laboratory variation Technology/assay variation Lack of suitable controls or international standards

9 Number of EQA participants increasing HIV HSV participants participants HBV N. gonorrhoeae Participants participants

10 Cumulative total datasets in HIV EQA since Cumulative total

11 What can we say about the performance of NATs for pathogen detection?

12 The issues False positives Assay contamination Assay specificity False negatives Assay sensitivity Strain variation Quantitative accuracy Assay/operator variation Lack of standards

13 False positives

14 False positive results False positives caused by Assay contamination Assay specificity issues Assay contamination an important concern in all assays (including molecular diagnostics) Assays may pick up on closely-related species

15 False positive results late 20 th Cent. Early-mid 1990s Late 1990s

16 False positive results early 21 st Cent. % Year C. trachomatis A VZV Enterovirus HSV JCV / BKV Cp / Mp HIV (RNA) HBV HCV HIV (DNA) B19 HHV6 CMV EBV L. pneumophila T. gondii MRSA C. difficile C. trachomatis B N. gonorrhoeae CMV Dried Blood Spot M. tuberculosis HPV B. pertussis Adenovirus Influenza A / B MPV / RSV Parainfluenzavirus Rhinovirus / Coronavirus Influenza typing HIV (RNA) (B) HBV (B) HCV (B) HIV (DNA) (B) Norovirus

17 Average false positivity early 21 st Cent % Average false positivity Year

18 Assay specificity Laboratories use a wide range of commercial and in-house assays to detect B. pertussis The majority of assays currently used IS481 for the detection of B. pertussis Issues with closely related species that also harbour IS481, such as B. bronchiseptica and B. holmesii

19 Assay specificity Results of the 2009 QCMD EQA programme for B. pertussis show evidence of false positives on closely related Bordetella spp. PCR Other Sample Sample Sample conc. Total Conventional Real time content CFU/m l datasets Commercial In-house Commercial In-house n=87 n=4 n=8 n=7 n=67 n=1 n % n % n % n % n % n % BP09-03 B. pertussis 1 x BP09-06 B. pertussis 1 x BP09-11 B. pertussis 1 x BP09-10 B. pertussis 1 x BP09-08 B. pertussis 1 x BP09-09 B. bronchiseptica (IS481-) 1 x BP09-04 B. bronchiseptica (IS481+) 1 x BP09-01 B. parapertussis 1 x BP09-12 B. hinzii 1 x BP09-02 B. holmesii (IS481+) 1 x BP09-07 Haemophilus influenzae 1 x BP09-05 Negative

20 Assay specificity Results from the 2009 QCMD EQA programme for B. pertussis by target gene Sample Sam ple Sample conc. Total IS481 PRN PT Multi-target Other Not content CFU/ml datasets reported n=87 n=55 n=3 n=3 n=9 n=9 n=8 n % n % n % n % n % n % n % BP09-03 B. pertussis 1 x BP09-06 B. pertussis 1 x BP09-11 B. pertussis 1 x BP09-10 B. pertussis 1 x BP09-08 B. pertussis 1 x BP09-09 B. bronchiseptica (IS481-) 1 x BP09-04 B. bronchiseptica (IS481+) 1 x BP09-01 B. parapertussis 1 x BP09-12 B. hinzii 1 x BP09-02 B. holmesii (IS481+) 1 x BP09-07 Haemophilus influenzae 1 x BP09-05 Negative PRN: pertactin gene. PT: pertussis toxin gene

21 False positives should we accept them? Is there an acceptable level of false positives? Should we expect laboratories to never experience a false positive result? What about the clinical impact of false positive results? From 2006 to 2009 no false positive results on true negative samples have been reported for N. gonorrhoeae No false positives found in over 500 datasets Wide range of technologies represented

22 False negatives

23 False negative results False negatives caused by Sensitivity issues Strain variation Assays vary considerably in reported limits of detection What is the clinically relevant level of a pathogen?

24 Assay sensitivity Reported lower limit of detection in the 2008 EQA for HIV RNA % Frequency Lower limit of detection (Cp/ml)

25 Assay sensitivity Results from the 2008 QCMD EQA programme for HIV RNA PCR NASBA TMA bdna Sam ple Sam ple Target value Total Conventional Real time content Copies/ml datasets Commercial In-house Commercial In-house n=191 n=52 n=2 n=90 n=10 n=25 n=4 n=8 n % n % n % n % n % n % n % n % HIVRNA08-03 HIV-1 Type A/G HIVRNA08-06 HIV-1 Type A/G HIVRNA08-02 HIV-1 Type B HIVRNA08-10 HIV-1 Type B HIVRNA08-04 HIV-1 Type C HIVRNA08-07 HIV-1 Type C HIVRNA08-09 HIV-1 Type C HIVRNA08-01 HIV-1 Type C HIVRNA08-08 HIV-1 Neg. Plasma HIVRNA08-05 HIV-1 Neg. Plasma

26 False negatives strain variation In 2006 a new variant of C. trachomatis was discovered in Sweden Surveillance studies showed an apparent decrease in the incidence of C. trachomatis Further work showed that the reduced detection rates was due to a new variant C. trachomatis lacking a section of the cryptic plasmid

27 False negatives strain variation Björn Herrmann et al 2008

28 False negatives strain variation QCMD included the C. trachomatis Swedish variant in the 2010 EQA panel PCR SDA TMA Other Sample Sample Sample conc. Total Conventional Real time content Copies/vial datasets Commercial In-house Commercial In-house n=211 n=35 n=4 n=91 n=28 n=29 n=23 n=1 n % n % n % n % n % n % n % n % CTA10-02 Chlamydia trachomatis CTA10-05 Chlamydia trachomatis CTA10-01 Chlamydia trachomatis CTA10-04 Chlamydia trachomatis CTA10-03 C. trachomatis (Sw edish Variant) 1.5x CTA10-06 Ct. negative urine CTA10-08 Chlamydia trachomatis CTA10-10 Chlamydia trachomatis CTA10-07 Chlamydia trachomatis CTA10-09 Ct. negative sw ab Approximately 20% of datasets returned by participants recorded a negative for the Swedish variant

29 Strain variation Results from the 2007 QCMD EQA programme for HIV Sam ple Sam ple Sam ple conc. Total content Copies/ml datasets n=171 n % HIVRNA07-06 HIV-1 Type A/G HIVRNA07-08 HIV-1 Type A/G HIVRNA07-09 HIV-1 Type A/G HIVRNA07-01 HIV-1 Type A/G HIVRNA07-03 HIV-1 Type B HIVRNA07-02 HIV-1 Type C HIVRNA07-07 HIV-1 Type C HIVRNA07-05 HIV-1 Type C HIVRNA07-10 HIV-1 Type C HIVRNA07-04 HIV-1 Negative

30 Strain variation y = 38.19x R² = Percentage positive results y = 39.48x R² = Type A/G Type C Linear (Type A/G) Linear (Type C) Log10 Copies/ml

31 Quantitative accuracy/variation

32 Quantitative accuracy/variation There is no such thing as a true gold standard in molecular diagnostics WHO supporting use of standardised international units (e.g for HIV) Still there is a large variation on reported quantitative results

33 Quantitative variation Results of the 2009 HSV EQA HSV -1 Sam ple Sam ple Sam ple Sam ple conc. content matrix Copies/ml HSVDNA10-07 HSV-1 VTM 9,016 HSVDNA10-02 HSV-1 VTM 968 HSVDNA10-06 HSV-1 VTM 129 HSVDNA10-10 HSV-1 VTM 116 HSVDNA10-01 HSV-2 VTM 4,710 HSVDNA10-05 HSV-2 VTM 1,294 HSVDNA10-09 HSV-2 VTM 178 HSVDNA10-03 HSV-2 VTM 69 HSVDNA10-04 VZV VTM HSVDNA10-08 Negative VTM

34 Technology variation Results of the 2009 HSV EQA (HSV-1) Technology Consensus (Log10 Copies/ml) Real time Commercial PCR y = x R² = Real time In house PCR y = x R² = Consensus (log10 Copies/ml)

35 Quantitative variation Results of the 2009 HHV6 EQA HHV 6-B Sam ple Sam ple Sam ple Sam ple conc. content m atrix Copies/m l HHV HHV6 type A Plasma 2,466 HHV HHV6 type A Plasma 685 HHV HHV6 type B Plasma 100,000 HHV HHV6 type B Plasma 9,354 HHV HHV6 type B Plasma 1,327 HHV HHV6 type B Plasma 1,019 HHV HHV6 type B Plasma 912 HHV HHV6 type B Plasma 166 HHV HHV6 type B Plasma 209 HHV HHV6 Negative Plasma

36 Technology variation Results of the 2009 HHV6 EQA (HHV6-B) Technology Consensus (log10 Copies/ml) y = 0.994x R² = y = x R² = Real time Commercial PCR Real time In house PCR Consensus (log10 Copies/ml) The lines predicting the Technology Consensus values have significantly different intercepts (difference = 0.436, 95% CI to 0.568; p = ).

37 How to improve quality in molecular diagnostics?

38 How to improve quality in molecular diagnostics Clear understanding of clinical requirements Careful planning and design Rigorous optimisation and continuous reevaluation Quality control

39 False positives Conclusions the issues Assay contamination Assay specificity False negatives Assay sensitivity Strain variation Quantitative accuracy Assay/operator variation Lack of standards

40 QUALITY CONTROL for MOLECULAR DIAGNOSTICS Quality issues highlighted through international external quality assessment William G MacKay PhD Neutral Office Coordinator QCMD The Altum Building, Todd Campus West of Scotland Science Park Glasgow G20 0XP williammackay@qcmd.org