QUALITY CONTROL for MOLECULAR DIAGNOSTICS Quality issues highlighted through international external quality assessment

QUALITY CONTROL for MOLECULAR DIAGNOSTICS Quality issues highlighted through international external quality assessment William G MacKay PhD Neutral Office Coordinator QCMD The Altum Building, Todd Campus West of Scotland Science Park Glasgow G20 0XP williammackay@qcmd.org

Quality Control for Molecular Diagnostics Provider of EQA programmes to the molecular diagnostics community worldwide An independent and international organisation Endorsed by the major scientific societies (ESCV & ESCMID)

QCMD: state of the art EQA panels Help labs to determine their performance Designed by experts Consist of 8-12 samples Include different serotypes / genotypes at various concentrations Reporting time 4-6 week Accompanied by questionnaire on technical details

QCMD: panels designed by experts Extended panels: dilution series, duplicates, negative samples, specificity samples Sam ple Sam ple Sam ple Sam ple conc. Stock Sam ple content matrix Copies/ml dilution type HSVDNA10-07 HSV-1 VTM 9,016 1.00x10-5 Core HSVDNA10-02 HSV-1 VTM 968 1.00x10-6 Core HSVDNA10-06 HSV-1 VTM 129 1.00x10-7 HSVDNA10-10 HSV-1 VTM 116 1.00x10-7 HSVDNA10-01 HSV-2 VTM 4,710 1.00x10-4 Core HSVDNA10-05 HSV-2 VTM 1,294 1.00x10-5 Core HSVDNA10-09 HSV-2 VTM 178 1.00x10-6 HSVDNA10-03 HSV-2 VTM 69 1.00x10-7 HSVDNA10-04 VZV VTM Core HSVDNA10-08 Negative VTM Core

QCMD: comprehensive feedback to participants State of the art scoring systems Covering qualitative and quantitative data, and genotyping / sequencing where applicable Detailed final reports with expert feedback Including region / country specific reports Individualised reports for each participant Supported through the QCMD Neutral Office

Diagnostics in laboratory medicine Serology Cell culture, antigen tests Direct IF Nucleic acid tests

NATS the upside Faster and easier Reliable and robust Scalable and cost effective Better sensitivity and specificity Ever widening range of target pathogens Greater contribution to patient care

Contamination NATS the downside Sensitivity and specificity Inter-laboratory variation Technology/assay variation Lack of suitable controls or international standards

Number of EQA participants increasing HIV HSV 250 250 200 200 150 150 participants participants 100 100 50 50 0 2004 2005 2006 2007 2008 2009 0 2004 2005 2006 2007 2008 2009 HBV N. gonorrhoeae 250 180 160 200 140 120 150 100 Participants 100 80 60 participants 50 40 20 0 2004 2005 2006 2007 2008 2009 0 2004 2005 2006 2007 2008 2009

Cumulative total datasets in HIV EQA since 2002 1400 1200 1000 800 600 Cumulative total 400 200 0 2002 2003 2004 2005 2006 2007 2008 2009

What can we say about the performance of NATs for pathogen detection?

The issues False positives Assay contamination Assay specificity False negatives Assay sensitivity Strain variation Quantitative accuracy Assay/operator variation Lack of standards

False positives

False positive results False positives caused by Assay contamination Assay specificity issues Assay contamination an important concern in all assays (including molecular diagnostics) Assays may pick up on closely-related species

False positive results late 20 th Cent. Early-mid 1990s Late 1990s

False positive results early 21 st Cent. % 50 45 40 35 30 25 20 15 10 5 0 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Year C. trachomatis A VZV Enterovirus HSV JCV / BKV Cp / Mp HIV (RNA) HBV HCV HIV (DNA) B19 HHV6 CMV EBV L. pneumophila T. gondii MRSA C. difficile C. trachomatis B N. gonorrhoeae CMV Dried Blood Spot M. tuberculosis HPV B. pertussis Adenovirus Influenza A / B MPV / RSV Parainfluenzavirus Rhinovirus / Coronavirus Influenza typing HIV (RNA) (B) HBV (B) HCV (B) HIV (DNA) (B) Norovirus

Average false positivity early 21 st Cent. 10.00 9.00 8.00 7.00 6.00 % 5.00 4.00 3.00 Average false positivity 2.00 1.00 0.00 Year

Assay specificity Laboratories use a wide range of commercial and in-house assays to detect B. pertussis The majority of assays currently used IS481 for the detection of B. pertussis Issues with closely related species that also harbour IS481, such as B. bronchiseptica and B. holmesii

Assay specificity Results of the 2009 QCMD EQA programme for B. pertussis show evidence of false positives on closely related Bordetella spp. PCR Other Sample Sample Sample conc. Total Conventional Real time content CFU/m l datasets Commercial In-house Commercial In-house n=87 n=4 n=8 n=7 n=67 n=1 n % n % n % n % n % n % BP09-03 B. pertussis 1 x 10 4 86 98.9 4 100.0 8 100.0 7 100.0 66 98.5 1 100.0 BP09-06 B. pertussis 1 x 10 3 84 96.6 4 100.0 8 100.0 7 100.0 64 95.5 1 100.0 BP09-11 B. pertussis 1 x 10 2 75 86.2 3 75.0 8 100.0 7 100.0 56 83.6 1 100.0 BP09-10 B. pertussis 1 x 10 2 75 86.2 3 75.0 8 100.0 7 100.0 56 83.6 1 100.0 BP09-08 B. pertussis 1 x 10 1 54 62.1 1 25.0 3 37.5 4 57.1 45 67.2 1 100.0 BP09-09 B. bronchiseptica (IS481-) 1 x 10 3 77 88.5 1 25.0 8 100.0 6 85.7 61 91.0 1 100.0 BP09-04 B. bronchiseptica (IS481+) 1 x 10 3 38 43.7 1 25.0 6 75.0 3 42.9 28 41.8 0 0.0 BP09-01 B. parapertussis 1 x 10 3 74 85.1 1 25.0 7 87.5 7 100.0 59 88.1 0 0.0 BP09-12 B. hinzii 1 x 10 3 80 92.0 4 100.0 8 100.0 6 85.7 61 91.0 1 100.0 BP09-02 B. holmesii (IS481+) 1 x 10 3 14 16.1 4 100.0 1 12.5 0 0.0 9 13.4 0 0.0 BP09-07 Haemophilus influenzae 1 x 10 3 83 95.4 4 100.0 8 100.0 7 100.0 63 94.0 1 100.0 BP09-05 Negative 87 100.0 4 100.0 8 100.0 7 100.0 67 100.0 1 100.0

Assay specificity Results from the 2009 QCMD EQA programme for B. pertussis by target gene Sample Sam ple Sample conc. Total IS481 PRN PT Multi-target Other Not content CFU/ml datasets reported n=87 n=55 n=3 n=3 n=9 n=9 n=8 n % n % n % n % n % n % n % BP09-03 B. pertussis 1 x 10 4 86 98.9 55 100.0 3 100.0 3 100.0 9 100.0 8 88.9 8 100.0 BP09-06 B. pertussis 1 x 10 3 84 96.6 55 100.0 3 100.0 3 100.0 9 100.0 6 66.7 8 100.0 BP09-11 B. pertussis 1 x 10 2 75 86.2 52 94.5 2 66.7 3 100.0 5 55.6 6 66.7 7 87.5 BP09-10 B. pertussis 1 x 10 2 75 86.2 49 89.1 1 33.3 3 100.0 8 88.9 7 77.8 7 87.5 BP09-08 B. pertussis 1 x 10 1 54 62.1 39 70.9 2 66.7 1 33.3 2 22.2 5 55.6 5 62.5 BP09-09 B. bronchiseptica (IS481-) 1 x 10 3 77 88.5 50 90.9 3 100.0 1 33.3 8 88.9 8 88.9 7 87.5 BP09-04 B. bronchiseptica (IS481+) 1 x 10 3 38 43.7 23 41.8 3 100.0 1 33.3 7 77.8 3 33.3 1 12.5 BP09-01 B. parapertussis 1 x 10 3 74 85.1 50 90.9 3 100.0 1 33.3 6 66.7 8 88.9 6 75.0 BP09-12 B. hinzii 1 x 10 3 80 92.0 50 90.9 3 100.0 3 100.0 8 88.9 9 100.0 7 87.5 BP09-02 B. holmesii (IS481+) 1 x 10 3 14 16.1 1 1.8 2 66.7 3 100.0 4 44.4 2 22.2 2 25.0 BP09-07 Haemophilus influenzae 1 x 10 3 83 95.4 51 92.7 3 100.0 3 100.0 9 100.0 9 100.0 8 100.0 BP09-05 Negative 87 100.0 55 100.0 3 100.0 3 100.0 9 100.0 9 100.0 8 100.0 PRN: pertactin gene. PT: pertussis toxin gene

False positives should we accept them? Is there an acceptable level of false positives? Should we expect laboratories to never experience a false positive result? What about the clinical impact of false positive results? From 2006 to 2009 no false positive results on true negative samples have been reported for N. gonorrhoeae No false positives found in over 500 datasets Wide range of technologies represented

False negatives

False negative results False negatives caused by Sensitivity issues Strain variation Assays vary considerably in reported limits of detection What is the clinically relevant level of a pathogen?

Assay sensitivity Reported lower limit of detection in the 2008 EQA for HIV RNA 60 50 90% Frequency 40 30 20 10 0 4 10 20 25 30 40 47 50 53 57 70 75 96 100 400 600 Lower limit of detection (Cp/ml)

Assay sensitivity Results from the 2008 QCMD EQA programme for HIV RNA PCR NASBA TMA bdna Sam ple Sam ple Target value Total Conventional Real time content Copies/ml datasets Commercial In-house Commercial In-house n=191 n=52 n=2 n=90 n=10 n=25 n=4 n=8 n % n % n % n % n % n % n % n % HIVRNA08-03 HIV-1 Type A/G 322 159 83.2 48 92.3 1 50.0 88 97.8 7 70.0 7 28 4 100.0 4 50.0 HIVRNA08-06 HIV-1 Type A/G 62 101 52.9 27 51.9 1 50.0 60 66.7 6 60.0 2 8 4 100.0 1 12.5 HIVRNA08-02 HIV-1 Type B 551 185 96.9 47 90.4 2 100.0 90 100.0 10 100.0 24 96 4 100.0 8 100.0 HIVRNA08-10 HIV-1 Type B 122 164 85.9 45 86.5 1 50.0 81 90.0 10 100.0 19 76 4 100.0 4 50.0 HIVRNA08-04 HIV-1 Type C 2767 188 98.4 49 94.2 2 100.0 90 100.0 10 100.0 25 100 4 100.0 8 100.0 HIVRNA08-07 HIV-1 Type C 294 169 88.5 47 90.4 2 100.0 89 98.9 10 100.0 14 56 4 100.0 3 37.5 HIVRNA08-09 HIV-1 Type C 263 169 88.5 46 88.5 2 100.0 88 97.8 10 100.0 15 60 4 100.0 4 50.0 HIVRNA08-01 HIV-1 Type C 65 109 57.1 29 55.8 0 0.0 64 71.1 8 80.0 4 16 4 100.0 0 0.0 HIVRNA08-08 HIV-1 Neg. Plasma 180 94.2 45 86.5 2 100.0 88 97.8 10 100.0 23 92 4 100.0 8 100.0 HIVRNA08-05 HIV-1 Neg. Plasma 179 93.7 46 88.5 1 50.0 89 98.9 9 90.0 22 88 4 100.0 8 100.0

False negatives strain variation In 2006 a new variant of C. trachomatis was discovered in Sweden Surveillance studies showed an apparent decrease in the incidence of C. trachomatis Further work showed that the reduced detection rates was due to a new variant C. trachomatis lacking a section of the cryptic plasmid

False negatives strain variation Björn Herrmann et al 2008

False negatives strain variation QCMD included the C. trachomatis Swedish variant in the 2010 EQA panel PCR SDA TMA Other Sample Sample Sample conc. Total Conventional Real time content Copies/vial datasets Commercial In-house Commercial In-house n=211 n=35 n=4 n=91 n=28 n=29 n=23 n=1 n % n % n % n % n % n % n % n % CTA10-02 Chlamydia trachomatis 5700 207 98.1 33 94.3 4 100.0 89 97.8 28 100.0 29 100.0 23 100.0 1 100.0 CTA10-05 Chlamydia trachomatis 280 198 93.8 33 94.3 3 75.0 86 94.5 28 100.0 25 86.2 22 95.7 1 100.0 CTA10-01 Chlamydia trachomatis 280 190 90.0 31 88.6 4 100.0 84 92.3 25 89.3 24 82.8 21 91.3 1 100.0 CTA10-04 Chlamydia trachomatis 57 150 71.1 27 77.1 3 75.0 68 74.7 20 71.4 11 37.9 21 91.3 0 0.0 CTA10-03 C. trachomatis (Sw edish Variant) 1.5x10 6 178 84.4 3 8.6 3 75.0 91 100.0 28 100.0 29 100.0 23 100.0 1 100.0 CTA10-06 Ct. negative urine 207 98.1 33 94.3 3 75.0 91 100.0 27 96.4 29 100.0 23 100.0 1 100.0 CTA10-08 Chlamydia trachomatis 5700 211 100.0 35 100.0 4 100.0 91 100.0 28 100.0 29 100.0 23 100.0 1 100.0 CTA10-10 Chlamydia trachomatis 57 177 83.9 30 85.7 4 100.0 85 93.4 23 82.1 14 48.3 20 87.0 1 100.0 CTA10-07 Chlamydia trachomatis 28 163 77.3 25 71.4 3 75.0 83 91.2 22 78.6 11 37.9 19 82.6 0 0.0 CTA10-09 Ct. negative sw ab 207 98.1 34 97.1 4 100.0 90 98.9 28 100.0 29 100.0 21 91.3 1 100.0 Approximately 20% of datasets returned by participants recorded a negative for the Swedish variant

Strain variation Results from the 2007 QCMD EQA programme for HIV Sam ple Sam ple Sam ple conc. Total content Copies/ml datasets n=171 n % HIVRNA07-06 HIV-1 Type A/G 971 166 97.1 HIVRNA07-08 HIV-1 Type A/G 256 143 83.6 HIVRNA07-09 HIV-1 Type A/G 95 100 58.5 HIVRNA07-01 HIV-1 Type A/G 78 94 55.0 HIVRNA07-03 HIV-1 Type B 193 154 90.1 HIVRNA07-02 HIV-1 Type C 625 166 97.1 HIVRNA07-07 HIV-1 Type C 226 152 88.9 HIVRNA07-05 HIV-1 Type C 66 111 64.9 HIVRNA07-10 HIV-1 Type C 15 93 54.4 HIVRNA07-04 HIV-1 Negative 163 95.3

Strain variation 120 100 y = 38.19x - 6.314 R² = 0.955 Percentage positive results 80 60 40 y = 39.48x - 17.93 R² = 0.954 Type A/G Type C Linear (Type A/G) Linear (Type C) 20 0 1.7 1.9 2.1 2.3 2.5 2.7 2.9 3.1 Log10 Copies/ml

Quantitative accuracy/variation

Quantitative accuracy/variation There is no such thing as a true gold standard in molecular diagnostics WHO supporting use of standardised international units (e.g for HIV) Still there is a large variation on reported quantitative results

Quantitative variation Results of the 2009 HSV EQA HSV -1 Sam ple Sam ple Sam ple Sam ple conc. content matrix Copies/ml HSVDNA10-07 HSV-1 VTM 9,016 HSVDNA10-02 HSV-1 VTM 968 HSVDNA10-06 HSV-1 VTM 129 HSVDNA10-10 HSV-1 VTM 116 HSVDNA10-01 HSV-2 VTM 4,710 HSVDNA10-05 HSV-2 VTM 1,294 HSVDNA10-09 HSV-2 VTM 178 HSVDNA10-03 HSV-2 VTM 69 HSVDNA10-04 VZV VTM HSVDNA10-08 Negative VTM

Technology variation Results of the 2009 HSV EQA (HSV-1) 5.000 Technology Consensus (Log10 Copies/ml) 4.500 4.000 3.500 3.000 2.500 2.000 Real time Commercial PCR y = 0.9768x + 0.0628 R² = 0.9983 Real time In house PCR y = 0.9452x + 0.1409 R² = 0.9931 1.500 1.500 2.000 2.500 3.000 3.500 4.000 4.500 Consensus (log10 Copies/ml)

Quantitative variation Results of the 2009 HHV6 EQA HHV 6-B Sam ple Sam ple Sam ple Sam ple conc. content m atrix Copies/m l HHV609-04 HHV6 type A Plasma 2,466 HHV609-09 HHV6 type A Plasma 685 HHV609-01 HHV6 type B Plasma 100,000 HHV609-06 HHV6 type B Plasma 9,354 HHV609-07 HHV6 type B Plasma 1,327 HHV609-02 HHV6 type B Plasma 1,019 HHV609-10 HHV6 type B Plasma 912 HHV609-03 HHV6 type B Plasma 166 HHV609-08 HHV6 type B Plasma 209 HHV609-05 HHV6 Negative Plasma

Technology variation Results of the 2009 HHV6 EQA (HHV6-B) 6.000 5.500 Technology Consensus (log10 Copies/ml) 5.000 4.500 4.000 3.500 3.000 2.500 2.000 1.500 y = 0.994x + 0.1484 R² = 0.9992 y = 1.0082x 0.288 R² = 0.9992 Real time Commercial PCR Real time In house PCR 1.000 1.000 2.000 3.000 4.000 5.000 6.000 Consensus (log10 Copies/ml) The lines predicting the Technology Consensus values have significantly different intercepts (difference = 0.436, 95% CI 0.305 to 0.568; p = 0.00002).

How to improve quality in molecular diagnostics?

How to improve quality in molecular diagnostics Clear understanding of clinical requirements Careful planning and design Rigorous optimisation and continuous reevaluation Quality control

False positives Conclusions the issues Assay contamination Assay specificity False negatives Assay sensitivity Strain variation Quantitative accuracy Assay/operator variation Lack of standards