Verification and validation of diagnostic laboratory tests in clinical virology

Transcription

1 Journal of Clinical Virology 40 (2007) Review Verification and validation of diagnostic laboratory tests in clinical virology Holger F. Rabenau a,, Harald H. Kessler b, Marhild Kortenbusch a, Andreas Steinhorst c, Reinhard B. Raggam b, Annemarie Berger a a Institute for Medical Virology, Johann Wolfgang Goethe University Frankfurt/Main, Germany b Molecular Diagnostics Laboratory, Institute of Hygiene, Microbiology and Environmental Medicine, Medical University of Graz, Austria c DACH Deutsche Akkreditierungsstelle Chemie, Frankfurt/Main, Germany Received 5 July 2007; accepted 23 July 2007 Abstract This review summarizes major issues of verification and validation procedures and describes minimum requirements for verification and validation of diagnostic assays in clinical virology including instructions for CE/IVD-labeled as well as for self-developed ( home-brewed ) tests or test systems. It covers techniques useful for detection of virus specific antibodies, for detection of viral antigens, for detection of viral nucleic acids, and for isolation of viruses on cell cultures in the routine virology laboratory Elsevier B.V. All rights reserved. Keywords: Verification; Validation; Diagnostics; Virology; IVD; CE Contents 1. Introduction Quality assurance/quality control Verification and validation of tests or test systems employed in the routine laboratory Minimum requirements for verification and validation procedures for virological tests or test systems Further considerations regarding validation procedures for virological tests or test systems Verification of IVD/CE-labeled tests or test systems for detection of virus specific antibodies, detection of viral antigens or NAT General considerations when establishing a home-brewed NAT assay Validation of home-brewed tests or test systems for detection of virus specific antibodies, detection of viral antigens or NAT Validation of isolation of viruses on cell cultures Conclusion References Introduction Corresponding author at: Institute for Medical Virology, Paul Ehrlich- Straße 40, Frankfurt/Main, Germany. Tel.: ; fax: address: rabenau@em.uni-frankfurt.de (H.F. Rabenau). Routine viral diagnostics includes techniques for indirect and those for direct detection of viruses. Indirect detection of viruses is performed by serological studies. Techniques for direct detection of viruses include detection of viral antigens, detection of viruses or viral components by isolation /$ see front matter 2007 Elsevier B.V. All rights reserved. doi: /j.jcv

2 94 H.F. Rabenau et al. / Journal of Clinical Virology 40 (2007) of viruses on cell cultures (or through animal experiments) and detection of viral nucleic acids also referred to as nucleic acid testing (NAT). Furthermore, viral morphologic structures can be investigated by means of transmission electron microscopy. Today, NAT has a major impact on viral diagnostics. Molecular assays are currently used routinely in almost all routine virological laboratories. Technological improvements, from automated sample preparation to real time amplification technology, provide the possibility to develop and introduce assays for most viruses of clinical interest. The risk of contamination has been reduced significantly and the turn-around time to generate results shortened. Both the sensitivity and the dynamic range have been improved. In contrast, standardization and quality assurance/quality control issues have often remained underdeveloped requiring urgent improvement. Moreover, it must be taken into consideration that reliable viral diagnostics depends on pre-analytical issues either such as choice of the correct specimen, optimal sampling time with regard to the course of disease, and both time and conditions of the specimen transport to the laboratory. 2. Quality assurance/quality control The European Union s Directive on In Vitro Diagnostic Medical Devices (98/79/EC) demands for data demonstrating that an IVD achieves the stated performance and will continue to perform properly after it has been shipped, stored, and put to use at its final destination (Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on in vitro diagnostic medical devices, 1998). Furthermore, in the international standard DIN EN ISO 15189, special requirements for medical laboratories have been established recently. Among several issues, this new standard demands for verification and validation procedures. Quality control systems have been implemented in many European laboratories. In contrast to certification which is mainly based on the supervision of description and conformity of processes, accreditation additionally focuses on the competence of the laboratory providing reliable test results and their correct interpretation. Quality assurance demands for careful documentation in the routine diagnostic laboratory. For each newly implemented test or test system, a standard operation procedure Table 1 Minimum requirements for verification or validation of a test or a test system in clinical virology Verification Validation Sensitivity Precision (intra- and inter-assay) Specificity Linearity (if quantitative) Precision (intra- and inter-assay) Linearity (if quantitative) must be available. Additionally, verification or validation data must be available for each test. 3. Verification and validation of tests or test systems employed in the routine laboratory 3.1. Minimum requirements for verification and validation procedures for virological tests or test systems Suitability of a technique does not necessarily mean that it is performed correctly and provides valid results. The IVD Directive 98/79/EC and the DIN EN ISO demand for verification or validation of each investigational procedure in order to prove both the correct application and the correct performance of a diagnostic test. The complexity and the extent of the verification or validation procedure depend on the fact whether an IVD-CE labeled test or a self-developed ( homebrewed ) test is concerned. It is important to mention that the expression in vitro diagnostic medical device as used in the IVD Directive 98/79/EC does not only mean test but also test system if more than a single component are required to generate a diagnostic result. For instance, molecular assays based on PCR usually consist of a combination of different reagents and instruments for nucleic acid extraction, amplification, and detection of amplification products. Verification or validation work has to be done if a new test or test system is introduced in the routine diagnostic laboratory. Additionally, any change of an existing test procedure requires further validation work. For a commercially available IVD-CE labeled test or test system, the manufacturer is responsible that the IVD achieves the performance as stated. Nevertheless, the user must verify that performance characteristics, such as accuracy and precision are achieved in the laboratory (Table 1). The accuracy (or trueness in the recent nomenclature) is defined as the degree of conformity of a measured or calculated quantity to its actual (true) value and can be estimated by analyses of reference materials or comparisons of results with those obtained by a reference method. These are the only accepted approaches to trueness. When neither is available, other evidence relevant to the ability of the method to measure the analyte is needed. The precision is defined as level of concordance of the individual test results within a single run (intra-assay precision) and from one run to another (interassay precision). Precision is usually characterized in terms of the standard deviation of the measurements and relative standard variation (variation coefficient). Because precision is a measure for the variation of test results, it is also referred to as imprecision. In case of a quantitative test or test system, the linearity must be evaluated additionally (Table 1). The linearity is defined as the determination of the linear range of quantification. Data for linearity studies should be subjected to linear regression analysis with an ideal regression coefficient of 1. In case of a nonlinear curve, any objective, statistically valid method may be used (Linnet and Boyd, 2006).

3 H.F. Rabenau et al. / Journal of Clinical Virology 40 (2007) In contrast, the clinical laboratory that develops (homebrewed) tests or test systems, employs research-use-only tests or combines different IVD/CE-labeled tests without recommendation of the manufacturer is acting as manufacturer of a medical device and thus responsible for both the suitability and the correct performance of the test. Therefore, homebrewed tests or test systems must be validated including accuracy, sensitivity, specificity, precision, and, if quantitative, linearity (Table 1). The sensitivity is defined as the proportion of true positives of all positives. Vice versa, the specificity is a measure how well a test or test system correctly identifies the proportion of true negatives. Minimum requirements for verification and validation procedures for virological tests or test systems are described in the following sections. A more simplified validation procedure may be applied only if a test or a test system for validation is based on a scientific publication in a highly recognized journal, calibrators are not commonly accessible or the significance of the parameter to be tested is very limited. In general, reference material, patient samples or pooled sera may serve as calibrators for a verification or validation experiment. If patient samples or pool sera are used, they must have been tested earlier with the existing gold standard, as far as available and/or defined. Calibrators are classified into positive controls (for detection of virus specific antibodies and detection of viral antigens defined as more than dilution factor 3 over the lower limit of detection of the test or test system and within the upper limit of linearity; for NAT defined as more than 1 log 10 over the lower limit of detection of the test or test system and within the upper limit of linearity), low positive controls (for detection of virus specific antibodies and detection of viral antigens defined as up to dilution factor 3 over the lower limit of detection of the test or test system; for NAT defined as up to 1 log 10 over the lower limit of detection of the test or test system), and negative controls. If more than one positive control is necessary to complete testing for a certain performance characteristics, they should always contain different concentrations (within the linearity range as defined above) of the parameter to be tested. Minimum requirements outlined in this review are valid for all verification and validation procedures in clinical virology. However, tests or test systems for parameters included in List A of Annex II to Directive 98/79/EC are not covered here. Common technical specifications enforced for tests or test systems on those parameters are outlined in the Commission Decision of 7 May 2002 on common technical specifications for IVD medical devices (Commission Decision of 7 May 2002 on common technical specifications for in vitrodiagnostic medical devices, 2002) Further considerations regarding validation procedures for virological tests or test systems Because the choice of the sample material also referred to as matrix may have a strong influence on the results, it is necessary to validate each matrix intended to be tested in the routine diagnostic laboratory. For estimation of possible matrix-induced effects, the matrix of specimens employed must be identical to that of the patient specimen. Additional validation experiments have thus to be done to check for a possible matrix-induced effect. At least nine specimens (three positives, three low positives, and three negatives) must be tested for each additional matrix. For home-brewed tests or test systems it is proposed to estimate the detection limit. The detection limit also referred to as limit of detection is defined as the lowest concentration or quantity of an analyte that can be detected with a stated reasonable uncertainty for a given analytical procedure. The operational definition of this limit must be stated clearly. If there is no reference material available, both the determination of the detection limit and absolute quantification are not possible. When, for instance, a home-brewed test or test system based on NAT is established, introduction of real-time PCR may be a possibility to achieve a relative statement on the viral load (Koidl et al., 2004; Rasmussen, 2001). If an existing test or test system is modified or even replaced, diagnostic accuracy must be included in the evaluation process. In studies of diagnostic accuracy, the outcome from a test or test system under evaluation is compared with the outcome from the reference test or test system. Proposed items to include in determination of diagnostic accuracy have been published recently (Bossuyt et al., 2003). In clinical virology, minimum requirement is the comparison of results obtained by the new test or test system with those obtained by the existing test or test. To fulfill this, 20 specimens (7 positives, 6 low positives, and 7 negatives) must be tested in parallel. 4. Verification of IVD/CE-labeled tests or test systems for detection of virus specific antibodies, detection of viral antigens or NAT If a new IVD/CE-labeled test or test system for detection of virus specific antibodies, detection of viral antigens or NAT is introduced in the routine diagnostic laboratory, verification experiments are performed to verify precision and, in case of a quantitative NAT test or test system, linearity (Table 2). In case of a qualitative test or test system, one positive and one low positive specimen are used for determination of intraassay precision. Each sample is tested three times within a run. For inter-assay precision, one positive and one low positive specimen are used. Each sample is tested one time on three different days. In case of a quantitative test or test system for detection of virus specific antibodies or viral antigens, four positive and three low positive specimens are used for determination of intra-assay precision (each sample tested three times within a run) and two positive and one low positive specimen for determination of inter-assay precision (each sample tested one time on three different days). Correspond-

4 96 H.F. Rabenau et al. / Journal of Clinical Virology 40 (2007) Table 2 Verification of a new IVD/CE-labeled test or test system for detection of virus specific antibodies, detection of viral antigens or viral nucleic acid testing Calibrator (specimen) No. of samples required Detection of antibodies or antigens Nucleic acid testing Qualitative Quantitative Qualitative Quantitative Positive a Low positive b Negative Intra-assay precision Positive a Low positive b Inter-assay precision Positive a Low positive b Linearity Positive a a For nucleic acid testing, more than 1 log 10 over the lower limit of detection of the test or test system and within the upper limit of linearity. b For nucleic acid testing, up to 1 log 10 over the lower limit of detection of the test or test system. ing data for a quantitative NAT test or test system are three positive and three low positive specimens each for determination of intra-assay precision and one positive and low positive specimen each for determination of inter-assay precision. In order to optimize the verification workflow, it may be useful to take the first result of intra-assay precision testing as first result of inter-assay precision testing thus allowing a reduction of the number of further runs for inter-assay precision testing to two. In case of a quantitative IVD/CE-labeled NAT test or test system, linearity must be verified additionally by analyzing a serial dilution (10-fold dilution series with at least three dilution steps) of one positive specimen in duplicate. format is required. Because amplification may fail in a reaction due to interference from inhibitors, an IC must be incorporated in every NAT assay to exclude false-negative results. To ensure an accurate control of the entire NAT assay, the IC should be added to the specimen before the start of the nucleic acid extraction procedure. This guarantees validation of the entire analytical testing process. Any IC (homologous or heterologous) must be added at a suitable concentration to prevent extreme competition with the target template for reagents. When PCR-based NAT assays are introduced, quantitation by end-point analysis should be avoided; instead, log-phase analysis is preferable. 5. General considerations when establishing a home-brewed NAT assay When establishing a home-brewed NAT assay, primer and probe sequences must be checked carefully by use of a genome sequence databank. It is advisable to verify the amplification product by means of sequencing and to use a primer pair that has already been published in a highly recognized journal. The latter helps to avoid extended specificity testing. However, the published sequences should always be subjected to an alignment analysis with a genome sequence databank to ensure that the correct sequence has been published. Furthermore, several issues including the molecular technique employed, the detection format, introduction of an internal control (IC), and quantitation must be addressed. With regard to the molecular technique employed, it must be taken into consideration that automation reduces hands-on work and thus helps to avoid human error. To guarantee maximum specificity, introduction of a probe detection 6. Validation of home-brewed tests or test systems for detection of virus specific antibodies, detection of viral antigens or NAT If a home-brewed test or test system for detection of virus specific antibodies, detection of viral antigens or NAT is introduced in the routine diagnostic laboratory, validation experiments are performed to validate sensitivity, specificity, precision and, in case of a quantitative test or test system, linearity (Table 3). The sensitivity is determined by testing 10 positive and 10 low positive specimens while the specificity is determined by analyzing 20 negative but potentially cross-reactive specimens including positives for antibodies against viruses of the same family, sera testing positive for rheumatoid factor, and sera containing auto-antibodies for serological tests or test systems. For tests or test systems based on NAT, samples positive for viruses of the same family and samples spiked with potentially cross-reactive reference material are analyzed. For each potentially crossreactive analyte, one high-positive (at least 10 5 TCID 50 /ml

5 H.F. Rabenau et al. / Journal of Clinical Virology 40 (2007) Table 3 Validation of a home-brewed test or test system for detection of virus specific antibodies, detection of viral antigens or viral nucleic acid testing Calibrator (specimen) No. of samples required Detection of antibodies or antigens Nucleic acid testing Qualitative Quantitative Qualitative Quantitative Positive a Low positive b Negative Sensitivity Positive a Low positive b Specificity Negative Intra-assay precision Positive a Low positive b Inter-assay precision Positive a Low positive b Linearity Positive a a For nucleic acid testing, more than 1 log 10 over the lower limit of detection of the test or test system and within the upper limit of linearity. b For nucleic acid testing, up to 1 log 10 over the lower limit of detection of the test or test system. or 10 5 genome equivalents/ml) must be tested. Determination of intra- and inter-assay precisions are similar to those for verification procedures except of an extension in validation of quantitative tests or test systems regarding positive specimens (use of six positives instead of three for determination of intra-assay precision and two instead of one for determination of inter-assay precision). In case of a quantitative home-brewed NAT test or test system, linearity must be validated additionally by analyzing serial dilutions (10-fold dilution series with at least four dilution steps) of two positive specimens in duplicate on 2 days. 7. Validation of isolation of viruses on cell cultures Virus isolation on cell cultures is a technique which is difficult to standardize, thus validation becoming particularly demanding. First of all, the suitability of the cells for the detection of a certain virus has to be examined. During the implementation of new cell lines as an indicator system, the cell line should be tested for its susceptibility with two concentrations of both a reference virus strain and a wild type isolate. After titration of the virus stock, the inoculums should contain a Multiplicity of Infection of 0.1 (positive) and 0.01 (low positive). Tests must be done in triplicate on 3 days. Determination of precision is performed by using 20 wild type samples which must be tested in parallel on the existing and the newly introduced cell line (Table 4). The viability of the cells and the influence of the sample matrix must be recorded carefully. Table 4 Validation of isolation of viruses on cell cultures Sample requirements Positive a 1 Low positive b 1 Precision Wild type 20 a Multiplicity of infection 0.1. b Multiplicity of infection Conclusion No. of samples required Susceptibility Implementation of a new test or test system in the routine diagnostic virological laboratory demands for verification or validation procedures in compliance with a quality management system and according to DIN EN ISO While a CE/IVD-labeled test or test system requires verification, a home-brewed test or test system must be validated. References Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003;49:7 18. Commission Decision of 7 May 2002 on common technical specifications for in vitro-diagnostic medical devices. Official J Eur Commun 2002;L131: Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on in vitro diagnostic medical devices. Official J Eur Commun 1998;L331:1 37.

6 98 H.F. Rabenau et al. / Journal of Clinical Virology 40 (2007) Koidl C, Bozic M, Berg J, Stöcher M, Mühlbauer G, Grisold AJ, Marth E, Kessler HH. Detection of transfusion transmitted virus DNA by real-time PCR. J Clin Virol 2004;29: Linnet K, Boyd JC. Selection and analytical evaluation of methods with statistical techniques. In: Burtis CA, Ashwood ER, Bruns DE, editors. Tietz textbook of clinical chemistry and molecular diagnostics. 4th ed. St. Louis: Saunders; p Rasmussen R. Quantification on the LightCycler. In: Meuer S, Wittwer C, Nakagawara K, editors. Rapid cycle real-time PCR, methods and applications. Berlin: Springer-Verlag; p