Confirmation and Certainty in Toxicology Screening

Transcription

1 CLIN. CHEM. 34/8, (1988) Confirmation and Certainty in Toxicology Screening Vine R. Splshler,1 C. Michael O Donnell,2 and D. V. Gokhale3 Confirmation of presumptive positive urine drug screens, necessary to minimize the reporting of false-positive results, can be costly and time-consuming. The predictive value model can be used to select the confirming tests and to calculate the confidence of the result. The predictive value of a test result is the probability, based on the sensitivity and specificity of the test, that the result is a true positive or a true negative. The predictive value model applied to toxicology screening tests for drugs of abuse showed that prevalence, in addition to sensitivity and specificity, was the factor controlling the confidence level of a result. For example, the predictive value of a positive result for a screening test that has a sensitivity of 99% and a specificity of 99%, applied to screening in a population with a prevalence of 1% is 0.50; for a prevalence of 10%, it is Confirmation with a second, chemically independent, test of equal sensitivity and specificity increases the predictive value to Additional Keyphrases: abused drugs confidence levels Bayes theorem forensic medicine economics of laboratory operation cutoff values test sensitivity and specificity is the duty of the toxicologist at ways to employ the various corroborative tests; by omitting to do this, his otherwise escellent evidence may be materially weakened. -J.J. Reese, A Manual of Toxicology, 1874 In analytical toxicology, the concept of confirmation is referred to as the cornerstone of toxicological investigations. Confirmation is especially important in screening for drugs of abuse to ensure accurate results. Clinical decision theory can make a contribution to toxicology confirmation by application of the procedures developed and used in medical diagnosis for evaluation of the predictive value and efficiency of screening tests. The purpose of this paper is to review the standard procedures (1, 2) for estimating confidence values for results of screening tests and to demonstrate how these procedures can be applied to confirmation in toxicology screening. In this paper we will primarily be dealing with the area of qualitative analysis for a general unknown and not quantitative analysis as used in therapeutic drug monitoring. Therefore, what is important is to determine with acceptable confidence or certainty the identity of a particular drug in a given biological matrix. To achieve this, the analyst oc Products Corp West 96th Street, Los Angeles, Ana1ytica1 Systems, Irvine, CA. 3University of California, Riverside, CA. Received December 28, 1987; accepted April 25, must define what level of confidence is acceptable-that is, 95% probability, 99% probability, or greater than 99% probability. The 95% confidence limit is accepted as a general standard in clinical laboratories. Those applications that are involved with civil law, including pre-employment screening, might require a confidence level equal to or greater than 95%. Applications in criminal law cases might require confidence levels >99%. Once it has been decided what confidence level is acceptable, the analytical technique can be selected, but the choice is not at all obvious. The analyst s first responsibility is to choose a technique that has as high efficiency as possible. The efficiency of the technique will be judged on the basis of such factors as its sensitivity, specificity, speed, and cost. Technically, the efficiency of a test is its performance in assigning positive and negative results to the proper classification. Efficiency can only be determined for the particular area of analysis to which a technique is being applied, and it depends on clearly defining the objective in making the determination. Recent published surveys, including those produced by the Centers for Disease Control and the College of American Pathologists, indicate the unacceptable performance of many laboratories doing qualitative toxicological screening. There may be many reasons for this deficiency, but the concept of confidence in confirmation of screening tests can be useful in evaluating the areas of performance that need improvement. Experience in toxicology leads the analyst to make certain intuitive decisions about the probability of identifying a particular analyte. Although these decisions may not be formalized, they generally are based on information as to the sensitivity and specificity of the analytical techniques being applied and the incidence of this analyte in the population being analyzed. The formal approach described in this paper will follow the scheme shown in Figure 1. The decision process begins with an initial or prior probability for finding a specific analyte in a matrix of interest. An example of this might be that 10% of all urine specimens submitted from an emergency room can be expected to be positive for ethyl alcohol. That is, without any further analysis we know that there is a 10% probability that we will find alcohol in any urine specimen submitted from the emergency area. After receiving the specimen, the analyst performs the chemical tests appropriate to the diagnostic hypothesis. The hypothesis may be, for instance, that the patient is intoxicated with ethyl alcohol or sedative drugs. The screening tests provide new information that can be used to update the probability of the hypothesis. Bayes theorem, described in the following Methods section, provides the means for making these probability calcula- CLINICAL CHEMISTRY, Vol. 34, No.8,

2 positive test result when the urine does contain the drug or drug metabolite, is equal to the sensitivity of the tests times the prevalence of the drug or drug metabolite in the population of urine specimens, e.g., those from the emergency-room. P(EIH) = (prevalence) (sensitivity) Fig. 1. Clinical decision theory in toxicology screening tions, based on the sensitivity and selectivity of the analytical tests used. The updated or posterior probability can then be compared with the desired level of confidence. If the posterior probability or predictive value of the completed test(s) exceeds the required level of confidence, the result can be reported; if not, further, confirmatory, tests must be performed and the probability of the test results with regard to the diagnosis hypothesis again updated. The definition and application of Bayes theorem is provided in the Methods section below. However, it is important to keep in mind that in all these computations, the sequence of steps is as follows: (a) determine prior probabilities, (b) add new information, (c) apply Bayes theorem, and (d) compare posterior probabilities to desired level of confidence. In the context of using Bayes theorem or any other approach to assignment of probabilities in toxicological analysis, we shall define confirmation as the increase in posterior probability or predictive value of the test(s) until it equals or exceeds a confidence level that is acceptable-for example, 95%, etc. Once we have reached such a probability for a given experiment, it is not necessary to do subsequent experiments. Likewise, in a toxicology screen, once confirmation is achieved, it is not necessary to do further chemical tests. Methods Predictive Value of a Screening Test Bayes formula or theorem states that the probability that hypothesis H is true, given experimental results E, is the quotient of the probability of those results E when the hypothesis was known to be true, divided by the sum of the occurrence of those experimental results E in all experiments, whether or not the hypothesis was true. P(HIE) = P(EIH)I[P(EIH) + P(EIH)J where H indicates an untrue hypothesis. Bayes theorem dates from the 18th century, and its proof can be found in most statistics textbooks (e.g., 3,4). The probability that the hypothesis is true given the test or experimental results is the a posteriori probability or predictive value of a positive test result. In drug screening the hypothesis is that the urine sample truly contains the substance of interest, a drug or drug metabolite. The experimental evidence comes from the chemical tests performed on the specimen. The numerator, the probability of a where sensitivity is defined (see Glossary) as the fraction of positive tests for all the actual positive specimens. The a priori probability is the prevalence of the condition (drug present in urine) in the population being tested. The denominator is the sum of all the positive test results in all the specimens. It is therefore the sum of the sensitivity times prevalence and 1 minus the specificity times 1 minus prevalence. The specificity is defined as the fraction of negative test results of all the specimens that did not contain the drug or drug metabolite. P(EIH) + P(E!H) = (prevalence sensitivity) + (1-prevalence) (1-specificity) Therefore, with use of Bayes formula, the predictive value for a screening test is (1): Predictive value = (prevalence. sensitivity)/[(prevalencesensitivity) + (1-prevalence) (1-specificity)] When prevalence is equal to 50% or 0.50, then prevalence = (1 - prevalence) and the predictive value depends only on the sensitivity and specificity of the test. Sensitivity and specificity data are most often given by test manufacturers or validation studies as a table of true positives, TP, false positives, FP, true negatives, TN, and false negatives, FN. Defining sensitivity and specificity in these terms (see Glossary) results in the following expression for predictive value of a test (2): Predictive value = [TP/(TP + FP)] x 100 Table 1 shows examples of these calculations for a screening test (5). Cumulative Predictive Value of Two Tests If the predictive value of the first test is not equal to or greater than the acceptable level of confidence, further tests Table 1. Some Calculations A. Calculating sensitivity and specificity of a test Drug present 48 Drug absent 1 B. Calculating predictive value of above test for a prevalence of 10% Prob. of Prob. of Prevalence positive teat negative test Drug present x x 0.10 = = Drug absent x 0.90 = Totals Test positive Test negative Totals Sensitivity = TP/(TP + FN) = 48/49 = 0.98 or 98% Specificity = TN/(TN + FP) = 24/25 = 0.96 or 96% Predictive value of a positive result 0.098/0.134 = 0.73 or 73% of a negative result = 0.864/0.866 = or 99.7% x 0.90 = CLINICAL CHEMISTRY, Vol. 34, No. 8, 1988

3 must be performed (Figure 1). To add information and to allow use of Bayes formula, the second test should be independent of the first. In analytical chemistry this means that the second test should be based on a different chemical or physical property of the analyte from that used in the first test. For instance, if the first test is based on the nonaqueous solubility of the analyte, the second test might be based on its molecular size and shape or the chemical reactivity of specific functional groups. Table 2. TestIng In Series A. Testing in series: Test A, then B A positive Sensitivity = 95/100 = 0.95 Specificity = 8910/9900 = 0.90 Predictive value of a positive test = 95/1085 = 8.76% B positive B negative Sensitivity = 76/ Specificity = 940/990 = 0.95 Predictive value of positive result (A+, B+) = 76/126 = 60% B. Testing in series: test B, then A B positive B negative Drug present Drug absent Sensitivity = 80/100 = 0.80 Specificity = 9400/9900 = 0.95 Predictive value of a positive test = 80/580 = 14% A positive A n.ativ. Drug present 95 5 Drug absent Drug present Drug absent A negative Drug present 76 4 Drug absent Sensitivity = 76/80 = 0.95 Specificity = 450/500 = 0.90 Predictive value of positive result (A+, B+) = 76/126 = 60% The cumulative predictive value of multiple independent tests can be calculated by using Bayes formula as shown above. After each test, the updated a posteriori probability becomes the a priori probability, or prevalence, for the next round (Figure 1). Table 2 shows an example calculation of the cumulative predictive value for two hypothetical tests, A and B. To the degree that the two tests are correlated (not independent), the predictive value of the combination will be overestimated. The degree of independence of analytical tests can be assessed experimentally by using discriminant analysis, cluster analysis (6), or other techniques. This type of analysis has been published, for example, for thin-layer- Totals chromatographic methods by the Systematic Toxicological Analysis (STA) Committee of the International Association of Forensic Toxicologists by Schepers et al. (7) and Masumarra et al. (8). If validation studies have established the sensitivity and specificity of the two tests in series, then the cumulative predictive value can be calculated from the overall positive Tota and negative results. Table 3 shows an example of calculation of the cumulative predictive value for a screening test 95 and a confirming test. In this case the two tests need not be 990 independent. However, the more independent the tests, the more effective the combination will be. Resufta Table IA shows the data and calculations of sensitivity and specificity for a screening test derived from analysis of 25 drug-free urine samples and 49 urine samples known to -!-! contain the drug (5). The sensitivity is 98% and the specific- 100 ity is 96%. Table lb shows the calculation of the predictive gg#{174} value for this screening test for a population with prevalence of the analyte of 10%. The predictive value of a positive result is 73% and the predictive value of a negative result is 99.7%. This analysis can be used to decide the most efficient order Totais of performing the tests, assuming the cost and time for both tests being equal. Table 2 shows the calculations for sensi- 500 tivity, specificity and predictive value for two different tests. In 2A, test A is performed first, then B. In 2B, the order is reversed. Test A had a sensitivity of 95% and a specificity of 90%. Test B had a sensitivity of 80% and a specificity of 95%. In either order, A then B or B then A, 76 positive results Table 3. CalculatIng Predictive Value A. Calculating predictive value of confirming test (9) Drug Drug present absent Test S and test C s+,c#{247} 108 <1 s+,c- 8a 28 Sensitivity 084/118 = Totals Specificity 1061/1062 = B. Predictive value of confirmation, S then C Drug present Drug absent Totals Probability of positive S and positive C x x 0.90 = Predictive value of a positive result = / = or equal to 99% of a negative result = / = or greater than 99% Probability of negative S x 0.10 = x 0.90 = Prevaience 10% 90% S is Abuscreen AlA (. C is GC/MS confirmation (#{149} a Positive by a second AlA but not confirmed by GC/MS (#{149}b Drug found in other tissues (10). None ound. dnegative by a second R1A test as well as GC/MS (. OAWN (13, 14). CLINICAL CHEMISTRY, Vol. 34, No. 8,

4 (A+ and B+) were reported. However, by performing the less sensitive but more specific test first, fewer specimens had to be tested (580 vs 1085). Although the final predictive value of the combined tests is the same in either case, it is more cost effective to perform the more specific, less sensitive, test first because it reduces the number of total tests performed. Table 3A shows the calculation of the sensitivity and specificity of two tests performed in series. The data used in this example have been reported previously (9, 10). The screening test, S, was the Abuscreen RIA for Cocaine Metabolite (9). The confirmation test, C, was the gas chromatographic/mass spectrometric analysis for simultaneous determination of cocaine and benzoylecgonine reported previously (10). The samples were whole blood taken from the heart at autopsy from coroner s cases. The combined sensitivity is 92.5% (positive reported only when both tests were positive). The sensitivity of parallel tests was 99% (positive reported when either test was positive). The specificity of both test in series was >99.9. The specificity of both tests in parallel was 97%. Table 3B shows the calculation of the predictive value of the two tests in series, for a prevalence of 10%. The predictive value for a positive result was 99.0% and for a negative result, 99.83%. The predictive value for a negative test is that for the screening test only. Because this value exceeds the acceptable confidence level, specimens that had a negative screening result need not be repeated in the second test. DIscussion Confirmation When the cumulative predictive value or a posteriori probability reaches or exceeds the acceptable confidence level, then the hypothesis (analyte present in specimen) is confirmed and the result can be reported. The time and costs required to reach this confidence will depend on the efficiency of the tests chosen, the order in which they are applied, and the prevalence of the condition in the population being screened. Table 4 shows the effect of prevalence on the predictive value of a positive test result with 99% sensitivity and 99% specificity. As has often been pointed out (1,2, 11, 12), this is why screening tests that look very promising in validation studies are sometimes disappointing in applications to general populations. In validation studies, the prevalence is often 50%. A group with the disease is matched by a control group of equal size known not to have the disease. When the number of cases who have the disease equals the number who do not, then the predictive value is independent of the prevalence. In screening general populations the prevalence is often <1%. The reported prevalence of drugs of abuse in random urines in different populations ranges from 2% to 20% (13). The importance of prevalence adds uncertainty to the estimate of predictive probability of screening test results to Table 4. Effect of Prevalence on PredIctive Value of a Test When SensItIvity and Specificity Equal 99% Prevalence, % Predictive value of a positive test, % the degree that the prevalence of the condition in the population being tested is not known exactly. The prevalence can be estimated and, for conditions of national concern, such as drug abuse, is available from government epidemiological studies such as DAWN (14), as well as private studies and published papers. A change in prevalence in the population under study can result in a marked change in predictive value. The markedly improved performance of screening tests for cocaine and cocaine metabolite noted by one of the authors (9) in the 1980 s was undoubtedly due to increases in cocaine abuse prevalence as well as improvements in the assay methods and increases in the concentrations encountered. Cutoff Concentrations Concentrations chosen for cutoff values in reporting a negative or positive result will, of course, affect the apparent sensitivity and specificity of the test. One school of thought is that the specificity of a procedure should be determined by measuring samples with concentrations of analyte distributed as expected in the population to be screened. In the screening test in Table 1, at a cutoff value of 750 ng of metabolite per milliliter, 43 of the 49 specimens with drug present were correctly identified. At a cutoff of 500 ng/ml the number was 44/49; at a cutoff of 300 ng/ml, 48/49; and at a cutoff of 100 ng/ml, 49/49, or a sensitivity of 100% (5). If the acceptable criterion for detection of positive samples is sensitivity 95%, then the cutoff of 300 ng/ml might be chosen for this test on the basis of the above data. In most toxicology screening tests, the predictive value of a positive test result will not be greatly affected by the choice of the cutoff. The predictive value of a positive result for the above cutoff values with resulting sensitivities of 87.8%, 89.8%, 98%, and 100%-given a specificity of 96%- is, respectively, 71%, 71%, 73%, and 74%. However, it is possible that a lower cutoff value can decrease the specificity performance. Predictive value analysis of the effect of raising cutoff concentrations can be used to improve the performance of the test by maximizing the test efficiency and the efficiency of test combinations. Summary Because of the demand for increased accuracy in drugabuse screening of unselected ( random ) urine samples, the present discussion emphasizes: #{149} the need to establish the objectives to be achieved by the testing program #{149} the need to establish the prevalence of abused drugs in the population tested #{149} the importance of determining the distribution of drug levels for each of the drugs abused in the population being screened #{149} the objective evaluation of all tests used in screening and confirmation with respect to accuracy, cost, and application of the results. Conclusions This paper has reviewed the procedures for calculating the predictive value of screening and confirmation tests and their application to toxicology tests. We demonstrate that use of Bayesian statistics is a formalization of intuitive good practice in toxicology. This formalization can be useful in explaining or justifying confirmation procedures to other professionals, for selecting tests and the order in which they are performed, and for balancing costs of further testing 1538 CLINICAL CHEMISTRY, Vol. 34, No. 8, 1988

5 against the information value obtained. It is useful in training because it gives a logical explanation of traditional choices in analytical toxicology. Application of the predictive value model for toxicology screening tests provides an objective measure based on performance of the cost effectiveness of confirmation testing. This measure is free of bias for or against any particular chemical or physical technique. As an objective measure it can show where no further testing is required (confirmation of negatives) or where more than one additional confirmation test might be required. In summary: this review is intended to provoke some discussion on objectives in drug screening, statistical approaches for identifying and reaching those goals, and a systematic way to combine both quality and cost effectiveness. Glossary Bayes theorem: a formula for combination of probabilities to reflect increasing or decreasing confidence in a hypothesis with additional information about a case. Confidence: the probability that the diagnostic hypothesis is true. Confirmation: The increase in the posterior probability or predictive value of the tests to equal or exceed an acceptable confidence level. Confirmation test(s): additional, chemically independent test(s) performed to increase the predictive value of test results. Efficiency: the proportion (percent) of specimens correctly classified as containing or not containing the analyte of interest. False negative: a negative test result for a specimen in which the analyte of interest was actually present. False positive: a positive test result for a specimen that did not contain the analyte of interest. Posterior probability: the updated probability of the hypothesis after additional data from tests or experiments (see Predictive value of test results). Prevalence: the proportion of positive cases per population. Predictive value of test results: the percentage of all positive (or negative) test results that are true positives (or true negatives): predictive value of a positive result = true positivesl(true positives + false positives) predictive value of a negative result = true negatives/(true negatives - false negatives) Prior probability: prevalence of a condition in a population. Sensitivity: the proportion (fraction) of all the actual positive specimens that test positive: sensitivity = true positives/(true positives + false negatives) Specificity: the proportion (fraction) of all the actual negative specimens that test negative: specificity = true negativesl(true negatives + false positives) True negative: a negative test result for a specimen that did not contain the analyte of interest. True positive: a positive test result for a specimen in which the analyte of interest was actually present. References 1. Tietz NW, ed. Textbook of clinical chemistry. Philadelphia: WB Saunders, Galen RS, Gainbino SR. Beyond normality; the predictive value and efficiency of medical diagnoses. New York: Wiley & Sons, Bayes, Rev. Thomas. An essay toward solving a problem in the doctrine of chance. Philos Trans R Soc London 1763;53: Mosteller FK, Rourke REK, Thomas GB. Probability and statistics Reading, MA: Addison-Wesley Publishing Co., Package insert, Abuscreen radioimmunoassay for cocaine metabolite. Nutley, NJ: Roche Diagnostics, Massars DL, Kaufinann L. The interpretation of analytical chemical data by the use of cluster analysis. New York: John Wiley, Schepers PG, Frank JP, de Zeeuw RA. System evaluation and substance identification in systematic toxicological analysis by the mean list length approach. J Anal Toxicol 1983;7: Masumarra F, Scarlata G, Cirma G, et al. Identification of drugs by principal component analysis of standardized thin-layer chromatographic data in four eluent systems. J Chromatogr 1985;350: Spiehier VR, Sedgwick P. Radioimmunoassay screening and Gd MS confirmation of whole blood samples for drugs of abuse. J Anal Toxicol 1985;9: Spiehler VR, Reed D. Brain concentrations of cocaine and benzoylecgonine in fatal cases. J Forensic Sci 1985;30: Sheps SB, Schechter MT. The assessment of diagnostic tests: a survey of current medical research [Review]. J Am Med Assoc 1984;252: Billings PR, Bernstein M. Physicians poor at prevalence and positive predictive value. J Am Med Assoc 1986;254: Hawks R, Chiang CN. Urine testing for drugs of abuse, NIDA Monograph No. 73. Washington, DC: U.S. Govt. Printing Office, Data from the Drug Abuse Warning Network, DAW?r, Natl Institutes on Drug Abuse, Statistical Series, Annual Data, Division of Epidemiology and Statistical Analysis, Rockville, MD. CLINICAL CHEMISTRY, Vol. 34, No. 8,