In-house* validation of Qualitative Methods

Transcription

1 Example from Gilbert de Roy In-house* validation of Qualitative Methods Aspects from a non forensic but analytical chemist *In-house in your own laboratory Presented at ENFSI, European Paint group meeting 27 of September 2010 in Krakow by Bertil Magnusson Contents Validation What is it and what is it not Qualitative what is it Expression of reliability and uncertainty false rates reliability % likelihood ratio posterior probability for postive results limit of detection/threshold Example with breast cancer Example with methadone in urine Validation of qualitative methods Conclusions Validation What is validation and what is is not Method validation is the process of proving that an analytical method is acceptable for its intended purpose The intended purpose is defined in the scope of the methods Validation can start in your lab when you have a method with scope detailed instructions for your personnel using your instruments, measuring interval specified performance Validation is done after method development Normally we will have a validation set a suite of test samples designed for validation (different from any calibration samples) For correct definition vocabulary in measurement VIM How to find VIM - Search on goggle: VIM and BIPM Validation definition according to VIM FULL DEFINITION: provision of objective evidence that a given item fulfils specified requirements, where the specified requirements are adequate for an intended use But consists of two parts 4.1 Verification provision of objective evidence that a given item fulfils specified requirements (VIM 2.44) 4.2 Validation verification, where the specified requirements are adequate for an intended use (VIM 2.45) Vali dati on of

2 Qualitative not defined in VIM Some examples Identifying petrol in a sample of debris from a fire Comparing a shoe mark at a scene with shoes taken from a suspect How to define qualitative? Proposed definition of qualitative analysis Classification according to specified criteria* criteria are related to information about the composition and/or structure of materials Examples from analytical chemistry: Chemical tests involving a visible change. For example, the production of a precipitate of BaSO 2-4 in testing for SO 4 Observation of the positions or pattern of particular peaks with XRD. Observation of the positions or pattern of particular absorption bands in an IR spectrum. The possible results are yes, maybe, no. We usually talk about binary response *According to a draft Eurachem guideline on qualitative analysis Expression of reliability and uncertainty The degree of confidence can be expressed in a number of ways. For a given test method, the basic properties that need to be measured are: False positive and false negative rates - the numbers of false positive and negative results obtained on a range of samples. From these numbers the false positive and false negative rates can be calculated. Reliability % - The false positive and negative rates can be combined into a single figure expressing a percentage of correct results using this test. Likelihood ratio - The false positive and negative rates can be combined into a single figure expressed by the Bayesian likelihood ratio. Posterior probability - If the analyst by experience knows the percentage positive for a particular sample (the prior information) before the test is applied then a further reliability measure in the form of a Bayesian posterior probability can be calculated. Limit of detection The limit of detection enables the analyst to select a method capable of satisfying the customer s requirement relating to minimum detectable amount. Lets look at an example where we know the false rates and the prior Reliability in qualitative analysis Test for breast cancer using mammography A women at age forty had a positive mammography in a routine screening. 80% of women with breast cancer will get positive mammography. 9.6% of women without breast cancer will also get positive mammography. What is the probability that she actually has breast cancer? Source of data: An Intuitive Explanation of Bayes' Theorem The reliability of the test 80% of women with breast cancer will get a positive mammography. (False negative rate 20 %) - 9.6% of women without breast cancer will also get positive mammography A 40-year old women is tested positive How reliable is this result? The answer is we do not know. We need more information! What info do we need? Likelihood ratio one parameter for assessing reliability in qualitative analysis Likelihood ratio is indication of the additional information provided by a positive test result. The increase in probability for a given positive test result is indicated by this ratio. For a ratio of 1 the test is useless - the probability is the same before and after the test. Our example - 80% of women with breast cancer will get a positive mammography and 20% will be negative. We have a sensitivity of 080(and 0.80 a false negative rate of 0.2) - is 9.6 % or Re liability(%) 100% FP rate FN % rate Likelihood ratio This does now answer our question A 40-year old women is tested positive How reliable is this result?

3 Likelihood ratio used by forensic analysts Hp suspected is guilty here patient having breast cancer (Hypothesis in favour of the prosecutor) E evidence the test gave positive results Likelihood ratio P( E Hp) P( E Hp) This does now answer our question A 40-year old women is tested positive How reliable is this result? Breast cancer - posterior probability Posterior probability: probability of an object fitting a given category given a positive test result Our test results is positive for a 40-year old lady what is the probability that this is true. Here we need more information than the likelihood ratio The PRIOR information: If we have that 1% of women at age forty who participate in routine screening have breast cancer. Out of 1000 women tested 10 have breast cancer and 8 of this ten are tested positive (sensitivity of 0.8). TP 8 Out of 1000 women 990 do not have breast cancer and 95 of them are tested positive (false positive rate of 9.6 %.) FP 95 We have in total positive results (Total P) but only 8 TP The probability that she has breast cancer is 8 % % TP 8 probability % Total P Methadone in urine using Immunoassay of 1000 test samples are positive - the prior Our test results for these 1000 samples using EMIT 5 FN Rate FP Rate Incorrect result false 1 semi-quantitative Enzyme Multiplied Immunoassay Technique *Ferrara S.D; Tedeschi L.; Frison G.; Brusini G.; Castagna F.; Bernadelli B.; Soregaroli D. J.Anal.Toxicol. 1994, 18, Methadone in urine - likelihood & posterior probability Our test results is positive for methadone in urine with EMIT 1 what is the probability that this is true. 26 % of incoming samples contain methadone - the prior is a prevalence of This information is in many cases not available Out of 1000 samples 260 contain methadone and 255 are tested positive (sensitivity of 0.982). TP 255 FN 5 Out of 1000 samples 740 does not contain methadone and 3 are tested positive (false positive rate of 0.004) FP 3 We have in total positive results (Total P) and 255 TP The probability TP 255 is over 98 %! % probability % Total P Sensitivit y Likelihood ratio semi-quantitative Enzyme Multiplied Immunoassay Technique *Ferrara S.D; Tedeschi L.; Frison G.; Brusini G.; Castagna F.; Bernadelli B.; Soregaroli D. J.Anal.Toxicol. 1994, 18, It is not easy to estimate reliability in qualitative analysis It is not easy to do a relevant estimation. The following is required: devotion competence in the measuring technique/method if applicable - knowledge of the threshold value knowledge of interference scope of the method a sound knowledge about the test items matrix and analyte variation other possible similar compounds stability, inhomogeneity The prior - % positive in a group of test samples basic know-how about different performance characteristics Validation of qualitative analysis Specify the measurement requirements Document in detail the procedure to be validated in your laboratory Plan the validation Perform the validation on selected parameters Decide if the method is fit for the intended purposet Reevaluate the validation using new data from internal quality control and proficiency testing

4 Conclusion for qualitative analysis Performance characteristics to validate Several important performance characteristics The most important are Detection limit/threshold value /False negative rate Likelihood ratio/reliability % Posterior probability NOTE the posterior probability can only be calculated if we know the probability that a certain group of test samples are positive the prior information No clear guidance or ISO standard for qualitative analysis Important performance characteristics What is false negative rate? Reliability False negative rate From these can be calculated % Reliability & Likelihood ratio Limit of detection Sampling For methods with an extraction step Recovery If prior knowledge is available Posterior probability The nomenclature for qualitative testing is not fully developed. The term false negative rate can, in principle, have two quite different interpretations. 1. The frequency of negative responses given that the response should be positive. Broadly, this is the fraction of true positive test items that return negative responses. 2. The frequency of incorrect negative responses in a series of tests, that is, the fraction of the (WHOLE) testing population which returns false negatives. Here is used definition Validation set & control samples Estimation of limit of detection - reliability zones Prepare a set of test samples to be used in validation analysed by a confirmary technique (reference method) Different from any calibration samples Similar to test samples Similar range of matrices/different types of samples Similar frequency of true positive and true negative The number of samples chosen to be fit for purpose dependent on the false rates and the confidence level needed. THIS is a critical step. Prepare also samples for ongoing quality control First estimation of reliability by varying for qualitative analysis analyte level and for comparative unclear to clear mark Analyte level/ n Results clear mark Positive/Negative / / / / / /

5 Threshold value cut-off level x 1 0 "negative" "positive" x 2 > 0 The validation route Specify the measurement requirements Document in detail the procedure to be validated in your laboratory Response Plan the validation false negative region false positive region Perform the validation on selected parameters Concentration 0 Figure 1: Response versus concentration and a cut-off level. Indicated in the graph are false positive from a blank sample and false negative for a sample with analyte present. Decide if the method is fit for the intended purposet Reevaluate the validation using new data from internal quality control and proficiency testing Working groups discussion items 4-5 people/group All groups - Method development and method validation. How are they interlinked and can we clearly distinguish between them? Discuss with examples from your field of expertise. 1 How to assess sensitivity - % true positive when test applied to positive samples. How to assess false positive rate - % positive when test applied to negative samples. How to choose validation samples for the test. Discuss with examples from your field of expertise. 2 In your field of expertise how can you prepare a validation set? Will this set reflect test items coming in to your lab? Can we assess the posterior probability and have a validation set with similar probability? Can the same set be used/prepared by other laboratories? 3 Quality control - 1) In your field of expertise how can you prepare or obtain these quality control samples. 2) Can you measure the repeatability for test samples e.g. several examinations, several people etc