Is it Method Verification or Validation, or Just Semantics? Michael Brodsky Brodsky Consultants

Transcription

1 Is it Method Verification or Validation, or Just Semantics? Michael Brodsky Brodsky Consultants

2 Objectives To define and differentiate method validation from verification Consider the requirements for method validation and expectation for in house verification from an ISO Standard perspective Review AOACI OMA and RI validation protocols Outline the requirements for a verification protocol that defines a method s fitness for purpose

3 What is Test Method Validation? Validation is the establishment of one or more performance characteristics for a test method By single laboratory validation By multi laboratory collaborative study Always by comparative analysis to a Reference Method

4 Performance Characteristics of Microbiological Methods Relative Accuracy (%Recovery) Precision (Repeatability and Reproducibility) Specificity (Selectivity for target analyte) Sensitivity (Distinguishing target from non target) Inclusivity (Range of target analytes detected by method) Exclusivity (Range of non target analytes excluded) False Positive and False Negative Rates Positive and Negative deviation Limits of Detection (LOD) Limit of Quantitation (LOQ) Scope of application

5 Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study Performance Characteristic Identification Quantitative Qualitative (P/A) Relative Accuracy No Yes No Matrix Effects (Scope) No Yes Yes Precision No Yes No Sensitivity Yes Yes Yes Specificity Yes Yes Yes Inclusivity Yes Yes Yes Exclusivity Yes Yes Yes False Positive Rate No Yes Yes False Negative Rate No Yes Yes LOD No Yes Yes LOQ No Yes No Ruggedness Yes Yes Yes Linearity/Range No Yes No

6 Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method Performance Characteristic Identification Quantitative Qualitative (P/A) Verification (Where Applicable) Relative Accuracy No Yes No Yes Matrix Effects (Scope) No Yes Yes Yes Precision No Yes No Yes Sensitivity Yes Yes Yes No Specificity Yes Yes Yes Yes Inclusivity Yes Yes Yes No Exclusivity Yes Yes Yes No False Positive Rate No Yes Yes No False Negative Rate No Yes Yes No LOD No Yes Yes No LOQ No Yes No No Ruggedness Yes Yes Yes No Linearity/Range No Yes No No

7 17025 Requirements Methods requiring validation are: Modified official methods In house developed methods Methods extended to a component, analyte or matrix not previously tested or included in validation Changes involving new technology or automation

8 Food Categories and Types

9 To help protect your privacy, PowerPoint has blocked automatic download of this picture.

10 Food Categories and Types (Based on Physiochemical and Innate Microbial Characteristics)

11 Food Categories/Matrices 1. Raw milk and dairy products 2. Heat processed milk and dairy products 3. Ready to eat, ready to reheat meat products 4. Raw poultry and ready to cook poultry products 5. Ready to eat, ready to reheat meat poultry products 6. Eggs and derivatives 7. Raw and ready to cook fish and seafoods (unprocessed) 8. Ready to eat, ready to reheat fishery products 9. Fresh produce and fruit 10. Processed fruits and vegetables 11. Infant formula and infant cereals 12. Dried cereals, fruits, nuts, seeds and vegetables 13. Chocolate, bakery products and confectionary 14. Multi component foods or meal components 15. Pet food and animal feed 16. Environmental samples (food or feed production)

12 1. Raw milk and dairy products : Types: i. Raw milks and/or fermented/acidified milks (not heat treated) ii. Raw milk based products, with high fat content and/or high background microflora

13 2. Heat processed milk and dairy products Types: i. Pasteurized dairy products ii. Sterilized or UHT dairy products iii. Pasteurized milk based products iv. Dry

14 3. Ready to eat, ready to reheat meat products: Types: i. Cooked meat products ii. Fermented or dried meat products iii. Raw cured (smoked) (aw > 0.92 iv. Raw cured (smoked) (aw < 0.92) v. Canned meat (ambient stable)

15 4. Raw poultry and ready to cook poultry products Types: i. Fresh (unprocessed) ii. Ready to cook products (processed)

16 5. Ready to eat, ready to reheat meat poultry products Types: i. Cooked products ii. Fermented or dried products iii. Raw cured (smoked) (aw > 0.92) iv. Raw cured (smoked) (aw < 0.92) v. Canned (ambient stable) i.

17 6. Eggs and derivates: Types: i. Eggs (unprocessed) ii. Egg products (heat processed) with additives (salt or sugar > 2%) iii. Egg products (heat processed) without additives iv. Dry

18 7. Raw and ready to cook fish and seafood (unprocessed) Types: i. Fish (unprocessed) ii. Shellfish (unprocessed) iii. Crustaceans (unprocessed) iv. Ready to cook fish and seafoods (processed)

19 8. Ready to eat, ready to reheat fishery products Types: i. Cooked products ii. Acidified and marinated products iii. Smoked or cured, and other processed products (aw > 0.92) iv. Smoked or cured, and other processed products (aw < 0.92) v. Canned (ambient stable fish)

20

21 9. Fresh produce and fruit Types: i. Cut ready to eat fruit ii. Cut ready to eat vegetables iii. Produce grown in or in contact with the ground iv. Sprouts v. Raw fruit/vegetable juices (unpasteurized) vi. Leafy greens vii. Vegetables and fruits (unprocessed) not described above i.

22 10. Processed fruits and vegetables Types: i. Heat processed fruit/vegetables juices ii. Canned fruits and vegetables (ambient stable) iii. Heat processed vegetables and fruits iv. Fermented/acidified vegetables i.

23

24 11. Infant formula and infant cereals Types: i. Probiotic ingredients ii. Non probiotic ingredients iii. Non probiotic infant formula iv. Probiotic infant formula v. Non probiotic infant cereals vi. Probiotic infant cereals i.

25 12. Dried cereals, fruits, nuts, seeds and vegetables Types: i. Low and IMF fruits (aw < 0.85) ii. Seasonings iii. Nuts and seeds iv. Dried fruits and vegetables (aw < 0.60) v. Dried cereals vi. Flours i.

26 13. Chocolate, bakery products and confectionary Types: i. Pastries ii. Dry powdered iii. Low moisture iv. Dry & sugared low moisture (aw < 0.85) v. Dry & sugared low moisture (aw < 0.65) i.

27 14. Multi component foods or meal Components Types: i. Composite foods with substantial raw ingredients (excluding patisserie) ii. Composite processed foods (cooked) iii. Ready to (re)heat food: refrigerated iv. Ready to (re)heat food: frozen v. Ready to (re)heat food: ambient stable (canned vi. Ready to (re)heat food: dry vii. Mayonnaise based delisalads (acid) with raw ingredients viii. Mayonnaise based delisalads (acid) with processed ingredients ix. Ambient stable acid foods (ph < 4.8) i.

28 15. Pet food and animal feed Types: i. Animal origin ingredients ii. Plant origin ingredients iii. Other ingredients iv. Dry food (aw 0.7) v. Wet food (aw > 0.7) vi. Canned vii. Animal feeds (bovine, ovine, pig) viii.animal feeds (poultry) ix. Animal feeds (fish) i. i

29 16. Environmental samples (food or feed production) Types: i. Equipment or production environment ii. Water used in the manufacturing process i. i.

30 Method Validation AOAC International

31 Performance Tested Methods Program Designed specifically to validate commercial test kits Speed and efficiency designed into the review process Single Laboratory Validation with an independent laboratory Test kit methods are reviewed annually Harmonized to determine the same performance characteristics as the OMA Pre collaborative study.

32 AOACI Pre Collaborative/PTM Study Design (SLV) (Quantitative Method) 9 Food categories (broad range of foods) ~ 20 foods (~2 food types/category)(5 portions of each) Low, medium and high levels of contamination and uncontaminated Comparison of recovery to reference method using ANOVA and comparison of means Inclusivity >50 strains of target microorganism (>100 serovars for Salmonella) Exclusivity >30 non target strains

33 AOACI Pre Collaborative/PTM Study Design (SLV) (Qualitative Method) 30 analyses for each food type: 5 replicate test portions per level the high inoculation level, 20 for the fractional positive level 5 for the uncontaminated level

34 AOACI Collaborative Study laboratories (Qualitative Method) Minimum of 10 labs with acceptable data Test samples 1 matrix 12 test portions per high analyte level 12 test portions per fractional Positive samples 12 uncontaminated test portions Comparison to reference method using Chi square analysis &/or Probability of Detection (POD models)

35 "I think you should be more explicit here in step two."

36 Qualitative Performance Characteristics (Paired Samples) Reference Method + _ Test Method + _ a c b d + or - : possible outcome of the test

37 Performance Characteristics for a Qualitative Method Sensitivity = a/a+c Specificity = d/b+d False Negative Rate = c/a+c False Positive Rate = b/b+d Positive Predictive Value= a/a+b Negative Predictive Value = d/c+d

38 Probability of Detection (POD) The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given analyte level or concentration. The POD estimate is calculated as the number of positive outcomes divided by the total number of trials. POD is concentration dependent. Several POD measures can be calculated; POD R (reference method POD), POD C (confirmed candidate method POD), POD CP (candidate method presumptive result POD) POD CC (candidate method confirmed result POD).

39 Difference of Probabilities of Detection (dpod Difference of probabilities of detection is the difference between any two POD values. dpod CP is the difference between the candidate presumptive result POD CP and the candidate confirmed result POD CC values. dpod CP = POD CP POD CC dpod C is the difference between the confirmed candidate method POD CC and reference method POD R values. dpod C = POD C POD R

40 POD Interpretation Calculate the 95% confidence interval on each dpod If the confidence interval of a dpod does not contain zero, then the difference is statistically significant at the 5% level The Excel spreadsheet for calculating RLOD values is freely available for download at:

41 AOACI Collaborative Study Quantitative laboratories Minimum of 8 labs with acceptable data Test samples 1 Matrix Low, medium and high contamination +uncontaminated Five samples tested at each level Both naturally and artificially contaminated food Compare against a reference method Repeatability, Reproducibility and differences between means

42 Method Verification

43

44 ISO/IEC17025 Requirements Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format. These official methods do not need to be fully validated. The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs

45 Method Verification Laboratory demonstrates the capability of achieving specified method performance characteristics before implementation and on an on going basis. Provides data to illustrate that the method is fit for (it s) purpose i.e. meets customer requirements Includes determination of uncertainty of measurement (quantitative methods).

46

47 Verification of Quantitative Microbiological Methods Precision (Repeatability and Reproducibility) Scope of application Uncertainty of Measurement Fitness for Purpose

48 Validation Data from Collaboratively Studied Quantitative Methods Provides reference values for RSD R and RSD r e.g. Pour Plate counting (SMEDP) e.g. RSD r 7.7% (0.077) (within analysts) e.g. RSD R 18.2% (0.182) (between analysts)

49 Validation Data from Collaboratively Studied Quantitative Methods Can also be used to estimate uncertainty of measurement e.g. Pour Plate counting (SMEDP) e.g. RSD r 7.7% (0.077) (within analysts) e.g. RSD R 18.2% (0.182) (between analysts)

50 Calculation of Combined Uncertainty Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the root sum of squares RSS).

51 Validation Data from Collaboratively Studied Quantitative Methods Calculation of Combined Uncertainty (Uc): Root Sum of Squares: (RSD r ) 2 + (RSD R ) 2 For Pour Plate (HPC) Sum of Squares: (0.077) 2 + (0.182) 2 = Uc = (0.0371) = 0.193=19.3% Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19.3% = 38.6% * 30 Observations

52 Expanded Uncertainty Expanded standard uncertainty is also known as measurement uncertainty (MU) calculated to give a confidence level of 95% using an expansion factor 'k' of: k = 2 when n is 30 or more (n = number of observations from which the SD is calculated). k = the appropriate (95% confidence level) Student distribution 't' (two tailed) factor for n<30 for n 1 degrees of freedom. (Note: Not applicable to qualitative analyses)

53 How Many Samples Are Needed for Verification??

54 How Many Samples Are Needed for Verification? No fixed number But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable 10 is the minimum requirement

55 Who s Doing What ISO: Microbiology of food and animal feed Method Verification Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing

56 ISO Protocol for the verification of reference and alternative methods implemented in a single laboratory (under review ) Minimum of 10 artificially inoculated samples Use CRM, if available For Qualitative methods inoculate with 1 5 CFU per test portion. For Quantitative methods, the levels of contamination shall cover the range of the method Analyse samples on 10 different occasions or days be performed by at least 2 technicians working independently and with separate samples and reagents.

57 How to Interpret Verification? For Quantitative methods, ISO proposes to accept: participation in interlaboratory comparisons such as proficiency testing and plotting z scores to show any trends; use of microbiological RM or CRM; recovery experiments with spiked samples using a nonselective method.

58 Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing by Health Canada (April 2015) For Qualitative Methods, Analyze 3 5 samples spiked at 3 5 times the reported or determined LOD (1 3 CFU/analytical unit) For Quantitative Methods using selective medium, measure repeatability by analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated

59 How to Interpret Verification? For Qualitative methods, ISO and HC agree:100% (comparative) sensitivity is required to demonstrate acceptability or fitness for purpose, i.e. All samples must be correctly identified For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2r < R. If R value if published. If there is no published performance characteristic data,??? Calculate Ue

60 Verification of Qualitative Microbiological Methods Scope of application Fitness for Purpose

61 Verification of a Rapid Screening Method for Salmonella Confirmed Salmonella Not-salmonella Known +ve -ve

62 ISO/IEC Requirements Fitness for Purpose The data obtained from the method approval process must show it is fit for the intended use and relevant to customers needs. Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?

63

64 References AOAC International Methods Committee Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J. The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group How to Meet ISO Requirements for Method Verification, 2015, AOAC Technical Division for Laboratory Management (TDLM), the Analytical Laboratory Accreditation Criteria Committee (ALACC) ISO/FDIS Microbiology of the food chain Method validation Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method (ISO/TC 34/SC 9) (Draft)

65 Thank you for your attention Any Questions?