Water Microbiology Proficiency Test Scheme. Overview & Description

Transcription

1 National Laboratory Association South Africa Association Incorporated under Section 21 Not for Gain) P.O. Box De Havilland Crescent Persequor Park Persequor Technopark Pretoria, South Africa, 0020 Reg. No: 1994/002856/08 Tel: (+27) Fax: (+27) Water Microbiology Proficiency Test Scheme Overview & Description Reviewed by: Hanlie Badenhorst Approved By: Steve Sidney NLA-PT-I Page 1 of 7

2 1. Background The National Laboratory Association South Africa (NLA-SA) is a representative body that evolved from the previous National Laboratory Accreditation Service. Following the assumption of overall responsibility for accreditation (including laboratory accreditation) by the South African National Accreditation System (SANAS) during 1998, it was decided that a new independent body should be created. This body, the National Laboratory Association, now looks after the interests of the large number of laboratories in South Africa. The acronym NLA, which was linked to the previous National Laboratory Accreditation service, was retained, both for the purpose of continuity and because it was well-recognised and respected. The NLA SA is a section 21, Not for Gain Company and has a board of directors who are elected from its members. In addition the National Metrology Institute of South Africa (NMISA), SANAS, the SABS (Standards Division) and more recently the National Regulator Compliance Specification (NRCS) are co-opted onto the board. The NLA-SA's focus is primarily on the representation of the activities, interests and opinions of all members laboratories. Membership of the Association is voluntary and open to all companies that have laboratories. In line with its mission the NLA-SA has undertaken to provide various Proficiency Testing and Inter-Laboratory Comparisons for both its members as well as the laboratory community as a whole. This activity which was originally focussed on the Calibration/Metrology environment has now grown into various analytical disciplines. Currently the NLA-SA offers a number of analytical PT Schemes, one of which is the Water Microbiological Scheme. 2. Scope The scheme involves the distribution of freeze dried pellets to each participating laboratory and consists of six rounds for the year, with three samples per round. The three samples consist of two drinking water samples and one raw water sample. The tests included in the samples are: Heterotrophic Plate Count Total Coliforms Faecal Coliforms E coli. Each laboratory is requested to perform duplicate analyses of each sample for all the determinands. At the beginning of 2011 a further four additional determinands were offered to participants during rounds 2, 4 and 6. The tests to be analysed in these rounds comprise of the following: Clostridia Enterococcus Faecal Streptococci Pseudomonas 3. Technical Committee A Technical Committee (TC) comprising of suitable experts in the field has been established. The committee will meet to discuss the results and any matters arising. They will also comment on individual results for future improvement and to determine training needs. NLA-PT-I Page 2 of 7

3 Programme Information regarding the annual programme is contained in the following document which can be obtained from the NLA-SA Office. NLA-PT-I-07-xx Water PTS brochure 4. Evaluation of Results General Comments The goal of this scheme is to provide water microbiology test laboratories, both accredited and non-accredited, the opportunity of participating in a proficiency scheme as a measure of their competency to perform various tests. The supplier and sample evaluator is IFMQS based outside Sydney in Australia. They are Guide 43 / ILAC G13 accredited as a PT Provider, as well as ISO/IEC accredited for their laboratory activity. System Design The objective of the NLA WM PT Scheme is to help laboratories with an external mechanism for having their results compared with a relatively large sample of other laboratories in the same field (water microbiology). The samples are evaluated by IFMQS for both stability and homogeneity and the three pairs of samples provided ((A, B & C) are manufactured to have different ranges of determinands and thus should enable the laboratories to get a good picture of how they perform. Whilst it is not possible to guarantee a single target value for evaluation purposes, the organizers are confident that the consensus value is representative of each sample. Duplicate measurements were requested, in order to help provide participating laboratories with a measure of both their repeatability (within lab result) as well as their overall measure of performance (between lab results) relative to all the other participants. Homogeneity testing For evaluating inter-laboratory comparisons the objective of homogeneity testing is to establish a suitably small sample variability, where the samples are sufficiently homogenous. Initial testing was conducted by IFMQS during the sample preparation stage, and once the samples were prepared and packaged, at least 10 samples were selected at random for homogeneity testing. The tests selected are those that are considered to best indicate any significant differences in the samples. The NLA has evaluated this data and also concluded that the samples are sufficiently homogenous and fit for the purpose of this scheme. Any outlier results identified are not considered to be attributable to sample variability. Statistics Employed ( See Annexure A for further details) In order to use as many results as possible and not have to make decisions with regard to outliers, robust indicators were used. NLA-PT-I Page 3 of 7

4 The Robust indicators include both the Median and the Normalised Inter Quartile Range (NIQR) and these were checked against the manufactured data to ensure that there was comparability. Where appropriate, all the results were transformed into log 10 (count) and these are reported. The raw results were sent to all participants about 2 weeks after the closing of data submission and these can be used for investigative purposes where necessary. The results were then used to establish Z Scores for Within Laboratory Results and Between Laboratory Results. 5. Z-scores & Evaluation Criteria A convenient and internationally accepted statistical method for analyzing test results is to calculate a Z-score for each laboratory's result. A Z-score is a normalised value which gives a "score" to each result, relative to the other numbers in the data set. A standard form for the calculation of Z-scores is: where ( x i ) is the ith result and ( x ) is the assigned value (eg mean or median) s is an estimate of the spread of all results (eg robust standard deviation or fitness for purpose criteria) A Z-score value close to zero therefore means that the result agrees well with those from the other laboratories. z i x i x s The Z-score approach described above, may be based on the mean ( x ) and standard deviation (s) of the set of results. However, these "classical" statistics are significantly influenced by the presence of extreme results (ie inordinately high or low values) in the data set. A robust alternative to the mean and standard deviation is the median and normalised inter-quartile range (IQR) respectively. Both the median ( x ) and the IQR (s) are based on the ordered results. The NLA WM PT Scheme has used these robust alternatives to evaluate the results. The Z-score for each individual determinand has been evaluated according to the recommendations of SANS17043 as follows: NLA-PT-I Page 4 of 7

5 A score of Z 2 is considered Satisfactory A score of between 2 < Z < 3 is considered Questionable A score of Z 3 is considered Unsatisfactory NLA-PT-I Page 5 of 7

6 Annexure A To statistically evaluate the participants results, the NLA uses Z-scores based on robust summary statistics (the median and normalised IQR). Where pairs of results have been obtained, two Z- scores are calculated - a between-laboratories Z-score and a within-laboratory Z-score. These are based on the sum and difference of the pair of results, respectively. Suppose the pair of results are from two samples called A and B. The median and normalised IQR of all the sample A results are denoted by median(a) and A), respectively. (Similarly for sample B.) A simple robust Z-score (denoted by Z) for a laboratory s sample A result would then be: Z A median( A) A) The standardised sum (denoted by S) and standardised difference (D) for the pair of results are: S ( A B) 2 ( B A) D if median(a)< median(b) or 2 ( A B) D otherwise 2 Each laboratory s standardised sum and difference are calculated, followed by the median and normalized IQR of all the S s and all the D s - i.e. median(s), D), etc. The between-laboratories Z-score (denoted by ZBW) is then calculated as the robust Z-score for S and the within-laboratory Z-score (ZWI) is the robust Z-score for D, i.e. ZBW S median( S) S) D median( D) ZWI D) The calculated Z-scores are tabulated in the report for each round of the scheme and these results are assessed, based on their Z-scores. An outlier is defined as any result/pair of results with an absolute Z-score greater than or equal to three, i.e. Z >= 3 or Z =< -3. This outlier criteria, Z >3, has a confidence level of about 99% (related to the normal distribution) - i.e. there is a less than 1% chance that the result(s) is a true member of the population and it is far more likely that there is a problem with this result/pair of results. Similarly a Z-score cut-off of two has a confidence level of approximately 95%. Laboratories which have a Z-score in this region (i.e. 2< Z <3) are encouraged to take a close look at their results. When interpreting results which have been identified as outliers, the sign of the Z-score and the design of the program must be considered. For both uniform and split pairs a positive betweenlaboratories outlier (i.e. ZBW > 3) indicates that both results for that pair are too high. Similarly a negative between laboratories outlier (i.e. ZBI < -3) indicates that the results are too low. For uniform pairs, where the results are on identical samples, a within-laboratory outlier of either sign (i.e. ZWI > 3) indicates that the difference between the results is too large. NLA-PT-I Page 6 of 7

7 In some circumstances it may not be possible to calculate a simple robust Z-score using the formula: [ X median( X )] Z X ) For example, if the normalised IQR was equal to zero (which could occur if more than 50% of the results submitted by participants were identical and equal to the median) then it is not possible to calculate robust Z-scores using this formula. In other circumstances it may be possible to calculate a simple robust Z-score using this formula, but the spread of results (as measured by the normalised IQR) might be so small that even a slight deviation from the median will result in an outlier. In yet other circumstances the spread of results (as measured by the normalised IQR) might be so large that it is extremely unlikely that any result would ever be classified as an outlier. If the normalised IQR is equal to zero, or if the spread of results is too large or too small, in the opinion of the technical adviser, then a target coefficient of variation (CV) is used to calculate Z- scores. These z scores are calculated by [ X median( X )] Z [target CV(X) median( X )] where the target CV is expressed as a decimal. The actual value used as the target CV to calculate such Z-scores is chosen in consultation with the technical adviser and usually takes into account historical data (most likely obtained from previous rounds of the program, or similar interlaboratory testing programs). NLA-PT-I Page 7 of 7