Mathematical Microbiologists: Why we have to return to our square roots to uncover uncertainty (of measurement) in Quantitative PCR (qpcr) J. Ian Stuart George Zahariadis Marina Salvadori
Objectives 1. Identify why Uncertainty of Measurement is required for qpcr and why it is important clinically 2. Demonstrate why you may be understating your uncertainty if you use a black box approach 3. Present appropriate methods for UOM calculation in qpcr 4. For discussion: A) The need for universal standards, B) Why clinical samples maybe better for UOM determination
Uses of viral load qpcr As surrogates for viral resistance For diagnosis of infection/disease As a trigger to start pre emptive therapy in order to prevent development of disease For treatment monitoring or safety monitoring Clearly a qpcr result has clinically important implications
What is our certainty in qpcr? Experience has taught us there can be a lot variability in measurements of any analyte This can occur both within a laboratory and between laboratories Clinicians use laboratory results as the foundation to implement treatment How can a laboratory handle this uncertainty when reporting a measured value?
Next 4 slides are a summary of a study by Dr. J Preiksaitis on CMV proficiency panels tested at international laboratories that highlight points regarding uncertainty of measurement(1) CMV DNA Copies/ml (log10) 7.00 6.50 6.00 5.50 5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Clinical sample CMV viral panel sample 02 09 07 08 04 11 03 12 06 10 05 01 Expected result based on stock quantified by reference laboratories Positive but not quantifiable (assigned lowest detectable value) CMV Sample Number
Summary of CMV Qualitative Results (constructed samples) 35 panels/33 labs Sample No EM based expected result copies/ml (log10) Reference lab expected result copies/ml (log10) Negative (%) Number of panels Positive-NQ (%) Positive-Q (%) 02 0.0 0.0 34 (97) 0 1 (3) 09 0.0 0.0 33 (94) 0 1 (3) 07 1.5 2.0 26 (74) 6 (17) 3 (9) 08 2.5 3.0 4 (11) 4 (11) 27 (77) 04 3.5 4.0 0 1 (3) 34 (97) 11 3.5 4.0 0 2 (6) 33 (94) 03 4.5 5.0 0 0 35 (100) 12 4.5 5.0 0 0 35 (100) 06 5.5 6.0 0 0 35 (100) One test was invalid Pos-NQ: positive but not quantifiable Pos-Q: positive with quantifiable results
Summary of CMV Quantitative results (constructed samples) 35 panels/33 labs Sample No EM based expected result copies/ml (log10) Reference lab expected result copies/ml (log10) Number positive GM ±SD copies/ml (log10) Median (range) copies/ml (log10) 07 1.5 2.0 08 2.5 3.0 04 3.5 4.0 11 3.5 4.0 03 4.5 5.0 12 4.5 5.0 06 5.5 6.0 9 2.2 ± 0.44 0 (0-2.78) 31 3.1 ± 0.58 3.01 (0-4.32) 35 3.89 ± 0.52 4.02 (2.33-5.08) 35 3.84 ± 0.52 3.95 (2.62-5.01) 35 4.83 ± 0.44 4.89 (3.42-5.89) 35 4.80 ± 0.49 4.90 (3.68-5.91) 35 5.59 ± 0.52 5.51 (4.65-6.73) Geometric mean; negative results were excluded
CMV Qualitative and Quantitative results (clinical samples) Clinical Sample Number #10 #05 #01 Qualitative Result Negative (%) 13 (37) 0 0 Pos-NQ (%) 9 (26) 1 (3) 0 Pos-Q (%) 13 (37) 34 (97) 35 (100) Quantitative Result copies/ml (log10) GM±SD 2.78 ± 0.72 3.89 ± 0.53 3.97 ± 0.47 Median (range) 2.24 (0-4.18) 3.87 (2.73-4.89) 3.99 (3.08-5.05) GM=Geometric mean; negative results were excluded
Ground Rules The methods in the following presentation are based on fictionalized data for CMV quantitative PCR. While some of the methods presented are sound, the results and error limits presented ARE NOT APPLICABLE TO ANY PATIENT POPULATION. Please consult your own laboratory for appropriate guidance on the significance of a result in quantitative PCR. Please reserve questions until the end of the presentation.
No! This is not Driver s Ed_ despite the figures to the right!
Some equations
The Problem with Inverse Regression
Another way to view this: From Miller 2006 (2)
The Wrong Way to do UOM
About ½ log difference over the entire range
Proper ways to Calculate UOM 1) The Box Method (note that the box is no longer black) 2) Taylor (Delta) Method 3) Bootstrapping 4) According to Fieller s Theorem
A graphical Approach
Problems with the box: Assumes normal distribution and homoscedastic data Only gives an approximation of UOM May underreport uncertainty Lacks mathematical proofs Advantages: Easy to use: Linest function in Excel Can be done manually for quick evaluations
About 0.7 to 0.95 log difference throughout the range
And you thought that you were done with calculus after your undergraduate courses!
About ½ log difference through the entire range. Note Convergence of the error bars around the mean Ct value (17)
Advantages: Mathematically more intuitive than other methods Easier to compute Disadvantages: Only gives an approximation of UOM Falls apart when the denominator of the ratio has a mean that approaches zero Assumes normal distributions and homoscedastic data
Fieller s method: Developed for The Biological Standardization of Insulin The Delta method only considers positive ratios: In Fieller s method, a negative ratio is considered and permits a quadratic solution for variance: Proof of Fieller s theorem is widely available and is beyond the scope of this discussion.? Could Fieller s conviction that a Ratio is a Ratio have been borrowed by a certain Canadian Politician who stated No, a proof is a proof. What kind of a proof? It's a proof. A proof is a proof, and when you have a good...?
About 1 log difference over the entire range
Advantages: Conservative estimates of error Quadratic formula permits easy computation One of the best recognized methods for evaluating error in ratios Disadvantages: Assumes normal distributions and homoscedastic data
THE MANHATTAN PROJECT? Now how can that apply to uncertainty of measurement for qpcr?
Bootstrapping (a special case of Monte Carlo Sampling) Resample the original sample with replacement k times k usually equals 1000 or greater Quote (Fox 2008): The population is to the sample as as the sample is to the bootstrap sample. So: 1)The bootstrap observations are analogous to the original observations 2)The bootstrap mean is analogous to the sample mean which is analogous to the population mean 3)The distribution of the bootstrap sample means is analogous to the distribution of means (n) drawn from the original sample
How to bootstrap
From (Ref. 6)
From (Ref. 7)
Advantages of Bootstrap: Relatively easy to perform Data does not have to be normally distributed or homoscedastic Mathematically simple Disadvantages of Bootstrap: May be computationally intensive (i.e. 1000 plus loop commands creating a matrix of 30 000 plus elements)
About 0.7 to 1 log difference over the entire range
For Discussion 1) The Need for standardized universal samples (such as those currently available for HIV, Hepatitis C and CMV): Healthcare portability Standardized reporting Inter laboratory comparisons 2) Standardized samples vs. patient samples for UOM determination: Precision vs. Accuracy
References 1) Preiksaitis, J. Hayden, R., Tong, Y. et al. Assessing the Impact of the 1rst WHO International Standard for Human Cytomegalovirus (HCMV) DNA on Result Harmonization. World Congress of Transplantation conference 2014. 2) Miller, J. N. Uncertainties in concentrations estimated from calibrations experiments. http://www.rsc.org/images/concentrations calibrationexperiments technical brief 22_tcm18 214840.pdf. 2006 3) Parker, P., Vining, G., Szarka III, J., Johnson, N. The Prediction Properties of Inverse and Reverse Regression for the Simple Linear Calibration Problem. http://ntrs.nasa.gov/search.jsp?r=20110016499. 2010 4) Beyene, J. and Moineddin, R. Methods for confidence interval estimation of a ratio parameter with application to location quotients. BMC Medical Research Methodology. 2005. 5: 31 38 5) Fox, F. and Weisberg, S. Bootstrapping Regression Models in R. An Appendix to An R Companion to Applied Regression, Second Edition. http://socserv.mcmaster.ca/jfox/books/companion/appendix/appendix Bootstrapping.pdf. 2012 6) http://www.texample.net/tikz/examples/bootstrap resampling/ 7) http://people.revoledu.com/kardi/tutorial/bootstrap/bootstrap.htm