MD-TIP Workshop at UVA Campus, 2011 Statistical Considerations: Study Designs and Challenges in the Development and Validation of Cancer Biomarkers Meijuan Li, PhD Acting Team Leader Diagnostic Devices Branch Division of Biostatistics, OSB/CDRH/FDA Silver Spring MD
Evaluation of Cancer Biomarkers FDA regulation and guidance on diagnostic devices for obtaining cancer biomarker measurements Intended uses of biomarkers (e.g., diagnosis, screening, monitoring, risk assessment, treatment selection) Study designs and methods for evaluating biomarkers according to their intended use Common statistical review issues arising from deficiencies in study design and study conduct Imputation of missing biomarker results (intent-todiagnose analysis) Bridging from a clinical trial assay to a market ready test
Intended Uses of Biomarkers Diagnosis, the identification of the presence of cancer Screening, enabling intervention at an earlier and potentially more curable stage than under usual clinical diagnostic conditions Monitoring, monitoring of cancer response during therapy, with potential for adjusting level of intervention (e.g. drug dose) Risk prediction, leading to preventive interventions for those at sufficient risk Treatment selection, predicts safety, efficacy of a specific therapy, thereby providing guidance in selecting it for patients or tailoring its dose.
Study Design Training Validation Description Exploratory Confirmatory Goal Identify markers performance for further study Unbiased estimation of marker performance Sample Size Small Moderate to large Subjects Design Representative of IU population Representative of IU population Avoid confounding with technical variations; align all statistical goals with clinical objectives based on IU/IFU; align the statistical design with study goals and the study populations
Analytical Validation of Cancer Biomarkers Precision (repeatability, reproducibility: closeness of repeated results, e.g. repeatability standard deviation) Sensitivity, limit of detection (limit of quantification) Specificity (interference, cross-reactivity) Sample type / matrix Performance around the cut-off Carryover, cross-hybridization, contamination Linearity
Clinical Validation: Performance Measures for Qualitative Biomarkers Sensitivity and specificity Likelihood ratio of positive test and likelihood ratio of negative test Positive predictive value and negative predictive value and prevalence Positive and negative percent agreement Receiver Operating Characteristic (ROC) plots and the area under the ROC (AUC) for ordinal or quantitative test with a cutoff The utility function
Clinical Validation: Performance Measures for Quantitative Biomarkers Trueness (closeness to correct result, on average, e.g. estimated bias with respect to a reference result) Slope and intercept from a linear regression (many different kinds), scatter plots Bias (mean difference) between the new biomarker test and the reference method 95% limits of agreement, Bland-Altman difference plots
A Biomarker Test Better than Chance Test Cancer Absent Present Total 65 10 75 + 20 42 62 Total 85 52 125 True positive rate (sensitivity) = 81% (42/52) False positive rate (1 specificity) = 24% (20/85) Positive predictive value = 68% (42/62) Negative predictive value = 87% (65/75 65/75)
Agendia Mammaprint Gene Signature for Time to Breast Cancer (N=302) Predictive value Years of Follow-Up Risk Group Low High 5 10 0.05 (0.01-0.09) 0.22 (0.16-0.28) 0.10 (0.04-0.15) 0.29 (0.22-0.35)
Proportion alive at 10 years Added Value Over Clinical Risk Groups Gene Clinical Signature N Survival Probability* Low Risk Low Risk 52 0.88 (0.74 to 0.95) Low Risk High Risk 28 0.69 (0.45 to 0.84) High Risk Low Risk 59 0.89 (0.77 to 0.95) High Risk High Risk 163 0.69 (0.61 to 0.76) * Buyse et al JNCI 2006
Challenges: Cancer Biomarker Performance Studies They are observational study; patients are not randomized; there are potential confounding factors Many sources of bias can be introduced causing a biased or spurious association Many cancer biomarkers are not a specific cancer marker, and its levels in a patient are affected by many other factors
Sources of Bias in Biomarker Performance Studies Selection bias convenience sampling of available specimens Spectrum bias advanced stage of disease vs. healthy patients enrichment with cases outside of IU population Verification bias disease status not verified in all subjects by the reference standard Imperfect Reference Standard Bias
Sources of Bias in Biomarker Performance Studies Ordering bias order in which results are taken by test, comparator, and reference standard is not randomized order in which disease and non-disease subjects are tested is not randomized. for predictive tests, test result is taken AFTER onset of target condition! Missing biomarker results Test interpretation, integrity, and context bias Device users / operators not masked to true disease status. Access to other clinical information not consistent with clinical practice.