Assessment of Audit Methodologies for Bias Evaluation of Tumor Progression in Oncology Clinical Trials

Transcription

1 CCR FOCUS Assessment of Audit Methodologies for Bias Evaluation of Tumor Progression in Oncology Clinical Trials Jenny J. Zhang 1, Lijun Zhang 1, Huanyu Chen 1, Anthony J. Murgo 2, Lori E. Dodd 3, Richard Pazdur 2, and Rajeshwari Sridhara 1 Abstract As progression-free survival (PFS) has become increasingly used as the primary endpoint in oncology phase III trials, the U.S. Food and Drug Administration (FDA) has generally required a complete-case blinded independent central review (BICR) of PFS to assess and reduce potential bias in the investigator or local site evaluation. However, recent publications and FDA analyses have shown a high correlation between local site evaluation and BICR assessments of the PFS treatment effect, which questions whether complete-case BICR is necessary. One potential alternative is to use BICR as an audit tool to detect evaluation bias in the local site evaluation. In this article, the performance characteristics of two audit methods proposed in the literature are evaluated on 26 prospective, randomized phase III registration trials in nonhematologic malignancies. The results support that a BICR audit to assess potential bias in the local site evaluation is a feasible approach. However, implementation and logistical challenges need further consideration and discussion. Clin Cancer Res; 19(10); Ó2013 AACR. Introduction Progression-free survival (PFS) is defined as the time from randomization to either disease progression or death, whichever occurs first. When PFS is the primary efficacy endpoint of a clinical trial, the U.S. Food and Drug Administration (FDA) and other regulatory authorities have generally required a blinded independent central review (BICR) under the assumption that the investigator or local site evaluation could potentially be biased because of the subjectivity in the measurement and interpretation of PFS. However, this approach may lead to more than 30% disagreement at the patient level between the BICR and local site evaluation assessments and/or among the independent reviewers themselves. These disagreements have been attributed to a variety of reasons, including selection of different target lesions by the reviewers (1). In addition, because treatment is generally changed after local site evaluation determined progression resulting in no further protocol-specified Authors' Affiliations: 1 Division of Biometrics V/Office of Biostatistics/ Office of Translational Sciences and 2 Office of Hematology and Oncology Products/Office of New Drugs, Center for Drug Evaluation Research, U.S. Food and Drug Administration, Silver Spring; and 3 Biostatistics Research Branch, National Institute of Allergy and Infectious Disease, NIH, Bethesda, Maryland Current address for J.J. Zhang: Gilead Sciences, Foster City, California. Corresponding Author: Rajeshwari Sridhara, FDA, WO21, New Hampshire Avenue, Silver Spring, MD Phone: ; Fax: ; rajeshwari.sridhara@fda.hhs.gov doi: / CCR Ó2013 American Association for Cancer Research. progression assessments before the BICR is conducted, missing data and informative censoring are limitations of BICR-determined PFS analyses that may result in biased treatment effect estimates (2). Informative censoring results, when patients declared to have progressive disease by the local site evaluation, are censored by the BICR due to lack of further tumor assessments after local site evaluation-determined progression. Note that local site evaluation assessments may include local radiographic reads and/or other clinical assessments. BICR radiologists are not part of the clinical trial investigation, typically do not have information about clinical assessments, and are blinded to treatment assignment. The role of BICR was first examined by Dodd and colleagues in 2008 in 6 phase III oncology trials (2), in which differences between local site evaluation and BICR did not result in different conclusions about treatment efficacy, despite relatively high discrepancy rates at the patient level. The Pharmaceutical Research and Manufacturers of America (PhRMA) PFS Working Group conducted a meta-analysis of 27 trials, which showed that while discrepancies in determining the progression dates were observed on average in 50% of patients, the relative treatment effect as measured by a HR was similar when assessed by either local site evaluation or BICR (3). To further confirm these results, the FDA conducted a metaanalysis of 28 randomized, phase III trials submitted for review in consideration of approval across 9 nonhematologic malignant tumor types with BICR- and local site evaluation assessed PFS results reported (4). Note that thereissomeoverlapofthetrialsincludedinthefda

2 CCR FOCUS analysis with that of the analyses in Dodd and colleagues 2008 (2) and Amit and colleagues 2011 (3). The FDA analysis showed that there existed a high degree of association between local site evaluation and BICR estimates of the PFS treatment effect [r ¼ 0.954, 95% confidence interval (CI): ]. The overall ratio of HRs (BICR vs. local site evaluation) was 1.03 (95% CI: ), indicating only a 3% difference between the 2 evaluations. Throughout, a HR less than 1 indicates a treatment effect in favor of the experimental therapy. Subgroup analyses of blinded versus open-label trials, interim versus final analysis results, and first-line versus subsequent line indications all showed similarly high correlations (4). Although an inherent measurement error exists in the reading of radiographic scans and disagreements between reviewers at the patient level are commonly observed, regulatory considerations for drug approval are based on the relative treatment effect at the population level. Given that the FDA meta-analysis corroborated the high degree of association between local site evaluation and BICR, PFS treatment effects purported in recent publications, complete-case BICR may not be necessary in many oncology trials. These findings motivate the exploration of a random sample-based BICR audit as an alternative method for bias evaluation to assess whether or not consistency of the treatment effect can be concluded. The idea behind the audit strategy is to increase our confidence in the local site evaluation results of PFS by conducting a BICR in a random sample of patients. The main saving of such a strategy lies in the situation in which there is no actual bias in the local site evaluation results and only a partial BICR audit is needed to confirm that fact. Other potential benefits include a reduction in trial complexity and burden to investigators and avoidance of some missing data issues. Currently, 2 methods have been proposed in the literature for a random sample-based BICR audit. In 2011, Dodd and colleagues (5) proposed a 2-stage procedure based on a HR estimator to evaluate the consistency of treatment effect (as measured by a HR) between the BICR-audited assessments and the local site evaluation assessments that is more efficient than the standard estimator based on the audit subset alone. Amit and colleagues (3) proposed a procedure based on differential discordance rates of local site evaluation versus BICR between the treatment and control arms. These 2 approaches will be respectively referred to herein as method A and method B. The objective of this article is to evaluate the performance characteristics of these 2 audit methods. Materials and Methods Brief summaries of the 2 audit methods are given below. We refer the reader to the original publications by Dodd and colleagues 2011 (2) and Amit and colleagues 2011 (3) for more details. The goal of method A is to provide assurance that the local site evaluation PFS treatment effect estimate is unlikely to be substantially biased. Thus, a BICR audit should only be considered when the local site evaluation HR indicates a clinically meaningful and statistically significant effect in favor of the experimental arm. The method is a 2-stage procedure. First, the BICR-based HR is estimated on the audited subset. Audited subjects were selected as a simple random sample from all subjects in the study. If the HR is confirmed to be significant, then the process concludes. If not, then a complete-case BICR is conducted. A more efficient estimator of the BICR HR simultaneously incorporates information from the patient-level local site evaluation data on all cases and the retrospective random sample BICR audit cases. A formula to estimate the audit size is also provided, which depends on the effect size and the minimum important difference (MID). The MID is a threshold value (e.g., HR ¼ 1) used in the proposed 2-stage testing procedure. The upper bound of the confidence interval of the BICR HR estimate is compared with the MID to determine whether consistency of the PFS treatment effect has been verified (5). The basis of method B is to use differential discordance as a measure to detect evaluation bias. Two measures are defined in Table 1: the early discrepancy rate (EDR) is the frequency that the local site evaluation declares progression earlier than BICR andthelatediscrepancyrate (LDR) is the frequency that the local site evaluation declares progression later than BICR. The differential discordance for each measure is the difference between the rate on the experimental arm and that on the control arm. A differential discordance beyond a certain threshold is suggestive of bias being present in the local site evaluation assessment, although the method does not quantify the uncertainty in estimation of the differential discordance. To implement the method, a threshold of differential discordance that triggers complete-case BICR Table 1. Agreement between LE and BICR from Amit and colleagues 2011 BICR PD No PD LE PD a ¼ a1 þ a2 þ a3 b No PD c d NOTE: Adapted from European Journal of Cancer, Amit and colleagues, Copyright 2011, with permission from Elsevier (3). Abbreviations: PD, progressive disease; LE, local site evaluation. a1: number of agreements on timing and occurrence of PD. a2: number of times LE declares PD later than BICR. a3: number of times LE declares PD earlier than BICR Clin Cancer Res; 19(10) May 15, 2013 Clinical Cancer Research

3 Evaluation of Audit Methodologies is required; increasing the threshold decreases the sensitivity of the method to detect bias while improving its specificity to rule it out. Threshold values ranging from to and BICR audit sizes of 100 to 160 patients were recommended on the basis of simulation studies in Amit and colleagues 2011 (3). A negative differential discordance for EDR and/or a positive differential discordance for LDR are indicative of bias in the local site evaluation result in favor of the experimental arm, which would trigger a complete-case audit. A negative differential discordance for EDR means a higher rate of local site evaluation progressions being called earlier than BICR on the control arm, and a positive differential discordance for LDR means a higher rate of local site evaluation progressions being called later than BICR on the experimental arm (3). EDR and LDR are calculated as EDR ¼ b a þ þ a3 b and LDR ¼ c þ a2 using the cell values from Table 1. b þ c þ a2 þ a3 The following hypothetical example will be used to illustrate the calculations. For example, in the treatment arm of a study, a total of 90 patients were assessed to have progressive disease by both BICR and local site evaluation (cell a in Table 1), of which 50 patients had agreement between BICR and local site evaluation on the timing and occurrence of progressive disease (a1), 25 patients had local site evaluation declaring progressive disease later than BICR (a2), and 15 patients had local site evaluation declaring progressive disease earlier than BICR (a3). A total of 30 patients were assessed to have progressive disease by local site evaluation but not BICR (cell b in Table 1), and 25 patients were assessed to have progressive disease by BICR but not local site evaluation (cell c in Table 1). Seventy patients did not have progressive disease by either local site evaluation or BICR assessments (cell d in Table 1). According to the formulas in Table 1, the corresponding EDR and LDR for the experimental arm are (30þ15)/(50þ25þ25þ30) ¼ 45/120 ¼ and (25þ25)/(30þ25þ25þ15) ¼ 50/ 95 ¼ 0.526, respectively. Similar calculations are needed for the control arm. The differential discordance for EDR/LDR would be the difference between arms for each rate. The performance evaluation of these 2 proposed audit methods is based on 26 randomized, superiority phase III trials submitted for review in consideration of approval in nonhematologic malignancies across 9 tumor types (Table 2). Note that these 26 trials are a subset of the 28 trials included in the FDA meta-analysis reported in Zhang and colleagues (4) with adequate data for evaluation to determine whether a random sample-based BICR audit is a viable alternative to a complete-case BICR with respect to its ability to detect bias in the local site evaluation. As a result of several trials having multiple cohorts or multiple treatment arms, the number of analysis units or randomized comparisons was greater than the number of trials (i.e., 31 instead of 26). Trials with Table 2. Summary of study characteristics Meta-analysis Study characteristics trials (N ¼ 26) Tumor type MBC 6 RCC 7 MCRC 4 Other a 9 Design A vs. B/AþB vs. A/Placebo or BSC 4/6/16 Open-label/double-blind 11/15 Interim/final analysis 9/17 1:1/2:1 randomization 18/8 First/subsequent b /maintenance line 12/12/3 Sample size Median Min, max 171, 1286 Discordance measure Control (n ¼ 31) Experimental (n ¼ 31) % LE-BICR discordance on censoring status Mean (SD) 23.8% (9.7%) 23.0% (7.7%) Median 23.7% 24.5% Min, max 9.1%, 43.3% 11.1%, 36.9% % LE-BICR discordance on timing of PD (7-day window) Mean (SD) 21.5% (8.1%) 20.4% (8.7%) Median 21.4% 20.2% Min, max 8.4%, 38.7% 6.1%, 41.4% Abbreviations: MBC, metastatic breast cancer; MCRC, metastatic colorectal cancer; RCC, renal cell carcinoma; BSC, best supportive care; LE, local site evaluation. a Includes trials in non small cell lung cancer (3), pancreatic neuroendocrine tumors (2), soft tissue sarcoma (2), gastrointestinal stromal tumor (1), ovarian cancer (1), carcinoid tumors (1). b One trial (#12 in Table 4) included both first and second line patients and is double counted here. multiple treatment arms were weighted accordingly to account for correlation. Table 3 provides a summary of the performance evaluation strategy and summary measures of the 2 audit methods. Because all trials evaluated had a complete-case BICR conducted, random sample audits are conducted 10,000 times for each trial to assess the performance of method A. Summary measures of the mean audit size, percentage of complete-case audits, and percentage of audits confirming the local site evaluation result (i.e., where consistency of the PFS treatment effect is concluded) out of the 10,000 simulated replicates were obtained. Note that these replicate Clin Cancer Res; 19(10) May 15,

4 CCR FOCUS Table 3. Performance evaluation summary measures of audit methods (for each study) Method A (NCI method) Method B (PhRMA method) Evaluation strategy Conduct 10,000 random-sample audits of varying size Conduct 10,000 random-sample audits of fixed size (n ¼ 160) Summary measures Mean audit size Mean differential discordance for EDR Percentage complete-case audits a Mean differential discordance for LDR Percentage audits confirming local site evaluation result % complete-case audit a recommended a A complete-case audit is an audit with size ¼ 100%. audits are conducted for performance evaluation purposes only; in practice, these procedures would be implemented a single time. For method B, our evaluations calculated the differential discordance for both the EDR and LDR. The audit size was fixed at 160 patients, the maximum audit size recommended by Amit and colleagues 2011 (3); analyses using other audit sizes were also conducted and gave similar results. For each trial, a random sample of 160 patients was drawn and the differential discordances for EDR and LDR were calculated. To assess performance characteristics, the random samples were repeatedly drawn 10,000 times; summary measures of the mean differential discordance values and the percentage of times a complete-case audit was recommended (i.e., bias was detected) out of the 10,000 replicates were obtained for each study. Under method B, a trial was considered to have a biased local site evaluation result, and thus a complete-case audit was recommended, if the differential discordance for EDR or LDR passed the specified threshold value. Results Table 2 summarizes the level of local site evaluation BICR discordance between treatment arms across the 26 trials, divided into disagreements on censoring status and the timing of progression given a 7-day window. Weseethatthediscordanceratesarearound20%across arms for both categories. Table 4 presents the performance measures for the 2 audit methods for all evaluated trials; these results will be discussed and described in the figures below and summarized in Table 5. Note that because the audit size for method B was fixed at 160 patients for all studies, no mean audit size was reported. Figure 1 assesses method A by looking into the relationship between the mean audit size for each trial and the upper bound of the 95% CI of the local site evaluation HR estimate. The cluster of circles at mean audit size of 100% are those trials for which complete-case BICR audits were needed in all 10,000 replicates. Those trials all had upper 95% CI bounds of the local site evaluation HR greater than This means that, as expected, trials with borderline or nonsignificant local site evaluation results would be recommended complete-case BICR audits. For all the other trials, the mean audit size decreases with the upper bound. This means that trials with larger, more significant local site evaluation results would obtain the most savings in terms of needing a much smaller audit size (most are below 50%). This general relationship between the mean audit size and the upper CI bound holds true across tumor types (results not shown). Figure 2 assesses method B by looking into the relationship between the HR ratio of BICR versus local site evaluation and the differential discordance for the EDR or LDR. Note that a HR ratio greater than 1 implies an overestimate of the treatment effect by the local site evaluation. Recall that EDR is the frequency that the local site evaluation declares progression earlier than BICR. As explained previously, a negative differential discordance for EDR is suggestive of bias in the local site evaluation result favoring the experimental arm. In support of this rationale, we see that the differential discordance for EDR decreases as the HR ratio increases. This means that, as more local site evaluation progressions are being called earlier than BICR on the control arm, the difference in BICR and local site evaluation HR estimates also increases. The reverse relationship is seen for the LDR because LDR is the complement of EDR. This general relationship between the HR ratio of BICR versus local site evaluation and the differential discordance for EDR/ LDR holds true across tumor types (results not shown). Table 5 summarizes measures from both methods by categorizing the trials with respect to their local site evaluation HR estimate. Of the 12 trials with a large observed local site evaluation-assessed PFS treatment effect (HR 0.5), the median across studies of the mean audit sizes over the 10,000 replicates (in short, the median mean audit size) from method A was 35% and the local site evaluation was confirmed in all 10,000 replicate audits for all 12 trials (i.e., consistency of the treatment effect was concluded for all replicates). This indicates very little (if any) loss in power from the 2-stage procedure of method A. Of the same 12 trials, the mean across studies of the proportion of times a complete-case audit was recommended over the 10,000 replicates from method B was 43% and 37% for threshold values of and 0.100, respectively. A complete-case 2640 Clin Cancer Res; 19(10) May 15, 2013 Clinical Cancer Research

5 Evaluation of Audit Methodologies Table 4. Evaluation results for method A and method B Method A Method B Study N Tumor type LE HR (95% CI) CC BICR HR (95% CI) HR ratio (BICR/LE) %CC Mean audit % replicate udits % CC audits % CC audits audits a size a confirming LE b (0.100) c (0.075) d MBC 0.79 (0.68, 0.91) 0.74 (0.64, 0.87) % 73% 100% 14.8% 23.2% MBC 0.49 (0.40, 0.59) 0.54 (0.44, 0.67) % 29% 100% 48.4% 57.7% 3-HER2-952 MBC 0.89 (0.76, 1.04) 0.96 (0.78, 1.20) % 100% % 37.1% 3-HER2þ 219 MBC 0.72 (0.54, 0.97) 0.67 (0.45, 0.99) % 100% % 39.6% 4-Anth 622 MBC 0.66 (0.54, 0.81) 0.79 (0.63, 1.00) % 86% 36% 66.0% 75.3% 4-Cap 615 MBC 0.67 (0.56, 0.82) 0.70 (0.56, 0.87) % 55% 100% 29.6% 38.7% MBC 0.81 (0.68, 0.95) 0.86 (0.72, 1.04) % 100% 0% 48.0% 58.0% MBC 0.44 (0.36, 0.55) 0.35 (0.27, 0.46) % 30% 100% 4.9% 8.0% RCC 0.44 (0.35, 0.54) 0.45 (0.37, 0.56) % 30% 100% 30.8% 38.9% RCC 0.41 (0.33, 0.52) 0.41 (0.31, 0.53) % 35% 100% 23.4% 30.2% 9 25mg 416 RCC 0.70 (0.58, 0.86) 0.68 (0.55, 0.85) % 61% 100% 15.6% 25.6% 9 15mg 417 RCC 0.75 (0.61, 0.92) 0.76 (0.62, 0.94) % 100% 100% 3.7% 7.6% RCC 0.33 (0.26, 0.42) 0.33 (0.26, 0.43) % 35% 100% 17.4% 24.1% RCC 0.62 (0.51, 0.75) 0.59 (0.47, 0.74) % 41% 100% 19.4% 27.3% RCC 0.43 (0.34, 0.54) 0.41 (0.32, 0.54) % 35% 100% 19.8% 29.0% RCC 0.68 (0.56, 0.82) 0.68 (0.56, 0.83) % 49% 100% 24.5% 33.7% MCRC 0.39 (0.32, 0.48) 0.55 (0.45, 0.67) % 29% 100% 98.2% 99.0% 15-Oxal 812 MCRC 1.35 (1.09, 1.67) 1.38 (1.08, 1.77) % 100% 0% 45.5% 54.7% 16-WT 656 MCRC 0.81 (0.67, 0.98) 0.80 (0.66, 0.98) % 100% 100% 25.6% 34.0% 16-Mu 527 MCRC 1.15 (0.95, 1.40) 1.22 (1.00, 1.50) % 100% 0% 16.4% 26.8% 17-WT 597 MCRC 0.71 (0.58, 0.87) 0.75 (0.62, 0.92) % 68% 100% 24.5% 33.4% 17-Mu 589 MCRC 0.82 (0.67, 0.99) 0.90 (0.74, 1.10) % 100% 0% 64.3% 74.6% NSCLC 0.50 (0.41, 0.60) 0.63 (0.52, 0.76) % 46% 100% 53.4% 62.2% NSCLC 0.71 (0.61, 0.82) 0.71 (0.60, 0.83) % 46% 100% 17.4% 24.8% PNET 0.42 (0.26, 0.66) 0.31 (0.18, 0.54) % 55% 100% 0.0% 0.0% PNET 0.38 (0.29, 0.48) 0.40 (0.30, 0.54) % 40% 100% 77.4% 84.7% STS 0.72 (0.61, 0.85) 0.76 (0.64, 0.90) % 48% 100% 41.4% 51.7% STS 0.35 (0.28, 0.45) 0.31 (0.24, 0.41) % 35% 100% 10.4% 16.0% GIST 0.29 (0.20, 0.40) 0.32 (0.23, 0.45) % 40% 100% 56.5% 65.5% Ovarian 0.69 (0.58, 0.82) 0.79 (0.65, 0.96) % 80% 100% 55.5% 66.0% Carcinoid 0.78 (0.62, 0.98) 0.93 (0.71, 1.22) % 100% 0% 22.1% 32.7% Abbreviations: LE, local site evaluation; MBC, metastatic breast cancer; Anth, anthracycline; Cap, capecitabine; CC, complete-case; DD, differential discordance; GIST, gastrointestinal stromal tumor; MCRC, metastatic colorectal cancer; Mu, mutant; NSCLC, non small cell lung cancer; Oxal, oxaliplatin; PNET, pancreatic neuroendocrine tumors; RCC, renal cell carcinoma; STS, soft tissue sarcoma; WT, wild type. a Over the 10,000 replicates per study. b Percentage of 10,000 audit replicates (whether partial or CC) per study in which consistency of the PFS treatment effect is concluded (i.e., the local site evaluation result is confirmed). c Percentage of 10,000 replicate audits per study for which CC audit is recommended (i.e., DD in EDR < or DD in LDR > 0.100). d Percentage of 10,000 replicate audits for which CC audit is recommended (i.e., DD in EDR < or DD in LDR > 0.075). Clin Cancer Res; 19(10) May 15,

6 CCR FOCUS Table 5. Method A versus method B Method A Method B All replicate audits Mean of % CC audit recommended d Local site Median mean audit confirmed local evaluation HR n a size b (min, max) site evaluation c Threshold ¼ Threshold ¼ % (29%, 55%) 12 (100%) 37% (0%, 98%) 43% (0%, 99%) % (41%, 100%) 10 (91%) 29% (4%, 66%) 39% (8%, 75%) > % (73%, 100%) 2 (25%) 33% (15%, 64%) 43% (23%, 75%) a n ¼ number of studies. b Median across studies of the mean over 10,000 replicates for each study. c Number of studies for which all 10,000 replicate audits concluded consistency of the PFS treatment effect (i.e., confirmed the local site evaluation result). d Mean across studies of % complete-case (CC) audit is recommended of 10,000 replicate audits; a CC audit is recommended for a study if the differential discordance (DD) in EDR is less than the negative of the threshold value or the DD in LDR is greater than the threshold value. audit is recommended for a study if the differential discordance in EDR is less than the negative of the threshold value or the differential discordance in LDR is greater than the threshold value. For smaller treatment effects (HR > 0.75), only 2 of the 8 trials resulted in all 10,000 replicate audits confirming the local site evaluation for method A and the median mean audit size was 100%. It should be noted that, for most trials, either all 10,000 replicate audits confirmed the local site evaluation or none did. Whereas there is an intuitive trend in decreased savings (larger median mean audit sizes) using method A as the observed local site Mean audit size (%) r = (0.617, 0.898) % CI upper bound of LE HR 2013 American Association for Cancer Research Figure 1. Relationship between mean audit size from method A and CI upper bound of local site evaluation (LE) HR. evaluation treatment effect becomes smaller, no such trend was seen for method B as the mean proportion of times a complete-case audit is recommended stayed fairly constant across the HR categories. Case Studies Carcinoid study One trial with evaluation bias present was the carcinoid trial (study 26 in Table 4), which was discussed by the Oncologic Drug Advisory Committee (ODAC) in April of 2011 (6). This was a phase III, randomized (1:1), placebo-controlled study of everolimus for the treatment of patients with unresectable or metastatic carcinoid tumor. The primary endpoint was PFS by BICR. At the second interim analysis, an unprecedented discordance of the PFS treatment effect was observed between the local site evaluation and BICR. The local site evaluation PFS result (HR ¼ 0.78; P ¼ 0.003) crossed the efficacy boundary of P ¼ 0.010, whereas the BICR PFS result (HR ¼ 0.93, P ¼ 0.233) crossed the futility boundary of P ¼ The boundary P values are the significance levels to conclude either efficacy or futility, while preserving the type I and II errors, respectively. Clearly, some bias was present in this trial. The HR ratio was Given that the local site evaluation and BICR gave such divergent views of efficacy, it was of particular interest how the 2 audit methods would perform for this study. Table 6 summarizes the discordance between arms with respect to censoring status, progression time, and censoring time. We see some discrepancies between the 2 arms. For method A, 100% of the 10,000 replicates resulted in complete-case audits with 0% being able to verify the consistency of the treatment effect. For method B, however, only 32.7% and 22.1% of the 10,000 replicates recommended a complete-case audit for threshold values of and 0.100, respectively, to 2642 Clin Cancer Res; 19(10) May 15, 2013 Clinical Cancer Research

7 Evaluation of Audit Methodologies Figure 2. Relationship between HR ratio [BICR vs. local site evaluation (LE)] and differential discordance in EDR/LDR from method B. HR ratio (BICR/LE) Early discrepancy rate (EDR) r = ( 0.821, 0.396) 1.6 r = (0.669, 0.914) Reference lines: HR ratio = 1 Threshold = Threshold = 0.1 HR ratio (BICR/LE) Late discrepancy rate (LDR) Reference lines: HR ratio = 1 Threshold = Threshold = Mean differential discordance in EDR Mean differential discordance in LDR 2013 American Association for Cancer Research support the conclusion that bias may be present. Note that the fixed audit size of 160 was 37% of the total number of patients. Soft tissue sarcoma study To illustrate the potential savings in audit size, another case study is presented (study 22 in Table 4), which was discussed by the ODAC in March of 2012 (7). This was a phase III, randomized (1:1), placebocontrolled, maintenance trial of ridaforolimus in 711 patients with soft tissue sarcoma (STS). The final local site evaluation and BICR PFS HR estimates were 0.72 and 0.76, respectively, with a HR ratio of Table 6 summarizes the discordance between arms for this study, which seems fairly balanced. For method A, only 28% of the 10,000 replicates resulted in complete-case audits with 100% verifying consistency of the treatment effect. The mean audit size was 48%, a savings of over 50% in audit size. For method B, the fixed audit size of 160 was 23% of the total number of patients. Of the 10,000 replicates, 51.7% and 41.4% recommended a complete-case audit for threshold values of and 0.100, respectively, to support the conclusion that bias may be present. This was largely due to increased LDR discrepancies, as 2.0% and 41.4% of the 10,000 replicates resulted in differential discordance values for EDR and LDR, respectively, exceeded a threshold of Thus, there is quite a bit of variability in the differential discordance for LDR making it difficult to determine whether or not bias was present in this study using method B. Discussion Although measurement error is inherent in the reading of radiographic scans, regulatory considerations for drug approval are based on the relative treatment effect at the population level. Given that multiple publications (2 4) have corroborated the strong correlation between local site evaluation and BICR-assessed PFS treatment effect estimates, there is a need for the exploration of alternative strategies to detect bias in the local site evaluation. The results of the analyses presented herein support that a random sample-based BICR audit is a viable alternative to a complete-case BICR and may be a Table 6. Discordance in case studies Case study 1 Case study 2 Discordance Placebo (n ¼ 213) Everolimus (n ¼ 216) Placebo (n ¼ 364) Ridaforolimus (n ¼ 347) Censoring status 38% 26% 14% 16% PD time a 15% 18% 21% 27% Censoring time a 1.4% 3.7% 0.8% 2.0% a PD/censoring discordant outside of 7-day window. Clin Cancer Res; 19(10) May 15,

8 CCR FOCUS more efficient strategy for bias evaluation of the local site evaluation. Note that, although there is general agreement, in one study reported here (study 26 in Table 4), there were divergent conclusions depending on the choice of local site evaluation and BICR, justifying a continued need for BICR to provide assurance about the local site evaluation based treatment effect. Method A seems to perform well in most situations; that is, it seems able to distinguish between trials with and without bias present. The savings with respect to audit size varies from case to case, depending on the study size and magnitude of the observed local site evaluation PFS treatment effect; larger effect sizes and/ or studies typically require smaller audit proportions. Method B is intuitively appealing, but needs further evaluation, particularly with respect to determination of the appropriate threshold value. Potential reasons for its variable performance that need further evaluation include loss of important information due to dichotomization and ignoring patients who were censored by both the local site evaluation and BICR (cell "d" in Table 1) in the definition of EDR and LDR. Method B counts discordances but not how far apart they are. For example, with the LDR, the local site evaluation could call progressive disease right after BICR or a long time after. If the late discrepancies occurred in the control arm many visits after BICR, this would produce more significant bias in the HR estimate. On a similar note, we could have a relatively small number of late discrepancies, but if they occur very late, then this would produce greater bias in the HR estimate. Although real-time BICRs are becoming more prominent with advances in modern digital technology, all the studies included in the analyses presented herein did not have realtime BICRs. It should be noted that real-time BICR would ameliorate bias concerns due to informative censoring, but does not alleviate potential bias due to the subjectivity inherent to the PFS endpoint. We acknowledge that the studies included in this analysis are limited to registration trials that all had complete-case BICR and may not be representative of the population of all clinical trials conducted. However, they are representative of the clinical trials that are generally submitted to the FDA for regulatory consideration and for which BICRs are usually required. An ODAC meeting was held in July 2012 to discuss whether the current practice of complete-case BICR should be replaced by a random sample-based BICR audit, based on the information and analyses presented herein (8). All committee members agreed that a random sample-based BICR audit should be considered; however, the potential merits must be viewed in tandem with the potential limitations and challenges. They also advised against the complete elimination of BICR, which could jeopardize the integrity of the local site evaluation. Although these analyses have showed that a BICR audit to assess potential bias in the local site evaluation is a feasible approach, the logistics of how the audit should proceed need further discussion and consideration. The method of selecting the random sample audit needs to be determined; a simple random sample audit may not be sufficient, for example, to ensure representation of all study sites. Efforts should also be made to minimize any additional burden that the audit may cause the investigator or sponsor without compromising the integrity of the study. Selection of the actual audit strategy to implement within a trial may need to be determined on a caseby-case basis. Although we focused on 2 methodologies, we expect that other approaches will become available for consideration in the future. Disclosure of Potential Conflicts of Interest No potential conflicts of interest were disclosed. Disclaimer This article reflects the views of the authors and should not be construed to represent the FDA s views or policies. Authors' Contributions Conception and design: J.J. Zhang, A.J. Murgo, R. Pazdur, R. Sridhara Development of methodology: J.J. Zhang, R. Sridhara, L.E. Dodd, R. Pazdur Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): J.J. Zhang, L. Zhang, H. Chen, L.E. Dodd, R. Sridhara Writing, review, and/or revision of the manuscript: J.J. Zhang, H. Chen, A.J. Murgo, L.E. Dodd, R. Pazdur, R. Sridhara Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): J.J. Zhang, R. Pazdur, R. Sridhara Study supervision: R. Sridhara Received October 28, 2012; revised February 20, 2013; accepted March 21, 2013; published OnlineFirst March 26, References 1. Ford R, Schwartz L, Dancey J, Dodd LE, Eisenhauer EA, Gwyther S, et al. Lessons learned from independent central review. Eur J Cancer 2009;45: Dodd LE, Korn EL, Freidlin B, Jaffe CC, Rubinstein LV, Dancey J, et al. Blinded independent central review of progression-free survival in phase III clinical trials: important design element or unnecessary expense? J Clin Oncol 2008;26: Amit O, Mannino F, Stone AM, Bushnell W, Denne J, Helterbrand J, et al. Blinded independent central review of progression in cancer clinical trials: results from a meta-analysis. Eur J Cancer 2011;47: Zhang JJ, Chen H, He K, Tang S, Justice R, Keegan P, et al. Evaluation of blinded independent central review of tumor progression in oncology clinical trials: a meta-analysis. Drug Inf J 2013;47: Dodd LE, Korn EL, Freidlin B, Gray R, Bhattacharya S. An audit strategy for progression-free survival. Biometrics 2011;67: Clin Cancer Res; 19(10) May 15, 2013 Clinical Cancer Research

9 Evaluation of Audit Methodologies 6. Oncologic Drug Advisory Committee everolimus for carcinoid tumors. [cited 2011 Apr 12]. Available from: CommitteesMeetingMaterials/Drugs/OncologicDrugsAdvisoryCommittee/ ucm htm 7. Oncologic Drug Advisory Committee ridaforolimus for STS. [cited 2012 Mar 20]. Available from: CommitteesMeetingMaterials/Drugs/OncologicDrugsAdvisory Committee/ucm htm 8. Oncologic Drug Advisory Committee evaluation of radiologic review of PFS in non-hematologic malignancies. [cited 2012 Jul 24]. Available from: Materials/Drugs/OncologicDrugsAdvisoryCommittee/ucm htm Clin Cancer Res; 19(10) May 15,

10 Assessment of Audit Methodologies for Bias Evaluation of Tumor Progression in Oncology Clinical Trials Jenny J. Zhang, Lijun Zhang, Huanyu Chen, et al. Clin Cancer Res 2013;19: Published OnlineFirst March 26, Updated version Access the most recent version of this article at: doi: / ccr Cited articles Citing articles This article cites 5 articles, 1 of which you can access for free at: This article has been cited by 4 HighWire-hosted articles. Access the articles at: alerts Sign up to receive free -alerts related to this article or journal. Reprints and Subscriptions Permissions To order reprints of this article or to subscribe to the journal, contact the AACR Publications Department at pubs@aacr.org. To request permission to re-use all or part of this article, use this link Click on "Request Permissions" which will take you to the Copyright Clearance Center's (CCC) Rightslink site.