University of Groningen

Transcription

1 University of Groningen ESMO-Magnitude of Clinical Benefit Scale version 1.1 Cherny, N. I.; Dafni, U.; Bogaerts, J.; Latino, N. J.; Pentheroudakis, G.; Douillard, J. -Y.; Tabernero, J.; Zielinski, C.; Piccart, M. J.; de Vries, E. G. E. Published in: Annals of Oncology DOI: /annonc/mdx310 IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below. Document Version Publisher's PDF, also known as Version of record Publication date: 2017 Link to publication in University of Groningen/UMCG research database Citation for published version (APA): Cherny, N. I., Dafni, U., Bogaerts, J., Latino, N. J., Pentheroudakis, G., Douillard, J. -Y.,... de Vries, E. G. E. (2017). ESMO-Magnitude of Clinical Benefit Scale version 1.1. Annals of Oncology, 28(10), DOI: /annonc/mdx310 Copyright Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons). Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Downloaded from the University of Groningen/UMCG research database (Pure): For technical reasons the number of authors shown on this cover page is limited to 10 maximum. Download date:

2 Annals of Oncology 28: , 2017 doi: /annonc/mdx310 Published online 4 September 2017 SPECIAL ARTICLE ESMO-Magnitude of Clinical Benefit Scale version 1.1 N. I. Cherny 1 *, U. Dafni 2, J. Bogaerts 3, N. J. Latino 4, G. Pentheroudakis 5, J.-Y. Douillard 4, J. Tabernero 6, C. Zielinski 7, M. J. Piccart 8 & E. G. E. de Vries 9 1 Cancer Pain and Palliative Medicine Service, Department of Medical Oncology, Shaare Zedek Medical Center, Jerusalem, Israel; 2 Laboratory of Biostatistics, School of Health Sciences, National and Kapodistrian University of Athens & Frontier Science Foundation-Hellas, Athens, Greece; 3 Methodology Direction, EORTC Headquarters, Brussels, Belgium; 4 ESMO Head Office, Lugano, Switzerland; 5 Medical Oncology Department, Ioannina University Hospital, Ioannina, Greece; 6 Medical Oncology Department, Vall d Hebron University Hospital, Barcelona, Spain; 7 Division of Oncology, Medical University Vienna, Vienna, Austria; 8 Jules Bordet Institute, Universite Libre de Bruxelles, Brussels, Belgium; 9 Department of Medical Oncology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands *Correspondence to: Prof. Nathan I. Cherny, Norman Levan Chair of Humanistic Medicine, Medicine (BGU), Cancer Pain and Palliative Medicine Service, Department of Medical Oncology, Shaare Zedek Medical Center, Jerusalem 91031, Israel. Tel: þ ; mcbs@esmo.org Background: The ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) version 1.0 (v1.0) was published in May 2015 and was the first version of a validated and reproducible tool to assess the magnitude of clinical benefit from new cancer therapies. The ESMO-MCBS was designed to be a dynamic tool with planned revisions and updates based upon recognition of expanding needs and shortcomings identified since the last review. Methods: The revision process for the ESMO-MCBS incorporates a nine-step process: Careful review of critiques and suggestions, and identification of problems in the application of v1.0; Identification of shortcomings for revision in the upcoming version; Proposal and evaluation of solutions to address identified shortcomings; Field testing of solutions; Preparation of a near-final revised version for peer review for reasonableness by members of the ESMO Faculty and Guidelines Committee; Amendments based on peer review for reasonableness; Near-final review by members of the ESMO-MCBS Working Group and the ESMO Executive Board; Final amendments; Final review and approval by members of the ESMO-MCBS Working Group and the ESMO Executive Board. Results: Twelve issues for revision or amendment were proposed for consideration; proposed amendments were formulated for eight identified shortcomings. The proposed amendments are classified as either structural, technical, immunotherapy triggered or nuanced. All amendments were field tested in a wide range of studies comparing scores generated with ESMO- MCBS v1.0 and version 1.1 (v1.1). Conclusions: ESMO-MCBS v1.1 incorporates 10 revisions and will allow for scoring of single-arm studies. Scoring remains very stable; revisions in v1.1 alter the scores of only 12 out of 118 comparative studies and facilitate scoring for single-arm studies. Key words: magnitude of clinical benefit scale, biostatistics, single arm studies, European Society for Medical Oncology Introduction The ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) v1.0 was published in May 2015 [1] and was the first version of a validated and reproducible tool to assess the magnitude of the clinical benefit from new and effective cancer therapies. The tool uses a rational, structured and consistent approach to derive a relative ranking of the magnitude of clinically meaningful benefit that can be expected from a new anti-cancer treatment. The ESMO-MCBS was designed to be a dynamic tool with planned revisions and updates based upon recognition of expanding needs and shortcomings identified since the last review. For instance, while there is a recent trend for more medication to be approved on the basis of single-arm studies, the first version only scored comparative studies. Therefore, there was a manifest imperative to formulate and to incorporate an approach to grade the clinical benefit data derived from the registration trials of medications approved on the basis of these single-arm studies. Additional impetus for revision was derived from experience accrued in evaluating a further large cohort of studies published between 2014 and 2016; feedback and queries from clinicians, patients, researchers and representatives of the pharmaceutical industry; and a dynamic process of internal peer review. VC The Author Published by Oxford University Press on behalf of the European Society for Medical Oncology. All rights reserved. For Permissions, please journals.permissions@oup.com.

3 Annals of Oncology Commitment to accountability for reasonableness, which represents the ethical gold-standard for a fair priority setting process in public policy [2, 3], demands a process for challenges and appeals and requires that the deliberative process be transparent. In accordance with this standard, the revised v1.1 of the ESMO-MCBS is presented with point-by-point deliberative explanation, including the shortcomings identified in v1.0, description of the amendment in v1.1, and explanation of the rationale including relevant index cases. Methodology The revision process for the ESMO-MCBS incorporates a ninestep process: 1. Careful review of critiques and suggestions, and identification of problems in the application of v1.0; 2. Identification of shortcomings for revision in the upcoming version; 3. Proposal and evaluation of solutions to address identified shortcomings; 4. Field testing of solutions; 5. Preparation of a near-final revised version for peer review for reasonableness by members of the ESMO Faculty and Guidelines Committee; 6. Amendments based on peer review for reasonableness; 7. Nearfinal review by members of the ESMO-MCBS Working Group and the ESMO Executive Board; 8. Final amendments; 9. Final review and approval by members of the ESMO-MCBS Working Group and the ESMO Executive Board. Twelve issues for revision or amendment were proposed for consideration (Table 1). Of these, one, the use of the lower limit of the 95% confidence interval (CI) rather than the point estimate, had been reviewed extensively by the Working Group and subjected to extensive statistical modelling (to be published in full in ESMO Open) and a decision was made not to subject this issue to revision. Three issues have been deferred for further consultation with patient representative organisations. These include toxicity penalties, credit for improvements in quality of life (QoL) or deferred deterioration in QoL and credits for single symptom improvements. Proposed amendments were formulated for the remaining eight identified shortcomings. The proposed amendments are classified as either structural, technical, immunotherapy triggered or nuanced. Structural amendments (n ¼ 2) involve either the addition of a new form or a restructuring of an existing form with the addition of a new prognostic stratification. Technical amendments (n ¼ 1) involve changes to the instructions regarding the application of the ESMO-MCBS. Immunotherapy triggered amendments (n ¼ 3) refer to amendments triggered by evolving issues derived from the expanding experience in evaluating new immunotherapy-based studies. Nuanced amendments (n ¼ 4) correct shortcomings identified in the scoring of existing forms. The amendments for each of the revised sections of the ESMO-MCBS: Introduction, form 1, forms 2a and 2b, and the new form 3 are detailed in this manuscript. Field testing, comparing scores generated on v1.0 and v1.1, was undertaken in 118 comparative studies (supplementary Tables S1 S15, available at Annals of Oncology online). A pre-final version and the field testing results derived from the proposed amendments were peer reviewed by members of the ESMO Faculty and Guidelines Committee for reasonableness. Based on feedback from this peer review process, amendments were further Table 1. Issues generating planned amendments in V1.1 ESMO-MCBS 1. Use of lower limit of 95% CI versus point estimate 2. Crediting 2- and 3-year survival advantage <10% 3. Unscorable randomised studies because of scale deficiencies 4. No mechanism to credit plateauing in PFS curve (when median PFS gain is small) 5. Toxicity penalties: differential or not, which adverse effects 6. Validity of QoL credit 7. Crediting of single symptom improvements 8. Inability to score FDA and EMA approved neoadjuvant therapies based on pcr 9. PFS studies with early termination (based on planned interim survival analysis) 10. Inability to score medications approved based on single-arm studies 11. No recognition that some patients treated with non-curable intent, may be cured 12. PFS penalty for no improvement in OS and QoL did not explicitly refer to mature survival data adjusted and revised. The final amendments resulted in a change in score for 12 studies: 4 comparative studies having scores upgraded by 1 point and 3 achieving scores in situations where they were not previously scorable (Table 2), and 5 having scores downgraded (Table 3); along with new scoring of 10 single-arm studies (Table 4). The final version (v1.1) is presented in Figure 1 and has been approved by the ESMO-MCBS Working Group and the ESMO Executive Board. INSTRUCTIONS: SHORTCOMINGS IN v1.0 AND AMENDMENT IN v1.1. Shortcoming: Instructions regarding subgroup analyses were not adequately clear. Amendment: Instructions regarding subgroup analyses were redrafted to more clearly reflect the published eligibility guidelines for the application of the ESMO-MCBS [1]. The rules for subgroup analyses are more clearly specified in the instructions as following: Studies with pre-planned subgroup analyses with a maximum of three subgroups can be graded (provided there is adjustment for multiple comparisons). When statistically significant results are reported for any subgroup, then each of these should be graded separately. Subgroups not showing statistically significant results are not graded. Except for studies that incorporate collection of tissue samples to enable re-stratification based on new genetic or other biomarkers, findings from un-planned (post hoc) subgroup analysis cannot be graded and they can only be used as foundation for hypothesis generation. Rationale: Point of clarification. Type: Technical. Special article Volume 28 Issue doi: /annonc/mdx

4 Special article Annals of Oncology Table 2. Randomized studies which were previously unscorable or whose scores are upgraded m ESMO-MCBS v1.1 Drug Trial name Disease Setting Primary outcome Melanoma 1st line metastatic OS (crossover allowed) 5-year survival 8.8% 9.40% 2-year 4 A/4 [15, 16] gain 11% PFS control PFS gain PFS HR OS control OS gain OS HR pcr QoL Toxicity ESM0- MCBS V1.0 ESM0- Ref. MCBS V months 2.1 months 0.33 ( ) KEYNOTE- Lung 2nd line after platinum OS All >1% 8.5 months (all) 1.9 months (2) 0.71 ( ) Reduced No score 3 [17] 010 based therapy OR TKI (for EGFR/ALK mutated) advanced NSCLC >1% tumour cell PD-L1 Ovarian cancer BRCA ovarian in remission NeoSphere Breast Neo-adjuvant HER2 overexpressed invasive ductal breast Basal Cell Superficial basal cell carcinoma in adults with normal immune systems when surgical methods are less appropriate Melanoma 1st line advanced or metastatic melanoma Dacarbazine6 ipilimumab Pembrolizumab 2 or 10 mg/kg versus docetaxel Olaparib versus placebo Trastuzumab and docetaxel6 pertuzumab Imiquimod 5% cream (5 or 7 days) versus placebo Combination versus nivolumab versus ipilimumab Pembrolizumab (2 versus 10 mg/kg) versus investigatorchoice KEYNOTE- 002 Melanoma 2nd line metastatic after failure of ipilimumab BRAF or MEK inhibitor 4.2 months (10) 0.61 ( ) grade 3/4 5 5 adverse PD-L1 >50% 8.2 months 6.7 months (2) 0.54 ( ) events months (10) 0.50 ( ) 5 5 PFS 4.3 months 6.9 months 018 ( ) 27.8 months 2 months 0.73 ( ) No benefit 2 a 3 a [25 27] pcr NS 29% versus No score C a [7, 33] 46% pcr (histologic clearance rates) PFS All 2.9 months Combination 8.6 months Nivolumab 4 months PD-L1þ: 3.9 months PD-L1 : 2.8 months Combination 10.1 months Nivolumab 10.1 months Combination 8.4 months Nivolumab 2.3 months PFS 2.6 months 1.6 months (2 mg/kg) 2 months (10 mg/kg) 0.42 ( ) Plateau with >10% gain at 1 year 3% versus No score C [34] 79% (5 days) 82% (7 days) Gr 3þ 55% 2 3 [23] versus 27% 0.57 ( ) Gr 3þ 16% 4 4 versus 29% Plateau with 2 3 >10% gain at 1 year 045 ( ) 3 a 4 a [24] Plateau PFS gain >10% gain at 1 year 039 ( ) 3 a 4 a Plateau with >10% gain at 1 year a Randomised phase II study Cherny et al. Volume 28 Issue

5 Annals of Oncology Table 3. Randomised studies whose scores are downgraded in ESMO-MCBS v1.1 PFS control PFS gain PFS HR OS control OS gain OS HR pcr QoL Toxicity ESM0- MCBS V1.0 ESM0- Ref MCBS V1.1 Disease Setting Primary outcome Drug Trial name 25.1 months 5.8 months 0.68 ( ) Delayed 5 4 [8, 9] deterioration [35, 36] PFS and OS 6.4 months 3.2 months 0.65 ( ) (All) 17.4 months 2.4 months 0.87 ( ) ( ) NS Minor 1 1 (high risk) deterioration PFS stratified for stage and risk of progression 10.5months 5.5 months 0.73( months 7.8 months 0.64 ( ) ) OS 11.7 months 1.6 months, 2-year 0.84 ( ) 3 1 [19] survival gain 5% 10% 1.8 months 0.72 ( ) 3 2 [20] 2-year survival gain 5% OS 6.7 months 2-year survival 4% OS 9.1 months 1.4 months 0.86 ( ) 2 1 [18] 2-year survival gain 3 5% EMILIA Breast 2nd line metastatic after trastuzumab failure ICON7 Ovarian High-risk, early stage post-resection or advanced ovarian or primary peritoneal RAISE Colorectal 2nd line metastatic after bevacizumab, oxaliplatin, fluoropyrimidine Pancreatic 1st line advanced or metastatic. Good performance status (KPS>70%) REVEL Lung 2nd line after platinum based therapy NSCLC T-DM1 versus lapatinibþ capecitabine Paclitaxel and carboplatin (5 or 6 cycles)6 bevacizumab till 18 cycles or progression FOLFIRI6 ramucirumab Gemcitabine6 nab-paclitaxel Docetaxel6 ramucirumab Index case: The SQUIRE study of cisplatin and gemcitabine with or without the addition of necitumumab in first-line treatment of patients with metastatic squamous non-small-cell lung cancer included a planned subgroup analysis for tumours expressing EGFR H-score less than or greater than 200 which did not demonstrate any predictive significance [4]. In a subsequent publication, the authors published results of an unplanned exploratory re-analysis of the data re-stratifying the cohorts to EGFRexpressing or non-egfr-expressing [5]. In the EGFR-expressing subgroup an OS advantage was reported: HR 0.79 ( ) absolute gain in median survival 1.7 months and with a 2-year survival gain of 3% 5%. An appeal was submitted by the study sponsor to have the unplanned exploratory re-analysis accredited with ESMO-MCBS score 3. The ESMO-MCBS Working Group evaluated the appeal and also noted the authors conclusions these data should be interpreted in the context of the SQUIRE ITT results and caution should be exercised, especially when it comes to the outcomes of smaller subgroups... Further analyses of the data are needed in order to better understand the potential predictive role of EGFR copy number gain in this setting. The appeal to score the unplanned exploratory analysis was declined. FORM 1: SHORTCOMINGS IN v1.0 AND AMENDMENT IN v1.1. Shortcoming: Version 1.0 had no framework to score adjuvant therapies approved on the basis of pathological complete remission (pcr). Amendment: New criteria for grade C have been inserted Improvements in pcr (pathological complete remission) alone (primary end point) by 30% relative gain AND 15% absolute gain in studies without mature survival data. Type: Nuanced. Special article Rationale: Pertuzumab was approved by both the European Medicines Agency (EMA) and US Food and Drug Administration (FDA) in addition to chemotherapy and trastuzumab in the neoadjuvant treatment of HER2 overexpressed invasive ductal breast cancer, on the basis of substantial improvement in the pcr rate of patients with HER2 overexpressing invasive ductal breast cancer. The approval by the FDA is an accelerated (provisional) approval. There continues to be substantial controversy regarding the validity of pcr as a surrogate for overall survival (OS) in neoadjuvant studies in solid tumours. It is graded at a level C because of this on-going controversy [6]. Index case: In the randomised phase II NeoSphere trial of neoadjuvant therapy with trastuzumab and docetaxel 6pertuzumab for HER2 overexpressed invasive ductal breast cancer, there was a 17% increase in the pcr rate (46% versus 29%) [7]. Based on this, the FDA granted accelerated (provisional) approval for the addition of pertuzumab to chemotherapy and trastuzumab in the neoadjuvant setting and the EMA approved the indication with a provision for additional monitoring. ESMO-MCBS v1.0 was Volume 28 Issue doi: /annonc/mdx

6 Special article Annals of Oncology Table 4. Single-arm studies scored using form 3 in ESMO-MCBS v1.1 Drug Disease Setting Stratification ORR CR DoR PFS Survival QoL Toxicity ESMO- MCBS 1.0 ESM0- MCBS 1.1 Ref. AZD9291 (Osimertib) Lung EGFR Inhibitor Resistant NSCLC phase 2, RR, stratified for EGFR T790M mutation Ceritinib Lung ALK mutated NSCLC stratified for prior/noprior crizotinib Crizotinib Lung 2nd line metastatic NSCLC whose tumours are ROS1 positive. Crizotinib Lung NSCLC whose tumours are ALK mutated Nivolumab Hodgkin s Classical Hodgkin lymphoma that has relapsed or progressed after autologous haematological stem cell transplantation and post-transplantation brentuximab vedotin Olaparib Ovarian After three lines therapy BRCA mutated (germline OR somatic) Atezolizumab Urothelial Platinum resistant locally advanced or metastatic transitional cell cancer stratified by PD-L1 immunohistochemistry score (IC 2/3) Everolimus Brain Subependymal giant-cell Pembrolizumab Head and Neck astrocytomas in tuberous sclerosis PD-L1-positive squamous cell HþN carcinoma after 1st line Pembrolizumab Colorectal 2nd lineþmetastatic colorectal cancer (MMRd) (MSI-high (microsatellite instability) who have received prior therapy MUT 61% 6þ months 9.6 months No score 3 [37] WT 21% 2.8 months No score 1 Prior crizotinib No prior crizotinib RR: tumour volume (vol.) 56% 8.2þ months 6.9 months No score 3 [38] 62% 10.2 months No score 3 72% 7% 17.6 months 19.2 months No score 3 [39] 61.8% 10 months 9.7 months No score 3 [40, 41] 87% 17% 8.7 months 24þ months No score 3 [42] 31% 3% 7 months 7 months 16 months No score 3 [43] 26% 11% 11.7 months No score 3 [44, 45] 100% (decrease volume >30%) 24þ months Improved No score 4 [46] 16% 5% 6þ months No score 1 [47] 62% 5.9þ months No score 3 [48] Lynch 27% Somatic 100% 2344 Cherny et al. Volume 28 Issue

7 Annals of Oncology not able to score this trial; in v1.1 it scores grade C (Improvements in pcr alone, as primary end point, by 30% relative AND 15% absolute gain in studies without mature survival data). FORM 2a: SHORTCOMINGS IN v1.0 AND AMENDMENT IN v1.1. Shortcoming: The thresholds for absolute gain in median survival in the subsection for patients with a median OS >12 months, provided inadequate constraints in the situation of clinical studies where the median OS was very long. Amendment: The prognostic stratification for form 2a has been revised, v1.1 incorporates a three-level prognostic stratification: 12 months, >12 to 24 months, and >24 months. The >24- month stratification is introduced to achieve maximal score if either: HR 0.70 AND Gain 9 months or increase in 7-year survival of >10%. Type: Structural. Rationale: For grade assignment in the ESMO-MCBS, the highest grade is assigned to successful trials providing a meaningful relative benefit, taking into account the variability of the estimate coupled with a minimum relevant observed absolute gain (quantitative component). It is thus guided by a dual rule based on the comparison to pre-specified threshold values of the observed relative benefit and the absolute benefit achieved by the therapy. Statistical simulation studies on the scoring criteria for v1.0 identified that for the subgroup of studies in which the median survival for the control arm was very long (>2 years), the absolute gain threshold restricted to 5 months did not apply adequate constraint on scores credited on the basis of HR This was identified as an important deficiency requiring amendment. Index cases: In the EMILIA study of T-DM1 versus the combination therapy of lapatinib and capecitabine, the control arm had an OS of 25.1 months and the survival gain was 5.8 months [8, 9], thus generating the same preliminary score of 4 as the CLEOPATRA study of paclitaxel and trastuzumab with or without the addition or pertuzumab in which the control arm OS was 40.8 months and the gain was 15.7 months [10 13]. In v1.1, this anomaly is corrected; the preliminary ESMO-MCBS score for the EMILIA study is downgraded from 4 to 3, and the final ESMO-MCBS score derived from the credit for co-primary outcome of PFS and quality of life benefit is 4. Shortcoming: Studies in which there was a long-term plateau in the survival curve (such as in the case of ipilimumab in melanoma) were not credited as having curative potential. Amendment: There is a new adjustment to the preliminary scoring: If there is a long-term plateau in the survival curve, and OS advantage continues to be observed at 5 years (or 7 years for diseases with median survival >24 months), also score according to form 1 (treatments with curative potential) and present both scores, i.e. A/4. Type: Immunotherapy triggered. Rationale: In some diseases exceptional responders may achieve protracted complete remission (CR) with the possibility of longterm disease control and survival. Historically, this has been described in the setting of small-cell-lung cancer [14]. More recently, long-term follow-up data after immunotherapy for metastatic melanoma suggested some patients may achieve long-term disease control with the potential for cure. The potential for cure is indicated by a sustained plateau in the long-term survival data. This amendment recognises that some treatments may have benefits both in a non-curative and curative setting and that the clinical benefit scale ought to credit both. It is important to emphasise that prolonged (5 or 7 year) survival advantage alone, without a plateau in the curve (as has been observed in some instances of breast and prostate cancer), does not meet this criterion. Index case: The study of dacarbazine 6ipilimumab in the treatment of metastatic melanoma initially demonstrated a modest survival gain of 2.1 months over a control of 9.1 months with HR 0.69 ( ) and with 2-year survival gain of 10.6% scoring ESMO-MCBS 4 [15]. Long-term follow-up subsequently demonstrated a plateau on the survival curve after 3 years, with a 9.4% absolute increase in 5-year OS [16] which meets form 1 criteria or grade A. Consequently, the ESMO- MCBS v1.1 score is A/4. Shortcoming: Anomalies in the absolute gain criteria for grade 2 scoring, did not allow scoring for some studies. Amendment: Grade 2 scoring across all prognostic groups (based on median OS of the control arm) has been amended to credit studies achieving optimal HR targets but small absolute gains, as well as studies achieving an intermediate HR which achieve median survival gains above a minimal threshold. Type: Nuanced. Special article Rationale: Statistical modelling of plausible outcome scenarios using v1.0, which incorporated both a lower and upper limit for median survival gain, identified the problem that some studies with intermediate HRs could not be scored because survival gain was outside the defining limits. This problem was also observed in the setting of some low HR scenarios in which the absolute gain in median OS was very modest. The revision corrects these anomalies in the scale function and corrects the lacunae in the scoring continuum. Index case: The KEYNOTE-010 study [17] compared docetaxel to pembrolizumab 2 or 10 mg/kg in advanced non-small-cell lung cancer with PD-L1 expression on at least 1% of tumour cells. The experimental arm of pembrolizumab 2 mg/kg achieved a 1.9- month gain in median survival (10.4 versus 8.5 months), HR 0.71 ( ). In v1.0, there was a grading lacuna that resulted in this scenario being unscorable. With the v1.1 amendment, this scenario is now scorable, and achieves a preliminary score of 2 which is upgraded to a final score of 3 based on reduced toxicity. Volume 28 Issue doi: /annonc/mdx

8 Special article Shortcoming: Grade 1 and grade 2 credit for low magnitude improvements in 2- and 3-year survival, corresponding to gains of <10%, lacked adequate statistical justification. Amendment: Grade 1 and grade 2 credit for low magnitude improvements of <10% absolute gain 2-year survival (for studies with control median OS <12 months) and 3-year survival in studies (for studies with median control OS >12 months) have been deleted. Type: Nuanced. Rationale: Published OS plots typically do not include confidence intervals. When small differences are identified in the tail of a survival curve, when the number of surviving cases is small, the likelihood that confidence intervals would overlap is high and therefore crediting benefit based on such small differences is not statistically justifiable. Index cases: According to ESMO-MCBS v1.0, the REVEL study of docetaxel versus ramucirumab in the second-line treatment of non-small-cell lung cancer [18] with control median OS of 9.1 months and median gain of 1.4 months with HR 0.86 ( ) would have qualified it for an ESMO-MCBS score of 1 on these considerations alone. However, the score was upgraded to 2 on the basis of a 2-year survival gain of 3% 5% that was observed in the tail of the survival curve. In another two instances, studies with relatively small gains in median survival had been upgraded on the basis of a 2-year survival gain of 5% 10% in the tail of the curve: The addition of ramucirumab to FOLFIRI in advanced colorectal cancer [19] was upgraded from 1 to 3 and the superiority of nab-paclitaxel in combination with gemcitabine over gemcitabine alone in advanced and metastatic pancreatic cancer [20] was upgraded from 2 to 3. In v1.1, these upgrades are rescinded because it is highly unlikely such small differences in the tail of the curve are statistically significant. In contrast, the CheckMate 057 study demonstrating superiority of nivolumab over docetaxel as second line therapy after platinum-based therapy for non-squamous non-small-cell lung cancer [21, 22] was upgraded in v1.0 from 3 to 4 on the basis of a >10% increase in 2-year survival. Although this credit may also be statistically dubious, the possibility of it being significant is considered more plausible, and in v1.1 credit continues to be accrued for 2- and 3-year survival gain of this magnitude. FORM 2b: SHORTCOMINGS IN v1.0 AND AMENDMENT IN v1.1. Shortcoming 1: In v1.0 there was no mechanism to credit plateauing in progression-free survival (PFS) curves. This resulted in low grading for treatments with a small gain in median PFS but improved duration of response (DoR) among the patients who did respond. Amendment: The addition of a new adjustment. Upgrade 1 level if there is a long-term plateau in the PFS curve, and there is 10% improvement in PFS at 1 year for control median PFS <6 months or 10% improvement in PFS at 2 years for median PFS >6months. Type: Immunotherapy triggered. Rationale: The rapidly evolving experience with immunotherapy interventions has highlighted the scenario in which very prolonged responses may be observed in a minority of patients. In this scenario, the relatively modest improvement in median PFS may be accompanied by a far more substantial widening of sustained PFS advantage with time, associated with a prolonged DoR. This amendment corrects the anomaly whereby scoring based only on median PFS, without looking at sustained advantage in PFS, understates the magnitude of clinical benefit. Index cases: In the study of combination therapy nivolumab plus ipilimumab versus nivolumab alone versus ipilimumab alone as first-line therapy for patients with locally advanced or metastatic melanoma [23], the combination therapy scores only a grade of 2 according to v1.0. The maximal preliminary score of 3 is achieved on the basis of improved median PFS gain of 7.6 months (control 2.9 months), HR 0.42 ( ). However, this is adjusted down to 2 because of high toxicity. In v1.1, additional credit is given for plateau in PFS curve with >10% gain at 1 year, thus upgrading the score to 3. Similarly, in the Keynote-002 study of pembrolizumab (2 versus 10 mg/kg) versus investigator-choice second-line therapy for metastatic melanoma after failure of ipilimumab, BRAF or MEK inhibitor [24], additional credit is given for plateau in PFS curve with >10% gain at 1 year, raising the score from 3 with v1.0 to 4 with the additional adjustment in v1.1. Shortcoming 2: PFS studies with early crossover because of early stopping or crossover based on detection of survival advantage at interim analysis, and with no mature OS advantage, are not credited with improving OS in v1.0. Amendment: Addition of a new adjustment to Upgrade 1 level if study had early crossover because of early stopping or crossover based on detection of survival advantage at interim analysis. Type: Immunotherapy triggered. Annals of Oncology Rationale: Preliminary PFS data are capped at a maximum of grade3becauseofthelimitationsofpfsasasurrogateforeither improvement in OS or improvement in QoL. ESMO- MCBS v1.0 was designed to allow upgrade for PFS studies in which secondary outcome data demonstrated improved OS. In v1.0, the only mechanism for this upgrade was based on improvement in median OS. This approach penalised studies which incorporated early stopping or crossover based on detection of survival advantage at interim analysis, particularly in situations where the interim analysis was performed before median survival was reached. This amendment corrects this anomaly and provides an extra mechanism for crediting improved magnitude of clinical benefit evidenced by survival advantage at interim analysis. This upgrade does not apply to situations in which early stopping or crossover are based on detection of PFS advantage at interim analysis Cherny et al. Volume 28 Issue

9 Annals of Oncology Index case: None. Shortcoming 3: The penalty for studies with PFS advantage only, with no gain in OS or QoL, did not explicitly indicate that survival data must be mature. Amendment: The adjustment criterion has been amended Downgrade 1 level if the drug ONLY leads to improved PFS (mature data shows no OS advantage) and QoL assessment does not demonstrate improved QoL. Type: Nuanced. Rationale: It is inappropriate to penalise a study with inconclusive immature interim survival data when mature data may refute the penalty. Index case: The phase II Study 19 which evaluated the role of olaparib for BRCA-mutated ovarian cancer in first remission [25 27] demonstrated a PFS advantage but no survival advantage at an interim analysis of immature survival data. Incorrect application of the penalty for PFS studies which yield neither survival nor QoL benefit led to a downgrading from 3 to 2. This penalty is rescinded in v1.1. NEW FORM 3: SINGLE-ARM STUDIES IN ORPHAN DISEASES AND FOR DISEASES WITH HIGH UNMET NEED WHEN PRIMARY OUTCOME IS PROGRESSION FREE SURVIVAL (PFS) OR OVERALL RESPONSE RATE (ORR). Shortcoming: In v1.0, there was no mechanism to evaluate medications approved on the basis of single-arm studies. Amendment: Development of a new form 3 For single-arm studies in orphan diseases and for diseases with high unmet need when primary outcome is PFS or ORR. Type: Structural. Rationale: In the past 5 years, an increasing number of agents have been approved by regulatory authorities on the basis of data derived from single-arm studies. Overwhelmingly, these approvals have been in the setting of either uncommon diseases, cohorts of small target populations identified by biological markers or situations in which there is no alternative treatment strategy and high unmet need for new therapeutic strategies. Even in the absence of a comparator arm, there is a perceived need to provide information regarding the magnitude of clinical benefit. The published papers describing the studies of agents which have been approved are characterised by a paucity of data regarding the natural history of untreated patients and the literature is characterised by a presumption of poor outcomes in the absence of disease modifying treatment. A review of published studies leading to license indications identified that three major indicators of clinical benefit leading to approval were response rate, either ORR or CR, DoR, and PFS. Infrequently, measurements of QoL have been incorporated. Based upon this review, the ESMO-MCBS Working Group has developed a three grade scale for single-arm studies in orphan diseases and for diseases with high unmet need when primary outcome is PFS or ORR. The highest grade of 3 is awarded to studies demonstrating either: median PFS >6 months; ORR > 60%; or ORR 20 <60% AND DoR 9 months. Preliminary scores are adjusted based on toxicity (Downgrade 1 level if there are 30% grade 3 4 toxicities impacting on daily well-being), QoL (Upgrade 1 level if improved QoL) or confirmatory phase 4 experience (Upgrade 1 level for confirmatory, adequately sized, phase 4 experience). Field testing All amendments incorporated into v1.1 have been field tested in 128 studies, reported in 142 publications across a range of 15 disease groups in the comparative studies and 5 disease groups in the single-arm setting (see Tables 2 4 and supplementary Tables S1 S15, available at Annals of Oncology online). Field testing has been reviewed by the ESMO Guidelines Committee and ESMO Faculty for reasonableness. The amended version resulted in scoring changes in 12 randomised studies (Tables 2 and 3) and facilitated scoring for 10 single-arm studies (Table 4). Discussion Special article The ESMO-MCBS has proven to be a valued and reproducible tool for the evaluation of the magnitude of benefit from clinical studies that is essential for both policy and clinical discourse. Dynamic discussion with end-users who have provided feedback, critiques and suggestions has helped facilitate a dynamic quality improvement process that is essential to maintain the integrity of this important endeavour. Since its launch in 2015, the ESMO-MCBS has been widely cited (source Scopus/Google Scholar). It has been applied in clinical and in health technology assessment settings [28, 29], successfully employed by the ESMO Guidelines Committee [30], demonstrated clinical utility [31] and has served as the basis for benchmarking studies [32]. ESMO-MCBS v1.1 incorporates 9 amendments in forms 1, 2a and 2b and adds a new form 3 for scoring of single-arm studies. For the most part, the scoring remains very stable. The 10 revisions in v1.1 will alter the scores of only 12 out of 118 comparative studies. In the seven studies impacted by the revision, the newer version either upgrades the score or enables scoring which was previously not possible and in five instances a score is lowered by the amendments. A transparent process of development with scope for peer review, appeal and revision is one of the critical criteria for accountability for reasonableness [2, 3]. The validity of this revision derives from a careful deliberative process followed by extensive field testing and peer review. This first revision is an integral part of the planned process for on-going development and the Working Group invites feedback, discussion, review and constructive critiques of the revised v1.1 as well as any appeals regarding scores derived using the new version. Volume 28 Issue doi: /annonc/mdx

10 Special article Not all issues could be addressed in this revision. For the next planned revision, the ESMO-MCBS Working Group will be soliciting stakeholder input from patients and patient advocates regarding the toxicity penalties, credit for improvements in QoL or deferred deterioration in QoL and credits for single symptom improvements. This process will involve formal consultations with patient advocacy groups and an open invitation for feedback or suggestions regarding these aspects of the scale. The amendments incorporated into v1.1 will be applied to studies previously scored and will prevail until the next planned revision. Conclusion The 10 amendments incorporated into ESMO-MCBS v1.1 improve the discriminatory capacity and utility of the grading system. These revisions are part of an organisational commitment by ESMO to the continual quality improvement that is essential for accountability for reasonableness, incorporating a transparent revision process based on accrued experience, feedback, consultation and the critical review necessary to promote the integrity of this important endeavour. Acknowledgements The authors wish to acknowledge the support and contribution of the ESMO Executive Board, the ESMO Faculty, Members of the ESMO Guidelines Committee and the logistics and organisational support provided by ESMO Staff. Appendix I lists our oncology colleagues who have participated on the field testing of ESMO-MCBS v1.1 and agreed to place their names in this publication. We also thank those who wished to remain anonymous. Funding ESMO (no grant numbers apply). Disclosure The authors have declared the following: JB: Director of EORTC. EORTC conducts many studies sponsored by, or otherwise supported by, a large number of companies. EORTC is an independent research organisation. MJP: Board member: Radius. Consultant (honoraria): AstraZeneca, Lilly, MSD, Novartis, Pfizer, Roche-Genentech, Crescendo Biologics, Periphagen, Huya, Debiopharm, PharmaMar. Research grants to Institute: AstraZeneca, Lilly, MDS, Novartis, Pfizer, Roche-Genentech, Synthon, Radius, Servier. Speakers bureau/stock ownership: none. GP: Consulting and advisory services, research support: Amgen, Merck, Astra Zeneca, Roche, BMS, MSD and Lilly. JYD: Compensated participation to Advisory boards, lecture, Symposia: Amgen, Merck Serono, Bayer, Roche/Genentech, Sanofi, AstraZeneca, Boehringer-Ingelheim, Sirtex until April No further compensated participation in industry events from May 2016 onwards. JT: Advisory boards for Amgen, Bayer, Boehringer Ingelheim, Celgene, Chugai, Genentech, Lilly, MSD, Merck Serono, Novartis, Pfizer, Roche, Sanofi, Symphogen, Taiho and Takeda. CZ: Honoraria: AstraZeneca, Celgene, Roche, Novartis, Bristol Myers Squibb, MSD, Ariad and Newgen. EGEV: Currently conducting research sponsored by the following companies: Amgen, Roche/Genentech, Chugai Pharma, Synthon, AstraZeneca, Radius Health, CytomX Therapeutics and Nordic Nanovector (all payments to the institution). Consulting or advisory role for the following companies: Synthon, Medication and Merck (all payments to the institution). Not a member of any speakers bureau. All remaining authors have declared no conflicts of interest. References Annals of Oncology 1. Cherny NI, Sullivan R, Dafni U et al. A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol 2015; 26: Gruskin S, Daniels N. Process is the point: justice and human rights: priority setting and fair deliberative process. Am J Public Health 2008; 98: Daniels N. Accountability for reasonableness. BMJ 2000; 321: Thatcher N, Hirsch FR, Luft AV et al. Necitumumab plus gemcitabine and cisplatin versus gemcitabine and cisplatin alone as first-line therapy in patients with stage IV squamous non-small-cell lung cancer (SQUIRE): an open-label, randomised, controlled phase 3 trial. Lancet Oncol 2015; 16: Paz-Ares L, Socinski M, Shahidi J et al. Correlation of EGFR-expression with safety and efficacy outcomes in SQUIRE: a randomized, multicenter, open-label, phase III study of gemcitabine-cisplatin plus necitumumab versus gemcitabine-cisplatin alone in the first-line treatment of patients with stage IV squamous non-small cell lung cancer. Ann Oncol 2016; 27: Rose BS, Winer EP, Mamon HJ. Perils of the pathologic complete response. J Clin Oncol 2016; 34: Gianni L, Pienkowski T, Im Y-H et al. Efficacy and safety of neoadjuvant pertuzumab and trastuzumab in women with locally advanced, inflammatory, or early HER2-positive breast cancer (NeoSphere): a randomised multicentre, open-label, phase 2 trial. Lancet Oncol 2012; 13: Verma S, Miles D, Gianni L et al. Trastuzumab emtansine for HER2- positive advanced breast cancer. N Engl J Med 2012; 367: Welslau M, Dieras V, Sohn JH et al. Patient-reported outcomes from EMILIA, a randomized phase 3 study of trastuzumab emtansine (T- DM1) versus capecitabine and lapatinib in human epidermal growth factor receptor 2-positive locally advanced or metastatic breast cancer. Cancer 2014; 120: Swain SM, Kim S-B, Cortés J et al. Pertuzumab, trastuzumab, and docetaxel for HER2-positive metastatic breast cancer (CLEOPATRA study): overall survival results from a randomised, double-blind, placebo-controlled, phase 3 study. Lancet Oncol 2013; 14: Baselga J, Cortés J, Kim S-B et al. Pertuzumab plus trastuzumab plus docetaxel for metastatic breast cancer. N Engl J Med 2012; 366: Swain SM, Baselga J, Kim S-B et al. Pertuzumab, trastuzumab, and docetaxel in HER2-positive metastatic breast cancer. N Engl J Med 2015; 372: Cortes J, Baselga J, Im YH et al. Health-related quality-of-life assessment in CLEOPATRA, a phase III study combining pertuzumab with trastuzumab and docetaxel in metastatic breast cancer. Ann Oncol 2013; 24: Takada M, Fukuoka M, Kawahara M et al. Phase III study of concurrent versus sequential thoracic radiotherapy in combination with cisplatin and etoposide for limited-stage small-cell lung cancer: results of the Japan Clinical Oncology Group Study J Clin Oncol 2002; 20: Robert C, Thomas L, Bondarenko I et al. Ipilimumab plus dacarbazine for previously untreated metastatic melanoma. N Engl J Med 2011; 364: Maio M, Grob JJ, Aamdal S et al. Five-year survival rates for treatmentnaive patients with advanced melanoma who received ipilimumab plus dacarbazine in a phase III trial. J Clin Oncol 2015; 33: Cherny et al. Volume 28 Issue

11 Annals of Oncology 17. Herbst RS, Baas P, Kim DW et al. Pembrolizumab versus docetaxel for previously treated, PD-L1-positive, advanced non-small-cell lung cancer (KEYNOTE-010): a randomised controlled trial. Lancet 2016; 387: Garon EB, Ciuleanu TE, Arrieta O et al. Ramucirumab plus docetaxel versus placebo plus docetaxel for second-line treatment of stage IV nonsmall-cell lung cancer after disease progression on platinum-based therapy (REVEL): a multicentre, double-blind, randomised phase 3 trial. Lancet 2014; 384: Tabernero J, Yoshino T, Cohn AL et al. Ramucirumab versus placebo in combination with second-line FOLFIRI in patients with metastatic colorectal carcinoma that progressed during or after first-line therapy with bevacizumab, oxaliplatin, and a fluoropyrimidine (RAISE): a randomised, double-blind, multicentre, phase 3 study. Lancet Oncol 2015; 16: Von Hoff DD, Ervin T, Arena FP et al. Increased survival in pancreatic cancer with nab-paclitaxel plus gemcitabine. N Engl J Med 2013; 369: Borghaei H, Paz-Ares L, Horn L et al. Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer. N Engl J Med 2015; 373: Borghaei H, Brahmer JR, Horn L et al. Nivolumab vs docetaxel in patients with advanced NSCLC: CheckMate 017/057 2-y update and exploratory cytokine profile analyses. J Clin Oncol 2016; 34 (suppl; abstr 9025). 23. Larkin J, Chiarion-Sileni V, Gonzalez R et al. Combined nivolumab and ipilimumab or monotherapy in untreated melanoma. N Engl J Med 2015; 373: Ribas A, Puzanov I, Dummer R et al. Pembrolizumab versus investigatorchoice chemotherapy for ipilimumab-refractory melanoma (KEYNOTE- 002): a randomised, controlled, phase 2 trial. Lancet Oncol 2015; 16: Ledermann J, Harter P, Gourley C et al. Olaparib maintenance therapy in platinum-sensitive relapsed ovarian cancer. N Engl J Med 2012; 366: Ledermann J, Harter P, Gourley C et al. Olaparib maintenance therapy in patients with platinum-sensitive relapsed serous ovarian cancer: a preplanned retrospective analysis of outcomes by BRCA status in a randomised phase 2 trial. Lancet Oncol 2014; 15: Ledermann JA, Harter P, Gourley C et al. Quality of life during olaparib maintenance therapy in platinum-sensitive relapsed serous ovarian cancer. Br J Cancer 2016; 115: Wild C, Grössmann N, Bucsics A et al. Utilisation of the ESMO-MCBS in practice of HTA. Ann Oncol 2016; 27: Hammerman A, Greenberg-Dotan S, Feldhamer I et al. The Esmo magnitude of clinical benefit scale for novel cancer medicines correspondence with prioritization decisions in updating the Israeli National List of Health Services. Value Health 2015; 18: A European Society for Medical Oncology. ESMO Clinical Practice Guidelines News. Lugano: ESMO 2016; Guidelines/Guidelines-News. 31. Kiesewetter B, Raderer M, Steger GG et al. The European Society for Medical Oncology Magnitude of Clinical Benefit Scale in daily practice: a single institution, real-life experience at the Medical University of Vienna. ESMO Open 2016; 1: e Del Paggio JC, Azariah B, Sullivan R et al. Do contemporary randomized controlled trials meet ESMO thresholds for meaningful clinical benefit? Ann Oncol 2017; 28: Gianni L, Pienkowski T, Im YH et al. 5-year analysis of neoadjuvant pertuzumab and trastuzumab in patients with locally advanced, inflammatory, or early-stage HER2-positive breast cancer (NeoSphere): a multicentre, open-label, phase 2 randomised trial. Lancet Oncol 2016; 17: Geisse J, Caro I, Lindholm J et al. Imiquimod 5% cream for the treatment of superficial basal cell carcinoma: results from two phase III, randomized, vehicle-controlled studies. J Am Acad Dermatol 2004; 50: Stark D, Nankivell M, Pujade-Lauraine E et al. Standard chemotherapy with or without bevacizumab in advanced ovarian cancer: quality-of-life outcomes from the International Collaboration on Ovarian Neoplasms (ICON7) phase 3 randomised trial. Lancet Oncol 2013; 14: Special article 36. Perren TJ, Swart AM, Pfisterer J et al. A phase 3 trial of bevacizumab in ovarian cancer. N Engl J Med 2011; 365: Janne PA, Yang JC, Kim DW et al. AZD9291 in EGFR inhibitor-resistant non-small-cell lung cancer. N Engl J Med 2015; 372: Shaw AT, Kim DW, Mehra R et al. Ceritinib in ALK-rearranged nonsmall-cell lung cancer. N Engl J Med 2014; 370: Shaw AT, Ou SH, Bang YJ et al. Crizotinib in ROS1-rearranged nonsmall-cell lung cancer. N Engl J Med 2014; 371: Kwak EL, Bang YJ, Camidge DR et al. Anaplastic lymphoma kinase inhibition in non-small-cell lung cancer. N Engl J Med 2010; 363: Camidge DR, Bang YJ, Kwak EL et al. Activity and safety of crizotinib in patients with ALK-positive non-small-cell lung cancer: updated results from a phase 1 study. Lancet Oncol 2012; 13: Ansell SM, Lesokhin AM, Borrello I et al. PD-1 blockade with nivolumab in relapsed or refractory Hodgkin s lymphoma. N Engl J Med 2015; 372: Kaufman B, Shapira-Frommer R, Schmutzler RK et al. Olaparib monotherapy in patients with advanced cancer and a germline BRCA1/2 mutation. J Clin Oncol 2015; 33: Powles T, Eder JP, Fine GD et al. MPDL3280A (anti-pd-l1) treatment leads to clinical activity in metastatic bladder cancer. Nature 2014; 515: Rosenberg JE, Hoffman-Censits J, Powles T et al. Atezolizumab in patients with locally advanced and metastatic urothelial carcinoma who have progressed following treatment with platinum-based chemotherapy: a singlearm, multicentre, phase 2 trial. Lancet 2016; 387: Krueger DA, Care MM, Holland K et al. Everolimus for subependymal giantcell astrocytomas in tuberous sclerosis. N Engl J Med 2010; 363: Seiwert TY, Burtness B, Mehra R et al. Safety and clinical activity of pembrolizumab for treatment of recurrent or metastatic squamous cell carcinoma of the head and neck (KEYNOTE-012): an open-label, multicentre, phase 1b trial. Lancet Oncol 2016; 17: Le DT, Uram JN, Wang H et al. PD-1 blockade in tumors with mismatch-repair deficiency. N Engl J Med 2015; 372: Figure 1. ESMO Magnitude of Clinical Benefit Scale v1.1. Instructions 1. There are five forms: Evaluation form 1: for new approaches to adjuvant therapy or new potentially curative therapies Hyper mature data from studies that were un-blinded after compelling early results with subsequent access to the superior arm are contaminated, subsequently late intention to treat (ITT) follow-up data are not evaluable. Evaluation form 2a: for therapies that are not likely to be curative with primary end point of OS with separate sheets for: IF median OS with the standard treatment is 12 months IF median OS with the standard treatment >12 months, 24 months IF median OS with the standard treatment >24 months Evaluation form 2b: for therapies that are not likely to be curative with primary end point PFS with separate sheets for: IF median PFS with standard treatment 6 months IF median PFS with standard treatment >6 months Evaluation form 2c: for therapies that are not likely to be curative with primary end point other than OS or PFS or equivalence (non-inferiority) studies. Evaluation form 3: for single-arm studies in orphan diseases and for diseases with high unmet need when primary outcome is PFS or ORR. Volume 28 Issue doi: /annonc/mdx