This study was designed to evaluate an online prognosis

Neuro-Oncology 15(8):1074 1078, 2013. doi:10.1093/neuonc/not033 Advance Access publication March 29, 2013 NEURO-ONCOLOGY Can the prognosis of individual patients with glioblastoma be predicted using an online calculator? Christopher Parks, James Heald, Gregory Hall, and Ian Kamaly-Asl Department of Neurosurgery, Salford Royal NHS Foundation Trust, Stott Lane, Salford (C.P., J.H., I.K.-A.), and Royal Preston Hospital, Sharoe Green Lane, Fulwood, Preston (G.H.), United Kingdom Background. In an exploratory subanalysis of the European Organisation for Research and Treatment of Cancer and National Cancer Institute of Canada (EORTC/NCIC) trial data, Gorlia et al. identified a variety of factors that were predictive of overall survival, including therapy administered, age, extent of surgery, mini-mental score, administration of corticosteroids, World Health Organization (WHO) performance status, and O-methylguanine-DNA methyltransferase (MGMT) promoter methylation status. Gorlia et al. developed 3 nomograms, each intended to predict the survival times of patients with newly diagnosed glioblastoma on the basis of individual-specific combinations of prognostic factors. These are available online as a GBM Calculator and are intended for use in patient counseling. This study is an external validation of this calculator. Method. One hundred eighty-seven patients from 2 UK neurosurgical units who had histologically confirmed glioblastoma (WHO grade IV) had their information at diagnosis entered into the GBM calculator. A record was made of the actual and predicted median survival time for each patient. Statistical analysis was performed to assess the accuracy, precision, correlation, and discrimination of the calculator. Results. The calculator gives both inaccurate and imprecise predictions. Only 23% of predictions were within 25% of the actual survival, and the percentage bias is 140% in our series. The coefficient of variance is 76%, where a smaller percentage would indicate greater precision. There is only a weak positive correlation between the predicted and actual survival among patients (R 2 of 0.07). Discrimination is inadequate as measured by a C- index of 0.62. Conclusions. The authors would not recommend the use of this tool in patient counseling. If departments were Received June 5, 2012; accepted February 15, 2013. Corresponding author: Christopher Parks, MBBS, FRCS(SN), Department of Neurosurgery, Salford Royal NHS Foundation Trust, Stott Road, Salford, M6 8HD, United Kingdom (cjcparks@gmail.com). considering its use, we would advise that a similar validating exercise be undertaken. Keywords: GBM calculator, glioblastoma, prognosis prediction. This study was designed to evaluate an online prognosis prediction calculator intended for use in patients with glioblastoma by comparison with actual outcomes in 2 UK-based neurosurgical units. Glioblastoma multiforme (GBM) is the most common and most aggressive primary brain tumor in adults. With an incidence rate of 2 3 cases per 100 000 per year, GBM accounts for 22.6% of all brain and central nervous system tumors. 1 The overall median survival is reported as 7.6 9.4 months, 2,3 but this can reach 14 months with a 2-year survival rate of 27% in clinical trial groups. 4 It is therefore a conspicuous clinical entity and conveys a poor prognosis. In 2005, a phase III randomized trial organized by the European Organisation for Research and Treatment of Cancer (EORTC) and the National Cancer Institute of Canada (NCIC) Clinical Trials Group demonstrated a survival benefit conveyed by radiotherapy plus concomitant and adjuvant temozolomide in the treatment of patients with newly diagnosed glioblastoma. 4 In an exploratory subanalysis of the EORTC/NCIC trial data, Gorlia et al. subsequently identified a variety of factors that were predictive of overall survival, including therapy administered, age, extent of surgery, mini-mental score examination, administration of corticosteroids, World Health Organization (WHO) performance status, and O-methylguanine-DNA methyltransferase (MGMT) promoter methylation status. 5 This is in agreement with the most consistently reported prognostic factors for GBM (age, extent of surgical resection, and performance status), 6 8 but promoter region methylation of the MGMT enzyme, which repairs DNA damaged by alkylating agents, and MMSE have also previously been associated with improved prognosis. 9,10 # The Author(s) 2013. Published by Oxford University Press on behalf of the Society for Neuro-Oncology. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

Parks et al.: Can the prognosis of individual patients with glioblastoma On the basis of their findings, Gorlia et al. developed 3 nomograms, each intended to predict the survival times among patients with newly diagnosed GBM on the basis of individual-specific combinations of prognostic factors. 5 These nomograms have been made available online as a GBM Calculator and are intended for use in patient counseling and treatment selection and in the design and interpretation of clinical trials. 11 The nomograms were developed by correlating all of the identified predictive factors with outcome and combining them to give a predicted median survival and a 2-year survival probability. This was done in 3 distinct populations to separate the potential skewing effect of MGMT promoter methylation status and treatment given. Population 1 was the entire group (n ¼ 573). Population 1 was stratified using the factors shown in Table 1. In the authors experience, many patients would like to know their prognosis at diagnosis to understand their condition and to plan practical and financial affairs, and it certainly helps them and their families to start the anticipatory grieving process necessary in terminally ill patients. 12 The GBM calculator promised to be a very helpful clinical tool in this respect. Before offering this information to patients, the authors wanted to validate its use in a local patient population, because it was felt that giving potentially incorrect information would be unhelpful and insensitive and would undermine confidence in the treating team. Materials and Methods Prospectively collected data from 187 patients from 2 UK neurosurgical units with histologically confirmed glioblastoma (WHO grade IV) who had been treated had their information at diagnosis retrospectively entered into the GBM calculator. All were patients during 2006 2010 and received treatment from a multidisciplinary team involving neurosurgeons, neurooncologists, neuro-radiologists, and neuropathologists. Model 1 of the calculator was used for all patients, because it was not routine in these units at the time of the study to collect MGMT status in all cases. Patients could only be included in the study if they had been treated surgically and received either radiotherapy and Table 1: Stratification criteria for GBM calculator model 1 Stratification Factor Treatment Options Radiotherapy and temozolomide Radiotherapy Age 50 51 60.60 Extent of surgery Total resection Partial resection Biopsy Mini-mental score examination 27 30,27 Corticosteroids No Yes chemotherapy or radiotherapy alone for their condition, enabling stratification on model 1 of the calculator. It was not possible to use the identical inclusion and exclusion criteria as was done for the original EORTC trial, because since the trial s publication, patients have received the new optimal standard of care, consisting of concomitant and adjuvant treatment with temozolomide and radiotherapy, if they met the criteria. Patients receiving radiotherapy alone without temozolomide were, therefore by definition, in a group with lower performance status. Concern about this potential bias has provoked a subgroup analysis using only the data on those who received both chemotherapy and radiotherapy. Comparison is made between our study group and the trial patients in Table 2. The calculator does not specify that patients should meet the trial inclusion criteria, and for example, the upper age stratification is given as.60 years, despite the exclusion from the trial of patients.70 years of age. The actual and predicted median survival time for each patient was recorded and statistically analyzed. The statistical analysis of these results needs to assess accuracy, precision, and correlation of the predictions. It would be possible for the predictions to be precise but biased by a factor that separated the local population from that used to generate the nomograms. The calculator could still be a very useful tool if a calibration factor could be used to give a precise and accurate prediction. The accuracy of each prediction can be given by dividing the predicted median survival by the actual survival. The accuracy can also be demonstrated using the mean magnitude relative error (MMRE) as a measure of spread and pred(25) as a measure of kurtosis of the distribution of prediction accuracies. The MMRE is the mean relative error that is calculated by finding the mean prediction error as a proportion of the actual survival. The pred(25) is the proportion of predictions within 25% of the actual survival. Precision of the predictions is assessed using the coefficient of variance performed on the accuracy (prediction/actual survival). It is derived by dividing the standard deviation by the mean accuracy and expressing it as a percentage. A lower percentage implies a more precise predictor. This, therefore, gives a measure of spread of the accuracy of the predictions. Correlation of the predictions with the actual survival using Pearson s correlation enables calculation of a coefficient of determination (R 2 ). This gives information about goodness of fit and is a measure of how likely the model is to accurately predict survival. It can be expressed as a percentage. Prediction models can also be assessed in terms of discrimination and calibration. 13 Gorlia et al. calculated the Harrell Concordance-index (C-index) as a measure of this. This is the probability that for 2 patients selected at random, the patient who died first had a higher probability of doing so according to the prediction model. The minimal value of c is 0.5; the maximum is 1. In their textbook, Hosmer and Lemeshow consider c values of 0.7 0.8 to show acceptable discrimination, values of 0.8 0.9 to indicate excellent discrimination, and values.0.9 to show outstanding discrimination. 14 NEURO-ONCOLOGY AUGUST 2013 1075

Parks et al.: Can the prognosis of individual patients with glioblastoma be predicted? Table 2. Comparison of study group and trial patients. Variable Percentage of EORTC patients (population 1) (n 5 573) Extent of surgery Biopsy 16 35 Partial 44 50 Complete 39 15 Treatment Radiotherapy and temozolomide 50 51 Radiotherapy only 50 49 Age (years) 50 32 21 51 60 38 24.60 30 55 Sex Male 61 65 Female 38 35 Not recorded,1 N/A Corticosteroids at randomisation No 24 1 Yes 75 99 Not recorded,1 N/A MMSE score,27 67 19 27 30 29 81 Not recorded 4 N/A Percentage of study population (n 5 187) Fig. 1. Diagram to clarify the concepts of accuracy and precision. (A) Inaccurate and imprecise. (B) Inaccurate but precise. (C) Accurate but imprecise. (D) Accurate and precise. Results One hundred eighty-seven patients had their actual survival compared with the survival predicted by model 1 of the EORTC GBM calculator. 11 Comparison of the study population and the trial patients used to design the calculator (Table 2) reveal that the sex ratios and the proportion of persons receiving radiotherapy alone were very similar. The study group had a higher proportion of patients in the older age group, which is a reflection of the expected age-related incidence of glioblastoma in the population. This also may explain why fewer had a complete resection. The study group had a higher proportion of patients treated with corticosteroids, reflecting local practice, but this may explain why the study group had a greater proportion of patients with a normal mini-mental test examination result. Having different proportions of the study group in each stratification of the calculator should not, however, bias the results, because stratifying the patients into risk groups is the way in which the calculator is supposed to work. The accuracy (predicted/actual) was calculated for each patient. A subgroup analysis was performed using the same statistical tests on the group of patients who were of a sufficient performance status to receive combined chemo- and radiotherapy and, thus, would have been eligible for the EORTC trial. In statistical analysis of the entire group of 187 patients, the pred(25) was 23% and the MMRE was 1.4, giving a percentage bias of 140% with a tendency to overestimate. These describe significant inaccuracy of the calculator in our series. As a measure of precision, the coefficient of variance was 76%, where a smaller percentage would indicate greater precision of the calculator. The correlation between the actual and predicted survivals was poor. Pearson s correlation coefficient was measured at 0.27, which gives a coefficient of determination (R 2 ) of 0.07, indicating that the calculator would have a 7% chance of predicting the survival accurately (Fig. 2). The c-index for our data is 0.62, which shows inadequate discrimination. It is, however, very similar to the c-index values quoted by Gorlia et al. of 0.62 0.66 for the different models. 5 When the statistics were repeated on patients receiving chemo- and radiotherapy (n ¼ 96), excluding those receiving radiotherapy only, they were similar or worse. The pred(25) was 27.08%, and the MMRE was 1.48, giving a percentage bias of 148%, still with a tendency to overestimate. The coefficient of variance for this subgroup is 88.5%, conveying even less precision. There was no correlation between the predicted and actual survivals of this patient group, with a Pearson s correlation coefficient of 0.056 and a coefficient of determination (R 2 ) of 0.003. The c-index for this subgroup was only 0.52 (Table 3). A proportion of our patients were.70 years of age (18%), which is not excluded by the calculator but 1076 NEURO-ONCOLOGY AUGUST 2013

Parks et al.: Can the prognosis of individual patients with glioblastoma Fig. 2. Scatter plot of predicted median survival plotted against actual survival in months. There is only a weak positive correlation (Pearson s correlation coefficient: 0.27) and a coefficient of determination R 2 of 0.07. Table 3. Results of statistical tests comparing the whole group with the subgroup excluding those treated with radiotherapy only. Statistical test Whole study group (n 5 187) Coefficient of 76% 88.5% variance Pred(25) 23% 27% Mean Magnitude 1.4 1.48 Relative Error Percentage Bias 140% 148% Correlation 0.27 0.056 Coefficient Coefficient of 0.07 0.003 Determination (R 2 ) C-Index 0.62 0.52 was from the original trial. This group had similar prediction quality to that of the younger groups. The accuracy was actually slightly better in this group than in the whole study group (pred[25] of 24% and MMRE of 0.99, compared with 23% and 1.4, respectively), but the precision was slightly worse (coefficient of variance, 89% vs 76%). Discussion Subgroup excluding radiotherapy only (n 5 96) In the population considered in this study, an individual patient s prognosis could not accurately or precisely be predicted by the EORTC online GBM calculator. Its use in counseling our patients would lead to the majority being misinformed about their prognosis. The majority of patients prefer a realistic and individualized approach from their cancer specialist and detailed information when discussing prognosis. Patients receive hope from clinicians whom they believe to be expert in their condition. 15 It would, therefore, be extremely useful for clinicians to be able to use a reliable predictor of an individual patient s prognosis. Clinicians are rarely able to give an accurate prognosis and often consciously over-estimate or avoid frank disclosure. 16 The use of a calculator giving inaccurate information would significantly undermine a patient s confidence in their clinician and could potentially cause loss of hope and a lack of preparation. It is often speculated that trial patients have better outcomes than the general population. This can be attributable to stringent inclusion and exclusion criteria but may also be attributable to the closer monitoring that the patients receive or merely the positive Hawthorne effect associated with inclusion in a clinical trial. 17 The use of the EORTC trial patients in generating the nomograms used in the calculator, therefore, raised concerns among the authors that they may not reflect a standard population. It was anticipated that a correction coefficient may need to be calculated to adapt the calculator for use in our patients. The use of a correction coefficient would only be appropriate if the results were precise but consistently inaccurate; however, this is not the case. A concern before this study was the statistical validity of comparing an actual survival with a median predicted survival. It was felt, however, that the prediction that a patient received should give a close indication of how long they would live even accounting for inaccuracy and individual variation, to be a valid adjunct in patient counseling. As an external validation for model 1of the GBM calculator, this study shows it to be a poor predictor of individual patient s survival. The reasons for poor performance of the calculator could be attributable to NEURO-ONCOLOGY AUGUST 2013 1077

Parks et al.: Can the prognosis of individual patients with glioblastoma be predicted? differences in the study populations or their treatment but the similarities of the c-index in our study to that calculated by Gorlia et al. suggest more fundamental flaws in the calculator. This is compounded by the fact that, since the EORTC trial, patients well enough to receive radiotherapy with concomitant and adjuvant chemotherapy would receive this treatment. The patients receiving radiotherapy only would be in a group with poor performance status. These patients, however, are easier for the calculator to discriminate from those receiving chemotherapy and radiotherapy, which explains why the correlation and c-index are so low when analyzing only the subgroup with better performance status. The inclusion of patients.70 years of age was not the reason for the inaccuracy and imprecision of the calculator in our study group. In conclusion, the authors would not recommend the use of this tool in patient counseling. If a unit were considering its use, we would advise that a similar validating exercise be undertaken. None declared. Funding Conflict of interest statement. None declared. Acknowledgments Christopher Parks: Principle author, data collection, data interpretation. James Heald: Data collection, data interpretation. Gregory Hall: Study design. Ian Kamaly-Asl: Senior author and study design. References 1. Surawicz TS, McCarthy BJ, Kupelian V, et al. Descriptive epidemiology of primary brain and CNS tumors: Results from the Central Brain Tumor Registry of the United States, 1990 1994. Neuro-oncol. 1999; 1(1):14 25. 2. Rock K, McArdle O, Forde P, et al. A clinical review of treatment outcomes in glioblastoma multiforme - the validation in a non-trial population of the results of a randomised Phase III clinical trial: has a more radical approach improved survival? Br J Radiol. 2012; 85(1017). 3. Bauchet L, Mathieu-Daudé H, Fabbro-Peray P, et al. Oncological patterns of care and outcome for 952 patients with newly diagnosed glioblastoma in 2004. Neuro-oncol. 2010;12(7):725 735. 4. Stupp R, Mason WP, van den Bent MJ, et al. Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. N Engl J Med. 2005;352(10):987 996. 5. Gorlia T, Van Den Bent MJ, Hegi ME, et al. Nomograms for predicting survival of patients with newly diagnosed glioblastoma: prognostic factor analysis of EORTC and NCIC trial 26981 22981/CE.3. Lancet Oncol. 2008;9(1):29 38. 6. Siker ML, Wang M, Porter K, et al. Age as an independent prognostic factor in patients with glioblastoma: a Radiation Therapy Oncology Group and American College of Surgeons National Cancer Data Base comparison. JNeurooncol. 2011;104(1):351 356. 7. Stummer W, van den Bent MJ, Westphal M. Cytoreductive surgery of glioblastoma as the key to successful adjuvant therapies: new arguments in an old discussion. Acta Neurochir. 2011;153(6): 1211 8. 8. Filippini G, Falcone C, Boiardi A, et al. Prognostic factors for survival in 676 consecutive patients with newly diagnosed primary glioblastoma. Neuro- Oncol. 2008;10(1):79 87. 9. Olson RA, Brastianos PK, Palma DA. Prognostic and predictive value of epigenetic silencing of MGMT in patients with high grade gliomas: a systematic review and meta-analysis. J Neurooncol. 2011;105(2): 325 335. 10. Buckner JC. Factors influencing survival in high-grade gliomas. Semin Oncol. 2003;30(6 Suppl 19):10 14. 11. Anon. http://www.eortc.be/tools/gbmcalculator/default.aspx. Available at: http://www.eortc.be/tools/gbmcalculator/default.aspx. Accessed February 5, 2012. 12. Clukey L. Anticipatory mourning: processes of expected loss in palliative care. Int J Palliat Nurs. 2008;14(7):316, 318 325. 13. Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338(May 28 1): b605 b605. 14. Hosmer D, Lameshow S. Applied Logistic Regression. 2nd ed. Hoboken, NJ: Wiley-Blackwell; 2000. 15. Hagerty RG, Butow PN, Ellis PM, et al. Communicating with realism and hope: incurable cancer patients views on the disclosure of prognosis. J Clin Oncol. 2005;23(6):1278 88. 16. Lamont EB, Christakis NA. Prognostic disclosure to patients with cancer near the end of life. Ann Intern Med. 2001;134(12): 1096 105. 17. Wolfe F, Michaud K. The Hawthorne effect, sponsored trials, and the overestimation of treatment effectiveness. J Rheumatol. 2010;37(11): 2216 20. 1078 NEURO-ONCOLOGY AUGUST 2013