Supplementary appendix This appendix formed part of the original submission and has been peer reviewed. We post it as supplied by the authors. Supplement to: Callegaro D, Miceli R, Bonvalot S, et al. Development and external validation of two nomograms to predict overall survival and occurrence of distant metastases in adults after surgical resection of localised soft-tissue sarcomas of the extremities: a retrospective analysis. Lancet Oncol 216; published online April 5. http://dx.doi.org/1.116/s147-245(16)1-3.
Appendix: supplemental statistical details Contents 1. Assessment of model performance Page 2 2. Generalized Boosted Regression Page 2 3. References Page 3 4. Supplemental Table 1. Model Performance Measures. Page 4 5. Supplemental Table 2. Overall survival nomogram scoring system. Page 5 6. Supplemental Table 3. Distant metastasis nomogram scoring system. Page 6 7. Supplemental Figure 1. Kaplan-Meier curves for overall survival. Page 7 8. Supplemental Figure 2. Decision curves for overall survival. Page 8 9. Supplemental Figure 3. Crude cumulative incidence curves of distant metastasis. Page 9 1. Supplemental Figure 4. Decision curves for distant metastasis. Page 1 1
1. Assessment of model performance We evaluated model performance considering the following features: - overall accuracy, as estimated by the Brier score (BS) 1 both on the absolute scale (Model BS) and as a relative improvement over the marginal Kaplan-Meier curve prediction (KM BS), namely (KM BS Model BS)/KM BS. - calibration, which was evaluated internally (in the developing set) and externally (in the validation sets) by elaborating the calibration plots and by performing the D Agostino and Nam 2 version of the Hosmer and Lemeshow test. 3 - discriminative ability, quantified by the Harrell C index 4 together with its 95% bootstrap confidence interval. 5 Decision curve analysis 6 was also used to assess the OS and DM models predictions on the development set; our application followed the approach proposed by Vickers. 7 The decision curves are used to assess the net benefit of nomogram-assisted decisions at different threshold probabilities, compared with the net benefit of treat all/treat none strategies. 2. Generalized Boosted Regression We applied the Generalized Boosted Regression (GBM) 8 in OS analysis only. GBM is an established exploratory tool able to handle a great number of predictor variables, possibly linked by complex nonlinear relationships or interactions, and to determine the relative influence (RI) of each variable (higher numbers indicating stronger influence on OS). We used the function gbm of the R library gbm, with the following specifications: 1 trees (this is equivalent to the number of iterations and the number of basis functions in the additive expansion); 5 cross validation folds; up to2-way interactions between the model factors; 1 minimum number of observations in the trees terminal nodes;.5 bag.fraction (the fraction of the training set observations randomly selected to propose the next tree in the expansion); 1% of observations used for GBM fitting; shrinkage parameter applied to each tree: we fixed a value corresponding to a very high precision level (.1); random seed: as different seeds generate different models, we considered 1 seeds randomly generated from a uniform distribution (,1). By changing the random seeds, 1 GBM Cox models were fitted and this allowed us to perform a sensitivity analysis of GBM model results. We described below the distributions of RI values for the 1 models: Age Depth Grading Histology Margins Size Min. 13.88. 16.44 2.875. 6.92 1st Qu. 15.12. 17.81 3.619. 62.17 Median 15.46. 18.19 3.815. 62.51 Mean 15.48.8355 18.17 3.821.84 62.51 3rd Qu. 15.8.1952 18.55 4.35.1986 62.87 Max. 16.99.1 19.75 4.792.8884 64.7 2
6 5 4 2 1 Age Depth Grading Histology Margins Size 3. References 1. Brier GW. Verification of forecasts expressed in terms of probability. Monthly Weather Review 195.78:1 3. 2. D Agostino RB Sr and Nam BH. Evaluation of the performance of survival analysis models: discrimination and calibration measures. In: Balakrishnan N and Rao C (eds) Advances in survival anlaysis: handbook of statistics. Amsterdam: Elsevier, 24, pp.1 25. 3. Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 1997; 16(9): 965-8. 4. Harrell FE,Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15(4): 361-87. 5. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Chapman and Hall, New York, NY; 1993. 6. Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol 215; 16(4): e173-8. 7. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 26; 26(6): 565-74. 8. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Springer Series in Statistics Springer New York Inc., New York, NY, USA; 21. 3
Supplemental Table 1. Model performance measures. Overall survival model Harrell C (95% CI).767 Brier score Absoute Relative INT IGR Mount Sinai Royal Marsden (.743-.789).112.2.698 (.638-.754).131.9.775 (.754-.796).143.173.762 (.72-.86).119.144 Hosmer and Lemeshow test p value 5 years 1 years.452.232 Distant metastasis Harrell C (95% CI).759 Brier score Absoute Relative Hosmer and Lemeshow test p value 5 years 1 years (.736-.781).129.178.33.176.73 <.1.652 (.65-.699).174.33.118.1 <.1 <.1.744 (.72-.768).176.27.98 <.1.48 --.749 (.77-.791).155.19.4 -- Abbreviations. CI: confidence interval. For the definition of absolute and relative Brier score, see the Supplemental statistical details. 4
Supplemental Table 2. Overall survival nomogram scoring system.* Age (years) 18 25 35 4 45 5 55 6 65 7 75 8 85 9 1 1 2 3 5 6 8 11 14 18 22 26 35 39 48 Tumor maximal size (cm) 1 5 1 15 2 25 35 Grading I II III Histology Leio DD/pleom lipo Myxoid lipo MPNST Myxofibro Other Synovial UPS Vascular 7 33 54 65 73 82 91 1 44 28 12 19 15 21 7 53 Total 18 169 16 152 144 124 11 87 65 5 37 5-year OS.1.2..4.5.7.8.9.95.97.98 Total 188 167 156 148 139 131 111 97 74 53 37 25 1-year OS.1.1.2..4.5.7.8.9.95.97.98. * OS predictions corresponding to total points not shown in the table may be obtained by linear interpolation. The same approach may be followed for obtaining specific points for values of age and size not included here. 5
Supplemental Table 3. Distant metastasis nomogram scoring system. Histology Leio DD/pleom lipo Myxoid lipo MPNST Myxofibro Other Synovial UPS Vascular 36 7 6 17 22 27 14 36 Grading I II III 39 53 Tumor maximal size (cm) 1 5 1 15 2 25 35 9 43 68 76 82 88 94 1 Total 15 88 111 126 137 146 163 172 183 5-year DM CCI.1.1.2..4.5.7.8.9 Total 11 84 17 121 133 142 159 168 179 187 1-year DM CCI.1.1.2..4.5.7.8.9.95 * DM predictions corresponding to total points not shown in the table may be obtained by linear interpolation. The same approach may be followed for obtaining specific points for values of size not included here. 6
Supplemental Figure 1 Supplemental Figure 1. Kaplan-Meier curves for overall survival of patients treated surgically for primary extremity soft tissue sarcomas in the development set and in the validation sets. Blue line: Fondazione IRCCS, Istituto Nazionale dei Tumori, Milan, Italy Red line: Institut Gustave Roussy, Villejuif, France Green line: Mount Sinai Hospital, Toronto, Canada Purple line: Royal Marsden Hospital, London, United Kingdom 7
Supplemental Figure 2 Supplemental Figure 2. Decision curves for overall survival at 5 (Panel A) and 1 years (Panel B). Solid thin line: net benefit of a strategy of treating all patients. Solid bold line: net benefit of treating no patients. Dotted line: net benefit of a strategy of treating patients according to the nomogram predictions. 8
Supplemental Figure 3 Supplemental Figure 3. Crude cumulative incidence curves of distant metastasis of patients treated surgically for primary extremity soft tissue sarcomas in the development set and in the validation sets. Blue line: Fondazione IRCCS, Istituto Nazionale dei Tumori, Milan, Italy Red line: Institut Gustave Roussy, Villejuif, France Green line: Mount Sinai Hospital, Toronto, Canada Purple line: Royal Marsden Hospital, London, United Kingdom 9
Supplemental Figure 4 Supplemental Figure 4. Decision curves for distant metastasis at 5 (Panel A) and 1 years (Panel B). Solid thin line: net benefit of a strategy of treating all patients. Solid bold line: net benefit of treating no patients. Dotted line: net benefit of a strategy of treating patients according to the nomogram predictions. 1