R. Harding, P. Trnková, S. J. Weston, J. Lilley, C. M. Thompson, S. C. Short, C. Loughrey, V. P. Cosgrove, A. J. Lomax, and D. I.

Transcription

1 Benchmarking of a treatment planning system for spot scanning proton therapy: Comparison and analysis of robustness to setup errors of photon IMRT and proton SFUD treatment plans of base of skull meningioma R. Harding, P. Trnková, S. J. Weston, J. Lilley, C. M. Thompson, S. C. Short, C. Loughrey, V. P. Cosgrove, A. J. Lomax, and D. I. Thwaites Citation: Medical Physics 41, (2014); doi: / View online: View Table of Contents: Published by the American Association of Physicists in Medicine Articles you may be interested in Comparison of organ-at-risk sparing and plan robustness for spot-scanning proton therapy and volumetric modulated arc photon therapy in head-and-neck cancer Med. Phys. 42, 6589 (2015); / Proton therapy dose distribution comparison between Monte Carlo and a treatment planning system for pediatric patients with ependymomaa) Med. Phys. 39, 4742 (2012); / Evaluation of a commercial biologically based IMRT treatment planning system Med. Phys. 35, 5851 (2008); / Treatment planning and verification of proton therapy using spot scanning: Initial experiences Med. Phys. 31, 3150 (2004); / APL Photonics

2 Benchmarking of a treatment planning system for spot scanning proton therapy: Comparison and analysis of robustness to setup errors of photon IMRT and proton SFUD treatment plans of base of skull meningioma R. Harding a) St James s Institute of Oncology, Medical Physics and Engineering, Leeds LS9 7TF, United Kingdom and Abertawe Bro Morgannwg University Health Board, Medical Physics and Clinical Engineering, Swansea SA2 8QA, United Kingdom P. Trnková Paul Scherrer Institute, Centre for Proton Therapy, Villigen 5232, Switzerland S. J. Weston, J. Lilley, and C. M. Thompson St James s Institute of Oncology, Medical Physics and Engineering, Leeds LS9 7TF, United Kingdom S. C. Short Leeds Institute of Molecular Medicine, Oncology and Clinical Research, Leeds LS9 7TF, United Kingdom and St James s Institute of Oncology, Oncology, Leeds LS9 7TF, United Kingdom C. Loughrey St James s Institute of Oncology, Oncology, Leeds LS9 7TF, United Kingdom V. P. Cosgrove St James s Institute of Oncology, Medical Physics and Engineering, Leeds LS9 7TF, United Kingdom A. J. Lomax Paul Scherrer Institute, Centre for Proton Therapy, Villigen 5232, Switzerland D. I. Thwaites St James s Institute of Oncology, Medical Physics and Engineering, Leeds LS9 7TF, United Kingdom and Institute of Medical Physics, School of Physics, University of Sydney, Sydney NSW 2006, Australia (Received 20 March 2014; revised 26 August 2014; accepted for publication 18 September 2014; published 21 October 2014) Purpose: Base of skull meningioma can be treated with both intensity modulated radiation therapy (IMRT) and spot scanned proton therapy (PT). One of the main benefits of PT is better sparing of organs at risk, but due to the physical and dosimetric characteristics of protons, spot scanned PT can be more sensitive to the uncertainties encountered in the treatment process compared with photon treatment. Therefore, robustness analysis should be part of a comprehensive comparison between these two treatment methods in order to quantify and understand the sensitivity of the treatment techniques to uncertainties. The aim of this work was to benchmark a spot scanning treatment planning system for planning of base of skull meningioma and to compare the created plans and analyze their robustness to setup errors against the IMRT technique. Methods: Plans were produced for three base of skull meningioma cases: IMRT planned with a commercial TPS [Monaco (Elekta AB, Sweden)]; single field uniform dose (SFUD) spot scanning PT produced with an in-house TPS (PSI-plan); and SFUD spot scanning PT plan created with a commercial TPS [XiO (Elekta AB, Sweden)]. A tool for evaluating robustness to random setup errors was created and, for each plan, both a dosimetric evaluation and a robustness analysis to setup errors were performed. Results: It was possible to create clinically acceptable treatment plans for spot scanning proton therapy of meningioma with a commercially available TPS. However, since each treatment planning system uses different methods, this comparison showed different dosimetric results as well as different sensitivities to setup uncertainties. The results confirmed the necessity of an analysis tool for assessing plan robustness to provide a fair comparison of photon and proton plans. Conclusions: Robustness analysis is a critical part of plan evaluation when comparing IMRT plans with spot scanned proton therapy plans. C 2014 American Association of Physicists in Medicine. [ Key words: proton therapy, IMRT, setup errors, robustness, conformity Med. Phys. 41 (11), November /2014/41(11)/111710/11/$ Am. Assoc. Phys. Med

3 Harding et al.: Benchmarking of proton therapy treatment planning system INTRODUCTION Both proton therapy (PT) 1,2 and photon radiation therapy 3,4 have been shown to be safe and effective treatments for meningioma. A prospective observational study of megavoltage photon intensity modulated radiation therapy (IMRT) for treatment of meningioma causing visual impairment demonstrated objective improvements in vision for 40% of patients. 3 The treatment dose prescribed was 50.4 Gy in 28 fractions and the study reported that 7 out of 30 cases experienced toxicity, 5 of which were grade I dry eye. For spot scanning proton therapy, a retrospective study 1 analyzed treatment of all grades of meningioma receiving a median dose of 56 GyE. In this study, 5 yr survival was 100% and grade 3 late toxicity was observed in 3 (13%) patients with benign meningioma. One of these had a very large tumor and suffered grade 3 brain necrosis/edema, two further patients with optic apparatus tumors suffered from grade 4 optic neuropathy. A review of radiation dose volume effects of optic nerve and chiasm by Mayo et al. 5 cited a strong link between dose to the optic structures and late toxicity, particularly for doses above 60 Gy of photons. Moreover, it indicated that dose tolerances for optical structures treated with protons were consistent with photons. Miralbell et al. 6 compared spot scanning PT with IMRT for meningioma and showed similar PTV coverage with lower doses to organs at risk (OARs) for protons, in particular to lacrimal glands and lenses, where protons were predicted to give lower risk of cataracts. Thus, it seems likely that dose tolerances to the OARs are similar for protons (GyE) and photons (Gy). Although there are no comparative clinical trials of photons with protons for skull base meningiomas, OAR sparing can be better with protons. Such sparing of, for example, temporal lobes could potentially lead to improved neurocognitive outcomes. Quality of life could also be improved if doses to other organs at risk such as the lacrimal glands are lower. A treatment modality allowing improved OAR sparing is advantageous for dose escalation studies for atypical and malignant meningioma. A potential risk in using proton therapy is that proton plans can be much more sensitive to uncertainties. 7 The precision of a spot scanned PT strongly depends on the spatial alignments of the patient, accuracy of the position of the Bragg peak, and range uncertainty. 8 Range uncertainties represent a crucial source of errors, as they will propagate through the whole treatment course. 7,8 In order to provide a comprehensive plan comparison between spot scanning PT and IMRT, a robustness analysis can be performed to quantify and understand the sensitivity of the treatment techniques to uncertainties. In this study, only robustness to setup uncertainties has been investigated as a first step. Robustness to range uncertainties is also important for proton plans but will be considered separately and reported in future planned work. A method for robustness analysis was proposed by Albertini et al. 7 whereby setup errors were simulated by recalculating the planned dose distribution in a number of spatially shifted versions of a patient s CT. Error bars reflecting the dose variations for each voxel were calculated. This method is implemented in the PSI-plan TPS, which is an in-house developed TPS which is not commercially available. To be able to perform such analysis for IMRT and PT plans from other TPSs, a new tool is required. The aim of this study was to benchmark a commercially available treatment planning system against an in-house developed TPS; PSI-plan. St. James s Institute of Oncology (SJIO) is currently evaluating the use of the XiO proton TPS (Elekta AB, Sweden) for spot scanning planning comparisons with IMRT. To ensure clinically acceptable PT plans were produced, guidance and peer review was provided by physicists from The Centre for Proton Therapy (CPT) at the Paul Scherrer Institute (PSI), as they have the greatest worldwide technical and clinical experience with spot scanning protons. 11,12 Thus, XiO plans were benchmarked against PSI-plans for dosimetric and conformity measures and robustness to setup errors. Part of this aim was to create a tool for robustness analysis, based on the method proposed by Albertini et al., 7 that could be usable in any clinic which does not have access to robustness analysis tools as a part of their TPS; to validate the tool; and to demonstrate its applicability on three meningioma cases. This enabled a comparison between IMRT and spot scanned proton therapy to be performed both dosimetrically and with regard to robustness to setup uncertainties. There are only limited planning comparisons in the literature which include robustness analysis of proton plans and IMRT plans in the same study. 9,10 2. MATERIALS AND METHODS 2.A. Patient selection Three patients with base of skull grade I meningioma were selected for the retrospective planning study. The patients had previously been imaged using MRI (Magnetom Symphony 1.5 T, Siemens AG, Erlangen, Germany) and CT (Somatom Sensation Open 40 CT Simulator, Siemens AG, Erlangen, Germany). The MRI and CT datasets were fused on the XiO version 4.4 TPS (Elekta AB, Sweden) to enable soft tissue delineation. The CTV and the OARs [brainstem, spinal canal, spinal cord planning risk volume (PRV), optic nerves, optic chiasm, retina - eye globe, lenses, left and right temporal lobes, lacrimal glands, and the cochlea] were contoured on the MRI. The CTV PTV margins were 5 mm for both modalities based on clinical experience at PSI for proton plans and clinical experience at Leeds for 3DCRT of photon plans. These margins were not determined by a calculation of setup errors or other relevant uncertainties such as range errors for proton plans. The volume of the PTVs was cm 3 (patient 1), cm 3 (patient 2), and cm 3 (patient 3). In all cases, a dose of 50.4 Gy (photons) or GyE (protons) (here using GyE for both photons and protons for simplicity) in 28 fractions was prescribed to the PTV based on the SJIO IMRT clinical protocol for benign meningioma. 3 The dose limits to the PTV (as a percentage of prescription dose) and the dose constraints for OARs (in GyE) are listed in Table I in brackets in the far left column.

4 Harding et al.: Benchmarking of proton therapy treatment planning system TABLE I. Dose volume evaluation of OARs and PTV for IMRT plan (Monaco), SFUD plan (XiO), and SFUD plan (PSI-plan): Dose constraints for each VOI are in brackets (in GyE for OARs and as percentage of prescription dose for PTVs). (BS: brainstem; SC: spinal canal; ON: optic nerve; OC: optic chiasm; Ret: retina (N.B. globe contoured); TL: temporal lobe; Coch: cochlea; Lach: lacrimal gland). Patient 1 Patient 2 Patient 3 IMRT XiO PSI-P IMRT XiO PSI-P IMRT XiO PSI-P Percentage of prescription dose (%) D98% ( 90) D95% ( 95) D5% ( 105) D2% ( 107) Maximum dose to organ at risk in GyE BS (55) SC (48) ON L (55) ON R (55) OC (55) Ret L (45) Ret R (45) Lens L (6) Lens R (6) TL L (30) TL R (30) Coch L (45) Coch R (45) Lach R (30) Lach L (30) B. IMRT treatment plans IMRT treatments for each patient were planned retrospectively by a trainee clinical scientist (R.H.) under guidance from experienced clinical scientists and clinicians, using the Monaco TPS (Elekta AB, Sweden). The IMRT plans comprised 5 coplanar 6 MV beams calculated with a Monte Carlo algorithm. The plans were optimized by adjusting the beam angles and optimization parameters in the prescription, particularly on the tissue surrounding the PTV (belonging to the patient contour), to produce the most conformal plan. 2.C. PT treatment plans The proton therapy plans were produced in two treatment planning systems: an in-house developed PSI-plan, 11,13 which is used clinically in PSI; a commercial TPS, XiO version 4.63 and 4.64 (Elekta AB, Sweden), which has a module for calculation of spot scanning proton plans. All PSI-plan treatment plans were planned by a trainee clinical scientist (R.H.) following the PSI protocol and under guidance from an experienced clinical scientist (PT). XiO plans were also produced by R.H., aiming to follow as closely as possible the PSI-plans in order to try to benchmark the two against each other, but without direct guidance. The PSI-plan dose calculation uses a modified ray casting algorithm, 14 which has been verified experimentally 15 as well as against Monte Carlo calculations. 16 Dose optimization is gradient based. 17 The relationship between Hounsfield units (HU) and stopping power (SP) was obtained by a stoichiometric calibration. 18 For all three patients, a calculation dose grid size of mm was used. This grid size provided sufficient accuracy and allowed acceptable dose calculation times. The XiO TPS uses a pencil beam dose calculation model 19 and dose optimization using a conjugate gradient algorithm (a gradient descent algorithm). 20 The HU calibration in XiO first converts HU into the mass density, from which energy-dependent stopping power is calculated. 21 In order to achieve the same calibration curve as in PSI-plan, the values of mass density that correspond to the same stopping power as the PSI stoichiometric calibration were calculated. The dose grid used in XiO was mm, which was as close as possible to the PSI grid, XiO being limited to symmetric dose grids. In both TPSs, a beam model for PSI Gantry 1 12 was defined and used for the plans to be compared. Nuclear interaction was not taken into account during treatment planning in PSI-plan. 12 In order to have the same conditions in XiO, nuclear interaction and subspot precision were switched off. There are two techniques for producing treatment plans for spot scanning PT: single field uniform dose (SFUD) plans and intensity modulated proton therapy (IMPT) plans. 7,22 For SFUD, every beam is optimized independently so that each one produces a uniform dose across the target. This is more robust than IMPT, 7 for which all beams are optimized together so that, potentially, each beam could produce a nonuniform dose, whilst the total dose in the absence of, e.g., setup errors, could be uniform across the target. In this study, only

5 Harding et al.: Benchmarking of proton therapy treatment planning system SFUD plans were considered since the focus of this study was that robustness to setup errors can be improved by the PTV approach. 7 SFUD plans with their relatively uniform dose patterns enable the sensitivity of a plan to setup errors to be evaluated because the optimization is simpler and the response of the plan to setup errors is less complex. All proton therapy plans were based on the PSI clinical experience. For each patient, one lateral beam plus two noncoplanar beam angles (nominally sup, ant beams) were selected, in order to give the most homogeneous dose. The lateral beam always had a lower weighting of 2/3 of the weighting of the other beams (3/3) in order to spare the cochlea (for 50.4 GyE, PTV prescribed doses of 12.6 and GyE, respectively). Identical beam angles were used in XiO as in PSI-plan. In order to achieve the same conformality in both TPS, expanded PTV structures were required for XiO. These structures were effectively rinds around the PTV as the voxels inside the PTV were excluded from them. They were set as OARs and empirically determined constraints, which were the same for each patient, were applied. This enabled some control of spot placement outside the PTV. Five different rind structures were used: Rind 0: directly around PTV (Fig. 1) Spacer rind 0.5 mm wide with no constraints to maximize PTV coverage. Rind 1 : (PTV cm to PTV + 1 cm) (Fig. 1) Max dose as prescribed dose per field to PTV (PDptv-f). Rind 2: (PTV + 1 cm to PTV cm) (Fig. 1) Max dose to Rind 2 as PDptv-f Second constraint of 80% of PDptv-f to 50% volume of Rind 2. For Rind 3: (PTV cm to PTV + 2 cm) (Fig. 1) The max dose to Rind 3 was set as 80% of PDptv-f Second constraint of 55% of PDptv-f to 50% of the volume of Rind 3. For Rind 4: (PTV + 2 cm to PTV cm) (Fig. 1) A max dose was set of 55% of PDptv-f 30% of PDptv-f was limited to 50% of Rind 4. Rinds were not necessary in PSI-plan because a virtual collimation is automatically applied to put spots only up to the distance of 5 mm from the PTV. 23 This corresponds to placing a layer of spots on the PTV boundary and a further single layer of spots 5 mm from the PTV. 2.D. Evaluation of the plans All three plans (Monaco IMRT, PSI-plan SFUD, XiO SFUD) were compared. For each plan, several dosimetric parameters were evaluated as described in Table I. In addition, robustness analysis to setup errors was performed and two conformity indices were calculated: conformity number (CN) 24,25 and conformity index COIN. 25,26 CN quantifies only conformity of dose to the PTV [Eq. (1)] whereas COIN quantifies both conformity of dose to the PTV and OARs sparing [Eq. (2)]. CN = TV RI TV TV RI V RI, (1) where TV RI is a target volume covered by reference isodose, TV is a target volume, and V RI is the volume of a reference isodose. COIN = CN N (OAR) i=1 1 V OARref,i, (2) V OAR,i where N(OAR) is the number of organs at risk selected, CN is the conformity number, V OARref,i represents the OAR volume receiving at least the reference dose, and V OAR,i represents the organ at risk volume. The conformity of the plans to the PTV was calculated for reference isodoses of 95% and 80% of prescribed dose (PD) for CN and 80% of PD for COIN. The reference isodose of 95% was chosen because the PTV should be covered by 95%. The reference isodose of 80% is not directly linked to a coverage goal for the plans but does indicate a level of dose which should be reduced as much as possible to reduce toxicity, e.g., to parallel organs at risk. The values of CN and COIN for all plans were evaluated in CERR (Ref. 27) software in order to eliminate uncertainties caused by different TPSs. All DVHs were also calculated in CERR software. 2.E. Analysis of robustness to setup errors FIG. 1. Scheme of the rind structures used in XiO proton around PTV. A tool for analysis of robustness to setup errors was created based on the method proposed by Albertini, 7 who recommended performing an error analysis of the worst case error distributions. In that work, 14 worst case shifts of the nominal plan were simulated, and for each of the shifted plans, the dose distribution was recalculated. For Albertini s method, for all 15 dose distributions (nominal plan + 14 shifted plans), the maximum dose and the minimum dose are subtracted for each voxel to produce an error bar on the dose. For this tool, all worst case dose distributions are kept and DVHs calculated. Values from the DVHs are evaluated to assess whether the plan is robust to worst case setup errors.

6 Harding et al.: Benchmarking of proton therapy treatment planning system This tool therefore allows the plan to be quantitatively assessed for robustness to worst case setup errors. To derive shifts that have to be simulated here, two studies have been considered. Bolsi et al. 28 showed that the mean systematic setup error for patients fixated with a head vacuum bite block is below 0.6 mm after correction on a daily basis following an internal (PSI) positioning protocol. Due to their small magnitude, these shifts can be considered as negligible. Albertini et al. 7 estimated that for the same internal positioning protocol, combined random daily setup errors are 2.2 mm. This daily setup error combined in quadrature over all directions, multiplied by a factor of 2.16 (Ref. 29) for a 3D distribution of setup errors at 80% confidence interval, results in a worst case random daily setup error of 4.8 mm. Albertini 7 used a smaller factor of nearly 1.5 times the combined setup error, which gives a confidence interval of 85% for a 1D error distribution. A confidence level of 80% seems more reasonable as a worst case for random setup errors, since recommendations 30 give a confidence interval of 90% for worst case systematic setup errors, whilst the effects of the latter on the dose distribution are expected to be very much more severe. The value of 80% is therefore an estimate of a sensible, probably conservative, value suitable for random daily setup errors and in the light of experience the value may be carefully amended. Based on this analysis, all treatment plans for three studied patients were shifted in 14 different directions (±x, 0, 0), (0, ±y, 0) (0, 0, ±z), i.e., 6 directions and (±x, ±y, ±z), i.e., an additional 8 directions by shifting the isocenter by 4.8 mm. The dose for all shifted plans was then recalculated in the corresponding TPS. The DVHs of the CTVs were calculated for all 14 shifted plans for each patient and the near minimum dose D 98 to the CTV was calculated from each shifted DVH. To assess the robustness of planned doses to organs at risk to worst case setup errors, the maximum dose to the brainstem was calculated in CERR for each of the 14 shifted plans associated with each nominal plan for each patient. 3. RESULTS 3.A. Dosimetric results The dosimetric evaluation of all three treatment plans for all three patients is shown in Table I. The PTV coverage fulfilled the prescription criteria (in brackets far left column Table I), except XiO SFUD plan for patient 2 and the PSI SFUD for patient 3. Both fail the D95% by 1%. The maximum doses to the OARs (Table I) mostly met the tolerances (in brackets far left column Table I), but there were differences among the patients and treatment plans. Note that it is considered desirable (rather than critical) that the temporal lobes, the cochlea, and the lachrymal glands meet their tolerances. In the cases where the dose to a critical OAR exceeded the tolerance, a discussion followed to decide whether it was possible to meet that dose constraint without sacrificing plan quality. All treatment plans created are therefore considered as the best reasonably achievable plans. In particular, for patient 1 the critical organs at risk above the tolerances are the left lens and the left retina. These exceeded the tolerances for all three planning modalities and a decision was made to accept these plans as the best achievable plans. There were slight differences dosimetrically between both proton SFUD plans for all three patients, which can be seen in Figs. 2 and 3 and in Table I. No statistical analysis has been performed to show whether this is significant. It appears that these differences can be accounted for by the different proton spot placements in these proton spot scanning plans. This aspect, and how this is believed to impact on plan robustness, is discussed in Sec. 4. Note that Fig. 2 shows the nominal plans, i.e., the plans before any setup error is applied, that is, the dose the patient would receive if the correction for setup errors was perfect. Looking at patient 1 [Fig. 2(a)], the dose to the temporal lobes was best spared with IMRT; however, the maximum dose constraint (30 GyE) could not be met (right temporal lobe IMRT Dmax = 52 GyE) because both temporal lobes overlap with the PTV. In contrast, the right lacrimal gland received a much lower dose for both SFUD plans in XiO (Dmax = 2.6 GyE) and in PSI-plan (Dmax = 2.5 GyE) compared to IMRT (Dmax = 17.7 GyE) (Table I). Figure 2(b) shows an example of the dose distribution for patient 2 of Monaco IMRT, PSI SFUD and XiO SFUD. From the Fig. 2, it is apparent that the IMRT plan has a much greater low dose bath to normal tissue compared to protons, as expected, because of the sharp distal fall off for the proton beams. Figure 3 displays a comparison of the DVHs for the same patient. PSI-plan SFUD gives similar sparing of OARs compared to XiO SFUD. In more detail, the maximum dose to the left temporal lobe was much lower with SFUD for both PSI-plan and XiO (Dmax = 39.7 GyE, 37.4 GyE, respectively) compared to the IMRT plan (Dmax = 47.2 GyE) without sacrificing PTV coverage (Table I). Likewise, the left cochlea received a much lower dose for both SFUD plans (6.6 GyE XiO, 14.4 GyE PSI-P) with respect to the IMRT plan (29.5 GyE) (Table I). The same was true for patient 3; e.g., the dose to the left temporal lobe was much lower for both SFUD plans (18.2 GyE XiO, 11.0 GyE PSI-P) compared to the IMRT plan (33.1 GyE) [Table I, Fig. 2(c)]. 3.B. Conformity The dosimetric results from the DVHs are reflected in the CN and COIN values (Table II). The difference in COIN at 80% of prescription dose (PD) between SFUD and IMRT corresponds to the dose values of OARs (Figs. 2 and 3). The evaluated OARs were always the same for all three plans of each patient but different for different patients (patient 1: right optic nerve, brainstem, left globe, left lacrimal gland, and right cochlea; patient 2: brainstem, right cochlea, and left temporal lobe; patient 3: right optic nerve, brainstem, chiasm, and spinal canal). That means that COIN in our study serves as a relative intrapatient measure only (while CN only considers PTV coverage, not OARs). For patient 1, the PSI-plan SFUD plan had slightly higher COIN and CN values than the XiO SFUD plan and the IMRT plan, which might indicate better overall OAR sparing and

7 Harding et al.: Benchmarking of proton therapy treatment planning system F. 2. Dose distribution of the nominal plans (i.e., without setup errors applied) for patient 1 (a), patient 2 (b), and patient 3 (c). improved conformity to the PTV. For patient 2, the higher values of CN at 80% PD and 95% PD for both the PSI-plan and XiO SFUD plans as compared to the IMRT plan showed an improved conformity of the proton therapy (PT) plans; however, the values for PSI-plan were better than the values for XiO. For patient 3, PSI-plan SFUD plan had the highest values of CN at 80% PD and 95% PD compared to the XiO SFUD plan and IMRT plan which had similar values. Looking only at the SFUD plans, the values of COIN at 80% PD for PSI-plan are much higher than for the values XiO for patient 2. However, this calculation was sensitive to the coverage of the right cochlea as it is an organ with a very small volume. For all three patients, the values of COIN were consistently higher for PSI-plan compared to XiO although larger errors in COIN need to be considered. 3.C. Robustness to setup errors To evaluate robustness according to worst case random setup errors, a DVH of the CTV was calculated for each of the 14 shifted, recalculated, plans on each patient. For each patient and each type of plan, the near minimum dose D98% to the CTV was calculated from each of the 14 DVHs from each shifted plan and the 14 values plotted as a box plot [Fig. 4 (Ref. 31)]. The median value of D98% to the CTV over 14 plans shifted by worst case setup errors for all three patients and all three treatment modalities were calculated. Their values were similar and all above 95% of the prescription dose (Fig. 4). The differences were seen in the minimum value of D98% to the CTV, i.e., the lowest value of D98% out of the 14 of the worst case shifts evaluated, which is the worst (least robust) shift direction. This is shown in the box plot as the lower whisker value [Fig. 4(a)]. Figure 4(a) shows that, for patient 1, the minimum value of D98% to the CTV for all three plans was 95% PD. For patient 2, for the SFUD XiO plan, the lower whisker shows that the minimum value of D98 to the CTV for one direction was only approximately 90% PD, whereas for the IMRT plan and SFUD PSI-plan, the minimum D98% to the CTV was over 95% PD. For patient 2, the least robust direction for the SFUD XiO plan, which is represented by the lower whisker on the box plot (Fig. 4), corresponds to the negative z direction which is a shift toward the posterior of the patient. In this posterior shift direction, the plan is less robust as it shows a worse CTV coverage. The dose distribution of the shifted, recalculated plan was compromised near the edge of the target and in a region containing inhomogeneities (air/bone interface). For patient 3, the box plot shows [Fig. 4(a)] that in certain directions, the XiO SFUD plan is also less robust to setup errors than the IMRT plan and the PSI SFUD plan. For the XiO SFUD plans on patient 3, over 25% of all D98% values to the CTV, for the 14 plans shifted by worst case setup errors, were below 95% PD. For the XiO SFUD plan, the lower whisker shows that the minimum value of D98% to the CTV was only 90% PD compared to 95% PD for the IMRT plan

8 Harding et al.: Benchmarking of proton therapy treatment planning system TABLE II. Conformity indices: COIN at the reference isodose of 80% of PD (prescription dose) and CN (conformity number) for the reference isodose of 80% PD and 95% PD. ±40%, ±1% are the estimated percentage errors on the values on COIN and CN, respectively, see text P1 = patient 1, P2 = patient 2, P3 = patient 3. COIN D80% ± 40% CN D80% ± 1% CN D95% ± 1% P1 SFUD XiO P1 SFUD PSI P1 IMRT Monaco P2 SFUD XiO P2 SFUD PSI P2 IMRT Monaco P3 SFUD XiO P3 SFUD PSI P3 IMRT Monaco to worse case setup errors, the three SFUD plans created in PSI-plans are robust. Two of these three particular SFUD plans created in XiO appear slightly more sensitive to worst case setup errors although no conclusions can be drawn about whether they are acceptably robust to setup errors from this conservative, worst case analysis. Figure 4(b) shows the box plot of the maximum doses to the brainstem for each of the 14 shifted plans for all three patients for each TPS. IMRT plans have a higher maximum brainstem dose compared to SFUD plans, with the XiO SFUD plans having the lowest brainstem maximum doses for the shifted plans. For all three patients, the PSI SFUD plan was similar or more robust to setup errors than the IMRT plan according to boxplots Fig. 4(a) for the CTV and Fig. 4(b) for the brainstem. These show that, for PSI-plan SFUD plans, the near minimum dose to the CTV is higher or approximately similar to that of the IMRT plans for all shift directions for all three patients [Fig. 4(a)]. Figure 4(b) shows that the maximum dose to the brainstem is similar to or lower for PSI-SFUD plans compared to IMRT plans for all shift directions for all three patients. Note that the maximum brainstem doses for PSI-plan plans may be artificially reduced due to larger voxel sizes. FIG. 3. DVHs of the right cochlea (top), the left temporal lobe (middle) and brainstem (bottom) for patient 2. and PSI SFUD plan. The lower whisker corresponds to the least robust shift direction, which is a shift in the left lateral direction. Again the CTV coverage is compromised near the edge of the target and near an air/bone interface. Our robustness analysis, which is expected to be conservative as a worst case analysis, nevertheless shows that even 4. DISCUSSION Performing a treatment plan comparison between different treatment planning modalities is not a trivial task. For estimation of the relative quality of photon treatment and proton treatment, dosimetric (DVH) evaluation alone is not sufficient. To understand the advantages of one method over the other, additional analysis, such as visual comparison of 2D dose distribution, conformity index calculation, and robustness assessment, should be performed. All these methods give a more detailed indication of the plan quality and help to define further possible sources of uncertainties and their possible influence on treatment outcome. In this study, two different methods for treatment of skull base meningioma have been evaluated: photon irradiation with IMRT and proton irradiation with the SFUD technique.

9 Harding et al.: Benchmarking of proton therapy treatment planning system FIG. 4. (a) A box plot of the near minimum dose (D98) to the CTV as a fraction of PD (Prescription Dose) to patient 1 (left), patient 2 (center) and patient 3 (right). Each box represents 14 worst case shifts. (b) A box plot of minimum dose Dmin in Gy (evaluated in CERR) to the brainstem for patient 1 (left), patient 2 (center), and patient 3 (right). Each box represents 14 worst case shifts. For SFUD, a commercially available TPS, XiO was benchmarked against an in-house developed TPS PSI-plan. Both institutions, SJIO and PSI, are currently evaluating XiO for the use of proton therapy planning. PSI has already treated many patients (e.g., Ref. 1) planned in PSI-plan. Therefore, plans deemed clinically acceptable in PSI-plan were considered as a gold standard in this study. Considering the dosimetric comparison, almost all plans met dose constraints to the PTV and maximum dose limits to OARs. When the dose coverage was not sufficient or the dose limit of an organ was exceeded, a discussion followed to ensure that the best reasonably achievable plan had been created. The SFUD plans were comparable to or better than the IMRT plans. The improvement of SFUD plans was mainly visible in better OARs sparing, in the values of COIN at 80% of the prescription dose, and in the DVH (Fig. 3), while the target coverage and conformity number at 80% and 95% of the prescription dose were similar. The better OAR sparing is due to higher conformity of proton SFUD plans with respect to IMRT plans, particularly at lower doses caused by sharper distal fall off and lower entrance dose (e.g., Ref. 32). Better sparing of the temporal lobes was observed for patient 1 for the IMRT plan with respect to both SFUD plans. For this patient, both temporal lobes were included in the IMRT prescription whilst they were not included in the SFUD plan prescriptions, in order to keep the latter as simple as possible. Moreover, beam angles for IMRT were specifically selected to avoid the temporal lobes. Rather than using five equally spaced coplanar beams, the beam angles were modified for each plan to avoid the contralateral temporal lobe. The same is true for IMRT of patient 2 where the left contralateral temporal lobe sparing was achieved by optimization and optimal selection of beam angles. For proton plans, the contralateral side of the brain was almost completely spared particularly due to the distal fall-off of the proton beam and the benefit of using noncoplanar, superior, beam angles. Noncoplanar angles were avoided for IMRT to reduce exit dose to the normal head and neck tissue. For patients 1 and 2, both SFUD plans were superior to IMRT. For patient 3, XiO SFUD plan resulted in a worse value of CN at 80% PD than IMRT. There was a consistent difference between XiO and PSI-plan plans where the latter had superior conformity according to values of CN and COIN for each of the three patients. However, it is not straight forward to directly identify the features of a plan that contribute to reduced conformity index values. 26,33 They are very sensitive to different spatial dose distributions. As both TPSs use different dose optimization algorithms, differences in these values are to be expected. Each optimization algorithm fulfills the constraints but results in different spatial distributions and different sizes of hot and cold spots. Therefore, the indices in this study should be understood only as a relative parameter to estimate the differences between the plans. They should not be used as a single parameter to evaluate the quality of the plan. It is also important to emphasize that in XiO, additional rinds around the PTV were necessary to improve the conformity of the dose to the PTV. The lower value of conformity indices may be caused by insufficient definition of the rinds. Since the spot distribution to deliver dose to the target is very degenerate in XiO, it is likely that without the right starting conditions, i.e., virtual collimation, SFUD plans will not result

10 Harding et al.: Benchmarking of proton therapy treatment planning system in the most conformal dose distribution. Although all conformity indices were calculated in CERR (Ref. 27) to limit uncertainties due to different segmentation algorithms in each TPS, 34 the voxel size differs in each TPS and therefore interpolation results in small differences within each structure set. The difference in voxel sizes remains in CERR with PSI-plan having larger voxel sizes. This can be a problem especially for very small volumes such as the cochleae since the error between the volumes in the different plans was estimated to be up to 40% for the right cochlea. Also the maximum dose is averaged over a larger volume for larger voxel sizes which may result in a slightly lower maximum dose value compared to the result for a smaller voxel size. This treatment planning study shows how it is possible to implement worst case robustness analysis to setup uncertainties in the planning process with commercial treatment planning systems without robustness calculations included [XiO proton and Monaco (IMRT)]. This is a trivial but rather time consuming task in practice. The robustness analysis method proposed by Albertini 7 for evaluation of the effects of treatment setup errors was followed. For proton treatments, robustness analysis should also include the effect of systematic range errors; these were out of the scope of this study because of their complexity and the necessity to investigate the uncertainty of the HU calibration curves. Future work is planned to address systematic range errors in these plans. In order to exclude the range uncertainty caused by different HU calibration curves in different planning systems, the curve obtained by stoichiometric calibration in PSI-plan was mimicked in XiO. Three reasons can be immediately identified from published literature as to why setup errors could compromise CTV coverage in spot scanning proton plans. The first is very simple; the PTV margin is not big enough compared to setup errors. 29 This is the case for a photon plan if the margin is not large enough for the setup errors and can also happen with proton plans. In these cases, the PTV seems large enough, given the photon plan has adequate CTV coverage with the applied setup errors. Second, the setup shifts bring different densities into the beam path. 7,35 The third reason identified for spot scanning proton plans is that a simple positioning error alone, due to a setup error, can cause a dose inhomogeneity. 35 This does not occur for SFUD plans since each individual beam is optimized to be uniform over the PTV. For all three patients, each of the PSI-plan SFUD plans were robust to the worst case random setup errors, maintaining the D98% value above 95% of the PD. However, the robustness analysis showed that for two out of three of the demonstration cases, XiO SFUD plans were less robust to worst case random setup errors than PSI-plan SFUD plans and IMRT plans. The difference in robustness may again be attributed to the method of controlling spot distribution outside the PTV. It is expected that, for SFUD, where using a PTV has been shown to improve robustness, 7 a uniform margin of spots around the PTV, i.e., effectively increasing the PTV margin would increase robustness. Nevertheless, the rind method was observed not to result in a uniform margin of spots around the PTV and therefore it results in a greater sensitivity to certain directions of worst case shift. Note that, as found by Albertini et al., 7 for SFUD plans, the areas of the patient associated with a worse robustness to worst case setup errors are around the edges of the CTV and not within the center of the CTV as is the case for IMPT plans. Thus for SFUD plans, the spot distribution around the edge of the PTV is important. However, for IMPT plans, which have dose inhomogeneity within the CTV in the presence of setup errors, 7 a uniform margin of spots around the PTV resulting in an effectively larger PTV margin may not improve robustness. The difference in brainstem maximum doses for the worst case setup errors between IMRT and proton plans [Fig. 4(b)] may be due to differences in penumbra gradient around the PTV between nominal IMRT and proton plans or may be due to different beam arrangements and/or nominal dose distributions between IMRT and proton plans. IMRT plans may have steeper dose gradients around the PTV due to the way they are optimized to spare organs at risk (in this case, the temporal lobes) and conform tightly to the PTV. On the other hand, SFUD plans are not optimized to give a tight dose distribution around the PTV, with the beam angles selected to give the most homogeneous dose to the PTV for that beam. The conformity is controlled rather by the spot distribution and spot size, which results in a less steep penumbra in the case of the PSI gantry 1 machine. The difference in brainstem maximum doses seen for patient 2 between XiO and PSI SFUD plans may be due to differences in spot distribution which may result in an effectively smaller margin of spots around the PTV for the XiO SFUD plan. Note that the maximum doses for PSI-plan plans may be artificially reduced due to larger voxel sizes. IMRT plans were robust to conservative worstcase random setup errors in all cases whilst, for the patient positioning protocol considered, systematic errors were negligible. This shows that, for this patient positioning protocol, the PTV margin is probably sufficient to protect against random setup errors, since our robustness analysis is expected to be conservative. However, a full analysis should be performed 29 including the CTV delineation error, which is beyond the scope of this paper. Clinical IMRT setup errors for the patient positioning protocol in Leeds from an unpublished audit were found to be larger than for the PSI positioning protocol considered here. Thus, cone beam CT with the continued use of rigid shells is considered desirable, as is a further audit, to find out whether the PTV margin of 5 mm is sufficient in practice. It is interesting to note that the two least robust XiO plans, corresponding to the whiskers on the box plots in the case of patients 2 and 3, were shift directions in the direction of air/bone interfaces. It is perhaps to be expected that the plan should be more sensitive to setup errors in these shift directions. 35 The observed difference in robustness to setup errors between XiO and PSI-plan plans is associated with different densities moving into the beam s path where the XiO plans are not as robust to this. N.B. the same beam angles in XiO and PSI-plans were chosen to ensure the same changes in density heterogeneities moving with respect to the beams when a particular setup error is applied. It was observed that the XiO plans were found to be less robust in a shift direction

11 Harding et al.: Benchmarking of proton therapy treatment planning system with a density heterogeneity. Although the PTV is identical in both plans, the spot distribution is different. For example, the PSI-plan has a uniform margin of spots outside the PTV which is not the case for the XiO plan; resulting in an effective uniform expansion of the PTV for PSI-plan plans. This points to the conclusion that XiO plans may possibly be made more robust with a larger PTV margin; however, this would be expected to worsen the conformity compared to PSI-plan plans. Further work could be done optimizing rind structures to try to mitigate this issue. For the three cases studied, PSI-plan SFUD plans had an improved nominal plan quality compared to IMRT as shown by higher values of conformity. They also had a similar or better robustness to IMRT plans to worst case setup errors for both the CTV coverage and the brainstem maximum dose. The comparison of SFUD plans with IMRT plans in terms of robustness has not previously been reported and it confirms that in the case of these three SFUD PSI-plan plans, the PTV margin of 5 mm is sufficient to produce a proton plan robust to daily setup errors at PSI as suggested in the work of Albertini. 7 Note that the margin recipe from van Herk et al. 27 cannot be applied to proton therapy due, for example, to the sensitivity of protons to density heterogeneities. 7 Nevertheless, a PTV margin for delineation errors should be included when treating patients with protons as with IMRT. It is also essential to ensure that the PTV margin is sufficient to protect against range errors; investigating robustness of these plans to range uncertainty is planned as part of future work where IMPT plans will be evaluated as well. The shallower penumbra for proton spot scanning plans compared to IMRT may result in an effectively larger margin around the CTV and this may be the explanation for SFUD PSI-plans having better robustness to setup errors compared to IMRT plans in patients 1 and 2. The results in this study on three demonstration cases are not sufficient to fully establish proton planning in XiO; further work is necessary. It is shown here that a robustness analysis is necessary to establish planning strategies with a new proton spot scanning planning system. Further work should include robustness to range error. Some of the apparent differences in XiO proton may be accounted for by lack of planner experience. Other methods of planning SFUD treatments (other than these rind constraints) in XiO may result in better robustness whilst still conforming the treatment to the PTV. Such studies are recommended to be performed in XiO and other commercial planning systems on larger patient numbers. It is emphasized that it is very useful to use such a tool in order to gain experience in robust planning in XiO proton and similar systems. 5. CONCLUSIONS SFUD PT plans created in both TPSs appear to offer good alternatives to IMRT treatment of skull base meningioma although the quality of XiO PT plans in terms of conformity and robustness could be improved by finding a better way to localize the scanned proton spots to the PTV. A robustness analysis method for random setup errors was successfully implemented and tested. It emphasized that assessing robustness of radiotherapy plans to setup uncertainties is an important part of evaluation of competing treatment techniques. However, proton plans cannot be considered as fully robust without analysis of range uncertainties. In future work, evaluation of the robustness of IMPT plans should be considered as well as evaluation of range uncertainty. Moreover, it appears necessary to improve the robustness analysis method in order to better reflect the actual clinical uncertainties. It would be beneficial if vendors of commercial planning systems such as XiO proton include automated robustness analysis. This would enable studies such as this to be performed routinely allowing professional planners to get further experience of planning robust spot scanning proton plans. ACKNOWLEDGMENTS The work in Leeds was funded by the Leeds Teaching Hospitals NHS Trust Charitable Trustees. ESTRO provided a Technology Transfer Grant to allow one of the authors (R.H.) to visit the PSI. Both are acknowledged with gratitude. The helpful support of Francesca Albertini is also acknowledged with thanks. Grateful acknowledgment also goes to Elekta (Elekta AB, Sweden) for providing XiO software to SJIO, Leeds. a) Author to whom correspondence should be addressed. Electronic mail: ruth.harding2@wales.nhs.uk 1 D. C. Weber, R. Schneider, G. Goitein, T. Koch, C. Ares, J. H. Geismar, A. Schertler, A. Bolsi, and E. B. Hug, Spot scanning-based proton therapy for intracranial meningioma: Long-term results from the Paul Scherrer Institute, Int. J. Radiat. Oncol., Biol., Phys. 83, (2012). 2 J. D. Slater, L. N. Loredo, A. Chung, D. A. Bush, B. Patyal, W. D. Johnson, F. P. K. Hsu, and J. M. Slater, Fractionated proton radiotherapy for benign cavernous sinus meningiomas, Int. J. Radiat. Oncol., Biol., Phys. 83(5), e633 e637 (2012). 3 J. Maclean, N. Fersht, F. Bremner, C. Stacey, S. Sivabalasingham, and S. Short, Meningioma causing visual impairment: Outcomes and toxicity after intensity modulated radiation therapy, Int. J. Radiat. Oncol., Biol., Phys. 85, e179 e186 (2013). 4 S. E. Combs, S. Adeberg, J. O. Dittmar, T. Welzel, S. Rieken, D. Habermehl, P. E. Huber, and J. Debus, Skull base meningiomas: Long-term results and patient self-reported outcome in 507 patients treated with fractionated stereotactic radiotherapy (FSRT) or intensity modulated radiotherapy (IMRT), Radiother. Oncol. 106(2), (2013). 5 C. Mayo, M. K. Martel, L. B. Marks, J. Flickinger, J. Nam, and J. Kirkpatrick, Radiation dose volume effects of optic nerves and chiasm, Int. J. Radiat. Oncol., Biol., Phys. 76, S28 S35 (2010). 6 R. Miralbell, L. Cella, D. Weber, and A. Lomax, Optimizing radiotherapy of orbital and paraorbital tumors: Intensity-modulated X-ray beams vs. intensity-modulated proton beams, Int. J. Radiat. Oncol., Biol., Phys. 47, (2000). 7 F. Albertini, E. B. Hug, and A. J. Lomax, Is it necessary to plan with safety margins for actively scanned proton therapy?, Phys. Med. Biol. 56, (2011). 8 A. J. Lomax, Intensity modulated proton therapy and its sensitivity to treatment uncertainties: 1. The potential effects of calculational uncertainties, Phys. Med. Biol. 53, (2008). 9 M. Stuschke, A. Kaiser, C. Pöttgen, W. Lübcke, and J. Farr, Potentials of robust intensity modulated scanning proton plans for locally advanced lung cancer in comparison to intensity modulated photon plans, Radiother. Oncol. 104, (2012). 10 P. Park, J. Cheung, Y. Chen, X. Zhu, N. Sahoo, R. Mohan, and L. Dong, TH- A-BRA-09: Statistical assessment of plan robustness under uncertainties: IMRT vs. proton therapy, Med. Phys. 39, 3985 (2012).