A CASE STUDY OF VALUE OF INFORMATION

A CASE STUDY OF VALUE OF INFORMATION, Research Fellow

1/19 Background The ISPOR good practices for performance-based risk-sharing arrangements task force recommends using value of information analysis when considering a coverage with evidence development (CED) arrangement (1). A risk-sharing arrangement that links reimbursement (e.g. PBS subsidy) to prospective data collection (e.g. the results of additional research). Aim: Demonstrate the analyses required to inform whether a drug is suitable for a CED arrangement.

Trastuzumab (Herceptin) for the treatment of HER2+ metastatic breast cancer 2/19 15% patients are HER2+ HER2+ associated with poorer prognosis breakthrough drug Expensive: $55,000 for 12 months PBAC rejected 3 submissions trastuzumab for HER2+ MBC in 2000-01. Unacceptably high ICER Government established Herceptin Program in December 2001 Rich observational dataset: Captured treatment of almost all patients Linked to MBS and PBS claims data

3/19 Methods: economic model Efficacy based on trial H0648g Trastuzumab + paclitaxel (n=92) vs paclitaxel alone (n=96) Only trial available in 2001 Digitised Kaplan-Meier estimates of overall survival Paclitaxel alone Trastuzumab + paclitaxel N 96 92 Odds ratio or HRR (95% CI) Response, n (%) 16 (16.7) 38 (41.3) 3.52 (1.79, 6.94)# Stable disease, n (%) NR NR NR Time to progression, median weeks (95% CI) 13.04 (9.13, 18.70) 30.00 (23.05, 43.05) 0.38 (0.27, 0.53) Overall survival, median weeks (95% CI) 80.01 (55.22, 106.10) 96.10 (73.48, 133.49) 0.80 (0.56, 1.11) Reasonably certain of a positive impact on health, but unsure of magnitude due to: cross-over of patients in the paclitaxel alone arm, and applicability to patients in clinical practice

4/19 Methods: economic model Markov model built in TreeAge based on best-available evidence in 2001 1 week cycle length and 10 year time horizon Considers only weekly trastuzumab, 150ml vial, and IHC testing. Utilities based on Hauser et al (2001) Disutilities for 20 adverse events Costs reported in 2001$A Drugs, administration, specialist visits, cardiac monitoring, blood tests HER2 testing decision tree Markov model

Results: deterministic and univariate sensitivity analysis 5/19 Treatment COST ($) Life years QALYs ICERs Cost/LYS ($) Cost/QALY($) Paclitaxel only available 16,731 1.94 1.24 Trastuzumab available 28,706 2.04 1.30 123,107 181,275 ICER is likely to be over-estimated due to cross-over. Most sensitive to: Baseline probability and odds ratio of an adverse event Baseline TTP and HRR of progression Various resource use parameters OR of AEs (lower 95%CI) Probability of AEs (95%CI) Duration of TTP with chemotherapy (95%CI) HRR of TTP (95%CIs) OR of response with trastuzumab (95%CIs) Probability of response with chemotherapy (95%CI) HRR of OS (95%CIs) Duration of OS with chemotherapy (95%C) Disutility AEs (95%CIs) Utility response (95%CIs) Utility stable disease (95%CIs) Utility progressive disease (95%CIs) OR of discontinuation (95%CIs) Probability of discontinuation (95%CI) Dose reduction (95%CIs) Frequency of cardiac testing, ECHO/MUGA and ECG (95%CIs) Frequency of CT scans (95%CIs) Probability outpatient (95%CIs) Trastuzumab cost -10% (0% in base case) Probability public given inpatient (95%CIs) Duration of hospitalisation for AEs (95%CIs) Duration of treatment with chemotherapy (upper 95%CI) Cost of palliative care (95%CIs) Cost of AE treatment in hospital (95%CIs) Cost of treatment administration in hospital (95%CIs) Probability trastuzumab weekly (95%CIs) Proportion using CVAD (95%CIs) Frequency of specialist visits(95%cis) Cost of treatment with chemotherapy post progression (95%CIs) Duration of treatment with chemotherapy post progression (95%CIs) Proportion using ECHO vs MUGA (95%CIs) Frequency of pathology tests (95%CIs) Discount rate = 3% and 7% (5% in base case) Time horizon +/-10% (520 in base case) Probability test results in true positive (95%CIs) Probability test results in true negative (95%CIs) HER2+ prevalence (95%CIs) Dominated $0 $50 $100 $150 $200 $250 ICER ($/QALY) Thousands

6/19 Results: threshold analysis In order for the deterministic ICER to be $80,000/QALY: a 81% price discount, or a 24 week dose cap, or a $8,000 per patient fixed price.

7/19 Methods: Probabilistic sensitivity analysis Monte Carlo simulation methods Type of Parameter Probability of response, stable disease or none of these Other probabilities (e.g. probability of being HER2+, probability of experiencing an adverse event) Odds ratios* and hazard rate ratios Duration and resource use variables (e.g. time between specialists visits) Utilities Disutilities Distribution Dirichlet Beta Log-normal Gamma 1 gamma -gamma

8/19 Methods: Probabilistic sensitivity analysis Repeated simulations until average ICER stable visually. 250,000 Average ICER ($/QALY) 200,000 150,000 100,000 50,000 0 0 1,000 2,000 3,000 4,000 Simulations No price discount With 81% price discount With 24 week dose cap With $8,000 per patient fixed price

9/19 Results: Probabilistic sensitivity analysis Only 0.10% chance trastuzumab is cost-effective at a threshold of $80,000/QALY at the current price. BUT with a price discount or cost-sharing arrangement there was a 35% to 48% chance trastuzumab is cost-effective. Incremental costs ($) 50,000 40,000 30,000 20,000 10,000 0-0.20-0.10 0.00 0.10 0.20 0.30 0.40 0.50 QALYs gained Probability Cost-effective 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 0 50,000 100,000 150,000 200,000 250,000 300,000 Willingness to pay ($/QALY) Chemotherapy only available, no price discount or dose cap With 81% price discount With 24 week dose cap With $8,000 per patient fixed price Trastuzumab available, no price discount or dose cap With 81% price discount With 24 week dose cap With $8,000 per patient fixed price

10/19 Proposed CED arrangement Assume that the manufacturer proposed that trastuzumab given interim funding: Subject to collecting observational data regarding the health outcomes and resource use in clinical practice. Parameters that could be captured using an observational dataset linked to PBS and MBS data Proportion of patients treated as outpatients versus inpatients Proportion of inpatients in public hospitals versus private hospitals Duration of treatment (weeks) with trastuzumab Duration of treatment (weeks) with paclitaxel Proportion of patients administered chemotherapy using a CVAD (vs IV) Average duration of time between subsequent specialist visits Average duration of time between pathology tests: FBC, EUC, LFT and calcium Average duration of time between CT scans of the chest/abdomen/pelvis Average duration of time between ECGs Average duration of time between ECHO or MUGA scans Proportion of patients whose cardiac function is tested using ECHO (vs MUGA) Duration of treatment (weeks) with chemotherapies following disease progression Cost of treatment with chemotherapies following disease progression Duration of TTP with chemotherapy alone* Parameters that could be captured if the observational dataset was also linked to hospital admission data Cost of chemotherapy administration in hospital, public inpatients Cost of chemotherapy administration in hospital, private inpatients Cost of hospitalised adverse events per day, public inpatients Cost of hospitalised adverse events per day, private inpatients Duration of hospitalisation for adverse events * The direct use of Kaplan- Meier curves means that sensitivity of the results to the baseline hazard rate of death was unable to be captured. Consequently the value of collecting the observational data with respect to overall survival is underestimated.

11/19 Results: Best-case scenario analysis Methods: Changed value of each parameter to upper or lower 95% confidence intervals, depending on which one resulted in a lower ICER. Treatment arm Cost ($) Incremental cost ($) QALYs QALYs gained ICER ($/QALY) No price discount Paclitaxel only available 19,093 1.23 Trastuzumab available 25,307 6,214 1.29 0.05 113,508 81% price discount Paclitaxel only available 19,093 1.23 Trastuzumab available 20,720 1,627 1.29 0.05 29,710 24 week dose cap Paclitaxel only available 18,647 1.24 Trastuzumab available 22,804 4,157 1.31 0.07 56,629 $8,000 fixed price Paclitaxel only available 18,647 1.24 Trastuzumab available 21,336 2,688 1.31 0.07 36,624 It is unlikely that the additional research will change the funding decision unless a price discount or cost-sharing arrangement applied.

12/19 Methods: Value of information analysis VOI analysis encompasses: Expected Value of Perfect Information (EVPI) Expected Value of Perfect Parameter Information (EVPPI) and Expected Value of Sample Information (EVSI) The EVPI assumes that the research will resolve uncertainty in every parameter. BUT the research proposed collects data on only a sub-set of parameters. EVSI is often the most appropriate measure of the value of research Most research is based on a sample of patients and results in uncertain point estimates (e.g. RCTs or surveys based on a sample of the population). Special case: the proposed research will capture perfect information on 100% of patients treated. Thus the EVSI = EVPPI.

13/19 Methods: EVPPI Two potential approaches: Normal distribution approach Monte Carlo simulation approach Used Monte Carlo simulation approach because: Collecting data on a sub-set of parameters Incremental net benefit was highly skewed 0.1200 Simulations 0.1000 0.0800 0.0600 0.0400 No price discount With 81% price discount With 24 week dose cap 0.0200 0.0000-20,000-15,000-10,000-5,000 0 5,000 Incremental Net Benefit (based on $80,000/QALY) With $8,000 per patient fixed price

Methods: EVPPI Step 1: Bootstrap a set of values for the sub-set of parameters (inner loop) Step 2: While holding the sub-set of parameters fixed, bootstrap the other parameter values (outer loop) Step 3: Calculate expected net benefit and select the treatment with the highest net benefit (optimal choice with perfect information about the sub-set of parameters). Repeat steps 1-3. Brennan et al. (2007) found that 500 inner loop simulations and 100 outer loop simulations (50,000 simulations in total) were capable of estimating the EVPPI reasonably well (2). Step 4: Estimate the expected (mean) net benefit with perfect information about the sub-set of parameters across all the simulations. Step 5: Estimate the EVPPI by subtracting the maximum expected net benefit given the mean parameter values (e.g. expected net benefit with treatment B) 14/19 Inner loop simulation Outer loop simulation Treatment A Treatment B Optimal choice with perfect information about sub-set of parameters Expected NB with perfect information about sub-set of parameters Expected NB with Treatment B Expected QALYs Expected Costs Expected NB Expected QALYs Expected Costs Expected NB 1 1, 2, 100 1.6 6000 9 2.1 6000 12 B 12 12 0 2 1, 2, 100 2.2 12000 12 1.7 2000 10 A 12 10 2 3 1, 2, 100 2.4 4000 14 3.4 4000 20 B 20 20 0 4 1, 2, 100 2 10000 11 1.8 8000 10 A 11 10 1 500 1, 2, 100 2.7 22000 14 2.4 14000 13 A 14 13 1 EE θθii max j NNNN(j, θ) max j E θ NB j, θ Expectation 12 13 Opportunity cost of incorrect decision EVPPI = 13.8 = 11 = 0.8

15/19 Methods: Population EVPPI The population EVPPI (or population EVSI) is estimated by multiplying the EVPPI by the expected number of patients to be affected by the decision and the expected discounted lifetime of the drug. Assumed that: Threshold = $80,000/QALY Interim funding provided (i.e. no opportunity cost of not treating patients involved in research). The maximum life-time of trastuzumab was 13 years, based on the period between 2001, when PBAC first considered trastuzumab for PBS-listing, and 2014, when trastuzumab will lose patent protection; and 464 patients receive trastuzumab per year (3,830 women were treated with trastuzumab from 12/2001 to 3/2010) Applied a discount rate of 5% was applied as per PBAC guidelines.

16/19 Results: Population EVPPI Population EVPPI = $0 Expected given univariate sensitivity analysis and best-case scenario analysis. Population EVPPI = up to $1.5 million if price discount or cost-sharing arrangement applied EVPPI with time taken to conduct the research, if interim funding was not possible. Population EVPPI = up to $2.2 million if linked to hospital administration data. EVPPI ($ per person ) $900 $800 $700 $600 $500 $400 $300 $200 No price discount, observational data + MBS data + PBS data (+/- admitted patient data) 81% price discount, observational data + MBS data + PBS data 81% price discount, observational data + MBS data + PBS data + admitted patient data 24 week dose cap, observational data + MBS data + PBS data Population EVPPI ($) Millions $2.5 $2.0 $1.5 $1.0 $0.5 No price discount, observational data + MBS data + PBS data (+/- admitted patient data) 81% price discount, observational data + MBS data + PBS data 81% price discount,observational data + MBS data + PBS data + admitted patient data 24 week dose cap, observational data + MBS data + PBS data $100 $0 $50,000 $70,000 $90,000 Willingness to pay ($ per QALY) $8,000 per patient fixed price, observational data + MBS data + PBS data $0.0 2 4 6 8 10 12 Time to collect data (years) $8,000 per patient fixed price, observational data + MBS data + PBS data

17/19 Costs A non-negligible EVPPI or EVSI is not a sufficient condition for recommending an CED arrangement. Are the benefits of research > costs? Costs of research: recruiting patients, treating patients in a clinical trial or administering a survey or registry data analysis Magnitude is determined by the parameters of interest and the research design requires a detailed research plan. Include only the costs of research that fall on the public budget (including hospitals, NHMRC etc).

18/19 Conclusions Value of information analysis can be used to support CED arrangements EVSI most likely relevant, but EVPPI in special cases. Note that other analyses can also be informative and should also be undertaken Univariate sensitivity analysis and best-case / worst-case scenario analysis. Need to specify research design Key limitation: EVPPI may be underestimated due to lack of access to patient-level data No uncertainty around Kaplan-Meier curves or coefficients of the fitted parametric functions, and unable to adjust for cross-over.

19/19 Issues encountered TreeAge cannot be used to calculate EVSI Very high computational demands (needed a new 8- core computer) Need to specify uptake Need to specify lifetime of drug

Questions?

Methods: Probabilistic sensitivity analysis Monte Carlo simulation method: Step 1: For each parameter specified parametric distributions Obtain means and standard errors reported in the literature. Can calculate standard errors using number sampled (if probability) or 95% confidence intervals or interquartile ranges OR Survey multiple experts (e.g. 30) to estimate standard errors OR Use non-informative priors where no data is available e.g. Beta(1, 1) Probability Cost ($) 0.14 0.12 0.10 0.08 0.06 0.04 Standard error 0.02 Mean 0.00 0 400 775 1150 1525 Parameter value Probability Baseline probability of response 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.00 0.05 0.10 0.15 0.20 0.25 Parameter value Probability Odds ratio of response 0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.9 1.025 1.15 1.275 1.4 1.525 Parameter value

Methods: Probabilistic sensitivity analysis Step 2: A potential value for each parameter is bootstrapped from each distribution. Probability Cost ($) 0.14 0.12 Each line represents 1 simulated 0.10 parameter value 0.08 0.06 0.04 0.02 0.00 0 400 775 1150 1525 Parameter value Probability Baseline probability of response 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.00 0.05 0.10 0.15 0.20 0.25 Parameter value Probability Odds ratio of response 0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 0.9 1.025 1.15 1.275 1.4 1.525 Parameter value Step 3: Using the new set of values as inputs into the model, the ICER was then recalculated. Repeat Steps 1 to 3 (each time is referred to as a simulation) until the estimated average ICER becomes stable (verified visually).

Methods: EVSI The EVSI is the difference between the expected net benefit following the collection of information about a sub-set of parameter values (θ I ) using a particular research design (D) and the subsequent choice of the treatment (from j alternatives) that maximises the net-benefit, compared to the expected net benefit with the current information (θ I and θ IC ) and the choice of the treatment with the highest probability of being the most cost-effective treatment. EEEEII = EEEEEEEEE NNNN wwww iiiiiiiiiii ffff a pppppppppp sssss dddddd eeeeeeee NNNN oo ccccccc iiiiiiiiiii EEEEII = E D max j EE θi C, θi D NNNN j, θ I, θ I C max j E θ NB j, θ I, θ I C Monte Carlo simulation method: Step 1: Bootstrap a set of true values for the sub-set of parameters (inner loop) Step 2: Draw a sample (D) of values that is likely to arise from research, given the true values for the sub-set of parameters in Step 1 and the research design (e.g. sample size of n). Calculate the estimated values arising from the research design for the sub-set of parameters. Step 3: While holding these values fixed, bootstrap the other parameter values (outer loop) Step 4: Select the treatment with the highest net benefit given (optimal choice with perfect information about the sub-set of parameters). Repeat steps 1-4. Step 5: Estimate the expected (mean) net benefit with sample information about the sub-set of parameters given the study design across all the simulations. Step 6: Estimate the EVSI by subtracting the maximum expected net benefit given the mean parameter values (e.g. expected net benefit with treatment B) from the expected net benefit with sample information about the sub-set of parameters.

References 1. Garrison LP, Jr., Towse A, Briggs A, et al. Performance-based risk-sharing arrangements-good practices for design, implementation, and evaluation: report of the ISPOR good practices for performance-based risksharing arrangements task force. Value Health. 2013; 16: 703-19. 2. Brennan A, Kharroubi S, O'Hagan A, et al. Calculating partial expected value of perfect information via Monte Carlo sampling algorithms. Med Decis Making. 2007; 27: 448-70. 3. Claxton K, Palmer S, Longworth L, et al. Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. Health Technol Assess. 2012; 16: 1-323. 4. Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Med Decis Making. 2004; 24: 207-27. 5. Felli JC, Hazen GB. Sensitivity analysis and the expected value of perfect information. Med Decis Making. 1998; 18: 95-109. 6. Briggs A, Sculpher M, Claxton K. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press, 2006.

Funding and ethics approvals This research forms part of the CHEETAH programme, which is funded by a NHMRC Capacity Building Grant in Health Services Research. The research also forms part of a research program approved by the University of Technology Sydney Research Ethics Committee (UTS HREC REF NO. 2009-143P).