Section 3: Economic evaluation PERSPECTIVES OF AN EVALUATOR DR BONNY PARKINSON SENIOR RESEARCH FELLOW MACQUARIE UNIVERSITY CENTRE FOR THE HEALTH ECONOMY (MUCHE)
Disclaimer The views presented are my own and do not represent those of other evaluation groups, the Department of Health, or the Pharmaceutical Benefits Advisory Committee (PBAC) and its subcommittees. I will not discuss any specific drugs or submissions, and will not disclose any committee discussions or discussions with the Department.
My perspective MAKING AN INCORRECT FUNDING DECISION SOCIETY S WELFARE If a drug is effective and cost-effective and not funded patients may be unable to access treatment ($$$) denied valuable health benefits If a treatment is not effective or not cost-effective but is funded resources may be diverted from other health programs other patients may be denied valuable health benefits 3
Key questions I ask myself 1) Is the evidence presented accurate? 2) Is any relevant evidence missing? 3) What is the quality of the evidence? 4) Is the evidence applicable to the Australian population? 5) Is there any risk of bias in the evidence, and what is the direction of that bias? 4/22
Is the evidence presented accurate? We need to verify that all inputs in the submission/model match the sources. Too many un-verifiable model inputs ESC/PBAC s confidence in the model. Base case is often re-specified if a mistake is identified. Too many mistakes scrutiny of the model. 5
Is the evidence presented accurate? SUGGESTIONS Example spreadsheet from a project I am currently working on (not a PBAC submission). Comments used liberally to explain things in the model Yellow = hard entered data Green = assumptions Blue = solved Different greys = different formulas Data sources (including table numbers or page numbers!) 6/22
Is the evidence presented accurate? SUGGESTIONS Example folder from a project I am currently working on (not a PBAC submission). Utilities in one place. Reference names match that in model. 7/22
Is the evidence presented accurate? SUGGESTIONS Example file from a project I am currently working on (not a PBAC submission). Inputs highlighted in yellow in PDFs OFFICE I FACULTY I DEPARTMENT 8
Is the evidence presented accurate? SUGGESTIONS Avoid hard entered numbers in models with no documentation of where numbers come from or how they were calculated. If there are any calculations made to the inputs before putting them in the model, please provide them. We would have to try to replicate them ourselves, which we might get wrong. OFFICE I FACULTY I DEPARTMENT 9
Is any relevant evidence missing? LITERATURE REVIEW What is the point of the literature review (Section 3.2.1)? Quick check whether there are other economic evaluations in Australia (unlikely). Whether the model structure and inputs differ from other published models, and if so why ( applicability or ICER?) Differences highlight potential missing evidence 10
Is any relevant evidence missing? EXTRAPOLATION Guidelines are clear regarding extrapolation: Check proportional hazards (NEED: log-cumulative hazard plots) Fit (at minimum): exponential, Weibull, log-normal, log-logistic, gamma, Gompertz (NEED: Stata/SAS/R code and output) Assess goodness of fit using visual inspection and AIC/BIC (NEED: Figures with functions overlapping Kaplan-Meier curves and Stata/SAS/R results) Bold = often not provided. Have to ask sponsors ask Global cause delays uncertainty. 11
What is the quality of the evidence? Use critical thinking to assess each model input is the data source appropriate (Section 3.2.2) and are any adjustments to the input data appropriate (Section 3.4). Inputs that are (almost) automatically accepted: Data sources previously accepted by the PBAC. Suggestion: Share your experiences with each other. Database of accepted sources: Whenever a new source is considered acceptable, report it so others can use it. Database of models: Have a database of accepted models that others can use (or use as a template). Utilities measured during the clinical trial. Don t have to worry about disutilities associated with adverse events Don t reference other cost-effectiveness models for utilities or costs. Reference the original source. Utilities or costs in the original source may be made up! 12
What is the quality of the evidence? ERRORS IN MODELS ARE VERY COMMON Get someone with fresh eyes to look at it before submission. Excel: Use Names Easier to avoid errors and makes long formulas understandable. Use sheet layouts that are as similar as possible. Easier to compare interventions. Make the tables in the results sheet look exactly like what is in the submission. Avoids mistakes when copying into submissions. Avoid hidden spreadsheets. It looks pretty, but actually results in a harder to understand model. Macros are useful, but too many are a pain and increase uncertainty. TreeAge: Use clones For each parameter include the source in the descriptions column. Both: Use graphs to check for sudden changes/sensible trends in the inputs/outputs (why model validation section asks for traces). Sense check the model results (especially in the control group). 13/22
Is the evidence applicable to the Australian population? Use your judgement. Sometimes there is a trade off between quality of evidence and Australian evidence. 14
Is there any risk of bias in the evidence, and what is the direction of that bias? Partial sensitivity analyses We need to run any missed. Use switches (IF S_Util = 1, 0.9, 0.8) to conduct sensitivity analyses. Diagrams and graphs! Tornado diagrams of the sensitivity analysis results (with the base case and range reported for each variable) Graph of ICER versus time horizon or other parameters if really uncertain. 15
Questions? The views presented are my own and do not represent those of other evaluation groups, the Department of Health, or the Pharmaceutical Benefits Advisory Committee (PBAC) and its subcommittees. I will not discuss any specific drugs or submissions, and will not disclose any committee discussions or discussions with the Department.