Bridging Trial and Decision: A Checklist to Frame Health Technology Assessments for Resource Allocation Decisions

Transcription

1 VALUE IN HEALTH 14 (2011) available at journal homepage: Bridging Trial and Decision: A Checklist to Frame Health Technology Assessments for Resource Allocation Decisions Janneke P. C. Grutters, PhD 1, *, Shanly C. Seferina, MD 2, Vivianne C. G. Tjan-Heijnen, MD, PhD 2, Roel J. W. van Kampen, MD 2, Wim G. Goettsch, PhD 3, Manuela A. Joore, PhD 4 1 Department of Health Organization, Policy, and Economics, CAPHRI School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands; 2 Division of Medical Oncology, Department of Internal Medicine, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands; 3 Dutch Health Care Insurance Board (CvZ), Diemen, The Netherlands; 4 Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Center, Maastricht, The Netherlands A B S T R A C T Objective: Health technology assessments (HTAs) intend to inform real-world decisions. They often draw on data from explanatory trials and hence are not always applicable to the decision problem. HTAs may therefore not meet the needs of decision makers. Our objective was to develop and apply a checklist to: 1) systematically frame HTAs in a way that they are applicable to the decision problem; and 2) assess if a decision problem can be informed by an available HTA. Methods: We reviewed published literature to identify factors that should be considered when framing HTAs for resource allocation decisions. The checklist was finalized in collaboration with clinicians and policy makers. We applied the checklist to the economic evaluation of trastuzumab in early breast cancer. We defined a reference case and for each study, retrieved through a systematic review, we examined if each factor was explicitly considered. Results: A checklist was developed with 11 factors (e.g., clinical practice, consequences, and patient use). In the case of trastuzumab, most factors were considered by the 11 retrieved economic evaluations. Two factors, being the inclusion of all relevant comparators and professional use, were considered by none of the studies. Conclusions: We developed a comprehensive checklist with 11 factors to frame HTAs and to assess the applicability of HTAs to resource allocation decisions. Economic evaluations on trastuzumab considered some of these factors, but overlooked others. The proposed checklist assists in systematically considering all factors in developing the conceptual model of an HTA, to make HTAs better reflect the decision problem. Keywords: breast cancer, cost-effectiveness, decision making, economic evaluation, evidence-based medicine. Copyright 2011, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc. Introduction Recently, the Centers for Medicare and Medicaid Services (CMS) decided not to reimburse computed tomographic colonography in the United States. The decision makers at CMS emphasized the lack of applicability of the evidence to the decision problem because the mean age of the trial participants was lower than that of the actual patient population [1]. Indeed, more and more technologies are not reimbursed because the health technology assessment (HTA) is not applicable to the decision problem [2]. Or, after a number of years technologies turn out to be wrongfully reimbursed [3]. In this article we describe the development and application of a checklist to systematically frame HTAs for resource allocation decisions in a way that they are applicable to the decision problem. One of the key principles to improve the use of HTA for resource allocation decisions is that the scope of the HTA should be relevant to its use [4,5]. HTA involves synthesis of evidence, and this evidence should be relevant to the decision. When designing an HTA, one of the first steps is the framing of the study, also referred to as the development of a conceptual model [6,7]. It is crucial that the conceptual model actually represents the decision problem; this is referred to as conceptual model validity [8,9]. To avoid withholding patients effective treatment, resource allocation decisions are often made directly after the efficacy of the intervention is shown in a trial [10]. As a result HTAs that are performed to inform these decisions are often based on data collection alongside these trials. It is well known that trial results do not necessarily reflect the real world [11,12]. For instance, patient mix, compliance, and cointerventions are known to differ substantially between trial and daily practice [13]. Earlier research showed that although it has a large influence on effectiveness and cost-effectiveness, only few analyses incorporate suboptimal patient compliance [13,14]. As a result, the relative effectiveness and cost-effectiveness of a new technology in practice can be substantially different from that observed in trials [11,12]. Although the prerequisite of applicability to the decision problem is well known and accepted, in practice the development of the conceptual model of HTAs often seems to be strongly influenced by the availability of data [15,16]. This may result in a low conceptual model validity [17]. * Address correspondence to: Janneke P.C. Grutters, Maastricht University, Department of Health Organization, Policy, and Economics, PO Box 616, 6200 MD Maastricht, The Netherlands. j.grutters@maastrichtuniversity.nl /$36.00 see front matter Copyright 2011, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc. doi: /j.jval

2 778 VALUE IN HEALTH 14 (2011) Handbooks on economic evaluation provide a guidance on how to frame an HTA, but do not pay explicit attention to the fact that trial evidence may not be directly applicable to the decision problem [6,7]. In the critical assessment of an HTA two questions are relevant: 1) Is the methodology appropriate and the results valid; and 2) are the results applicable to the decision problem [10]. Comprehensive checklists exist to answer the first question, assessing the quality of the HTA [8,10,18,19]. No checklists exist to answer the second question, assessing the applicability of the HTA to the decision problem [8]. Although the issue of applicability is acknowledged, it is said to be difficult to evaluate this in any formal way [8]. In this study we aim to provide a helpful tool for this evaluation. Our objective was two-fold. First, to develop a checklist to frame HTAs in a way that they are applicable to the decision problem. The checklist can both guide the design and analysis of an HTA for resource allocation decisions and assess if a specific decision problem can be informed by an existing HTA. It can then function as a guide to which factors should be adapted to make an existing HTA applicable to the decision problem. Second, we illustrate the use of the checklist by assessing if economic evaluations of trastuzumab in early stage breast cancer explicitly consider factors that may hamper the applicability of results to the decision problem. We used this example because of the availability of a number of economic evaluation studies and the availability of data from randomized controlled trials [20]. Methods Framing an HTA When framing an HTA, one needs to be aware of the applicability of the study to the decision problem. This awareness will enhance the conceptual model validity and may increase use of the results for decision making. Two handbooks describe the process of framing an HTA [6,7]. First, the objective, audience, and perspective of the analysis should be defined. Further, the population, intervention, and comparators need to be specified. Also, the time horizon and boundaries of the analysis should be defined, as should the type of analysis. Although sometimes in this process authors implicitly state that one should evaluate the intervention as it will be used in daily practice, this is not made explicit. Checklist development To develop the checklist we used the process of framing an HTA as described above as a starting point. In addition, we reviewed published literature and handbooks to identify factors that can influence the applicability of HTAs for real-world resource allocation decisions. Because we examine if trial evidence is directly applicable to the decision problem, in our review we focused on the generalizability of scientific evidence. The Cochrane library, PubMed, and MEDLINE database were searched until March The search included the following terms: ( Randomized Controlled Trials as Topic [MeSH Terms] OR Clinical Trials as Topic [MeSH Terms]) and ( generalizability OR generalisability OR real world ). Based on the title, abstract, and if necessary the full article, studies were included if they were written in English, had a methodological focus, and provided a list of issues that could affect the applicability of scientific evidence. We also included three frequently used handbooks on general, model-based, and trialbased economic evaluation [10,21,22]. Issues that could influence the applicability of HTAs to the decision problem were identified by two authors (Grutters and Joore) independently and listed per source. Overlapping factors were clustered. Both the clustering of the factors and the development of the final checklist were discussed with clinicians (Tjan-Heijnen, Seferina, and Van Kampen) and policy makers (Goettsch) during consensus meetings. The clinicians were chosen because of their expert knowledge of breast cancer and trastuzumab treatment in particular. The policy maker was chosen based on his longstanding experience with the use of HTAs for resource allocation decisions. Application to the case of trastuzumab Breast cancer is the most common type of cancer in women worldwide [23]. Women with human epidermal growth factor receptor 2 (HER2) positive breast cancer have a poor prognosis after diagnosis if not treated with HER2-targeted therapy [24]. Trastuzumab is a humanized monoclonal antibody that binds to HER2. Recent randomized controlled trials have shown a positive treatment effect of trastuzumab in women with HER2 positive early breast cancer [25 30]. This has resulted in a number of economic evaluations that are all based on these randomized controlled trials [20]. Trastuzumab is therefore deemed a suitable case to examine if the checklist is useful in assessing if efforts were made to make the economic evaluation applicable to the real-world decision problem. We examined if economic evaluation studies on trastuzumab in early breast cancer explicitly considered all factors of the checklist in framing their economic evaluation. For this purpose we first had to define a reference case. In close collaboration with clinicians and decision makers we framed the economic evaluation of trastuzumab in early breast cancer to define this reference case. Next, a systematic literature review was performed to identify original HTAs of trastuzumab in HER2 positive early breast cancer. For illustration purposes we limited the search to economic evaluations. We used the electronic databases Cochrane library, PubMed, and MEDLINE to search for peer-reviewed full articles, published before March The search included the following terms: ( trastuzumab OR herceptin ) and ( breast cancer ) and ( cost-effectiveness OR economic evaluation ). Additional references were searched through manual searches of the reference lists and specialist journals. Studies were included only if they were written in English and focused on early breast cancer. A data extraction tool to extract relevant information from each included study was developed and used by two authors (Grutters, Joore) independently. Both authors completed the data extraction tool for all studies. Discrepancies were solved in consensus meetings with the other authors (Tjan-Heijnen, Seferina, and Van Kampen). Results Results of the literature search The first literature search identified 882 articles. After checking the inclusion criteria five peer-reviewed articles were included: a comprehensive HTA report on generalizability [31] and four other publications [11,12,32,33]. Five handbooks were included [6,7,10,21,22]. Because three handbooks [10,21,22] named issues that were reasons to use modelling instead of trial-based economic evaluation, we additionally included a comprehensive article that concerns the reasons for using model-based economic evaluation [34] (see Table 1 for an overview of the issues we derived from each source). As was mentioned in the Methods section, two handbooks mentioned the objective, audience, and perspective, which can affect the applicability of an HTA [6,7]. Issues concerning patient mix [11,31,32], population [6,7,33,34], patient selection [12], and representative sample [22] were clustered. A fifth issue, listed in almost all sources, concerned the comparators [6,7,10,11,21,22,31 34]. Issues regarding the intervention [6,7], clinical practice [11,22,33], and differences between trial protocol and routine practice [12,32,34] all relate to clinical practice, and were therefore clustered. The time horizon of the analysis was mentioned in all but one of the sources [6,7,10-12,21,22,32 34]. The issues analysis

3 Table 1 Results of the literature review on issues that should be considered when framing an HTA: identified issues in each source. Gold (6) Fox-Rushby (7) Sculpher (31) Revicki (11) Rothwell (12) Baltussen (32) O Brien (34) Garrison (33) Drummond (10) Briggs (21) Glick (22) 1. Objective Objective 2. Audience Audience 3. Perspective Perspective 4. Population Population Patient mix Patient mix Patient selection Patient mix Patient population 5. Comparators Comparators Clinical practice (availability, comparators) Comparators 6. Intervention Intervention Clinical practice Differences between trial protocol and routine practice Artificial design/ interactions with other diseases Study population Representative sample Comparators Comparators Comparators Comparators Comparators Protocol-driven costs & outcomes 7. Time horizon Horizon Time horizon 8. Analysis type Endpoints Outcome measures External effects Measurement of outcomes Clinical practice Clinical practice/ settting Time horizon Time horizon Time horizon Time horizon Time horizon Time horizon Time horizon Types of outcomes Endpoints Outcome measure Boundaries Sample size Sample size 9. Compliance Compliance Compliance Compliance 10. Skills/experience Physician characteristics Belief/incentives Education/training Learning effects 11. Price/resource use Resource use Setting of the trial Health service characteristics Selected provider population Price VALUE IN HEALTH 14 (2011)

4 780 VALUE IN HEALTH 14 (2011) type [6], endpoints [10,11], and outcomes [12,22,32 34] were clustered as they all relate to consequences. Other clusters of issues were boundaries [6], sample size [22,34], and compliance [11,31 33]. In the cluster of issues related to physician characteristics [32] or the provider population [34] we included skills/experience [31], beliefs [31], and training or learning effects [31,32]. Finally, issues regarding price/resource use [11,31] were clustered with issues on setting [12] and health service characteristics [32]. Checklist development Each cluster of issues was converted into one factor in the checklist. Boundaries of the HTA can concern consequences (e.g., ripple effects) or patient population (which patients are considered in the analysis?). Sample size is also related to the consequences cluster. Therefore, boundaries and sample size are already covered in the factors consequences and patient population. The clusters regarding compliance and physician characteristics were defined more broadly as patient use and professional use, respectively. This resulted in a final checklist with eleven factors: objective, audience, perspective, population, comparators, clinical practice, time horizon, consequences, patient use, professional use, and price and resource use (Table 2). Table 2 Checklist to frame HTAs to the decision problem. Factor Question 1. Objective How will the HTA be used? (e.g., contribute to evidence, inform adoption decision) 2. Audience What is the audience (principal users) for the HTA? (e.g., government, pharmaceutical companies, insurance companies, patient groups, jurisdiction) 3. Perspective Which viewpoint or perspective is relevant for the HTA? (e.g., societal, health care, insurer, payer) 4. Population What is the patient population relevant for the decision problem? (e.g., age, health status, sex, other characteristics) 5. Comparators What are relevant comparators for the decision problem? (e.g., care as usual, alternative technologies) 6. Clinical practice How are the technologies embedded in clinical practice? (e.g., diagnostics, clinical instead of research protocol) 7. Time horizon Which time horizon is relevant for the decision problem? (e.g., lifetime, one year) 8. Consequences Which consequences are relevant for the decision problem? (e.g., final versus intermediate outcomes, indirect and/or rare consequences) 9. Patient use What is the patient use that is relevant for the decision problem? (e.g., uptake, compliance, adherence) 10. Professional use What is the use of the technology by health care professionals that is relevant for the decision problem? (e.g., skills, experience, beliefs) 11. Price and resource use What price level and resource use are relevant for the decision problem? (e.g., personnel providing the intervention) Besides the objective, audience, and perspective of the HTA [6,7], the patient population may also affect the applicability of an HTA. In defining the population it is important to be aware that, as trials often apply strict in- and exclusion criteria, the population in a trial may differ from the population that is relevant to the decision problem [11,12,22,31 35]. Trials often compare a new intervention to care as usual or placebo. As a result, comparative treatments relevant to the decision problem may well be different from the direct comparisons available in the scientific evidence [10,11,21,22,31,33,34]. It is therefore important to consider all relevant comparators in an HTA. Clinical practice may well differ from the restricted trial setting (11,12,22,31 34]. For example, an intervention may have been doctor-led in the trial, whereas it is nurse-led in practice. Or, different diagnostics may be used than in the trial. Therefore, it is important that the interventions in the HTA reflect how they will be embedded in clinical practice [6,7]. Variations in clinical practice patterns may not only influence the intervention, but may also affect the presentation and outcomes of a disease, and in turn can affect the effectiveness and/or costeffectiveness of a treatment. Also, trials often use relatively short time horizons that do not reflect the time horizon of the decision problem, which is generally longer. Time horizon is therefore a factor that may impact the applicability of an HTA [10-12,21,22,32 34]. Consequences may not be measured in a way that they are applicable to the decision problem. This may be caused by intermediate endpoints or limited sample sizes that do not represent the actual decision problem [10 12,22,32 34]. For example, only survival may be incorporated in an HTA, whereas adverse events that reduce quality of life are very important and therefore the decision problem is focused on maximizing quality adjusted life years (QALYs) [15]. Or, productivity losses due to absenteeism may occur that were not measured in the trial. It is also important to be aware of potential ripple effects that may occur, such as parents who experience productivity losses because of an ill child. Additional factors that may impact the applicability of an HTA are patient use, health care professional use, and price and resource use. Patient use may be influenced by compliance or uptake, and may be different in a trial than in daily practice [11,31 33]. Factors that may influence professional use include skills, experience, belief, and incentives [31,32,34]. A useful division can be made here between whether and how a health care professional uses a technology. For example, a general practitioner may not prescribe a certain drug because he does not believe in its effectiveness. Or, a surgeon may perform a new surgery differently than intended, which may hamper the effectiveness of the surgery, or may even cause malpractice. Finally, price and resource use change over time and may be different when a treatment is implemented than it was during the trial [11,12,22,31,32]. For example, the price of a diagnostic procedure may vary across settings and over time. Framing the economic evaluation of trastuzumab in early breast cancer The reference case for the economic evaluation of trastuzumab in early breast cancer is listed in Table 3. Various objectives and audiences exist for the economic evaluation of trastuzumab in early breast cancer. For example, the study can be performed to assist decision makers in making a reimbursement decision, or it can be performed for researchers to add to the existing knowledge. The perspective of the analysis depends on this audience and objective. The population consists of women with HER2 positive early breast cancer. The population in the trastuzumab trials is the result of very strict eligibility criteria, which will probably be less strictly applied in the real world [26 28]. This could for example lead to patients with a lower left ventricular ejection fraction receiving trastuzumab, possibly resulting in more cardiac adverse events and presumably higher incremental cost effectiveness ratio. The population could also be of a different age. The exact age

5 VALUE IN HEALTH 14 (2011) Table 3 Consideration of factors in the checklist by economic evaluations of trastuzumab in early breast cancer*. Essers (47) Vlaenderen (46) Shiroiwa (45) Norum (44) Neyt (43) Millar (42) Lidgren (41) Liberato (40) Kurian (39) Garrison (38) Reference case Dedes (37) 1. Objective Any????? 2. Audience Any 3. Perspective Any 4. Population Context specific 5. Comparators CAU and all trastuzumab regimens 6. Clinical practice HER2 diagnosis Breast cancer care 7. Time horizon Lifetime? 8. Consequences Cardiac adverse events Costs QALYs 9. Patient use Similar to observed in trial 10. Professional use Start and stopping rules 11. Price and resource use Context specific CAU, Care as usual; HER2, Human epidermal growth factor receptor 2; QALYs, quality adjusted life years. * In this table,? unknown; corresponds to reference case; does not correspond to reference case. The trastuzumab regimens are the HERA regimen (1 year, every 3 weeks, after chemotherapy); the NSABP B31/NCCTG N9831 regimen (1 year, weekly, concurrent with chemotherapy); and the FinHer regimen (9 weeks, weekly, concurrent with chemotherapy). and left ventricular ejection fraction of the patient population depends on the decision context. With regard to the comparators, three different strategies of trastuzumab treatment are evaluated in the existing trials. The first regimen, referred to as the HERA regimen, consists of one year trastuzumab, every 3 weeks after chemotherapy [25,27,29]. The second regimen, referred to as the NSABP B31/NCCTG N9831 regimen, consists of 1 year of trastuzumab weekly, concurrent with chemotherapy [28]. The third regimen, called FinHer, consists of nine weekly trastuzumab infusions, concurrent with chemotherapy [26,30]. These three regimens as well as care as usual are all relevant comparators. As for clinical practice, trastuzumab is only effective in patients with a HER2-positive tumor, which is tested in the trials by immunohistochemistry and, if indicated (HER2 ), also fluorescence in situ hybridization tests. There is a large variety worldwide, however, which test is used in clinical practice and how test results are interpreted [36]. This testing strategy should therefore be considered in framing the economic evaluation. Additionally, breast cancer care is very context-specific, and should therefore be considered in the framing process. The relevant time horizon for the analysis is lifetime. Because beyond 5 years no data on trastuzumab in early breast cancer are currently available from trials, the results need to be extrapolated to assess long-term cost-effectiveness. The consequences that should be considered are adverse events, costs and QALYs. Because compliance or adherence is not expected to be relevant for this problem, patient use is expected to be as it was in the trials. Professional use on the other hand is expected to be relevant. In the trials strict criteria are applied regarding when to stop trastuzumab treatment if a patient shows cardiac reactions. It is possible that professionals use the treatment differently and stop trastuzumab sooner or later than outlined by stopping rules. Price and resource used are context specific and should be adapted to the relevant context. Economic evaluations of trastuzumab in early breast cancer The second literature search resulted in 87 full articles on the economic evaluation of trastuzumab in early breast cancer. A total of 11 publications met the inclusion criteria [37 47]. All studies used decision-analytic modeling. The complete checklist for all studies is listed in Table 3. In most of the studies the objective, audience and perspective were specified. Five out of 11 studies used the population that was included in the trials for their economic evaluation [37 39,44,47]. The other six studies did incorporate the fact that the population is context specific, through varying patient age [40 43,46] or weight [45]. No study investigated the impact of treating patients with a worse or better left ventricular ejection fraction than included in the trials, or incorporated the effect of patient age on cardiac problems. Liberato et al. [40] mentioned in their discussion that they, due to lack of data, could not fully consider the patient s case mix. No study examined all relevant comparators. Seven studies compared one trastuzumab regimen to care as usual [38,39,41,44 47]. Two studies combined data from different trastuzumab regimens into one intervention that was compared to care as usual [42,44]. More than one trastuzumab regimen was considered by four studies [37,40,42,43]. In these studies the different trastuzumab regimens were all compared to care as usual, which means that they did not calculate the cost-effectiveness of one strategy to the next best strategy. With regard to clinical practice, all studies defined breast cancer care as it is organized in their country. Only two studies explicitly considered the HER2 test strategy [38,41]. All studies extrapolated the trial data. The time horizon was lifetime in eight studies [38,41 47], two studies analyzed costs and effects over 15 years [37,40]. One study used a not specified long-term time horizon [39].

6 782 VALUE IN HEALTH 14 (2011) Adverse events and costs were incorporated as consequences by all studies. Eight studies used QALYs as the outcome measure of interest [38 42,44,46,47]. The other three studies only examined life years [37,43,45]. All studies incorporated patient use as it was observed in the trials. Professional use that may result in different start and stopping rules were not considered by any of the studies. Contextspecific price and resource use were incorporated by all studies. Discussion We reviewed the literature and developed a checklist containing 11 factors that assists in framing HTA studies in such a way that they are applicable to the decision problem. Additionally, the checklist can be used to determine the applicability of HTAs to resource allocation decisions, and to adapt existing HTAs to make them applicable to the decision problem. We illustrated the use of the checklist by applying it to the case of trastuzumab in early breast cancer. The illustration of the checklist by applying it to the case of trastuzumab in early breast cancer indicated that the checklist is feasible. We have no reason to assume that the checklist is casespecific and that it would not be feasible in other disease areas or different types of interventions. Further use in other case studies is necessary to confirm the feasibility and practical value of the checklist. In the case of trastuzumab it was found that some factors were explicitly considered by all or a majority of studies (such as the time horizon), whereas other factors were not considered by any study (e.g., health care professional use and comparators) or only few (e.g., clinical practice and population). The latter also applies to work by our own group [47]. To improve the applicability of HTAs for decision making, it is important that all considerations are explicitly stated, even when factors are not deemed relevant. The checklist can then act as a guide to which factors should be adapted in an HTA to make it applicable to the decision problem. Although considering factors like time horizon (through extrapolation) and patient population (through subgroup analyses) may be common practice, other factors tend to be overlooked more often. The factors that are more commonly considered are also the factors that are listed in handbooks or national pharmacoeconomic guidelines (e.g., [4,21,22,48], and are well-known principles of good economic evaluation. Factors that are not generally accepted principles of good economic evaluation seemed more likely to be overlooked. However, although it is recommended to include all relevant comparators in an economic evaluation and compare one strategy with the next best strategy [10,21], none of the economic evaluations of trastuzumab considered all trastuzumab regimens that are described in the literature. A reason why some factors are not explicitly considered may be the absence of data. In our opinion, the absence of data is no reason to not incorporate a relevant parameter into the analysis [49]. The checklist can both guide the design and analysis of an HTA for resource allocation decisions, and assess if existing HTAs are applicable to the decision problem. The purpose of the checklist will depend on the stage of the HTA. The first purpose of guiding the design, or framing, of an HTA, is especially relevant in the stage of developing the conceptual model [6]. For both trial-based and model-based HTAs the use of the checklist aims to improve the conceptual model validity by providing a systematic tool to consider all factors and adapt the conceptual model to the decision problem. Of course, an HTA is only as accurate as its inputs, and for factors like professional use no evidence may be available. Torrance et al. [6] introduced what-if analyses, which can be used when data that are applicable to the decision context are lacking but a policy decision is still needed. In what-if analyses one should anticipate on the fact that it may not be cost-effective to use (that is: adopt and implement) a technology, for example because of the low professional use. A possible solution for this is decision analytical modeling, as it allows for the synthesis of evidence from different sources, and the exploration of the influence of deviations between trial and practice on cost-effectiveness results [50]. When evidence on these factors is absent, experts could be consulted and best/worst scenarios could be examined. Coverage with evidence development schemes are also useful to collect these types of evidence, because patients in these schemes are being monitored in the real world [51]. Additionally, the checklist could assist in designing pragmatic trials for which the hypothesis and study design are formulated based on information needed to make a decision [52]. This would assist in targeting research to the areas where we need more evidence, which may result in the collection of evidence that is more suited to the decision we need to make. The second purpose of the checklist, assessing if a specific decision problem can be informed by an existing HTA, is relevant in a later stage, when an HTA is readily available. In this stage it can for example help decision makers assess if the HTA is useful for their decision problem. We did not intend, however, to develop a checklist in a sense that boxes can be ticked to calculate a score of exactly how applicable an HTA is. Rather, we intended to provide a systematic tool that can act as a guide to which factors in an HTA should be adapted to make it more applicable to the decision problem. Whether an HTA that, based on the checklist, was found to be not fully applicable will lead to poor decisions is highly case-specific. It should be assessed if adapting the HTA to make it more applicable, significantly affects the policy recommendation. In the critical assessment of an HTA two questions are relevant: 1) is the methodology is appropriate and the results are valid; and 2) are the results applicable to the decision problem [10]. Comprehensive checklists exist to answer the first question, assessing the quality of the HTA [8,10,18,19]. To date no checklist existed to answer the second question, assessing the applicability of the HTA to the decision problem [8]. Our checklist aims to answer this second question. It should be emphasized that our checklist should not be used instead of, but complementary to the quality checklists. Although related, the problem of HTAs not being applicable to the decision problem is conceptually different from HTAs not being generalizable or transferable across clinical settings and jurisdictions [31,53,54]. Discount rate for example is important in assessing transferability, as it differs across jurisdictions, whereas the discount rate is not important when assessing the applicability of an HTA to the decision problem. On the other hand the relevant time horizon of an analysis will not differ across jurisdictions, and is therefore not listed as a factor in these checklists [31,54]. However, time horizon is an important factor when examining if an HTA is applicable to the decision problem. Also, adverse events are relevant in all jurisdictions and therefore not a factor that is considered with regard to transferability. In a trial, however, adverse events may not have been measured although they are relevant for the allocation decision. In examining the applicability of HTAs to the decision problem it is therefore not only important if the numbers are correct, but also if all relevant information is included. Additionally, some of the factors in the proposed checklist resemble part of a set of morally relevant questions with respect to assessing health technologies [55]. For example, one of these questions is what the morally relevant consequences of the technology are. If these morally relevant consequences are relevant to the decision problem, they should indeed be addressed when framing the HTA. When using the proposed checklist, they should come up at the consequences factor. The checklist and the set of questions have a very different purpose, and therefore do not substitute but complement each other. In our application of the checklist we focused on economic evaluations. We believe, however, that the checklist is useful for HTAs in general, considering both effectiveness and cost-effectiveness. The checklist provides a tool to systematically frame an HTA in a way that

7 VALUE IN HEALTH 14 (2011) it is applicable for the decision problem, and to formally evaluate the applicability of HTAs to the decision problem, which is currently lacking [8]. Although the practical value of the checklist needs to be revealed through future use, use of the presented checklist may improve the applicability of HTAs to decision problems. Making HTAs more applicable to the decision problem hopefully increases the impact of HTA on policy decision making and on practice [56 58], and may eventually lead to better policy decisions. Acknowledgment The authors thank six anonymous reviewers for their contributions and Ties Hoomans for his helpful comments. Source of financial support: This study was supported by a research grant (No ) from the Dutch Organization for Health Research and Development. Part of this research was presented at the Health Technology Assessment International Conference, June 6 9, 2010, Dublin, Ireland. REFERENCES [1] Dhruva SS, Phurrough SE, Salive ME, et al. CMS s landmark decision on CT colonography examining the relevant data. N Engl J Med 2009; 360: [2] Neumann PJ, Tunis SR. Medicare and medical technology the growing demand for relevant outcomes. N Engl J Med 2010;362: [3] Raftery J. Multiple sclerosis risk sharing scheme: a costly failure. BMJ 2010;340:c1672. [4] Drummond MF, Schwartz JS, Jonsson B, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care 2008; 24: [5] Neumann PJ, Drummond MF, Jonsson B, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care 2010;26:71 8. [6] Torrance GW, Siegel JE, Luce BR. Framing and designing the costeffectiveness analysis. In: Gold MR, Siegel JE, Russell LB, et al., eds., Cost-Effectiveness in Health and Medicine. Oxford: Oxford University Press; [7] Fox-Rushby J, Cairns J. Economic evaluation. Maidenhead, UK: Open University Press; [8] McCabe C, Dixon S. Testing the validity of cost-effectiveness models. Pharmacoeconomics 2000;17: [9] Sargent R. Validation and verification of simulation models. Proceedings of the Winter Simulation Conference 2004 (WSC 2004): Volume 1, pp [10] Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. New York: Oxford University Press; [11] Revicki DA, Frank L. Pharmacoeconomic evaluation in the real world. Effectiveness versus efficacy studies. Pharmacoeconomics 1999;15: [12] Rothwell PM. External validity of randomised controlled trials: to whom do the results of this trial apply?. Lancet 2005;365: [13] Hughes DA, Bagust A, Haycox A, et al. The impact of non-compliance on the cost-effectiveness of pharmaceuticals: a review of the literature. Health Econ 2001;10: [14] Rosen AB, Spaulding AB, Greenberg D, et al. Patient adherence: a blind spot in cost-effectiveness analyses? Am J Manag Care 2009;15: [15] Craig D, McDaid C, Fonseca T, et al. Are adverse effects incorporated in economic models? A survey of current practice. Int J Technol Assess Health Care 2010;26: [16] Mullins CD, Montgomery R, Tunis S. Uncertainty in assessing value of oncology treatments. Oncologist 2010;15(Suppl. 1): [17] Ramsey S, Willke R, Briggs A, et al. Good research practices for costeffectiveness analysis alongside clinical trials: the ISPOR RCT-CEA Task Force report. Value Health 2005;8: [18] Evers S, Goossens M, de Vet H, et al. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care 2005;21: [19] Hailey D. Toward transparency in health technology assessment: a checklist for HTA reports. Int J Technol Assess Health Care 2003;19:1 7. [20] Chan AL, Leung HW, Lu CL, et al. Cost-effectiveness of trastuzumab as adjuvant therapy for early breast cancer: a systematic review. Ann Pharmacother 2009;43: [21] Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. New York: Oxford University Press; [22] Glick HA, Doshi JA, Sonnad SS, et al. Economic Evaluation in Clinical Trials. Oxford: Oxford University Press; [23] Jemal A, Siegel R, Ward E, et al. Cancer statistics, CA Cancer J Clin 2009;59: [24] Slamon DJ, Clark GM, Wong SG, et al. Human breast cancer: correlation of relapse and survival with amplification of the HER-2/ neu oncogene. Science 1987;235: [25] Spielmann M, Roche H, Delozier T, et al. Trastuzumab for patients with axillary-node-positive breast cancer: results of the FNCLCC-PACS 04 trial. J Clin Oncol 2009;27: [26] Joensuu H, Kellokumpu-Lehtinen PL, Bono P, et al. Adjuvant docetaxel or vinorelbine with or without trastuzumab for breast cancer. N Engl J Med 2006;354: [27] Piccart-Gebhart MJ, Procter M, Leyland-Jones B, et al. Trastuzumab after adjuvant chemotherapy in HER2-positive breast cancer. N Engl J Med 2005;353: [28] Romond EH, Perez EA, Bryant J, et al. Trastuzumab plus adjuvant chemotherapy for operable HER2-positive breast cancer. N Engl J Med 2005;353: [29] Smith I, Procter M, Gelber RD, et al. 2-year follow-up of trastuzumab after adjuvant chemotherapy in HER2-positive breast cancer: a randomised controlled trial. Lancet 2007;369: [30] Joensuu H, Bono P, Kataja V, et al. Fluorouracil, epirubicin, and cyclophosphamide with either docetaxel or vinorelbine, with or without trastuzumab, as adjuvant treatments of breast cancer: final results of the FinHer Trial. J Clin Oncol 2009;27: [31] Sculpher MJ, Pang FS, Manca A, et al. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technol Assess. 2004;8:iii iv. [32] Baltussen R, Ament A, Leidl R. Making cost assessments based on RCTs more useful to decision-makers. Health Policy 1996;37: [33] Garrison LP, Jr., Neumann PJ, Erickson P, et al. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health 2007;10: [34] O Brien B. Economic evaluation of pharmaceuticals. Frankenstein s monster or vampire of trials? Medical Care 1996;34:DS [35] Bartlett C, Doyal L, Ebrahim S, et al. The causes and effects of sociodemographic exclusions from clinical trials. Health Technol Assess 2005;9:iii iv, ix x, [36] Phillips KA, Marshall DA, Haas JS, et al. Clinical practice patterns and cost effectiveness of human epidermal growth receptor 2 testing strategies in breast cancer patients. Cancer 2009;115: [37] Dedes KJ, Szucs TD, Imesch P, et al. Cost-effectiveness of trastuzumab in the adjuvant treatment of early breast cancer: a model-based analysis of the HERA and FinHer trial. Ann Oncol 2007;18: [38] Garrison LP Jr, Lubeck D, Lalla D, et al. Cost-effectiveness analysis of trastuzumab in the adjuvant setting for treatment of HER2-positive breast cancer. Cancer 2007;110: [39] Kurian AW, Thompson RN, Gaw AF, et al. A cost-effectiveness analysis of adjuvant trastuzumab regimens in early HER2/neu-positive breast cancer. J Clin Oncol 2007;25: [40] Liberato NL, Marchetti M, Barosi G. Cost effectiveness of adjuvant trastuzumab in human epidermal growth factor receptor 2-positive breast cancer. J Clin Oncol 2007;25: [41] Lidgren M, Jonsson B, Rehnberg C, et al. Cost-effectiveness of HER2 testing and 1-year adjuvant trastuzumab therapy for early breast cancer. Ann Oncol 2008;19: [42] Millar JA, Millward MJ. Cost effectiveness of trastuzumab in the adjuvant treatment of early breast cancer: a lifetime model. Pharmacoeconomics 2007;25: [43] Neyt M, Huybrechts M, Hulstaert F, et al. Trastuzumab in early stage breast cancer: a cost-effectiveness analysis for Belgium. Health Policy 2008;87: [44] Norum J, Olsen JA, Wist EA, et al. Trastuzumab in adjuvant breast cancer therapy. A model based cost-effectiveness analysis. Acta Oncologica 2007;46: [45] Shiroiwa T, Fukuda T, Shimozuma K, et al. The model-based costeffectiveness analysis of 1-year adjuvant trastuzumab treatment: based on 2-year follow-up HERA trial data. Breast Cancer Res Treat 2008;109: [46] Van Vlaenderen I, Canon JL, Cocquyt V, et al. Trastuzumab treatment of early stage breast cancer is cost-effective from the perspective of the Belgian health care authorities. Acta Clin Belg 2009;64: [47] Essers BA, Seferina SC, Tjan-Heijnen VC, et al. Transferability of model-based economic evaluations: the case of trastuzumab for the adjuvant treatment of HER2-positive early breast cancer in the Netherlands. Value Health 2010;13:

8 784 VALUE IN HEALTH 14 (2011) [48] Pharmacoeconomic guidelines around the world. Available from: [Accessed February 4, 2011]. [49] Briggs AH. Handling uncertainty in cost-effectiveness models. Pharmacoeconomics 2000;17: [50] Brennan A, Akehurst R. Modelling in health economic evaluation. What is its place? What is its value? Pharmacoeconomics 2000;17: [51] Briggs A, Ritchie K, Fenwick E, et al. Access with evidence development in the UK: past experience, current initiatives and future potential. Pharmacoeconomics 2010;28: [52] Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003;290: [53] Drummond M, Barbieri M, Cook J, et al. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value Health 2009;12: [54] Welte R, Feenstra T, Jager H, et al. A decision chart for assessing and improving the transferability of economic evaluation results between countries. Pharmacoeconomics 2004;22: [55] Hofmann B. Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Assess Health Care 2005;21: [56] Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technology Assessment (Winchester, England). 2007;11: iii iv. [57] Neumann PJ. Why don t Americans use cost-effectiveness analysis? Am J Manag Care 2004;10: [58] Oortwijn WJ, Hanney SR, Ligtvoet A, et al. Assessing the impact of health technology assessment in the Netherlands. Int J Technol Assess Health Care 2008;24: