Valuing Health-Related Quality of Life A Review of Health State Valuation Techniques

Transcription

1 REVIEW ARTICLE Pharmacoeconomics 2000 Feb; 17 (2): /00/ /$20.00/0 Adis International Limited. All rights reserved. Valuing Health-Related Quality of Life A Review of Health State Valuation Techniques Colin Green, John Brazier and Mark Deverill Sheffield Health Economics Group, School of Health and Related Research, University of Sheffield, Sheffield, England Contents Abstract Description of Health State Valuation Techniques Visual Analogue Scale (VAS) or Rating Scale Magnitude Estimation (ME) Standard Gamble (SG) Time Trade-off (TTO) Person Trade-off (PTO) Details of the Literature Search Methodology and Results Criteria for Reviewing Performance Practicality Reliability Validity Comparison of Health State Valuation Techniques Practicality Reliability Theoretical Basis of Valuation Techniques VAS ME TTO PTO SG Empirical Validity Theory Predicting Preferences Mapping from Visual Analogue Scale to Standard Gamble or Time Trade-off Conclusion Abstract Given the growing need to value health-related quality of life, a review of the literature relating to health state valuation techniques was undertaken to appraise the current theoretical and empirical evidence available to inform on the techniques, to identify consensus, identify disagreement and identify important areas for future research. A systematic search of the literature was conducted, covering standard gamble (SG), time trade-off (TTO), visual analogue scale (VAS), magnitude estimation (ME) and person trade-off (PTO) techniques. The basic concepts of practicality, reliability, theoretical validity and empirical validity formed the criteria for reviewing the performance of valuation techniques.

2 152 Green et al. In terms of practicality and reliability, we found little evidence relating to ME and PTO. SG, TTO and VAS have been shown to be practical on a range of populations. There is little difference between the reliability of SG, TTO and VAS, and present evidence does not offer a basis to differentiate between them. When considering the theoretical basis of techniques, we conclude that choicebased methods (i.e. SG, TTO and PTO) are best placed to reflect the strength of preference for health, with the choice between these techniques depending on the study characteristics and the perspective employed. Empirical evidence relating to the theoretical perspective of the techniques has shown that there are problems with all techniques in terms of descriptive validity. There is growing acceptance within the healthcare profession that people s preferences regarding the benefit from, or effectiveness of, different interventions should have a role in decision-making. Given this belief, it is becoming increasingly necessary not only to describe the benefits or otherwise of different interventions but also to give an indication of the value attached to such benefits in terms of health-related quality of life (HR-QOL). A number of informative reviews have been conducted to assess the main areas that require addressing in the valuation of HR-QOL. [1-4] These reviews consider, in a broad manner, how to describe health (i.e. what to value), who to involve in the valuation task and the techniques that can be used for the purpose of health valuation. Each of these areas contains a number of unresolved issues, both empirical and theoretical. In considering the techniques for health valuation, a number of approaches have been used to determine the values that might be attached to HR- QOL. The 3 methods that have been widely used to date are: (i) contingent valuation (CV), which places a monetary value on health changes; (ii) conjoint analysis (CA), which indirectly attaches a relative value to health changes; and (iii) the health state valuation or health state utilities approach, [1] which elicits preferences for individual health states in order to represent health on a metric scale, usually between 1 (full health) and 0 (dead) although scale values less than 1 can be used to represent health state preferences regarded as worse than death. [1] Both CV and CA have been the subject of increasing attention, but it is the latter of the 3 approaches, the health state valuation approach, which we consider in this review. This is the approach employed to determine quality weights (preferences) in the calculation of quality-adjusted life-years (QALYs). Much debate exists, in a wide literature, as to the relative merits of the techniques used to elicit health state valuations. By examining the health state valuation techniques available, this review focuses attention on one important area of inquiry in order to offer practical advice to those wishing to employ such techniques in a particular setting. We review both the theoretical and empirical evidence which informs on the specific techniques available to value multidimensional health state descriptions. The valuation techniques covered in this review are the standard gamble (SG), time trade-off (TTO), visual analogue scale (VAS) or rating scale (RS), magnitude estimation (ME) and person trade-off (PTO). These techniques have been used to elicit a strength of preference for health states on a health metric scale. For the purpose of this review, such a measure of strength of preference is considered to be a utility measure, leaving to one side, for simplicity, the debate relating to the interpretation of utility and the debate concerning the distinction between utilities and values (for further information on these issues, refer to Richardson [3] who offers an excellent contribution to the debate). The findings of this review are relevant regardless of the interpretation of measures of strength of preference as utilities or values. The aim of the review has been to appraise the current theoretical and empirical evidence avail-

3 Valuing HR-QOL 153 able on techniques for the valuation of HR-QOL in order to identify consensus, disagreement and important areas for future research. It is the first time that such a descriptive review has been undertaken based on a systematic search of the healthcare literature. Issues surrounding what to value, whose values to elicit and, in a wider sense, the issues relating to framing and context effects have not been addressed as part of the review, unless such considerations appear to be specific to particular valuation techniques. These topics are considered to be more general issues, rather than method-specific concerns and, although we advocate further research in these areas, we are unable to address all of the topics together. We do not wish to give the impression that these issues are insignificant and would recommend the interested reader to consult, as a starting point, Froberg and Kane [5] on population and context effects, and a more recent review on many of these issues by Rabin. [6] 1. Description of Health State Valuation Techniques The techniques reviewed have been described in detail by many commentators [1-4] and we provide only summary descriptions in this paper. We would encourage the reader to consult earlier work [7] for further clarification where necessary Visual Analogue Scale (VAS) or Rating Scale VAS, sometimes referred to in the literature as the category rating scale (CR) or just RS, is simply a line, usually with well defined end-points on which respondents are able to indicate their preferences. There are many variants of the technique. The lines can vary in length, be vertical or horizontal and may or may not have intervals marked out with different numbers. 1.2 Magnitude Estimation (ME) Developed in psychometrics as an alternative to VAS, the phrasing of the ME question was intended to generate data with ratio properties (i.e. one state is so much better or worse than another), and therefore, it is often referred to in the psychometric literature as ratio scaling. ME asks patients to provide the ratio of undesirability of pairs of health states, e.g. patients may say that state A is x times worse than state B. Therefore, the undesirability (disutility) of state A is x times as great as that of state B. By asking a series of questions, all states can be related to each other on the undesirability scale. [1] 1.3 Standard Gamble (SG) SG asks respondents to make a choice between alternative outcomes, where one of them involves uncertainty. They are asked how much in terms of risk of death, or some other outcome worse than the one being valued, they are prepared to accept in order to avoid the certainty of the health state being valued. This technique is based on the Expected Utility Theory (EUT) of decision-making under uncertainty developed by Von Neuman and Morgenstern, [8] which rests on a set of axioms about the nature of individual preferences over uncertain prospects. 1.4 Time Trade-off (TTO) The TTO technique was developed [9] as an alternative to SG, and was designed to overcome the problems of explaining probabilities to respondents. The TTO technique asks the respondent to choose between 2 alternatives, both with certain prospects, i.e. years (x) in full health and years (t) in the health state being valued. The respondent is asked to consider trading a reduction in their length of life for a health improvement. The health state valuation is the fraction of healthy years equivalent to a year in a given health state, i.e. x/t. 1.5 Person Trade-off (PTO) PTO is a way of estimating the social value of different health states. As with TTO and SG, the PTO technique asks the respondent to make a choice between alternatives. The crucial difference is that

4 154 Green et al. the respondent makes a choice in the context of a decision involving other people rather than themselves. PTO basically consists of asking an individual how many outcomes of one kind (e.g. outcome A) they consider to be equivalent in social value to X outcomes of another kind (e.g. outcome B). [10] By asking a series of such questions, all conditions can be related to each other on the undesirability scale. [1] 2. Details of the Literature Search Methodology and Results Search terms were identified to capture literature relating to valuation techniques and, where necessary (i.e. VAS literature), a subset of literature was constructed to attract only those references related to health and healthcare using specific search terms (table I). All searches covered the period from the date of commencement of each database service (e.g. Medline 1966) to November 1997, but were restricted to English language materials. The health-related databases used were Medline, EMBASE (Excerpta Medica), HealthSTAR and CINAHL. The general science and social sciences databases used were the Science Citation Index (SCI), the Social Science Citation Index (SSCI) and the International Bibliography of the Social Sciences (IBSS). An economicspecific database, EconLit, was searched and the National Health Service (NHS) Economic Evaluation Database (NEED) from the NHS Centre for Table I. Search terms used to identify literature relating to valuation techniques Medline exemplar 1. (rating scale*) or (categor* near2 scal*) or (linear scal*) or (linear analog*) or (visual analog*) 2. (magnitude estimation) or (ratio scal*) 3. standard gamble* 4. (timetradeoff) or (time tradeoff) or (time trade off) or (time trade*) 5. (persontradeoff) or (person tradeoff) or (person trade off) or (person trade*) or (equivalen* near2 number*) Health-related literature subset where appropriate (health near2 state*) or (health near2 status) or health-status* or (health near2 utilit*) or (quality near2 life) or quality-of-life* Reviews and Dissemination was also searched. All citations resulting from the search strategy were retrieved and articles were reviewed for relevance on the basis of their abstracts. Details of the literature identification process are shown in figure 1. The identified abstracts and bibliographic details were reviewed according to the following predetermined criteria: (i) references had to present discussion/results relating to at least one of the designated health state valuation techniques; (ii) references had to discuss/present health state valuation techniques in the context of health and healthcare evaluation, i.e. literature relating to environmental or transport applications, for example, were not selected for review; and (iii) the valuation technique(s) had to be discussed/applied in the context of eliciting values for general multidimensional health states, i.e. techniques which had been discussed/applied in the context of valuing unidimensional health state descriptors (e.g. pain or disability only) were not selected for review. By applying these criteria at the initial sifting stage, a significant number of references were excluded (1204 at this stage). Many of these related to VAS and, on inspection, they were using the technique to measure individual dimensions of health status, e.g. pain. The remaining articles (377) were obtained and, once again, the above criteria were applied. The resulting review literature (259 references) was supplemented with a further 26 references identified from bibliographic information contained in the identified references. These were classed as ad hoc references. Including the ad hoc references, we found a literature of 285 articles for review. 3. Criteria for Reviewing Performance The basic concepts of practicality, reliability and validity form the criteria used for reviewing the performance of the valuation techniques. We have presented our views on these criteria in some detail elsewhere; [11] however, these views can be summarised in the context of valuation techniques, as follows.

5 Valuing HR-QOL 155 Database Reference 3.2 Reliability Medline 846 Embase 812 Cinahl 316 EconLit 7 Healthstar 149 SSCI/SCI 432 Need 15 IBSS 13 Reliability is the ability of a measure to reproduce the same quality adjustment values (preference scores) on 2 separate administrations when there has been no change in the health state being valued. This can be over time, known as retest reliability, within the rating task or between raters. Correlation coefficients are the commonly presented measure of reliability. Whilst we recognise the arguments against the use of correlation as a measure of agreement between measures, [12] most studies have only reported on reliability using this type of summary statistic. Combined excluding duplicates Selected for review Final review references Accept 259 Plus adhoc references Practicality Excluded Excluded Fig. 1. Overview of the literature identification process. IBSS = International Bibliography of the Social Sciences; SCI = Science Citation Index; SSCI = Social Science Citation Index. The practicality of a health state valuation technique depends on its acceptability to respondents. Such acceptability can be a function of length, complexity and respondents interest in the task. It can be assessed by examining issues such as completion, response and the level of missing data. 3.3 Validity Validity was considered on both theoretical and empirical grounds. We examined the economic validity of theoretical arguments presented in the literature to support the use of health state valuation techniques, e.g. the use of EUT as a basis for SG. The assessment of the theoretical basis of the techniques also helped identify the testable predictions of any underlying theory. In the assessment of the empirical validity of a technique, we require a test of the extent to which the technique was able to predict those preferences revealed from actual decisions, i.e. does it measure what it intends to measure? Unfortunately, there is a general absence of revealed preference data relating to health state valuation techniques. In the absence of such data, we assess empirical validity on the basis of: (i) evidence on the extent to which the theoretical basis of the techniques correctly describe individual preferences; and (ii) the evidence available to predict stated preferences (stated ordinal preferences and convergence) and hypothetical preferences [such as consistency with recognised descriptive orderings, e.g. EQ-5D (EuroQoL instrument). For example, where studies report the respondents ordinal ranking of health states (i.e. stated preferences), the ability of each valuation technique to correctly predict the ranking of the states provides some evidence of their ordinal properties.

6 156 Green et al. When seeking to test the techniques against hypothetical preferences, studies may hypothesise that one health state should be preferred to another. For example, it could be that on the basis of past experience, patients with renal failure are expected to prefer to be in a health state following a successful transplant than to depend on dialysis. Again, the ability of each valuation technique to correctly predict the hypothesised preferences provides some evidence of their empirical properties. We accept that some definitional uncertainty exists in the present literature surrounding the criteria to be used for the assessment of health state valuation techniques, for use in economic evaluation, especially surrounding the validity of techniques. [11] However, the criteria used here have been broadly accepted in earlier reviews [1,2] and offer an opportunity to assess the performance of health state valuation techniques. Although all the criteria are seen as addressing important issues, with each being considered as necessary but not sufficient, we have found no empirical literature on which to base any ranking or weighting of the criteria in terms of their relative importance. We view empirical validity as the most important criterion upon which to assess the valuation techniques, whilst accepting that there will be trade-offs between particular criteria depending on study characteristics (e.g. population, size of sample). 4. Comparison of Health State Valuation Techniques A review of the literature relating to each of the techniques was undertaken using the review criteria discussed. The general findings are presented in the form of a comparison of valuation techniques. A more detailed review on each of the techniques has been reported elsewhere. [7] 4.1 Practicality Although the 5 methods of health state valuation discussed in this review have previously been reported to be practical and acceptable, [2] we have found a lack of empirical evidence to demonstrate the acceptability of ME and PTO. The PTO has not been widely used to value health states. Applications of the PTO have been in a methodological environment [10,13,14] and, to date, the feasibility and acceptability of the PTO in health is relatively unknown. The lack of evidence for ME also reflects the limited use made of ME in healthcare. Further evidence is required from empirical studies to demonstrate the practicality of ME and PTO. There is empirical evidence to support the practicality of the SG, [15-20] TTO [15,17,21-27] and VAS [2,28-34] methods, with high completion and response rates reported. SG and TTO have been found to be practical for most populations studied, but VAS is usually slightly better in terms of response rate and cost. Although SG and TTO have demonstrated a similar performance in terms of completion, TTO has outperformed SG by a small margin in a number of studies. [15,35,36] Earlier reviews [2] have suggested that the SG is complex and not intuitively obvious to most respondents, stressing that it may be too complex for population studies. However, empirical evidence supports the successful use of SG. A number of studies across different respondent groups (including a general population sample) have reported completion rates between 95 and 100%, [15,16,18-20] and studies by Revicki [37] and Gage et al. [38] report completion rates over 80%. Whilst some studies report SG to be feasible, they do not report quantitative details (e.g. Llewellyn Thomas et al. [39] in their study of patients with cancer). Although SG has shown some completion problems within particular studies, these have been no worse than similar difficulties associated with other techniques used at the same time. For example, where completion problems are reported by Patrick et al. [17] and Van der Donk et al. [36] (both studies used SG, TTO and VAS), SG was not seen to be more burdensome than the other methods employed. Stiggelbout et al. [40] report completion problems with SG; however, the authors concluded that the questions themselves were too hypothetical for the patient group (disease-free patients with testicular cancer).

7 Valuing HR-QOL 157 The TTO technique has also proved to be a practical technique in general population studies [4,15] and in a wide variety of empirical studies. [21-27] For example, Dolan et al. [15] and Fryback et al. [25] both report completion rates over 95% in population based-studies, and Glasziou et al. [23] report a completion rate of 91% in a large self-completed sample survey of patients 6 months after myocardial infarction (n = 714). Both SG [4] and TTO [23] have been shown to be feasible in a self-completed format, but both techniques are relatively unproven in a postal format. The VAS technique has been used more widely, with broader evidence of acceptability [28-34] and empirical evidence to support its use in postal surveys. [33,41] As one would expect, given the more complex cognitive task, it would appear that the choice-based SG and TTO techniques result in a larger number of refusals, missing values and inconsistent responses. Some of these completion difficulties are due to misunderstandings, conflicts with personal beliefs or straight-forward difficulties in understanding the tasks. [25,38] Other studies have reported that respondents have difficulties dealing with small probabilities [42] or overestimate small probabilities. [43] Furthermore, although respondents find the SG and TTO techniques acceptable, they are often unwilling to make a sacrifice in the valuation task. [44] That is, respondents have been unwilling to trade any of their remaining life-expectancy (not even a few days or hours) or have been unwilling to accept any level of risk (not even a very small level of risk) to obtain an improved health state, even when respondents are experiencing a large number of health problems. [45] There has been a growth in the number of studies utilising computer-based methods for the administration of valuation tasks. Empirical evidence has been reported to show that computer-based methods (e.g. U Titer) [46,47] have proved feasible and acceptable for the presentation of SG, TTO and VAS. [18,46,48,49] 4.2 Reliability Froberg and Kane [2] report an acceptable level of intrarater reliability for all 5 of the techniques discussed and good to moderate levels of inter-rater reliability for RS, ME and PTO (equivalence). However, they do comment on the general lack of evidence surrounding the reliability of methods. Ten years after the study by Froberg and Kane [2] this problem still exists, with many studies either failing to undertake or failing to report tests of reliability. Given the data available, we have focused on reliability over time, i.e. test-retest reliability. There is little evidence surrounding the test-retest reliability of PTO and ME techniques. However, of the 2 techniques, ME would appear to be the most promising in terms of reliability. [50] Rosser and Kind [50] report test-retest reliability for ME at 97.2%, measured by the percentage of agreement. Nord [51] reports poor test-retest findings for the PTO (40% measured by the percentage of agreement) and expresses concerns over a strong random element in individual PTO responses. PTO is being further developed and more evidence is expected in the near future. Table II reports empirical evidence covering SG, TTO and VAS techniques. All of the techniques demonstrate an acceptable level of reliability and there is little to choose between them on the grounds of reliability. In 3 of the 5 comparative studies reported in the table, choice-based methods outperform VAS, with 1 case showing similar results at 1-week retest. SG and TTO techniques display similar results across comparative studies in terms of reliability. Empirical evidence would suggest that, although it would be difficult to express a preference over the 2 techniques on the basis of reliability, the TTO offers slightly better performance statistics on test-retest reliability, as can be seen by its greater reliability in 3 of the 5 comparative studies cited in table II. Yet, as shown in the table, the differences between the retest reliability of the 3 techniques are small.

8 158 Green et al. Table II. Test-retest reliability for SG, TTO and VAS techniques (maximum value of 1). Intraclass correlation coefficient applied in references Dolan et al., [4] Bakker et al., [32] Gudex et al., [34] Gage et al. [38] and O Brien & Viramontes. [52] Pearson correlation coefficient applied in references Dolan et al. [15] and Ramsey et al. [19] and others were unspecified Test-retest reliability time period Techniques SG TTO VAS 1-week 0.80, [53] [32] 0.87 [53] 0.77, [53] [32] 4-week 0.82 [52] 0.81, [54] 0.63 [54] 0.62, [52] 0.89 [54] 3- to 6-week [21] 6-week , [54] 0.85 [55] 10-week 0.73 [4] 0.78 [34] 6- to 16-week 0.63 (props), [15] 0.74 (no props) [15] 0.83 (props), [15] 0.55 (no props) [15] 1-year 0.53 [28] 0.62 [28] 0.49 [28] Other (time-specified) 0.82, [35] 0.80 [38] 0.74, [35] [38] SG = standard gamble; TTO = time trade-off; VAS = visual analogue scale. 4.3 Theoretical Basis of Valuation Techniques VAS The VAS is not a choice-based technique and this raises doubts as to its ability to reflect strength of preference on a cardinal scale. [56] Due to the absence of choice and opportunity cost within the VAS technique, one common view is that it can have no basis in economic theory. [3,57] Yet, there are supporters of the VAS techniques who present them as a cardinal measure of strength of preference, [30,37] and indeed the techniques have been widely used in such a manner. The strongest theoretical argument for the VAS has been provided by Dyer and Sarin, [58] who suggest a measurable value function providing a link between such a value function and utility. Essentially, the function represents preferences under certainty. Dyer and Sarin [58] postulate that utility represents preferences under uncertainty and a stable relationship exists between values and utilities. Torrance et al. [59] have interpreted this to provide a link between VAS and SG, and hence a means of estimating SG values from VAS responses. However, such a theoretical basis is not established in economics. This leaves the foundation of VAS in psychometrics and psychophysics, which have no direct link with the measurement of strength of preferences ME Although ME has a fairly long history in the healthcare decision-making literature, [60] it has been largely unused and remains theoretically undeveloped. ME does not present the respondent with a choice and some find its meaning obscure and inappropriate. [3] The theoretical appeal of ME rests on an assumption that it is able to provide ratioscale properties, [50,60,61] yet there are serious doubts concerning the basis of this assumption. [62] ME is not related to economic theory and the assumption surrounding its ratio level properties remain unsupported TTO The TTO technique presents a choice which involves an opportunity cost, a sacrifice, and as such finds some association with theories of consumer choice and welfarism. Theoretical support has been sought amongst those theories surrounding equivalent and compensating variation. Mehrez and Gafni [63] discuss the TTO in the context of value function theory (as set out by Dyer and Sarin [58] ) due to the identification of differing trade-off combinations of health and duration. Buckingham et al. [64] align the TTO method with the welfare economic approach of compensating variation, where welfare gain is measured by compensating loss of something else that is valuable so that the respondent is returned to their original level of welfare. Although these theoretical arguments emphasise the conceptual advantages of choice-based methods in economics, they remain undeveloped within the current literature and TTO remains unrelated in a specific way to any underlying economic theory. Should TTO be considered in the context of consumer theory, 3 concerns would present themselves:

9 Valuing HR-QOL 159 the effect of duration, [65,66] the impact of time preference [67] and the incorporation of uncertainty. [68] PTO PTO is a choice-based technique; however, it relates to social choice, i.e. choices concerning the welfare of others and the opportunity cost is not directly borne by the individual. Due to the relation between the choice presented within the PTO technique, standard theories of consumer choice can not be applied. Although the technique is seen as intuitively appealing, [10] there are no theoretical underpinnings advocated in the current literature other than psychometric qualities surrounding adjustment or equivalent stimuli. [61] There is support for the potential interval scale properties of the PTO due to the fact that there is a clear and comprehensible meaning to the PTO (where the numbers are specified). [3,13] Nevertheless, there presently appears to be no formal theoretical support within economics to underpin the PTO technique. It may be that the PTO, due to its social preference perspective, can be linked to the economics literature surrounding the valuation of externalities, but this has not been pursued so far SG The SG is undoubtedly the most theoretically appealing of the techniques reviewed here. It has rigorous theoretical foundations in the form of the EUT which has proved to be the dominant theory of decision-making under uncertainty since the 1950s. Although its restrictive axiomatic approach has many critics (e.g. Loomes & McKenzie [43] ) due to the uncertain nature of medical decisionmaking, EUT finds eminent supporters for its application to health state valuation. [1,69] Due to its link with EUT, the SG has been put forward as the criterion or reference method of health state valuation, often referred to as the gold standard. [28,70] However, in the face of theoretical arguments against the favoured use of EUT and SG in health state valuation, [3,71,72] andinconsiderationofthe limited empirical support for the application of EUT (discussed in section 4.4.1), we can not support the gold standard status of the SG even though we do recognise the potential theoretical strengths of the SG technique. 4.4 Empirical Validity Theory In terms of empirical support for the theoretical underpinnings of the techniques, we have reported that both PTO and ME lack theoretical support and have therefore been unable to comment further on these techniques. The theoretical argument associated with the VAS technique, i.e. measurable value function, has been challenged by Bleichrodt and Johannesson [73] who present findings to suggest such a function is not present (see also Loomes et al. [74] ). There is also evidence reported to suggest that the presence of response spreading is a significant VAS problem. [30,57,75] Although the TTO is not directly linked to specific theoretical foundations, there is evidence to suggest that duration effects and time preference effects can have an impact on the elicitation and use of TTO values. [66,67] The absence of uncertainty in the TTO task has been addressed by Stiggelbout et al. [40] and Cher et al. [76] who report that it is possible to adjust TTO elicited values to address the absence of uncertainty (e.g. risk adjustment). However, we have found no further empirical literature to demonstrate such an adjustment. The SG is the only technique with clear theoretical foundations (i.e. EUT) but there is evidence showing that SG values can be strongly influenced by the outcomes used in the task (i.e. nonindependence) and by the manner in which the task is presented. [38,39] There is also evidence to suggest that attitude to risk is not constant. [43,49,77,78] Further evidence that respondents systematically violate the axioms of EUT in the health context has been reported by Dolan et al. [4] and Read et al., [79] and more generally by Hershey et al. [80] and Schoemaker. [81] Such evidence suggests that the axioms of EUT are often violated. This has led some to consider the SG to be the best available technique on the basis of the normative appeal of the axioms of EUT. [70,82] Torrance and Feeney [70] set the value of EUT as indicating how individuals preferences should behave even if the

10 160 Green et al. outcome does not correspond to their own choice. Richardson [3] counters the normative theoretical arguments in favour of the SG, citing defects in the normative approach. Richardson [3] states that the EUT is unable to take account of the specific utility of risk and of the associated emotions. He also accuses the normative justification for SG of being logically tautologous. Although the theoretical underpinnings of the SG are beneficial in understanding the decisionmaking process, if the axioms of EUT are empirically flawed in both its positive and normative forms, as many commentators suggest, there can be no justification for the SG as the reference method or gold standard for health state valuation Predicting Preferences We have not found a large literature reporting on the empirical validity of valuation techniques in terms of their ability to predict preferences. Researchers have for some time found the assessment of the empirical validity of techniques a difficult task, this being due to the absence of any reference unit of measurement such as revealed preferences. We have found evidence to report on the empirical validity of techniques on the basis of their performance against stated and hypothesised preferences. In terms of stated preferences, we have considered techniques in relation to available measures of stated preference, for example ordinal ranking of health states and also in the context of valuations elicited from other techniques employed at the same time (i.e. convergent validity). Although many studies consider such issues in their protocol, we have found that often results are not reported. There is evidence of a poor to moderate correlation between VAS values and values from choicebased techniques undertaken at the same time (i.e. SG and TTO). [32,49,83,84] This finding, together with significant evidence to suggest a strong correlation between VAS and measures of health status [e.g. pain, functioning, clinical symptoms and instruments such as the Sickness Impact Profile (SIP) and Arthritis Impact Measurement Scale (AIMS)], [30,83,85] raises concerns over the ability of VAS methods to elicit strength of preference for health states (these concerns are also compounded by the findings relating to response spreading). Such concerns are further supported by qualitative data reported in the study undertaken by Robinson et al., [86] where respondents indicated that their VAS responses did not truly reflect their preference. Although the ordinal properties of VAS methods are largely unchallenged, findings of this nature cast doubt on whether VAS values are able to reflect respondent strength of preference. It would appear from the empirical findings of this review that VAS methods are measuring aspects of HR- QOL which differ from those being considered by the SG and TTO. There is little evidence concerning the performance of ME and PTO against stated preferences. Gudex et al. [87] report a comparison between ME, VAS and TTO, with some evidence of convergence between ME and VAS; although overall, they cite tests (Friedman test statistic) indicating important differences between valuations produced by the different methods. These studies are limited and further evidence is required to support any relationship. SG and TTO values have been shown to correlate reasonably well with one another, [1,15,84,88,89] suggesting they may be valuing similar aspects of HR-QOL. However, a study by Hornberger et al. [90] reports only a poor correlation between SG and TTO. Whilst we have found no direct evidence to inform on the performance of SG against stated ordered preferences, Ashby et al. [21] report that the TTO produced mean values which were consistent with respondent rank ordering of health states and Robinson et al. [86] present qualitative evidence to indicate TTO responses reflect the stated preferences of individuals. Evidence relating to consistency of response with aprioripreferences indicates that SG, TTO and VAS have demonstrated good levels of consistency with multiattribute utility scales (MAUS), 1 whilst 1 Note that in this review, we have not concerned ourselves in any detail with the presentation of MAUS or the description of health states. We have undertaken a detailed review of MAUS which can be found in Brazier et al. [7]

11 Valuing HR-QOL 161 a lack of evidence surrounding PTO and ME leaves the consistency of these methods unproven. Although consistency may be a function of the medium used to present health states for valuation (e.g. EQ- 5D), thereby making the comparison of techniques dubious, 2 of the studies reviewed report comparative results. Gudex et al. [87] report that consistency of TTO responses was superior to VAS, although the difference was small (reversals of logical ordering was found in 6 of 28 TTO mean/median responses as opposed to 7 of 28 VAS responses); however, ME was found to be superior to both of these techniques (with only 3 inconsistent responses of 28). Gudex et al. [87] report that inconsistency may have been related to one particular level of disability. Dolan et al. [15] report a TTO consistency rate (91.7% for props and no props versions) superior to SG (83.8 to 87.5%). Gudex et al. [34] and Dolan et al. [4] report high levels of consistency (from VAS and TTO, respectively) amongst a large general population sample (n = 3395). However, quantitative information is not presented in the latter study to allow a direct comparison. Further to the above study by Gudex et al., [87] the consistency of ME is not discussed in the literature reviewed. The consistency of PTO responses is also relatively untouched. However, Ubel et al. [14] have reported (self-administered survey) a high number of inconsistent responses and concerns over the internal inconsistency of PTO, i.e. in relation to a relative unit of measure. A number of studies have reported the performance of SG and TTO against hypothesised or expected (apriori) preferences. Gage et al. [38] report that TTO values reflected the expected ordinal ranking of stroke severity (mild, moderate, severe) and that TTO and SG values for moderate stroke were not significantly different (SG was used to value moderate stroke only). Dolan et al. [15] have reported that both SG and TTO performed well against hypothesised preferences with respect to the background characteristics of respondents. Churchill et al. [91] have reported that TTO values reflected hypothesised preferences within the valuation of endstage renal failure treatment modalities. There is very limited evidence on empirical validity. In relation to consistency with MAUS, the evidence marginally favours TTO over SG, but this is not sufficient to say one technique is more valid than another on empirical grounds. 5. Mapping from Visual Analogue Scale to Standard Gamble or Time Trade-off Part of the literature review examined whether it is possible to estimate unique and stable relationships between VAS and SG, and VAS and TTO. Given the potential cost savings associated with the use of VAS, together with its high rate of completion and reliability, there would be significant practical advantages to being able to map from VAS to one of the choice-based techniques. However, the amount of published evidence available to address this question is limited, studies were based on very small numbers in some instances (e.g. Torrance et al. [59] used 4 pairs of mean health state values), and none reported diagnostic test results for evaluating the appropriateness of different specifications. Findings are discussed in detail elsewhere. [7] However, we believe that the lack of evidence would seem to provide an insecure foundation on which to conclude that there is a unique and stable relationship between VAS and SG or TTO. We therefore recommend SG and TTO values are obtained directly, rather than estimated from VAS values until better evidence becomes available at the group level. 6. Conclusion This is the first review of health state valuation techniques to be based on a systematic search of the literature. As such it adds a further 10 years of theoretical work and empirical evidence to the well referenced reviews undertaken by Torrance [1] and Froberg and Kane. [2] We find that there has been a lack of progress amongst economists in deciding which valuation technique is the most appropriate. We present a greater consensus on the failure of VAS and ME to reflect strength of preference for health states. Yet, when considering the choice

12 162 Green et al. between SG, TTO and PTO, our conclusions reflect the genuine lack of agreement which characterises the present literature. However, our task was not to resolve existing disputes but to reflect them (and to reflect on them). This review has added to the synthesis of evidence on the acceptability and reliability of techniques, providing specific references from the latest empirical literature. More specifically, we have included evidence on the acceptability of computerbased methods which are now becoming more widely available. We assess validity in a specific way, considering proxy measures of revealed preference in terms of available evidence, i.e. stated preferences and hypothesised preferences, this being a development within the generally contentious area of empirical validity. As part of the review, we have attempted to directly assess the theoretical basis (assumptions) of techniques and we believe that this has not been done in any detail in a review of this nature to date. The review also incorporates consideration of the evidence on the mapping of values from VAS to SG and TTO. Although the review has added to the evidence on the application of techniques, more importantly, it highlights that the evidence base is growing very slowly and encourages researchers to undertake tests of acceptability, reliability and validity, or alternatively, to report them where they may not currently do so. In terms of practicality, SG, TTO and VAS have proved to be acceptable and practical on a number of different populations, with little to choose between these 3 techniques. However, VAS techniques have performed slightly better and have cost advantages. Contrary to earlier evidence presented by Froberg and Kane, [2] we have found that SG should not be disregarded on the grounds that it is complex and not intuitively obvious to most respondents. Also contrary to earlier evidence in the review by Froberg and Kane, [2] we have found that ME and PTO lack empirical support to demonstrate acceptability to respondents. When considering reliability, present evidence does not offer a basis to differentiate between the SG, TTO and VAS techniques, whilst again, there is little evidence relating to PTO and ME. Given our examination of the theoretical underpinnings of the techniques and the role of the techniques in acting as a proxy for revealed preferences, only choice-based techniques should be used (SG, TTO and PTO). The choice between SG, TTO and PTO will depend on the perspective employed. The debate surrounding SG versus TTO is unresolved and depends on the belief in the descriptive validity of the theory or its normative basis. Empirical evidence relating to the theoretical perspective of the techniques has shown that there are problems with all techniques in terms of descriptive validity. The evidence available to inform on the performance of techniques against preferences, empirical validity, would suggest that: (i) VAS techniques may be measuring aspects of health status rather than valuing health states; and (ii) choice-based methods are best placed to reflect strength of preference for health states. Given the limited evidence against empirical validity, there is little to choose between the choicebased valuation techniques. SG and TTO are the most developed techniques, with PTO being relatively undeveloped as well as focusing on social preference. SG and TTO have been found to give similar results, although SG values tend to be greater than those of TTO. At present, the empirical literature informing on empirical validity would favour TTO, although this currently reflects the sparse literature available. Further research evidence is required for all techniques to inform on their performance in terms of practicality, reliability, theoretical validity and especially empirical validity. We would prioritise research towards the choice-based techniques (SG, TTO and PTO) with an emphasis on empirical validity and theoretical support. We believe that there can be little benefit in pursuing research on ME. Whilst we focus research on areas of empirical and theoretical validity, we encourage researchers to

13 Valuing HR-QOL 163 include and report tests of acceptability and reliability where possible. Further research on techniques in terms of their ability to proxy revealed preferences for health states will undoubtedly require more qualitative elements of research to assess whether health state values actually correspond to respondent choices, as in the study by Robinson et al. [86] We suspect that quantitative evidence can not provide a definitive answer to the debates surrounding validity, and questions about the cognitive processes and factors determining the response to valuation techniques require further qualitative examination. We would advocate a considered approach to the selection and use of health state valuation techniques. Although we have highlighted the fact that the present literature covering the valuation techniques is sparse, we feel it can offer a valuable insight, both theoretically and empirically to those wishing to elicit health state valuations. Acknowledgements The research reported in this paper was funded by the National Health Service (NHS) Executive through the Health Technology Assessment Programme. We are grateful to Andrew Booth (ScHARR, Sheffield, England) for his help in conducting the literature search and to members of the Health Economists Study Group (Galway, Ireland; July 1998), for their helpful comments. References 1. Torrance GW. Measurement of health state utilities for economic appraisal: a review. J Health Econ 1986; 5: Froberg DG, Kane RL. Methodology for measuring health-state preferences. II: scaling methods. J Clin Epidemiol 1989; 42: Richardson J. Cost utility analysis: what should be measured? Soc Sci Med 1994; 39: Dolan P, Gudex C, Kind P, et al. Valuing health states: a comparison of methods. J Health Econ 1996; 15: Froberg DG, Kane RL. Methodology for measuring health-state preferences. III: population and context effects. J Clin Epidemiol 1989; 42: Rabin M. Psychology and economics. J Econ Lit 1998; XXXVI: Brazier JE, Deverill M, Green C, et al. A review of the use of health status measures in economic evaluation. Health Technol Assess 1999; 3 (9) 8. Von Neumann J, Morganstern O. Theory of games and economic behaviour. Princeton (NJ): Princeton University Press, Torrance GW, Thomas WH, Sackett DL. Autility maximization model for evaluation of health care programs. Health Serv Res 1972; 7: Nord E. The person-trade-off approach to valuing health care programs. Med Decis Making 1995; 15: Brazier JE, Deverill M. A checklist for judging preference-based measures of health related quality of life: learning from psychometrics. Health Econ 1998; 8: Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; I (8476): Pinto Prades JL. Is the person trade-off a valid method for allocating health care resources? Health Econ 1997; 6: Ubel PA, Loewenstein G, Scanlon D, et al. Individual utilities are inconsistent with rationing choices: a partial explanation of why Oregon s cost-effectiveness list failed. Med Decis Making 1996; 16: Dolan P, Gudex C, Kind P, et al. The time trade-off method: results from a general population study. Health Econ 1996; 5: Rabin R, Rosser RM, Butler C. Impact of diagnosis on utilities assigned to states of illness. J R Soc Med 1993; 86: Patrick DL, Starks HE, Cain KC, et al. Measuring preferences for health states worse than death. Med Decis Making 1994; 14: Morss SE, Lenert LA, Faustman WO. The side effects of antipsychotic drugs and patients quality of life: patient education and preference assessment with computers and multimedia. Proc Annu Symp Comput Appl Med Care 1993: Ramsey SD, Patrick DL, Lewis S, The University of Washington Medical Center Lung Transplant Study Group, et al. Improvement in quality of life after lung transplantation: a preliminary study. J Heart Lung Transplant 1995; 14: Lenert LA, Morss S, Goldstein MK, et al. Measurement of the validity of utility elicitations performed by computerized interview. Med Care 1997; 35: Ashby J, O Hanlon M, Buxton MJ. The time trade-off technique: how do the valuations of breast cancer patients compare to those of other groups? Qual Life Res 1994; 3: Detsky AS, McLaughlin JR, Abrams HB, et al. Quality of life of patients on long-term total parenteral nutrition at home. J Gen Intern Med 1986; 1: Glasziou PP, Bromwich S, Simes RJ, for the AUS-TASK Group. Quality of life six months after myocardial infarction treated with thrombolytic therapy. Med J Aust 1994; 161: Johnson ES, Sullivan SD, Mozaffari E, et al. A utility assessment of oral and intravenous ganciclovir for the maintenance treatment of AIDS-related cytomegalovirus retinitis. Pharmacoeconomics 1996; 10: Fryback DG, Dasbach EJ, Klein R, et al. The Beaver Dam Health Outcomes Study: initial catalog of health-state quality factors. Med Decis Making 1993; 13: Kreibich DN, Vaz M, Bourne RB, et al. What is the best way of assessing outcome after total knee replacement? Clin Orthop 1996; Krumins PE, Fihn SD, Kent DL. Symptom severity and patients values in the decision to perform a transurethral resection of the prostate. Med Decis Making 1988; 8: Torrance GW. Social preferences for health states: an empirical evaluation of three measurement techniques. Soc Econ Plan Sci 1976; 10: Torrance GW. Utility approach to measuring health-related quality of life. J Chron Dis 1987: 40: Kaplan RM, Feeny D, Revicki DA. Methods for assessing relative importance in preference based outcome measures. Qual Life Res 1993; 2: