Methods for the Statistical Analysis of Discrete-Choice Experiments: A Report of the ISPOR Conjoint Analysis Good Research Practices Task Force

Transcription

1 Methods for the Statistical Analysis of Discrete-Choice Experiments: An ISPOR Conjoint Analysis Good Research Practices Task Force Report Methods for the Statistical Analysis of Discrete-Choice Experiments: A Report of the ISPOR Conjoint Analysis Good Research Practices Task Force A. Brett Hauber, PhD, Senior Economist and Vice President, Health Preference Assessment, RTI Health Solutions, Research Triangle Park, NC USA Juan Marcos Gonzalez, PhD, Senior Research Economist, Health Preference Assessment, RTI Health Solutions, Research Triangle Park, NC, USA Catharina G.M. Groothuis-Oudshoorn, PhD, Assistant Professor, Health Technology and Services Research, University of Twente, Enschede, Netherlands Thomas Prior, PhD Candidate, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA Deborah A. Marshall, PhD, Canada Research Chair, Health Services and Systems Research; Associate Professor, Department of Community Health Sciences, Faculty of Medicine, University of Calgary; Alberta Bone and Joint Health Institute, Calgary, AB, Canada Charles Cunningham, PhD, Professor, Department of Psychiatry & Behavioural Neuroscience, Jack Laidlaw Chair in Patient-Centered Health Care, Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada Maarten J. IJzerman, PhD, Professor of Clinical Epidemiology & Health Technology Assessment, Chair, Department Health Technology & Services Research, University of Twente, Enschede, the Netherlands. John F.P. Bridges, PhD, Associate Professor, Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA * Authors represent the ISPOR Conjoint Analysis Statistical Analysis Good Research Practices Task Force 1

2 Abstract Conjoint analysis is a stated-preference survey method that can be used to elicit responses that reveal preferences, priorities, and the relative importance of individual features associated with health care interventions or services. Conjoint analysis methods, particularly discretechoice experiments (DCE), have been increasingly used to quantify preferences of patients, caregivers, physicians, and other healthcare stakeholders. Recent consensus-based guidance on good research practices, including two recent ISPOR task force reports Conjoint Analysis Use in Health Studies - a Checklist: A Report of the ISPOR Conjoint Analysis in Health Good Research Practices Task Force (the Checklist) (Bridges et al., 2011) and Constructing Experimental Designs for Discrete-Choice Experiments: Report of the ISPOR Conjoint Analysis Experimental Design Task Force (Johnson et al., 2013) have aided in improving the quality of conjoint analyses and discrete-choice experiments in health economics and outcomes research. However, uncertainty regarding good research practices for the statistical analysis of data from DCEs persists. There exist multiple methods for analyzing DCE data. Each analysis method has different characteristics and can potentially yield outputs that differ quantitatively. Understanding the characteristics and appropriate use of different analysis methods is critical to conducting a well-designed DCE study. This report will assist researchers in evaluating and selecting from among alternative approaches to conducting statistical analysis of DCE data. In this report, we first present a very basic DCE example and a simple method for using the resulting data. We then present an archetypal case of a DCE a three-attribute, two-alternative, forced-choice and one of the most common approaches to analyzing data from such a question format multinomial logit. We then describe some common alternative methods for analyzing these data. This report briefly introduces the theoretical underpinnings of using DCEs to elicit preference information and the strengths and weaknesses of each alternative analysis method. This report does not endorse any specific method; however, it does provide a guide for selecting a method that is appropriate for a particular study and for understanding the implications of the selection of statistical analysis method on what conclusions can be supported by the results. Keywords: conjoint analysis, discrete-choice experiment, statistical analysis, stated preference methods, patient-centered outcomes research. 21 2

3 22 Background to the Task Force The ISPOR Conjoint Analysis Statistical Analysis Good Research Practices Task Force is the third ISPOR Conjoint Analysis Task Force. It builds on two previous task force reports, Conjoint Analysis Applications in Health A Checklist: A Report of the ISPOR Good Research Practices for Conjoint Analysis Task Force (Bridges et al., 2011) and the ISPOR Conjoint Analysis Experimental Design Task Force (Johnson et al., 2013). The Conjoint Analysis Checklist report developed a 10-point checklist for conjoint analysis. Checklist items included: 1) the research question, 2) the attributes and levels, 3) the format of the question, 4) the experimental design, 5) the preference elicitation, 6) the design of the instrument, 7) the data-collection plan, 8) the statistical analysis, 9) the results and conclusions, and 10) the study s presentation. This first task force determined that several items, including experimental design (Checklist Item #4) and methods for analyzing data from conjoint analysis studies (Checklist Item #8), deserved more detailed attention. Thus, the ISPOR Conjoint Analysis Experimental Design Task Force focused on experimental design to assist researchers in evaluating alternative approaches to this difficult and important element of a successful conjointanalysis study. This third task force report focuses on the range of options available to researchers to analyze data generated from studies using a particular type of conjoint analysis the discrete-choice experiment and the types of results generated by each method. This report also describes the issues researchers should consider when evaluating each analysis method and factors to consider when choosing a method for statistical analysis. The Conjoint Analysis Statistical Analysis Good Research Practices Task Force proposal was submitted to the ISPOR Health Science Policy Council for evaluation in December The Council recommended the proposal to the ISPOR Board of Directors, and it was subsequently approved in January Researchers experienced in stated preferences and discrete-choice experiments working in academia and research organizations in Canada, the Netherlands and the United States were invited to join the task force s leadership group. The leadership group met via regular teleconference to identify and discuss current analytical techniques, develop the topics and outline, and prepare drafts of the manuscript. An international group of analytical experts was consulted during this process to discuss methods for analysis of conjoint analysis studies and to review the task force s draft reports. (This background to the task force section will include details of the review process and be completed before submission to Value in Health.) 3

4 INTRODUCTION Over the past two decades there has been a rapid increase in the use of conjoint analysis to measure the preferences of patients and other stakeholders in health applications (Ryan and Gerard, 2003; Bridges et al., 2008; Marshall et al., 2010; debekker-grob et al., 2012; Hauber et al., 2013a, Clark et al., 2014). While early applications highlight the importance of process utility (Ryan, 1999; Longworth et al., 2001), applications now frequently focus on patient preferences for health status (Hauber et al., 2010; Mohamed et al., 2010), screening (Marshall et al., 2010), prevention (Poulos et al., 2011; van Gils et al., 2011), pharmaceutical treatment (Bridges et al., 2012; Hauber et al., 2013b), therapeutic devices (Ho et al., 2015; Sanders et al., 2010), diagnostic testing (Groothuis-Oudshoorn et al., 2014; Plumb et al., 2014) and end-of-life care (Fishman et al., 2009; Hall et al., 2013). In addition, conjoint analysis methods have been used to study decision making among stakeholders other than patients including clinicians (Nathan et al., 2012; Faggioli et al., 2011; Arellano et al., 2015), caregivers (Morton et al., 2012; Faggioli et al., 2011), and the general public (Honda et al., 2105; Regier et al., 2015). Conjoint analysis is a broad term that can be used to describe a range of stated-preference methods that have respondents rate, rank, or choose from among a set of experimentally controlled profiles consisting of multiple attributes varying across levels. The most common type of conjoint analysis used in health economics and outcomes research is the discrete-choice experiment (DCE) (Clark et al., 2014; de Bekker-Grob et al., 2012). This Task Force report focuses on motivating and reviewing the most common statistical methods that are currently used to analyze data from a DCE. The primary objective when analyzing DCE data is to statistically decompose the determinants of the choice between profiles. In doing so, one can infer the importance of attributes and levels included in the profiles. The premise of DCE is that choice is motivated by the attributes of the alternatives considered by decision makers. By controlling these attributes experimentally and observing choices among populations of interest, DCE provides the principles that reverse engineer choice to estimate the implied importance of the attributes and attribute levels behind these choices. We refer to these estimates of strength-of-preference, which are sometimes called part-worth utilities, as preference weights. Because preference weights in a DCE are estimated on a common scale, they can be used to calculate ratios describing the tradeoffs respondents are willing to make among the attributes. Examples of these tradeoffs include estimates of money equivalence (willingness to pay) (Gray et al., 2015; Johnson P et al., 2014), risk equivalence (maximum acceptable risk) (Ho et al., 2015; Kauf et al., 2015), or time equivalence (Whitty et al., 2015; Gonzalez et al., 2013) for various changes in attributes or attribute levels. 4

5 In contrast to choice-based conjoint analysis studies conducted in market research or marketing science, DCEs in health economics and outcomes research tend to focus more on quantifying preferences for changes in attributes and attribute levels and less on predicting choices. To that end, most statistical analyses of DCE in health applications data are geared toward estimating a model to test hypotheses regarding strength of preference, importance, and tradeoffs among attributes rather than estimating the model with the greatest predictive power. The application of DCEs to measuring preferences for health and healthcare has benefited from a growing literature on methods (Ryan and Farrar 2000; Bridges, 2003; Carlsson and Martinsson 2003; Viney et al., 2005; Lancsar and Louviere 2008; Johnson and Mansfield 2008) including the two previous task force reports from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Conjoint Analysis Good Research Practice. This report builds on the first of these reports, Conjoint Analysis Applications in Health A Checklist: A Report of the ISPOR Good Research Practices for Conjoint Analysis Task Force (Bridges et al., 2010). The Checklist outlines the steps to take for the development, analysis, and publication of conjoint analyses, including experimental design (Checklist Item #4) and methods for analyzing data from conjoint analysis studies (Checklist Item #8). While there are several other key methodological references that may be useful to more experienced researchers (Louviere et al., 2000; Hensher et al., 2005; Orme, 2010), this task force report is targeted to a more general audience of healthcare researchers. Understanding the characteristics and appropriate analysis of preference data generated by DCE surveys is critical to conducting a welldesigned DCE. This report will assist in understanding some aspects of the preference information obtained through DCEs that should be taken into account as researchers evaluate and select between alternative statistical analyses methods to analyzing DCE data. As such, this Task Force report is a primer on statistical methods as much as a statement of good research practices. The Task Force agreed on this approach because, despite the growing use of conjoint analysis methods in health economics and outcomes research, there remains inconsistency in the statistical methods used to analyze data from DCEs (Bridges et al., 2011; Bridges et al., 2008; Marshall et al., 2010; Johnson and Mansfield, 2008). This report also highlights the fact that there is no single best way to analyze DCE data. Therefore, this report concludes with a checklist of principles to consider when designing a DCE and selecting an appropriate approach to statistical analysis. This report starts with the basic idea behind deriving preference weights from DCE data by describing two simple approaches to calculating preference weights from a DCE with a very limited number of attributes and levels. The purpose of providing these examples is to help readers understand some of the basic properties of choice data. We then present an archetypal case a three-attribute, two-alternative, forced-choice DCE analyzed using a multinomial logit model consistent with the random utility model of choice (McFadden, 1974; Louviere et al 5

6 ; Bateman et al, 2002; Ryan and Ferrar, 2000). Because most of the other commonly used methods for analyzing DCE data are, in effect, variations on the multinomial logit model, we then describe extensions to multinomial logit that can be used to analyze the same DCE data. These extensions include random-parameters (mixed) logit, hierarchal Bayes, and latent class analysis. To demonstrate the differences in the properties of each of these analysis methods we present the results of each method as applied to a common simulated data set. A SIMPLE EXAMPLE To understand the basic concepts underlying the analysis of DCE data, consider the case of a hypothetical, over-the-counter analgesic. If we assume that the relevant analgesics can be described by three attributes with two possible levels (time to onset of action can be 30 minutes or 5 minutes; duration of pain relief can be 4 hours or 8 hours; and formulation can be either tablets or capsules), then these can be combined into 8 possible profiles. While these 8 profiles can be combined into a very large number of distinct pairs, we could use a main-effects orthogonal design (Johnson et al., 2013) to generate an experimental design consisting of four paired profile tasks (Table 1). Table 1 Profile A Profile B Task 1 30 minute onset, 4 hour duration, tablet 5 minute onset, 8 hour duration, capsule Task 2 30 minute onset, 8 hour duration, capsule 5 minute onset, 4 hour duration, tablet Task 3 5 minute onset, 4 hour duration, capsule 30 minute onset, 8 hour duration, tablet Task 4 5 minute onset, 8 hour duration, tablet 30 minute onset, 4 hour duration, capsule If there were three respondents who completed all four choice tasks, we would have 12 observations with which to estimate a model. Assume respondents answered as follows: Respondent 1 (B, B, A, B), Respondent 2 (A, B, A, A), and Respondent 3 (A, B, B, A). The simplest form of analysis is to count how many times each level of each attribute was chosen. In Table 2, we count the number of times each attribute level was chosen by each respondent, sum these totals across all respondents and then divide this sum by the number of times each attribute was presented across the three respondents to calculate a score for each attribute level. Although there appears to be some heterogeneity in preferences among the respondents (e.g., Respondent 1 appears to prefer capsules to tablets while Respondents 2 and 3 appear to prefer tablets to capsules), we can still infer sample-level preferences from these data. Across the sample: 5 minute onset was preferred to 30 minute onset, 4 hour duration was preferred to 8 hour duration, and tablets were preferred to capsules. 6

7 Table 2 Level Resp. 1 Resp. 2 Resp. 3 Sum Score 30 minute onset minute onset hour duration hour duration Tablet Capsule We can also use regression analysis to linearly relate the probability of choosing one profile over another to the characteristics of the profiles. This model is known as a linear probability model and assumes that the probability of choosing one profile (we assume that is Profile A in our example) over another is a linear function of attribute level differences between profiles in each choice task. Thus, in our example above the model can be described as, Pr(choice = Profile A) = β o + β i (X i, A X i, B ) i [1] Where X i is the level of attribute i, β 0 is the intercept, and β i is the preference weight for attribute i. The linear probability model defines a relationship between choices and attribute levels that can be leveraged to estimate the preference weights through various linear regression models. One such estimator is ordinary least squares (OLS). With OLS, the value of the set of β in the equation above is determined by determining the value for the set that minimizes the sum of the squared residuals. To set up the data for regression analysis in this example, let Quick=1 if the onset of action is 30 minutes and Quick=0 if it is 5 minutes in profile A, Duration=1 if the duration is 4 hours and Duration=0 if it is 8 hours in profile A, and let Tablet=1 if the formulation is a tablet and Tablet=0 if the formulation is a capsule in Profile A. Finally, let Choice=1 if profile A was chosen and 0 if Profile B was chosen. Table 3 presents the data from this experiment, using the above coding. Table 3 Respondent Choice Task Choice Quick Duration Tablet

8 Using OLS on the binary choice variable Y (1 for Profile A and zero if Profile B is chosen) to estimate preference weights for the attribute levels used to define the analgesics in this example, the model takes the form: Y = α + β 1 Quick + β 2 Duration + β 3 Tablet + ε, [2] and the conditional expectation of the binary variable Y equals the probability that Profile A is chosen. The estimated linear probability function is then: Pr(choice=Profile A) = Pr(Y=1) = Quick Duration Tablet. [3] The coefficients from this simple model can be interpreted as marginal probabilities: an analgesic with a 30-minute onset is 33% less likely to be chosen than an analgesic with a 5-minute onset, an analgesic with a 4-hour duration is 33% more likely to be chosen than an analgesic with an 8-hour duration, and an analgesic in the form of a tablet is 33% more likely to be chosen than an analgesic in the form of a capsule. From this we can infer that, on average, respondents in the sample prefer faster onset of action, shorter duration of effect, and tablets over capsules. Note that with results from a linear probability model, all the preference information we have is perfectly confounded with the probability of choice associated with changes in attribute levels. That is, the measure of how much more a respondent prefers a 5-minute onset versus a 30-minute onset (as with any other change in attribute levels) is the marginal change in the probability of choice for treatments that differ only in the time of onset of action. Comparing the results of this simple regression to the results of the counts analysis, we see that the estimated regression parameter on each attribute is simply the difference between the sample-level scores for the levels of that attribute. Specifically, the score for 30-minute onset 8

9 was 0.33 less than the score for 5-minute onset, the score for 4-hour duration was 0.33 higher than the score for 8-hour duration, and the score for tablet was 0.33 higher than the score for capsule. Although users can expect OLS estimates to have standard errors associated with each preference weight, these standard errors are problematic and inferences about the significance of estimates are typically not done with OLS results. This is because the OLS estimator is subject to severe limitations when it is used to analyze DCE data. When using OLS, a researcher must assume that the errors with which they measure choices are independent and identically distributed with mean zero and constant variance (Cox, 1970). In reality, the variance in DCE data changes across choice tasks. In addition, even with estimators other than OLS, linear-probability models can produce choice probabilities that are greater than one or less than zero for certain combinations of attribute levels. For these reasons, among others, linear probability methods are rarely used to analyze DCE data. THE ARCHETYPAL CASE To describe the more common alternatives for analyzing data from a DCE, we define an archetypal case. As above, we define a DCE in which each choice task presents a pair of alternatives. Respondents are asked to choose between the two profiles in each pair. Each profile is defined by three attributes, and each attribute has three possible levels. This case can thus be referred to as a three-attribute, two-alternative, forced-choice experiment. In the archetypal case, each profile is a medication alternative. The three attributes that define each medication alternative are efficacy, a side-effect, and a mode of administration. Efficacy is the treatment outcome measured on a numeric scale between 0 and 10 where higher values represent better outcomes. The levels of the side effect are levels of severity (mild, moderate, severe). The levels of the mode of administration are daily tablets, weekly subcutaneous injections, and monthly intravenous infusions. The attributes and levels used to create the profiles in the archetypal case are presented in Table 4 and an example of a choice task is presented in Figure Table 4. Attributes and Attribute Levels in the Archetypal Case Attributes Levels A1 Efficacy L1 10 (best level) L2 5 (middle level) L3 3 (worst level) A2 Side effect L1 Mild 9

10 L2 Moderate L3 Severe A3 Mode of administration L1 1 tablet once a day L2 Subcutaneous injection once a week L3 Intravenous infusion once a month Figure 1. An Example of a Choice Task for the Archetypal Case Feature Medicine A Medicine B Efficacy Severity of side effects How you take the medicine 10 on a scale from 1 to 10 where 10 it the best Severe Subcutaneous injection once a week 5 on a scale from 1 to 10 where 10 is the best Mild Intravenous infusion once a month Which medicine would you choose? Variable Coding in the Archetypal Case Defining the levels in each row of data for the archetypal case can be accomplished in multiple ways. Attributes with numeric levels (e.g., survival time, risk, cost) can be specified as continuous variables. Using this approach, the level value of the attribute in the profile will appear in the appropriate place in the row. In the archetypal case, only efficacy can be logically coded as a continuous variable because the levels of the side effect and mode of administration are descriptive and thus categorical. Two commonly used methods for categorical coding of attribute levels are dummy-variable coding and effects coding (Hensher, Rose and Green 2005; Bech and Gyrd-Hansen, 2005). In each of these coding approaches, one level of each attribute must be omitted. In both dummy- 10

11 variable coding and effects coding each non-omitted attribute level is assigned a value of 1 when that level is present in the corresponding profile and 0 when another non-omitted level is present in the corresponding profile. The difference between the two coding methods is related to the coding of the non-omitted levels when the omitted level is presented in the profile. With dummy variable coding, all non-omitted levels are coded as 0 when the omitted level is present. With effects coding, all non-omitted levels are coded as -1 when the omitted level is present. Table 5 presents dummy coding and effects coding for an attribute with three levels when L3 is the omitted level. The same coding can be applied to each attribute in the archetypal case. Table 5. Dummy-Variable Coding and Effects Coding for the Archetypal Case Attribute level Dummy-variables Effects-coded variables presented in the L1 L2 L1 L2 profile L L L With dummy-variable coding, all coefficients estimated by the model represent a measure of preference for each level of an attribute relative to the omitted level of that attribute. The coefficient on the omitted category of an effects-coded variable can be recovered as the negative sum of the coefficients on the non-omitted levels of that attribute and thus yields a unique coefficient for each attribute level included in the study. However, each coefficient is estimated relative to the mean attribute effect and statistical tests of significance for each coefficient are not direct tests of the statistical significance of differences between estimated coefficients on two different levels of the same attribute. Dummy or effects coding lead to the same estimates of differences in preference weights between attribute levels (Mark and Swait, 2004). For this reason, the decision to use dummy or effects coding of variables should be based on ease of interpretation of the estimates from the model, not based on the expectation that one type of coding will provide more information than the other. It is important to note that with dummy coding, whenever a constant is added to a model to characterize relative preferences for any specific alternative in the choice questions, the value of this constant represents the preference for the alternative when the alternative assumes 11

12 all the omitted attribute levels in the model 1. The same is true with effects-coded variables, except that with effects coding, the omitted levels correspond to the average effect of the study attributes. Thus, the constant can be interpreted as the preference for the alternative when all attributes are set at their average effect. In other words, the constant in such a case becomes a measure of the average preference for the alternative given the attribute levels in the study. Data Generated by the Archetypal Case One way to set up the data generated by this DCE is to construct two rows of data for each choice task for each respondent, one row per alternative. So, if each respondent is presented with 10 choice tasks generated by an appropriate experimental design, and there are 200 respondents, the total data set will have 4000 rows. In our example with 2 alternatives, the first row of data for each choice task will include the attribute levels that appear in the first profile in the pair presented in that choice task. The second row of data for the same choice task will include the attribute levels for the second profile in that pair. In addition, each row of data will include a choice dummy variable equal to 1 if the chosen profile for the choice task corresponds to that row of data or 0 otherwise. In Table 6, we present an example of this type of data setup for the example choice task presented above using effects coding for all attributes assuming that level 3 is the omitted level for each attribute. Table 6. Data Setup for one Response to the Example Choice Task Using Effects Coding ID Task Choice Eff (L1) Eff (L2) SE (L1) SE (L2) Mode (L1) Mode (L2) The ID number in the first column of Table 6 is the respondent number. Task is the number of the choice task in the series. Choice indicates that the first respondent choose Medicine B when presented with the choice task in Figure 1 above. The level of side effects in Medicine A in the example choice task is severe (the omitted level). Therefore, the non-omitted levels of this attributes are coded as -1 in the first line of data in 1 This may be desirable if the combination of all omitted levels represents a clinically meaningful profile. Then, all preference weights in the model describe deviations from this profile and statistical differences from the levels in the profile. 12

13 Table 6. In addition, the level for the mode of administration in Medicine B is the omitted level for this attribute and the non-omitted levels for this attribute are both coded as -1 in the two columns farthest to the right in the second line of data in Table MULTINOMIAL LOGIT The most common approach to analyzing data generated by this type of experiment is the use of a limited-dependent variable model that relates the probability of choices among two or more alternatives to the characteristics of the individuals making the choices and/or the elements describing those alternatives. This model is commonly known as a multinomial logit (MNL) model or a conditional logit (CL) model. Both models rely on the same statistical assumptions about the relationship between choice and the variables used to help explain choice. MNL is often used to describe the model that relates choices to the characteristics of the respondents choosing, while CL relates choices to the elements describing those alternatives. When this distinction is made in software packages, MNL does not produce estimates for variables that only change across alternatives, and CL does not produce estimates for variables that only change across respondents. Often, however, the distinction between the two models is not made and the two models are characterized as MNL. In a DCE, the elements describing the alternatives are the attribute levels used to define each profile in the choice task. The MNL model was shown by McFadden (McFadden, 1974) to be consistent with random utility theory (RUT) 2. McFadden originally applied this framework to observed transportation choices. His work laid the foundation for what is now known as conjoint analysis (Louviere et al 2000; Bateman et al, 2002) involving hypothetical or stated choices. Using RUT, the utility associated with an alternative or profile is assumed to be a function of observed characteristics (attributes levels) and unobserved characteristics of the alternative. This theoretic framework also assumes that each individual, when faced with a choice between 2 or more alternatives, will choose the alternative that maximizes his or her utility. The utility function is specified as an indirect utility function defined by the attribute levels in the alternative plus a random error term reflecting the researcher s inability to perfectly measure utility: U i = V(β,X i ) + ε i [4] 2 The novelty in McFadden s use of the multinomial logit model is that he applied this model to choice behavior that was consistent with economic theory and derived a regression model that relates choices to the characteristics of the alternatives available to decision makers. McFadden used the term conditional logit to describe this innovation. For simplicity, we will use multinomial logit to describe both the estimator and the application of the estimator to choice experiments. 13

14 where V is a function defined by the attribute levels for alternative i, ε i is a random error term, X i is a vector of attribute levels defining alternative i; and β is a vector of estimated coefficients. Each estimated coefficient is a preference weight and represents the relative contribution of the attribute level to the utility that respondents assign to an alternative. In the MNL model, ε j is assumed to follow an independently and identically distributed type 1 extreme value distribution. The assumption of the extreme-value distribution of ε j results in a logit model: 223 Pr(choice = i) = ev(β,x i ) e V(β,x j ) j [5] where V(β,x i ) is the observed portion of the function for alternative i, and i is one alternative among a set of j alternatives. Simply stated, the probability of choosing alternative i is a function of both the attribute levels of alternative i and the attribute levels of all other profiles presented in a choice task. In the case of the two-alternative, forced-choice DCE, there are two alternatives in each choice task, so j=2. The probability of choosing one profile from the set of two alternatives is 1 minus the probability of choosing the other profile in that choice task. Therefore, neither alternative in the choice task has a choice probability less than 0% or greater than 100%. In addition, this specification implies that the closer the probability of choosing an alternative in a 2-alternative choice task is to 50%, the more sensitive the probability of choosing that alternative is to changes in the attribute levels that define the alternative Results of the Multinomial Logit Model Using Effects-Coding and Dummy-variable Coding Table 7 presents a set of example results from a MNL model regression with effects-coded variables, while Table 8 presents results from the same model where the variables were dummy coded. Details regarding the simulated data set used to demonstrate the models described in the remainder of this report are presented in an appendix. In a MNL model, a coefficient (or preference weight) and a corresponding standard error are estimated for all but one level of each attribute. A t-value (or sometimes a z-value 3 ) and a p-value or probability that the estimated coefficient 3 With large sample sizes, parameters in the regression models are assumed to be approximately normally distributed. With such an assumption about the distribution of the parameters, statistical significance can be tested using a z-value that indicates how many standard deviations from the parameter mean is the value for the most common null hypothesis value, zero. 14

15 is zero, are often calculated for each estimated preference weight. In some statistical packages, confidence intervals at the 95% confidence level are also provided for each preference weight. The table also includes AIC, BIC, and log likelihood values; all measures of model fit discussed later in the report. Table 7. Example Results from a Multinomial Logit Model Using Effects Coding Attribute Level Coefficient Std. Error t-value p-value Efficacy L <0.01 L Side Effect L <0.01 L Mode of Administration L L <0.01 Log Likelihood Log likelihood of model without predictors AIC BIC Table 8. Example Results from a Multinomial Logit Model Using Dummy-variable Coding Attribute Level Coefficient Std. Error t-value p-value Efficacy L <0.01 L Side Effect L <0.01 L Mode of Administration L L <0.01 Log Likelihood of model Log likelihood of model without predictors AIC BIC

16 With effects-coded variables, the preference weight on the omitted level is recovered by calculating the negative of the sum of the estimated preference weights for all non-omitted levels of the attribute. For example, in the results in Table 7 the estimated preference weight for the omitted level of efficacy would be equal to (-0.28 = -[ ]). If the attribute-level variables in the regression model are dummy coded, the estimated preference weight can be interpreted as difference in preference weights between that level and the omitted level. Also, since the preference weight for the omitted attribute level is assumed to be zero, the t-value and p-value in the output are associated with the likelihood that the estimated preference weights are statistically different from the estimated preference weight of the omitted level. Interpreting the Results of the Multinomial Logit Model The preference weights for the effects-coded and dummy-coded MNL models are presented in Figures 2 and 3, respectively. Both figures clearly indicate that higher levels of efficacy are preferred to (have higher preference weights than) lower levels of efficacy, less severe side effects are preferred to more severe side effects, and subcutaneous injections are preferred to 1 tablet once a day which, in turn, is preferred to an intravenous infusion once a month. 16

17 256 Figure 2. Preference Weights for the Archetypal Case Estimated Using the Multinomial Logit Model and Effects-Coded Attribute Levels

18 Figure 3. Preference Weights for the Archetypal Case Estimated Using the Multinomial Logit Model and Dummy-Coded Attribute Levels Figures 2 and 3 demonstrate that the differences in the preference weights among levels within each attribute are the same in both models. The effect of increasing efficacy from L2 to L1 is 0.24 (0.24 = ) in the dummy-coded model and 0.24 (0.19 = ) in the effects-coded model. The effect of reducing side effects from L2 to L1 is 0.30 (0.30 = ) in the dummy-coded model and 0.30 (0.30 = 18

19 ) in the effects-coded model. The similarities between the effects-coded and the dummy-coded models highlight an important aspect of the MNL model. That is, the absolute values of preference weights alone have no meaningful interpretation. Preference weights measure relative preference, which means that only changes between attribute-level estimates and the relative size of those changes across attributes have meaningful interpretations. In the effects-coded and dummy-coded models that we have used in the examples, directly comparing the parameter estimates for the efficacy attribute in each model would have erroneously suggested that the two models provided different results or different information on the relative impact that these attribute levels have on choice. As the figures demonstrate, differences between the two model specifications are merely due to differences in variable coding and no additional information is obtained from either model. This implies that estimates from MNL models cannot be compared across models directly 4. This interpretation of the model estimates, in fact, applies to all the models presented later in this report. Knowing that the two models provide the same information, we will use only the results of the effects-coded model presented in Table 7 and Figure 2 to further interpret the results generated by the MNL model as applied to the archetypal case. First, the difference in preference weights between the best or most preferred level of an attribute and the worst or least preferred level of the same attribute provides an estimate of the relative importance of that attribute over the range of levels included in the experiment. For example, the relative importance of a change in efficacy from a value of 10 to a value of 3 (a 7-point change) is 0.54 (0.26 [-0.28]). The value of a change in side-effect severity from severe to mild is 0.66 (0.32 [-0.34]). Therefore, reducing side-effect severity from severe to mild yields approximately 1.2 (0.66 / 0.54) times as much utility as increasing efficacy by 7 points (from 3 to 10). Likewise, changing the mode of administration from an intravenous infusion once a month to a subcutaneous injection once a week yields only 0.72 (0.39 / 0.54) times as much utility as increasing efficacy by 7 points. These results can also be used to estimate the rate at which respondents would be willing to trade off among the attributes in the experiment. For example, a reduction in side-effect severity from moderate to mild yields an increase in utility of The reduction in efficacy (from the highest level of 10) that exactly offsets this increase in utility would be approximately 5.4 points. This change in efficacy is calculated as the 0.24 reduction in utility achieved by moving from an efficacy value of 10 to an efficacy value of 5, plus the change in efficacy between a value of 5 and an efficacy value of 3 that would yield a reduction in utility of The necessary additional change in utility is equal to one fifth of the difference (0.06 / 0.30) between an efficacy value of 5 and an efficacy value of 3. Interpolating between these 2 points and adding the value to 4 Results from models that share the same specification can be compared following Swait and Louviere, 1993 or Hensher, Rose and Greene,

20 the change from an efficacy value of 10 to and efficacy value of 5 results in a total change in efficacy from 10 to 4.6 necessary to offset the increase in utility from reducing the severity of the side effect. Preference Weight Estimates Although the results presented in Table 7 (Figure 2) and Table 8 (Figure 3) appear to be different at first glance, this difference is solely due to the reference point for the estimated preference weights in each model. The relationship between the two sets of results can be confirmed by transforming the results from the dummy-coded model as deviations from the attribute mean effects. For example, based on the results in Table 8 the mean of the preference weights for the efficacy attribute is 0.29 (0.29 = ( )/3). Subtracting this mean effect from each estimated preference weight for the efficacy attribute we obtain estimates of the preference weights that are equal to the preference-weight estimates obtained using the effects-coded data as shown blow. Efficacy L1 = = 0.26 Efficacy L2 = = 0.02 [6] Efficacy L3 = = The relationship between attribute levels across models is the same for all three attributes. For example, in the case of the side-effect attribute the preference weights of the dummy-coded results can be transformed as shown below: Side effect L1 = = 0.32 Side effect L2 = = 0.02 [7] Side effect L3 = = Goodness of Fit The relationship between the dummy-variable coded and effects-coded models can also be confirmed by looking at a measure of goodness of fit for each model. Contrary to OLS, MNL models do not allow for the calculation of an R-squared measure of goodness of fit. Instead, several measures that mimic an R-squared calculation have been developed using the log likelihood (LL) of these models. The measures are known as Pseudo R-Squared measures and provide a way to determine relative model fit, as opposed to absolute fit as R-squared measured do with OLS results. Table 7 and Table 8 each present two LL values for the effects-coded and dummy-coded models, respectively. In 20

21 each table, there is one LL value for the model specification, and another one for a model without predictors (a model that includes only a constant for all but one of the alternatives available). Log likelihood is an indicator of the maximum explanatory power of a model. Higher (less negative) values in the LL are associated with greater ability of a model to explain the pattern of choices in the data. Although LL values alone cannot be used as a measure of model fit because they are a function of sample size, they can be used to calculate goodness of fit measures like the likelihood ratio chi-square test and McFadden s Pseudo R-Squared, which are commonly reported in software packages. The likelihood ratio chi-square test determines significance based on a Chi-squared distribution and provides a way to determine whether the fit of a model is significantly improved with the attribute level variables relative to a model without any of these attribute level variables in essence providing users with a test equivalent to an F-test in OLS regressions. That is, indicating whether one or more of the preference weights is expected to be different from zero. The critical value for this test can be simply calculated as the following: Likelihood Ratio Χ 2 = 2(LL of model LL model without predictors). [8] The likelihood ratio has a chi-squared distribution with degrees of freedom equal to the number of preference weights estimated in the model. Models that improve fit more significantly with the attribute level variables can be assumed to have better fit. Although the calculation above is what typically is provided in software packages, extensions of this tests provide a way to test for improvements in fit between an unrestricted model (that estimates all preference weights in a model) and a restricted model (that forces one or more preference weights to be zero). In that case, the critical value for this test can be simply calculated as the following: Likelihood Ratio Χ 2 = 2(LL of restricted model LL of unrestricted model) [9] With degrees of freedom equal to the number of restrictions (preference weight estimates forced to be zero). For example, an unrestricted model may be one that estimates preference weights for all attributes in a DCE, whereas the restricted model forces the weights for one of the attributes to be zero (in essence eliminating the attribute from the analysis). In such a case, the likelihood ratio chi-square test informs the user whether estimating the weights for the eliminated attribute improves model fit in a statistically significant way. 21

22 McFadden s Pseudo R-Squared is calculated using the following formula: McFadden s Pseudo R 2 = 1 LL of model LL model without predictors [10] McFadden s Pseudo R-Squared can be zero if all the preference weights are constrained to be zero, but the measure can never reach 1. Note that McFadden s Pseudo R-Squared for the effects-coded model in Table 7 is ( = 1-[ / ]). McFadden s Pseudo R-Squared for the dummy-coded model in Table 8 is also ( = 1-[ / ]). The two models have the same exact goodness of fit indicating that both models explain choices equally well. Although McFadden s Pseudo R-Squared provides a measure of relative (rather than absolute) model fit, a measure between 0.2 and 0.4 can be considered a good model fit. (McFadden, 1978) Other measures of model fit that are commonly provided by statistical packages are the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Both criteria are specified as -2LL+Kγ, where LL is log likelihood value of the full model, K is the number of parameter estimates, and γ is a penalty constant that changes between AIC (γ=2) and BIC (γ=ll*sample size). The premise of the calculations behind these measures is different from the calculation of Pseudo R-Squared measures. AIC and BIC evaluate the plausibility of the models focusing on minimizing information loss rather than evaluate improvements in the adequacy of the model to explain responses in the data as done in the Pseudo R-Squared measures. Also, contrary to Pseudo R-Squared measures, models with lower AIC and BIC measures are preferred over models with higher measures. As with McFadden s Pseudo R-Squared, AIC and BIC measures are identical across the models in Table 7 and Table 8. The Multinomial Logit Model with a Continuous Variable An alternative specification of the MNL as applied to the archetypal case is presented in Table 9. In this specification, all attribute levels are effects-coded except for the efficacy attribute, which was coded as a linear continuous variable with the efficacy values shown (3, 5, and 10) in each alternative. Since only one efficacy variable was necessary to characterize treatment efficacy in the model, the output in Table 9 has only one preference-weight estimate for the efficacy attribute. The preference-weight represents that relative marginal utility of a 1-unit change in the efficacy outcome. All changes in the efficacy of more than one unit are assumed to be proportional to a 1-unit change. That is, the effect of a 1- unit change in efficacy on utility is assumed to be linear over the range of levels of this attribute included in the experiment. For example, the 22

23 utility change resulting from a 9-unit change in the efficacy measure is assumed to be 9 times the marginal utility of a 1-unit change in the efficacy measure. Two pieces of evidence suggest that this specification may be inappropriate given the data in this experiment. First, the effects-coded and dummy-variable coded models suggest that the relative marginal utility of a 1-unit change in the efficacy measure is more important when the change in efficacy is between 3 (Level 3) and 5 (Level 2) than when the change in efficacy is between 5 (Level 2) and 10 (Level 1). This is determined because the difference in the relative preference weights between levels 1 and 2 of the efficacy attribute is 0.24 (implying a 0.05 change in preference weight associated with a 1-unit change in the level of efficacy over this range of levels) and the difference in the relative preference weights between levels 2 and 3 of the efficacy attribute is 0.30 (implying a 0.15 change in preference weight associated with a 1-unit change in the level of efficacy over this range of levels). The second indication that the continuous specification of efficacy may be incorrect is the difference in the goodness of fit between the model in which the efficacy attribute is modeled as a linear continuous variable and the models in which the efficacy attribute is modeled as a categorical variable. This change in model fit is evident if we calculate McFadden s Pseudo R-Squared, which is ( = 1-[ / ]). The reduction in the model goodness of fit suggests that the proportionality assumption imposed with the marginal effect of the efficacy attribute does not fit the data as well as the categorical preference weights in the effects-coded and dummy-variable coded models. In addition, the calculated AIC and BIC results suggest that the use of a linear continuous efficacy coding is less appropriate than the previous models because both measures are higher (i.e., less desirable) for the model with a continuous specification for efficacy than the equivalent measures calculated for the effects-coded and dummy-coded models. Table 9. Example Results from a Multinomial Logit Model with Continuous Efficacy Attribute Level Coefficient Std. Error t-value p-value Efficacy Marginal effect 0.07 < <0.01 Side Effect L <0.01 L Mode of Administration L L <0.01 Log Likelihood of model