increasing effort Petr Mariel Department of Applied Economics III (Econometrics and Statistics), University of the Basque Country

Transcription

1 Hybrid discrete choice models: gained insights versus increasing effort Petr Mariel Department of Applied Economics III (Econometrics and Statistics), University of the Basque Country Avda. Lehendakari Aguirre, 83 E48015 Bilbao, Spain Tel: Fax: Jürgen Meyerhoff a,b a) Institute for Landscape Architecture and Environmental Planning Technical University of Berlin D Berlin, Germany b) The Kiel Institute for the World Economy, Duesternbrooker Weg 120, Kiel, Germany Corresponding author 1

2 Abstract Hybrid choice models expand the standard models in discrete choice modelling by incorporating psychological factors as latent variables. They could therefore provide further insights into choice processes and underlying taste heterogeneity but the costs of estimating these models often significantly increase. This paper aims at comparing the results from a hybrid choice model and a classical random parameter logit. Point of departure for this analysis is whether researchers and practitioners should add hybrid choice models to their suite of models routinely estimated. Our comparison reveals, in line with the few prior studies, that hybrid models gain in efficiency by the inclusion of additional information. The use of one of the two proposed approaches, however, depends on the objective of the analysis. If disentangling preference heterogeneity is most important, hybrid model seems to be preferable. If the focus is on predictive power, a standard random parameter logit model might be the better choice. Finally, we give recommendations for an adequate use of hybrid choice models based on known principles of elementary scientific inference. Keywords: discrete choice, hybrid choice model, land use, random parameter logit, marginal willingness to pay, latent variable JEL: Q51, C35 2

3 1. Introduction Hybrid Choice Models (HCM) have recently become more popular in discrete choice modelling as they expand standard choice models by incorporating psychological factors that may affect decision making. Generally, HCMs extend the specification of the traditional random utility model (RUM) by incorporating additional decision protocols in order to relax the simplifying assumptions and enrich the underlying behavioural characterizations. These extensions comprise, among others, flexible disturbances (e.g., factor analytic) to mimic more complex error structures and to allow for the explicit modelling of latent psychological factors such as attitudes (Ben-Akiva et al., 2002). This paper aims at contributing to the current literature by investigating whether HCMs provide further insights that justify the higher costs of estimation. The occasion for this question is the experience gained from the development and estimation of a couple of latent variable models (Bartczak et al., 2015; Hoyos et al., 2015; Mariel et al., 2015). In all studies we found that the modelling process was very complex and costly because of the high number of coefficients and the complex likelihood function with numerous local maxima that make its maximization tricky. On the other hand, the models resulted in new insights compared to more conventional models like the Random Parameter Logit (RPL) or the Latent Class Model. Thus, more knowledge about the potential gains of HCMs seems to be valuable as the majority of choice experiment applications nowadays apply routinely approaches capable of capturing unobserved taste heterogeneity such as RPL or latent class models. Given this, the question is whether researchers and practitioners mainly interested in the outcome of choice experiments should also move ahead and include HCM to the suite of models routinely estimated as discrete choice models are being increasingly used in environmental valuation studies (see e.g. Can and Alp, 2012 or Justes et al.,2014). 3

4 Accordingly, in this paper we focus on a closer comparison of the results from an HCM and those from an RPL. As case for the comparison we choose to analyse the effect of different design dimensions on the propensity to select the status quo (SQ) option in discrete choice tasks. The literature generally provides evidence that people have a tendency to choose the SQ option disproportionately often and that this behaviour is, at least partially, triggered by the design characteristics of the choice sets used in the survey (Boxall et al. 2009; Rolfe and Bennett, 2009; Zhang and Adamowicz, 2011). The data are from a study applying a design-ofdesigns approach (Caussade et al., 2005) resulting in 16 different choice designs. Across those designs the following five design dimensions vary systematically: the number of choice sets, the number of alternatives, the number of attributes, the number of levels and the range of attribute levels (see Meyerhoff et al. 2015). Moreover, we use an attitudinal scale developed for measuring impulsivity. We were motivated to add this scale by the recently increasing interest in whether personal traits explain (stated) choice behaviour (e.g., Grebitus et al., 2013). In the following we compare the HCM and the RPL with respect to the impact of the design dimensions on the frequency of SQ choices, the distribution of the marginal WTP estimates, and the extent to which they allow insights into respondents decision making. The paper is organized as follows. Section 2 discusses the definition of the latent variables used in HCM, Section 3 describes the methodological framework and Section 4 the case study. Afterwards, Section 5 presents the main results and, finally, Section 6 is devoted to discussions and conclusions. 2. Latent variables and structural equation models A latent variable is one of the foundation stones of a structural equation model, but there is no general definition of a latent variable that includes all its applications. Non-formal definitions consider latent variables as hypothetical variables that cannot be directly 4

5 measured (MacCallum and Austin, 2000). Among the formal definitions, we find the local independence definition (Hambleton et al., 1991), the expected value definition (Lord and Novick 1968, pp ), the definition of them as nondeterministic functions of observed variables (Bentler, 1982) or the sample realization definition suggested by Bollen (2002, p. 612), which seems to be simple and flexible: A latent random (or nonrandom) variable is a random (or nonrandom) variable for which there is no sample realization for at least some observations in a given sample. Structural Choice Models (Rungie, Coote and Louvieree, 2011, 2012) combine structural equation modelling (SEM) with discrete choice models, assuming that the latent variables have random coefficients with multivariate distributions with unknown parameters. The model incorporates factors that influence the random coefficients and can influence each other through links in the structural equations. We focus in the following on related but different models called HCMs in which the latent variables represent the characteristics of individuals, typically constructs like attitudes (Ben-Akiva et al., 2002). These latent variables are treated as endogenous and related to sociodemographic characteristics in structural equations, but, at the same time, they are explanatory in measurement equations relating them to observed indicators. This type of model has been increasingly used in all fields in which discrete choice models are applied. Nevertheless, criticism of them has increased at the same pace. The most frequently used latent variable models thus far have been applied and supported in transportation by, among others, Abou-Zeid et al. (2010), Walker et al. (2010), Daly et al. (2012), Prato et al. (2012), Glerum et al. (2014), Kamargianni and Polydoropoulou (2014), Kim et al. (2014) and Paulssen et al. (2014). In a recent paper, also in transportation, Hess et al. (2013) find better performance of HCMs in terms of efficiency, represented by lower standard errors, and argue that this approach presents a theoretical advantage in terms of endogeneity bias and measurement error, but its practical implications seem limited. 5

6 Chorus and Kroesen (2014) go even further in their criticism. They state that HCMs do not support the derivation of travel demand policies that aim to change travel behaviour through changes in a latent variable, because of the non-trivial endogeneity of the latent variable regarding travel choice and the cross-sectional nature of the latent variable which does not allow for claims concerning changes in the variable at the individual level. The first argument is probably highly case specific as the endogeneity of the latent variable can be an empirically non-relevant issue. The second argument definitely needs future research, as it is not obvious how strongly the cross-sectional nature of the attitudinal information affects the performance of the HCMs. Recently, Dekker et al. (2014) investigated to what extent choices for leisure activities and related travels are driven by the satisfaction of needs of a particular leisure activity. They include in their choice model latent variables representing the anticipated level of individual needs-satisfaction by a particular leisure activity. Using a stated choice-dataset involving choices between leisure activities, they contrast regret-minimisation based discrete choice models including and excluding the subjective measurements of need-satisfaction. Their empirical results show that, not unexpected, a big portion of the unobserved heterogeneity (around 40%) in the activity specific utility levels can be attributed to anticipated needs satisfaction. In environmental valuation, the HCM has been applied, among others, by Hess and Beharry-Borg (2012), Bartczak et al. (2015), Hoyos et al. (2015), Mariel et al. (2015), and Lundhede et al. (2015). In general, they all support the finding that HCMs provide greater insights into attitudes as additional drivers of choices. Both Lundhede et al (2015) and Bartczak et al (2015) found, for example, a significant influence of age on the latent variable and subsequently on WTP estimates. In some case also gains in efficiency were achieved. Nevertheless, Dekker et al. (2013), who additionally asked follow-up questions to record 6

7 respondents response certainty, note, in a rather critical way, that this additional information does not significantly improve the explanation of the observed choices. Kløjgaard and Hess (2014), applying the HCM approach in order to investigate data from a health survey, also express scepticism about latent variable models. They found that only a small share of the overall heterogeneity was linked to the latent variable. According to their interpretation, an explanation for the weak link could be the fact that preference heterogeneity is unrelated to attitudes and perceptions, or, more precisely, that the specific attitudinal statements measured in the survey are not directly linked to preference heterogeneity. Some of the issues related to the use of latent variables in HCMs might be avoided by learning from the SEM literature. Cliff (1983), for example, gives some warnings and advice to structural modellers, reminding them of four principles of elementary scientific inference that are perfectly applicable to discrete choice models with latent variables. The first principle is that data do not confirm a model; they only fail to refute it. That is, an estimated model cannot tell us about what is not in it. Generally, it is thus recommended to estimate multiple specifications and functional forms of a model in order to better understand the underlying generating process. The second principle is that post hoc does not imply propter hoc; that is, a significant coefficient in an estimated model does not always mean causality. That principle can be related to critique by Chorus and Kroesen (2014) regarding the cross-sectional nature of the latent variable. Due to this characteristic it is not appropriate for analysis of changes in the variable at the individual level. The third principle is crucial in HCM as it states that just giving something a name does not mean that we understand it. This is directly related to the definition of a latent variable, which usually is defined through associations with a set of indicators. Cliff (1983, p. 121) states:... we can only interpret our results very cautiously unless or until we have included enough 7

8 indicators of a variable in our analysis, and have satisfied not only ourselves but sceptical colleagues and critics that we have done so. The meaning of the latent variable will always, to some extent, be wrong, and our indicators will, to some extent, be unreliable. Moreover, in HCMs the definition of the latent variables is usually neither based on theoretical foundations nor proved through empirical work. There are, however, accepted scales to measures, for example, attitudes with a tested set of questions, like locus of control (Rotter, 1975) or environmental beliefs (Stern, 2000), which can easily be incorporated in choice models. If the set of follow-up questions has not been based on theoretical findings, a preliminary exploratory multivariate analysis should at least be applied to confirm the structure of the underlying constructs. The fourth principle is that ex post facto explanations are untrustworthy. If a model has been adjusted on the basis of its fit or lack of fit to a particular data set, its statistical status is precarious until it can be tested on a new data set. Regarding that principle, a simple prediction, such as the one used in this application, can help in model comparison and can shed light on the real performance of the model and on how close the model is to the true data-generating process. 3. Model specification We use two model specifications in this paper to investigate the influence of the design dimensionality on stated choices. The first is a HCM consisting, apart from measurement equations for attitudinal indicators, of two types of structural equation, one for the choice model and one for the latent variable model. The structural equation for the choice model is based on random utility theory (RUM), which is used to link the deterministic model with a statistical model of human behaviour. Under this framework, the utility U int of alternative i for respondent n in choice situation t (from a total of T n choice occasions) is given by: 8

9 U int = V int + ε int, (1) where V int in a classical logit model depending on observable explanatory variables, which are usually attributes (x int ) and vectors of attribute parameters β. The term ε int is a random variable following an extreme value distribution with location parameter 0 and scale parameter 1. In a HCM, V int also depends on the latent variable LV n and a vector of parameters α usually representing the interaction terms of the latent and explanatory variables. Now let j n,t be the alternative chosen by consumer n in choice situation t, such that P n,t (j n,t ) gives the logit probability of the observed choice for consumer n in choice situation T t. The logit probability of consumer n s observed sequence of choices is P n = n t=1 P n,t (j n,t ). The second structural equation for the latent variable is given by LV n = h(z n, γ) + ω n, (2) where h(z n, γ) represents the determinist part of LV n and the specification is h( ), which is in our case linear, with Z n being a vector of the socio-demographic variables of respondent n, and γ being a vector of parameters. Additionally, ω n is a normally distributed random disturbance with zero mean and standard deviation σ ω. In our case, the latent variable should represent the level of impulsivity of the respondents. Measurement equations use the values of the attitudinal indicators as dependent variables, and explain their values with the help of the latent variables. The l th indicator (of the total of L indicators) for respondent n is therefore defined as: I ln = m(lv n, ζ) + v n, (3) where the indicator I ln is a function of the latent variable LV n and a vector of parameters ζ. The specification of v n determines the behaviour of the measurement model and depends on the nature of the indicator. Responses to impulsivity statements in our case study are collected 9

10 using a Likert type response scale, so that the measurement equations are given by typical ordinal logit (Mariel et al., 2015) in which, apart from the parameters ζ, the corresponding thresholds τ need to be estimated. The model is finally estimated by maximum simulated likelihood. The estimation involves maximizing the joint likelihood of the observed sequence of choices (P n ) and the observed answers to the attitudinal questions (L Iln ). The two components are conditional on the given realization of the latent variable LV n. Accordingly, the log-likelihood function of the model is given by integration over ω n : N L l=1 LL(β, γ, ζ, τ) = n=1 ln (P n L Iln ) g(ω)dω. (4) ω Thus, the joint likelihood function (4) depends on the parameters of the utility functions included in (1), the parameters for the socio-demographic interactions in the latent variable specification defined in (2), and the parameters for the measurement equations defined in (3). Daly et al. (2012) describe different identification procedures. We follow the Bolduc normalization by setting σ ω equal to 1. All model components are estimated simultaneously and are contrasted using PythonBiogeme (Bierlaire, 2003, 2008) and Ox (Doornik, 2001). The benchmark model for the hybrid setting described above is a typical RPL model in which we assume that β n is a vector of the true, but unobserved, taste coefficients for consumer n. We assume that β n is distributed over consumers with density g(β, Ω). In this case, if P R n,t (j n,t β) gives the logit probability of the observed choice for consumer n in choice situation t, the logit probability of consumer n s observed sequence of choices is: T n t=1 P R n (Ω) = P R n j n,t β g(β Ω)dβ. (5) β The log-likelihood function for the observed choices is then: N LL(Ω) = n=1 ln (P R n (Ω)). (6) 10

11 4. Case study The survey aimed at measuring preferences for land use changes in Germany. 1 Thus, the selected choice attributes comprise share of forest, land consumption, biodiversity conservation and a price attribute (Table 4.1). All attributes except those concerning biodiversity conservation were presented in all designs, while the biodiversity attributes were used to adjust the number of attributes according to the design plan proposed by Hensher (2004). Following this approach, 16 separate efficient designs were created using C-efficiency allowing for minimizing the variance of WTP (Scarpa and Rose 2008). The designs were optimized for a MNL model. Table 4.1: Attributes used in the Choice Experiment Attribute FOREST LAND BIO BIO_AGRAR BIO_FOREST BIO_URBAN BIO_OTHER1 BIO_OTHER2 BIO_OTHER3 COST Description Percentage changes in the share of forest (positive and negative) Percentage changes in land conversion for housing development and traffic (positive and negative) Biodiversity in the whole landscape including all landscape types Agricultural landscape biodiversity Forest landscape biodiversity Urban area biodiversity Biodiversity in other landscape types: forests, urban areas, mountains, water Biodiversity in other landscape types: urban areas, mountains, water Biodiversity in other landscape types: mountains, water Contribution to a landscape fund in per year Table 4.2 provides an overview of the 16 designs and of how the dimensions of the choice sets vary across designs. All choice tasks included an SQ alternative, i.e., a zero price option with no environmental changes, plus two or more alternatives depending on the design-of-designs plan. Choices in the choice experiment regarding landscape changes had to be made by considering the landscape within a distance of about 15 kilometres from the respondent s place of residence. Respondents for the nationwide online survey were recruited 1 See Meyerhoff et al. (2015) for more details of the design of the choice experiment and the survey. 11

12 from a panel of a survey company. Each respondent was randomly allocated to one of the 16 designs. Table 4.2: Design overview Design Sets Alternatives Attributes Levels Range Interviews completed Base % % Base % % % % Base % % % Base % Base % 76 Note: The number of interviews does not include those respondents who always chose the SQ option. The questionnaire also included scales to capture different attitudes or personality traits of the respondents. One of these was a scale developed for measuring impulsivity. The scale is meant to provide a measurement instrument that allows the psychological trait of impulsivity to be recorded in an economic way, i.e., in a way that consumes only a small amount of interview time. The scale follows the UPPS (Urgency Premeditation Perseverance and Sensation Seeking Impulsive Behavior Scale) approach. Kovaleva et al. (2012) point out that there is still no standard definition of impulsiveness but that it is assumed that the construct is multidimensional and thus comprises various aspects of impulsive behaviour. These include, among others, i) the tendency to act without thinking and without sufficient information for a decision, ii) the tendency to prefer a smaller immediate reward, and iii) the tendency to choose riskier alternatives or the inability to assess the risks associated with decisions correctly. Therefore, the UPPS approach comprises the four subscales urgency, 12

13 intention, endurance and willingness to take risks. Each subscale is addressed using two items. Table 4.3 reports the wording of the attitudinal statements and the direction of the association with the latent construct impulsivity. Kovaleva et al. (2012) show that their scale performs well and allows a reliable and valid measurement of impulsivity. Table 4.3: Attitudinal questions impul1 urgency + Sometimes I do things impulsively that I shouldn't do impul2 + I sometimes do things to cheer myself up that I later regret impul3 intention - I usually think carefully before I act impul4 - I usually consider things carefully and logically before I make up my mind impul5 endurance - I always bring to an end what I have started impul6 - I plan my schedule so that I get everything done on time impul7 willingness to + I am willing to take risks impul8 take risks + I am happy to take chances The scale was added to the survey in order to shed light on the link between respondents psychological traits and their stated choices in the survey. We expect that respondents who tend to be more impulsive are more likely to choose alternatives with a positive price, i.e., not the SQ option, and that this intensifies when the choice sets become more complex with a higher dimensionality. The reason for this is that people who are said to be more impulsive are, among other things, expected to be more likely to act without reflecting on the consequences and to be more likely to take risks (Kovaleva et al., 2012). To some extent, however, the scale, which was provided by a leading social science research centre in Germany (GESIS - Leibnitz Institute for the Social Sciences), was added in an experimental manner as we expected it to be a reliable measurement instrument enabling us to estimate HCMs. The literature applying latent variable models indicates that not using reliable measurement instruments reduces the possibility of estimating an HCM. 13

14 5. Results Table 5.1 describes the variables used in the econometric models, along with their descriptive statistics. Non-responses to items mean that the useable sample comprises 23,118 responses from 1,661 individuals. Briefly, the mean age is 42.3 years, the share of female respondents is 53% and the mean disposable income of the respondents households is 17,500 Euros. As the survey was conducted as an online survey, we did not expect the sample to be representative for the population in Germany. Not all people have access to the Internet and, above all, not all use it regularly. Obvious deviations exist for the variables education and income. Compared to the German population, the share of respondents with higher education is too large and thus the disposable incomes are also too high. However, as we did not plan to aggregate, for example, welfare measures based on the model results, we assume for the following that the model comparison is not affected by the sample composition. Table 5.1: Summary statistics Variable (Attribute) Description Mean Std.Dev. Min Max AGE Age MAN Gender: Male HIGHEDUC Level of education > secondary INCOME Income 17, , ,000 POSITION Position of the choice set ALTERNATIVES Number of alternatives ATTRIBUTES Number of attributes WIDE Wide level range NARROW Narrow level range LEVEL3 Three level range LEVEL4 Four level range In addition to the socio-economic information, the respondents were asked a series of attitudinal questions regarding impulsivity, as presented in Table 4.3. Table 5.2 shows the response distributions on a 5-point Likert scale. For each statement, values closer to five would 14

15 equate to stronger agreement while values closer to one would equate to stronger disagreement. Table 5.2: Responses to the impulsivity attitudinal questions impul1 4% 35 % 24% 32% 5% urgency impul2 10% 40% 24% 24% 2% impul3 1% 10% 17% 58% 14% intention impul4 1% 8% 18% 58% 15% impul5 1% 3% 11% 60% 25% endurance impul6 2% 16% 16% 50% 16% impul7 3% 29% 26% 37% 5% willingness to take risks impul8 1% 20% 30% 43% 6% Note: 1 = doesn t apply at all, 5 = applies completely As a first step, an exploratory factor analysis was conducted on the responses to the attitudinal questions. The exploratory factor analysis employed principal axis factor analysis. According to Table 5.3, it seems reasonable to choose a two-factors solution, as the percentage of variance explained decreases sharply in the third factor, and the highest factor loadings appear in the columns for Factors 1 and 2. A HCM with all eight attitudinal questions and two latent variables would have a very high number of parameters (82), which could lead to numerical issues in the estimation procedure. As parsimony is also an important issue for model development, we estimated numerous alternative model specifications and selected a subset of questions using as criterion the significance of the parameters ζ in the measurement equations (3). This is the reason why only three attitudinal questions (impul1, impul7 and impul8) have finally been included in the HCM incorporating therefore only one latent variable representing the first factor (Table 5.3). This one latent variable solution is also in line with the definition of our attitudinal questions, as impul1 is related to urgency and impul6 and impul7 to willingness to take risks. Our latent variable represents, therefore, urgency and risk propensity. What is pursued here is the satisfaction of the third principle introduced by Cliff (1983) stating that just giving something a name does not mean that we understand it. In our 15

16 case we chose three indicators of clearly stated theoretical concepts basing our decision on a factor analysis of our data. Table 5.3: Exploratory factor analysis Eigenvalues and percentages Factor loadings Factor Eigenvalue Proportion Cumulative Variable Factor1 Factor2 Factor3 Factor4 Factor impul Factor impul Factor impul Factor impul Factor impul Factor impul Factor impul Factor impul As outlined in Section 3, the specification of a HCM requires the specification of two types of structural equations, one for the choice model and one for the latent variable model. Following equation (1), the structural equation for the choice model has a deterministic term V int, defined in our case as: V int = β X int = (ASC i + α ASCi LV n ) + (β FOREST + α FOREST LV n ) FOREST int +(β LAND + α LAND LV n )LAND int + β BIO BIO int exp(β COST +α COST LV n ) COST int, (7) where FOREST, LAND, BIO and COST are the choice attributes described in Table 4.1 and α ASCi = 0 i SQ. The attribute BIO is substituted by the corresponding split attributes in designs including BIO_AGRAR, BIO_FOREST, BIO_URBAN, BIO_OTHER1, BIO_OTHER2 and BIO_OTHER3. In addition, we include alternative specific constants ASC i for all but one of the alternatives. Note that the functional form of equation (7) resembles an RPL with the key attributes (FOREST, LAND and COST) being random to allow for a more straightforward comparison of the results obtained from the two models. According to (7), and apart from the key attributes, ASC SQ is also assumed to be random, which allows a possible SQ effect caused 16

17 by impulsivity and/or complexity of the design to be analysed. In the RPL the coefficients β FOREST, β LAND and ASC SQ are assumed to be normally, and β COST to be log-normally, distributed, which is in line with (7). Moreover, we assume that there is a vector of individual characteristics and complexity variables that affects the mean of these random parameter distributions. To make the two competing models similar, we include in the vector affecting the mean of the random parameters the same variables as those included in the determinist part of the latent variable LV n defined in (2). Figure 5.1: Empirical distributions of random parameters Share of forest Land conversion Cost ASCsq Note: solid line represents the individual contributions of each random parameter and dashed line a normal density (log-normal for Cost). As the selection of the parameters distribution is a key issue in the RPL methodology we applied the empirical approach proposed by Hensher and Greene (2003) to describe graphically the empirical distributions for the random parameters. Due to this procedure the same model for different data subsets are estimated. These subsets are created by leaving one individual out. The differences in the parameter estimates obtained by the use of these subsets and the parameter estimates of the whole sample provide the contribution (incremental marginal utility) of a specific individual to the overall sample mean parameter estimate and they can, therefore, indicate the type of underlying individual preference heterogeneity. Figure 5.1 shows the shape of these individual contributions for each random parameter (solid line) together with a normal density (dashed line). The cost coefficient, however, is plotted with lognormal dashed density. The lognormal distribution (with a sign 17

18 change), assumed for the cost parameter, assures finite moments for the WTP distributions (Daly, Hess, and Train; 2012). Figure 5.1 shows that there are no sizeable deviations of the individual contributions from the previously assumed density shapes for the random parameters. Table 5.4 presents the maximum simulated log-likelihood estimation obtained from the RPL using 200 Halton draws. The high number of observations and the high number of different utility function specifications due to the complex design do not allow for using more Halton draws as this would increase estimation costs drastically. However, both models were estimated by two different software packages (PythonBiogeme and Ox) and by using various sets of starting values to prove the stability of presented results. The estimated means and standard deviations of all random coefficients are presented in the upper part of Table 5.4, together with estimated coefficients representing the heterogeneity in mean. The lower part of the same table presents the estimations of the non-random coefficients. Table 5.4: Random parameter logit estimation Observations: Log_L: Respondents: 1661 AIC: Parameters: 58 BIC: CAIC: Share of Land forest conversion Cost ASC SQ p- p- Value value Value p-value Value value Value Mean *** < < St. Dev *** < < < <0.01 Mean heterogeneity: Choice task position ** *** < *** <0.01 Number of alternatives *** < *** <0.01 Number of attributes *** < *** < Wide level range *** < ** *** < *** <0.01 Narrow level range *** < *** < Three levels *** < ** 0.01 Four levels ** *** < Age *** < ** 0.01 Male *** < Higher Education *** < Other coefficients: Biodiversity-Whole *** <0.01 Biodiversity- Agricultural *** <0.01 Biodiversity-Forest *** <0.01 Biodiversity-Urban *** <0.01 Biodiversity-Other *** <0.01 Biodiversity-Other *** <0.01 p- value 18

19 Biodiversity-Other *** <0.01 ASC *** <0.01 ASC *** <0.01 ASC *** <0.01 Table 5.5 presents the maximum simulated log-likelihood estimation results of the HCM obtained using also only 200 Halton draws. The upper part of the table presents the estimations of the key attributes together with corresponding LV effect coefficient (α). The coefficients (γ) of the structural equation of the LV defined in (2) are on the left hand side of the table and the coefficients of the measurement equations (ζ) on the right hand side. These are presented together with the thresholds estimated using the ordinal logit model (defined as τ l, τ l + δ 1l, τ l + δ 2l, τ l + δ 3l ) for the three attitudinal response scales. Table 5.5: HCM model estimation Observations: Log_L: Respondents: 1661 AIC: Parameters: 43 BIC: CAIC: Share of forest Land conversion Cost ASC SQ Value p-value Value p-value Value p-value Value p-value - Coefficient *** < *** < *** < *** <0.01 Effect of the LV *** < *** < *** < *** <0.01 Structural equation Measurement equation parameters Choice task position *** <0.01 Thresholds and constants Number of alternatives *** <0.01 τ *** <0.01 Number of attributes ** <0.01 δ *** <0.01 Wide level range δ *** <0.01 Narrow level range δ *** <0.01 Three levels *** <0.01 Four levels τ *** <0.01 Age δ *** <0.01 Male *** <0.01 δ *** <0.01 Higher Education δ *** <0.01 Other coefficients: Biodiversity-Whole *** <0.01 τ *** <0.01 Biodiversity-Agricultural *** <0.01 δ *** <0.01 Biodiversity-Forest *** <0.01 δ *** <0.01 Biodiversity-Urban *** <0.01 δ *** <0.01 Biodiversity-Other *** <0.01 Biodiversity-Other *** <0.01 Biodiversity-Other *** <0.01 Coefficients of the LV ASC *** 0.01 ζ ** 0.03 ASC *** <0.01 ζ ** 0.05 ASC *** <0.01 ζ **

20 A comparison of Tables 5.4 and 5.5 leads to the following conclusions. The estimates of the non-random coefficients, as well as the coefficient mean values of the Share of forest and Land conversion coefficients, are very close. The main difference between the models can be found only in the Cost and ASC SQ coefficients. Many design dimensions and sociodemographic variables have significant effects on the mean of the random coefficients in RPL and on the LV in the HCM model. A direct comparison, however, is not possible because the RPL is more flexible, in the sense that it allows different impacts of these variables on the mean of each random coefficient, whereas in the HCM this effect is modelled through a latent concept of impulsivity and is therefore assumed to be the same for all coefficients. Given that the ζ coefficients in (3) are negative, high values of the latent variable correspond to less impulsive, and more risk averse, individuals. Thus, as α ASCSQ is negative, more impulsive individuals with high risk propensity are more likely to choose an alternative different from the SQ option, and that confirms our a priori expectations. Next, based on the results from Tables 5.4 and 5.5, we simulate the marginal WTP values and the distribution of the ASC SQ for the sample population of respondents, using 10,000 draws of the corresponding normal (ASC SQ, Share of forest, Land conversion) and lognormal (Cost) distributions, taking into account the heterogeneity in mean coefficients. Similarly, the simulated marginal WTP and the distribution of ASC SQ for the HCM model are computed by using 10,000 draws for the LV of each respondent and taking into account the coefficients of the structural equations for the LV. Table 5.6 presents the distribution of the WTP of the two models obtained for the two attributes Share of forest and Land conversion. As can easily be seen, the median values are similar, but the distribution of the WTP obtained by RPL is much wider. This could indicate a better performance of the HCM in terms of less variation of WTP values (Hess et al., 2013). However, we have to be cautious with the comparison as the large intervals in the RPL are likely to be, at least partially, driven by the 20

21 heavy tailed lognormal distribution of the cost coefficient. Nevertheless, the cost coefficient in the HCM is also modelled in lognormal-like way ( exp(β COST +α COST LV n ) COST int ). Table 5.6: Distribution of marginal WTP obtained by RPL and HCM models RPL 25th percentile Median 75th percentile Share of forest Land conversion HCM 25th percentile Median 75th percentile Share of forest Land conversion As the means of the random coefficients in the RPL model, as well as the latent variables, depend on various design dimensions and socio-demographic variables, the WTP values can be simulated for specific subgroups of respondents. The Tables 5.7 and 5.8 demonstrate how the distributions of WTP change under different scenarios characterized by different values of two design dimension variables. This allows us to analyse the effect of these variables on the WTP distributions. Table 5.7: Effects of position in a series of choice occasions on the marginal WTP distribution Share of forest - RPL Land conversion - RPL Position 25th perc. Median 75th perc. Position 25th perc. Median 75th perc. Low (<5) Low (<5) High (>13) High (>13) Share of forest - HCM Land conversion - HCM Position 25th perc. Median 75th perc. Position 25th perc. Median 75th perc. Low (<5) Low (<5) High (>13) High (>13)

22 Table 5.8: Effects of number of alternatives on the marginal WTP distribution Alternatives 25th perc. Share of forest - RPL Median 75th perc. Alternatives 25th perc. Land conversion - RPL Median Low (3) Low (3) High (5) High (5) Alternatives 25th perc. Share of forest - HCM Median 75th perc. Alternatives 25th perc. Land conversion - HCM Median Low (3) Low(3) High (5) High (5) th perc. 75th perc. The effects in Tables 5.6 and 5.7 are, as expected, in the direction of the sign of the corresponding heterogeneity in mean coefficient (RPL) or the structural equation coefficient (HCM). These effects are not always in the same direction in both approaches as the models rely on different assumptions. If we focus on shifts in the median values, we conclude that these are not as large as we would expect in all cases presented above but, nevertheless, they are too large to be ignored. For example, the mean WTP value for the Share of forest attribute changes from 1.2 to 4.2 in RPL and from 2.2 to 3.0 in HCM as a consequence of the change in the number of alternatives from 3 to 5. Using the same procedure, the distribution of ASC SQ was simulated in the RPL and HCM models under different scenarios. Table 5.9 characterizes the changes in those distributions attributable to design dimension variables. For example, the two approaches confirm that a choice task appearing later in the sequence of tasks increases the utility of the SQ alternative, leading to a higher probability of it being chosen. This can be due to the fatigue effect (e.g., Boxall et al., 2009). On the other hand, as expected, in the two approaches a higher number of alternatives has an opposite effect that is, more alternatives leads to a lower probability for the SQ choice. The same result was obtained in Oehlmann et al. (2014). 22

23 Table 5.9: Distribution of ASC SQ under different scenarios RPL HCM 25th perc. Median 75th perc. 25th perc. Median 75th perc. Position Position Low Low High High Alternatives Alternatives Low Low High High Attributes Attributes Low Low High High Wide level range Wide level range No No Yes Yes Narrow level range Narrow level range No No Yes Yes Next, we compared the performance of the two models by two simple approaches. First, in a similar way to the simulation of the marginal WTP distributions, we simulated the probabilities of each alternative based on the sample population of respondents, using 10,000 draws. If we assume that the highest probability coincides with the choice prediction, we get the classification tables of observed and predicted outcomes presented in Table The results are presented in percentages. As can be observed, there are only minor differences between the results for both models. Both models, RPL and HCM, predict very similarly, but at the same time also poorly. Table 5.10: Classification table of observed and predicted outcomes RPL Predicted Predicted Observed HCM 23

24 2 If we transform the information into one indicator, defined as R Count = ( 1 ) n N j jj, where n jj is the number of correct predictions for outcome j that are located on the diagonal 2 2 cells of the two tables, we get R Count = for the RPL model and R Count = for the HCM model. If we make our prediction more realistic and use in each simulation step a draw from a uniform [0,1] to generate a choice prediction based on the predicted probabilities, then 2 the values R Count drop slightly to and in the RPL and HCM models respectively. Unsurprisingly, the difference is also very small. If we analyse the contribution of the attributes and attitudinal questions to the 2 prediction in more detail, we can subtract from the numerator and denominator of R Count the number of cases in the outcome with the highest frequency (in our case outcome 3), and we 2 obtain an adjusted R Count which is, in our case, for RPL and for HCM. Our knowledge of attributes and attitudinal questions, compared to a prediction based only on the marginal distributions, reduces the error in prediction by only 11.4% and 11.8% respectively. There are other simple indicators related to observed and unobserved heterogeneity that can be used to compare the RPL and HCM. The random coefficients are an appealing part of the RPL, but we would certainly prefer to interpret a model in which the unobserved heterogeneity represents only a small part of the random coefficients. The same is true for the HCMs. Actually, an RPL-like definition of the HCM coefficients (β FOREST + α FOREST LV n ) is a nice way to disentangle the preference heterogeneity through the use of the underlying construct. To achieve this goal, the coefficients γ in (2) should be sufficiently big so that h(z n, γ) represents a high proportion of the total variation of the latent variable. Table 5.11 represents the ratios of the variances of observed and unobserved heterogeneity. For the HCM model, the table represents the ratio of the variances of h(z n, γ) and ω n defined in (2) and computed by the use of the same simulations as those used in the above prediction exercise. The values in the RPL column have been computed in a similar way. 24

25 Table 5.11: Observed/unobserved heterogeneity ratios RPL HCM Share of forest Land conversion Cost ASC SQ As can be observed from Table 5.11, the ratios are low but this finding is not unusual in the literature (Dekker et al., 2013; Kløjgaard and Hess, 2014). 6. Discussion The objective of this paper was to investigate whether the insights gained from HCMs, which have been applied more frequently in the recent literature, justify the additional effort. We used as a case a data set based on design-of-designs approach allowing for the analysis of the influence of choice task complexity on model outcomes. Regarding the influence of the design dimensions we find that both the HCM and the RPL model show that the design dimensions influence the WTP distribution. The results are obviously not exactly the same for the two models, as the more flexible RPL specification allows us to see different effects of the design dimensions on WTP for each attribute. Both approaches, moreover, confirm that all the design dimensions in the analysis influence the marginal WTP values, and, subsequently some conclusions can be drawn. Firstly, it is important to choose the design dimensions of choice sets carefully as they can significantly influence the outcomes. Our results show that the highest influence corresponds to the number of alternatives and the number of attribute levels. Secondly, the design dimensions are also related to the frequency of SQ choices. According to our results, more alternatives for the choice set have a negative impact on the frequency of SQ choices. This can be explained by the so-called preference matching effect (Zhang and Adamowicz, 2011), i.e., giving respondents more alternatives on a choice set increases the probability that they find an alternative that matches their preferences. By 25

26 contrast, the number of choice tasks faced by a respondent positively affects the frequency of SQ choices, i.e., the later in the sequence of choice sets, the higher the propensity to choose the SQ alternative. This might be caused by respondent fatigue at the end of the sequence of choice sets. To what extent learning and fatigue take place while responding to a discrete choice experiment is, however, still under investigation (see for a recent study Campbell et al., 2015). In this study we have only focused on the design dimensions and have not incorporated other aspects of complexity such as the total number of level changes or the similarity of alternatives measured, for example, through entropy (e.g., Zhang and Adamowicz, 2011). Therefore, we might not have captured all those aspects of complexity that influence the propensity to choose the SQ alternative. The reason for this is that we wanted to focus here on the comparison of the models. Readers interested in the relationship between the other aspects complexity and SQ choices are thus referred to Oehlmann et al. (2014). Finally, regarding the effect of impulsivity on the propensity to choose the SQ option, we conclude that more impulsive and risk-seeking people are more likely to choose a non-sq alternative. The findings add to an increasing evidence about the relationship between personality traits and choices (Farizo et al., 2016). The main objective of this paper, as stated in the introduction, was to compare, more closely than is usually done, an HCM with the more commonly used RPL model. The comparison includes performance, the insights gained through the estimation and the subsequent post-estimation analysis. We therefore believe that our results add new insights to the ongoing debate regarding the performance and additional value of HCMs (Chorus and Kroesen, 2014; Dekker et al., 2014, Kløjgaard and Hess, 2014; Vij and Walker, 2015). The two competing models in our case study were specified in a similar way so that their comparison would be relatively easy. The two models allow for preference heterogeneity of three key attributes. One part of this heterogeneity is linked to the dimensionality of the choice tasks and to socio-demographic variables. The other part remains random. The main difference 26

27 between the two approaches is that the taste heterogeneity in the RPL model is not linked to any underlying latent attitudes. Thus, a comparison in terms of model fit is not straightforward. Some authors compute the LogL-value of competing models corresponding only to the choice part of the model. However, this procedure is debateable as the loglikelihood function is maximized taking into account all the parameters of the model. This is why in the literature the debate about the suitability of the HCM usually remains in the discussion of the actual differences in the implied sensitivities of alternative model specifications. The work of Glerum et al. (2014) is an exception, presenting an interesting validation of their model in relation to the fourth principle of the SEM literature that ex post facto explanations are untrustworthy. They estimate the HCM on 80% of the data and compute the choice probabilities for the remaining 20% of the data. Assuming that the highest predicted probability corresponds to the chosen alternative, the authors compare this to the actual choice. They also use the ρ 2 as an additional indicator of the validity of the HCM in comparison to a plain MNL model, concluding, unsurprisingly, that the HCM performs better. A different approach was presented by Kløjgaard and Hess (2014), who try to disentangle the influence of the latent variable, but their conclusion is not very optimistic. Only a small share of the overall heterogeneity is linked to the latent variable that explains only slightly more than 6% of the total variance. The validation of the HCM should therefore be an important part of any empirical application based on HCM methodology, as the criticism of this approach basing on empirical evidence (Kløjgaard and Hess, 2014) and theoretical foundations (Chorus and Kroesen, 2014, Dekker et al., 2013) has increased considerably. If we focus on our comparison of the performance of the two models, the first conclusion, based on the prediction exercise (Table 5.10), is that the two models perform very similarly and that no great differences can be found in their prediction outcomes. The second 27