Measuring Risk and Time Preferences and Their Connections with Behavior

Transcription

1 Measuring Risk and Time Preferences and Their Connections with Behavior Julian Jamison, Dean Karlan, and Jonathan Zinman * August 19, 2012 CITES AND SUPPORTING MATERIALS INCOMPLETE, BUT READY FOR COMMENTS * julison@gmail.com, Federal Reserve Bank of Boston, IPA; dean.karlan@yale.edu, Yale University, IPA, J-PAL, and NBER; jzinman@dartmouth.edu, Dartmouth College, IPA, J-PAL, and NBER. Thanks to Lynn Conell-Price, Hannah Trachtman, and Gordon Vermeer for outstanding research assistance, and to the Russell Sage Foundation for funding. 1

2 I. Introduction The economics profession builds models based on individual preferences. Estimating values for preference parameters is important for both testing and applying our models. For testing models, preference parameter estimates allow us to assess the predictions models make about relationships between preferences and choices or outcomes. For applying models, preference parameter estimates are inputs for measuring welfare and conducting policy analysis. We examine methods for measuring individual preferences directly not because observing choices is bad, but specifically because we want to understand better the link between preferences and choices. A revealed preference approach alone yields inferences that conflate other issues. For example, data on insurance choices may be used to estimate risk preferences, but time preferences, perceptions and unobserved individual heterogeneity in true underlying risk also drive insurance purchase decisions [cite/fn on work that does this by imposing assumptions?]. Similarly, is our tardy production of this paper due to our stable preferences for leisure versus work; to aspects of our choice set that may be mistakenly confounded with our preferences; to procrastination deriving from time-inconsistent preferences (and the absence of a commitment device with a higher cost of failure than the shame of receiving polite but increasingly firm warning s from the editors); or to a planning problem in which we systematically underestimate the time for the remaining tasks? By eliciting measures of preferences directly, and linking such measures to behavior, we can better validate the underlying model to explain choices. We consider evidence on the direct elicitation of three broad classes of individual preferences. Our coverage of risk preferences includes risk aversion as classically defined, and also ambiguity aversion and loss aversion. Our coverage of time preferences includes the classic issue of how to disentangle preference from other determinants of discount rates, and also time-inconsistency and other sources of costly self-control. Our coverage of process preferences (includes regret and transaction utility) is much shorter, reflecting the lack of similar work on measurement. We thus focus more on identifying key gaps in our knowledge for further research. We do not cover atemporal preferences over different goods in a consumption bundle, or social preferences (see [cites] for reviews). Nor do we cover meta-awareness of one s (changing) preferences e.g., projection bias, sophistication/naivete about self-control problems-- about which there has been less work on direct elicitation. For both risk and time preferences we address four types of questions: methods, predictive power of actual behavior, heterogeneity across people, and within-subject stability. On methods, we describe the various commonly used elicitations, and examine how different elicitation and estimation methods affect parameter estimates. Key examples include the roles of monetary incentives versus hypothetical questions; quantitative versus qualitative questions; potential confounds researchers should consider when choosing an elicitation method (e.g., how numeracy may influence lottery choices, and disentangling risk perception from risk preference); and some quick-and-dirty methods for researchers facing budget and/or time constraints in the lab or the field. 2

3 Second, we examine how measures predict actual behavior. The primary challenge with posing this question is simple: one needs clean measurement on both sides of the correlation, free of alternative explanations that happen to also correlate with each other. Let s start with a simple example of the problem. Suppose we want to validate a model of time preferences and investment in new agricultural technology. The policy idea is simple: farmers may not invest in highly profitable investments if they are impatient and thus prefer current consumption to considerably more future consumption. So researchers conduct simple time preference elicitation questions (would you prefer money now or more money later?), and observe if they invest in a higher yield agricultural technology. The researcher finds they are correlated. What can be concluded? Perhaps also the higher yield agricultural technology requires trusting the agricultural extension agent. And so does accepting more money later rather than money immediately. If this were the case, trust, but not necessarily time preferences, would be the underlying mechanism at work. These problems with interpreting correlations between preferences and behavior are likely difficult to overcome perfectly, but must be whittled down as much as possible in order to advance our knowledge on the validity of such measures. Timing is helpful, but not dispositive. If measurement occurs, and then much later the behavior is observed, this may help eliminate reverse causality (although naturally does not remove a myriad of other unobservable correlates). This was the approach taken, e.g., in Karlan (2005) in which trustworthy behavior in a trust game was found to predict repayment of loans a year later, and in Ashraf, Karlan and Yin (2006) in which inconsistent responses to time preference questions predicted later adoption of a commitment savings product. To make this link from preferences to behavior, however, requires a strong methodological emphasis on the details on elicitation methods, and also strong contextual understanding of the decisions people face in the real world, that one is trying to model. Third, we ask how heterogeneous are estimates cross-subjects, and what are the determinants of heterogeneity? This speaks to, among other things, the descriptive power of representative agent models, and to our (limited) ability to explain, or at least fit, preferences with observable characteristics. Fourth, we examine how (un)stable parameter estimates are within-subject, across time. This question speaks to meta-questions about what preferences capture (something inviolable and deep, versus something more malleable and highly context-specific). In this context, we also discuss interventions designed to change preferences. We know of only a few cases of the latter, primarily in the context of impatience and self-control. Our approach here errs on the side of breadth, not depth. We hope that each section highlights the challenges involved in the direct elicitation of that set of preferences and their impacts on behavior, in ways that encourage further progress in the development of elicitation methods. We try to be comprehensive about identifying the issues that researchers need to confront, rather than being comprehensive about resolving said issues. Part of this is due to necessity: for most types of preferences, we could find little warranted consensus on best-practice methods. Part of this is due to taste, in that we believe many of the most important research questions here revolve 3

4 around how to disentangle specific types of preferences from other preferences, from other cognitive inputs into decision making (like expectations, price perceptions, and memory), and from elements of choice sets (like liquidity constraints, and returns to capital). II. Uncertainty A. Overview of Theories and Concepts Preferences with regard to uncertainty are generally regarded as one of the most fundamental or primitive aspects of someone s utility function. As such, inferences about the nature of these preferences are of vital importance to virtually every discipline and subfield in the social sciences. The experimental study of preferences over risk/uncertainty has breadth and depth commensurate with its importance; a thorough study would be at least book-length (as evidenced by [Cox and Harrison 2008]). As such we provide a primer rather than a manual, and refer the interested reader to Cox and Harrison [2008], and other references below, for further details. We start with the important distinction between attitudes towards and preferences over uncertainty. Attitudes, as typically defined, are a reduced-form combination of both preferences and perceptions about risk likelihood (and/or the cost/benefit of different states of the world conditional on their realization). E.g., someone may exhibit risk averse behavior because of their underlying preferences, conditional on (possibly distorted) expectations, and/or they may exhibit risk averse behavior because of their expectations. So a question that asks: Do you tend to take risks in choosing when to harvest your crops? may pick up elements of risk preference (I don t take the risky action because I have very concave utility and am not willing to expose myself to variance in income), and/or of risk perception (I don t take the risky action because I perceive bad states of the world, e.g. a heavy rainfall at harvest time, to have a high probability). We avoid the notion of strategic uncertainty, which is due to another agent s behavior rather than states of nature. 1 A brief overview of different theories helps set the stage. Expected utility theory (EUT) reduces preferences over uncertainty to risk. Agents face known distributions of probabilities of all states of the world, and perceive these probabilities accurately. Risk preferences can then be categorized in one of three ways: risk aversion (preferring a certain payoff lower than the expected value of a gamble to the gamble itself), risk-seeking (preferring the gamble to a certain payoff equal to the expected value) or risk-neutrality (linear utility). In order to better facilitate this categorization, Arrow [(1965) and Pratt [(1964) formalized two local measures of riskaversion, relative risk aversion (RRA) and absolute risk aversion (ARA,) 2 both of which are positive for risk aversion and negative for risk seeking. Studying risk preference under EUT frequently involves estimating these parameters. It can also involve sketching the utility function more globally, since the study of risk preferences under EUT is equivalent to studying the function s curvature. However, EUT implies approximate risk neutrality over anything but large 1 For instance, by observing the relative frequency of choices of the risk-dominant action in the stag-hunt game, one could measure the strategic risk-aversion of an individual. 2 RRA is defined as and ARA is defined as. 4

5 stakes [Rabin 2000], which is a problem for elicitation methods that use small stake questions to calibrate EUT models. 3 Other theories posit more complicated structures of preferences over uncertainty. Some complications are due to allowing for added richness or bias in risk perception (e.g., lack of clear or accurate expectations), as in cumulative prospect theory [Tversky and Kahneman 1992], salience theory [Bordalo et al 2012] or ambiguity aversion. Other complications are due to nonlinearities or other discreteness in preferences; e.g., loss aversion in prospect theory, preference for certainty [see, e.g., Andreoni and Sprenger _UCE_ 2012] or rank-dependent utility [Quiggin 1982]; see also Fudenberg and Levine [2012]. Hey [1997] and Starmer [2000] provide a more complete set and description of alternatives to EUT. As with EUT, researchers can use data elicited from choice tasks, in combination with various econometric approaches, to estimate model parameters and/or utility functions under various (often testable) assumptions. B. Methods 1. Methods: Elicitation and Estimation 4 There are several general methods to elicit preferences or attitudes over uncertainty. We describe each in brief, along with a summary of key limitations or concerns. Our Appendix contains a concrete example, for each elicitation method, of a choice task or survey question used in that method. We focus on direct elicitation and do not cover papers that infer preferences from field data; see e.g., [Einav et al] for a recent paper using this approach. The Multiple List Price (MPL) method [Miller et al 1969] offers choices between two or more uncertain prospects with fixed payoff amounts and varying probabilities. In the widely-used Holt and Laury [2002] version of MPL, subjects face a single list (visible all at once) of binary decisions between two gambles. The payoffs remain the same in each decision but the probabilities vary, meaning that any respondent with consistent risk preferences should have a switch point between preferring gamble A or gamble B (we discuss models that allow for a trembling hand or other types of choice-inconsistency in Section []). Tanaka et al [2010] offer a new variant of MPL that elicits utility curvature, curvature of the probability weighting, and the degree of loss aversion (and of time preference) under various assumptions. MPLs have been criticized for assuming linear utility (as discussed in Section III, discount rate estimates from MPLs are biased upward if utility is concave), and for providing interval rather than point identification of preference parameters. The Ordered Lottery Selection method [Binswanger 1980; 1981; Barr 2003] offers choices between two or more uncertain prospects with fixed probabilities and varying payoff amounts. As Harrison and Rutström [2008] discuss, it has been conventional to use 0.5 as the fixed probability, and to offer a certain option along with (several) gambles. These conventions create difficulties for estimating non-eut models but are not intrinsic features of the design: one could 3 For example: say a person turns down a gamble where he has equal probability of losing $100 or gaining $110. If the only explanation is curved utility, this implies that he will spurn any gambles with equal probability of losing $1000 or gaining any positive amount, no matter how large it is.. 4 Here we borrow especially heavily from the taxonomies and discussions in Harrison et al [2007] and Harrison and Rustrom [2008]. 5

6 use a range of probabilities to allow for estimates of probability weighting, and one could eliminate the certain option, and/or vary how the different gambles are arrayed, to test or control for framing/reference point effects [Engle-Warnick et al 2006] A few studies use methods that provide a continuum of choices (in contrast to the discrete choices posed by MPL and OLS), using linear budget constraints. (We discuss the convex time budget method later in this sub-section, and again briefly in the section on time preferences.) Andreoni and Harbaugh [2010] provide choices between gambles with probability of winning an amount x with probability p<1. Choices trade off probability for prize. Choi et al [2011] provide choices between two differently priced assets that each have the same gross payoff per unit, and a chance of paying out or paying nothing. So a (locally) risk neutral subject should allocate all experimental income to the cheapest asset, and the share of income allocated to the cheapest asset is a measure of risk tolerance. The certainty equivalent (CE) method adapts the Becker, DeGroot, and Marschack [1964] (BDM) procedure by endowing a subject with a one or more lotteries and then asking for her selling price for each lottery [Harrison 1986]. 5 The subject is told (and, in some implementations, shown) that a buying price will be picked at random, and that if the buy price exceeds (does not exceed) the subject s stated sell price, the lottery will be sold (not sold) and the subject will receive her sell price (play the lottery); i.e., she will receive (not receive) her CE. Other CE methods use a list of choices, in the spirit of MPL, rather than direct elicitation; e.g., in Tversky and Kahneman [1992] the respondent must choose between a gamble (which stays constant throughout the trial) and a series of sure payoffs. 6 The use of this choice-based method is motivated in part by work showing that it produces more internally-consistent results than direct elicitation of CEs [Bostic et al 1990]. Plott and Zeiler [2005] and Harrison and Rutström [2008] raise some concerns with standard implementations of the CE method: subjects may misunderstand the payoff structure of BDM elicitation and engage in misguided strategic misstatement of their CE s, and within-session (co-mingled) income and learning effects can confound inferences about preference parameters. Both papers offer tweaks to instructions and treatments that are designed to mitigate these concerns. The Barsky et al [1997] lifetime income gamble (LIG) battery is a (relatively small) set of hypothetical choices between a job that, with probability, either doubles lifetime income or cuts it by a (varying) fraction, and a job that pays a certain lifetime income. So the general elicitation method is a hybrid CE/MPL. LIG s innovations are the contextual focus on lifetime income (which, for many theoretical and practical applications is more interesting/important than the siloed choices often used to elicit risk preferences), and tractability: LIG is relatively quick and easy to administer. LIG has been adopted and adapted by nationally representative household surveys across the world. Its main limitations are its coarseness (although Kimball et al [2008, 2009] provide an econometric method for imputing quantitative estimates of the 5 The probability equivalent (PE) method, where a subject is asked to supply a probability at which she would play a lottery with two specified payoffs instead of taking a certain payoff, seems to us justifiably less popular (in economics at least) these days than the other methods discussed here, given concerns about probability weighting and numeracy (see Section []). 6 Specifically, the range of sure payoffs is then adjusted to include values spaced closer together and near the switching point from the first series, in order to refine the certainty-equivalent estimate. 6

7 coefficient of relative risk tolerance from LIG responses under CRRA, EUT, and other functional form assumptions), its reliance on hypothetical stakes (see Section [] below), and a potential confound with discount rates (depending on what respondents assume about the exact meaning of double ; e.g., timing of income flows). The trade-off (TO) method [Wakker and Deneffe 1996] is a variant of CE where subjects choose between pairs of lotteries. TO permits inference about risk parameters without any assumptions about whether the subject weighs or misperceives probabilities (in contrast, standard CE methods assume no probability weighting, which is paradoxical because prospect theory, the motivating theory behind many applications of CE methods, posits probability weighting). Abdellaoui et al [2007] extend the TO method to facilitate estimation of a loss aversion parameter. However, there are two thorny problems with the TO method. The first and arguably most serious is a lack of incentive compatibility that, as far as we can tell, is intrinsic to the method: subjects have an incentive to overstate their CE. The second, somewhat more tractable problem is that, since responses to any choice after the first one depend on previous responses in the TO method, there is potential for error propagation [Abdellaoui 2000]. The uncertainty equivalent (UCE) method identifies indifference between a gamble, and the probability mixture over the gamble s best outcome and zero [Andreoni and Sprenger 2012; Callen et al 2012]. To take Andreoni and Sprenger s example, consider a gamble G1 that pays $10 probability p or $30 with probability 1-p. Now consider a gamble G2 that pays $30 with probability q, and $0 with probability 1-q. The UCE is the probability q that makes the subject indifferent between G1 and G2. Andreoni and Sprenger [2012] use UCEs to test the independence axiom (see Section [] below) independence implies a linear relationship between p and q-- and the relative descriptive power of EUT, CPT, and u-v models away from and near certainty (i.e., over choices involving only risky options, and over choices involving a safe option). [critiques of this method?] Fehr and Goette [2007] provide a quick method for estimating loss aversion from two choices. One choice is between zero with certainty versus Lottery A : a 50% chance of winning 8 and a 50% chance of losing 5. The other choice is between zero with certainty versus playing six independent repetitions of Lottery A. Identifying loss aversion per se with these tasks requires two potentially problematic assumptions: linear utility over the stakes under consideration, and no probability weighting. Several surveys have asked purely qualitative questions that are designed to elicit risk attitudes. Noteworthy among these are those included in several European household surveys, which ask Do you consider yourself a risk-taker [overall, or in domain X]?, and the Survey of Consumer Finances question about the amount of financial risk that you are willing to take (with response options ranging from Take substantial financial risks expecting to earn substantial returns to Not willing to take any financial risks ). 7 The merits of this sort of approach are the easy and low-cost implementation, and the strong, positive correlations between measures of risk 7 See also Meertens and Lion s (2008) Risk Propensity Scale, which asks respondents to agree or disagree with seven unrelated statements about their risk propensity; e.g., I really dislike not knowing what is going to happen (which economists would think of as pertaining to ambiguity more than risk; see following section), and I do not take risks with my health. 7

8 tolerance inferred from responses and behavior/outcomes like wealth or risky asset holdings [Dohmen et al 2011; Stango and Zinman 2009]. It may be the case that short qualitative questions are more easily understood by respondents, and/or completed with less effort, and thereby deliver more precise and/or more informative estimates. But what exactly do these qualitative responses identify? Clearly they do not readily yield point identification of any preference parameter (although in principle they could be useful for ordinal ranking). Indeed, the evidence on whether and how responses to qualitative questions are correlated with parameter estimates obtained from choice task methods is limited, and mixed (see Dohmen et al (2011) vs. Lonnqvist et al [2011]). The data thus far is also reduced-form in the additional sense that the survey responses presumably reveal something about risk perception (e.g., the likelihoods of various states) as well as risk preference (how I value, in utility terms, the realizations of various states). In principle, one could elicit perceptions indeed, there is a large literature on eliciting risk perceptions along with attitudes, and then use the information on perceptions to help back out (ordinal) preferences. We are not aware of any papers taking this exact approach with qualitative questions, but it seems like a potentially profitable one, and has been employed with quantitative elicitations by Andersen et al [2010]. Most of the work comparing different elicitation methods uses within-subject designs, and not all of these studies (fully) control for order effects. This is a real concern given the possibility of learning, fatigue, wealth effects, and other sources of within-subject preference instability (Section []). So one should be circumspect when drawing inferences from these studies, and it seems to us that there would be value in new studies that use between-subject designs (as in Harrison et al s [2007] analysis of prize type and lottery framing), and/or within-subject designs that carefully control for order effects. More studies using non-eut specifications would also help better link this line of inquiry to the rest of the literatures. The general consensus seems to be that inferences depend significantly, and often dramatically, on elicitation method (Anderson and Mellor [2009]; Dave et al [2010]; Deck et al [2008]; Hey et al [2009]; Reynaud and Couture [2010]; Andreoni and Sprenger [2012 Uncert Equiv]; Andreoni and Sprenger [2012 estimation]). 8 This can hold even within classes of the general elicitation methods described above, as in Isaac and James [2000] and Berg et al s [2005] comparisons of different CE methods. Harrison and Rutström [2008] is an exception to the consensus that elicitation method matters a lot: they obtain similar estimates for CRRA from the MPL of Holt-Laury [2002], the random lottery pairs of Hey and Orme [1994], and the ordered lottery selection of Binswanger [1980; 1981]. Several studies conclude that some methods produce less noise (or, more precisely, greater within-subject, within-task consistency) than others. Hey et al [2009] find that MPL is more precise than CE methods. Anderson and Mellor [2009] find that correlations between Holt-Laury MPL and hypothetical gambles with time horizons are stronger among subjects who make consistent choices (a la [Choi et al 2011]). Dave et al [2010] find that a simplified Binswanger 8 Hershey and Schoemaker [1985] find substantial differences across CE vs. PE tasks for hypothethical gambles, and review the early literature on response mode biases and other sources of differences across elicitation methods using hypotheticals. 8

9 task (a la Eckel and Grossman [2002, 2008]) produces more informative CRRA estimates, among low-numeracy subjects, than a more complicated MPL task. Harrison and Rutström [2008] cover the what to do with the data question in detail, for both EUT and non-eut models, so we focus on some recent developments. One recent development is the joint estimation (and elicitation) of risk and time preferences, which has been motivated primarily by the concern that moving payoffs to the future (in order to elicit time preferences) introduces risk. Another, less-scrutinized source of motivation is that evaluating risk preferences over time horizons is often of interest (e.g., with respect to lifetime income), and this introduces potential confounds with time preference. There are two main approaches for jointly eliciting risk and time preferences (see also Section [] below). Andersen et al [2008] employ separate multiple price lists, one for riskless choices over time, or one for risky choices paid out immediately (see also Andersen et al [2011]). Laury et al. (2012 JRU) are able to approximately replicate these discounting results by using a novel risk-free intertemporalchoice task. Andreoni and Sprenger use convex time budgets (CTBs), where subjects allocate a convex budget of experimental tokens to sooner and later payments. Variation in sooner and later times, slopes of the budgets, and relative risk are used to identify parameters and test theories. Andreoni and Sprenger [2012 Estimation] use riskless choice over time to estimate utility function curvature, which is equivalent to risk aversion under expected utility. Andreoni and Sprenger [2012 RiskTime] uses CTBs with risky choices to test the common ratio prediction of discounted expected utility theory. For discussions of the pros and cons of each method, see the papers cited in this paragraph. Another recent development is the application of mixture models for classifying individuals as EUT or non-eut (e.g., CPT, or Rank-Dependent Expected Utility) types [Harrison and Rutström 2009 EE; Bruhin et al ECMA 2010; Conte et al 2011 Journal of Econometrics]. See also Choi et al [2007 AER]. These approaches seem fruitful in light of the mounting evidence of substantial preference heterogeneity across individuals (see Section []). The use of dynamic survey design (relying on real-time estimation of which subsequent question/task, from a pre-set menu, will elicit the most precise information on a subject s preferences) also seems promising. Toubia et al [2011] adapt a dynamic method used in conjoint analysis to the estimation of CPT parameters (and to estimation of a quasi-hyperbolic time discounting model as well). 2. Methods: Real Stakes versus Hypothetical The question of whether/when/why a researcher should use real stakes, rather than hypothetical questions, to elicit risk preferences has been ably addressed by, among others, Camerer and Hogarth [1999], and Harrison [2007]. 9 Focusing on comparisons of the same method with versus without real stakes, we have not been able to find much work that post-dates Harrison [2007 book chapter]. One exception is Laury and Holt [2008], which finds that evidence of asymmetry in risk preferences over gains versus losses is larger with hypothetical than real stakes, and further attenuated as the stakes get larger. Another recent contribution is von Gaudecker [AER 9 Note we refer to this as real stakes versus hypothetical, not incentivized versus not incentivized, out of respect for the great American philosopher Calvin, who said to Hobbes, verbing weirds language. 9

10 2011] which finds that stakes (or lack thereof) change parameter estimates for risk aversion and loss aversion. In all, it seems that there is a consensus that stakes matter: estimates of various risk preference moments are different under different-sized stakes. It bears noting, however, that merely finding that the two methods yield different answers does not mean that using stakes is necessarily the better one method (particularly given the cost, i.e, if merely to reduce measurement error, there is a genuine tradeoff between sample size and measurement error). More direct comparisons of how real stakes versus hypothetical elicitations correlate with (real-world) behavior would help (see section below on predicting real behavior). There are some compelling reasons for relying on hypothetical questions. For example, stakes may exceed research budgets, or gambles may be over long horizons, or complications may exist with respect to corruption and cash management in handling the cash in the field. These reasons may come to include an inherent tension between an assumption needed for standard 1-in-K payment mechanisms to make sense (the independence axiom), and the assumption (and accompanying evidence) that that same assumption is systematically violated when subjects make multiple choices (see our discussion of Harrison and Swarthout [2012], and related work, below). Ethical considerations involved in imposing losses on subjects does not, in and of itself, strike us as a compelling reason to rely on hypotheticals. One can provide subjects with an endowment and design gambles such that, even under the experiment s worst-case scenario, the subject does not lose more than her experimental endowment (although this does require some assumptions regarding how the show-up money is accounted for, mentally, by the participant). So if the default is to use real stakes, the next question is: how? Important work is emerging on how different types of incentive mechanisms affect choices, and the inferences we can make from them. Cox et al [2011] and Harrison and Swarthout [2012] find that estimates of rank-dependent utility risk preferences differ depending on the payment protocol (see also [Laury Pay One or Pay All 2006]): 1-1, where the subject makes only one choice and is paid based on that choice [Conlisk 1989], versus the standard 1-in-K payment protocol [Starmer and Sugden 1991]. This finding suggests, perhaps not surprisingly (given evidence on probability weighting), that the oftinvoked convenient assumption of an isolation effect (subjects view each choice as independent of the others) is violated. Violation of this independence axiom (IA) makes it challenging to construct incentivized tasks that elicit valid, high-powered inferences on non-eut theories where the IA does not hold. Cox et al [2011] test two new payment mechanisms that should, in theory, be incentive-compatible under special cases: Yaari s [1987] particular form of rank-dependent utility, and Schmidt and Zank s [2009] linear cumulative prospect theory. Cox et al [2011] also test two other payment mechanisms and find that they induce wealth effects, portfolio effects, and adding-up effects that can confound inferences. Andreoni and Harbaugh [2010] and Andreoni and Sprenger [2011 UCE] test the independence axiom using two different methods, and find that it holds only away from certainty (see also Callen et al [2012]). 10

11 3. Methods: Other Design Considerations Apart from the issues with stakes discussed directly above, and some rather obvious considerations (allowing for losses if you want to study loss aversion, allowing for certain options if you want to study direct risk aversion), there are several other important considerations when designing an elicitation method. For some lines of inquiry using multiple-decision tasks it may also be important to consider how visible the subject s prior decisions are, in order to disentangle effects of memory, salience, and/or framing on choices from effects of preferences One fundamental consideration, embedded in many non-eut theories, is a distinction between preferences and perceptions. If people do not have accurate perceptions of likelihoods if they subjectively weight probabilities then choices can reflect distorted perceptions (which may be amenable to correction) rather than preferences (which are inviolate over some timeframe). There are various methods for eliciting and/or estimating subjective probabilities or decision weights ; see e.g., Andersen et al [2010], Blais et al [2002], [Bruhin et al 2010], [Bruine de Bruin et al 2007], [Toubia et al 2011]. Another consideration is framing: how a task is presented or described (e.g., as an investment or a gamble ) can affect choices, independent of its economic content as defined by EUT [Peters et al 2006; Deck et al 2008]. Depending on the line of inquiry, framing effects may be second-order and easily correctable (e.g., Harrison and Rutström [2008, section 1.1]), while in other instances framing may be more central. For example, work on non-eut theories where reference points are critical (prospect theory, salience theory) must think carefully about how subjects bring reference points into, and/or form them under, the elicitation task (see, e.g., Ericson and Fuster [2010]). Work on rank-dependent utility models must pay particularly close attention to whether a given outcome is framed as a gain or a loss [e.g., Sitkin and Weingart (1995)]. Frames can have unexpected effects, however. Harrison et al [2007 ECMA] find that a frame designed to make people less risk averse (by providing smaller increments in the probability for risk loving choices than for risk averse choices) actually made people more risk averse. Beauchamp et al [2012] find that a frame designed to make people more risk-neutral (by providing the expected values of risky prospects) had no effect. A related consideration is that loss aversion creates incentives for people to use creative mental accounting [see Thaler 2004 Mental Accounting Matters ]. E.g., accounting for a loss of 10 and a gain of 20 as a net gain of 10 may deliver greater utility than booking the two transactions separately. (We are distinguishing between underlying preferences and decision making approaches heuristics, etc. operationalized in service of those preferences.) Understanding how and when subjects (strategically) bracket decisions and outcomes, on the narrow vs global continuum, may be critical for obtaining an accurate picture of preferences per se. Another emerging consideration is whether to jointly elicit multiple aspects of preferences (e.g., risk and time, risk and ambiguity). Joint elicitation can save time (Section [], and may be needed to produce unbiased estimates of preferences over uncertainty (Section []). 11

12 Another consideration is how to deal with within-subject internal inconsistency that is not explicable by simple/literal versions of any of the leading models of preferences over uncertainty. Some approaches provide econometric corrections or allowances for response error (see, e.g., Hey and Orme [1994]; Harrison and Rutström [2008]; Toubia et al [2011]). Other approaches take internal inconsistency as a behavioral object of interest [von Gaudecker et al AER 2011; Choi et al Who is More Rational; Parker and Fischoff 2005], and/or as a cognitive factor that may interact with risk preferences [Jacobson and Petrie 2009]. A closely related consideration, long a focus of experimenters, is whether subjects understand the task/question. Simpler tasks, even if they map imperfectly to theory, may produce better estimates of a concept more precise, and/or less confounded with omitted variables. Similarly, controlling for likely omitted variables (such as numeracy or cognitive ability see e.g. Frederick s [2005] cognitive reflection task ) may improve estimates but of course risk problems if the underlying preference also shifts the now-included omitted variable (e.g., if being impatient leads one to invest less in education, and thus core worse on cognitive tests, then including a control for cognition would bias the true effect of impatience on the outcome of interest). Background risk (a risk that is correlated with a risk used to elicit preferences) is another important consideration. Harrison et al [ECMA 2007] find that background risk can affect inferences, and they discuss implications for field and lab design, including controlling for and/or experimentally manipulating background risk. Harrison et al [ECMA 2007] also study the closely related question of what good to gamble over (money vs. rare coins, in their case). C. Predictive Power of Measures on Real Behavior Evidence on whether and how elicited estimates of preferences over uncertainty correlate with real-world behavior/outcomes is limited, mixed, and inconclusive, in large part due to the potential for massive omitted variables problems stemming from the (often unobserved) heterogeneity in preferences and behavior described in Sections [] and []. Cardenas and Carpenter [2010] nicely summarize the mixed state of prior evidence (focusing on studies set in developing countries), and conduct their own analysis by linking rich survey data from subjects in six Latin American cities on economic well-being (including access to credit) with incentivized tasks to elicit preferences over risk, ambiguity, and losses. They find no robust conditional correlations between risk preferences (estimated using a short Binswanger-like task where subjects were shown a ring of six binary lotteries and asked to pick one to play) and economic status, but do find some evidence that ambiguity aversion is correlated with poverty and that loss aversion is correlated with wealth. Tanaka et al [2010] find that MPL estimates of risk aversion and loss aversion are not conditionally correlated with household income (Table 4). Choi et al [2011] find no significant conditional correlation (p-value 0.12) between risk tolerance (estimated using one of the linear budget constraint methods decribed in Section []) and wealth in a representative sample of Dutch households, although the point estimate implies an economically large negative association. Of course, one can debate whether or not wealth is a behavior in the sense that smoking or stock 12

13 market participation (see below) unequivocally are. We could as easily have included these studies in the follow subsection on individual heterogeneity and correlates. Johnson et al [2010] find that risk aversion and loss aversion (estimated using the Toubia et al [2011] method) are uncorrelated with being under water on a mortgage. In the same vein, but turning more specifically to individual choice behavior, risk tolerance inferred from the lifetimeincome gamble questions are (fairly) strongly conditionally correlated with share of financial wealth in stocks [Barsky et al 1997; Kimball et al 2008], stock market participation [Hong et al 2004 Table III] and other risky behaviors like smoking, drinking, and not having insurance [Barsky et al 1997], as measured in U.S. surveys. Anderson and Mellor [2008] find that risk aversion (estimated using Holt-Laury MPL) is negatively and significantly conditionally correlated with several unhealthy behaviors and outcomes in a sample of Virginia adults. Lusk and Coble [2005] find that risk aversion (estimated using Holt-Laury MPL) is negatively correlated with eating genetically-modified food among student subjects, conditional on risk perception and some demographics. Guiso and Paiella [2008] find that risk aversion (as inferred from a CE hypothetical on a risky security) is strongly unconditionally correlated with some (risky) behaviors but not others in an Italian survey. Finally, Fehr and Goette [2007] find that only loss-averse individuals (as measured using their method described above) exhibit less effort when wages increase. Qualitative measures of risk attitudes are strongly conditionally correlated with various outcomes in intuitive ways, as discussed in Section [] above. D. Correlates (heterogeneity across subjects) Harrison and Rutström [2008, p. 130] conclude that: At a substantive level, the most important conclusion is that the average subject is moderately risk averse, but there is evidence of considerable individual heterogeneity in risk attitudes in the laboratory. The first conclusion is critical in that it casts doubt on the usefulness of EUT as a (leading) workhorse model [Rabin 2000]. The second conclusion effectively lays out a challenge: our workhorse model(s) of decision making under risk/uncertainty must be able to accommodate substantial heterogeneity in preferences (or at least in behavior). Nothing has emerged, insofar as we have seen, to change or modify those conclusions based on method, mode, setting, or subject population. To take just a few recent examples from studies based on large representative samples, 10 Von Gaudecker et al [AER 2011] find substantial dispersion in risk aversion and loss aversion inferred from incentivized MPL choices offered to a nationally representative sample of Dutch survey-takers (see also Choi et al [2011]). Kimball et al [2008; 2009] find substantial heterogeneity in CRRA, among a nationally representative sample of U.S. survey-takers, estimated from the Barsky et al [1997] hypothetical gambles over lifetime income. Dohmen et al [JEEA 2011] find substantial heterogeneity in responses to qualitative questions about risk attitudes (Section 2.3.b) in a nationally representative sample of German survey-takers. Cardenas and Carpenter [2010] find substantial heterogeneity in risk [and ambiguity?] preferences, inferred from incentivized Binswanger-like tasks, among large representative samples from six Latin American cities. 10 Other studies finding substantial heterogeneity among non-student samples include Andersen et al [JEBO 2010], Attanasio et al [forthcoming AEJ: Applied], Burks et al [2007, Figure 10], and Tanaka et al [2010 AER]. 13

14 A key issue is whether convenience samples (usually university students) have preferences that are representative of the broader population(s) of interest. Andersen et al [JEBO 2010] examine this question in Denmark using MPL and obtain similar estimates of the CRRA on average, with substantially more heterogeneity in the field (non-student-specific) sample. 11 It is worth noting that barriers to sampling from (more) representative and other non-student samples have never been lower, and seem to be falling, with development of internet survey (panels), Time-Sharing Experiments for the Social Sciences, and other channels (see the Conclusion for further discussion). Much has been made about heterogeneity in risk preferences by gender, race, parental background, height, etc. [in addition to the papers cited in this section, see also Donkers et al [2001]; Eckel and Grossman [2008]; Croson and Gneezy [2009]], but overall, there is evidence that far more cross-subject heterogeneity is due to unobserved than observed characteristics (e.g., Sahm [2007; Guiso and Paiella [2008]; von Gaudecker et al [2011]). Coupled with a subtle but fundamentally thorny issue with incentive designs (Section []), this suggests that we should be very circumspect about our ability, at this juncture, to describe the nature of heterogeneous preferences over uncertainty, in terms of the relative predictive power of specific non-eut models. Inferences about how preferences over uncertainty drive behavior (Section []) often rest on assumptions about how these preferences are (un)correlated with other inputs into decision making. This helps motivate the growing body of work that directly examines correlations between different types of preferences, cognitive abilities, and personality traits. Relationships between preferences and what we label cognitive skills (i.e., the ability to solve problems correctly) have received the most visibility thus far [Benjamin et al; Burks et al 2009 PNAS; Dohmen et al 2010 AER; Frederick 2005; Li et al 2011; Oechssler et al 2009]. All of these studies find evidence of significant (and sometimes large) conditional positive correlations between risk tolerance and cognitive skills. Correlations between preferences over risk/uncertainty and other aspects of preferences have received less attention, but strike us as no less important for informing the proper specification of theoretical and empirical models. Barsky et al [1997] find no correlation between risk tolerance and the elasticity of intertemporal substitution. Andersen et al [2008 ECMA] find a small but significant positive correlation between risk aversion and impatience (see also their Section 4 for a discussion of a few related prior studies). Wang et al [2009] find evidence of correlations between loss aversion, risk preferences, and discounting. Li et al [2011] find some weak evidence of correlations between loss aversion and discounting. See also Epper et al [2011] on the strong correlation between probability distortions and the degree of decreasing discount rates. Understanding correlations, and construct relations, between preferences and personality may also be important [Weber, Blais, and Betz 2002; Deck et al 2008; Anderson et al 2011]. Although psychologists often think of personality traits as relatively fixed and preferences as 11 See also Akay et al [forthcoming Theory and Decisions], and Drichoutis and Koundouri [2011], although DM is unclear on where their student and general population subjects are drawn from. 14

15 relatively malleable, Almund et al [2011] models preferences as fixed and personality traits as endowments that can be altered by experience and investment. E. Within subject stability How stable are preferences over uncertainty? Work on this critical question was surprisingly dormant until recently. We focus on one particular metric of stability: within-subject and withinmethod, over time periods longer than an elicitation session. (To our minds, it seems quite likely that within-session, within-method instability, which could simply be classical measurement error, is due to something other than true preferences; see, e.g., the various design considerations discussed above.) Another important metric that we do not grapple with here is (in)stability across different domains (e.g., money vs. health); see, e.g, Weber, Blais, and Betz [2002], Harrison et al [2007 ECMA], and Dohmen et al [JEEA]. There are several important issues to keep in mind when trying to evaluate (studies) of preference stability. One of course is measurement error: what appears to be instability may instead be confusion, lack of effort, etc. Another issue is attrition: many of the panel studies below often lack a thorough exploration of whether and how attrition affects the results. A key conceptual issue is the relationship between preference (in)stability and state contingencies (Hirschleifer and Riley [1992]; Chambers and Quiggin [2000]), as articulated nicely by Andersen et al [2008 IER]: preferences are still functionally stable even if they change with states of nature/opportunities ( states ), provided that the relationship between preferences and states is stable, and provided that states are exogenous to choices of the agent. Most work on risk preference stability has assumed and/or estimated CRRA. Harrison et al. [2005] finds no significant shift in risk preferences inferred from a Holt and Laury [2002] MPL task (31 student subjects over 6-months). Sahm [2007] finds that risk tolerance inferred from the lifetime income gamble questions in the Panel Survey of Income Dynamics (12,000 U.S. subjects over up to 10 years) is relatively stable (albeit noisy), and largely consistent with CRRA, although there is some evidence that aging and macroeconomic conditions affect preferences. Andersen et al [2008 IER] find that estimates of CRRA from a Holt-Laury [2002] task are largely stable (correlated about 0.4 to 0.5 within-subject) on average (97 representative Danes over up to 17 months), although they do find some substantial variation, and variation that is correlated with the state of personal finances (but not with macroeconomic conditions as in Sahm [2007]). Goldstein et al [2008] also find fairly stable CRRA inferred from choices over hypothetical wealth distributions in retirement (75 geographically diverse U.S. subjects with mean age of 42 and median income of $50,000, over one year). Smidts [1997] finds a fairly strong (0.45) within-subject correlation in CARA inferred from a CE task on the market price of potatoes (253 Dutch farmers, over one year). Zeisberger et al [2012] find remarkable aggregate stability of prospect theory parameters elicited using a CE method (86 students over one month), although about 1/3 of subjects show significant instability. Tanaka et al [2010] use instrumental variables for income and prospecttheoretic model of preferences, and find that risk aversion moves (weakly) with village income but not with household income. See also Guiso and Paiella [2008]. 15