On Adaptation, Maximization, and Reinforcement Learning Among Cognitive Strategies

Transcription

1 1 On Adaptation, Maximization, and Reinforcement Learning Among Cognitive Strategies Ido Erev and Greg Barron Columbia University and Technion Abstract Analysis of choice behavior in iterated tasks with immediate feedback reveals robust deviations from maximization that can be attributed to the effects five distinct properties of the different alternatives. Decision-makers behave as if they tend to: (1) explore too much when the payoff variability is high, (2) prefer alternatives with high payoff rank, (3) prefer alternatives with low loss rate, (4) prefer alternatives that are displayed next to other relatively good alternatives, and (5) take too much risk when the amount of information provided in the feedback increases. Although the implications of the five factors appear to interact with each other, their joint effect can be summarized with a simple 2-parameter model that assumes reinforcement learning among cognitive strategies. With a single set of two parameters, the model captures the 48 experimental conditions used to demonstrate the five types of deviations. Moreover, with the same parameters, the model provides good predictions of behavior in 39 other repeated choice tasks. The theoretical and practical implications of these results are discussed. Address all correspondence to Ido Erev, Graduate School of Business, Columbia University, ie61@columbia.edu. This research was supported by a grant from NSF and the USA--Israel Binational Science Foundation. We thank Ernan Haruvy, Al Roth, Yoav Ganzach and the participants of seminars at Harvard, Columbia, NYU, Univ. of Michigan, Univ. of Chicago, and Univ. of Maryland for useful comments.

2 2 On Adaptation, Maximization, and Reinforcement Learning Among Cognitive Strategies Experimental studies of human decision-making in iterated tasks reveal a general tendency to respond to immediate feedback in an adaptive fashion. In line with the Law of Effect (Thorndike, 1898), the probability of successful responses tends to increase with time. Nevertheless, under certain conditions adaptation does not seem to lead to selection of the alternative that maximizes expected payoff. For example, decision-makers behave as if they are insufficiently responsive to rare outcomes (Barron & Erev, 2001); when rare outcomes are important, immediate feedback can increase the proportion of suboptimal choices. The main goal of the current paper is to improve our understanding of the conditions under which human adaptation does not lead to expected value maximization (at least not during the 200 trials of an experimental session). To achieve this goal, the paper reviews the relevant experimental results and proposes a model that summarizes the basic regularities. The results reveal that the main deviations from maximization can be attributed to sensitivity to five properties of the decision problems that can be negatively correlated with expected payoffs. The five properties involve payoff variability, payoff rank, loss rates, relative location (of the alternatives), and information concerning rare but high payoffs. Interestingly, the results also show that in three important cases the deviations from maximization in feedback-based decisions are in the opposite direction of the known deviations from maximization in one-shot tasks 1. This observation implies that March s (1996) elegant theoretical speculation concerning the possible relationship between decision-making in one-shot tasks and feedback-based decisions was too optimistic. It seems that different variables affect choice behavior in the two situations. The positive finding of the current research concerns with the predictability of the outcomes of the adaptive learning process across tasks. A twoparameter model built on a cognitive interpretation of the Law of Effect provides a good summary of all the 48 experimental conditions used to demonstrate the deviations from maximization. Moreover, the model, with the same two parameters, provides good

3 3 predictions of the 27 tasks studied by Myers, Reilly, and Taub (1961, in a systematic examination of the relative effect of incentives and probabilities) and captures the results obtained in studies of repeated play of matrix games with unique mixed strategy equilibrium (the 12 conditions reviewed in Erev & Roth, 1998). 1. Three Paradigms and Five Deviations from Maximization The current review focuses on three related experimental paradigms. In all three paradigms the decision-makers (DMs) are faced with the same decision problem many times and have minimal initial information. The DMs are instructed (and motivated) to try to use the (immediate and unbiased) feedback they receive after each choice in order to maximize earning. 2 The first paradigm involves choice between two or more unmarked buttons on the computer screen (see first column in Figure 1 for an example). In each trial the DM is asked to click on one of the buttons. Each click leads to a random draw from a payoff distribution (a play of a gamble) associated with the selected key. The distributions do not change during the experiment. The DM receives no prior information concerning the relevant payoff distributions, but can see the drawn value (the payoff) after each trial. We refer to this basic paradigm as unmarked buttons, 1 draw (UB1). <Insert Figure 1> The second paradigm is a variant of the basic paradigm with more complete feedback. After each choice in this paradigm, the DM is presented with a random value drawn from the payoff distributions of each of the (M) buttons (but payoffs are determined based on the value of the selected button). The additional feedback is often 1 In one-shot tasks, the decision makers are asked to make a single choice based on a description of the decision problems. The main deviations from maximization in this setting are summarized by Prospect Theory (Kahneman & Tversky, 1979). 2 The focus on immediate feedback implies that the current review does not include the important effects of delayed payoff and the related Melioration phenomenon explored by Herrnstein and his associates (see Herrnstein et al., 1993). The relationship between Melioration and the results reviewed here is discussed in Section 5. In addition, to facilitate the relationship to the economic literature, the current review does not consider studies that did not use monetary incentives (see Hertwig & Ortmann, 2001 for a discussion of this issue).

4 4 referred to as information concerning forgone payoffs. A typical trial in this unmarked buttons, M draws (UBM) paradigm is presented in the central column of Figure 1. The third paradigm considered here, known as probability learning (PL), was extensively studied in the 1950 s and 60 s (see reviews in Lee, 1971; Luce & Suppes, 1965; Estes, 1964; 1976). In each trial of a typical PL study, the DM is asked to predict which one of two mutually exclusive events (E or not-e) will occur. For example, which one of two lights will be turned on. The probability of the two events is static throughout the multi-trial experiment. The DMs receive no prior information about probabilities but know the payoff from correct and incorrect predictions of E and not-e. Immediately after each prediction, the DMs can see which light is on and can calculate their payoffs (and the forgone payoffs). The right-hand column in Figure 1 summarizes a typical trial. In order to demonstrate the observed deviations from maximization, 3 the current section summarizes the results of 48 experimental conditions. Each of the 48 conditions involves at least 200 trials. To facilitate an efficient summary of the large set of data, the current analysis focuses on the aggregate proportion of maximization in blocks of 100 trials. In the 2-alternative case this statistic, referred to as Pmax, coincides with the proportion of choices of the alternative that maximizes expected value. More generally this statistic reflects the relative location of the obtained payoffs in the interval between the minimal and maximal payoffs. For example, if the minimal expected payoff is 10 the maximal is 20 and the DM obtained 17, Pmax is Recall that at the first trial of the studies considered here subjects are expected to respond randomly (expected maximization rate of 0.50). Over the 48 conditions the rate of maximization in the second experimental block (Pmax2) was over Thus, on the average, experience leads toward maximization. 1.1 The Payoff Variability Effect The most obvious class of failures to maximize immediate payoffs involves situations with high payoff variability like Casino slot machines (see Haruvy et al., 2001). 3 Notice that in the current minimal information settings almost any behavior can be justified as rational ; it is possible to find a particular set of prior beliefs that would lead Bayesian agents to exhibit this behavior. Footnote 4 discusses one example. Thus, the current paper focuses on deviations from maximization that may not imply violations of the rationality assumption.

5 5 A particularly clear and elegant demonstration of this effect is provided by a series of papers by Jerome Myers and his associates (see Myers, Suydam & Gambino, 1965; and recent analysis by Busemeyer & Townsend, 1993). A simplified replication of this demonstration is summarized in the top panel of Figure 2. All three problems displayed in this panel present a choice between alternative H with an expected value (EV) of 11 points and alternative L with an EV of 10 points. The problems differ with respect to the variance of the two payoff distributions: Problems 1-3 (UB1, 200 trials, n=14, 0.25 ): Problem 1 H 11 points with certainty Pmax2 =.90 L 10 points with certainty Problem 2 H 11 point with certainty Pmax2 =.71 L 19 points with probability.5 1 otherwise Problem 3 H 21 points with probability.5 1 otherwise L 10 points with certainty Pmax2 =.57 These problems were examined (see Haruvy & Erev, in press) using the UB1 paradigm in a 200 trial experiment with conversion rate of 0.25 per point. The proportion of maximization in the second block, referred to here as Pmax2, was.90 in Problem 1 4,.71 in Problem 2, and.57 in Problem 3. <Insert Figure 2> 4 Informal conversations with the participants in the experiments suggests that the failure to maximize in this trivial condition can be rationalized with the assertion that at least some of the DMs thought at least some of the time that the 10 option provides high payoffs with small probability.

6 6 Notice that the difference between Problem 1 and 3 appears to reflect risk aversion (H is less attractive when its payoff variability increases), but the difference between Problem 1 and 2 appears to reflect risk seeking (L is more attractive when its payoff variability increases). This observation suggests that the risk attitude concept that was found to provide a useful summary of choice behavior in one-shot tasks, might be less useful in summarizing feedback-based choices (see Section 6 for a discussion of the relationship of the current findings to mainstream models of choice behavior like expected utility theory and prospect theory that use this concept). Rather, it seems that the results exhibit the following payoff variability effect: variability moves behavior toward random choice, and this effect is particularly strong when the variability is associated with the high EV alternative. The lower panels of Figure 2 present additional demonstrations of this payoff variability effect. The second panel (Problems 4, 5, and 6) shows that the effect is robust to the payoff domain (gain or loss). (These problems were run using the same procedure as Problems 1, 2, and 3 with the exception that the DMs received a high show-up fee to insure similar total expected payoff in the gain (Problems 1-3) and loss (Problem 4-6) domains.) The observed robustness implies an important difference between decisionmaking in one-shot tasks and in feedback-based decisions. In one-shot tasks DMs tend to be risk-averse in the gain domain and risk-seeking in the loss domain. The pattern is referred to as the reflection effect (Kahneman & Tversky, 1979). Feedback-based decisions, on the other hand, can lead to a reversal of this pattern. The third panel of Figure 2 (Problems 7-10) shows robustness to the number of observed draws. Problems 7-10 are replications of Problems 1, 3, 4, and 5 using the UBM paradigm. After each choice the DMs observed two draws, one from each distribution. The results show that in the current setting the additional information does not have a significant effect (but see Section 1.5 for obvious positive and negative effects of similar information). The forth panel of Figure 2 (Problems 11-14) shows robustness of the effect in a multi-outcome environment. These problems studied by Haruvy and Erev (2001) (200 trials, UB1 paradigm, n=12, 0.15 per point) involve choice among normal distributions.

7 7 Comparison of Problems 11 and 12 shows a replication of the observation that increase in payoff variability can increase the attractiveness of the low EV gamble. The fifth panel of Figure 2 (Problems 15-20) summarizes the examination of the interaction between the variability effect and payoff magnitude conducted by Myers et al. (1963). All six problems used following format: Problems (PL, 400 trials, n=20, p(e) >.5): H x if E occurs, -x otherwise L x if E does not occur, -x otherwise Myers et al. manipulated the value of x (1 or 10 cents) and p(e) (.6,.7, and.8) in a 2x3 design. The gambles were presented in a probability learning framing: The DMs were asked to predict which of two mutually exclusive events (E or no-e) would occur. Correct predictions paid x, while incorrect responses led to a loss of x cents. The results show slow adaptive learning and relatively weak and insignificant payoff magnitude effect. Notice that after 100 trials, the proportion of maximization is close to the value of p(e). This finding is often referred to as "probability matching." A payoff variability effect can be observed in decision-making in one-shot tasks (see Busemeyer & Townsend, 1993). Yet the effect in repeated tasks appears to be more robust. An interesting example is provided by the study of the certainty effect (Kahneman & Tversky, 1979) using a variant of Allais (1953) common ratio problems. Barron and Erev (2001) examined the two problems presented in the lower panel of Figure 1 (400 trials, UB1 paradigm, n=24, 0.25 per point). Notice that Problem 22 was created by dividing the probability of winning in Problem 21 by 4. This transformation does not affect the prediction of expected utility theory (von Neumann & Morgenstern, 1947) but does affect behavior. In the one-shot task it reduces the attractiveness of the safer alternative (L). Kahneman and Tversky (1979) called this pattern the certainty effect. Interestingly, in the repeated task the division by 4 increases the attractiveness of the safer alternative. Barron and Erev noted that the repeated choice pattern can be a result of the payoff variability effect: The payoff variability (relative to the differences in

8 8 expected value) is larger in Problem 22 than in Problem 21. As a result, DMs are less sensitive to the expected values and behave as if L is more attractive The Payoff Rank Effect (and Underweighting of Rare Outcomes) A second class of problems in which adaptation does not lead to maximization involves situations in which the expected rank of the gambles is incongruent with their expected payoff. In feedback-based decisions DMs behave as if they are sensitive to the expected rank. In a two-alternative case, expected rank is defined as the expected number of trials in which the gamble yields highest payoff. Sensitivity to expected rank implies low sensitivity to (or underweighting of) important but rare outcomes. One situation in which DMs appear to be more sensitive to expected rank than to expected payoffs is demonstrated in Problem 23 (studied by Barron and Erev, 2001). Problem 23 (UB1, 400 trials, n=24, 0.25 per point): H 32 points with probability.1, Pmax2 =.24 0 point otherwise L 3 points with certainty Notice that H has higher expected value, but L has higher expected rank. The proportion of maximization in trials 101 to 200 with immediate feedback was only Moreover, initially, maximization rate reduced with experience (see Figure 3). <Insert Figure 3> Figure 3 shows two additional conditions (run using Problem 23 s procedure) that demonstrate the robustness of the payoff rank effect. Problem 24 shows payoff rank effect when both alternatives are risky. Problem 25 shows a similar effect in the loss domain. Recall that studies of decision-making in one-shot tasks reveal a strong tendency to overweight small probabilities that lead to important deviations from maximization. This tendency is captured by the weighting function of prospect theory (see Gonzalez & Wu, 1999 for a careful analysis of this concept). The results of all three problems

9 9 summarized in Figure 3 show that in feedback-based decision DMs tend to deviate from maximization in the opposite direction. The average DMs behave as if they underweight rare events The Loss Rate Effect Examination of behavior in the stock market has led to the discovery of a third class of maximization failures. Thaler et al. (1997, and see Gneezy & Potter, 1997) show that when the action that maximizes expected value increases the probability of losses, people tend to avoid it. This observation provides an explanation to the equity premium puzzle, the finding that people are under-invested in the stock market. In a replication of these studies, Barron and Erev considered the following problem. Problem 26 (UB1, 200 trials, n=12, 1 per 100 points): H A draw from a normal distribution with a mean of 100 and standard deviation of 354. L A draw from a truncated (at zero) normal distribution with a mean of 25 and standard deviation of Pmax2 =.32 The proportion of maximization after 200 trials with immediate payoff was Similar results were obtained in the original studies (Thaler et al., and Gneezy & Potter) in which the subjects received more information. To establish that the suboptimal behavior observed in Problem 25 is a result of loss aversion, Thaler et al. compared this condition to an inflated condition. In Barron and Erev s replication, adding 1200 to the means of the two distributions created the inflated condition. That is, Problem 27 (UB1, 200 trials, n=12, 1 per 100 points): H A draw from a normal distribution with a mean of 1300 and standard deviation of 354. L A draw from a normal distribution with a mean of 1225 and standard deviation of Pmax2 =.56 5 The current results could also be captured with the assumption of an extreme reflection effect. Yet this assumption is inconsistent with the results summarized in Section 1.1.

10 10 The elimination of the losses increased the proportion of maximization in the last block to The top panel in Figure 4 summarizes the learning curves in the two replications of Thaler et al. described above. In addition, Figure 4 shows a third condition (Problem 28) that reveals a further and larger increase in maximization that was obtained by reducing payoff variance. <Insert Figure 4> Conditions that interact with the effect of losses. Additional data concerning the effect of losses comes from probability learning studies that manipulated payoff sign. Seven experimental conditions focused on the following task: Problems (PL, trials, p(e) >.5): H G if E occurs, B otherwise L G if E does not occur, B otherwise The second panel in Figure 4 summarizes Siegel and Goldstein s (1959) study that compares two problems with p(e) =.75 and G = 5 cents. The value of B was 0 in Problem 29 and 5 cents in Problem 30. The proportion of maximization in the second block was 0.85 and 0.95 respectively. The higher maximization rate in Problem 30 can be a result of the larger difference between G and B (5 or 10) and/or a positive effect of the losses. To evaluate the possibility that losses can facilitate maximization Erev, Bereby-Meyer, and Roth (1999, and see Bereby-Meyer & Erev, 1998) examined Problems (n = 14). They used the parameters p(e) =.7, G = 6, 4, 2, 0 or 2, and B = G 4 (with the conversion rate of.25 per point). The results, presented in the third panel of Figure 4, show a nonlinear effect of G on learning speed: Maximal learning speed was observed when G was 2 and 0. To compare alternative explanations of the nonlinear effect of G in Problems 30-34, Erev et al. ran four additional conditions that can be summarized as follows.

11 11 Problems (PL and UB1, n= 9, 550 trials, P(E) =.7, P(F) =.9,.25 per point): H G if E and F occur, B if not-e and F occur, 0 otherwise L G if not-e and F occurs, B if E and F occur, 0 otherwise The value of G was 6 (Problems 36 and 38) or 2 (Problems 37 and 39). As in Problems B = G 4. Problems 36 and 37 were studied in a PL paradigm, and Problems 38 and 39 in a UB1 paradigm. The results (forth panel of Figure 4) show very small (and insignificant) differences between the four conditions. Evaluation of Problems suggests that the main results can be summarized as a loss-aversion EV congruency effect. The possibility of losses: impairs maximization when the strategy that maximizes EV maximizes the probability of losses (Problem 26), facilitates maximization when the same alternative both maximizes EV and, minimizes losses (G= 2 and 0) and does not have a clear effect in the other cases. The lower panel in Figure 4 presents a comparison that highlights the possible tradeoffs and interactions between the loss rate and payoff variability effects. Notice that payoff variability effect implies higher maximization rate in Problem 40 than in Problem 21 (because in Problem 40 H involves less variability). The loss rate effect implies the opposite (because in Problem 40 H involves higher loss rate). The results (UB1, 400 trials, n=24,.25 per point) show insignificant difference between the two problems. Thus, in the current example the two effects appear to cancel each other The Relative Location Effect Less surprising, but not less important, sets of situations in which adaptation does not lead to maximization involve UB1 problems with a large strategy space. Consider situations in which the DMs have to select among 400 alternatives presented in a 20X20 payoff matrix. When the experiment is short (relative to the number of alternatives) the DMs should not try to explore the entire set of alternatives. Thus, when forgone payoffs are not known, adaptive learning processes are not likely to lead to the global maximum.

12 12 Experimental studies of repeated decision-making given a large set of alternatives reveal that performance is sensitive to the location of the alternatives in the strategy space (Busemeyer and Muyng, 1989). Maximization is more likely when the payoff matrix has a single peak (an optimal alternative surrounded by increasingly suboptimal alternatives). Busemeyer and Muyng note that this regularity can be captured by the assertion of a hill climbing search process. Yechiam, Erev, and Gopher (in press) have replicated Busemeyer and Muyng s results in a comparison of two types of 20x20 matrices. They compare 5 problems with an approximately single peak payoff matrix to 5 problems with multiple maxima in which the global maximum had a narrow basin of attraction. 6 A clear demonstration of this effect is provided by a comparison of Problem 41 and 42. Both problems involve a choice among the same 400 alternatives using the UB1 paradigm. Each alternative is associated with only one outcome. The two problems differ with respect to the location of the 400 alternatives in the 20X20 matrix presentation. The top panel in Figure 5 shows a three-dimensional summary of the two matrices. It shows that both matrices have two maximum points (a local maximum of 32 and a global maximum of 52). The conversion rate was.25 per point. In Problem 41 the local maximum (32) had a wide basin of attraction. Problem 42 was creating by replacing the location of the two maxima; thus, the global maximum (52) had the wide basin of attraction. <Insert Figure 5> The lower panel in Figure 5 presents the proportion of maximization statistic in the two conditions. In line with Busemeyer and Myung s and Yecham et al. s findings, the DMs were closer to maximization in Problem 42 (global maximum with wide basin of attraction) than in Problem 41. Whereas these results are not surprising, they have nontrivial implications. First, they suggest that in multi-peak environments performance can be improved by exploration enhancing training methods (see Yechiam et al, in press). Second, as 6 The basin of attraction of a maximum is defined as all the cells that are connected to the maximum with a chain of cells with non-decreasing values. In a 3-dimension presentation the basin of attraction is the hill

13 13 emphasized by Busemeyer and Muyng, they show that hill climbing or a similar process plays an important role in human adaptation The Effect of Information Concerning High but Rare Payoffs In an exploration of the effect of forgone payoffs, Grosskopf, Erev and Yechiam (2001, and see Grossskopf, 2000) studied choice among 100 alternatives in 10x10 and 20x20 matrices. In the 10x10 matrix conditions (Problems 43 and 44), the payoff of button ij at round t was determined by a random draw from one of two truncated normal distributions, either N(10,3) or N(11,1) 7 (with the conversion rate of.25 per point). The random draws were rounded to the nearest integer. The matrix included 50 buttons of each type. The locations of the different buttons were randomly determined for each subject (and did not change during the experiment). Notice that in this design the higher payoffs in each specific trial tend to come from the alternatives with the lower mean (the N(10,3) distribution). Problem 43 used the UB1 paradigm (1 draw), and Problem 44 used the UBM paradigm (M draws). The results (200 trials) presented in the upper panel of Figure 6 show an initial negative effect of the additional forgone payoff information. In the first 100 trials, Pmax with M draws was below the 0.50 chance level (the difference between the two conditions is significant). Erev and Rapoport (1998) who observed a similar pattern in a 12-person game referred to it as the big eyes effect. <Insert Figure 6> The positive effect of forgone payoffs. An obvious but important observation involves the positive effect of multi-draws in problems with large number of possible actions and low variance. Van Huyck et al. (1996) demonstrated this effect in a coordination game environment. Grosskopf et al. (2001, see Grosskopf, 2000) replicated this effect in a 20x20 matrix in which the payoff of alternative j (j =1,2 400) was j (the conversion rate was 1 per 120 points). The locations of the 400 alternatives in the matrix were randomly determined for each subject around the peak. 7 Both normal distributions were truncated at ±3 std and the payoffs were rounded to the nearest integer.

14 14 before the beginning of the experiment. This matrix was studied both with 1 and M draws (Problems 45 and 46 respectively). The results (200 trials) presented in the middle panel of Figure 6 shows a clear positive effect of additional forgone payoffs information in this setting. The additional information moved all the DMs toward maximization in few trials. To evaluate the robustness of the positive effect of forgone payoff information to payoff magnitude, Grosskopf et al. replicated Problems 45 and 46 after dividing the conversion rate by 15 (to 1 per 1800 points). The results (lower panel in Figure 6) show that this manipulation did not decrease the positive effect. In fact, the effect appears to increase. 2. The Potential Value of Simple Models Attempts to derive the practical implications of the five effects summarized above reveal that this task is not trivial. As suggested by the comparison of Problems 21 and 40 in many cases the different effects can lead to contradicting predictions. For an applied example consider the optimal tradeoff between the magnitude and likelihood of a fine (e.g., a rule enforcement system designed to reduce the frequency of reckless behavior, like running red lights; see Perry, Erev & Haruvy, in press). The payoff rank effect (the finding that rare outcomes are underweighted) suggests that likelihood is more important (because large but rare fines will be underweighted). Yet, if the expected loss from violating the rule is not high enough (and DMs learn to violate the rule), it is possible that increasing variability by using large but rare fines will reduce the violation rate. Moreover, in some situations the different effects interact which each other. For example, payoff variance increases exploration and, for that reason, can facilitate maximization in problems with multiple maxima like Problem 41 (see Busemeyer and Muyng, 1989). These observations imply that summarizing the 48 data sets reviewed here with a list of qualitative phenomena will be insufficient. The existence of tradeoffs and interactions requires a quantitative summary by a general model. Generality is essential in order to capture the possible interactions between the five phenomena. Quantification is important in order to capture the relative importance of the different effects when they

15 15 contradict each other. The current section takes one step towards developing a simple model of the 48 data sets. To facilitate the clarity of the description of the models considered here (especially for readers who may want to program and modify them 8 ), we chose to start the presentation with some technical definitions Technical Definitions Acts: The prospects presented before the DM by the experimenter. Strategies: Rules that condition a selection of one of the acts on available information. Neighbors: Acts b and c are neighbors unless they are placed in the space such that another action d is physically between them. Thus, in Problems 1-40 the two acts are neighbors. In Problems most acts (all the acts with the exception of the ones located on the boundary of the matrix) have 8 neighbors. A(1): The expected payoff to a DM who randomly selects among the possible acts. For example, in Problem 3 (10 or (1,.5; 21) ), A(1) =.5(10) +.5[.5(1) +.5(21)] =10.5. S(1): The expected absolute difference between A and the obtained payoffs (reinforcements) from random choice. For example, in Problem 3, S(1) = ( ) = Notice that in all interesting choice problems S(1) > 0. The current analysis ignores the trivial situation in which all the actions always yield the same outcome and S(1) = 0. R (t): The obtained payoff (reinforcement) at trial t. 8 Readers interested in deriving the prediction of the model with less effort can use the program available on our web cite at In addition, the program s source code has been made readily accessible for those wishing to construct a simulation themselves.

16 16 The loss minimizing (LM) set: Let PL b (t) be the proportion of times in which the observed payoff of act b was negative (during the first t trials). The PL b (t) value of a particular act can be unknown (before the first observation of its payoff), minimal (the lowest value among all acts with known value), or high (not minimal). The LM set that affects the decision at trial t+1 includes all the acts whose PL b (r) value was never (for r=1 to t) high. 9 A continuation act: Let D(t) be the direction of the move between trial t-1 and t. When actions are placed in a 2 dimensional matrix, the move can be classified as one of eight basic directions: up, up/right, right, right/down, down, down/left, left, left/up. If the payoff in trial t is higher than the payoff in t-1, the continuation act (at trial t+1) is the closest neighbor of the act selected at t in direction D(t). If the payoff in trial t is lower than the payoff in t-1, the continuation act is the closest neighbor of the act selected at t in the opposite direction. When a move in the relevant direction is not possible or when actions are not ordered, the continuation act is unknown A 2-parameter model: Reinforcement Learning Among Cognitive Strategies (RELACS) Although our goal is to derive a general model (a joint quantification) of the different regularities it is convenient to start by stating the simplest behavioral principles that could give rise to the different phenomena. The five assumptions listed below represent the simplest quantification (we found) of these principles. A1. Hill climbing. As noted by Busemeyer and Muyng (1989), the observation that DMs tend to converge to a local maximum (Section 1.4) can be abstracted with the assumption that 9 Notice that this definition implies that in the long term (following enough experience) an act will not belong to this set unless it minimizes the probability of losses. When only one act avoids losses (Problems 25, 26, 40) this act will eventually be the sole member of the set. When the loss probabilities are identical (e.g., Problems 36-39) experience leads to an empty set. When the loss likelihoods are not identical, the probability of an empty set decreases with the probability ratio. For example, the probability of an empty set is 0.65 in Problem 15 and 16, 0.41 in Problem 17 and 18, and 0.25 in Problems 19 and 20.

17 17 they behave as if they use, at least sometimes, a hill climbing strategy. A related idea, referred to as direction learning, is presented in Selten and Buchta (1999). Hill climbing is abstracted here as follows: One of the strategies considered by the DM is Hill Climbing (HC). This strategy (if used at trial t+1) prescribes a selection of the act with the highest recent payoff record RPR b (t+1) in the set that includes the acts whose payoffs were observed in the last period and their neighbors. If R b (t), the payoff of act b at trial t, is observed then the record of b is set to equal the mean of the new payoff and the previous record. That is, RPR b (t+1) = Mean[RPR b (t), R b (t)]. If the payoff of b is not observed at t, then RPR b (t+1)= RPR b (t). The initial value is RPR b (1) = A(1). When a payoff from a neighbor of act b is observed before the first selection of b, b s record is updated based on the neighbor s payoff. When more than one act in this set have the same maximal RPR b (t+1) value, the DM selects the continuation act if it is defined, and randomly (among the set members) otherwise. Notice that this assumption implies an implicit conditioning of behavior on the available information. When forgone payoffs are unknown (the UB1 paradigm), HC implies a slow step-by-step search. When forgone payoffs are known, or when acts are not on a continuum (they are all neighbors ) 10, HC implies a jump to the act with the highest RPR b (t) value. This property of the definition was introduced to capture the positive effect of forgone payoff information described in Section 1.5. Initially we preferred this abstraction of hill-climbing over the abstraction proposed by Busemeyer and Myung because it saves parameters (yet more parameters will be needed to capture the continuous case examined by Busemeyer and Muyng). Later we discovered that in addition to capturing the convergence to a local maximum, this abstraction captures three of the other four deviations from maximization summarized above. The payoff rank effect (Section 1.2) is captured because hill climbing 10 In the three paradigms studied here actions are ordered in a two-dimension continuum. For an example of paradigms in which actions are not ordered, consider a choice between different names.

18 18 tends to imply a move to the top-ranked recent observed outcome. 11 The effect of payoff variability (Section 1.1) is captured because noise reduces the probability that the EV maximizing outcome has the highest recent payoff record. Finally, a big eyes effect (Section 1.5) is predicted in a noisy environment in which one of the low EV actions that is not selected is likely to yield the highest payoff. Obviously, however, under the assumption that DMs always follow the HC strategy, the predicted deviations from maximization are much larger than the violations summarized above. This observation implies that the other assumptions of the model have to achieve two goals: Moderate the extreme HC predictions, and capture the fifth violation (loss aversion). Both goals are addressed by assuming that HC is only one of the possible strategies the DM considers and that the propensity to select each strategy changes as a function of the obtained reinforcements. A2. Loss avoidance To capture the robustness of the loss aversion phenomenon (Section 1.3), we assume that in some cases DMs simply try to avoid losses. Formally, One of the strategies considered by the DM is Loss Avoidance (LA). A selection of the LA strategy at trial t+1 implies a selection of one of the acts in the loss minimizing set. If this rule implies indifference (because the set is empty or has more than one member), then the indifference is resolved by selecting one of the acts using the choice rule described in equation 1 below The assumption that behavior is strongly influenced by recent experience is not unique to the current model. Gigerenzer & Goldstein (1999) assume the heuristic take the last in describing how people choose which of two alternatives score higher on a given criterion. According to this heuristic, people choose the alternative that was most effective (in terms of the criterion) the last time. Earlier, Gestalt psychologists called this strategy an Einstellung set and demonstrated that people tend to use the strategy that worked on the last problem when working on a series of problems (Duncker, 1935). Messick and Liebrand (1995) call a similar assumption the Win-Continue-Lose-Change strategy. Previous research suggests that memory, as well as behavior, is also biased by last experiences. Patients memories of painful medical procedures are largely determined by the intensity of pain at the worst part and at the final part of the experience (Redelmeir & Kahneman, 1996). 12 For example, if the set has two members, the tie-breaking rule implies a selection between the direct strategies associated with these actions. If the LA strategy was selected, it is the only strategy to be reinforced independently of a possible use of the tie-breaking rule.

19 19 This formulation is preferred over more conventional abstractions of loss aversion (like Kahneman and Tversky s prospect theory s abstraction) because it is parameter free. In addition, it is not obvious how the traditional abstraction that was developed to capture the results of one-shot studies can capture the non-monotonic pattern found in repeated tasks (Problems 31-39). A3. A two-stage choice process. Comparison of Problem 8 with Problem 47 suggests that in a noise-free environment in which forgone payoffs are available, the probability of maximization does not decrease with the number of acts. Whereas this suggestion does not seem surprising, it turns out that all the learning models we examined in our previous research (and summarized below) cannot capture it. To address this difficulty the current model assumes a two-stage choice process in each trial. At the first stage the DM selects a set of strategies: Direct strategies (one of the available acts) or Cognitive strategies (HC or LA as defined above). Thus, the probability of selecting each of the cognitive strategies (and the probability of maximization) can be approximately independent of the number of acts. Two choices are made in each trial: The DM first selects a set of strategies ( cognitive or direct ). At the second stage the DM selects one of the strategies in the selected set. The details of the two stages choice process are presented below. A4. Reinforcement learning. The Law of Effect, the finding that good outcomes increase the probability of similar choices, is quantified here with the assumption that the choice probabilities of sets and strategies are sensitive to their average payoffs. Specifically:

20 20 The probability p j (t,d) that a DM selects option j in decision d (d=1 involves the decision among the sets, d=2 is the choice among the strategies in the selected set) at time t is given by the following variant of Luce s (1959) choice rule: p q ( t) = e j j ( t) λ( t) n d k= 1 q e k ( t) λ( t) (1) where λ(t) is a payoff sensitivity (defined in A5), q j (t) is the propensity to select option j, and n d is the number of relevant options (n d = 2 in the choice among the two sets, and the number of strategies in the selected set in the second choice). If option j is selected at trial t, the propensity to select it in trial t+1, q j (t+1), is a weighted average of the propensity in t and the obtained payoff R (t): q ( t + 1) = q ( t)[1 w( t, d, j)] R( t) w( t, d, j) j j + (2) The weight of the new reinforcement is w(t,d,j)=1/[η/[(d)(n d )]+C j (t)] where η is a parameter that captures the strength of the initial value q j (1), C j (t) is number of times j was selected in the first t trials. The initial propensity q j (1) is assumed to equal A(1) (the expected payoff from random choice). On each trial two propensities are updated, one for the chosen set and one for the chosen act or strategy. Propensities of options that were not selected are not updated. Thus, Equation 1 is used at least twice in each trial: It determine the selection of the set (at d=1), the strategy (at d=2), and the tie breaking rule when LA is selected and implies indifference. Equation 2 is used twice, to update the propensities of the selected set and strategy. The exponential response rule with average updating is used here (eq. 1) because it is the simplest rule we know that captures the finding that the payoff domain (all gains

21 21 or all losses) does not appear to affect behavior (see a comparison of Problems 1 with 4 and Problems 31 with 35). 13 Notice that eq. 2 involves three implicit simplification assumptions. The first is the assumption of initial indifference between all strategies and between the two sets. Whereas initial indifference between the acts is natural (because the DMs have no prior information to distinguish between them), the implied initial indifference between the two sets and the two cognitive strategies might be an oversimplification. We address this possibility below. A second assumption implies that the DMs behave as if they know the value of A(1). Although this assumption cannot be correct, we believe that it provides a reasonable simplification of the prior information. A third implicit assumption involves the idea that all the reinforcements are equally weighted, and the weight of the initial propensity to select each option is identical to the weight of η trials in which the selection among the sets and strategies is random (the random rule implies η/[(d)(n d )] choices of each option). This assumption ignores the reasonable possibility that recent reinforcements have higher weight. This possibility is evaluated below. A5. Payoff sensitivity The current attempt to model payoff sensitivity is built upon related ideas presented by Busemeyer and Townsend (1993) and Erev et al. (1999) (and see Weber, Shafir, & Blais, 1999, and Friedman and Mezzetti, 2001for similar ideas): The payoff sensitivity at trial t is λ(t) = λ/s(t) where λ is a payoff sensitivity parameter and S(t) is a measure of observed payoff variability: S(t + 1) = S(t)[1 - w' (t)] + AD( t ) w' (t) (3) 13 As shown by Bereby-Meyer & Erev (1998), a summation rule (without additional parameters) predicts a lower maximization rate (and a violation of the Law of Effect) in the loss domain. For example, consider Problem 4 (-10 or 11). Under the summation rule each selection of the optimal action (-10) will decrease the likelihood of additional choice of this action.

22 where ' ( t ) = 1 ( t +η) w and AD(t) is the perceived payoff deviation in trial t. The exact value of AD(t) depends on the available forgone payoff information (FPI): 22 AD( t ) = MIN R(t) A(t) ( R(t) A(t), R(t) - R (t) ) k nofpi with FPI (4) where A(t)is a measure of payoff average and act k is the act whose direct strategy has the highest propensity among all the acts not selected at t. The average payoff measure, A(t), is updated in the same way as the payoff variability term: A(t + 1) = A(t)[1- w' (t)] + R(t)w' (t). (5) When forgone payoff information is not available, the current abstraction is identical to the abstraction proposed in Erev et al. (1999). This abstraction was introduced to capture the finding that in the current setting payoff magnitude does not appear to have a significant effect on maximization rate. The effect of forgone payoffs on payoff sensitivity was introduced following Busemeyer and Townsend (1993) to address situations in which the payoffs are noisy but positively correlated. An extreme example involves a problem in which H dominates L, but the same noisy factor is added to the two alternatives. The current formulation implies that in this setting forgone payoff information will increase payoff sensitivity. The Minimum function was added to capture the intuition that the availability of forgone payoffs will not reduce payoff sensitivity Descriptive value. Under the optimistic hypothesis that the five assumptions listed above capture the main psychological factors that affect behavior in the current task, RELACS should be able to reproduce the 48 learning curves. To evaluate this hypothesis, we ran computer simulations in which virtual DM s that behave according to the five assumptions

23 23 participated in a virtual replication of the 48 experimental conditions. 14 A grid search estimation method was used. A wide set of possible values for the two parameters was considered (on the plane 1< η <1000 and 1< λ< 10). Then one hundred simulations were run in each of the 48 problems under each set of parameter values. The simulations were run for the same number of trials as the experimental DMs (e.g., 200 trials in Problems 1). The initial values (A(1) and S(1)) were computed based on the payoff matrix. The following steps were taken in trial t of each simulation: (1) The virtual DM selected a Set and then a Strategy that determined an action. (2) The payoff of the observed actions was determined based on the problem payoff rule. 15 (3) The relevant statistic (the Pmax) that was recorded in the original experiment) was recorded. (4) The propensities and the other values assumed to affect future choices (e.g., C j (t), PL j (t), RPR j (t), S(t), A(t)) were updated. Each set of simulations resulted in a predicted learning curve in each of the 48 problems under each set of parameters. Like the experimental data set, the predictions were summarized with the proportions of Pmax in blocks of 100 trials. The fit of each set of simulations was summarized with the Mean Squared Deviation (MSD) between the aggregated observed and the predicted curves. The average MSD over the 2 to 5 blocks in each condition received the same weight. The parameters that minimize this score are λ = 4.5 and η = Table 1 summarizes the MSD scores of the models studied here. It shows that the MSD score (multiplied by 100) of RELACS is This score implies that the average distance between the observed and predicted proportion (that can assume values between 14 To reduce the risk of programming errors, the simulations were run using two independent computer programs (each written by one of the co-authors). The first author used SAS; the second author used Visual Basic. 15 To facilitate comparison with models that imply sensitivity to payoff magnitude, the payoffs were converted to their current value in cents. The conversion rate from the late 50 s early 60 s data was 1:5.

24 24 0 to 1) is below.07. In an initial evaluation of the model, it was compared to three baseline models: expected value maximization (Pmax of 1), random choice (Pmax of 0.50), and a 5-parameter model that fits the mean of each block (over the 48 curves) to each of the curves. The MSD scores of the baseline models (Cf. second panel in Table 1) are 8.63, 4.71, and 3.53 respectively. The large advantage of RELACS over the 5- parameter baseline suggests that it captures some of the robust differences between the different problems. <Insert Table 1> 2.3 Is the Model Too Complex? It is important to emphasize that RELACS is only one possible summary of the current data. We are confident that it is possible to find better models. The current model is presented here because it is the simplest model that we could find. To demonstrate that our failure to find a simpler model does not reflect a lack of effort, the current section summarizes some of the simpler models that were considered. Readers who are not interested in the details of this and similar analyses (that examine more complex models) can skip the rest of Section 2 (and see the summary in Table 1) Simpler models proposed in previous research. Most of the learning models proposed in recent research are not simpler (at least with respect to the number of parameters) than RELACS. Indeed, relatively little attention was paid in that research to the current goal: developing simple models that can capture the observed learning curves in a wide set of problems with a single set of parameters. Nevertheless, some interesting simple models were proposed. The three models proposed by March (1996). March (1996) showed that a tendency for risk aversion in the gain domain and risk seeking in the loss domain (summarized by Prospect Theory) could be a result of a simple learning process. He used 16 The relatively round values of these and other parameters presented below reflect the fact that the MSD function has a flat minimum: A large subspace of the parameter space yields (including λ = 4 to 6,