Higher-order retrospective revaluation in human causal learning

Transcription

1 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2002, 55B (2), Higher-order retrospective revaluation in human causal learning Jan De Houwer University of Southampton, Southampton, UK Tom Beckers University of Leuven, Leuven, Belgium Previous studies demonstrated that participants will retrospectively adjust their ratings about the relation between a target cue and an outcome on the basis of information about the causal status of a competing cue that was previously paired with the target cue. We demonstrate that such retrospective revaluation effects occur not only for target cues with which the competing cue was associated directly, but also for target cues that were associated indirectly with the competing cue. These second-order and third-order retrospective revaluation effects are compatible with certain implementations of the probabilistic contrast model and with a modified, extended comparator model, but cannot be explained on the basis of a revised Rescorla Wagner model or a revised SOP model. In human causal learning studies, participants are asked to make judgements about the relationship between potential causes and outcomes on the basis of information about the presence and absence of these causes and outcomes in various situations. In some situations, the information about the relationship between causes and outcomes is ambiguous. For instance, when two potential causes are present and the outcome follows, it is unclear which of the two causes actually produced the outcome. Recent studies have demonstrated that when participants are subsequently given additional information that allows them to determine the causal status of one of these two causes, they will also use this information to retrospectively adjust their judgements about the second cause (e.g., Dickinson & Burke, 1996; Larkin, Aitken, & Dickinson, 1998; Shanks, 1985). In a typical retrospective revaluation study, participants are first exposed to situations in which a compound stimulus consisting of a target cause T and a potentially competing cause C is followed by an outcome (CT+). During the second phase, cue C is presented alone and is either followed by (C+) or not followed by (C ) the outcome. At Requests for reprints should be sent to Jan De Houwer, Department of Psychology, University of Southampton, Highfield, Southampton, SO17 1BJ, UK. jandh@soton.ac.uk Tom Beckers is a Research Assistant of the Fund for Scientific Research Flanders (Belgium). We thank David Shanks for his suggestions regarding the research and the write-up of the manuscript as well as two anonymous reviewers for their helpful comments on an earlier draft of the manuscript. Ó 2002 The Experimental Psychology Society DOI: /

2 138 DE HOUWER AND BECKERS the end of the experiment, participants are asked to rate the strength of the relation between the target cause T and the outcome. Retrospective revaluation is evidenced by lower ratings for T when C was paired with the outcome in the second phase (C+) than when C was not followed by the outcome in the second phase (C ). Several models have been proposed that provide an explanation for such retrospective revaluation effects. First, Van Hamme and Wasserman (1994) showed that a modification of the Rescorla Wagner model (Rescorla & Wagner, 1972) allows this model to predict such effects. According to the original Rescorla Wagner model, causal judgements are a reflection of the acquired strength of the association between two stimuli. If a cue is present on trial n, the strength of the association between that cue and the outcome is updated according to Equation 1. DV n = ab( l - SVn -1 ) (1) The magnitude of the change in associative strength on trial n (DV n ) depends on the difference between l (which equals 1 when the outcome is present and 0 when the outcome is absent) and the sum of the existing associative strengths of all cues that are present on trial n (SV n 1 ). In the revised model, the associative strength of a cue can change on trials where the cue is present and also on trials where the cue is expected but not present. When the cue is present, a will be positive; when it is expected but absent, a will be negative. In the retrospective revaluation study described earlier, T is presumed to be expected on the C trials of Phase 2 because C and T were associated on the CT+ trials of Phase 1. However, because T is absent, its a will be negative. When C is paired with the outcome during Phase 2 (C+), (l SV n 1 ) in Equation 1 will be positive because the outcome is present (l = 1) but not fully predicted (l > SV n-1 ), but thea for T will be negative. Therefore, DV n will be negative and the associative strength of T is expected to decrease. When C is not followed by the outcome during Phase 2 (C ), (l SV n-1 ) in Equation 1 will be negative because the outcome is absent (l = 0) but was predicted (l < SV n-1 ) and the a for T will also be negative. Therefore, DV n will be positive, and the associative strength of T is anticipated to increase. Therefore, participants should judge the relation between T and the outcome to be less strong after C+ trials than after C trials. Dickinson and Burke (1996; also see Dwyer, Mackintosh, & Boakes, 1998; Larkin et al., 1998) adapted the SOP model of Wagner (1981) in such a way that it can account for retrospective revaluation. The SOP model is based on the assumption that stimuli (both cues and outcomes) are represented in memory as nodes that consist of several elements. Each of the elements in a node can be in one of three states: an inactive state (I), a primary active state (A1), and a secondary active state (A2). Whereas the initial activation of a stimulus representation produced by the presentation of the stimulus reflects the A1 state, A2 is the initial state achieved by the associative activation of a stimulus representation. The revised SOP model of Dickinson and Burke postulates that the excitatory association between two representations will increase as a function of the number of elements of both stimulus representations that are in the same active state (A1 A1 or A2 A2). Apart from excitatory associations, there are also inhibitory associations that will increase in strength whenever the elements of two stimulus representations are in different active states (A1 A2). Dickinson and Burke furthermore postulate that causal judgements reflect the difference between the strength of the excitatory and inhibitory association between the representations of a cue and an outcome.

3 HIGHER-ORDER RETROSPECTIVE REVALUATION 139 Consider the first phase of the retrospective revaluation study described earlier. On each CT+ trial, the elements of the representations of C, T, and the outcome will be in the A1 state simply because these three stimuli are present. Because the elements of all three representations are in the same active state, an excitatory association will be formed between the representations of C and T, C and the outcome, and T and the outcome. On the C+ trials that follow the CT+ trials, the representations of C and the outcome will be in the A1 state. Although T is absent, its representation will be activated in an associative way by C through the excitatory association between C and T that was built up during the CT+ trials. Because the representation of T is activated in an associative manner, its elements will be in the A2 state. The strength of the inhibitory association between the representations of T and the outcome will increase because the elements of T are in the A2 state whereas the elements of the outcome are in the A1 state. When the CT+ trials are followed by C trials, however, the elements of both T and the outcome will be associatively activated by C and will thus both be in the A2 state. As a result, the excitatory association between T and the outcome will be strengthened on C trials. Because the difference between the strength of the excitatory and inhibitory association between T and the outcome is greater after C trials than after C+ trials, participants will give higher causal judgements for T in the former case. The comparator hypothesis (Blaisdell, Bristol, Gunther, & Miller, 1998; Miller & Matzel, 1988) offers a third explanation for retrospective revaluation. Although it was developed as a model of Pavlovian conditioning in animals, it can also account for most findings within the literature of human causal learning, including retrospective revaluation. According to the comparator hypothesis, causal judgements are not a direct reflection of the strength of the association between a target cause and an outcome. Rather, judgements depend on the comparison between the strength of this association and the strength of association between the outcome and potential competing causes that were presented together with the target cause. Such a comparison takes place at the moment when the judgement needs to be made. This model provides a straightforward explanation of retrospective revaluation. Because C and T are paired during the CT+ trials, C will be a potential competing cause for T. Therefore, responding to T will depend on the strength of the association between T and the outcome relative to the strength of the association between C and the outcome. On C+ trials, the association between C and the outcome strengthens, whereas on C trials this association decreases in strength. Therefore, if the CT+ trials are followed by C+ trials the association between C and the outcome will gain strength relative to the association between T and the outcome, and responding to T will decrease. When the CT+ are followed by C trials, however, the association between C and the outcome will decrease in strength relative to the association between T and the outcome, and responding to T will increase. Finally, the probabilistic contrast model of causal learning (Cheng & Holyoak, 1995; Cheng & Novick, 1990; Waldmann & Holyoak, 1992) also provides an explanation for retrospective revaluation effects. The model postulates that causal judgements reflect the outcome of probabilistic contrasts. For instance, judgements about the relation between a cause T and an outcome will correspond to the outcome of probabilistic contrasts that evaluate the influence of the presence of T on the presence of the outcome while keeping the presence or absence of competing causes C constant for example, p(outcome C.T) p(outcome C.~T). The model readily explains retrospective revaluation effects. When CT+ trials are followed by C+ trials, the probability of the outcome in the presence of C and T,

4 140 DE HOUWER AND BECKERS p(outcome C.T), equals the probability of the outcome in the presence of C, but in the absence of T, p(outcome C.~T). Because the result of the probabilistic contrast is zero, causal judgements for T will be low. However, when the CT+ trials are followed by C trials, the outcome is more likely to be present when both C and T are present than when only C is present, p(outcome C.T) > p(outcome C.~T). Therefore, the result of the probabilistic contrast for T is positive, and participants should give a high rating for the relation between T and the outcome. Until now, causal learning studies have only looked at retrospective revaluation that depends on first-order relations. Two stimuli are entered into a first-order relation if these stimuli have actually co-occurred in space and time. A higher-order relation refers to a relation between two stimuli that have not co-occurred in space and time but are related as a result of associations with other stimuli. For instance, if cue C has been associated with cue T1, and cue T1 has co-occurred with cue T2, cues C and T2 are said to have a second-order relation. Recently, Denniston, Savastano, Blaisdell, and Miller (2001, cited in Denniston, Savastano, & Miller, 2001) provided a demonstration of second-order retrospective revaluation in the context of Pavlovian conditioning with animals. During a first phase, animals were exposed to a compound of cues C and T1 that was always followed by an outcome (CT1+). Next, T1 was presented together with a second target cue (T2), and this compound was also followed by the outcome (T1T2+). During the third phase, C was either presented on its own without the outcome (C ) or not presented (control). Animals responded less strongly to T2 in the condition with C trials than in the control condition without C trials. Note that T2 was not directly associated with C but only with T1. There was only a second-order relation between T2 and C: C was associated with T1, and T1 was associated with T2. The results of Denniston et al. (2001) demonstrate that information about the causal status of a cue (i.e., the fact that C on its own is not paired with the outcome) can retrospectively influence reactions towards stimuli with which that cue has a second-order relation (i.e., T2). This finding is important because it is incompatible with both the revised Rescorla Wagner and the revised SOP model. These models only allow for first-order retrospective revaluation effects because the associative strength of an absent cue can only change if it has entered into a first-order relation with the cue that is present. In the study of Denniston et al. (2001), C was associated with T1 but not with T2. Within a revised Rescorla Wagner model, only T1 (not T2) is expected to occur on the C trials. Therefore, whereas the associative strength of T1 should change on the C trials, the associative strength of T2 should not change. Note that even if T2 was, for some reason, expected on the C trials, its a should be negative because T2 was not physically present. As such, the associative strength of T2 should increase on C trials because both T2 and the (expected) outcome are absent (see Equation 1). However, the results of Denniston et al. showed that responding to T2 decreased as a result of the C trials. Likewise, the revised SOP model would predict that presenting C cannot put the elements of the representation of T2 in an A2 state simply because C and T2 have not been paired. Even if presenting C could somehow activate the representation of T2 (e.g., presenting C could activate T1, and activation could then spread to the representation of T2), both the representation of T2 and the representation of the outcome should be in the A2 state, and the associative strength of T2 should thus increase rather than decrease. The results of Denniston et al. (2001) can be explained on the basis of the other two models. According to the extended comparator model, which was recently proposed by Blaisdell et al.

5 HIGHER-ORDER RETROSPECTIVE REVALUATION 141 (1998; also see Denniston et al., 2001), the modulating effect of competing causes is itself modulated by competitors of the competing cause. With a design such as the one used by Denniston et al., C becomes the competitor for T1 because these stimuli are paired on the CT1+ trials. Likewise, T1 will become the competitor for T2 as a result of the T1T2+ trials. Without going into detail (see Denniston et al. for an in-depth discussion), the extended comparator model postulates that reactions towards T2 will be determined by a comparison of the associative strength of T2, on the one hand, and the associative strength of T1 relative to the associative strength of C, on the other hand. In other words, just as the impact of the associative strength of T2 on behaviour will depend on the associative strength of its competitor T1, so will the modulating effect of T1 on T2 depend on the associative strength of C, which is the competitor of T1. According to the extended comparator hypothesis, the associative strength of C, will be smaller when C is presented on its own during the C trials (C condition) than when C is not presented during the last phase (control condition). As a result, the difference between the associative strength of C and T1 will be larger in the C condition than in the control condition. Because the extent to which T1 modulates the effect of the associative strength of T2 on behaviour depends on the associative strength of T1 relative to the associative strength of C, T1 will counteract the effects of the associative strength of T2 to a larger extent in the C condition than in the control condition. Therefore, animals should respond less strongly to T2 in the C condition than in the control condition. The results of Denniston et al. (2001) can also be explained if certain process assumptions are made about how the probabilistic contrast model can be implemented algorithmically (also see Cheng & Holyoak, 1995). As we explained earlier, the model postulates that judgements will reflect the outcome of probabilistic contrasts that compare the probability of the outcome in the presence of the target cue with the probability of the outcome in the absence of the target cue within a focal set of situations in which the presence of other potential causes is kept constant. The reason for selecting situations in which the presence of competing causes is constant is that it allows one to determine whether the target cue has an effect on the outcome independent of other causes of the outcome. As T2 has only been presented in compound with T1 on the T1T2+ trials, the focal set would ideally consist of the T1T2+ trials and trials on which only T1 is present. The latter trials are not available because T1 was always presented in compound with C in those cases where T2 was absent (i.e., CT1+ trials). However, one could assume that the information that is presented during the third phase allows one to retrospectively recode the CT1+ trials. If the third phase contains C trials, the appropriate probabilistic contrast for T1 is positive, p(outcome C.T1) p(outcome C.~T1) = 1 0 = 1, whereas the appropriate probabilistic contrast of C is zero at most, p(outcome Context.C) p(outcome Context.~C) = 0 p(outcome Context.~C) 0. Given the natural tendency to assume that the causal impact of a cue is constant across situations (Waldmann, 2000), C can be dismissed as a potential cause of the outcome on the CT1+ trials (also see Cheng & Holyoak, 1995, p. 285), which implies that the CT1+ trials can be recoded into T1+ trials. This in turn provides the information that is necessary to compute the probabilistic contrast for T1, p(outcome T1.T2) p(outcome T1.~T2). Because this contrast is zero, T2 should receive a low rating. The aim of the present studies was to examine higher order revaluation effects within the context of human causal learning. In our experiments, participants saw a series of situations in which so-called weapons (represented by squares at certain positions on a computer screen)

6 142 DE HOUWER AND BECKERS fired at tanks (which were also drawn on the computer screen). In each situation, the tank either exploded or did not explode. After observing these events, participants were asked to judge for each weapon how likely it was that the tank would be destroyed if that weapon fired. De Houwer, Beckers, and Glautier (in press) demonstrated that within certain parameters, reliable first-order retrospective revaluation effects do occur within this task (also see Shanks, 1985). The designs of the present experiments are summarized in Table 1. All experiments consisted of three phases. In the first phase of Experiment 1, weapons C and T1 always fired together and always destroyed the tank (CT1+). During the second phase, T1 always fired together with weapon T2, and the tank was always destroyed (T1T2+). The third phase contained the crucial manipulation. Half of the participants saw that the tank was always destroyed when C fired on its own (condition C+), whereas for the other participants, the tank was never destroyed when C fired on its own (condition C ). In all phases, we also presented trials on which weapon A destroyed the tank on its own (A+) and trials on which weapon B fired alone but did not destroy the tank (B ). These trials were added to avoid the outcome (tank destroyed or not destroyed) being the same on all trials within a certain phase. All four models previously described predict that a first-order retrospective revaluation effect should emerge. That is, the ratings for T1 should be lower in the condition with C+ trials than in the condition with C trials. However, only the extended comparator hypothesis and the probabilistic contrast model predict a second-order retrospective revaluation effect in that ratings for T2 should be higher after C+ trials than after C trials. In Experiment 2, we went one step further and examined, for the first time, whether thirdorder retrospective revaluation effects also occur in human causal learning. As in Experiment 1, the tank was always destroyed when weapons C and T1 fired together during the first phase (CT1+). In the second phase, two compound stimuli were reinforced. Weapon T1 fired together with weapon T2 on some trials (T1T2+), and weapon T2 fired together with weapon T3 (T2T3+) on other trials. We again manipulated the relation between C and the outcome during the third phase. For half of the participants, C on its own was paired with tank destruction (condition C+), whereas for other participants C was never followed by the destruction of the tank (condition C ). First-order retrospective revaluation would be evidenced by lower ratings for T1 after C+ trials than after C trials. Second-order retrospective revaluation TABLE 1 Design of Experiments 1, 2, and 3 Phase Experiment Condition C+ CT1+ T1T2+ C+ C CT1+ T1T2+ C 2 C+ CT1+ T1T2+, T2T3+ C+ C CT1+ T1T2+, T2T3+ C 3 CrT1r+, CuT1u+ T1rT2r+, T1uT2u+ Cr+, Cu- Note: All phases of all experiments and conditions also included A+ and B trials. + stands for the presence of the outcome; stands for the absence of the outcome.

7 HIGHER-ORDER RETROSPECTIVE REVALUATION 143 would lead to higher ratings for T2 after C+ trials than after C trials. Third-order retrospective revaluation would result in lower ratings for T3 after C+ trials than after C trials. Although the idea of third-order revaluation might seem far-fetched, it can be justified on the basis of certain implementations of the probabilistic contrast model (e.g., Cheng & Holyoak, 1995). C trials allow the participants to infer that T1 caused the outcome to occur on the CT1+ trials and to recode these trials into T1+ trials. This recoded information can be used to infer that T2 is not a cause of the outcome, p(outcome T1.T2) p(outcome T1.~T2) = 1 1 = 0, and thus that p(outcome T2) = 0. This in turn allows participants to calculate the appropriate probabilistic contrast for T3, p(outcome T2.T3) p(outcome T2.~T3) = 1 0 = 1, and to infer that T3 is a cause of the outcome. When participants are exposed to C+ trials during the third phase, CT1+ trials can be recoded into C+ trials and T1T2+ trials into T2+ trials. Because the probabilistic contrast for T3 that can be performed on the recoded focal set produces a value of zero, p(outcome T2.T3) p(outcome T2.~T3) = 1 1 = 0, T3 can be dismissed as a cause of the outcome. As such, ratings for T3 should be higher after C trials than after C+ trials. The extended comparator model as it was formulated by Denniston et al. (2001) does provide for the possibility of beyond second-order comparison processes. The modulatory effects of a (first-order) comparator are influenced by the comparator of that comparator (i.e., the second-order comparator). In theory, the influence of this second-order comparator can itself be modulated by a third-order comparator. Thus, if the results of Experiment 2 demonstrate that C trials influence the ratings for T3, this would imply that the modulation by the comparator of T3 (i.e., its first-order comparator: T2) is modulated by the comparator for T2 (i.e., second-order comparator: T1), which in turn is modulated by the comparator for T1 (i.e., third-order comparator: C). However, Denniston et al. suppose that the possible contribution of higher order comparator stimuli will be increasingly limited. Method Participants EXPERIMENT 1 A total of 26 psychology undergraduates at the University of Leuven participated for course credit. None had participated in a similar experiment before. Stimuli A custom-made Turbo Pascal 7.0 program was used to present the instructions and stimuli and to register the ratings. This program was similar to the programs used in the experiments reported by De Houwer (in press) and De Houwer et al. (in press) except with regard to the number of weapons and the type and number of trials. The program was implemented on an IBM-compatible P166 computer. On the 15" SVGA screen, a tank was 4 cm long and 2 cm high. It moved in a continuous manner from the left to the right side of the screen on a straight line that was situated 10 cm from the top of the screen. It took about 6 s for the tank to get from the left to the right side of the screen. If the tank exploded, this always occurred 2 s after the tank appeared. Explosions were always located at a point 12 cm from the left side of the screen. When a tank exploded, it disappeared from the screen and was replaced by 10 lines that gradually increased in length from 1 cm to 7 cm and then decreased in length until they disappeared. The

8 144 DE HOUWER AND BECKERS lines diverged as they became longer, thus forming a fan-like shape. This explosion took about 1 s. The five weapons were represented by rectangles that were situated at the bottom of the screen. Each rectangle was 3 cm wide and 2.5 cm high. The rectangles were numbered 1 to 5, 1 being the rectangle on the far left side of the screen, 5 being the rectangle on the far right side. A weapon fired when a solid white square measuring cm appeared in the rectangle that represented the weapon. Participants entered their ratings using the keyboard. All stimuli were drawn with white lines on a black background. Procedure At the beginning of the experiment, participants received written instructions in Dutch (see Appendix). They were told that army tanks would ride across the computer screen. There would also be five weapons represented by five squares at the bottom of the screen. When a weapon fired, a white light would appear in the square that represented the weapon that fired. Participants were asked to determine for each weapon separately how effective it was. Their task would be complicated by the fact that sometimes two weapons fired together. But on each trial, information about the combined impact of all fired weapons would be displayed (see later). After the participants read the instructions, a screen appeared on which the five squares and the horizontal line on which the tank would ride were visible. The experimenter briefly repeated the instructions while pointing at the relevant sections of the screen. First, 10 CT1+, 5 A+, and 5 B trials were presented. These trials were followed by 10 T1T2+, 5 A+, and 5 B trials. For half of the participants (condition C+), 10 C+, 5 A+, and 5 B trials were presented during the last phase of training, whereas for the other participants (condition C ), ten C, five A+, and five B trials were presented. There was no break between the different phases. Within each phase, trials were presented in a random order that was determined separately for each participant, with an inter-trial interval of 3000 ms. Similarly, which square functioned as which weapon was also determined randomly for each participant. When a weapon fired, a solid white square appeared in the square that represented that weapon. The solid square was presented for 300 ms, during which time the tank kept on moving at the same speed as previously. Tank explosions occurred immediately after the solid white square disappeared. At the same time the message IMPACT 10/20 appeared on the screen. This message remained on the screen for 3 s. If the tank did not explode, the message IMPACT 0/20 appeared on the screen after the weapon(s) fired. The impact message remained on the screen until the tank, which drove on, had reached the right side of the screen. Information about the impact of the weapons was added for both theoretical and empirical reasons. Cheng (1997; Cheng & Holyoak, 1995) argues that participants are aware of the fact that probabilistic contrasts do not provide a valid basis for contingency judgements when the outcome is an event that always occurs within the relevant focal set of situations. Therefore, blocking should not occur when the outcome is always present on both C+ and CT+ trials. However, Cheng (1997, p. 384) points out that participants are also aware of the fact that such probabilistic contrasts can be interpreted when the outcome is a continuous event that occurs at a non-maximal rate. The results of recent studies conducted in our laboratory supported this hypothesis. In a study that used the same task as the present study, De Houwer et al. (in press) only observed a significant blocking effect when the tank explosion always occurred at a non-maximal level on C+ and CT+ trials (i.e., an impact of 10 when the maximal explosion impact that could be measured corresponded to a value of 20) but not when the tank explosion always occurred at a maximal level (i.e., an impact of 10 while the maximal explosion impact was also 10; see De Houwer et al., for a more detailed discussion). After observing all 60 trials, participants were asked to judge for each weapon how likely it would be that a tank would be destroyed if that weapon fired. They could give a rating by entering a score between 0 (very unlikely) and 100 (very likely). All participants first rated the weapon that was represented by Square 1 (the square on the far left side), then the weapon represented by Square 2, and so. During this rating phase, the rectangles were presented on the screen in the same way as during the rest of the

9 experiment. A 10-cm rating scale ranging from 0 to 100 was also present on the screen together with the question How likely is tank-destruction if weapon x fires? After they entered their likelihood rating for a weapon, a second rating scale appeared on the screen together with the question How sure are you? Participants were instructed to express their confidence in the accuracy of the likelihood rating by giving a score between 0 and 100, where 0 stands for very unsure and 100 stands for very sure. 1 Results and discussion HIGHER-ORDER RETROSPECTIVE REVALUATION 145 The likelihood ratings for each of the weapons were analysed using a Condition (C+ or C ) Weapon (C, T1, T2, A, or B) analysis of variance (ANOVA) with repeated measures on the latter variable. A priori t tests were used to examine the effect of condition for each of the weapons separately. The significance level was set at p <.05 for all tests. Where necessary, Greenhouse Geisser corrections were performed. The ANOVA revealed a main effect of condition, F(1, 24) = 6.84, MSE = , a main effect of weapon, F(2.68, 64.34) = 42.02, MSE = , and an interaction between both variables, F(2.68, 64.34) = 19.79, MSE = The means for this interaction can be found in Table 2. A priori t tests showed that condition had a significant effect on the ratings of C, T1, and T2. As could be expected, C received lower ratings when participants were exposed to C trials than when they saw C+ trials. The fact that T1 received lower ratings after C+ trials than after C trials provides evidence for first-order retrospective revaluation. Most importantly, we also found evidence for second-order retrospective revaluation. As was expected on the basis of the extended comparator model and our implementation of the probabilistic contrast model, T2 received higher ratings after C+ trials than after C trials. This result demonstrates that ratings about the relation between a target cause and an outcome can be influenced by manipulations that affect the associative strength of a cause that has a second-order relation with the target cause. TABLE 2 Mean likelihood ratings as a function of condition and weapon in Experiment 1 Condition C+ C Weapon M SE M SE t(24) p C <.001 T T A n.s. B The confidence ratings were added for exploratory reasons only. We had no predictions about differential effects of condition on the confidence in the likelihood rating for the different weapons. Results showed that in both conditions of Experiments 1 and 2 and in Experiment 3, participants were more confident about their ratings for the weapons that (sometimes) fired alone (i.e., A, B, and to a lesser extent C) than for the weapons that only fired together with another weapon (i.e., T1 and T2 in Experiments 1 and 3; T1, T2, and T3 in Experiment 2). This suggests that participants were more confident about their likelihood ratings if they could directly observe what the effect of a weapon was when it fired on its own compared to when they could only determine the effect of a weapon indirectly.

10 146 DE HOUWER AND BECKERS TABLE 3 Mean likelihood ratings as a function of condition and weapon in Experiment 2 Condition C+ C Weapon M SE M SE t(14) p C T T T A n.s. B n.s. Method EXPERIMENT 2 Experiment 2 was identical to Experiment 1 except with regard to the following points. A total of 16 firstyear psychology students at the University of Leuven participated for partial fulfilment of course requirements. None had participated in the previous study. As we now examined both second-order and third-order retrospective revaluation, we had six rather than five weapons and thus six instead of five squares at the bottom of the screen. Phases 1 and 3 were identical to the corresponding phases in Experiment 1. However, the second phase consisted of 10 T1T2+, 10 T2T3+, 5 A+, and 5 B trials (see Table 1). Results and discussion The likelihood ratings for the six weapons were analysed using a Condition (C+ or C ) Weapon (C, T1, T2, T3, A, B) ANOVA with repeated measures on the second variable. A priori t tests were used to examine the effect of condition on the ratings of the six weapons separately. The ANOVA revealed a main effect of weapon, F(2.95, 41.31) = 22.50, MSE = , and an interaction between condition and weapon, F(2.95, 41.31) = 16.30, MSE = The main effect of condition was not significant, F < 1. Table 3 shows that the effect of condition on C, T1, and T2 was exactly the same as that in Experiment 1. Ratings for C and T2 were higher after C+ trials than after C trials, whereas ratings for T1 were lower after C+ trials than after C trials. Most importantly, condition also had a significant effect on the ratings for T3: T3 received lower ratings after C+ trials than after C trials. The results thus revealed a third-order retrospective revaluation effect. EXPERIMENT 3 The conclusion that higher-order retrospective revaluation occurred in Experiments 1 and 2 was based on a between-subjects comparison of the ratings for target cues in a condition with C+ trials and a condition with C trials. There are, however, a number of confounds that are inherent to the between-subjects design we used. For example, participants in the C

11 HIGHER-ORDER RETROSPECTIVE REVALUATION 147 condition were exposed to more unreinforced trials than participants in the C+ condition. Also, in the C condition participants might have perceived a conflict between the C trials of the third phase and the CT1+ trials of the first phase: Whereas C was followed by the outcome on the CT1+ trials, it was not followed by the outcome on the C trials. Although such a conflict can be interpreted in a number of ways, some participants in the C condition might have believed that (some of) the contingencies had reversed from Phase 3 onwards, a belief that might have resulted in systematic differences between how the cues were rated in both conditions. Examples of such reversal or miscueing effects have been reported in other studies (e.g., Lipp, Siddle, & Dall, 1993). 2 Although it is unlikely that such confounds could account for the complex pattern of performance observed in Experiments 1 and 2, we decided to examine second-order revaluation using a within-subject design (see Table 1). Rather than presenting one set of competing and target cues, we now presented two sets that each consisted of a competing cue C, a first-order target cue T1, and a second-order target cue T2. Within both sets, we created CT1+ and T1T2+ trials. The only difference was that the competing cue of set R (i.e., Cr) was reinforced during the third phase (Cr+) whereas the competing cue of set U (i.e., Cu) was unreinforced during Phase 3 (Cu ). First-order retrospective revaluation would be evidenced by lower ratings for T1 of set R than for T1 of set U (T1r < T1u). Second-order retrospective revaluation would be revealed by higher ratings for T2 of set R than for T2 of set U (T2r > T2u). We did not examine third-order retrospective revaluation in this experiment because it would have forced us to increase the number of trials and trial types even further. Method Experiment 3 was identical to Experiments 1 and 2 apart from the following points. A total of 10 psychology graduate students at the University of Leuven volunteered to participate. None had participated in a previous study on causal learning. There were eight weapons represented by eight squares at the bottom of the screen. Phase 1 consisted of 10 CrT1r+, 10 CuT1u+, 5 A+, and 5 B trials. During Phase 2, 10 T1rT2r+, 10 T1uT2u+, 5 A+, and 5 B trials were presented. Finally, Phase 3 consisted of 10 Cr+, 10 Cu, 5 A+, and 5 B trials. Results and discussion The likelihood ratings for Cr, T1r, T2r, Cu, T1u, and T2u were entered into a Set ( R or U) Weapon (C, T1, or T2) ANOVA with repeated measures on both variables. The ratings for weapons A and B were not entered into the ANOVA or subjected to a t test because there was only one weapon A and one weapon B rather than one A and B for each of the two sets. Weapon A received a mean rating of 85 (SE = 8) whereas weapon B received a mean rating of 0 (SE = 0). The mean ratings of the other weapons are listed in Table 4. The ANOVA revealed a significant main effect of set, F(1, 9) = 43.55, MSE = , and a significant interaction between set and weapon, F(1.28, 11.52) = 18.06, MSE = The main effect of weapon did not reach significance, F(1.42, 12.75) = 1.49, MSE = The t tests showed that Cr obtained a higher rating than Cu and that T1r received a lower rating than T2r. Most importantly, we 2 We thank David Shanks for drawing our attention to these problems and for encouraging us to conduct a withinsubject test of higher-order retrospective revaluation.

12 148 DE HOUWER AND BECKERS TABLE 4 Mean likelihood ratings as a function of set and weapon in Experiment 3 Set R (C+) U (C ) Weapon M SE M SE t(9) p C <.001 T T again observed second-order retrospective revaluation as evidenced by the higher rating for T2r than for T2u. GENERAL DISCUSSION The present experiments provide the first evidence for higher-order retrospective revaluation in human causal learning. At a functional level, our results demonstrate that participants can use additional information about the causal status of a competing cue to retrospectively adjust their judgements about target cues that have a first-order, a second-order, or a third-order relation with the competing cue. Note that the studies reported here do not allow us to differentiate between retrospective revaluation that is due to the C+ treatment and retrospective revaluation that is due to the C treatment, relative to no revaluation treatment (the same holds for most retrospective revaluation studies). This issue needs to be addressed in future research. Provided that certain process assumptions are made (e.g., Cheng & Holyoak, 1995), the probabilistic contrast model is compatible with the reported results. According to this model, causal judgements reflect the result of probabilistic contrasts that compare the probability of the outcome in the presence of the target cue with the probability of the outcome in the absence of the target cue. When the target cue is accompanied by a potential competing cause, participants will try to select a focal set of situations in which the presence of the competing cause is kept constant. If the competing cause is sometimes presented on its own, this not only provides the necessary information to compute the appropriate probabilistic contrast for the first-order target cue, p(outcome C.T1) p(outcome C.~T1), but also allows participants to recode trials in such a way that the appropriate probabilistic contrasts for higher-order target cues can be calculated (given that people tend to assume that the causal status of a cue remains stable across situations; Waldmann, 2000). For instance, with a second-order retrospective revaluation design (CT1+ and T1T2+ followed by C+ or C ), C trials allow participants to infer that T1 produced the outcome on the CT1+ trials and thus to recode these trials as T1+ trials. After such a recoding, participants can calculate the probabilistic contrast for T2, p(outcome T1.T2) p(outcome T1.~T2) = 1 1 = 0, which will lead to a low rating for T2. On the other hand, C+ trials allow participants to determine that T1 is not a cause of the outcome, p(outcome C.T1) p(outcome C.~T1) = 1 1 = 0, and thus that p(outcome T1) = 0. This, in turn, provides the necessary information to calculate an appropriate probabilistic contrast for T2, p(outcome T1.T2) p(outcome T1.~T2) = 1 0 = 1, and to infer that T2 causes the outcome.

13 HIGHER-ORDER RETROSPECTIVE REVALUATION 149 The extended comparator model seems fairly able to cope with our results too. In fact, the extended comparator model formulated by Denniston et al. (2001) already provides for the theoretical possibility of third-order retrospective revaluation, albeit cautiously. Our results suggest that third-order modulation can be observed. That is, not only can the effect of the associative strength of a target cue on behaviour be modulated by the associative strength of the competitor of this cue (first-order modulation), but also the degree to which the competitor modulates the effects of the associative strength of the target cue depends on the associative strength of the competitor of the competitor of the target cue (second-order modulation). Apparently, this second-order modulation is also influenced by the associative strength of the competitor of the second-order modulating cue. According to this analysis, in Experiment 2, ratings for weapon T3 reflect the associative strength of T3 as modulated by the associative strength of its competitor T2. The degree of modulation by T2 depends on the associative strength of its competitor, namely weapon T1, and the degree to which T1 modulates the modulation by T2 depends on the associative strength of its competitor, namely C. Our results seem incompatible with both the revised Rescorla Wagner model and the revised SOP model. However, one can think of ways to further revise these models so that they can cope with our results. According to the revised Rescorla Wagner model (Dickinson & Burke, 1996; Van Hamme & Wasserman, 1994), the associative strength of a target cue will change on trials where the target cue is present and on trials where another cue is present that has a first-order relation with the target cue. Changes in associative strength will be opposite on trials where the target cue is present than on trials where the target cue is absent because a is positive in the former case but negative in the latter case (see Equation 1). One could propose that the associative strength of a target cue can also change on trials where the target cue is absent but another cue is present that has a second-order or third-order relation with the target cue. In order to explain that first-order retrospective revaluation has an opposite effect to second-order retrospective revaluation (e.g., with CT1+ and T1T2+ trials, C+ trials lead to a lower rating for T1 but a higher rating for T2) but the same effect as third-order retrospective revaluation (e.g., with CT1+, T1T2+, and T2T3+ trials, C+ trials lead to a lower rating for T1, a higher rating for T2, and a lower rating for T3), one needs to assume that the sign of a depends on the type of relation between the cue that is present and the target cue that is absent: With first- and third-order relations, a is negative, but with second-order relations, a is positive. Although technically one can revise the Rescorla Wagner model in this way, logically it would make little sense. The a parameter is intended to model cue salience. Although one could accept that this parameter is positive when a cue is present and negative when a cue is expected but absent, it makes little sense to assume that a parameter that models stimulus salience could be positive for cues that are absent (as needs to be assumed in order to explain second-order effects). The revised SOP model can also be revised further in order to make it compatible with the present results. The revised SOP model as formulated by Dickinson and Burke (1996; see also Larkin et al., 1998) postulates that presenting a cue can put the elements of the representation of a target cue in the A2 state if the presented cue and the target cue share a first-order relation. In order to explain the results of the present studies, one needs to make the additional assumption that a cue can put the elements of a target cue in the A1 state when the cues share a secondorder relation and in the A2 state when the cues share a third-order relation. As was the case with the revision of the Rescorla Wagner model that was proposed earlier, this revision of the

14 150 DE HOUWER AND BECKERS SOP model is technically possible but logically implausible. The SOP model was built upon the assumption that the A1 state reflects the actual presence of a stimulus. However, in order to explain the present results, one needs to assume that the elements of an absent cue can also be in the A1 state. This assumption is inconsistent with the central tenets of the SOP model. Although our results are thus most compatible with the ideas that underlie the probabilistic contrast model and the extended comparator model, one should note that some previously reported findings are not compatible with the logic of these models. Most importantly, both the probabilistic contrast model and the extended comparator model predict that the order in which trials are presented should have no effect on behaviour. However, the results of numerous studies contradict this prediction (e.g., Lopez, Shanks, Almaraz, & Fernandez, 1998; Shanks, Lopez, Darby, & Dickinson, 1996). These models need to be modified, therefore, before they can be considered to be valid accounts of human causal learning (e.g., by incorporating recency weights, see Maldonado, Catena, Candido, & Garcia, 1999). REFERENCES Blaisdell, A.P., Bristol, A.S., Gunther, L.M., & Miller, R.R. (1998). Overshadowing and latent inhibition counteract each other: Support for the comparator hypothesis. Journal of Experimental Psychology: Animal Behavior Processes, 24, Cheng, P.W. (1997). From covariation to causation: A causal power theory. Psychological Review, 104, Cheng, P.W., & Holyoak, K.J. (1995). Complex adaptive systems as intuitive statisticians: Causality, contingency, and prediction. In J.-A. Meyer & H. Roitblat (Eds.), Comparative approaches to cognition (pp ). Cambridge, MA: MIT Press. Cheng, P. W., & Novick, L. R. (1990). A probabilistic contrast model of causal induction. Journal of Personality and Social Psychology, 58, De Houwer, J. (in press). Forward blocking depends on retrospective inferences about the presence of the blocked cue during the elemental phase. Memory and Cognition. De Houwer, J., Beckers, T., & Glautier, S. (in press). Outcome and cue properties modulate blocking. Quarterly Journal of Experimental Psychology: A. Denniston, J.C., Savastano, H.I., & Miller, R.R. (2001). The extended comparator hypothesis: Learning by contiguity, responding by relative strength. In R.R. Mowrer & S.B. Klein (Eds.), Handbook of contemporary learning theories (pp ). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Dickinson, A., & Burke, J. (1996). Within-compound associations mediate the retrospective revaluation of causality judgements. Quarterly Journal of Experimental Psychology, 49B, Dwyer, D.M., Mackintosh, N.J., & Boakes, R.A. (1998). Simultaneous activation of the representations of absent cues results in the formation of an excitatory association between them. Journal of Experimental Psychology: Animal Behavior Processes, 24, Larkin, M.J.W., Aitken, M.R.F., & Dickinson, A. (1998). Retrospective revaluation of causal judgments under positive and negative contingencies. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, Lipp, O.V., Siddle, D.A.T., & Dall, P.J. (1993). Effects of miscuing on Pavlovian conditioned responding and on probe reaction time. Australian Journal of Psychology, 45, Lopez, F.J., Shanks, D.R., Almaraz, J., & Fernandez, P. (1998). Effects of trial order on contingency judgements: A comparison of associative and probabilistic contrast accounts. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, Maldonado, A., Catena, A., Candido, A., & Garcia, I. (1999). The belief revision model: Asymmetrical effects of noncontingency on human covariation learning. Animal Learning and Behavior, 27, Miller, R.R., & Matzel, L.D. (1988). The comparator hypothesis: A response rule for the expression of associations. In G.H. Bower (Ed.), The psychology of learning and motivation, Vol. 22: Advances in research and theory (pp ). San Diego, CA: Academic Press.