Individual Differences and the Belief Bias Effect: Mental Models, Logical Necessity, and Abstract Reasoning

Transcription

1 THINKING AND REASONING, 1999, THE 5 (1), BELIEF 1 28 BIAS EFFECT 1 Individual Differences and the Belief Bias Effect: Mental Models, Logical Necessity, and Abstract Reasoning Donna Torrens and Valerie A. Thompson University of Saskatchewan, Canada Kenneth M. Cramer University of Windsor, Canada This study investigated individual differences in the belief bias effect, which is the tendency to accept conclusions because they are believable rather than because they are logically valid. It was observed that the extent of an individual s belief bias effect was unrelated to a number of measures of reasoning competence. Instead, as predicted by mental models theory, it was related to a person s ability to generate alternative representations of premises: the more alternatives a person generated, the less likely they were to show a belief bias effect. In contrast to belief effects, which were predicted by the number of alternatives generated, scores on logical reasoning tasks were predicted by an individual s understanding of the concept of logical necessity. These two mediating variables, ability to generate alternatives and understanding of logical necessity, were themselves predicted by similar variables, such as cognitive motivation and abstract reasoning ability. Our findings strongly support the mental models interpretation of the belief bias effect, and suggest that individual differences in this effect are associated with a specific ability to generate alternatives, rather than with a general logical competence. INTRODUCTION The ability to reason and use logic is one of the distinguishing characteristics of human intelligence. Nonetheless, much research has shown that people are subject to belief bias, and tend to accept conclusions that they believe to be true, regardless of whether these conclusions follow validly from their premises (e.g. Evans, Barston, & Pollard, 1983; Oakhill, Johnson-Laird, & Garnham, Requests for reprints should be sent to Dr Valerie Thompson, Department of Psychology, University of Saskatchewan, 9 Campus Drive, Saskatoon, SK. Canada, S7N 5A5. thompsonv@sask.usask.ca This research was supported by an operating grant from the Natural Sciences and Engineering Research Council of Canada to V.A. Thompson. We would like to thank Jim Cheesman and Jamie Campbell for helpful comments on an earlier draft of this manuscript Psychology Press Ltd

2 2 TORRENS, THOMPSON, CRAMER 1989). People also tend to accept conclusions drawn from premises that they believe, regardless of whether they are valid (Thompson, 1996). To date, most research has examined the effects of belief on logical thinking by manipulating the characteristics of the problems used to measure belief effects (e.g. Newstead, Pollard, Evans, & Allen, 1992; Oakhill et al., 1989; Thompson, 1996). Surprisingly, there has been little research examining the characteristics of individuals that may contribute to those effects. Consequently, the purpose of this paper was to examine individual differences in the extent to which beliefs influence logical thinking. Mental Models Theory The most successful framework for explaining the belief bias effect in deductive reasoning is mental models theory (Johnson-Laird, 1983; Oakhill & Johnson- Laird, 1985). The belief bias effect is a specific type of belief effect, namely the tendency to accept conclusions that accord with one s beliefs regardless of the actual validity of those conclusions. According to this view, people reason deductively by constructing an initial, quasi-imaginal model of the relationship among the premises. A prudent reasoner would check if there were any other models that were also consistent with the premises, and draw a final conclusion that was consistent with all models generated. According to mental models theory, the effect of beliefs is to pre-empt the search for alternative models. If the first model of the premises yields a believable conclusion, then the search for further models may be terminated. However, if the first conclusion is unbelievable, then the search for alternatives may be continued and a different conclusion reached. In this manner, people accept more believable than unbelievable conclusions, even when those conclusions are not necessarily valid (Newstead et al., 1992; Oakhill & Johnson- Laird, 1985; Oakhill et al., 1989). Individual Differences in Generating Alternatives. According to mental models theory, the search for alternative models is a key part of the belief bias effect. To the extent that this is true, then people who generate alternative representations should be more resistant to the effects of belief than those who fail to do so. However, as mentioned earlier, there is little direct evidence in support of this conclusion. Indeed, the studies that have supported the mental models interpretation have investigated the properties of the reasoning materials, rather than properties of the individuals who are reasoning. For example, Newstead et al. (1992) manipulated the complexity of syllogistic problems (i.e. the number of alternative representations that a set of premises affords), and found that mental models theory explained how problem complexity interacted with a conclusion s believability and validity. However, although mental models theory has successfully predicted the nature of the belief bias effect in specific

3 THE BELIEF BIAS EFFECT 3 problems, the relationship between an individual s ability to construct alternative mental representations and their susceptibility to the belief bias effect has yet to be determined. Generating Alternatives and Reasoning Performance. Although we know of no studies that have specifically examined the relation between one s ability to generate alternatives and the belief bias effect in deductive reasoning, there is evidence to support the role of alternative-generation in other aspects of reasoning performance. For example, Galotti, Baron, and Sabini (1986) found that good and poor reasoners may differ with respect to their ability to generate alternative representations. Participants, who pretested as good or poor reasoners, were presented with conditional syllogisms and asked to deduce both initial and final conclusions from the premises. Good reasoners spent more time reaching their final conclusion than poor reasoners, and were also more likely to change their initial conclusion to a correct final conclusion. The researchers suggested that good reasoners may have used their time to explore and consider more alternative models of the situation than did poor reasoners. Using a different paradigm, Markovits (1984) has also found that individual differences in the ability to consider alternatives may be related to deductive reasoning ability. He provided people with a conditional relation (e.g. when David has homework to do, he gets in a bad mood), and asked them to provide reasons for why David was in bad mood. Markovits observed that performance on a subsequent deductive reasoning task was related to performance on this alternatives task: participants who generated an alternative reason were less likely to reach fallacious conclusions than those who did not generate an alternative. Evidence from the domain of informal reasoning bears more directly on the relationship between generating alternatives and belief effects. Specifically, research on informal reasoning suggests that a process similar to the generation of alternative mental models may contribute to the extent to which beliefs influence informal reasoning. Perkins (1989) observed that biased, one-sided thinking (i.e. reasoning that accords with one s preconceived beliefs) may be associated with a failure to explore alternative solutions to a problem. Although the task demands of informal and formal reasoning are quite different, Perkins study lends at least tangential support to the hypothesis that the ability to generate alternatives may play an important role in mediating belief effects in more formal tasks. Specifically, the ability to question one s own beliefs, or at least to ignore these beliefs in order to generate alternative solutions to a problem, may determine an individual s susceptibility to the influence of beliefs in deductive reasoning. In summary, existing evidence suggests a link between the ability to generate alternatives and deductive reasoning ability, and between alternative-generation and belief effects in informal reasoning. Thus, it seems reasonable to hypothesise

4 4 TORRENS, THOMPSON, CRAMER that the ability to generate alternative representations of the premises is related to belief effects in deductive reasoning. To investigate this hypothesis, participants in the present study were given conditional syllogisms that employed both believable and unbelievable premises and conclusions, and were asked to determine which conclusions followed validly from the premises. This permitted us to assess the extent to which an individual s reasoning performance varied as a function of the believability of the materials with which they were reasoning. To assess their ability to generate alternatives, participants were also presented with two quantified syllogisms (e.g. Some of the A are not B; none of the B are C) and asked to draw as many diagrams as possible that were consistent with the premises. It was predicted that the extent of the belief bias effect observed on the conditional syllogisms task should be negatively related to the number of alternative representations generated on the alternatives generation task; that is, people who generated more alternatives should be less influenced by beliefs. In addition to testing the hypothesised relations between beliefs and alternatives, we also tested several alternative hypotheses that could potentially explain individual differences in the belief bias effect. Specifically, we examined the relationship between the belief bias effect and an individual s understanding of logical necessity, as well as their performance on other deductive reasoning tasks. We also examined the possibility that global characteristics, such as intelligence and personality variables, may underlie belief effects in reasoning. The rationale for each of these hypotheses is outlined below. Logical Necessity and Deductive Ability An alternative hypothesis for individual differences in belief effects concerns an individual s general logical abilities. For example, it is possible that people who misunderstand the concept of logical necessity, or who have poor deductive reasoning skills in general, might be susceptible to the effects of belief in deductive reasoning tasks. According to the misinterpreted necessity hypothesis, belief bias effects arise because people fail to understand that a valid conclusion must necessarily (not possibly) follow from the premises (Evans et al., 1983). According to this view, if one interpretation of the premises leads to a believable conclusion, then the conclusion may be accepted because the reasoner does not understand that a valid conclusion must be consistent with all, rather than some, interpretations. In a series of experiments, Newstead et al. (1992) contrasted predictions derived from the misinterpreted necessity and the mental models views, and found that the data supported the mental models theory over the misinterpreted necessity hypothesis. As in other studies, however, Newstead et al. investigated the properties of the reasoning problems, not the properties of the reasoners. Thus, it is still plausible

5 THE BELIEF BIAS EFFECT 5 that individuals who understand the concept of logical necessity will be less susceptible to the effects of beliefs on a logical reasoning task. For example, reasoners who understand that valid conclusions always follow from their premises may be less inclined to accept invalid, but believable, conclusions that only possibly follow from their premises. To test this hypothesis, we derived a measure of participants understanding of logical necessity by asking them to solve both deductive and inductive reasoning problems. The deductive problems produced conclusions that necessarily followed from their premises, whereas the inductive problems produced conclusions that only possibly followed. To the extent that one understands the concept of logical necessity, one should accept more deductive than inductive conclusions. Moreover, the logical necessity hypothesis suggests that this difference should be related to belief effects on the conditional syllogisms task described in the preceding section. An alternative possibility is that good deductive reasoners are more resistant to the effects of belief than poor deductive reasoners. Like the misinterpreted necessity hypothesis, existing data suggest that this alternative is unlikely. Specifically, Markovits and Nantel (1989) failed to find a relationship between performance on abstract conditional reasoning problems and the extent of an individual s belief bias effect. However, two aspects of their findings prevent firm conclusions. First, although the difference was not significant, the authors observed a tendency for belief effects to be stronger in those with low scores on the abstract reasoning task; thus, the failure to find a difference may be due to lack of statistical power. Second, it is problematic to draw general conclusions on the basis of a single null finding. Consequently, the relationship between abstract reasoning ability and the belief bias effect was examined in detail in the present investigation. To test this relationship, we provided two measures of deductive reasoning ability: first, similar to Markovits and Nantel (1989), participants were asked to solve a conditional syllogism that produced a belief-neutral conclusion. This problem allowed us to assess reasoning performance when beliefs were not an issue. In addition, we devised a test of abstract reasoning that used propositional connectives other than if, then. To the extent that the belief bias effect is predicted by deductive reasoning ability, people who do well on these two tests should be less likely to show a belief bias effect than reasoners who perform poorly. Intelligence and Personality Variables Finally, several hypotheses were investigated concerning variables that potentially predict individual differences in a reasoner s ability to generate alternatives. These hypotheses, which are outlined here, were derived from the literature on informal reasoning, which suggests that fairly global characteristics, such as intelligence and motivation, may be related to one s ability to generate

6 6 TORRENS, THOMPSON, CRAMER alternatives, and therefore, be indirectly related to individual differences in the belief bias effect. Thus, we investigated the possibility that the ability to generate alternatives is a mediating variable that directly influences the belief bias effect, and which is itself influenced by intelligence and motivation variables. To this end, we administered measures of cognitive motivation and open-mindedness to participants, together with a short-form intelligence test, the Shipley Institute of Living Scale. This latter test consisted of abstract reasoning and vocabulary subscales. Intelligence and Belief Effects in Deductive Reasoning. Perkin s (1985) work suggested that there might be a link between intelligence and the ability to generate alternatives. For example, the ability to generate alternatives in informal arguments may be related to intelligence variables, such as verbal and abstract intelligence. However, Perkins (1985) also found that intelligence did not predict the ability to generate alternatives that contradicted a reasoner s beliefs. Instead, intelligent reasoners generated more arguments supporting their views but not more arguments contradicting their views than did less intelligent reasoners. This suggests that although intelligent reasoners may be expected to generate a larger number of believable mental models, they should be no more likely to generate models that contradict their beliefs than would less intelligent reasoners. To the extent that an analogous set of relations holds for deductive reasoning tasks, one would expect general intelligence to be related to the ability to generate alternatives, but not necessarily to belief effects on the conditional syllogisms task. Personality and Belief Effects in Reasoning. Perkins (1985) suggests that factors other than those associated with intelligence may be necessary to explain why some intelligent people are capable of reasoning logically in spite of their beliefs, whereas others are not. These factors may include personality variables associated with the generation and consideration of alternative viewpoints. Consistent with this suggestion, Furlong (1993) found that the ability to generate and consider multiple arguments that contradict beliefs was related to personality characteristics associated with critical-thinking abilities, such as openmindedness and enthusiasm for performing cognitive tasks. Moreover, individuals who are highly motivated to perform cognitive tasks may think in more depth about issues, and therefore be more likely to explore and consider alternative viewpoints that contradict their beliefs than are individuals who are less motivated (e.g. Cacioppo, Petty, Kao, & Rodriguez, 1986). Thus, two personality traits, cognitive motivation and open-mindedness, seem to be related to the ability to generate and consider solutions that both support and contradict one s beliefs. Although these latter studies were concerned with reasoning in informal domains, it seems reasonable to hypothesise that these personality traits may also predict individual differences in belief effects for formal, deductive

7 THE BELIEF BIAS EFFECT 7 tasks. Specifically, we predicted that the variables of open-mindedness and cognitive motivation would predict scores on the alternatives generation task, and would also predict belief effects on the conditional syllogisms task. Present Study Working from a mental models point of view, we hypothesised that a person s ability to generate alternative representations of a set of premises should predict the extent to which he or she is susceptible to beliefs while reasoning. The ability to generate alternatives was, in turn, predicted to be influenced by general intelligence (verbal ability and abstract reasoning as measured by the SILS), cognitive motivation (as measured by the short form of the Need for Cognition Scale; Cacioppo, Petty, & Kao, 1984), and openness to new ideas (as measured by the Openness Scale of the NEO Personality Inventory; Costa & McRae, 1985). In addition, we tested two plausible alternative hypotheses, namely that belief bias effects would be influenced by a person s understanding of logical necessity, or by their overall deductive reasoning ability. Path analyses of these variables were performed in order to test alternative models of the direction and degree of contribution each predictor variable made to belief effects in reasoning. Participants METHOD A total of 87 female and 41 male introductory psychology students at the University of Saskatchewan participated in this study for partial course credit. Students ranged from 17 to 45 years of age (M=19.25 years), and were tested as a single group. Measures Predicted Variables. The effects of beliefs on reasoning were assessed using two-term conditional syllogisms taken from Thompson (1996). These problems had been rated for believability (Thompson, 1996) and were based on four types of subject-matter, namely birds, mammals, polio, and plants. Participants were presented with four problems, each of which consisted of two premises followed by four putative conclusions for the reasoners to evaluate. The two premises had the generic form If p, then q and If q, then r. Two of the four conclusions followed validly from the premises (i.e. if p, then r, and if ~r, then ~p ), and two did not (i.e. if r, then p, and if ~p, then ~r ). See Appendix A for example problems. Reasoners evaluated a total of 16 conclusions (two valid and two invalid conclusions for each of four problems). Participants were asked to assess the validity of each conclusion by indicating whether each conclusion followed Always, Sometimes, or Never, from the premises. A conclusion was defined as accepted if a subject indicated that it Always

8 8 TORRENS, THOMPSON, CRAMER followed from the premises. The conclusions were presented in one of four pseudo-random orders. Each participant received one problem in each of four believability conditions. The first problem was a neutral condition, in which both the premises and conclusions were neither believable nor unbelievable. This problem was the same for all participants, and used plants as the subject matter. Of the remaining three problems, one problem had believable premises and believable conclusions, a second had unbelievable premises and believable conclusions, and the third had unbelievable premises and unbelievable conclusions; the order of these conditions was counterbalanced across participants. In addition, the remaining three subject matters (birds, mammals, and polio) were counterbalanced so that they were used equally often in each believability condition. From these problems we derived the four dependent measures described here. Premise believability scores reflected the effect of premise believability on the judged acceptability of conclusions. These scores were calculated by subtracting the number of conclusions accepted when the premises were unbelievable from the number accepted for believable premises, holding the believability of the conclusion constant. In other words, premise believability was computed as the number of conclusions accepted in the believable premise/ believable conclusion condition minus those accepted in the unbelievable premise/believable conclusion condition (max = +4; min = 4). Conclusion believability scores were determined in a similar manner by subtracting the number of conclusions accepted in the unbelievable premise/unbelievable conclusion condition from those accepted in the unbelievable premise/believable conclusion condition (max = +4; min = 4). 1 For both measures, positive scores reflected belief effects (more conclusions accepted for believable than unbelievable materials). The number of valid conclusions accepted in all three believability conditions minus the number of invalid conclusions accepted in these three conditions determined the belief logic scores, representing an individual s logical performance on belief-laden material (max = +6; min = 6). Similarly, the number of valid minus invalid conclusions accepted in the neutral premise/ neutral conclusion condition determined the neutral logic scores, representing logical performance when beliefs were not a consideration (max = +2; min = 2). For both measures, positive scores reflected strong logical performance while negative scores represented poor logical performance. 1 These combinations of premise and conclusion believability were chosen to manipulate the largest number of variables using the smallest number of problems. The rationale for doing so was based on Thompson s (1996) findings, which indicated that the effects of premise and conclusion believability were independent, so that the effects of one variable were observed across all levels of the other variable.

9 THE BELIEF BIAS EFFECT 9 Predictor Variables. Three clusters of predictor variables, associated with intelligence, personality, and logical reasoning were included. The variables for the intelligence cluster included both verbal ability and abstract reasoning ability (as measured by the Shipley Institute of Living Scale or SILS); the variables for the personality cluster included motivation to perform cognitive tasks (as measured by the short form of the Need for Cognition Scale) and openmindedness (as measured by the Openness Scale of the NEO Personality Inventory); the variables for the logic cluster included the ability to generate alternatives, the ability to reach abstract deductive conclusions, and understanding of logical necessity, measures of which were designed for this study. Thus, a total of seven predictor variables were examined. The measures used to assess the seven predictor variables were all chosen or developed to be amenable to group testing in a limited (one-hour) time. In addition, the standardised measures of verbal ability, abstract reasoning, cognitive motivation, and openness were chosen to have high reliability and good construct validity. The Shipley Institutes of Living Scale, (SILS; Shipley, 1940) comprises two sub-scales, which measure verbal ability and abstract reasoning ability. The SILS was chosen because it is a short-form measure of intelligence that is highly correlated with a wide number of intelligence and academic achievement measures, in particular, both the verbal and the performance scales of the Wechsler Adult Intelligence Scale (WAIS) (see Buros, 1992). The verbal ability scale consisted of a 40-item vocabulary test, which required participants to choose the one word that meant the same thing as a given word, from a list of four alternatives. Abstract reasoning ability was measured by a 20- item scale requiring the subjects to determine the next item in a sequence of numbers, letters, or words. Scores for verbal ability were derived by adding the number of items correctly answered by the subject on the SILS. Abstract reasoning ability was likewise determined by adding the number of items correctly answered on the SILS. Motivation to perform cognitive tasks was assessed by the short-form of the Need for Cognition Scale (NCS; Cacioppo et al., 1984). This scale measures the extent to which a person likes to think and perform cognitive tasks. It is internally consistent and reliable in several studies (e.g. Cacioppo et al., 1986; Furlong, 1993). Construct validity of the NCS has also been established by a number of studies (e.g. Cacioppo et al., 1986). For the NCS, participants rated 18 statements on a five-point scale relating to their enthusiasm for performing cognitive tasks. High scores indicated high levels of cognitive motivation. Open-mindedness was measured by the Openness Scale of the NEO Personality Inventory (NEO-PI; Costa & McRae, 1985). The Openness Scale assesses an individual s tendency to be open to experiencing new situations or ideas. The NEO-PI is a widely used measure in personality research, which defines personality in terms of a five-factor model, including openness, extroversion, neuroticism, conscientiousness, and agreeableness. Each of these

10 10 TORRENS, THOMPSON, CRAMER five factors is considered to be a broad personality dimension. The reliability and validity of the NEO-PI, including each of the scales measuring the five factors, has been extensively tested (see Buros, 1992; Costa & McCrae, 1985). For the Openness Scale, participants rated 48 statements (on a five-point scale) that concerned their openness to new experiences or ideas. High scores indicated openness to new ideas and experiences and thus greater open-mindedness. Three new measures of logical reasoning were created for use in this study: one measured abstract deductive reasoning, the second was the alternatives generation test, and the third tested participants understanding of logical necessity. The abstract deductive reasoning measure assessed an individual s ability to recognise a valid deductive argument when presented in an abstract form (i.e. where only the form of the argument and not the content is being assessed). The alternatives generation test measured an individual s ability to derive different relational representations of premise information (in the form of Venn diagrams). The logical necessity measure assessed an individual s ability to distinguish between valid deductive conclusions (which necessarily follow from premise information) and inductive conclusions (which may or may not follow from premise information). These measures are described in detail next. Abstract deductive reasoning was measured using four abstract reasoning problems, taken from Braine, Reiser, and Rumain (1984). For these problems, participants were to indicate whether a given conclusion followed for each problem (e.g. It is not true that there is both a K and an L. It is false that there is not a K. Conclusion There is an L. ). Example problems are presented in Appendix B. Problems were selected on the basis of difficulty levels (Braine et al., 1984) and on the basis of a small pilot study. For the pilot study, eight questions of varying difficulty level (range 1.42 to 6.00) were solved by eight students. Because these items appeared to be relatively easy, (six of the eight pilot subjects had scores of seven out of eight or better), we selected the four most difficult problems for use in the current study, on the basis that (a) at least one of the eight pilot subjects had incorrectly answered the problem or, (b) that the difficulty level of the problem was 5.00 or higher based on Braine et al. s ratings. This task shall hereafter be referred to as abstract deduction ; a person s abstract deduction score was computed by adding the number of conclusions correctly deduced by a subject (min = 0; max = 4). For the alternatives generation task, participants were presented with two quantified syllogisms, consisting of two premises and five putative conclusions (see Appendix B). Participants were told to draw as many diagrams as possible that could represent the premises of the two problems. The alternatives generation score was assessed by adding the number of different diagrams that correctly represented the premises of the two syllogisms. Internal consistency for this measure was established by calculating a correlation between the number of diagrams generated for each of the two problems (r =.7805, P <.01).

11 THE BELIEF BIAS EFFECT 11 The measure of logical necessity required reasoners to distinguish between conclusions that necessarily and possibly followed from given information. For this purpose, participants were presented with four deductive and four inductive reasoning problems modelled on problems used by Galotti, Komatsu, and Voelz (1997); these problems contained neutral content so that believability of the materials would not be a confounding factor (see Appendix B). Deductive problems led to conclusions that necessarily followed from the given information, whereas inductive problems led to conclusions that might or might not follow from the given information. Reasoners who understood the concept of logical necessity would accept deductive conclusions but not inductive conclusions. Understanding of logical necessity was calculated as the difference between the number of deductive and inductive conclusions accepted (min = 4; max = +4). A conclusion was scored as accepted if the participant indicated the conclusion certainly or certainly did not follow from the premises, according to whether the problem was framed in a positive or negative manner. The higher the score, the better the understanding of logical necessity. Procedure Participants completed a booklet measuring each of the predictor and predicted variables examined. The four conditional syllogisms were presented at the beginning of the booklet; these were used to derive the four dependent measures (i.e. neutral logic, belief logic, conclusion believability, and premise believability). The remaining tests followed in a different random order for each participant. Participants also provided basic demographic information, such as age and gender, along with information as to whether or not they had taken a class in formal logic or statistics prior to participating in this study. It took approximately 40 minutes to complete all of the tasks. RESULTS Reasoning Performance with Believable and Unbelievable Problems The first set of analyses was performed as a manipulation check of the conditional syllogisms task, in order to confirm that reasoning performance differed as a function of the believability of premises and conclusions. For these analyses, the number of conclusions accepted in each of the three belief by validity conditions (max = 2) was the dependent variable. Table 1 summarises these data. The effects of conclusion believability and validity were examined by comparing performance for problems having unbelievable premises and

12 12 TORRENS, THOMPSON, CRAMER TABLE 1 Number of Conclusions Accepted as a Function of Validity and Believability Condition Believability Condition Neutral Believable Unbelievable Unbelievable Premise Premise Premise Premise Neutral Believable Believable Unbelievable Conclusion Conclusion Conclusion Conclusion Valid M SD Invalid M SD N = 128. believable conclusions to those having both unbelievable premises and unbelievable conclusions. A 2 (conclusion believability) 2 (validity) withinsubject analysis of variance (ANOVA) was conducted. As expected, significant main effects were found for both conclusion believability, F(1, 127) = 43.25, MSe = 26.74, P <.001; and conclusion validity, F(1, 127) = 51.61, MSe = 19.14, P <.001. Participants accepted more believable than unbelievable conclusions, and accepted more valid than invalid conclusions. The interaction between these variables was not significant, F(1, 127) = 2.84, MSe =.71, P >.05. In order to analyse the effects of premise believability, we compared performance for problems having believable premises and conclusions to those having unbelievable premises and believable conclusions. A 2 (premise believability) 2 (validity) within-subject ANOVA found no main effect for premise believability, F(1, 127) = 1.13, MSe =.78, P >.05, indicating that the number of conclusions accepted did not vary as a function of premise believability. However, a significant main effect for validity was found, F(1, 127) = 64.79, MSe = 27.20, P <.001, whereby participants accepted more valid than invalid conclusions. Again, the interaction between these variables was not significant, F < 1. This failure to find an effect of premise believability is puzzling, given that such an effect has been found previously, using similar materials (Thompson, 1996). However, the two studies used slightly different procedures, which could account for the discrepancy in findings. In the present study, both premise and conclusion believability were manipulated within subject, whereas in Thompson s study conclusion believability was held constant for each participant. It is possible that the effects of conclusion believability are more

13 THE BELIEF BIAS EFFECT 13 salient than those of premise believability, so that conclusion believability masked the effects of premise believability. Regardless, given that there was no systematic effect of premise believability, this effect was not analysed further (although correlations with this variable are reported for the sake of completeness). The focus of this study will therefore lie with the analysis of conclusion believability. Individual Differences in Conclusion Believability Correlational Analyses. Summary data for the seven predictor variables (verbal ability, abstract reasoning, abstract deduction, alternatives, logical necessity, cognitive attitudes, and open-mindedness) are reported in Table 2; the correlations between these variables and the four predicted variables (belief logic, neutral logic, conclusion believability, and premise believability) are presented in Table 3. Note that the Ns vary because some participants failed to complete all of the tasks. There are two general points to be made before the data presented in Tables 2 and 3 are interpreted. First, because the hypotheses outlined in the introduction make predictions regarding both the presence and absence of correlations, it is relevant to establish that our experiment had sufficient power to detect relationships among variables. A power analysis established that an N of 125 produced sufficient power to be almost certain of detecting a small correlation (r =.30) with a =.05 ([1 ß] =.91 for a one-tailed and.95 for a two-tailed test). Indeed, even for very small correlations (.20 r.25), 125 participants afforded a high probability to detect a correlation if present (.60 [1 ß].80 for twotailed tests, and.71 [1 ß].88 for one-tailed tests). Thus, it would appear reasonable to infer that the absence of a correlation between two variables is not due to a lack of statistical power to observe an effect. 2 A second general point concerns the measurement of belief effects. To compute the effects of belief, we subtracted the number of unbelievable conclusions accepted from the number of believable conclusions accepted; this gave a score ranging from - 4 to +4. This procedure has been used previously as an index of belief effects (Evans, Newstead, Allen, & Pollard, 1994), and provided a straightforward measure of the effects of belief on reasoning: the higher the score, the more likely it is that a person accepts conclusions that accord with their beliefs. However, there is a potential problem in interpreting negative scores, which would seem to indicate that a reasoner was less likely to accept 2 A related concern is that the measure of belief effects is based on a relatively small number of problems. This might reduce the probability of observing a correlation by restricting the possible range of values that belief scores might take. This argument is made less plausible by the fact that the neutral logic scores varied in an even narrower range, but still correlated with a large number of variables. Thus, the fact that the belief scores were related only to the number of alternatives generated and not to the other measures is not likely to be due to the restricted range of values.

14 14 TORRENS, THOMPSON, CRAMER TABLE 2 Summary Statistics for Predicted and Predictor Variables Variable (N) M SD Range Premise Believability (128) Conclusion Believability (128) Belief Logic (128) Neutral Logic (128) Verbal Ability (128) Abstract Reasoning (127) Cognitive Motivation (128) Open-mindedness (113) Abstract Deduction (118) Alternatives (123) Logical Necessity (125) TABLE 3 Correlations Among Predicted and Predictor Variables Predicted Variable Premise Conclusion Belief Neutral Predictor Variable Believability Believability Logic Logic Verbal Ability ** 0.35*** Abstract Reasoning *** 0.30** Cognitive Motivation ** 0.28** Open-mindedness * Abstract Deduction 0.24* Alternatives * 0.27** 0.26** Logical Necessity *** 0.31** *P <.05. ** P <.01. *** P <.001 (all P-values are two-tailed). believable than unbelievable conclusions. In our sample, there were a total of 18 people with negative belief scores. The most likely explanation for these people s scores is that they had a different perspective on which were believable and unbelievable conclusions; alternatively, their reasoning may have been motivated by variables not considered in this study. Because it seemed likely that these reasoners represented the normal variability expected in a sample of this type, we decided not to discard these individuals from the sample. Instead, we performed all of our analyses twice; once including the 18 negative scores, and a second time excluding them. Their presence or absence made very little difference to any of the analyses we report here, and for the sake of brevity, we will report only the analyses that include the negative scores, and note any changes that occurred when the 18 scores were excluded.

15 THE BELIEF BIAS EFFECT 15 The correlations reported in Table 3 provided an initial test of our main hypotheses: 1. Mental Models and Belief Effects. The prediction derived from mental models theory, namely that an individual s belief scores would be related to their ability to generate alternative representations of deductive premises was supported. As indicated in Table 3, there was a reliable, negative correlation between the number of alternatives generated and a reasoner s belief scores (r =.23, P <.05), indicating that the more alternatives a person generated, the less susceptible they were to belief effects when reasoning. 2. Alternative Sources of Belief Effects. We had suggested two alternative hypotheses regarding the relationship between belief effects and two other individual differences measures, namely a person s understanding of the concept of logical necessity, and their overall deductive reasoning ability. Neither of these hypotheses was supported by our data. The correlation between our measure of logical necessity and our measure of conclusion believability was not significant (r =.02; P >.05), nor were the correlations between our measures of deductive reasoning and conclusion believability for the abstract deduction task, (r =.07, P >.05) and for the neutral logic measure, (r =.11, P >.05). 3. Intelligence and Belief Effects. It was predicted that measures of intelligence, such as the verbal and abstract ability scales from the SILS, would correlate with a person s ability to generate alternatives but would not necessarily correlate directly with belief effects. This pattern was observed. Both the verbal and abstract scales from the SILS were positively related to the alternatives generation scores (r =.20 and.25 respectively, P <.05), but not with the conclusion believability scores (r =.01 and.11, P >.05). 4. Motivation, Open-mindedness, and Belief Effects. It was predicted that two variables, cognitive motivation and open-mindedness, would be associated with the ability to generate alternatives as well as with belief effects. These predictions were only partially supported. The measure of cognitive motivation correlated with the alternatives measure (r =.23, P <.05), but was related to conclusion believability only when a one-tailed test was used (r =.16, P =.035 one-tailed,.071 two-tailed). This suggests that people who enjoy thinking and performing cognitive tasks generated more alternatives than those who do not, and were also less influenced by beliefs when reasoning. The open-mindedness measure, however, did not correlate with either the alternatives measure or the conclusion believability scores (P >.05). 5. Individual Differences in Logical Reasoning. Although specific predictions were not generated for the logical reasoning variables, we examined the relationship between our two measures of conditional reasoning, namely neutral logic and belief logic, and the predictor variables. The correlations suggest that these two logic scores were associated with a different cluster of variables than the belief scores. Unlike the belief scores, these variables were associated with

16 16 TORRENS, THOMPSON, CRAMER both of the SILS sub-scales (r.24, P >.05), and with the logical necessity measure (r.31, P >.05). 3 Further support for the conclusion that reasoning competence and belief effects are influenced by separate factors was found when the effects of prior training in logic were examined. We found that participants self-reports of prior training were related to several measures of logical performance, including an understanding of logical necessity, verbal ability, abstract reasoning ability, and performance on the neutral logic task (r =.18,.18,.20, and.20 respectively, P <.05), but not with either the alternatives generation task or the conclusion believability scores (r =.07 and.02, P >.05). These data suggest that formal logical training, like intelligence and understanding logical necessity, predicts an individual s deductive reasoning ability, but not the extent to which he or she manifests belief effects. In summary, whereas logical competence, as measured by the neutral logic and belief logic scores, appears to be related to a number of individual difference variables, conclusion believability appears to be primarily related to the alternatives measure. Yet, as is shown in Table 4, five of the seven predictor variables, including ability to generate alternatives, were inter-correlated. This suggests that one of these other variables, rather than alternatives generation per se, might predict conclusion believability. Accordingly, path analyses were performed to confirm these initial conclusions. 3 Step-wise multiple regression analyses confirmed that logic scores are predicted by a different set of variables than belief scores. Two analyses (one including and one excluding the 18 participants with negative belief scores) were performed on three of the dependent variables; conclusion believability, belief logic, and neutral logic. In both analyses, only one factor predicted conclusion believability; namely the number of alternatives generated (R =.27, MSe = 2.19, P <.01 with all participants included and R =.24, MSe = 1.64, P <.05 with 18 participants excluded). This factor accounted for 7.3% of the total variance in the first analysis and 5.9% in the second. In the case of neutral logic and belief logic, a cluster of three variables (namely verbal ability, cognitive motivation, and understanding of logical necessity) entered the regression equation as predictors. Which of these factors were entered depended slightly on whether the 18 subjects were included or excluded. For neutral logic, verbal ability and cognitive motivation entered the equation when all subjects were included (R =.43, MSe =.88, P <.001), accounting for 14.6% and 4.0% of the variance respectively; with the 18 subjects excluded, verbal ability and logical necessity entered the equation (R =.48, MSe =.81, P <.001), accounting for 19.2% and 4.2% of total variance. Similarly, for belief logic, logical necessity and cognitive motivation entered the equation when the whole sample was analysed (R =.54, MSe = 2.13, P <.001), accounting for 21.8% and 7.5% of the variance respectively. When the 18 participants were excluded, logical necessity and verbal ability entered (R =.58, MSe = 1.9, P <.001), accounting for 29.1% and 4.7% of the variance. Regardless of the specific variables that entered the equation, the general conclusion is the same: different factors predict belief effects and logic scores.

17 TABLE 4 Inter-correlations Among Predictor Variables THE BELIEF BIAS EFFECT Verbal Ability 0.38*** 0.35*** * 0.33*** 2. Abstract Reasoning 0.19* ** 0.41*** 3. Cognitive Motivation 0.48*** * 0.29** 4. Open-mindedness Abstract Deduction Alternatives 0.22* 7. Logical Necessity *P <.05. **P <.01. ***P <.001 (all P-values two-tailed) Path Analyses. The correlational analyses suggested a model for how to integrate the variables tested. The hypothesised model, presented in Fig. 1, suggests that the effects of conclusion believability are mediated directly by the number of alternatives generated, whereas both neutral and belief logic are mediated by logical necessity. In turn, understanding of logical necessity and generating alternatives are linked to both verbal and abstract reasoning ability, as well as to cognitive motivation. In the model, cognitive motivation predicts both verbal ability and abstract reasoning, and is predicted by openness to new ideas. Note that the abstract deduction measure was not included in the analysis because it did not correlate with any of the other variables. This model could be tested using several multiple regression analyses; however, to minimise the inflation of Type I error, a path analysis (EQS; Bentler, 1995) was selected because it could test the feasibility of all regression models simultaneously. There is a potential concern that path analysis may not be appropriate given the low number of observations (N < 150). Despite this concern, however, according to Cole (1987), the present sample size is adequate, albeit small, to produce a stable path structure. Initially, it was of interest to test the hypothesised model using a confirmatory analysis. However, the viability of alternative models derived by multivariate Wald and Lagrange Multiplier tests (Bentler & Chou, 1987; Lee & Bentler, 1980; Satorra, 1989) was evaluated using exploratory post-hoc analyses. The shaded and un-shaded rectangles in Fig. 1 represent, respectively, the exogenous and endogenous variables in the model; absence of a line connecting variables implies no hypothesised direct effect. The assumptions of linearity and normality were evaluated, and the hypothesis that all variables were un-correlated was easily rejected x 2 (36, N = 128) = , P <.05. The degree to which the hypothesised model fit the sample data was assessed by several popular measures, namely the x 2 goodnessof-fit test statistic (assessing the magnitude of residual covariance or error) and both the comparative fit and Bentler-Bonett non-normed fit indices (CFI and NNFI, respectively; Bentler & Bonett, 1980). Because the test statistic measures

18 18 18 TORRENS, THOMPSON, CRAMER OPENNESS TO IDEAS FIG. 1. Hypothesised path model.

19 THE BELIEF BIAS EFFECT 19 unexplained variance, a non-significant value suggests good fit; and index values above.90 also suggest good fit. For the hypothesised model, both the test statistic and fit indices indicated poor fit: x 2 (22, N = 128) = 57.76, P <.001; NNFI =.687; CFI =.809. Post hoc modifications were performed to develop a better fitting and possibly more parsimonious model. Results of the multivariate Wald and Lagrange Multiplier tests (for parameter exclusion and inclusion, respectively) suggested three changes to the model: (1) like openness to ideas, abstract reasoning should be an exogenous variable influenced by no other variables, but should itself influence both verbal ability and belief logic; (2) verbal ability should influence neither logical necessity nor the number of generated alternatives, but should influence neutral logic; and (3) the number of generated alternatives should influence neutral logic. This exploratory model showed very good fit by all fit indices: x 2 (23, N = 128) = 32.66, P =.0968; NNFI =.933; CFI =.949. Figure 2 includes the standardised parameter estimates in the resulting diagram, which indicates that openness to ideas influenced cognitive motivation which, together with abstract reasoning, influenced each of, verbal ability, logical necessity, and the number of generated alternatives. All three moderators influenced neutral logic, only logical necessity and abstract reasoning influenced belief logic, and only the number of generated alternatives influenced conclusion believability. The same final path structure emerged when the 18 participants with negative belief scores were eliminated from the analysis. Conclusions. The path analysis confirmed several hypotheses. First, as predicted by mental models theory (e.g. Johnson-Laird, 1983; Oakhill & Johnson-Laird, 1985), the magnitude of the belief bias effect was predicted by the number of alternative models that reasoners were able to generate. Second, different variables predicted the extent to which reasoners were influenced by beliefs, and the number of valid inferences that they were able to make. Specifically, whereas belief effects were directly influenced by the number of alternatives generated, the two logic scores were directly influenced by the logical necessity measure. Finally, the path analysis supported the hypothesis that even though different measures of performance (i.e. conclusion believability, belief logic, neutral logic) may be predicted by different specific abilities (i.e. generating alternatives and understanding logical necessity), these abilities in turn are predicted by an interrelated cluster of more general abilities, such as abstract reasoning and cognitive motivation. DISCUSSION Past research has investigated the belief bias effect by manipulating properties of the stimulus materials about which reasoners are asked to draw inferences. In contrast, the current study investigated properties of the reasoner that may