FULL REPORT OF RESEARCH ACTIVITIES Background There has been a recent upsurge of interest in individual differences in reasoning which has been well summarised by Stanovich & West (2000). The reason for this is that certain types of individual differences can shed light on unresolved theoretical debates within the psychology of thinking and reasoning. One of the major debates concerns human rationality: the extent to which humans are capable of rational, logical thought. There are numerous instances of people giving non-rational answers to reasoning problems and the question arises as to whether this demonstrates lack of rationality or whether there are alternative explanations. It has been claimed by some that the incorrect normative model has been applied, and that humans are rational but not in the sense of adhering to the tenets of logic. One line of evidence which goes against this is the well-established finding that people higher in intelligence are more likely to produce the logically correct answer to reasoning problems. This seems to indicate that the correct normative model has been applied (Stanovich & West, 2000). A related issue concerns the existence of two different types of thinking. It has been claimed by many that there are two ways of thinking, one slow, conscious, rational and controlled, the other fast, unconscious, experiential and automated. Unfortunately these are given different names by different people and are also defined in different ways (see Newstead, 2000). Nor is there any accepted way of determining which type of thinking is being used or of determining how they relate to each other. Once again, individual differences may throw light on this. Recently-developed tests such as the Rational- Experiential Inventory (REI) (Pacini & Epstein, 1999) have the potential to measure the extent to which a person relies on rational thought or prior experience. Investigating how such thinking styles relate to reasoning performance can indicate which mode of thinking has been applied. Previous research, principally on statistical reasoning, has indicated that rationality as measured by the REI, which is essentially a measure of the willingness to engage in logical thinking, is related to accuracy on these problems. On the other hand experientiality, which is the tendency to use intuition and to rely on prior experience, is related to susceptibility to biases.
Objectives There were a number of objectives in the present research programme. To determine the reliability and usefulness of the REI. This measure has been developed and used only in the USA, and an important preliminary to the present research was to establish whether the scales are robust and suitable for use on a UK sample. To replicate the correlation between intelligence and performance on deductive reasoning tasks such as syllogistic reasoning and the Wason Selection Task. In addition, the aim was to look at the relationship between intelligence and statistical reasoning tasks, and to extend this to other reasoning tasks not previously used. To investigate the relationship between thinking styles and performance on reasoning tasks. The initial interest was primarily in rationality and experientiality as measured by the REI. It was predicted that rationality would tend to correlate with overall accuracy and experientiality with biases in responding. Our research also went beyond these measures of thinking style and also used the Thinking Disposition Composite, the Approaches to Studying Inventory, and a range of measures designed to investigate the ability to produce different representations of the information presented. It should be noted that the objectives of the present research were to some extent modified as the research progressed in order to take account of novel findings which emerged. In particular, the first series of experiments produced results which were completely unexpected and flew in the face of previous experimental findings. Rather than blindly following our original plan of research, which was in any case substantially undermined by these anomalous findings, we were inevitably side-tracked into investigating the source of the discrepancies. This led to us carrying out extended studies using a much wider range of individual differences measures than anticipated and a much wider selection of deductive reasoning tasks. This search for an explanation of our findings was ultimately successful and led to novel empirical findings which have considerable theoretical importance. However, it meant that we had less time to spend on the latter part of our initial project proposal,
with the result that we did not, as planned, investigate individual differences in statistical reasoning. Methods and Results In total eight studies were conducted, three more than were outlined in the original proposal. The method and results for each of these will be outlined in turn. Study 1 In Study 1 we validated the REI using a postal sample of 128 members of the general population and 98 undergraduate students. For the student sample we also administered a test of general intelligence. The internal and test-retest reliabilities of all the REI scales were high and the factor structure was exactly as expected. Consistent with previous findings, men scored higher than women on rationality whereas women scored higher than men on experientiality. These scales were unrelated to general intelligence. This study confirms that the REI is a scale with good psychometric properties and one which could be used effectively on a UK sample. Study 2 Study 2 examined the relationship between the REI, intelligence, and performance on abstract and realistic versions of the Wason selection task. This task is probably the most widely used in the psychology of reasoning. In the abstract version people are given four cards which have a letter on one side and a number on the other. They can see the top side of the cards, for example A, D, 4 and 7, and are asked which cards they need to turn over to determine whether the rule If there is an A on one side of the card then there is a 4 on the other side is true. The correct answer is to turn over the A and 7 cards, the latter because it would disprove the rule if there was an A on the other side. However, the great majority of people give the wrong answer, saying it is necessary only to turn over the A and 4 cards. Certain realistic versions of the task, e.g. using the rule If a person is drinking beer they are over 18, have been shown to produce massive facilitation (Griggs & Cox, 1982). It was expected that there would be a correlation between intelligence and performance on abstract selection tasks since this is the typical finding in previous research (e.g. Valentine, 1975; Stanovich & West, 1998a). It was
also expected that the measure of experientiality would correlate with performance on realistic tasks since the correct response has been assumed to derive in part from the use of prior experience. No prediction was made concerning correlations with rationality. The 98 participants, were given the 40-item version of the REI (Pacini & Epstein, 1999), and Part 1 of the AH5 intelligence test (Heim, 1968). Four selection tasks were used, two abstract and two deontic. The abstract problems were the original letter-number version ( If a card has an A on its letter side then it has 4 on its number side ) and a destination version ( If Glasgow is on one side of the ticket, then train is on the other side of the ticket ). The instructions asked participants to indicate which of the four cards (e.g., A, K, 4, and 5 in the first example) needed to be turned over to test whether the rule was true or false. The two realistic tasks were an anglicised version of the Sears problem in which Sears became Debenhams ( Any sale over 30 must be approved by the manager, Mr Jones ) and the drinking age problem ( If a person is drinking beer then the person must be over 18 years of age ). The instructions asked participants to choose the cards they thought needed to be turned over in order to test whether the rule was being violated. There was the usual facilitation of realistic over abstract selection tasks. However, the correlations between the selection tasks and the individual differences measures were quite unexpected. In contrast to previous findings, there was no correlation between intelligence and performance on abstract tasks. However, there was a significant correlation between intelligence and the realistic problems. These surprising findings challenge the received wisdom in this area. Neither of the REI scales correlated with performance on the selection tasks. We had predicted that experientiality would correlate with accuracy on the deontic (realistic) tasks. One possible explanation for the discrepancy with previous findings is that we had used an inappropriate measure of intelligence or that the measures of deductive reasoning were unreliable or inaccurate in some way. In order to check on this, we decided to bring back as many of the 98 participants as we could to repeat and extend the testing we had carried out. Study 3
In this study, 44 of the participants used in Study 2 were given the REI and the same selection tasks as before. In addition they were given a further measure of ability, the AH4 Parts 1 and 2 (Heim, 1967). Letter distance estimation, a measure of working memory, was also used. Sixteen syllogistic reasoning problems were used which lead to conclusions which were either valid or invalid and either believable or unbelievable. This is an example of a valid, believable syllogism: No dogs are unhappy Some mammals are unhappy Therefore, Some mammals are not dogs The short form of the Approaches to Studying Inventory was used (Newstead, 1992) which measures three approaches to studying. This was included mainly for exploratory purposes. The various individual differences measures were found to be generally reliable. However, the selection tasks were less reliable, giving low test-retest scores. Nevertheless, the classic facilitation of realistic over abstract tasks was once again obtained. There was again no correlation between intelligence and performance on abstract selection tasks; in fact the correlations were (nonsignificantly) negative. As in Study 2 there were significant correlations between intelligence and performance on realistic selection tasks. There were no significant correlations between either experientiality or rationality and performance on any of the selection tasks. The unusual and completely unexpected pattern of findings on the selection task observed in Study 2 was thus confirmed in Study 3. With the syllogisms, the classic findings were broadly confirmed. There was a main effect of logic, with participants endorsing more valid than invalid conclusions. They also endorsed more believable than unbelievable conclusions, though this effect was not significant, and there was an interaction between these two variables. The main interest is in how these variables relate to measures of individual differences. Intelligence was significantly negatively related to belief; in other words, people with higher levels of intelligence were less likely to respond simply in line with the believability or otherwise of the conclusion. The correlation between intelligence and logic was not significant overall, but on those syllogisms where belief and logic were in conflict the correlation achieved significance. Participants of higher ability are thus able to
resist the belief-based response and provide a solution consistent with the logical requirements of the task. Rationality was related to neither belief nor logic, but experientiality showed a negative relationship with logic. The findings on syllogistic reasoning were thus similar to what has been obtained in previous research. Although Study 3 replicated the findings of Experiment 1, the sample size was relatively small. The next study used a larger number of participants and additional tasks. Study 4 This study used 152 participants and similar materials to Study 3. The AH4 and REI were used, and the same 16 syllogisms. Eight selection tasks were used, four abstract and four realistic. The ASI and the letter distance tasks were dropped, but a further measure of thinking style, the Thinking Dispositions Composite (Stanovich and West, 1998b) was included. It was hoped that this might provide a measure related to intuitive thinking, though in actuality it produced few results of interest. There were no significant correlations between selection task performance and any of the individual differences measures. The unexpected positive correlation between intelligence and deontic tasks was not significant in the present study. However, it was positive and on two of the tasks (the two we had used in Experiments 1 and 2), the correlation was significant. There was no overall correlation between intelligence and abstract selection tasks though for the first time in this series of studies there was such a correlation on one of the tasks, the original letter-number version. On the syllogisms, intellectual ability correlated with logic scores and with scores on conflict problems (i.e., those where belief and logic point to different conclusions). In addition, rationality (as measured by the REI) correlated positively with performance on nonconflict problems. Experientiality did not correlate with any aspect of reasoning performance. Thus these results broadly confirmed the findings of the earlier experiments. The curious correlation between ability and performance on realistic selection tasks was found once again, though only on the two selection tasks used in earlier studies. In this experiment, for the first time, we observed a link between ability and
abstract tasks, though this was observed on only one problem, the standard letter-number task. We were still unsure as to why our results were out of line with previous research and decided to carry out another study, this time on sixth form school students. Study 5 Participants were 126 secondary school children aged between 16 and 18. The REI and AH4 were used and also an additional measure of intellectual ability, Raven s Progressive Matrices (Raven, 1983). Eight selection tasks were used, four abstract and four realistic, and the syllogisms task as presented in Studies 3 and 4. AH4 and Raven s Matrices scores correlated significantly and hence a composite measure of intellectual ability was used. On the selection tasks, there was a significant but much smaller than usual facilitation on the realistic tasks. The ability composite correlated with only one of the four abstract tasks and rationality correlated significantly, but negatively, with one of the other abstract problems. In contrast to the previous experiments there was no evidence that realistic tasks correlated with ability. The pattern of results for the syllogisms was somewhat different to that normally observed. There was a smaller than usual effect of validity but a very large effect of belief, and the interaction between these factors did not approach significance. Intellectual ability correlated with logic scores and with accuracy on nonconflict problems. However, in contrast to the previous experiments, ability did not correlate with conflict problems nor did it correlate with belief. There were no correlations between syllogistic reasoning and either rationality or experientiality. ***[more here on data?] In some ways the results from this study are even more discrepant with previous findings than our earlier studies, though the participants were younger than are typically used. We began to suspect that our samples might have been different in some ways to those used in earlier studies, though at this stage we were not sure whether this was to do with motivation, ability or some other factor. We felt we needed more data, and using different kinds reasoning tasks and different measures of individual differences. Study 6
In addition to the measures used in previous studies (AH4 and REI), Study 6 used a new type of measure of thinking style involving measures of the ability to generate alternative representations. The highly influential mental models theory of reasoning (Johnson- Laird & Byrne, 1991) claims that the construction of alternative representations is a key factor in reasoning. People are assumed to construct a representation (or model) of the premises, generate a conclusion, and then attempt to produce alternative mental models in which the conclusion does not hold. If this cannot be done, then the conclusion is assumed to be valid. Torrens, Thompson & Cramer (1999) used a measure of alternatives generation involving the construction of diagrams of different models of syllogistic reasoning premises. They found that it predicted well certain aspects of reasoning performance, a finding that has been confirmed by Newstead, Thompson & Handley (in press). We used this measure and also a well known measure of creativity, the uses of objects test, and a measure of possible antecedents for conditional statements first used by Markovits (1984). This latter involves producing alternative ways in which the consequent of a conditional might be true. For example, given the conditional statement If it rains then the road will get wet participants were asked to indicate as many different ways as possible for the road to become wet. In addition we used two tasks not previously used. One of these involved constructing unlikely scenarios to explain apparently impossible events, for example how a murder suspect might be guilty even though he was on a train some miles away when the murder occurred. The final task involved listing potential experimental confounds in a simple study. Our interest was both in how these various measures of alternatives generation related to each other and in how they related to reasoning performance. In addition to selection tasks and syllogisms we also used four versions of the THOG problem (Wason and Brooks, 1979) and a conditional reasoning task involving shapes and colours. 70 participants were used in this study. Two new measures of individual differences used for the first time in this study produced results of particular interest. The alternatives generation task in which diagrams representing syllogisms had to be produced correlated with intelligence and with performance on abstract selection tasks. Multiple regression showed that it was
predicting variance in performance on these abstract tasks independently of intelligence. The other measures of alternatives generation correlated with each other but did not serve as good predictors of reasoning performance. The second measure of interest was one derived from the conditional reasoning task: the tendency to resist the fallacious inferences. The fallacious inference of affirmation of the consequent involves inferring that p is true when given the conditional statement If p than q, and the further information that q is true. Denial of the antecedent involves fallaciously inferring not q when given the same conditional statement and the further information that p is true. Both of these errors are extremely common. However, there are many people who resist them and this tendency correlated with intelligence, with performance on abstract selection tasks, with performance on deontic tasks (though to a lesser extent), with logic scores on the syllogisms and with accuracy on conflict syllogisms. Perhaps the most surprising aspect of the present results, however, was that the classic pattern of correlations on the selection tasks was obtained: a significant correlation between intelligence and abstract tasks and a positive but nonsignificant correlation on the realistic tasks. What is more, our data give a powerful clue as to why this pattern should have returned. In the present study the overall level of intelligence was in line with the published norms for the AH4 test. Re-analysis of the scores from Studies 2-5 revealed that in each case the intelligence scores were lower than the norms. In other words, we had, quite by chance, sampled the lower end of the ability range in our earlier studies. But why should this lead to the correlations with intelligence changing? We believe we have evidence that there are three main levels of performance on the selection task. At the first level are context specific reasoners who treat every task as independent from others and use whatever contextual cues may be present in the scenario. At the second level are pragmatic reasoners who realise that there are commonalities between different selection tasks but fall prey to the pragmatic inference present in conditionals, for example interpreting them as biconditionals. At the highest level are the formal reasoners who can overcome contextual cues and analyse the logical structure of the problems. We believe that in Studies 2-5 we had virtually no participants in the formal reasoning group, hence there were very few correct answers on the abstract tasks and no
correlations with intelligence. Since we were sampling only from the context specific and pragmatic reasoners, a correlation between intelligence and performance on realistic tasks would be expected, since the more intelligent participants were those who are capable of using the pragmatic cues present in realistic tasks. To test these suggestions we conducted extensive re-analyses of the data from Studies 2-5. We assumed that context specific reasoners would give different responses to the various abstract tasks that were given while pragmatic reasoners would give consistent responses. In all of the experiments, consistent responders were of higher intelligence than inconsistent responders. Pragmatic reasoners would be expected to give the cards named in the rule as their responses to abstract tasks since these are cued by the conditional statement. They would also be expected to give the correct answer on the deontic tasks. Analyses indicated that those who selected the cards named in the rule were of higher intelligence than those who did not, and those who gave this response plus the correct answer on deontic tasks were also of higher intelligence. The findings in Study 6 on conditional reasoning were also of interest in their own right. In particular, it was found that logical performance on most logical inferences correlated with intelligence and alternatives generation, but that the modus tollens inference behaved very differently. This is the (valid) inference that not p follows from not q. This inference was independent of intelligence but correlated with belief bias, suggesting that it may be drawn though pragmatic cues rather than logical analysis. Study 7 The aim of this study was to look in more detail at individual differences in a variety of propositional reasoning tasks, thus replicating and extending the findings of Study 6. This study used a variety of individual differences measures including the AH4, Markovits alternative antecedents task, uses of objects, experimental confounds, unusual scenarios, the REI, and possible diagrams. In addition to a variety of propositional reasoning tasks, various types of abstract selection tasks were also used. In the propositional reasoning task, the 102 participants were given 64 statements containing abstract statements involving letters and numbers. The connectives used were if, either or but not both,
either or or both, and if and only if. Negation was systematically manipulated in the antecedent and consequent of the first premise, as was affirmation or denial of the antecedent and consequent in the second premise. An example of a statement using if and only if, and with negated antecedent and affirmative consequent is and with a second premise containing a denial of the consequent is: If and only if there is not a 3, then there is a Z There is not a Z Which of the following conclusions is valid: (a) There is a 4 (b) There is not a 4 (c) It is uncertain whether or not there is a 4 Participants also received abstract selection tasks using letters and numbers, the same connectives, and with negation of the antecedent and consequent systematically varied. As in Study 6, intelligence correlated with correctly resisting the affirmation of the consequent and denial of the antecedent fallacies. It also correlated with correct performance on inclusive and, to a lesser extent, exclusive disjunctives. The alternatives generation task was slightly less good a predictor in the present study, but nevertheless correlated significantly with performance on exclusive disjunctives and conditionals. However, neither task predicted performance on biconditionals, suggesting once again that performance on these is affected by more pragmatic factors. *** Study 8 The AH4, the Markovits possible antecedents task, uses of objects, unlikely scenarios and uses of objects were all used in this study. As in Study 7, the main experimental tasks involved conditional reasoning and the selection task. In the conditional reasoning task, half the 114 participants were presented with conditional statements in which there were many alternatives, for example If it is a trout then it has gills. Since there are many animals other than trout which have gills, it is easy to bring to mind instances in which the antecedent is false and the consequent true (i.e. non-trout which have gills). The other half of the participants received statements in which
there were few alternatives, for example If it is a fish then it has gills. They also received statements using abstract material. In the selection task, the participants were given problems which also used statements with either many or few alternatives. For example, in the many alternatives condition one of the selection tasks used the statement If a card has cat on one side then it has backbone on the other side. Abstract selection tasks were also used. *** References Griggs, R.A., & Cox, J.R. (1982). The elusive thematic materials effect in the Wason selection task. British Journal of Psychology, 72, 407-420. Heim, A.W. (1967). AH4 group test of intelligence. Manual. London: National Foundation for Educational Research. Heim, A.W. (1968). AH5 group test of intelligence. Manual. London: National Foundation for Educational Research. Johnson-Laird, P.N., & Byrne, R.M.J. (1991). Deduction. Hove, UK: Lawrence Erlbaum. Markovits, H. (1984). Awareness of the possible as a mediator of formal thinking in conditional reasoning problems. British Journal of Psychology, 75, 367-376. Newstead, S.E. (1992). A study of two quick and easy methods of assessing individual differences in student learning. British Journal of Educational Psychology, 62, 299-312. Newstead, S. E. (2000). Are there two different types of thinking? Behavioral and Brain Sciences, 23, 690-691. Newstead, S.E., Thompson, V.A. & Handley, S.J. (in press). Generating alternatives: A key component in human reasoning? Memory & Cognition. Pacini, R., & Epstein, S. (1999). The relation of rational and experiential processing styles to personality, basic beliefs, and the ratio-bias phenomenon. Journal of Personality and Social Psychology, 76, 972-987. Raven Stanovich, K.E., & West, R.F. (1998a). Cognitive ability and variation in selection task performance. Thinking and Reasoning, 4, 193-230.
Stanovich, K.E., & West, R.F. (1998b). Individual differences in rational thought. Journal of Experimental Psychology: General, 127, 161-188. Stanovich, K.E., & West, R.F. (2000) Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 22, Torrens, D., Thompson, V.A., & Cramer, K.M. (1999). Individual differences and the belief bias effect: Mental models, logical necessity, and abstract reasoning. Thinking and Reasoning, 5, 1-28. Valentine, E.R. (1975). Performance on two reasoning tasks in relation to intelligence, divergence and interference proneness. British Journal of Educational Psychology, 45, 198-205. Wason, P. C., & Brooks, P. G. (1979). THOG: The anatomy of a problem. Psychological Research, 41, 79-90. Activities Attendance by investigators at the following conferences: Experimental Psychology Society/Canadian Brain and Behavior Society conference, Cambridge, July 2000 (Newstead and Handley). Fourth International Conference on Thinking, Durham, August 2000 (Newstead and Handley). BPS Centenary Conference, Glasgow, March 2001 (Newstead). *** [others?] Outputs Newstead, S.E., Handley, S.J., Harley, C. & Wright, H. (2000). Individual differences in deductive reasoning. Paper presented to the Fourth International Conference on Thinking, Durham, August. Newstead, S.E. (2001). Individual differences in reasoning. Paper presented to the BPS Centenary Conference, Glasgow, March. Newstead, S.E. (2000) Are there two different types of thinking? Behavioral and Brain Sciences, 23, 690-691. Newstead, S.E., Handley, S.J., Harley, C. Wright, H. & Farrelly,D. (in submission). Individual differences in deductive reasoning.
Submitted to Quarterly Journal of Experimental Psychology. [Enclosed] Handley, S, Newstead, S.E. & Wright, H. (2000) Rational and experiential thinking: a study of the REI. International perspectives on individual differences, 1, 97-113. [Enclosed] Impacts The research has already attracted considerable interest both via conference presentations and published articles. Stanovich has reanalysed some of his own data and has confirmed our claim that the correlation between intelligence and performance on abstract selection tasks is moderated by the level of intelligence of the participants. This does not undermine Stanovich's theoretical approach but it does mean that certain caveats and restrictions are required. *** Future Research Priorities Our research has confirmed the usefulness of individual difference measures in the study of individual differences. However, thus far such research has only scratched the surface. Clearly the range of individual difference measures and reasoning tasks that could be used is large. At the same time our work also illustrates some of the problems with such research. Because the individual difference measures are not perfectly valid and reliable and because there are differences between populations, it is quite common for results not to replicate perfectly. In our research we found a plausible and theoretically interesting explanation for the discrepancy between our data and those of earlier studies, but this will not necessarily always be the case. Our results point in particular towards there being important differences between individuals of different levels of intelligence in the way in which reasoning tasks are carried out. They suggest that there is a progression from responding purely on the basis of contextual cues, to the use of pragmatic, conversationally-based cues, to the ability to decontextualise material and engage in high level abstract thought. Further study of this progression on other tasks is clearly indicated. More importantly, these progressions probably occur developmentally as well as in adults of differing levels of intelligence. There is existing evidence to support this claim but
further investigation of the development of the understanding of logical connectives and their use in logical reasoning tasks is needed. ***