Bargain Chinos: The influence of cognitive biases in assessing written work

Transcription

1 Bargain Chinos: The influence of cognitive biases in assessing written work Daragh Behrman INTO London Tom Alder Kings College

2 Why is this focus important? Considerable body of work on cognitive bias Considerable body of work on assessment and rating Relatively little that combines these two themes within the field of EAP

3 Outline Today, our talk will focus on the following topics: Cognitive biases in general / the anchoring bias specifically Description of our research Findings / implications Conclusions and suggestions

4 written work What are cognitive biases and where to do they come from? Bounded rationality ( Herbert Simon 1950s): decision makers operate with 3 constraints: Limited access to information Inherent cognitive limitations of the human mind Time limit of decision making (Mission Command Centre of Excellence, U.S. Army 2015)

5 written work To simplify complex and uncertain situations (ibid) decision makers use mental shortcuts. Mental shortcuts (heuristics) biases errors in judgements Cognitive biases are therefore mental errors caused by our simplified information processing strategies (Heuer, Jr. 1999: 111).

6 Anchoring bias written work when we have a number FIXED in our minds and this affects subsequent numerical judgements e.g. buying & selling People make estimates by starting from an initial value that is adjusted to yield the final answer. Adjustments to the initial starting point are typically insufficient. (Tversky & Kahneman 1974: 1128)

7 written work Anchoring bias example 1 (Williams Cited in Mission Command Centre of Excellence, U.S. Army 2015) American troop levels in Iraq from US Army - average of 138,000 troops deployed

8 written work Anchoring bias example 1 cont d. Increasing evidence to suggest that troop levels needed to be significantly strengthened in order to accomplish stated aims Williams suggests that decision makers refused to properly adjust figure because they were anchored to original estimates of what was necessary (Williams 2010: 48).

9 written work Anchoring bias example 2 Ariely s auction (2008) Subjects were given a list of six items and told a little about them Subjects were then asked to write the last two digits of their social security number at the top of their list and then next to each of the six items in dollars (23 = $23)

10 written work Anchoring bias example 2 cont d. Subjects then indicated if they would pay that amount for each of the six items YES / NO answer Anchor planted in 3 movements pushed towards internalisation Subjects then decided their maximum bid

11 written work Anchoring bias example 2 cont d. Results showed that the students with the highest-ending social security digits (from 80-99) bid highest, while those with the lowest-ending numbers bid lowest (Ariely 2008: 28).

12 written work Anchoring bias - Why focus on it? We felt this bias may be prevalent when giving numerical grades Co-operative nature of many staffrooms.

13 written work Anchoring bias - Why focus on it? Awareness doesn t help! Very hard to overcome these biases biases persist after test subjects are informed of them and instructed to try to avoid them or compensate for them (Alpert & Raiffa, Cited in Heuer Jr. 1999: 152). Ever-present anchors: pass marks, entrance requirements, the first essay graded on pile etc.

14 written work Research project: 1 sample student EAP essay Our anchors: 3 pairs of anchors : Strong anchor low / high grade Medium anchor low / high mean averages (for last 5 years) Weak anchor low / high required progression grade

15 Version 1: Strong anchor, low grade

16 Version 2: Strong anchor, high grade

17 Version 3: Medium anchor, low grades

18 Version 4: Medium anchor, high grades

19 Version 5: Weak anchor, low grade

20 Version 6: Weak anchor, high grade

21 RESPONDENTS The influence of cognitive biases in assessing written work research results Range of scores FINAL MARK Finding 1: There was a 31 point difference between the highest and lowest scores given by respondents for the same piece of work.

22 written work research results Finding 2: When more subtle anchors were used, the average mark was significantly less impacted. When the subtle anchors were used the standard deviation was greater.

23 Finding 3: written work research results When explicit anchors were used, the average mark showed their impact. When the anchoring mark was 59, the mean average of the results was 60 When the anchoring mark was 70, the mean average of the results was 68

24 Summary of Results STRONG ANCHOR Grades MEDIUM ANCHOR Historic grades MILD ANCHOR Target grades Anchor Mean mark Stndrd dev

25 MEAN GRADES Anchors and Averages ANCHOR

26 MEAN GRADES 70 Strong and Medium Anchor ANCHOR

27 written work research results Strong and Medium anchors: correlation coefficient of (but this will need to be tested further on a larger data set).

28 Discussion written work 1. The more explicit the anchor the closer it was to the final score care needs to be taken when discussing grades with colleagues. 2. Need to examine what is unsubtle about unsubtle anchors e.g. social proof? 3. Just because you resist our anchors does not mean that you are free! Auto-anchoring!

29 written work Grades cluster between 55 and % of what? 50% extra value!

30 Suggestion 1 written work Make sure that blind (double) marking takes place Not everywhere does this and from our research we can see that an anchoring mark can have a significant impact on the final mark. EAP context: making sure blind marking takes place and if it does not then raise the spectre of bias!

31 written work Suggestion 2 Marking work in advance of deadlines Making choices further in advance of their consequences less impulsive, suboptimal decisions (Milkman, Rogers & Bazerman Cited in Milkman, Chugh & Bazerman 2009: 382).

32 written work Suggestion 2 cont d. EAP context: making sure marking/blind double marking takes place with enough time before feedback needs to be given to the student. This is not always feasible / may overload already busy teachers.

33 Suggestion 3 Consider the opposite written work Encouraging others to consider the opposite when making decisions reduces errors in judgement (Milkman, Chugh & Bazerman 2009: 381). EAP context: asking teachers to consider a radically different grade before finalising the mark. Anchor on anchor!

34 Suggestion 4 Group marking written work Having groups instead of individuals make decisions has been found to partially debias errors in judgement typically classified as the result of biases and heuristics. (Tversky & Kahneman, Cited in Milkman, Chugh & Bazerman 2009: 381).

35 written work Suggestion 4 cont d. EAP context: groups of teachers mark a piece of work and come to a consensus Time consuming / individuals may be dominant or passive within the group.

36 Next time: A much larger data set Greater range of anchors Ariely-style multi-stage anchor implantation Marking in groups vs. marking individually How much impact a deadline may have on choice of final grade Impact of marking time (e.g. 5 mins vs. 15 mins)

37

38 Reference list Ariely, D. (2008) Predictably Irrational: The Hidden Forces That Shape Our Decisions: United States: Harper Collins Heuer Jr., R. J. (1999) Psychology of Intelligence Analysis Centre for the study of Intelligence: Central Intelligence Agency Milkman, K. L., Chugh, D. & Bazerman, M. (2009) How Can Decision Making Be Improved? Perspectives On Psychological Science. 4/ Mission Command Centre of Excellence, (2015) Cognitive Biases and Decision Making: A Literature Review and Discussion of Implications for the US Army. [online] America, The United States Army. Available from 0and%20Decision%20Making_Final_2015_01_09_0.pdf [Accessed 21 March 2016] Tversky, A. & Kahneman, D. (1974) Judgement Under Uncertainty: Heuristics and Biases. Science 185/