Bargain Chinos: The influence of cognitive biases in assessing written work Daragh Behrman INTO London Tom Alder Kings College
Why is this focus important? Considerable body of work on cognitive bias Considerable body of work on assessment and rating Relatively little that combines these two themes within the field of EAP
Outline Today, our talk will focus on the following topics: Cognitive biases in general / the anchoring bias specifically Description of our research Findings / implications Conclusions and suggestions
written work What are cognitive biases and where to do they come from? Bounded rationality ( Herbert Simon 1950s): decision makers operate with 3 constraints: Limited access to information Inherent cognitive limitations of the human mind Time limit of decision making (Mission Command Centre of Excellence, U.S. Army 2015)
written work To simplify complex and uncertain situations (ibid) decision makers use mental shortcuts. Mental shortcuts (heuristics) biases errors in judgements Cognitive biases are therefore mental errors caused by our simplified information processing strategies (Heuer, Jr. 1999: 111).
Anchoring bias written work when we have a number FIXED in our minds and this affects subsequent numerical judgements e.g. buying & selling People make estimates by starting from an initial value that is adjusted to yield the final answer. Adjustments to the initial starting point are typically insufficient. (Tversky & Kahneman 1974: 1128)
written work Anchoring bias example 1 (Williams 2010. Cited in Mission Command Centre of Excellence, U.S. Army 2015) American troop levels in Iraq from 2003-2007 US Army - average of 138,000 troops deployed
written work Anchoring bias example 1 cont d. Increasing evidence to suggest that troop levels needed to be significantly strengthened in order to accomplish stated aims Williams suggests that decision makers refused to properly adjust figure because they were anchored to original estimates of what was necessary (Williams 2010: 48).
written work Anchoring bias example 2 Ariely s auction (2008) Subjects were given a list of six items and told a little about them Subjects were then asked to write the last two digits of their social security number at the top of their list and then next to each of the six items in dollars (23 = $23)
written work Anchoring bias example 2 cont d. Subjects then indicated if they would pay that amount for each of the six items YES / NO answer Anchor planted in 3 movements pushed towards internalisation Subjects then decided their maximum bid
written work Anchoring bias example 2 cont d. Results showed that the students with the highest-ending social security digits (from 80-99) bid highest, while those with the lowest-ending numbers bid lowest (Ariely 2008: 28).
written work Anchoring bias - Why focus on it? We felt this bias may be prevalent when giving numerical grades Co-operative nature of many staffrooms.
written work Anchoring bias - Why focus on it? Awareness doesn t help! Very hard to overcome these biases biases persist after test subjects are informed of them and instructed to try to avoid them or compensate for them (Alpert & Raiffa, 1968. Cited in Heuer Jr. 1999: 152). Ever-present anchors: pass marks, entrance requirements, the first essay graded on pile etc.
written work Research project: 1 sample student EAP essay Our anchors: 3 pairs of anchors : Strong anchor low / high grade Medium anchor low / high mean averages (for last 5 years) Weak anchor low / high required progression grade
Version 1: Strong anchor, low grade
Version 2: Strong anchor, high grade
Version 3: Medium anchor, low grades
Version 4: Medium anchor, high grades
Version 5: Weak anchor, low grade
Version 6: Weak anchor, high grade
RESPONDENTS The influence of cognitive biases in assessing written work research results Range of scores 50 40 30 20 10 0 40 45 50 55 60 65 70 75 80 FINAL MARK Finding 1: There was a 31 point difference between the highest and lowest scores given by respondents for the same piece of work.
written work research results Finding 2: When more subtle anchors were used, the average mark was significantly less impacted. When the subtle anchors were used the standard deviation was greater.
Finding 3: written work research results When explicit anchors were used, the average mark showed their impact. When the anchoring mark was 59, the mean average of the results was 60 When the anchoring mark was 70, the mean average of the results was 68
Summary of Results STRONG ANCHOR Grades MEDIUM ANCHOR Historic grades MILD ANCHOR Target grades Anchor 59 70 61 66 55 68 Mean mark 60 68 66 67 62 59 Stndrd dev. 3.5 5.1 4.9 1.7 8.4 8.8
MEAN GRADES 70 68 66 64 62 60 58 56 54 52 50 Anchors and Averages 50 55 60 65 70 75 ANCHOR
MEAN GRADES 70 Strong and Medium Anchor 69 68 67 66 65 64 63 62 61 60 59 58 60 62 64 66 68 70 72 ANCHOR
written work research results Strong and Medium anchors: correlation coefficient of 0.822 (but this will need to be tested further on a larger data set).
Discussion written work 1. The more explicit the anchor the closer it was to the final score care needs to be taken when discussing grades with colleagues. 2. Need to examine what is unsubtle about unsubtle anchors e.g. social proof? 3. Just because you resist our anchors does not mean that you are free! Auto-anchoring!
written work Grades cluster between 55 and 75 100% of what? 50% extra value!
Suggestion 1 written work Make sure that blind (double) marking takes place Not everywhere does this and from our research we can see that an anchoring mark can have a significant impact on the final mark. EAP context: making sure blind marking takes place and if it does not then raise the spectre of bias!
written work Suggestion 2 Marking work in advance of deadlines Making choices further in advance of their consequences less impulsive, suboptimal decisions (Milkman, Rogers & Bazerman 2008. Cited in Milkman, Chugh & Bazerman 2009: 382).
written work Suggestion 2 cont d. EAP context: making sure marking/blind double marking takes place with enough time before feedback needs to be given to the student. This is not always feasible / may overload already busy teachers.
Suggestion 3 Consider the opposite written work Encouraging others to consider the opposite when making decisions reduces errors in judgement (Milkman, Chugh & Bazerman 2009: 381). EAP context: asking teachers to consider a radically different grade before finalising the mark. Anchor on anchor!
Suggestion 4 Group marking written work Having groups instead of individuals make decisions has been found to partially debias errors in judgement typically classified as the result of biases and heuristics. (Tversky & Kahneman, 1974. Cited in Milkman, Chugh & Bazerman 2009: 381).
written work Suggestion 4 cont d. EAP context: groups of teachers mark a piece of work and come to a consensus Time consuming / individuals may be dominant or passive within the group.
Next time: A much larger data set Greater range of anchors Ariely-style multi-stage anchor implantation Marking in groups vs. marking individually How much impact a deadline may have on choice of final grade Impact of marking time (e.g. 5 mins vs. 15 mins)
Reference list Ariely, D. (2008) Predictably Irrational: The Hidden Forces That Shape Our Decisions: United States: Harper Collins Heuer Jr., R. J. (1999) Psychology of Intelligence Analysis Centre for the study of Intelligence: Central Intelligence Agency Milkman, K. L., Chugh, D. & Bazerman, M. (2009) How Can Decision Making Be Improved? Perspectives On Psychological Science. 4/4 379-383 Mission Command Centre of Excellence, (2015) Cognitive Biases and Decision Making: A Literature Review and Discussion of Implications for the US Army. [online] America, The United States Army. Available from http://usacac.army.mil/sites/default/files/publications/hdcdtf_whitepaper_cognitive%20biases%2 0and%20Decision%20Making_Final_2015_01_09_0.pdf [Accessed 21 March 2016] Tversky, A. & Kahneman, D. (1974) Judgement Under Uncertainty: Heuristics and Biases. Science 185/4157 1124-1131