The Moral Psychology Simplexity of Acceptable Risk in Safety Standards

Transcription

1 The Moral Psychology Simplexity of Acceptable Risk in Safety Standards by Andreas Johnsen April, 2016

2 Safety-critical systems

3 Certification through safety standards Safety of electrical and/or electronic systems (known as func7onal safety) is the absence of unreasonable risk due to hazards caused by malfunc3oning behaviour of the system unreasonable risk: risk judged to be unacceptable in a certain context according to valid societal moral concepts ALARP ( as low as reasonable prac7cal ) is the general criterion risk: combina7on of the probability of occurrence of harm and the severity of that harm hazard: poten7al source of harm malfunc3oning behaviour: failure or unintended behaviour of a system with respect to its design intent

4 Risk assessment and mitigation... Safety measures: fault prevention, fault removal, fault tolerance

5 Safety measures

6 Ethics in a nutshell TRUSTWORTHINESS CITIZENSHIP RESPECT ETHICS CARING RESPONSIBILITY FAIRNESS

7 Is the acceptable risk ethical?

8 What is acceptable for one may be unacceptable for another acceptable risk acceptable risk acceptable risk acceptable risk acceptable risk acceptable risk acceptable risk acceptable ris acceptable risk acceptable risk acceptable risk acceptable risk acceptable risk acceptable risk

9 The simplexity of ethically acceptable risk

10 Right or Wrong? It depends

11 Continuous awareness and adaptive regulation is the only practical solution

12 Our awareness is full of systematic errors

13 Deriving truth from both feelings and statistical data The emo3onal tail wags the ra3onal dog (decisions are based on emo7ons) Irra3onal feeling of risk: people become risk seeking when all their op7ons are bad and risk aversive when all op7ons are good Availability and risk: The feeling of risk is dependent on the 7me and frequency of accidents We have a hard 7me imagining something worse than experienced Highly shaped by media (extreme events are given far to much alen7on) Small risks: we either give them far too much weight or ignore them Above-average effect: 90% of drivers believe they are beler than average

14 We are bad at deriving truth from both feelings and statistics (data) Denominator neglect: Representa7on of numbers irra7onally affect our interpreta7on of them Most people feel a 8/100 chance is higher than a 1/10 chance Confirma3on bias: we tend to search for, interpret, favor, and recall informa7on that confirms ones believes Contrary to clinical science, where hypotheses are tested by trying to refute them, we tend to search for informa7on that support them Causes belief perseverance (inability to change beliefs, even when exposed to evidence) The halo effect: we tend to generalize few bits informa7on of something into believing they are true for all its proper7es Availability effect: we tend to assess the rela7ve importance of issues by the ease with which they are retrieved from memory (largely determined by the extent of coverage in the media)

15 We are bad at deriving truth from both feelings and statistics (data) Reliance on heuris3cs : we (including experts ) tend to make opinions based on memory availability, guesses, and feelings Cause biased opinions Cause truth illusions: familiarity à cogni3ve ease à perceived truthiness Ques3on subs3tu3on: when faced with a difficult ques7on, we tend to answer an easier ques7on instead, usually without no7cing the subs7tu7on Is it safe to drive to work? à Have I ever been in a car accident when driving to work? WYSIATI (What You See Is All There Is): we tend to treat problems in isola7on

16 We are bad at deriving truth from both feelings and statistics The affect heuris3c: we tend to let conclusions dominate over arguments jump into conclusions regardless of the quality and amount of informa7on Most o[en leads to false conclusions We tend to have intui3onal faith in conclusions, which strengthens believes in arguments (even unsound) that support them We are prone to think of causality rather than luck We tend to overes7mate causality and exper7se in environments with many uncertain7es (e.g. stockbrokers do not perform beler than random)

17 We are bad at deriving truth from both feelings and statistics The possibility effect: Highly unlikely events are given more weight than their probabili7es jus7fies. The certainty effect: Highly likely events are given less weight than their probabili7es jus7fies Priming effect: our thoughts are highly affected by unconsciously processed informa7on Our opinions are highly affected by the sequence with which pieces of informa7on are perceived, even if they are not correlated The first and last pieces have the highest effect

18 We are bad at deriving truth from both feelings and statistics We tend to have an excessive confidence in what we believe we know We tend to have an inability to acknowledge our ignorance and the uncertainty of the world we live in, even when evidence is presented Argument to modera3on (compromise fallacy): when faced with two opposite opinions, we tend to believe that the truth lies between them Focusing illusion: Nothing in life is as important as you think it is when you are thinking about it How much pleasure do you get from your car? Means How much pleasure do you get from your car when you think about it?

19 We are bad at deriving truth from both feelings and statistics Conjunc3on fallacy: hard to see difference between plausible/coherency vs. probability E.g. What is more likely: Linda is a bank teller OR Linda is a bank teller and is ac7ve in the feminist movement (up to 90% say the laler) Induc3on over Deduc3on fallacy: we are unwilling to deduce the par7cular from the general willing to infer the general from the par7cular Hindsight bias: the outcome of events will change memories of what we believed prior to the event in the direc7on of the outcome. Good decisions with unlucky outcomes will be unjus7fiably cri7cized, and bad decisions with lucky outcomes will be unjus7fiably graced

20 We are bad at deriving truth from both feelings and statistics Planning fallacy: we tend to plan unrealis7cally close to best-case scenarios Coherency-induced blindness: subjec7ve confidence is determined by the coherence of the story one has constructed, not by the quality and amount of informa7on that supports it Theory-induced blindness: A theory, when accepted, will be seen as truth to the point where its flaws will be invisible to its subjects Status quo bias: when a state is changed, we tend to weight losses more than its wins (wins must exceed over weighted losses to be accepted)

21 We are bad at deriving truth from both feelings and statistics Sunk-cost fallacy (pot commitment): keeps people for too long in poor jobs, unhappy marriages, and unpromising research projects To resist our illusions, there is only one thing we can do: we must learn to mistrust our impressions of the world. To implement that rule, we must be able to recognize the illusory palerns and recall what we know about it.

22 True morally acceptable risk!

23 Avoiding errors in regulation of risk levels Decorrelate errors (suppresses the halo effect): Make sources of informa7on independent from each other The law of small number: Extreme results are more likely to appear in small samples compared to larger (large samples are more precise than small) The world makes much less sense than we think: randomness is o[en perceived as regularit Suppress availability effect: People are less confident in a choice when they are asked to produce more arguments to support it

24 Avoiding errors in regulation of risk levels The public is beler at deciding good accidents and bad accidents compared to experts who only look at numbers rather than feelings Ra7onal or not, fear is painful and debilita7ng, and policy makers must endeavor to protect the public from fear, not only from real dangers Predic3on formulas that are simple and based on few variables with few weights per variable produce best results Cau7on Their ra7onal argument is compelling, but it runs against a stubborn psychological reality: for most people, the cause of a mistake malers. The story of a child dying because an algorithm made a mistake is more poignant than the story of the same tragedy occurring as a result of human error, and the difference in emo7onal intensity is readily translated into a moral preference. Predic7ons are best based on both sta7s7cs and intui7ve case-specific info. 1. Start with baseline (average) 2. Es7mate evidence by intui7on 3. Es7mate correla7on between evidence and the data set 4. Move the length of the correla7on towards to the intui7ve es7mate

25 Avoiding errors in regulation of risk levels The wisdom of Crowds: If individuals of a crowd are independent form each other, the average of their guess will be highly accurate (errors will cancel each other out), even though each individual is very poor in guessing. Lawyer cri3que methods: raise doubts about the strongest argument (logics and evidence) and focus on the weakest part of tes7monies (subjec7ve informa7on) Recall sta3s3cs when case-specific informa3on: sta7s7cal base rates are generally underweighted, and some7mes neglected altogether, when specific informa7on about the case at hand is available Take height for unknown unknowns

26 Avoiding errors in regulation of risk levels Stereotyping oren improves the accuracy of judgment Moral conflict: We consider it morally desirable for base rates to be treated as sta7s7cal facts about the group rather than as presump7ve facts about individuals Op7mism is suppressed by construc7ng counter-examples (worst case scenarios) Broad framing (decision making): It is beneficial to make decisions based on an analysis of all possible op7ons rather than making individual decisions on each op7on in isola7on. A culture that truly values ethical (and safe) behavior must be led by men and women commiled to principle for its own sake, not solely for the purpose of compliance. Compliance alone does not require a deeper understanding, and without a deeper understanding, the ability to make func7onal safety safer is reduced.

27 What do you think? Contact: