Expert judgements in risk and reliability analysis Jan-Erik Holmberg M2-E2117 Risk analysis, April 17, 2016 Aalto university
Contents Definitions Applications Bayesian approach Examples 2
Background Objective, statistical data do not exist for many input data needed in risk analysis Examples Human reliability analysis Phenomenological uncertainties related to severe accidents Expert judgements are then needed and applied Plenty of literature exists 3
Some commonly appearing definitions in risk analysis Expert judgement Engineering judgement Screening values Generic data 4
Expert judgement expression of opinion, based on knowledge and experience, which experts make in responding to technical problems. Specifically, the judgment represents the expert s state of knowledge at the time of response to the technical question 5
Use of experts in qualitative risk analysis Qualitative Risk analysis methods like FMEA, HAZOP rely on expert judgements, especially when documentation is missing E.g. design phase analysis Experts are asked to identify failure modes, causes and impacts to give qualitative estimate of the likelihood and severity of failures 6
Expert judgements including a model of reasoning Not only a numerical, qualititative or verbal answer is asked but also a logic model representing the reasoning and assumed causal relationships Examples Bayesian belief network Safety case ROAAM (see later slides) 7
Use of experts in decision analysis Is very similar to risk analysis, except that in risk analysis preferences are seldom asked 8
Type of experts Domain expert Knows the subject (scientific/technical discipline) Provides judgements Method expert Risk analysis, probability calculus, expert judgement method expert Elicits judgements (facilitator) 9
Building blocks of elicitation 10
Elicitation situations 11
Qualification of experts In principle there should be some criteria how to qualify a person as an expert Years of experience Educational background In practice, best available experts must be used in a risk analysis project 12
Training of experts If possible, an expert should be trained before the elicitation process At minimum Explain the scope, purpose and context of the analysis Explain the method Train with some examples 13
Judgement biases Stakeholder bias Group effect Effect of the training examples Probability judgement biases Effect of the facilitator 14
Some expert judgement methods Delphi NUREG-1150 ROOAM Bayesian approach 15
Delphi Interactive forecasting method relying on an expert panel Experts answers in two or more rounds After each round the facilitator distributes anonymous summary of the responses to the panel Experts may revise their responses based on the replies of other members 16
NUREG-1150 Comprehensive risk analysis of five US nuclear power plants Published 1990 Still a kind of reference how to do level 2 and level 3 PRA Extensive use of expert judgements especially in level 2 PRA 17
NUREG-1150 method 18
Selection of issues If one or more of the following situation exists: No other means are available for quantifying an important issue The information available is characterized by high variability Some experts question the applicability of the available data The existing results from code calculations need to be supplemented, interpreted or extended due to recent experimental results or deficiencies in the codes Analysts need to determine the current state of knowledge 19
NUREG-1150 safety issues for expert opinion elicitation systems analysis (seven issues); in-vessel melt progression (six issues); containment structural response (five issues); molten core-concrete interactions (three issues); containment loads (three issues); source term analysis (eight issues). 20
NUREG-1150 example LOCA frequency LOCA = loss of coolant accident Caused by a check valve failure Five experts developed a model of the frequency of interfacing system LOCA to address the three failure scenarios and the common-cause failure of the two check valves 21
NUREG-1150 example Individual and aggregated estimates 22
ROOAM - Risk Oriented Accident Analysis Methodology Theofanous, T.G. On the proper formulation of safety goals and assessment of safety margins for rare and high-consequence hazards, Reliability Engineering and System Safety 54 (1996) 243-257. Used in some level 2 PRA:s (e.g. Loviisa PRA) Probability Description 1 Behavior is possible. It is conservatively assumed that the probability is 1 1/10 Behavior is within known trends but obtainable only at the edge-of-spectrum parameters 1/100 Behavior cannot be positively excluded, but it is outside the spectrum of reason 1/1000 Behavior is physically unreasonable and violates well-known reality. Its occurrence can be argued against positively 23
ROAAM philosophy Rather than the usual formal treatments on how to combine expert opinions that diverge widely, such uncertainty must be approached in each case as a research question encompasses frame of assessment, approach methodology, risk management, and safety goals the aim of obtaining resolution in a clear, consistent, and complete manner Resolution of major severe accident issues is relied heavily on developing an understanding of the underlying physics of the relevant phenomena a comprehensive review effort that involves all experts in the field, through an iterative and fully documented process towards resolution 24
ROAAM key elements Physically-based decomposition that allows transparency in separating out the essential portions of epistemic uncertainty Probabilistic framework made up of causal relations and intangible parameters Causal relations (key physics) represent well-posed problems, i.e., not subject to major discontinuities of a stochastic nature. Uncertainty can be reduced to the parameter level (no major modelling uncertainty) "Splinter" scenarios in combination with conservative estimates of epistemic uncertainty, so as to obtain convincingly conservative results 25
ROAAM example H2 combustion 26
Treatment of expert judgements Judgement are data as any observations or measurements In principle, anything can be elicited from experts but it may be better to ask concretely observable entities or indicators than directly probabilities Experts are equal Simple aggregation method Compute mean value of responses and standard deviation or some other metric representing the distribution of the responses 27
Bayesian approach Responses given by experts 1,.., n are interpreted as data from i.i.d random variables X 1,, X n, X i ~ F(x; θ) Data is used to estimate the model parameter(s) θ, which is used to derive the quantity needed in risk analysis, e.g., probability of an event Method produces an uncertainty distribution for the variable 28
Human reliability analysis example Enhanced Bayesian THERP Qualitative analysis Block diagram analysis of the event sequence Qualitative analysis of boundary conditions (specification, instructions, MMI, time, etc.) Quantitative analysis 29
Generic human failure event model for post-initiator actions Correct outcome Diagnosis and decisionmaking Postdiagnosis action Correct outcome Ok Failed outcome Recovery Correct outcome Failed outcome Recovery Correct outcome Failed outcome Failed outcome Failure 30
Quantitative analysis of the p t K 1,, K 5 diagnosis error 5 = min{1, p 0 (t) i=1 K i } p 0 (t)= the basic human error probability from Swain s handbook, t = available time identification and decision making K 1,, K 5 = the performance shaping factors = {1/5, 1/2, 1, 2 or 5} Procedures Training Man-machine interface Mental load Coordination and communication 31
Prior probability p 0 (t) 1E+0 1E-1 Probability of failure 1E-2 1E-3 1E-4 1E-5 1E-6 Swain 95% Swain Median Swain 5% Base probability Bayes THERP 1E-7 1 10 100 1000 10000 t, time for identification and decision making[min] 32
Performance shaping factors Quality and importance of procedures Quality and importance of training Performance shaping factors Feedback from process, Mental load quality of MMI Communication and coordination Factor K procedure K training K feedback K stress K coordination Questions Are there procedures? Are they needed? Do they give support? K i = 5 K i = 2 K i = 1 K i = 0,5 K i = 0,2 No instructions or misleading instructions, instructions would be needed Instructions are important but they are imperfect Instructions play no major role in the situation Good instructions, applicable for the situation and they support well the selection of correct actions Very good instructions, operators should not make any mistake Has the situation been trained? What kind of training? How often it has been trained? Is the action well known? No training or misleading training, training would be needed Some training has been given but it is not fully applicable for the situation Training plays no major role in the situation Situation has been trained (in a simulator), an important theme in the training Situation is often trained in a simulator, a very important theme Is critical information available for the operators? How easily, understandably, rapidly? Are there redundant information? Can there be misleading signals? No feedback from process or misleading information or too late feedback Feedback is obtained but there are defects in the presentation of the critical information Feedback plays no major role in the situation Symptoms can be easily observed and identified It is practically impossible to miss the symptoms, several redundant indications Is the situation or action unusual? Are there any special uncertainties in the situation? Mental load is so high that it nearly hinders to make a rational decision, situation is chaotic, an extreme decision needs to made Mental load is considerable, situation is serious, a serious decision needs to be made Mental load plays no major role in the situation NA NA Is it a scattered disturbance? Are actions needed inside/outside of control room (CR)? Does communication work? Is coordination of actions needed? Information must be obtained from inside and outside of CR, coordination of many activities, poor conditions for communication Information must be obtained from inside and outside of CR, coordination of many activities, good conditions for communication Communication and coordination play no major role in the situation Operator(s) can act directly based on available information without further communication NA 33
Assessment of performance shaping factors Given by experts and integrated together in a Bayesian manner Each gives judgement for each K i P(K i = q θ j, q) = θ j, q = 1/5, ½, 1, 2, 5 Multinomial, whose prior is Dirichlet D(α1, α1, α 1, α 2, α 5 ) 5 2 Posterior is also Dirichlet, D(α1 5 Y 1, α 2 + Y 2, α 5 + Y 5 ), where + Y1, α1 5 2 + Y1 2, α 1 + Y q = number of experts giving the response q Prior is chosen so that the expected value of the product term is 1 and the entropy of the distribution is maximal 34
HRA Example Recovery of loss of offsite power Initiating event: loss of offsite power Additional failure: common cause failure of emergency diesel generators => Station blackout Time windows 30 min from reactor scram to core damage Offsite power return within 20 min =>10 min time to recover power supply and core cooling 35
Fictive expert judgements Two experts: PSF Instructions Training Manmachine interface Stress Coordination, communication Expert 1 1 1 0,5 2 1 Expert 2 1 1 1 5 1 Base probability p(10 min) = 0,1 Posterior probability 0,31 36
Impact of several experts Case P Base probability (prior) 0,1 Posterior probability 0,31 expert 1 (1; 1; 0,5; 2; 1), expert 2 (1; 1; 1; 5; 1) Both give 5 for all PSF:s 0,88 Both give 0,2 for all PSF:s 0,002 6 experts: 0,34 1 expert (1; 1; 0,5; 2; 1) and 5 experts (1; 1; 1; 5; 1) 11 experts: 0,4 1 expert (1; 1; 0,5; 2; 1) and 10 experts (1; 1; 1; 5; 1) 37
Experience from use of expert judgements in HRA Good way to communicate with experts and can produce a lot valuable information Consistency checks useful Difficult to apply with living PRA => it can be problematic if all/same experts are not later available when the PRA is updated Bayesian approach requires that several experts give judgements (at least five?) 38
Conclusions Expert judgements are needed and commonly applied e.g. in qualitative risks analysis Also many probability assessments are based on expert judgements Traceability of judgements important Rather simple mathematical models and elicitation procedures preferable 39