Expert judgements in risk and reliability analysis

Size: px
Start display at page:

Download "Expert judgements in risk and reliability analysis"

Transcription

1 Expert judgements in risk and reliability analysis Jan-Erik Holmberg M2-E2117 Risk analysis, April 17, 2016 Aalto university

2 Contents Definitions Applications Bayesian approach Examples 2

3 Background Objective, statistical data do not exist for many input data needed in risk analysis Examples Human reliability analysis Phenomenological uncertainties related to severe accidents Expert judgements are then needed and applied Plenty of literature exists 3

4 Some commonly appearing definitions in risk analysis Expert judgement Engineering judgement Screening values Generic data 4

5 Expert judgement expression of opinion, based on knowledge and experience, which experts make in responding to technical problems. Specifically, the judgment represents the expert s state of knowledge at the time of response to the technical question 5

6 Use of experts in qualitative risk analysis Qualitative Risk analysis methods like FMEA, HAZOP rely on expert judgements, especially when documentation is missing E.g. design phase analysis Experts are asked to identify failure modes, causes and impacts to give qualitative estimate of the likelihood and severity of failures 6

7 Expert judgements including a model of reasoning Not only a numerical, qualititative or verbal answer is asked but also a logic model representing the reasoning and assumed causal relationships Examples Bayesian belief network Safety case ROAAM (see later slides) 7

8 Use of experts in decision analysis Is very similar to risk analysis, except that in risk analysis preferences are seldom asked 8

9 Type of experts Domain expert Knows the subject (scientific/technical discipline) Provides judgements Method expert Risk analysis, probability calculus, expert judgement method expert Elicits judgements (facilitator) 9

10 Building blocks of elicitation 10

11 Elicitation situations 11

12 Qualification of experts In principle there should be some criteria how to qualify a person as an expert Years of experience Educational background In practice, best available experts must be used in a risk analysis project 12

13 Training of experts If possible, an expert should be trained before the elicitation process At minimum Explain the scope, purpose and context of the analysis Explain the method Train with some examples 13

14 Judgement biases Stakeholder bias Group effect Effect of the training examples Probability judgement biases Effect of the facilitator 14

15 Some expert judgement methods Delphi NUREG-1150 ROOAM Bayesian approach 15

16 Delphi Interactive forecasting method relying on an expert panel Experts answers in two or more rounds After each round the facilitator distributes anonymous summary of the responses to the panel Experts may revise their responses based on the replies of other members 16

17 NUREG-1150 Comprehensive risk analysis of five US nuclear power plants Published 1990 Still a kind of reference how to do level 2 and level 3 PRA Extensive use of expert judgements especially in level 2 PRA 17

18 NUREG-1150 method 18

19 Selection of issues If one or more of the following situation exists: No other means are available for quantifying an important issue The information available is characterized by high variability Some experts question the applicability of the available data The existing results from code calculations need to be supplemented, interpreted or extended due to recent experimental results or deficiencies in the codes Analysts need to determine the current state of knowledge 19

20 NUREG-1150 safety issues for expert opinion elicitation systems analysis (seven issues); in-vessel melt progression (six issues); containment structural response (five issues); molten core-concrete interactions (three issues); containment loads (three issues); source term analysis (eight issues). 20

21 NUREG-1150 example LOCA frequency LOCA = loss of coolant accident Caused by a check valve failure Five experts developed a model of the frequency of interfacing system LOCA to address the three failure scenarios and the common-cause failure of the two check valves 21

22 NUREG-1150 example Individual and aggregated estimates 22

23 ROOAM - Risk Oriented Accident Analysis Methodology Theofanous, T.G. On the proper formulation of safety goals and assessment of safety margins for rare and high-consequence hazards, Reliability Engineering and System Safety 54 (1996) Used in some level 2 PRA:s (e.g. Loviisa PRA) Probability Description 1 Behavior is possible. It is conservatively assumed that the probability is 1 1/10 Behavior is within known trends but obtainable only at the edge-of-spectrum parameters 1/100 Behavior cannot be positively excluded, but it is outside the spectrum of reason 1/1000 Behavior is physically unreasonable and violates well-known reality. Its occurrence can be argued against positively 23

24 ROAAM philosophy Rather than the usual formal treatments on how to combine expert opinions that diverge widely, such uncertainty must be approached in each case as a research question encompasses frame of assessment, approach methodology, risk management, and safety goals the aim of obtaining resolution in a clear, consistent, and complete manner Resolution of major severe accident issues is relied heavily on developing an understanding of the underlying physics of the relevant phenomena a comprehensive review effort that involves all experts in the field, through an iterative and fully documented process towards resolution 24

25 ROAAM key elements Physically-based decomposition that allows transparency in separating out the essential portions of epistemic uncertainty Probabilistic framework made up of causal relations and intangible parameters Causal relations (key physics) represent well-posed problems, i.e., not subject to major discontinuities of a stochastic nature. Uncertainty can be reduced to the parameter level (no major modelling uncertainty) "Splinter" scenarios in combination with conservative estimates of epistemic uncertainty, so as to obtain convincingly conservative results 25

26 ROAAM example H2 combustion 26

27 Treatment of expert judgements Judgement are data as any observations or measurements In principle, anything can be elicited from experts but it may be better to ask concretely observable entities or indicators than directly probabilities Experts are equal Simple aggregation method Compute mean value of responses and standard deviation or some other metric representing the distribution of the responses 27

28 Bayesian approach Responses given by experts 1,.., n are interpreted as data from i.i.d random variables X 1,, X n, X i ~ F(x; θ) Data is used to estimate the model parameter(s) θ, which is used to derive the quantity needed in risk analysis, e.g., probability of an event Method produces an uncertainty distribution for the variable 28

29 Human reliability analysis example Enhanced Bayesian THERP Qualitative analysis Block diagram analysis of the event sequence Qualitative analysis of boundary conditions (specification, instructions, MMI, time, etc.) Quantitative analysis 29

30 Generic human failure event model for post-initiator actions Correct outcome Diagnosis and decisionmaking Postdiagnosis action Correct outcome Ok Failed outcome Recovery Correct outcome Failed outcome Recovery Correct outcome Failed outcome Failed outcome Failure 30

31 Quantitative analysis of the p t K 1,, K 5 diagnosis error 5 = min{1, p 0 (t) i=1 K i } p 0 (t)= the basic human error probability from Swain s handbook, t = available time identification and decision making K 1,, K 5 = the performance shaping factors = {1/5, 1/2, 1, 2 or 5} Procedures Training Man-machine interface Mental load Coordination and communication 31

32 Prior probability p 0 (t) 1E+0 1E-1 Probability of failure 1E-2 1E-3 1E-4 1E-5 1E-6 Swain 95% Swain Median Swain 5% Base probability Bayes THERP 1E t, time for identification and decision making[min] 32

33 Performance shaping factors Quality and importance of procedures Quality and importance of training Performance shaping factors Feedback from process, Mental load quality of MMI Communication and coordination Factor K procedure K training K feedback K stress K coordination Questions Are there procedures? Are they needed? Do they give support? K i = 5 K i = 2 K i = 1 K i = 0,5 K i = 0,2 No instructions or misleading instructions, instructions would be needed Instructions are important but they are imperfect Instructions play no major role in the situation Good instructions, applicable for the situation and they support well the selection of correct actions Very good instructions, operators should not make any mistake Has the situation been trained? What kind of training? How often it has been trained? Is the action well known? No training or misleading training, training would be needed Some training has been given but it is not fully applicable for the situation Training plays no major role in the situation Situation has been trained (in a simulator), an important theme in the training Situation is often trained in a simulator, a very important theme Is critical information available for the operators? How easily, understandably, rapidly? Are there redundant information? Can there be misleading signals? No feedback from process or misleading information or too late feedback Feedback is obtained but there are defects in the presentation of the critical information Feedback plays no major role in the situation Symptoms can be easily observed and identified It is practically impossible to miss the symptoms, several redundant indications Is the situation or action unusual? Are there any special uncertainties in the situation? Mental load is so high that it nearly hinders to make a rational decision, situation is chaotic, an extreme decision needs to made Mental load is considerable, situation is serious, a serious decision needs to be made Mental load plays no major role in the situation NA NA Is it a scattered disturbance? Are actions needed inside/outside of control room (CR)? Does communication work? Is coordination of actions needed? Information must be obtained from inside and outside of CR, coordination of many activities, poor conditions for communication Information must be obtained from inside and outside of CR, coordination of many activities, good conditions for communication Communication and coordination play no major role in the situation Operator(s) can act directly based on available information without further communication NA 33

34 Assessment of performance shaping factors Given by experts and integrated together in a Bayesian manner Each gives judgement for each K i P(K i = q θ j, q) = θ j, q = 1/5, ½, 1, 2, 5 Multinomial, whose prior is Dirichlet D(α1, α1, α 1, α 2, α 5 ) 5 2 Posterior is also Dirichlet, D(α1 5 Y 1, α 2 + Y 2, α 5 + Y 5 ), where + Y1, α Y1 2, α 1 + Y q = number of experts giving the response q Prior is chosen so that the expected value of the product term is 1 and the entropy of the distribution is maximal 34

35 HRA Example Recovery of loss of offsite power Initiating event: loss of offsite power Additional failure: common cause failure of emergency diesel generators => Station blackout Time windows 30 min from reactor scram to core damage Offsite power return within 20 min =>10 min time to recover power supply and core cooling 35

36 Fictive expert judgements Two experts: PSF Instructions Training Manmachine interface Stress Coordination, communication Expert ,5 2 1 Expert Base probability p(10 min) = 0,1 Posterior probability 0,31 36

37 Impact of several experts Case P Base probability (prior) 0,1 Posterior probability 0,31 expert 1 (1; 1; 0,5; 2; 1), expert 2 (1; 1; 1; 5; 1) Both give 5 for all PSF:s 0,88 Both give 0,2 for all PSF:s 0,002 6 experts: 0,34 1 expert (1; 1; 0,5; 2; 1) and 5 experts (1; 1; 1; 5; 1) 11 experts: 0,4 1 expert (1; 1; 0,5; 2; 1) and 10 experts (1; 1; 1; 5; 1) 37

38 Experience from use of expert judgements in HRA Good way to communicate with experts and can produce a lot valuable information Consistency checks useful Difficult to apply with living PRA => it can be problematic if all/same experts are not later available when the PRA is updated Bayesian approach requires that several experts give judgements (at least five?) 38

39 Conclusions Expert judgements are needed and commonly applied e.g. in qualitative risks analysis Also many probability assessments are based on expert judgements Traceability of judgements important Rather simple mathematical models and elicitation procedures preferable 39

Use of the deterministic and probabilistic approach in the Belgian regulatory context

Use of the deterministic and probabilistic approach in the Belgian regulatory context Use of the deterministic and probabilistic approach in the Belgian regulatory context Pieter De Gelder, Mark Hulsmans, Dries Gryffroy and Benoît De Boeck AVN, Brussels, Belgium Abstract The Belgian nuclear

More information

PANDAROSA. Probabilistic ANd Deterministic Analysis for Risk-Oriented Severe Accidents Vapor Explosion Risk in Nordic LWRs.

PANDAROSA. Probabilistic ANd Deterministic Analysis for Risk-Oriented Severe Accidents Vapor Explosion Risk in Nordic LWRs. PANDAROSA Probabilistic ANd Deterministic Analysis for Risk-Oriented Severe Accidents Vapor Explosion Risk in Nordic LWRs Introduction Proposed by Park, Hyun Sun, Ph.D. SINEWSTEK Gustav III:s Boulevard

More information

John Quigley, Tim Bedford, Lesley Walls Department of Management Science, University of Strathclyde, Glasgow

John Quigley, Tim Bedford, Lesley Walls Department of Management Science, University of Strathclyde, Glasgow Empirical Bayes Estimates of Development Reliability for One Shot Devices John Quigley, Tim Bedford, Lesley Walls Department of Management Science, University of Strathclyde, Glasgow This article describes

More information

EXPLORING SUBJECTIVITY IN HAZARD ANALYSIS 1

EXPLORING SUBJECTIVITY IN HAZARD ANALYSIS 1 EXPLORING SUBJECTIVITY IN HAZARD ANALYSIS 1 Felix Redmill Redmill Consultancy Email: Felix.Redmill@ncl.ac.uk INTRODUCTION In an earlier paper (Redmill 2002) risk analysis was defined as comprising four

More information

Workshop Information IAEA Workshop

Workshop Information IAEA Workshop IAEA Training Course on Safety Assessment of NPPs to Assist Decision Making Diablo Canyon NPP Risk-Informed In-service Inspection Lecturer Lesson Lesson IV IV 3_11.3 3_11.3 Workshop Information IAEA Workshop

More information

Outline. What s inside this paper? My expectation. Software Defect Prediction. Traditional Method. What s inside this paper?

Outline. What s inside this paper? My expectation. Software Defect Prediction. Traditional Method. What s inside this paper? Outline A Critique of Software Defect Prediction Models Norman E. Fenton Dongfeng Zhu What s inside this paper? What kind of new technique was developed in this paper? Research area of this technique?

More information

Task Group on the CSNI Safety Margins Action Plan (SMAP)

Task Group on the CSNI Safety Margins Action Plan (SMAP) Unclassified Unclassified Organisation de Coopération et de Développement Economiques Organisation for Economic Co-operation and Development 30-Aug-2006 English - Or. English NUCLEAR ENERGY AGENCY COMMITTEE

More information

Model calibration and Bayesian methods for probabilistic projections

Model calibration and Bayesian methods for probabilistic projections ETH Zurich Reto Knutti Model calibration and Bayesian methods for probabilistic projections Reto Knutti, IAC ETH Toy model Model: obs = linear trend + noise(variance, spectrum) 1) Short term predictability,

More information

C/S/E/L :2008. On Analytical Rigor A Study of How Professional Intelligence Analysts Assess Rigor. innovations at the intersection of people,

C/S/E/L :2008. On Analytical Rigor A Study of How Professional Intelligence Analysts Assess Rigor. innovations at the intersection of people, C/S/E/L :2008 innovations at the intersection of people, technology, and work. On Analytical Rigor A Study of How Professional Intelligence Analysts Assess Rigor Daniel J. Zelik The Ohio State University

More information

Generalization and Theory-Building in Software Engineering Research

Generalization and Theory-Building in Software Engineering Research Generalization and Theory-Building in Software Engineering Research Magne Jørgensen, Dag Sjøberg Simula Research Laboratory {magne.jorgensen, dagsj}@simula.no Abstract The main purpose of this paper is

More information

Modeling Terrorist Beliefs and Motivations

Modeling Terrorist Beliefs and Motivations CREATE Research Archive Research Project Summaries 2010 Modeling Terrorist Beliefs and Motivations Richard John University of Southern California, richardj@usc.edu Heather Rosoff University of Southern

More information

On the Use of a Bayesian Updating Technique to Get Probabilistic Seismic Hazard Assessment More Rugged

On the Use of a Bayesian Updating Technique to Get Probabilistic Seismic Hazard Assessment More Rugged On the Use of a Bayesian Updating Technique to Get Probabilistic Seismic Hazard Assessment More Rugged ABSTRACT : E. Viallet 1, N. Humbert 1, C. Martin 2, R. Secanell 2 1 EDF SEPTEN, Dept. of Dynamics

More information

Educational Research

Educational Research Educational Research What is experimental research in education Started in educational psychology Issues of Transfer Thorndike Woodworth Used in each of the educational paradigms behaviorism cognitivism

More information

Fundamental Concepts for Using Diagnostic Classification Models. Section #2 NCME 2016 Training Session. NCME 2016 Training Session: Section 2

Fundamental Concepts for Using Diagnostic Classification Models. Section #2 NCME 2016 Training Session. NCME 2016 Training Session: Section 2 Fundamental Concepts for Using Diagnostic Classification Models Section #2 NCME 2016 Training Session NCME 2016 Training Session: Section 2 Lecture Overview Nature of attributes What s in a name? Grain

More information

GOOD SCIENCE Vs UNCERTAIN REGULATORY GUIDANCE or HOW CHI/Q AFFECTS NUCLEAR SAFETY

GOOD SCIENCE Vs UNCERTAIN REGULATORY GUIDANCE or HOW CHI/Q AFFECTS NUCLEAR SAFETY GOOD SCIENCE Vs UNCERTAIN REGULATORY GUIDANCE or HOW CHI/Q AFFECTS NUCLEAR SAFETY ABSTRACT For many years in the nuclear industry the calculation of Chi/Q was a dry subject that was limited to meteorologists

More information

Research and science: Qualitative methods

Research and science: Qualitative methods Research and science: Qualitative methods Urban Bilstrup (E327) Urban.Bilstrup@hh.se 140922 2 INTRODUCTION TO RESEARCH AND SCIENTIFIC METHODS Outline Definitions Problem formulation? Aim and goal operational

More information

Quantifying uncertainty under a predictive, epistemic approach to risk analysis

Quantifying uncertainty under a predictive, epistemic approach to risk analysis Quantifying uncertainty under a predictive, epistemic approach to risk analysis S.Apeland Stavanger University College, Norway T.Aven Stavanger University College, Norway T.Nilsen RF Rogaland Research,

More information

Bayes Theorem Application: Estimating Outcomes in Terms of Probability

Bayes Theorem Application: Estimating Outcomes in Terms of Probability Bayes Theorem Application: Estimating Outcomes in Terms of Probability The better the estimates, the better the outcomes. It s true in engineering and in just about everything else. Decisions and judgments

More information

Supplementary notes for lecture 8: Computational modeling of cognitive development

Supplementary notes for lecture 8: Computational modeling of cognitive development Supplementary notes for lecture 8: Computational modeling of cognitive development Slide 1 Why computational modeling is important for studying cognitive development. Let s think about how to study the

More information

Analysis of Competing Hypotheses using Subjective Logic (ACH-SL)

Analysis of Competing Hypotheses using Subjective Logic (ACH-SL) Analysis of Competing Hypotheses using Subjective Logic (ACH-SL) Simon Pope and Audun Jøsang, DSTC, CRC for Enterprise Distributed Systems Technology ICCRTS, 13-16 June 2005 June 22, 2005 DSTC 2005. All

More information

Improving Construction of Conditional Probability Tables for Ranked Nodes in Bayesian Networks Pekka Laitila, Kai Virtanen

Improving Construction of Conditional Probability Tables for Ranked Nodes in Bayesian Networks Pekka Laitila, Kai Virtanen Improving Construction of Conditional Probability Tables for Ranked Nodes in Bayesian Networks Pekka Laitila, Kai Virtanen pekka.laitila@aalto.fi, kai.virtanen@aalto.fi Systems Analysis Laboratory, Department

More information

Risk Assessment Terminology, Expression of Nature and level of Risk and Uncertainties

Risk Assessment Terminology, Expression of Nature and level of Risk and Uncertainties Risk Assessment Terminology, Expression of Nature and level of Risk and Uncertainties Professor Tony Hardy, Chair of EFSA PPR Panel Brussels 13 th November 2008 Comparative Review of Risk Terminology of

More information

Statistical Issues in Road Safety Part I: Uncertainty, Variability, Sampling

Statistical Issues in Road Safety Part I: Uncertainty, Variability, Sampling Statistical Issues in Road Safety Part I: Uncertainty, Variability, Sampling Shrikant I. Bangdiwala, PhD McMaster University Hamilton, Canada 1 Why statistics in road safety research? Our questions are

More information

FMEA AND RPN NUMBERS. Failure Mode Severity Occurrence Detection RPN A B

FMEA AND RPN NUMBERS. Failure Mode Severity Occurrence Detection RPN A B FMEA AND RPN NUMBERS An important part of risk is to remember that risk is a vector: one aspect of risk is the severity of the effect of the event and the other aspect is the probability or frequency of

More information

School of Nursing, University of British Columbia Vancouver, British Columbia, Canada

School of Nursing, University of British Columbia Vancouver, British Columbia, Canada Data analysis in qualitative research School of Nursing, University of British Columbia Vancouver, British Columbia, Canada Unquestionably, data analysis is the most complex and mysterious of all of the

More information

Funnelling Used to describe a process of narrowing down of focus within a literature review. So, the writer begins with a broad discussion providing b

Funnelling Used to describe a process of narrowing down of focus within a literature review. So, the writer begins with a broad discussion providing b Accidental sampling A lesser-used term for convenience sampling. Action research An approach that challenges the traditional conception of the researcher as separate from the real world. It is associated

More information

Correlation Neglect in Belief Formation

Correlation Neglect in Belief Formation Correlation Neglect in Belief Formation Benjamin Enke Florian Zimmermann Bonn Graduate School of Economics University of Zurich NYU Bounded Rationality in Choice Conference May 31, 2015 Benjamin Enke (Bonn)

More information

Bayes, Data and NUREG/CR-6928 Caveats Nathan Larson, Carroll Trull September 2017

Bayes, Data and NUREG/CR-6928 Caveats Nathan Larson, Carroll Trull September 2017 Bayes, Data and NUREG/CR-6928 Caveats Nathan Larson, Carroll Trull September 2017 1 Overview State-of-Knowledge Correlation Change-of-State Failure Mode LOCA Treatment Plant Availability Factor Zero Event

More information

THE STAMP PYRAMID: Vertical Thinking for Leverage in a Horizontally Focused World. April 19, 2012 STAMP/STPA Workshop

THE STAMP PYRAMID: Vertical Thinking for Leverage in a Horizontally Focused World. April 19, 2012 STAMP/STPA Workshop THE STAMP PYRAMID: Vertical Thinking for Leverage in a Horizontally Focused World April 19, 2012 STAMP/STPA Workshop William E. Young, Jr. Colonel, USAF PhD Student, Engineering Systems Division Massachusetts

More information

A Cue Imputation Bayesian Model of Information Aggregation

A Cue Imputation Bayesian Model of Information Aggregation A Cue Imputation Bayesian Model of Information Aggregation Jennifer S. Trueblood, George Kachergis, and John K. Kruschke {jstruebl, gkacherg, kruschke}@indiana.edu Cognitive Science Program, 819 Eigenmann,

More information

An Emerging New Risk Analysis Science: Foundations and Implications

An Emerging New Risk Analysis Science: Foundations and Implications Risk Analysis, Vol. 38, No. 5, 2018 DOI: 10.1111/risa.12899 An Emerging New Risk Analysis Science: Foundations and Implications Terje Aven To solve real-life problems such as those related to technology,

More information

Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research

Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research Other Methodology Articles Bayesian and Classical Hypothesis Testing: Practical Differences for a Controversial Area of Research J. E. Kennedy Version of 10/12/2014 (in press) Abstract: The use of Bayesian

More information

Discussion Meeting for MCP-Mod Qualification Opinion Request. Novartis 10 July 2013 EMA, London, UK

Discussion Meeting for MCP-Mod Qualification Opinion Request. Novartis 10 July 2013 EMA, London, UK Discussion Meeting for MCP-Mod Qualification Opinion Request Novartis 10 July 2013 EMA, London, UK Attendees Face to face: Dr. Frank Bretz Global Statistical Methodology Head, Novartis Dr. Björn Bornkamp

More information

What Case Study means? Case Studies. Case Study in SE. Key Characteristics. Flexibility of Case Studies. Sources of Evidence

What Case Study means? Case Studies. Case Study in SE. Key Characteristics. Flexibility of Case Studies. Sources of Evidence DCC / ICEx / UFMG What Case Study means? Case Studies Eduardo Figueiredo http://www.dcc.ufmg.br/~figueiredo The term case study frequently appears in title and abstracts of papers Its meaning varies a

More information

382C Empirical Studies in Software Engineering. Lecture 1. Introduction. Dewayne E Perry ENS present, Dewayne E Perry

382C Empirical Studies in Software Engineering. Lecture 1. Introduction. Dewayne E Perry ENS present, Dewayne E Perry Introduction Dewayne E Perry ENS 623 perry@mail.utexas.edu 1 Course Goals Create/evaluate empirical studies/data Understand underlying scientific and theoretical bases Understand fundamentals of experimental

More information

References. Christos A. Ioannou 2/37

References. Christos A. Ioannou 2/37 Prospect Theory References Tversky, A., and D. Kahneman: Judgement under Uncertainty: Heuristics and Biases, Science, 185 (1974), 1124-1131. Tversky, A., and D. Kahneman: Prospect Theory: An Analysis of

More information

High-level Vision. Bernd Neumann Slides for the course in WS 2004/05. Faculty of Informatics Hamburg University Germany

High-level Vision. Bernd Neumann Slides for the course in WS 2004/05. Faculty of Informatics Hamburg University Germany High-level Vision Bernd Neumann Slides for the course in WS 2004/05 Faculty of Informatics Hamburg University Germany neumann@informatik.uni-hamburg.de http://kogs-www.informatik.uni-hamburg.de 1 Contents

More information

Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Ryan Adams, Hugo LaRochelle NIPS 2012

Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Ryan Adams, Hugo LaRochelle NIPS 2012 Practical Bayesian Optimization of Machine Learning Algorithms Jasper Snoek, Ryan Adams, Hugo LaRochelle NIPS 2012 ... (Gaussian Processes) are inadequate for doing speech and vision. I still think they're

More information

The objectives behind the preparation of this document, including the list of core subjects of risk analysis, are:

The objectives behind the preparation of this document, including the list of core subjects of risk analysis, are: May 15, 2017 Core subjects of Risk Analysis The Society for Risk Analysis (SRA) Specialty Group on foundational issues in risk analysis and the SRA Committee for Specialty Groups have established a group

More information

Confirmation Bias. this entry appeared in pp of in M. Kattan (Ed.), The Encyclopedia of Medical Decision Making.

Confirmation Bias. this entry appeared in pp of in M. Kattan (Ed.), The Encyclopedia of Medical Decision Making. Confirmation Bias Jonathan D Nelson^ and Craig R M McKenzie + this entry appeared in pp. 167-171 of in M. Kattan (Ed.), The Encyclopedia of Medical Decision Making. London, UK: Sage the full Encyclopedia

More information

We Don t Measure Moisture... We Calculate Risk

We Don t Measure Moisture... We Calculate Risk We Don t Measure Moisture... We Calculate Risk Qualitative versus Quantitative Inquiry Why point testing for insitu %RH as the sole criterion for assessing risk of flooring failure is fraught with danger.

More information

Statistical Decision Theory and Bayesian Analysis

Statistical Decision Theory and Bayesian Analysis Statistical Decision Theory and Bayesian Analysis Chapter 3: Prior Information and Subjective Probability Lili MOU moull12@sei.pku.edu.cn http://sei.pku.edu.cn/ moull12 11 MayApril 2015 Reference 3, James

More information

Structural Modelling of Operational Risk in Financial Institutions:

Structural Modelling of Operational Risk in Financial Institutions: Structural Modelling of Operational Risk in Financial Institutions: Application of Bayesian Networks and Balanced Scorecards to IT Infrastructure Risk Modelling Inaugural-Dissertation zur Erlangung des

More information

Exploring the Influence of Particle Filter Parameters on Order Effects in Causal Learning

Exploring the Influence of Particle Filter Parameters on Order Effects in Causal Learning Exploring the Influence of Particle Filter Parameters on Order Effects in Causal Learning Joshua T. Abbott (joshua.abbott@berkeley.edu) Thomas L. Griffiths (tom griffiths@berkeley.edu) Department of Psychology,

More information

CS343: Artificial Intelligence

CS343: Artificial Intelligence CS343: Artificial Intelligence Introduction: Part 2 Prof. Scott Niekum University of Texas at Austin [Based on slides created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All materials

More information

A Brief Introduction to Bayesian Statistics

A Brief Introduction to Bayesian Statistics A Brief Introduction to Statistics David Kaplan Department of Educational Psychology Methods for Social Policy Research and, Washington, DC 2017 1 / 37 The Reverend Thomas Bayes, 1701 1761 2 / 37 Pierre-Simon

More information

MTAT Bayesian Networks. Introductory Lecture. Sven Laur University of Tartu

MTAT Bayesian Networks. Introductory Lecture. Sven Laur University of Tartu MTAT.05.113 Bayesian Networks Introductory Lecture Sven Laur University of Tartu Motivation Probability calculus can be viewed as an extension of classical logic. We use many imprecise and heuristic rules

More information

Probabilistic Information Processing Systems: Evaluation with Conditionally Dependent Data 1

Probabilistic Information Processing Systems: Evaluation with Conditionally Dependent Data 1 ORGANIZATIONAL BEHAVIOR AND I-III~ IAN PERFORNiANCE 7, 77-85 (1972) Probabilistic Information Processing Systems: Evaluation with Conditionally Dependent Data 1 PATRICIA ANN ])OMAS AND CAMERON ~. PETERSON

More information

School of Population and Public Health SPPH 503 Epidemiologic methods II January to April 2019

School of Population and Public Health SPPH 503 Epidemiologic methods II January to April 2019 School of Population and Public Health SPPH 503 Epidemiologic methods II January to April 2019 Time: Tuesday, 1330 1630 Location: School of Population and Public Health, UBC Course description Students

More information

Bayesian Belief Network Based Fault Diagnosis in Automotive Electronic Systems

Bayesian Belief Network Based Fault Diagnosis in Automotive Electronic Systems Bayesian Belief Network Based Fault Diagnosis in Automotive Electronic Systems Yingping Huang *, David Antory, R. Peter Jones, Craig Groom, Ross McMurran, Peter Earp and Francis Mckinney International

More information

Responsiveness to feedback as a personal trait

Responsiveness to feedback as a personal trait Responsiveness to feedback as a personal trait Thomas Buser University of Amsterdam Leonie Gerhards University of Hamburg Joël van der Weele University of Amsterdam Pittsburgh, June 12, 2017 1 / 30 Feedback

More information

Using historical data for Bayesian sample size determination

Using historical data for Bayesian sample size determination Using historical data for Bayesian sample size determination Author: Fulvio De Santis, J. R. Statist. Soc. A (2007) 170, Part 1, pp. 95 113 Harvard Catalyst Journal Club: December 7 th 2016 Kush Kapur,

More information

Behavioural Economics University of Oxford Vincent P. Crawford Michaelmas Term 2012

Behavioural Economics University of Oxford Vincent P. Crawford Michaelmas Term 2012 Behavioural Economics University of Oxford Vincent P. Crawford Michaelmas Term 2012 Introduction to Behavioral Economics and Decision Theory (with very large debts to David Laibson and Matthew Rabin) Revised

More information

A Conceptualistic Pragmatism in a Risk Assessment Context

A Conceptualistic Pragmatism in a Risk Assessment Context International Journal of Performability Engineering, Vol. 8, No. 3, May 2012, pp. 223-232. RAMS Consultants Printed in India A Conceptualistic Pragmatism in a Risk Assessment Context TERJE AVEN 1* and

More information

Artificial Intelligence Programming Probability

Artificial Intelligence Programming Probability Artificial Intelligence Programming Probability Chris Brooks Department of Computer Science University of San Francisco Department of Computer Science University of San Francisco p.1/25 17-0: Uncertainty

More information

Heuristics & Biases:

Heuristics & Biases: Heuristics & Biases: The Availability Heuristic and The Representativeness Heuristic Psychology 355: Cognitive Psychology Instructor: John Miyamoto 05/29/2018: Lecture 10-2 Note: This Powerpoint presentation

More information

Bayesian Joint Modelling of Benefit and Risk in Drug Development

Bayesian Joint Modelling of Benefit and Risk in Drug Development Bayesian Joint Modelling of Benefit and Risk in Drug Development EFSPI/PSDM Safety Statistics Meeting Leiden 2017 Disclosure is an employee and shareholder of GSK Data presented is based on human research

More information

Ambiguous Data Result in Ambiguous Conclusions: A Reply to Charles T. Tart

Ambiguous Data Result in Ambiguous Conclusions: A Reply to Charles T. Tart Other Methodology Articles Ambiguous Data Result in Ambiguous Conclusions: A Reply to Charles T. Tart J. E. KENNEDY 1 (Original publication and copyright: Journal of the American Society for Psychical

More information

IDENTIFYING AND CORRECTING COGNITIVE BIASES IN SUBJECTIVE PROBABILITY ELICITATION SURVEYS: MODEL BASED APPROACHES

IDENTIFYING AND CORRECTING COGNITIVE BIASES IN SUBJECTIVE PROBABILITY ELICITATION SURVEYS: MODEL BASED APPROACHES IDENTIFYING AND CORRECTING COGNITIVE BIASES IN SUBJECTIVE PROBABILITY ELICITATION SURVEYS: MODEL BASED APPROACHES by Venkataramaavadhanulu Prava A thesis submitted to Johns Hopkins University in conformity

More information

CARISMA-LMS Workshop on Statistics for Risk Analysis

CARISMA-LMS Workshop on Statistics for Risk Analysis Department of Mathematics CARISMA-LMS Workshop on Statistics for Risk Analysis Thursday 28 th May 2015 Location: Department of Mathematics, John Crank Building, Room JNCK128 (Campus map can be found at

More information

How to weigh the strength of prior information and clarify the expected level of evidence?

How to weigh the strength of prior information and clarify the expected level of evidence? How to weigh the strength of prior information and clarify the expected level of evidence? Martin Posch martin.posch@meduniwien.ac.at joint work with Gerald Hlavin Franz König Christoph Male Peter Bauer

More information

Analysis of Competing Hypotheses using Subjective Logic (ACH-SL)

Analysis of Competing Hypotheses using Subjective Logic (ACH-SL) Analysis of Competing Hypotheses using Subjective Logic (ACH-SL) Simon Pope and Audun Jøsang, DSTC, CRC for Enterprise Distributed Systems Technology ICCRTS, 13-16 June 2005 June 22, 2005 DSTC 2005. All

More information

A Mini-conference on Bayesian Methods at Lund University 5th of February, 2016 Room C121, Lux building, Helgonavägen 3, Lund University.

A Mini-conference on Bayesian Methods at Lund University 5th of February, 2016 Room C121, Lux building, Helgonavägen 3, Lund University. A Mini-conference on Bayesian Methods at Lund University 5th of February, 2016 Room C121, Lux building, Helgonavägen 3, Lund University. www.lucs.lu.se/bayes-at-lund-2016/ Program 12.15 12.45 Welcome and

More information

Bayesian Tolerance Intervals for Sparse Data Margin Assessment

Bayesian Tolerance Intervals for Sparse Data Margin Assessment Bayesian Tolerance Intervals for Sparse Data Margin Assessment Benjamin Schroeder and Lauren Hund ASME V&V Symposium May 3, 2017 - Las Vegas, NV SAND2017-4590 C - (UUR) Sandia National Laboratories is

More information

Inconsistent Inference in Qualitative Risk Assessment

Inconsistent Inference in Qualitative Risk Assessment Inconsistent Inference in Qualitative Risk Assessment November 10, 2013 Prepared by Kailan Shang 1 1 Kailan Shang, FSA, CFA, PRM, SCJP, of Manulife Financial, can be reached at klshang81@gmail.com. Page

More information

SLAUGHTER PIG MARKETING MANAGEMENT: UTILIZATION OF HIGHLY BIASED HERD SPECIFIC DATA. Henrik Kure

SLAUGHTER PIG MARKETING MANAGEMENT: UTILIZATION OF HIGHLY BIASED HERD SPECIFIC DATA. Henrik Kure SLAUGHTER PIG MARKETING MANAGEMENT: UTILIZATION OF HIGHLY BIASED HERD SPECIFIC DATA Henrik Kure Dina, The Royal Veterinary and Agricuural University Bülowsvej 48 DK 1870 Frederiksberg C. kure@dina.kvl.dk

More information

Actor (and network) models

Actor (and network) models Actor (and network) models EPA2142 Policy and Decision Models Research Seminar Leon Hermans 29-06-10 Delft University of Technology The original lecture slides by Leon Hermans have been adapted for OpenCourseWare

More information

Cognitive Modeling. Lecture 9: Intro to Probabilistic Modeling: Rational Analysis. Sharon Goldwater

Cognitive Modeling. Lecture 9: Intro to Probabilistic Modeling: Rational Analysis. Sharon Goldwater Cognitive Modeling Lecture 9: Intro to Probabilistic Modeling: Sharon Goldwater School of Informatics University of Edinburgh sgwater@inf.ed.ac.uk February 8, 2010 Sharon Goldwater Cognitive Modeling 1

More information

Therapy symposium Formal Radiation Therapy Safety Processes

Therapy symposium Formal Radiation Therapy Safety Processes Therapy symposium Formal Radiation Therapy Safety Processes M. Saiful Huq, PhD, FAAPM, FInstP, Professor and Director University of Pittsburgh Cancer Institute and UPMC CancerCenter Pittsburgh, Pennsylvania,

More information

Cognitive Modeling. Mechanistic Modeling. Mechanistic Modeling. Mechanistic Modeling Rational Analysis

Cognitive Modeling. Mechanistic Modeling. Mechanistic Modeling. Mechanistic Modeling Rational Analysis Lecture 9: Intro to Probabilistic Modeling: School of Informatics University of Edinburgh sgwater@inf.ed.ac.uk February 8, 2010 1 1 2 3 4 Reading: Anderson (2002). 2 Traditional mechanistic approach to

More information

Wendy J. Reece (INEEL) Leroy J. Matthews (ISU) Linda K. Burggraff (ISU)

Wendy J. Reece (INEEL) Leroy J. Matthews (ISU) Linda K. Burggraff (ISU) INEEL/CON-98-01115 PREPRINT Estimating Production Potentials: Expert Bias in Applied Decision Making Wendy J. Reece (INEEL) Leroy J. Matthews (ISU) Linda K. Burggraff (ISU) October 28, 1998 October 30,

More information

What Is Science? Lesson Overview. Lesson Overview. 1.1 What Is Science?

What Is Science? Lesson Overview. Lesson Overview. 1.1 What Is Science? Lesson Overview 1.1 What Science Is and Is Not What are the goals of science? One goal of science is to provide natural explanations for events in the natural world. Science also aims to use those explanations

More information

A Case Study: Two-sample categorical data

A Case Study: Two-sample categorical data A Case Study: Two-sample categorical data Patrick Breheny January 31 Patrick Breheny BST 701: Bayesian Modeling in Biostatistics 1/43 Introduction Model specification Continuous vs. mixture priors Choice

More information

UNIT 4 ALGEBRA II TEMPLATE CREATED BY REGION 1 ESA UNIT 4

UNIT 4 ALGEBRA II TEMPLATE CREATED BY REGION 1 ESA UNIT 4 UNIT 4 ALGEBRA II TEMPLATE CREATED BY REGION 1 ESA UNIT 4 Algebra II Unit 4 Overview: Inferences and Conclusions from Data In this unit, students see how the visual displays and summary statistics they

More information

Who? What? What do you want to know? What scope of the product will you evaluate?

Who? What? What do you want to know? What scope of the product will you evaluate? Usability Evaluation Why? Organizational perspective: To make a better product Is it usable and useful? Does it improve productivity? Reduce development and support costs Designer & developer perspective:

More information

Introduction to Behavioral Economics Like the subject matter of behavioral economics, this course is divided into two parts:

Introduction to Behavioral Economics Like the subject matter of behavioral economics, this course is divided into two parts: Economics 142: Behavioral Economics Spring 2008 Vincent Crawford (with very large debts to Colin Camerer of Caltech, David Laibson of Harvard, and especially Botond Koszegi and Matthew Rabin of UC Berkeley)

More information

Chapter 2: Research Methods in I/O Psychology Research a formal process by which knowledge is produced and understood Generalizability the extent to

Chapter 2: Research Methods in I/O Psychology Research a formal process by which knowledge is produced and understood Generalizability the extent to Chapter 2: Research Methods in I/O Psychology Research a formal process by which knowledge is produced and understood Generalizability the extent to which conclusions drawn from one research study spread

More information

Steve Eisenhawer Terry Bott D.V. Rao. American Nuclear Society Annual Meeting June 1-4, 2003 San Diego, CA

Steve Eisenhawer Terry Bott D.V. Rao. American Nuclear Society Annual Meeting June 1-4, 2003 San Diego, CA LA-UR-03-3467 Approved for public release; distribution is unlimited. Title: Assessing the Risk of Nuclear Terrorism Using Logic Evolved Decision Analysis Author(s): Steve Eisenhawer Terry Bott D.V. Rao

More information

Introduction to Bayesian Analysis 1

Introduction to Bayesian Analysis 1 Biostats VHM 801/802 Courses Fall 2005, Atlantic Veterinary College, PEI Henrik Stryhn Introduction to Bayesian Analysis 1 Little known outside the statistical science, there exist two different approaches

More information

Workshop Information IAEA Workshop

Workshop Information IAEA Workshop IAEA Training Course on Safety Assessment of NPPs to Assist Decision Making Incident Evaluation and Ranking of Operational Events Lecturer Lesson Lesson IV IV3_II.13 Workshop Information IAEA Workshop

More information

Introduction to Computational Neuroscience

Introduction to Computational Neuroscience Introduction to Computational Neuroscience Lecture 5: Data analysis II Lesson Title 1 Introduction 2 Structure and Function of the NS 3 Windows to the Brain 4 Data analysis 5 Data analysis II 6 Single

More information

Bayesian and Frequentist Approaches

Bayesian and Frequentist Approaches Bayesian and Frequentist Approaches G. Jogesh Babu Penn State University http://sites.stat.psu.edu/ babu http://astrostatistics.psu.edu All models are wrong But some are useful George E. P. Box (son-in-law

More information

Risk Interpretation and Action

Risk Interpretation and Action Risk Interpretation and Action J. Richard Eiser 1, Ann Bostrom 2, Ian Burton 3, David Johnston 4, John McClure 5, Douglas Paton 6, Joop van der Pligt 7, Mathew White 8. 1 University of Sheffield, UK; 2

More information

Deviations Inspection Observations & Issues to Consider for Achieving Compliance

Deviations Inspection Observations & Issues to Consider for Achieving Compliance Deviations Inspection Observations & Issues to Consider for Achieving Compliance Kevin O Donnell, Ph.D. Market Compliance Manager Irish Medicines Board IMB GMP & Market Compliance Information Day Dublin,

More information

Continuous/Discrete Non Parametric Bayesian Belief Nets with UNICORN and UNINET

Continuous/Discrete Non Parametric Bayesian Belief Nets with UNICORN and UNINET Continuous/Discrete Non Parametric Bayesian Belief Nets with UNICORN and UNINET R.M. Cooke 1, D. Kurowicka, A. M. Hanea, O. Morales, D. A. Ababei Department of Mathematics Delft University of Technology

More information

A Proposal for the Validation of Control Banding Using Bayesian Decision Analysis Techniques

A Proposal for the Validation of Control Banding Using Bayesian Decision Analysis Techniques A Proposal for the Validation of Control Banding Using Bayesian Decision Analysis Techniques Paul Hewett Exposure Assessment Solutions, Inc. Morgantown, WV John Mulhausen 3M Company St. Paul, MN Perry

More information

Probabilistic Reasoning with Bayesian Networks and BayesiaLab

Probabilistic Reasoning with Bayesian Networks and BayesiaLab The presentation will start at: 13:00:00 The current time is: 13:00:42 Central Standard Time, UTC-6 Probabilistic Reasoning with Bayesian Networks and BayesiaLab Introduction Your Hosts Today Stefan Conrady

More information

METHODOLOGY FOR DISSERTATION

METHODOLOGY FOR DISSERTATION METHODOLOGY FOR DISSERTATION In order to expose the methods of scientific work, it is necessary to briefly clarify the terms of methodology, methods and scientific methods. The methodology comes from the

More information

Application of Analytical Hierarchy Process and Bayesian Belief Networks for Risk Analysis

Application of Analytical Hierarchy Process and Bayesian Belief Networks for Risk Analysis Volume 2 Application of Analytical Hierarchy Process and Bayesian Belief Networks for Risk Analysis A. Ahmed, R. Kusumo 2, S Savci 3, B. Kayis, M. Zhou 2 and Y.B. Khoo 2 School of Mechanical and Manufacturing

More information

Safety Analysis. Workshop Information IAEA Workshop

Safety Analysis. Workshop Information IAEA Workshop IAEA Training Course on Safety Assessment of NPPs to Assist Decision Making Safety Analysis Lecturer Lesson Lesson IV IV1_1 Workshop Information IAEA Workshop City, XX XX - City -XX, Country Month, Year

More information

Verification and Validation of Accident Management Guidelines in Indian PHWRs

Verification and Validation of Accident Management Guidelines in Indian PHWRs Verification and Validation of Accident Management Guidelines in Indian PHWRs By RS&A, NPCIL Mumbai, India riteshraj@npcil.co.in IAEA Headquarters, Vienna, Austria, 12 14 December 2016 01/20 NPCIL Nuclear

More information

The role of sampling assumptions in generalization with multiple categories

The role of sampling assumptions in generalization with multiple categories The role of sampling assumptions in generalization with multiple categories Wai Keen Vong (waikeen.vong@adelaide.edu.au) Andrew T. Hendrickson (drew.hendrickson@adelaide.edu.au) Amy Perfors (amy.perfors@adelaide.edu.au)

More information

Introduction to Applied Research in Economics

Introduction to Applied Research in Economics Introduction to Applied Research in Economics Dr. Kamiljon T. Akramov IFPRI, Washington, DC, USA Training Course on Introduction to Applied Econometric Analysis November 14, 2016, National Library, Dushanbe,

More information

Introduction to Applied Research in Economics

Introduction to Applied Research in Economics Introduction to Applied Research in Economics Dr. Kamiljon T. Akramov IFPRI, Washington, DC, USA Regional Training Course on Applied Econometric Analysis June 12-23, 2017, WIUT, Tashkent, Uzbekistan Why

More information

Bayesian Methods in Regulatory Science

Bayesian Methods in Regulatory Science Bayesian Methods in Regulatory Science Gary L. Rosner, Sc.D. Regulatory-Industry Statistics Workshop Washington, D.C. 13 September 2018 What is Regulatory Science? US FDA Regulatory Science is the science

More information

New Effects-Based Operations Models in War Games. Lee W. Wagenhals Larry K. Wentz

New Effects-Based Operations Models in War Games. Lee W. Wagenhals Larry K. Wentz New Effects-Based Operations Models in War Games Lee W. Wagenhals Larry K. Wentz 2003 ICCRTS 8th International Command and Control Research and Technology Symposium June 19, 2002 National Defense University

More information

Psychotherapists and Counsellors Professional Liaison Group (PLG) 15 December 2010

Psychotherapists and Counsellors Professional Liaison Group (PLG) 15 December 2010 Psychotherapists and Counsellors Professional Liaison Group (PLG) 15 December 2010 Standards of proficiency for counsellors Executive summary and recommendations Introduction At the meeting on 19 October

More information

PHO MetaQAT Guide. Critical appraisal in public health. PHO Meta-tool for quality appraisal

PHO MetaQAT Guide. Critical appraisal in public health. PHO Meta-tool for quality appraisal PHO MetaQAT Guide Critical appraisal in public health Critical appraisal is a necessary part of evidence-based practice and decision-making, allowing us to understand the strengths and weaknesses of evidence,

More information

Bayesian Models for Combining Data Across Subjects and Studies in Predictive fmri Data Analysis

Bayesian Models for Combining Data Across Subjects and Studies in Predictive fmri Data Analysis Bayesian Models for Combining Data Across Subjects and Studies in Predictive fmri Data Analysis Thesis Proposal Indrayana Rustandi April 3, 2007 Outline Motivation and Thesis Preliminary results: Hierarchical

More information

Stepwise Knowledge Acquisition in a Fuzzy Knowledge Representation Framework

Stepwise Knowledge Acquisition in a Fuzzy Knowledge Representation Framework Stepwise Knowledge Acquisition in a Fuzzy Knowledge Representation Framework Thomas E. Rothenfluh 1, Karl Bögl 2, and Klaus-Peter Adlassnig 2 1 Department of Psychology University of Zurich, Zürichbergstraße

More information