Uncertainty, Variability and Weight of Evidence: How Well Do We Know Environmental Risks?

Transcription

1 Uncertainty, Variability and Weight of : How Well Do We Know Environmental Risks? Glenn Suter U.S. Environmental Protection Agency EFSA Scientific Conference, Milan, Italy The views expressed in this presentation are those of the author and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency

2 What do decision makers & stakeholders need? Qualitative information Does this chemical cause cancer? Is this ecosystem impaired? What is the cause of observed effects? Quantitative information What concentration is protective? How much salmon habitat would be lost? How long will natural remediation require? Confidence in both

3 What is Confidence?

4 What is Scatter? Anything expressed as a numerical distribution Variability natural variation in some quantity Cannot be reduced Uncertainty Incomplete or imprecise data Can be reduced Degree of belief Subsumes variability and uncertainty for subjectivists If you were betting on the teratogenicity of a chemical in use, What odds would you accept? This is statistics controversial but familiar

5 What is weight of evidence? Metaphor based on the scales of justice The influence evidence should have on an inference The weight of a piece of evidence dropped in a pan A summary interpretation of multiple pieces of evidence The weight of the body of evidence in one pan relative to the other An inferential approach that uses this concept Often simply an expert narrative We are trying to make it more formal, transparent and defensible

6 How do we weigh evidence? Assemble Evaluate and Weight Integrate and Weigh Body of

7 Assembling evidence Assemble Search Literature Design and Conduct Studies Screen Categorize Extract and Analyze Weight Weigh Body of

8 Assembling evidence Systematic Review thorough and unbiased Assemble Search Literature Design and Conduct Studies Screen Categorize Extract and Analyze Weight Weigh Body of

9 Assembling evidence Systematic Review Assemble Search Literature Screen Design and Conduct Studies Complement existing evidence to create a coherent body of evidence Categorize Extract and Analyze Weight Weigh Body of

10 Assembling evidence Systematic Review Eliminate irrelevant and unacceptable studies Assemble Search Literature Screen Categorize Design and Conduct Studies Complement existing to create a coherent body of evidence Extract and Analyze Weight Weigh Body of

11 Assembling evidence Systematic Review Eliminate irrelevant and unacceptable studies Assemble Search Literature Screen Categorize Design and Conduct Studies Extract and Analyze Complement existing to create a coherent body of evidence Put the studies in bins that may be treated as a type of evidence Weight Weigh Body of

12 Assembling evidence Systematic Review Eliminate irrelevant and unacceptable studies Extract the data, reanalyze, and combine with other information to create evidence Assemble Search Literature Screen Categorize Design and Conduct Studies Extract and Analyze Weight Weigh Body of Complement existing evidence to create a coherent body of evidence Put the studies in bins that may be treated as a type of evidence

13 Weighting evidence Assemble Evaluate with respect to 3 attributes and assign scores Weight Relevance Weight Weight Strength Combine Weights Weight Reliability Weigh Body of

14 Weighting evidence Assemble Evaluate with respect to 3 attributes and assign scores Weight Relevance Weight Weight Strength Combine Weights Weight Reliability Combine into weight of piece or type of evidence Weigh Body of

15 Weighing bodies of evidence Assemble Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency Weight Weigh Body of Integrate Pieces or Categories of Interpret Bodies of Explain Ambiguities & Discrepancies

16 Weighing bodies of evidence Assemble Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency Weight Weigh Body of Integrate Pieces or Categories of Interpret Bodies of Explain Ambiguities & Discrepancies Which, if any, hypothesis is supported by the weight of evidence and with how much weight?

17 Weighing bodies of evidence Assemble Determine the overall weight of the body of evidence based on the weights of pieces, or types of evidence and consistency Deal with cases in which the evidence does not support any hypothesis Weight Weigh Body of Integrate Pieces or Categories of Interpret Bodies of Explain Ambiguities & Discrepancies Which, if any, hypothesis is supported by the weight of evidence and with how much weight?

18 Expressing qualitative results Example Conclusion Example Weight The chemical is a vertebrate teratogen High relevance Moderate strength Moderate reliability High consistency

19 Linking qualitative to quantitative results Summary of previous multicolored boxes Weigh for the Quality to be Quantified Merge Quantitative Choose Best Quantitative Define Confidence in Quantitative

20 Linking qualitative to quantitative results Summary of previous multicolored boxes Weigh for the Quality to be Quantified Weighted mean or other metaanalysis Merge Quantitative Choose Best Quantitative Define Confidence in Quantitative

21 Linking qualitative to quantitative results Summary of previous multicolored boxes Weigh for the Quality to be Quantified Weighted mean or other metaanalysis Merge Quantitative Choose Best Quantitative Weigh evidence to choose best source of the quantity Define Confidence in Quantitative

22 Linking qualitative to quantitative results Summary of previous multicolored boxes Weigh for the Quality to be Quantified Weighted mean or other metaanalysis Merge Quantitative Choose Best Quantitative Weigh evidence to choose best source of the quantity Integrate results Define Confidence in Quantitative

23 Expressing quantitative results Quality Protection from deformities Number 22 Units mg/l Scatter ± 3 Weight Moderate Each has supporting information and detail, but this is the basic information (tip of the hat to Ravetz and Funtowicz s NUSAP system)

24 Summary Formal weighing of evidence can Provide greater utility of results Increase defensibility Increase transparency With flexibility and adaption to purpose, WoE need not be onerous In combination with conventional statistics WoE can convey confidence in results The views expressed in this presentation are those of the author and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency