ELICITING EXPERT KNOWLEDGE IN SUPPORT OF PLANNING AND ADAPTIVE MANAGEMENT

Transcription

1 ELICITING EXPERT KNOWLEDGE IN SUPPORT OF PLANNING AND ADAPTIVE MANAGEMENT C. Ashton Drew Research Coordinator North Carolina Fish & Wildlife Cooperative Research Unit

2 Outline Expert Knowledge Applications Characteristics of Expert Knowledge NOT issues of bias, dominance, etc Forms of expert knowledge Domains of expertise Elicitation Methods NOT group elicitation, consensus methods Statistical encoding of individual knowledge for Bayesian regression models

3 Decision Case Studies King Rail Habitat Valuation What land to acquire/manage for King Rail and what land to manage for other species? Biodiversity Impacts of Agriculture How can a farmer maximize the biodiversity value of a field in commercial production? Freshwater Mussel Recovery Which stream reaches are candidates for protection, restoration, resettlement, introduction, rescue, or no action?

4 Some Common Challenges Data AND knowledge are limited together Experts fear mis-use of their knowledge Scale of observations rarely match scales of decision models Highly dynamic systems Experts also managers Expert exhaustion Some strategies Two stages: Qualitative and Quantitative Specific experts for specific tasks Distinguish biological vs management input Hypotheses NOT predictions in a Bayesian framework

5 Expert Knowledge Applications Gap In Ecology & Biosecurity In Medicine & Social Science Urgent outcome Invasive pest response Military threat response Transfer knowledge to new situation (pioneering investigation) Empirical measures poor Expected outcomes of novel crop (biofuels) on wildlife Biomass trends based on aerial imagery Expected outcomes of a foreign education program applied in locally Occurrence of rape and abuse based on police reports Data insufficient Climate change impact studies Rare disease prognosis Distill current knowledge on complex process Initiate continual learning cycle Impacts of flow on native species in waterways Onset of toxic algal blooms Impacts of media campaigns on childhood obesity Effectiveness of drug treatment for an individual s depression

6 Who are our experts? Anyone with substantive knowledge and experience relevant to the domain of a particular problem Deliberate practice After completion of formal or informal training, they intentionally practice for many years during which they receive regular, unambiguous feedback on their judgements. Expert knowledge personal beliefs that are in agreement with facts.

7 Forms of Knowledge Expression Existential Statements Qualitative Information Quantitative Information Relationships & Synthesis Evaluation & Decisions A and B exist C does not A is greater than B A = 10 B = 5 C = 0 A, B, and C are linked A = 2(B+C) Need more A Get more by increasing B or C

8 Domains of Expertise (What knowledge is available?)

9 Domains of Expertise (Should I weight experts?) No known consistent correlate to predict expert accuracy and precision (Burgman et al. 2011) Use multiple experts Allow iterative response Weight all experts equally Use domain information to indicate strengths & weaknesses of expert pool, rather than individual experts

10 Domains of Expertise (Should I calibrate experts?)

11 Forms of Knowledge Expression Existential Statements Qualitative Information Quantitative Information Relationships & Synthesis Evaluation & Decisions A and B exist C does not A is greater than B A = 10 B = 5 C = 0 A, B, and C are linked A = 2(B+C) Need more A Get more by increasing B or C

12 Two Elicitation Methods (to estimate beta coefficients in logistic regressions) Requires specialized software? Can be accomplished remotely? Elicitation is repeatable? Statistical model structure is flexible? Easily handles multiple experts? Provides model feedback during elicitation? Requires knowledge of basic probability theory? Requires quantitative knowledge (or hypotheses)? Incorporates measures of confidence? Simple, Direct (O Leary) Indirect, Graphically Assisted (Low Choy & James)

13 Simple, Direct Method Experts identify regression variables Provide qualitative, univariate response predictions Example question for decisions among categorical variables: Compared to the baseline choice (e.g. pasture), I would expect this choice (e.g. plant corn) to the occurrence of golden mice. Example question for decisions on continuous variables: As the decision value increases (e.g. mowing frequency), I would expect a in the occurrence of golden mice. Score Definition +1 Increase 0 No Change -1 Decrease

14 Simple, Direct Method Experts indicate their level of confidence Response plus confidence information are combined to generate probability distribution for estimate of beta coefficient Score Text Definition Odds Ratio Definition Probability 1 Fully confident this is true All odds on Very confident this is true Better than 10-1 or 20-1 odds on Somewhat confident this is true Better than 2-1 or 3-1 odds on I could be right or wrong odds of being right 0.50 Operational Decision: Plant Corn Species: Golden mice

15 Indirect, Graphically Assisted Method Elicitator Software Scenario-based elicitation Sampling expert experience

16 Indirect, Graphically Assisted Method ORIENTATION INTRODUCE SCENARIO Imagine 100 sites with <30 % forested riparian, 2-5 m bank full depth, significant groundwater input,

17 Indirect, Graphically Assisted Method ORIENTATION INTRODUCE SCENARIO ELICIT EXTREMES What is the minimum number of sites you would expect to maintain substrate temperatures within range suitable for TRSM?

18 Indirect, Graphically Assisted Method ORIENTATION INTRODUCE SCENARIO ELICIT EXTREMES REFINE EXPECTATIONS Now bring in these limits to be more informative so that you re 95% sure. Bring in further so you re 50% sure... Now what s your best estimate of...

19 Indirect, Graphically Assisted Method ORIENTATION INTRODUCE SCENARIO ELICIT EXTREMES REFINE EXPECTATIONS ESTIMATE ACCURACY

20 One expert, many scenarios

21 One expert, many scenarios Each level of each variable is represented multiple times but in different combinations

22 Thank you! Drescher et al. (2013) Toward rigorous use of expert knowledge in ecological research. Ecosphere 4(7) Article 83 Burgman et al. (2011) Expert status and performance. PLoS ONE 6(7) e22998 O Leary et al. (2008) Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush-tailed rock-wallaby. Environmetrics 20: James et al. (2010) Elicitator: an expert elicitation tool for regression in ecology. Environmental Modeling and Software 25: Ashton Drew: cadrew@ncsu.edu

23