Presentation to the Committee on Assessment of Agent-Based Models to Inform Tobacco Product Regulation

Transcription

1 Presentation to the Committee on Assessment of Agent-Based Models to Inform Tobacco Product Regulation This work was funded by the U.S. Food and Drug Administration (FDA) through a contract with the U.S. Department of Energy/Sandia National Laboratories. The information included in this presentation reflects the views of the authors and should not be construed to represent FDA s positions or policies. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy s National Nuclear Security Administration under contract DE-AC04-94AL85000.

2 Presentation Outline Introduction Agent-Based Modeling Lessons Learned SnapDragon Conceptual Model Model Development and Verification Model Verification and Validation Process Next Steps 2

3 Agent-Based Modeling A computer-based modeling approach, often associated with modeling complex systems Agents Can be individuals, groups, organizations, or other system components Are often part of a network Rules (typically simple rules) determine agent interaction and agent behavior Agents act and interact in the system Agents/nodes are joined by links on a directed network Arrows indicate nomination, influence flows in the opposite direction 3

4 Utility of Agent-Based Models (ABMs) Useful when there is no equation for an emergent behavior Explains system behavior using lower-level dynamics. Generate the behavior of interest: agents act based on lower-level rules, community behavior emerges Lower-level assumptions, perhaps coming from a variety of domains, can be analyzed Perturbations (e.g., policies/interventions) are introduced, agent actions can cascade into changes at higher levels of the system Useful for analysis of conditions not seen before Useful for representing heterogeneity - Agent-based modeling is more useful for tracking multiple individual attributes Agent-based models (ABMs), as well as other models, Are abstract representations Are not crystal balls Provide an in-silico workbench/ quantitative exploratory testbed on which policy comparisons can be conducted 4

5 Regulatory Context The Family Smoking Prevention and Tobacco Control Act (FSPTCA), signed into law by President Barack Obama on June 22, 2009: Recognizes the FDA as the primary Federal regulatory authority with respect to the manufacture, marketing, and distribution of tobacco products FSPTCA authorizes the FDA to: Require disclosure of tobacco product ingredients Require stronger health warnings on packaging and in advertisements Create standards for tobacco products Restrict tobacco sales, distribution, and marketing Mandates regulatory decisions be based on the protection of public health, which shall be determined with respect to the risks and benefits to the population as a whole, including users and nonusers of the tobacco product 5

6 Social Network Analysis for Policy on Directed Graph Networks (SnapDragon) SnapDragon: An agent-based model Represents individuals (or other system components) within a social network Desired Model Functionality Provides prevalences for specified products (characterized by properties) and conditions Allows identification/comparison of impacts of policies and policy combinations Informs population health models (changes in initiation and cessation rate, product transition distributions) Integrates multiple aspects of theory, policy and data 6

7 Lessons Learned from Modeling for Health and Regulatory Analyses Public health Pandemic modeling (beginning in 2005) Zoonotic (Afghan) Medical facilities Hepatitis C, Legionella, Varicella Food supply chains Regulatory Nuclear Waste Disposal Compliance Assessments Nuclear Energy Safety Chemical Facilities 7

8 Lessons Learned When to use Models Future time frame, new conditions, regulatory impacts difficult to foresee Data limited or ambiguous Can t experiment in advance Modeling for Policy Multiple modeling approaches Don t over-constrain problem boundaries Agent-Based Models Emergent behavior in response to interactions Conditions not seen before Heterogeneity - Agent-based modeling is more useful for tracking multiple individual attributes Relatively easy to verify, more difficult to validate 8

9 Lessons Learned: Synthesized in Process Conceptual Model Development Define system and boundaries Identify model requirements Existing models; Relevant models, draw from information disciplines Develop model(s) Keep simple as possible Build and test incrementally Build confidence in model structure and results Re-evaluate model as system changes, or as data are collected 9

10 Stages of Model Development and Use Required confidence depends on use of the model Greater output fidelity possible as model matures Different uses in different stages of development Stage Conceptual Exploration Explanation Level of Confidence Minimal Low Medium Utility Problem Framing Hypothesis Formulation Develop Intuition Hypothesis Testing Evaluating trade-offs Screening options Confirmation High Inform high-consequence decisions 10

11 Our Clearly Posed Problem FSPTCA: Regulatory decisions to be based on the protection of public health, which shall be determined with respect to the risks and benefits to the population as a whole, including users and nonusers of the tobacco product 11

12 Conceptual Model Development Entities and Relationships Building the Big Picture 12

13 Problem Space: Tobacco Use Dynamics Population Non-Users Tobacco Users Morbidity Mortality Social Norms Former Users 13

14 Tobacco Use Dynamics Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Users Former Users HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments Tobacco Regulation 14

15 Tobacco Use Dynamics Tobacco Control Community Tobacco Product Retailers Manufacturers of Tobacco Products Public Health Advocacy Research Lobbying CDC, NGOs, Other International Population Non-Users Tobacco Users Product Importing Tobacco Product Advertising & Marketing Product Research Production and Distribution PH Advertising & Education Morbidity Public Relations & Proxies Mortality Social Norms Former Users Tobacco Products Lobbying Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments Tobacco Regulation 15

16 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Use Dynamics Tobacco Users Former Users Tobacco Product Retailers Product Importing Manufacturers of Tobacco Products Tobacco Product Advertising & Marketing Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments Tobacco Regulation US Congress State Legislatures City Councils Legislation Supreme Courts Appellate Courts District Courts (Federal or State) Judicial 16

17 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Use Dynamics Tobacco Users Former Users Tobacco Product Retailers Product Importing Manufacturers of Tobacco Products Tobacco Product Advertising & Marketing Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Finance Stock Markets Commodity Markets HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments Tobacco Regulation US Congress State Legislatures City Councils Legislation Supreme Courts Appellate Courts District Courts (Federal or State) Judicial 17

18 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Use Dynamics Tobacco Users Former Users Tobacco Product Retailers Product Importing Manufacturers of Tobacco Products Tobacco Product Advertising & Marketing Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Pharmaceuticals Nicotine Products as Quitting Aids Other (Nicotine Devices) Pharma Advertising & Marketing Treatment & Cessation Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Finance Stock Markets Commodity Markets Supreme Courts Treatment of Individuals Appellate Courts Research Health Care Health Life Liability HHS/FDA/CTP Treasury/ATF/TTB State Governments US Congress State Legislatures City Councils District Courts (Federal or State) Judicial Insurance Local Governments Tobacco Regulation Legislation 18

19 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Use Dynamics Tobacco Users Former Users Tobacco Product Retailers Tobacco Products Product Importing Tobacco Products Manufacturers of Tobacco Products Tobacco Product Advertising & Marketing Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Pharmaceuticals Nicotine Products as Quitting Aids Other (Nicotine Devices) Pharma Advertising & Marketing Treatment & Cessation Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Tobacco Leaves Finance Stock Markets Commodity Markets Supreme Courts Treatment of Individuals Appellate Courts Research Health Care Health Life Liability HHS/FDA/CTP Treasury/ATF/TTB State Governments US Congress State Legislatures City Councils District Courts (Federal or State) Judicial Insurance Local Governments Tobacco Regulation Legislation Arrows show flow of: Tobacco 19

20 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Morbidity Mortality Social Norms Population Non-Users Tobacco Use Dynamics Tobacco Users Former Users Tobacco Product Retailers Tobacco Products Product Importing Tobacco Products Manufacturers of Tobacco Products Tobacco Product Advertising & Marketing Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Pharmaceuticals Nicotine Products as Quitting Aids Other (Nicotine Devices) Pharma Advertising & Marketing Treatment & Cessation Quitting Aids Treatment Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Finance Stock Markets Commodity Markets Supreme Courts Treatment of Individuals Appellate Courts Pharmaceuticals Research Health Care Health Life Liability HHS/FDA/CTP Treasury/ATF/TTB State Governments US Congress State Legislatures City Councils District Courts (Federal or State) Judicial Insurance Local Governments Tobacco Regulation Legislation Arrows show flow of: Tobacco Goods 20

21 Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Education & Warnings Pharmaceuticals Nicotine Products as Quitting Aids Other (Nicotine Devices) Pharma Advertising & Marketing Treatment & Cessation Morbidity Mortality Social Norms Population Non-Users Quitting Aids Tobacco Use Dynamics Tobacco Users Former Users Warnings Treatment Tobacco Product Retailers Product Importing Tobacco Tobacco Products Products Advertisements Content, Information Access & Age Limits Tobacco Product Standards Manufacturers of Tobacco Products Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Tobacco Product Advertising & Marketing Trade Agreements Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying Tobacco Leaves Data, Applications Finance Stock Markets Commodity Markets Legal Ruling Stock Price Economic Pressure (indicators) Treatment of Individuals Political Pressure Supreme Courts Appellate Courts Pharmaceuticals Research Health Care Tax $$ Health Life Liability Insurance HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments Tobacco Regulation Legal Guidance Laws Data US Congress State Legislatures City Councils Legislation Legal Reading District Courts (Federal or State) Judicial Political Pressure (Campaign Contributions $$) Arrows show flow of: Tobacco Goods Data Data / Info Political Pressure Information 21

22 Connections between Tobacco Entities Tobacco Control Community Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education $$ Pharmaceuticals Nicotine Products as Quitting Aids Other (Nicotine Devices) Pharma Advertising & Marketing Treatment & Cessation Pharmaceuticals Education & Warnings Treatment of Individuals Research $$ Health Care Tax $$ Morbidity Mortality Social Norms Population Non-Users Quitting Aids $$ Payments Tobacco Users Former Users Warnings Treatment Health Life Liability $$ Insurance $$ Tobacco Product Retailers Product Importing Tobacco Tobacco Products Products Advertisements Content, Information Access & Age Limits HHS/FDA/CTP Treasury/ATF/TTB State Governments Local Governments $$ Tobacco Product Standards Manufacturers of Tobacco Products Tobacco Growers Tobacco Regulation US Agriculture Imported Tobacco Tobacco Lobbying Political Pressure Tobacco Product Advertising & Marketing Trade Agreements Legal Guidance $$ Laws Data Product Research Production and Distribution Public Relations & Proxies Tobacco Products Lobbying $$ Tobacco Leaves Tax $$, Fine $$ US Congress State Legislatures City Councils Legislation Tax $$ Data, Applications Finance Stock Markets Commodity Markets Data Legal Reading Data / Info Political Pressure Legal Ruling Stock Price Economic Pressure (indicators) Supreme Courts Appellate Courts District Courts (Federal or State) Judicial Political Pressure (Campaign Contributions $$) Arrows show flow of: Tobacco Goods Information Money 22

23 Multiple Connected Feedback Loops Public Health Advocacy Research Lobbying CDC, NGOs, Other International PH Advertising & Education Education & Warnings Pharmaceuticals Research Pharmaceuticals $$ Nicotine products as quitting aids (patches, gum) Other (nicotine devices) Pharma advertising and marketing Others for treatment and cessation (antidepressants, nicotine vaccine, chemo, radiation sources) Health Care Non-Users Morbidity Mortality Treatment of Individuals Social Norms Public Health Cycle Tax $$ Quitting Aids $$ Population Payments Tobacco Users Former Users Warnings Treatment $$ Insurance Tobacco Product Retailers $$ Tobacco Products $$ Product Importing Tobacco Products Advertisements Content Information Access & Age Limits Tobacco Product Standards Tobacco Growers US Agriculture Imported Tobacco Tobacco Lobbying Regulations/ Legislative Cycle FDA Political Pressure Tobacco Product Regulation Tobacco Product Manufacturers Tobacco Product Advertising and Marketing Product Research Labor Production and Distribution (legal and illegal) Public Relations & Proxies Products Tobacco Products Lobbying Trade Agreements Legal Guidance $$ Laws Data Tobacco Leaves $$ Tax $$, Fine $$ Federal State Local Legislation Tax $$ Money Advertising Data, Applications Finance Stock Markets Commodity Markets Legal Interpretation Data Economic Pressure (indicators) Data / Info Political Pressure Legal Ruling Stock Price Judicial Political Pressure (Campaign Contributions $$) Arrows show flow of: Tobacco Goods Information Money 23

24 Model Purview Public Health Cycle Regulatory Cycle 24

25 Social Influence and Information Flow The evidence is sufficient to conclude that there is a causal relationship between peer group social influences and the initiation and maintenance of smoking behaviors during adolescence. - Preventing Tobacco Use Among Youth and Young Adults, A Report of the Surgeon General,

26 Conceptual Model Development We are here 26

27 Multiple Modeling Approach Existing models examine essential components of the system Population models (macro-level e.g., Nation, State): Project future population health states, use historical and hypothesized changes in initiation and cessation rates Agent-based models that examine specific issues (reactance, points of sale) SnapDragon is useful for examining the use of tobacco products as an emergent behavior due to changes in products and regulations 27

28 SnapDragon Model Goals Comparative analysis of policy option impacts on individual/group initiation and cessation behaviors Inform population health models - changes in initiation and cessation rate, product transition distributions Explore a future that may be very different from the past Evolving market / new products New/evolving regulatory environment 28

29 SnapDragon Conceptual Model SnapDragon Conceptual Model 29

30 Key Model Elements Population of individuals Tobacco products and their attributes Regulations and interventions that limit access to tobacco products and their use, facilitate quitting assistance (quit lines, aids) Information about tobacco products (marketing and risk) Peer/family/social influence Individual differences (risk takers, adults, youth, role models, addiction) Tobacco use Tobacco Products Information Individuality Regulations Family/Friends 30

31 Model Tobacco-Related Influences Model Requirement Represent individuals integrated view of tobacco Represent influences and propagate changes in individual views Model Representation Opinion variable Opinion Dynamics High school seniors social network (From: Variations in network boundary and type: A study of adolescent peer influences, Thomas W. Valente et al., [in press: Social Networks, an international journal of structural analysis]) 31

32 Opinion An Integrated View of a Product Opinion variable represents: Integrated view of a product The result of multiple influences and perceptions Range 0 (lowest opinion of a product) 1 (highest opinion of a product) 32

33 Positive Negative Opinion: Integrated View of Tobacco Social Influences (Affect) Sophisticated Cool Group identity Adult Independence Déclassé Smell/taste Corporate Practical Influences (Perceived Utility) Stress relief Concentration Avoiding the effects of withdrawal Weight control Disease Addiction Dangers of Environmental Tobacco Smoke Positive Negative 33

34 Applications of Opinion Dynamics Models Opinion dynamics models of information spread have been demonstrated and validated in many domains Initially proposed in 1999 to model uptake of environmentally sustainable agricultural practices in Europe (Deffuant et al. 2000) Political opinions (Galam 2012) Political extremism and terrorism (Deffuant 2006, Amblard and Deffuant 2002) Gender-based social networks in tribal cultures (Moore et al. 2012) 34

35 Simplified Conceptual Diagram Family/Friends Individual Opinion Products Information 35

36 Simplified Conceptual Diagram Family/Friends Products Individual Opinion Individual Tobacco-Use Behavior Information 36

37 Simplified Conceptual Diagram Family/Friends Barriers to Access Products Individual Opinion Costs Addiction Risk Affinity Individual Tobacco-Use Behavior Information 37

38 SnapDragon Modeling Approach SnapDragon Modeling Approach 38

39 Simplified Modeling Approach Family/Friends Opinion Dynamics: An approach for the analysis of social influences on individual Individual opinions, the emergence Opinion of community-scale Products patterns Barriers to Access Costs Addiction Risk Affinity Individual Tobacco-Use Behavior Information 39

40 Simplified Modeling Approach Family/Friends Opinion Dynamics: An approach for the analysis of social influences on individual Individual opinions, the emergence Opinion of community-scale Products patterns Barriers to Access Costs Opinion-to-Behavior Mapping Addiction Risk Affinity Individual Tobacco-Use Behavior Information 40

41 Summary of SnapDragon Modeling Approach Model Purpose Assess changes in future prevalence Comparison of impacts, potential effects of policies and policy combinations Inform population models Inform regulatory design Model Requirements Based on system view and regulatory authorities Model Development Agent-based technique most suitable Emergent behavior in response to interactions Heterogeneity - Agent-based modeling is more useful for tracking multiple individual attributes Opinion dynamics describes movement of ideas on social networks Opinion-to-behavior mapping incorporates Regulatory components such as access, costs Individual characteristics, e.g., risk-taking, addiction 41

42 SnapDragon Modeling Approach We are here Model results are discussed in terms of tobacco use behaviors, consistent with model design. The scenarios presented are hypothetical and show model behavior. Results are not intended to represent a specific forecast. 42

43 Model Implementation Starting with Opinion Spread Use software agents to represent a population of individuals Each individual assigned two values to start: Opinion: Integrated view of the issue under study Tolerance: Degree of openness to consider other ideas At each time step, agents adjust their opinion based on opinions of neighboring agents and their own tolerance 43

44 Individual Opinion: Bounded-Confidence Formulation x i N ( t i 1) S i : x x k i ( t) ( t) 1 k x i N i [ x ik ( t) i k ( t) x i ( t)] Update Rule: Adjust individual agent s opinion by mean scaled opinion differences of opinion and neighbors opinions within tolerance limits S i Tolerance (Do you influence i Plasticity (How much you influence i x Opinion (My integrated N i Set of ( t) Subset of k number of out degree S i neighbors neighbors view) me?) within tolerance within tolerance me) bounds at time t bounds 44

45 Opinion Model: Effect of Tolerance B B and C contribute to new opinion value for A D s opinion excluded since it is outside of tolerance A ε: Tolerance x: Opinion (integrated view of tobacco use) C D Directionality of arrows represents nominations (individuals nominate others as their friends) Influence flows in opposite direction (from the nominee to the individual) 45

46 Opinion Model: Effect of Plasticity and Tolerance At each step, the new opinion is original value plus mean residual scaled by plasticity B D C A A D B C µ: Plasticity ε: Tolerance 46

47 Opinion Model: Information Effects Information sources modeled as social network nodes broadcasting messages to individuals Pro-tobacco information sources represented as highopinion messages, values closer to 1.0 A B Media Node D 0.1 for all Public-health information sources represented as lowopinion messages, values closer to 0.0 C 47

48 Opinion Model: Information Effects Due to tolerance and changes in opinion: B D C A Media the Media node primarily impacts Agent B s opinion Agent A s opinion continues to be influenced by those of Agent B and Agent C B A D C 48

49 Opinion Model: Default Parameter Values Networks ER graphs (directed random graphs with a Poisson degree distribution) 250 nodes Probability of connection of Initial Agent Opinion Distribution (uniform 0-1) Tolerance 0.21 Plasticity 0.05 Note: See Table 1 in the manuscript An Opinion-Driven Behavioral Dynamics Model for Addictive Behaviors for a complete list of parameter default values, ranges and explanations 49

50 Agents SnapDragon Behavior: Media Broadcast Impact on Average Opinion Media Node Broadcast opinion sweep [0-1] Default values for all other parameters 10,000 runs 250-agent network 2.5 million agents total 50

51 Media Broadcast Impact on Individual Opinion 51

52 Mapping of Opinion to Behavior: Smoking If the agent s opinion about smoking is greater than threshold, the agent is a smoker The higher the initiation threshold the greater the barriers to the behavior The initiation threshold can be age dependent Initiation Threshold Opinion of Smoking Smoker. x 0 52

53 Mapping of Opinion to Behavior: Addiction Because of addiction, behavior is path dependent Non-Smoker Cessation Threshold Initiation Threshold The effect of addiction is represented with a cessation threshold A smokers opinion must fall below cessation threshold to overcome addiction Opinion of Smoking Smoker 53

54 Mapping of Opinion to Behavior: Addiction Factor Cessation Threshold Initiation Threshold The addiction factor is the difference between the initiation and cessation thresholds Non-Smoker D x Smoker Addiction Factor 54

55 Examine: Cessation Support Effect on Prevalence Effect of cessation support modeled as an increase in the Cessation Threshold Non-Smoker New Cessation Threshold Initiation Threshold Result - agents become former-smokers at a higher opinion value D x Smoker Addiction Factor 55

56 Testing Impact of Cessation Threshold on Modeled Prevalence Addiction Factor = 0 Cessation Threshold = Initiation Threshold Implications: If cessation support helps overcome addiction, then the model implementation of raising the Cessation Threshold value is appropriate Model behaves as designed for the full range of Cessation Threshold values 56

57 Cessation Threshold Impact on Modeled Prevalence Heat map shows the range of input where simulated prevalence is reduced 57

58 Combined Effects of Cessation Threshold and Messaging No messaging Public Health messaging Public Health and Pro-Smoking messaging Increasing the Cessation Threshold can reduce modeled prevalence Simulated prevalence reduction is enhanced with a low broadcast opinion Hypothetical public health message scenario (Broadcast Opinion = 0.35) With high and low broadcast opinions active, a larger increase in the Cessation Threshold is required before prevalence is reduced Hypothetical multi-messaging scenario (Broadcast Opinions = 0.75 and 0.35) 58

59 Representing Interventions Examples Intervention Model or Parameter Change Provide cessation support Decrease access through indoor air laws, age limits, or taxes and enforce regulations Increase risk awareness Raise the cessation threshold Raise the initiation threshold for specified age group or population Add education node(s) 59

60 Next Phase: Modeling Multiple Products Representing product switching and alternate product uptake Add risk affinity attribute for individual agents Represent product-specific risk perception as a distinct component of opinion 2-product model that has product switching and dual use 60

61 Modeling Multiple Tobacco Products: Risk Affinity Effect Risk Affinity Effect Decreases the range of opinion values under which the agent is a non-user Non-User User Increases the range of opinion values under which the agent is a user Cessation Threshold Initiation Threshold Opinion of Product Initiate product if: Opinion Risk Perception > Initiation threshold Risk Affinity Quit product if: Opinion Risk Perception < Cessation threshold Risk Affinity 61

62 Examine Potential Prevalence of Different Products Compare model behavior Run single-product model for cigarettes and then for the alternate product Examine effects of risk affinity and risk perception on model behavior 62

63 Effect of Perceived Risk, Risk Affinity and Broadcast Opinion on Modeled Prevalence Lower perceived risk increases modeled prevalence for the same ranges of risk affinity and broadcast opinion (greater volume under the surface) The effectiveness of broadcast opinion is altered by the combined effects of perceived risks and risk affinity 63

64 Modeling Product Switching Add switching behavior for addicted smokers with opinion below the smoking threshold Quits smoking Non-Smoker May switch from smoker to user of alternative product Does not switch, continues to smoke Smoker Cessation Threshold Risk Affinity Initiation Threshold Risk Affinity Opinion of Cigarettes Switch when: opinion product2 - risk perception product2 + (initiation threshold cig - opinion cig - risk perception cig ) > cessation threshold cig - risk affinity 64

65 Modeling Two Products: Possible Agent Behaviors Nonsmoker Smoker Nonsmoker Smoker CT-RA IT-RA 0 Cigarette Opinion 1 CT-RA: Cessation Threshold less Risk Affinity IT-RA: Initiation Threshold less Risk Affinity 65

66 Alternate -Product Opinion Modeling Two Products: Possible Agent Behaviors 1 Independent product use can result in dual use Nonsmoker/User Smoker/User IT- RA CT-RA Nonsmoker / Nonuser Smoker/ Nonuser 0 CT-RA IT-RA 0 Cigarette Opinion 1 CT-RA: Cessation Threshold less Risk Affinity IT-RA: Initiation Threshold less Risk Affinity 66

67 Alternate -Product Opinion Modeling Two Products: Possible Agent Behaviors Independent product use can result in dual use 1 Nonsmoker/User Smoker/User Nonsmoker/User Smoker/User IT- RA Nonsmoker/User Nonsmoker/Nonuser Smoker/User Smoker/Nonuser CT-RA Nonsmoker / Nonuser Smoker/ Nonuser 0 CT-RA IT-RA 0 Cigarette Opinion 1 CT-RA: Cessation Threshold less Risk Affinity IT-RA: Initiation Threshold less Risk Affinity 67

68 Alternate -Product Opinion Modeling Two Products: Possible Agent Behaviors Independent product use can result in dual use 1 Nonsmoker/User Smoker/User Nonsmoker/User Smoker/User IT- RA Nonsmoker/User Nonsmoker/Nonuser Smoker/User Smoker/Nonuser Nonsmoker/User Nonsmoker/Nonuser Smoker/User Smoker/Nonuser CT-RA Nonsmoker / Nonuser Smoker/ Nonuser 0 CT-RA IT-RA 0 Cigarette Opinion 1 CT-RA: Cessation Threshold less Risk Affinity IT-RA: Initiation Threshold less Risk Affinity 68

69 Alternate -Product Opinion Modeling Two Products: Possible Agent Behaviors Independent product use can result in dual use 1 Nonsmoker/User Smoker/User Nonsmoker/User Smoker/User IT- RA Nonsmoker/User Nonsmoker/Nonuser Smoker/User Smoker/Nonuser Nonsmoker/User Nonsmoker/Nonuser Smoker/User Smoker/Nonuser CT-RA 0 Nonsmoker / Nonuser CT-RA Smoker/Nonuser Nonsmoker/User (only due to switching) Nonsmoker/Nonuser IT-RA Smoker/ Nonuser 0 Cigarette Opinion 1 CT-RA: Cessation Threshold less Risk Affinity IT-RA: Initiation Threshold less Risk Affinity 69

70 Multi-Product Model Development Status Model Testing Underway Parameter sweeps to examine model behavior compared to single product model Next steps in verification testing include: Sensitivity Analysis Evaluating switching assumptions Progression from alternate product to cigarettes and additional dual use Validation Long-term goals Provide distributions for product-transition probabilities for population health models for different demographic and tobacco-use groups Provide distributions for product-specific initiation and cessation rates as necessary for demographic groups 70

71 Summary SnapDragon is an opinion-driven behavior dynamics model that provides the capability to: Explore single and multi-product environments and their effect on prevalence Explore interventions The model development process is incremental Additional capability is added only as needed and is kept simple to maintain model utility Verification tests and face validation are part of the development process 71

72 SnapDragon Verification and Validation Building Confidence in an Agent Based Model Iterative model development and testing 72

73 Characteristics of a Trusted Model Based on solid science Accepted theories Validated studies Well built Code is reliable Results are intuitive Final results corresponds to real-world examples 73

74 Processes that Build Confidence Verification Confirms model is correctly designed and implemented Applies Software Engineering processes to ensure quality Run the model with synthetic data sets to test functionality Validation Results correspond to real-world data Model assumptions are justified Rigor depends on intended use Uncertainty Analysis Identify sources of uncertainty and effects on model results Sensitivity Analysis determines response of output to inputs Uncertainty Quantification propagates uncertainties through model 74

75 SnapDragon Verification and Validation Status Stage Level of Confidence SnapDragon Status Processes Conceptual Minimal Advanced Documentation and Review Exploration Low Active Sensitivity Analysis, Verification Explanation Medium Initial Assumption Validation Confirmation High Future Validation 75

76 Model Verification Confirms model is correctly designed and implemented Software Engineering Process Professional design and coding standards Exhaustive testing of model outputs Unit tests Integration tests Regression tests Archive all code, executables, data inputs, model outputs, experimental designs for tracking and reproducibility Analyze Code Results Run the model in variety of scenarios Do results make sense? Is model running correctly over multiple trials? 76

77 Verification Early in Model Development Developers and analysts use SnapDragon to develop intuition and perform simple verification tests 77

78 Results of Large-Scale Verification Testing Confirmation of proper model functioning: Intended mathematics correctly transformed to code Software defects are detected early and fixed before causing harm Full documentation trail from code -> output -> interpretation Identifying outliers which may represent unintended consequences Sometimes the defects found aren t in the model code: Discovered the BA scale-free network algorithm in widely-used opensource network mathematics package was wrong. Replaced it with correct code Gold Standard sensitivity analysis method (Satelli s Variance Decomposition) found to be incorrect, correction of method published in

79 Validation: Which type of model? Output corresponds with real-world data Physics and engineering models: Well-defined mathematics and constraints Validation methods sophisticated and reliable Rigorous validation enables model-based prediction Models of populations and social processes: Includes complex social dynamics, humanin-the-loop, and data gaps Validation is challenging because of complexity and limited data Models often project tendencies and trends rather than exact outcomes 79

80 Agent-Based Model Validation Agent-based models are very challenging to validate Real-world data: fragmentary, contradictory or unavailable Emergence of group dynamics from agent interactions Validate agents and interactions Validate emergent large scale phenomena Agent-based modeling tradeoff: ABMs can tackle very difficult problems with little data Rigorous prediction-level validation often not feasible ABM Validation: Determine defensible probability distributions for outcomes from limited data and large uncertainties 80

81 Agent-Based Model Validation Methods Validate Assumptions Test each model assumption Basis for validating assumptions Accepted theory Previously validated studies Tests against real-world Sensitivity analysis showing effects of assumption on model Document assumptions, bases, and degree of confidence ID Assumption Basis N023 N024 N025 Normal distribution of edge weights Ignore edge reciprocity Directionality important Referenc e Confidenc e SA SAND Moderate SA EPJ High SA TBD High Compare Output with Real-World Data Accepted procedures to match distributions and trends of data sets Required statistical significance depends on intended use N026 Percolation at n/log(n) Theory TBD High 81

82 Determine sources of uncertainty and effect on model outputs Key to effective verification and validation Parametric Uncertainty Lack of knowledge about true values of inputs Input value ranges based on expert opinion or prior studies Stochastic Uncertainty Account for random nature of real-world SnapDragon stochastic components Social network topology Uncertainty Analysis Agents receiving media messages 82

83 SnapDragon Uncertainty Analysis Design experiment Start 10,000-run scenario Generate set of input parameters Repeat 100 times Repeat 100 times Create stochastic network, 250 nodes Model network tobacco prevalence Write results to output file End 10,000-run scenario Analyze data file Interpret results 83

84 Uncertainty Quantification Assign Probability Distributions of Inputs Calculate spread of possible outputs Model 84

85 SnapDragon UQ Example Uncertainty Quantification: propagate uncertain inputs through model to generate expected range of outputs CDF CDF Quantiles 85

86 Single-Factor Sensitivity Analysis Analyze effect of one uncertain input on output values Scatterplot shows effect of single input on output 86

87 Network Media Effect on Opinion and Tobacco Use Prevalence Mean Results Number of Runs Settings: 10,000 runs, 250-agent networks Media node broadcast opinion sweep Default values for all other parameters Media Node 87

88 Multi-Factor Sensitivity Analysis Analyze effect of multiple uncertain inputs on output values Sensitivity Index measures effect of inputs on output A Input Sensitivity Index B A 0.5 B 0.3 C C