Using Risk Analysis to Manage Terrorism Risk

Transcription

1 CREATE Research Archive Research Project Summaries 2010 Using Risk Analysis to Manage Terrorism Risk Henry H. Willis RAND, Follow this and additional works at: Recommended Citation Willis, Henry H., "Using Risk Analysis to Manage Terrorism Risk" (2010). Research Project Summaries. Paper 5. This Article is brought to you for free and open access by CREATE Research Archive. It has been accepted for inclusion in Research Project Summaries by an authorized administrator of CREATE Research Archive. For more information, please contact

2 Using Risk Analysis to Manage Terrorism Risk Henry H. Willis, RAND Corporation 1. Executive Summary Applying risk analysis to support intelligence analysis Comparing risks in the homeland security mission space Applied risk management and analysis Research Accomplishments Applying risk analysis to support intelligence analysis Comparing risks in the homeland security mission space Applied risk management and analysis Applied Relevance Applying risk analysis to support intelligence analysis Comparing risks in the homeland security mission space Applied risk management and analysis Collaborative Projects Research Products Publications and Reports Presentations Models, Databases, and Software Tools and Products Education and Outreach Products Executive Summary The primary focus of research at RAND Corporation has been to apply tools from disciplines of risk and decision analysis to support risk management challenges salient to the national homeland security enterprise. Over the past year, this work has been directed towards three related efforts: Applying risk analysis to support intelligence analysis Comparing risks in the homeland security mission space Applied risk management and analysis The results of this work have been disseminated through consultations with local, state, and federal homeland security professionals, leadership at DHS, and to the scholarly community through journal papers, book chapters, poster and oral presentations at conferences Applying risk analysis to support intelligence analysis Prior work in this topic for CREATE developed a prototype resource allocation tool for use in assigning technology or personnel to intelligence requests. The prototype tool was intended to assist local law enforcement agencies in efficiently managing requests for surveillance or for triaging and prioritizing tips and leads. Work during the current year has focused on collaborating with local law enforcement organizations to identify potential applications for this tool and understand technical and practical barriers to successful implementation of using this type of analysis. In collaboration with Dr. Michael Orosz and Errol Southers (USC) we have been interacting with three local law enforcement organizations in Southern California: the Los Angeles Police Department, the Los Angeles Joint Terrorism Task Force, and the Los Angeles Airport Police Department. Conversations This research was supported by the United States Department of Homeland Security through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE.

3 with each of these organizations have identified a common challenge: analyzing information collected through Suspicious Activity Reporting programs (i.e., SARs). Initial discussions reveal that current challenges for analyzing SARs involve detecting a signal of terrorist activity in a large and unstructured dataset, if a signal exists. Currently the volume of SARs received and processes through which they are handled does not appear to overwhelm personnel available to analyze SARs. However, concerns exist that law enforcement are not able to systematically identify connections between SARs collected across the country and that future programs may lead to significantly greater numbers of SARs; thereby exacerbating challenges of both identifying potential signals of terrorism and prioritizing requests for surveillance that would be generated by the SARs. The conclusions of this work have been presented academic meetings, such as the 2009 DHS University Programs Summit in Washington DC Comparing risks in the homeland security mission space The Department of Homeland Security (DHS) is responsible for managing many types of risks to the US. Some risks are naturally occurring, like hurricanes, earthquakes, and pandemic flu outbreaks. Others are the result of deliberate acts, like terrorist attacks on chemical facilities. Some are catastrophic and likely infrequent, like the detonation of a nuclear device. Others are endemic and of moderate consequences at the event level, like smuggling drugs or illegal immigration. As DHS looks to align its budget with its strategic goals, one of the first questions being asked is, What are the greatest risks to homeland security in the US? The DHS Office of Risk Management and Analysis is beginning to answer this question for the first time by completing a National Homeland Security Risk Assessment. This assessment will raise several fundamental questions including: What do existing estimates suggest the greatest risks are? Can and should these estimates about a disparate set of risks be compared to each other? What are the appropriate characteristics for describing homeland security risks? How should value judgments be included in the assessment? This study aimed to support DHS strategic planning by providing insights into the answers for these types of questions. Specifically, the study: (a) reviewed existing estimates of risks to the US in the homeland security mission space, (b) reviewed studies of different ways of describing natural disaster, terrorism, and accident hazards, and (c) summarized the findings into comparisons of available estimates of homeland security risks in the US. These reviews led to recommendations for a concise set of characteristics that should be used to comprehensively describe hazards in the homeland security mission space and risk summary sheets for 10 representative hazards that describe the current state of knowledge about each hazard in terms of the identified characteristics. On going work is directed towards integrating these materials into processes for eliciting perceptions of policymakers and the general public about their concerns about homeland security risks. The results of this work have been disseminated at academic meetings and through consultations with DHS officials at the Office of Risk Management and Analysis Applied risk management and analysis To compliment the research projects described previously in this section, this study incorporated smaller initiatives to apply research conducted at CREATE to current policy challenges that DHS professionals are addressing. These collaborations resulted in requests for review of interim analytic products by DHS- NPPD, DHS-ORMA, and DHS-Policy. They also resulted in one peer-reviewed journal paper in the Page 2 of 12

4 Journal of Risk Research (published in collaboration with Rae Zimmerman) and a working paper that was submitted to the journal Decision Analysis (published in collaboration with Vicki Bier). Keyword 1: homeland security national risk assessment Keyword 2: intelligence analysis Keyword 3: comparative risk 2. Research Accomplishments 2.1. Applying risk analysis to support intelligence analysis This project has focused on applying a prototype decision support tool to help local law enforcement agencies analyze information contained in suspicious activity reports and prioritize resulting requests for surveillance that stem from these leads. By framing this problem, this research has identified priorities for future research to develop tools to support local law enforcement surveillance and intelligence efforts. The prototype decision support tool frames the prioritization problem that law enforcement agencies face as a single decision (i.e., whether or not to assign surveillance to a request) for which outcomes are determined by four events (i.e. whether or not a crime exists, whether or not it is disrupted, whether or not an arrest is made, and whether or not a conviction is made); see Figure 1. Conviction Arrest There is an underlying crime Crime disrupted No Arrest Arrest No Conviction Conviction Assign Crime No Conviction intel asset occurs There is no underlying crime No Arrest Arrest Conviction There is an underlying crime Crime disrupted No Arrest Arrest No Conviction Conviction Don't assign Crime No conviction intel asset occurs There is no underlying crime No Arrest Figure 1 Decision tree for whether or not to assign intelligence or surveillance assets to a request The value of surveillance is based on changing the probabilities of each of the latter three of these events occurring, and thus the decision problem can be framed as an optimization of the objective function shown below: Page 3 of 12

5 Increased benefit from increased likelihood of arrest Increased conviction benefit due to increased arrest Increased conviction benefit from conviction itself Decreased damage due to disruption of crime In this function, ρ is the baseline probability, δ is the increased probability when asset is assigned, I is an identity function =1 when asset is assigned and 0 when asset is not assigned, and A,C, and D are the benefits associated with alternate resolutions of the events shown in Figure 1. In collaboration with Dr. Michael Orosz and Erroll Southers, work during the year involved working with three local law enforcement agencies (the Los Angeles Police Department, the Los Angeles Joint Terrorism Task Force, and the Los Angeles Airport Police) to identify challenges for which this prototype tool could be applied. These discussions revealed that these law enforcement agencies are not currently constrained by the surveillance resources available to follow leads that they generate based on review of suspicious activity reports. Instead the greatest challenges revolve around how to identify the signal of a potential terrorist threat within an unstructured data set collected across several jurisdictions. This problem, has become the focus of related CREATE work on intelligence analysis being conducted by Dr. Orosz Comparing risks in the homeland security mission space The terrorism, natural disaster, and accident hazards that DHS is responsible for addressing affect communities in a variety of ways. A first step in managing this broad set of risks is to better understand how to compare them and, having done so, to understand what drives the public s concerns about the risks the hazards pose. Over the past year, this study extended empirically validated methods for comparative risk assessment to applications in the domain of homeland security policy 1. These methods are based on the principal that risk management policy should be grounded in a comprehensive assessment of what the social, natural, and physical scientific communities can tell the public about hazards and should reflect individual preferences about managing these risks, which are captured using appropriate methods of social science. Meeting these goals required first completing two research accomplishments: determining what characteristics to use when comprehensively describing homeland security hazards and reviewing existing studies to describe hazards in terms of these characteristics. Determining characteristics to describe terrorism, accident, and natural disaster hazards 1 See the following studies published in peer review journal articles: Willis, H. H., J. MacDonald, R. Shih, S. Geschwind, S. Olmstead, J. Hu, A. E. Curtright, G. Cecchine, and M. Moore (2010). Prioritizing Environmental Health Risks in the UAE. Risk Analysis, 20(12), Willis, H. H., M. L. DeKay, B. Fischhoff, P. S. Fischbeck, H. K. Florig, M. G. Morgan (2004). Ecological risk ranking: Evaluation of a method for improving the quality of public participation in environmental decision making. Risk Analysis, 24, Page 4 of 12

6 The notion that individual s concerns about risks are based on informed judgments of multiple factors is supported by a well-established body of empirical research 2. The challenge this presents is identifying a set of characteristics that is comprehensive yet concise. The set of characteristics must be complete enough to allow any person to express their concerns. At the same time, it must be concise enough so that the amount of information provided does not overwhelm individuals abilities to make informed judgments. With these goals in mind, we have selected a set of 14 risk attributes to describe homeland security disasters (see Table 1). These 14 attributes were selected based on a survey of literature that describes risks from natural disaster, accidents, and terrorism and factors that affect individual perceptions of these risks. These studies reveal characteristics of risks related to three dimensions of risks: public health and safety, economic consequences, environmental and social consequences, and dread and uncertainty. Overall, the literature review covered more than 90 studies and resulted in more than 50 potential different characteristics that could be used. The ultimate set of characteristics was selected so that each of the larger set of characteristics could be described using them. This required selecting six 4 measures describing health and safety consequences, two 2 describing economic consequences, and three 3 describing other environmental and social concerns. Additionally, the set includes 5 characteristics of hazards that risk perception research has shown to influence how individuals perceive risks. Table 1 Characteristics for describing risks from terrorism, natural disasters, and accidents Public Health and Safety Average number of deaths per year Greatest number of casualties in a single episode More severe injuries/illnesses Less severe injuries/illnesses Other Economic, Environmental and Social Consequences Average economic damages per year Greatest economic damages in a single episode Average environmental damage per year Average households displaced per year Expected disruption of government operations Aspects of Dread and Uncertainty Cause of event (Natural or human-induced) Ability of individual to control their exposure Time between exposure and health effect Quality of scientific understanding Combined uncertainty in health and economic consequences 2 For example, see the following studies published in peer review journal articles: McDaniels, T., Axelrod, L. J., & Slovic, P. (1995). Characterizing perception of ecological risk. Risk Analysis, 15, Slovic, P., Fischhoff, B., & Lichtenstein, S. (1985). Characterizing perceived risk. In R. W. Kates, C. Hohenemser, & J. X. Kasperson (Eds.), Perilous progress: Managing the hazards of technology (pp ). Boulder, CO: Westview Press Willis, H. H., M. L. DeKay, B. Fischhoff, M. G. Morgan (2005). Aggregate- and individual-level analysis of risk perception data. Risk Analysis, 25, Page 5 of 12

7 Reviewing existing studies of hazard risk To satisfy the objective that risk management be informed by the best available scientific knowledge of hazards, the study developed risk summary sheets for ten hazards (See Table 2). These hazards were selected to be representative of the types of events that DHS is responsible for managing. They stem from both natural and human-induced causes. They produce different combinations of economic, health and safety, and environmental outcomes. They also span the range of well understood and newly recognized hazards. By describing current knowledge about these hazards, this study aimed to demonstrate the feasibility of producing concise summaries available information about risks that can be used as a foundation for informed discussions about risk management priorities. Table 2 Hazards described by risk summary sheets Natural Disasters Terrorism Accidents Earthquakes Hurricanes Pandemic influenza Tornados Nuclear detonation Anthrax attacks IED attacks Cyber attacks Oil spills Industrial gas release The format for these risk summary sheets followed experience from prior comparative risk assessment studies 3. Consistent with the approaches used in these studies, the risk summary sheets answer four questions, in order: 1. What is included within the scope of the hazard being described? 2. What is known about the causes and consequences of the hazard in the U.S.? 3. What is known about exposures to the hazard in the U.S.? 4. What has already been done to reduce or control risks from the hazard in the U.S.? These materials will form the basis of continued research at CREATE planned for fiscal year 2011 to better understand how different groups perceive risks from these types of hazards, what factors most influence their concerns, and what implications such information would have for national homeland security risk management Applied risk management and analysis The applied risk management activities conducted during the past year led to two research accomplishments. The first was in collaboration with Rae Zimmerman and colleagues at New York University. This work applied statistical analysis of geographic information that Dr. Zimmerman and her colleagues had collected to develop a measure of community vulnerability. The measure related to the amount of critical infrastructure within a community and how close the community was to other critical infrastructure. This measure could be potentially relevant to policymakers allocating funding for emergency preparedness because the requirements for emergency response capability depend both upon risks within a region s jurisdiction as well as mutual-aid agreements that have been made with other regions. In general, regions in close proximity to infrastructure would benefit more from resources to 3 Willis, H. H., M. L. DeKay, B. Fischhoff, P. S. Fischbeck, H. K. Florig, M. G. Morgan (2004). Ecological risk ranking: Evaluation of a method for improving the quality of public participation in environmental decision making. Risk Analysis, 24, Page 6 of 12

8 improve preparedness because there is a greater potential for an event requiring emergency response to occur if there are more facilities at which such events could occur. The resulting study described a methodology for measuring a region s exposure to infrastructure-related risks that captures both a community s concentration of infrastructure systems and its proximity to surrounding infrastructure systems. These measures are based on smoothing-based nonparametric probability density estimators, which are then used to estimate the probability of all of the infrastructure occurring within any specified distance of critical infrastructure in a county. The set of critical infrastructure used in the paper to illustrate the use of this methodology consists of facilities identified as vulnerable through the California Buffer Zone Protection Program. The results are forthcoming in the Journal of Risk Research and show that the methodology provides information about patterns of critical infrastructure in regions that is relevant for decisions about how to allocate terrorism security and emergency preparedness resources. The second accomplishment was in collaboration with Vicki Bier and Nara Paoprasert at the University of Wisconsin. This study focused on understanding the role of nuclear detection technologies in deterring nuclear terrorism. By applying a model developed by Paoprasert and Bier, we were able to provide analytic insights into the value of increased investment in nuclear detection programs like the Secure Freight Initiative. The preliminary results suggest that unless the defender imposes high retaliation costs on the attacker, 100% inspection is likely to be needed, and deterrence with partial inspection may not be achievable in practice even though it is possible in theory. On the other hand, when the defender can credibly threaten the attacker with costly retaliation, partial inspection may be sufficient to deter nuclear smuggling attempts. Thus, for policy debates about how to prevent nuclear terrorism, consideration of the diplomatic stance on retaliation is as important as, or maybe even more important than, debate about the optimal percentage of containers to inspect. The study has been submitted to a peer-reviewed journal for review. 3. Applied Relevance By its nature, the work conducted in these studies was conceived of and carried out in a manner to enhance the applied relevance of study results. In each case, research was conducted in collaboration with homeland security professionals and results of interim or final analysis were shared through both formal and informal collaborations with those collaborators. Specific examples that emphasize the applied relevance can be drawn from each component of this effort Applying risk analysis to support intelligence analysis Intelligence analysis uses qualitative and quantitative reasoning tools to translate what is known about an adversary s intentions and capabilities into actionable advice on how respond in anticipation of the adversaries actions. As part of a Year 4 CREATE study of the value of information of intelligence analysis, we reviewed what types of intelligence analysis the private sector receives and seeks and for what purposes. At the same time, local law enforcement is conducting parallel searches for and reviews of intelligence products to anticipate and disrupt terrorist attacks. One challenge this presents is how both the private sector and local law enforcement decide when and how to allocate intelligence assets. A specific example of such a decision is that of how law enforcement decides how many officers to assign to investigate terrorist related cases, gang related cases, drug related cases, or other crimes. In a Year 5 study, this problem served as the basis for developing tools for helping local law enforcement best utilize resources to collect intelligence. This study developed a strawman resource allocation tool and began working with law enforcement organizations to apply the tool. In Year 6, we continued to refine and apply the tool with law enforcement organizations. Page 7 of 12

9 As described in Section 2 of this report, this work involved consultation with local law enforcement organizations in the Los Angeles metropolitan area that are responsible for identifying, preventing, and persecuting potential crimes of terrorism. These discussions suggest that at the present time, law enforcement intelligence operations are constrained more by the ability to detect the signal of terrorism than by having enough resources available to follow potential terrorism leads that are identified. However, anticipated development of new approaches to collect and analyze suspicious activity reports could result in greater numbers of leads, at which time resource allocation questions will likely become more salient Comparing risks in the homeland security mission space The Department of Homeland Security (DHS) is responsible for managing many types of risks to the US. Some risks are naturally occurring, like hurricanes, earthquakes, and pandemic flu outbreaks. Others are the result of deliberate acts, like terrorist attacks on chemical facilities. Some are catastrophic and likely infrequent, like the detonation of a nuclear device. Others are endemic and of moderate consequences at the event level, like smuggling drugs or illegal immigration. As DHS looks to align its budget with its strategic goals, one of the first questions being asked is, What are the greatest risks to homeland security in the US? During the Quadrennial Homeland Security Review, the DHS Office of Risk Management and Analysis began to answer this question for the first time by providing goals and a framework for completing a National Homeland Security Risk Assessment. This assessment would raise several fundamental questions including: What do existing estimates suggest the greatest risks are? Can and should these estimates about a disparate set of risks be compared to each other? What are the appropriate characteristics for describing homeland security risks? How should value judgments be included in the assessment? The results of this study help to answer questions like this, which remain relevant for ongoing DHS strategic planning efforts. To increase the applied relevance of this project to DHS, the study was conducted in consultations with analysts from DHS Office of Risk Management and Analysis. Consultations occurred through requests to the study team for formal comments on interim results of QHSR study teams on the national homeland security risk assessment framework and on the strategic goals for domestic security. Based on these types of discussions, the results are anticipated as being useful to this office as it completes the National Homeland Security Risk Assessment and continues to improve the Risk Assessment Process for Investment Decisionmaking Applied risk management and analysis The applied risk management tasks conducted as part of this effort were largely generated in response to specific policy questions that resulted from discussions with homeland security professionals at the federal, state, and local levels. For example, research with New York University on developing a measure of a communities proximity to concentrations of critical infrastructure stemmed from CREATE discussions with the California Office of Homeland Security through assistance provided in allocating homeland security grants across counties in California. Similarly, collaboration with the University of Wisconsin was motivated by and contributed to consultations with analysts and leadership at Customs and Border Protection, DHS Headquarters, the Domestic Nuclear Detection Office, and the DHS Office of Policy about how to assess the benefits of nuclear detection programs and technologies. Some of these consultations were conducted as part of Page 8 of 12

10 service on the National Academies of Sciences Committee on Advanced Spectroscopic Portals. Others were conducted as part of informal briefings of interim and final study results. 4. Collaborative Projects This project involved collaboration both across CREATE and with homeland security professionals at federal, state and local levels. Collaborations within CREATE included: Appling risk analysis to intelligence analysis: Working with Erroll Southers and Dr. Michael Orosz, collaborations led to a comprehensive assessment of how risk analysis tools can assist local law enforcement in analyzing suspicious activity reports and prioritizing requests for surveillance that result from this analysis. This work was presented at the 2010 University Programs Summit in Washington, DC in collaboration with Commander Joan McNamara (retired) of the Los Angeles Police Department. Incorporating risk perceptions into comparative risk analysis: Through work with Dr. Adam Rose and Dr. Bill Burns, this study incorporated implications of CREATE research about economic resilience and risk perceptions into identifying the characteristics that must be addressed when describing terrorism, accident, and natural disaster hazards. The results of this collaboration were presented at the CREATE terrorism risk perceptions workshop organized by Dr. Burns in Applying risk analysis to preventing nuclear terrorism: Through collaboration with Vicki Bier and her graduate student Nara Paoprasert, the applied studies aspect of this research developed insights on the value of nuclear detection technologies for preventing nuclear terrorism in the U.S. As described in Sections 2.3 and 3.3, this research concluded that the value of nuclear detection technologies was strongly dependent on policy decisions made with respect to threats of retaliation against would-be or successful nuclear terrorist. These findings were briefed to analysts at the DHS Office of Policy and Domestic Nuclear Detection Office during discussions of the National Strategy for Cargo Security, described in more detail below. Results were also documented in a working paper that has been submitted to a peer review journal for publication. Allocating Emergency Preparedness Resources: Collaboration with Rae Zimmerman and colleagues at New York University leveraged on-going work on GIS-based statistical analysis and prior analytic support to the CA Office of Homeland Security to advise on risk-based resource allocation. The result of this collaboration was the description and demonstration of a new metric for describing a community s vulnerability to accidents, natural disasters, and terrorism that could affect critical infrastructure systems that is based on the density and proximity of infrastructure facilities to a county. The work has been accepted for publication in the Journal of Risk Research. Collaborations with federal, state and local homeland security professionals have included: Applying risk analysis evaluating emergency preparedness: Through a contract to CREATE from the FEMA Preparedness Directorate, the results of current and prior years CREATE analysis were extended to describe and demonstrate an approach to assessing how prepared a community was to respond to future disasters and prioritizing options for improving a the community s emergency response capabilities. The work was published as a RAND report and as a working paper that has been submitted to a peer reviewed journal for publication 4. This study led to 4 Jackson, B.A., K. Sullivan Faith, H. H. Willis (2010). Evaluating the Reliability of Emergency Response Systems for Large-Scale Incident Operations. MG-994-FEMA, RAND Corporation, Santa Monica, CA. Page 9 of 12

11 follow-on applications of the method for the Department of Health and Human Services (DHHS), Office of the Assistant Secretary for Preparedness and Response to evaluate a community s readiness to distribute medical countermeasures following a bioterrorism incident. In addition, the work has been briefed to local emergency response planners in Los Angeles, CA and Tampa, FL to understand the potential value of future applications of the methods. The Contracting Office Technical Representative for work with FEMA was Ms. Laureen Daly and for work with DHHS was Mr. James Mulikin. Support to the Quadrennial Homeland Security Review: Current and prior year CREATE research created an understanding of homeland security missions that was relevant to studies DHS conducted as part of the Quadrennial Homeland Security Review. Specifically, results of prior work on allocating resources for critical infrastructure protection and on applying risk analysis to intelligence analysis enable support of the NPPD-led study on national objectives for domestic security led by Mr. Brandon Wales. Similarly, ongoing work on comparing risks in the homeland security mission space enabled support to the RMA-led study on completing a Homeland Security National Risk Assessment led by Mr. Robert Kolasky. This support was provided through comment on interim study reports and participation in study team meetings. Support to development of the National Strategy for Cargo Security: As described already in this section, collaboration with Dr. Vicki Bier directly supported parallel efforts at DHS to develop the strategy for securing the global containerized supply chain. This effort was led for the DHS Office of Policy by Ms. Christa Brzozowski. Collaboration occurred through participation in stakeholder meetings, briefings of CREATE study results to the intra-agency study team led by Ms. Brzozowski, and formal comment on interim reports produced by this study team. Using risk analysis to support law enforcement intelligence and surveillance efforts: As described already in this section, research collaborations to link intelligence and risk analysis involved direct collaboration with the Los Angeles Police Department. 5. Research Products Research Products (Please detail below) # 5a # of peer-reviewed journal reports published 1 5a # of peer-reviewed journal reports accepted for publication 1 5a # of non-peer reviewed publications and reports 2 5a # of scholarly journal citations of published reports? 5b # of scholarly presentations (conferences, workshops, seminars) 5 5b # of outreach presentations (non-technical groups, general public) 2 5c # of products delivered to DHS, other Federal agencies, or State/Local 0 5c # of patents filed 0 5c # of patents issued 0 5c # of products in commercialization pipeline (products not yet to market) 0 5c # of products introduced to market Publications and Reports Jackson, B.A., K.S. Faith, H.H. Willis (2010) Are We Prepared? Using Reliability Analysis to Evaluate Emergency Response System. Submitted to Journal of Contingencies and Crisis Management. Page 10 of 12

12 Simonoff, J.S., C. E. Restrepo, R. Zimmerman, Z. S. Naphtali, and H. H. Willis (forthcoming). Resource Allocation and Infrastructure Density around Priority Sites. Journal of Risk Research. Jackson, B.A., K. Sullivan Faith, H. H. Willis (2010). Evaluating the Reliability of Emergency Response Systems for Large-Scale Incident Operations. MG-994-FEMA, RAND Corporation, Santa Monica, CA. Jackson, B.A., K.S. Faith, H.H. Willis (2010) Are We Prepared? Using Reliability Analysis to Evaluate Emergency Response System. Submitted to Journal of Contingencies and Crisis Management. Haphuriwat, N., V. Bier, H. H. Willis (2010). Deterring the Smuggling of Nuclear Weapons in Container Freight through Detection and Retaliation. Submitted to Decision Analysis Presentations Willis, H. H. (2010). The Science of Listening: How Understanding Risk Perceptions Can Lead to Better Policy, Paper presented at the Society for Risk Analysis Annual Meeting, Salt Lake City, Utah, December 8, Willis, H.H. (2010). What Have We Learned About Assessing the Benefits of Security?, Plenary Address to the Annual Meeting of the Security Analysis and Risk Management Association, Arlington, VA, October 5, Willis, H. H. (2010). The Science of Listening: How Understanding Risk Perceptions Can Lead to Better Policy, Paper presented at the USC CREATE Workshop on Risk Perception and Risk- Related Behaviors: Anticipating and Responding to Crisis, Los Angeles, CA, March 6, Willis, H.H. (2010). Managing the Unimaginable: Lessons from Terrorism Security and Disaster Response Planning, Paper presented at the International Disaster Risk Conference, Davos, Switzerland, June 2, Willis, H. H. (2010). Allocating Law Enforcement Resources for Information Gathering. Paper presented at the DHS University Programs Summit, Washington, DC, March 11, Willis, H.H., (2009). Issues and Opportunities in Homeland Security Research, Paper presented at the INFORMS Annual Meeting, San Diego, CA, October 12, Willis, H.H., J. E. Predd (2009). Bandits for Border Security, Paper presented at the INFORMS Annual Meeting, San Diego, CA, October 12, Models, Databases, and Software Tools and Products This research has not resulted in prototype models, databases, or software tools. 6. Education and Outreach Products Education and Outreach Initiatives (Please detail below) # # of students supported (funded by CREATE) 1 # of students involved (funded by CREATE + any other programs) 2 # of students graduated 0 Page 11 of 12

13 # of contacts with DHS, other Federal agencies, or State/Local (committees) 6 # of existing courses modified with new material 0 # of new courses developed 1 # of new certificate programs developed 0 # of new degree programs developed 0 Education Homeland Security Risk Assessment and Management, Course for DHS Analysts Delivered as part of the RAND Homeland Security Symposium, November 2009 Student supported by CREATE funding: Russell Lundberg, Pardee RAND Graduate School Student supported by other CREATE-related funding: Kay Sullivan, Pardee RAND Graduate School Outreach DHS Policy, Support to development of the National Cargo Security Strategy Related additional consultations as part of this: - DNDO, Director, Consultation on Approaches to Analyzing GNDA - Member, NAS Committee on Advanced Spectroscopic Portals DHS Office of Risk Management and Analysis, Support to QHSR study team on conducting a Homeland Security National Risk Assessment DHS NPPD, Support to QHSR Domestic Security Study Team Los Angeles Police Department, Consulting on applying risk analysis to analyze suspicious activity reports. Page 12 of 12