The Chemical, Biological, Radiological/Nuclear Explosive (CBRNE) Consolidated Risk Assessment (CRA) Rating Tool Guide

Transcription

1 The Chemical, Biological, Radiological/Nuclear Explosive (CBRNE) Consolidated Risk Assessment (CRA) Rating Tool Guide Shaye Friesen DRDC Centre for Security Science Defence Research and Development Canada Reference Document DRDC-RDDC-2017-D001 January 2017

2 Template in use: Normal.dotm Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2017 Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2017

3 Abstract The aim of this document is to provide a basic user s guide of the Chemical, Biological, Radiological/Nuclear Explosive (CBRNE) Consolidated Risk Assessment (CRA) process and rating tool. Defence Research and Development Canada s Centre for Security Science (DRDC CSS) developed the CRA rating tool to aid risk analysts in evaluating the risks posed by a broad range of CBRNE hazard scenarios. Additionally, the background to the CRA tool and scoring approach is included for reference purposes, so that end users can understand how risk ratings are generated and calculated. The CRA tool is a Microsoft Excel file and is available from DRDC CSS. DRDC-RDDC-2017-D001 i

4 Résumé L objectif du présent document est d offrir un guide d utilisation de base pour l outil de traitement et de cotation de l évaluation consolidée des risques (ÉCR) relatifs aux agents chimiques, biologiques, radiologiques, nucléaires et explosifs (CBRNE). (RDDC CSS) a élaboré l outil de cotation de l ÉCR pour aider les analystes à évaluer les risques courus dans divers scénarios mettant en jeu des agents CBRNE. Le document comprend également, à titre de référence, des renseignements généraux sur l outil ÉCR et l approche de cotation afin que les utilisateurs finaux puissent mieux comprendre comment générer et calculer les cotes de risques. L outil ÉCR est un fichier Microsoft Excel disponible auprès de RDDC CSS. ii DRDC-RDDC-2017-D001

5 Table of Contents Abstract i Résumé ii Table of Contents iii List of Figures iv 1 Introduction Aim Background Consolidated Risk Assessment (CRA) Process Overview The CRA Formula and Rating Tool Characteristic Scenarios Rating Impacts Rating Relative Technical Feasibility (RTF) Confidence Intervals Risk Plots Conclusion References Annex A Rating Criteria DRDC-RDDC-2017-D001 iii

6 List of Figures Figure 1: CRA process and methodology Figure 2: Example Worksheets from CRA Tool Figure 3: Example Worksheet from CRA Tool Figure 4: Screenshot of dropdown menu Figure 5: Screenshot of the RTF ratings note, the final score is derived by selecting the lowest sub-category rating Figure 6: Screen shot of confidence levels Figure 7: Example of Vulnerability Scores (with Error Bars) displayed within the CRA Tool Figure A.1: Impact Grid Figure A.2: RTF Grid iv DRDC-RDDC-2017-D001

7 1 Introduction 1.1 Aim The aim of this document is to provide a basic user s guide of the Chemical, Biological, Radiological/Nuclear Explosive (CBRNE) Consolidated Risk Assessment (CRA) process and rating tool. Defence Research and Development Canada s Centre for Security Science (DRDC CSS) developed the CRA rating tool to aid risk analysts in evaluating the risks posed by a broad range of CBRNE hazard scenarios. Additionally, the background to the CRA tool and scoring approach is included for reference purposes, so that end users can understand how risk ratings are generated and calculated. The CRA tool is a Microsoft Excel file and is available from DRDC CSS. DRDC-RDDC-2017-D001 1

8 2 Background The CBRNE CRA was originally designed in 2002 by the Canadian Security Intelligence Service (CSIS) with support from the Department of National Defence Intelligence (DND), and the Canadian Food Inspection Agency (CFIA). The methodology is based on the RAND Corporation Delphi Method a structured process developed for technology forecasting that collects and distils knowledge from a group of experts. The methodology has evolved over time, with refinements to the criteria set, scoring methods, the number of threat scenarios assessed, and the inclusion of a process to capture the science and technology (S&T), capability and response gaps. The CRA was augmented in 2004 to include explosives threat scenarios. The CRA is a comprehensive, bottom-up process that engages experts from federal S&T, operational, law enforcement, and intelligence communities. The assessment has been conducted annually since 2002, normally through two-day workshops, where a representative set of scenarios are rated for technical feasibility, impact, vulnerability, likelihood and risk. The ratings are achieved through consensus of various stakeholders and subject matter experts (SMEs). 2.1 CRA Process Overview The CRA process occurs in four steps (see Figure 1). In the first step, SMEs identify scenarios. In step two, SMEs evaluate the vulnerabilities by estimating the impact and relative technical feasibility (RTF) of the CBRNE hazard scenarios. In step three, the intelligence community is engaged to provide an intelligence judgement, based on current intelligence reporting, and aims to portray the history of attacks, terrorist capability and intent. In the final step, the impact and RTF combine to form a vulnerability rating, which, together with the intelligence judgement, result in a priority risk rating for each hazard scenario. CBRNE Consolidated Risk Assessment S&T - Operations - Law Enforcement - Intel - Allied Identify Scenarios Scoping, Generation, Agreement Determine Vulnerabilities Relative Technical Impact Feasibility Intelligence Judgement Assess Likelihood Assess Risk ~1 month ~3-4 months ~2 months ~2-3 months Develop characteristic scenarios that describe CBRNE events Describes how a threat/hazard could exploit a target vulnerability List of characteristic scenarios Evaluate vulnerabilities of each scenario for Relative Technical Feasibility: Access to Materials Equipment Access to Target Technical Expertise Access to Critical Information Evaluate vulnerabilities of each scenario for Impact: Human Losses Dead Injured Magnitude of Response Economic Damage ($) Assess the likelihood of each scenario by factoring in Intelligence Judgement Determine the risk for each scenario (combine Vulnerability and Intelligence Judgement) Prioritize risks Identify key S&T mitigating factors Figure 1: CRA process and methodology. 2 DRDC-RDDC-2017-D001

9 3 The CRA Formula and Rating Tool The CRA tool was created to help risk analysts define scenarios and rate them across a broad range of different criteria. The tool is built in Microsoft Excel. See Figure 2 below for a screen shot of the main page, which is displayed upon opening the tool. Figure 2: Example worksheets from CRA Tool. Within the CRA risk tool, there are a number of worksheets at the bottom to input data. These include: 1. Results: worksheet for hazard scenarios; the ratings for impact, RTF, intelligence judgement and confidence estimates are inserted here by risk analysts. 2. Impact Grid: displays the impact scoring matrix for malicious CBRNE hazard scenarios (see Annex A, Figure A.1). 3. RTF Grid: displays the RTF rating scale for the malicious CBRNE hazard scenarios (see Annex A, Figure A.2). 4. Confidence: displays the metrics for measuring the confidence intervals associated with the CBRNE hazard scenarios. 5. Plot: displays a graphical depiction of the results of the ratings, according to Vulnerability (impact and RTF) + Risk. 6. Scoring lists: displays the tables for converting the impact and RTF scores to a common scale. Dollars ($) are used as the common unit due to the widespread practice in other areas of risk assessment [1,2]. In the CRA, the equivalency of deaths to $ is set to 1 death = $10 M. DRDC-RDDC-2017-D001 3

10 The tables can be modified, if a different set of common measurement units is selected by the risk analyst. To do this, the measurement units need to be specified or adjusted in the tables, and the results will be translated in the drop down menus on the Results table (i.e., the worksheet where the scenarios and rating values are inserted). After converting all the components of the impact scores to $, resulted values are added linearly, and the logarithm (log) of the sum of the scores is taken, as shown in the equation below. (1) The RTF ratings follow a similar order of magnitude approach, to facilitate the comparison of multiple CBRNE hazard scenarios across target categories, such as people, critical infrastructure, food and consumer products, and agro system (plant and animal). The RTF tables are formulated with a baseline vignette (i.e., 1x (baseline vignette), with the remainder of the scenarios being assessed as 100x less likely than base 1, or 100x more likely than base 1, where rtf l is the associated RTF rating (5 being the maximum rating considered, and the rating scheme adopted was designed to identify the easiest hazard scenarios to execute). Thus, if Vulnerability= Impact + RTF, 1 then we follow a logarithmic scale that implies the following: If one incorporates intelligence judgement, then we define risk as: This can be used to calculate the risk, using iso-risk contours (i.e., lines of equal risk) on an Impact vs. RTF plot, or a Vulnerability vs. Intelligence Judgement plot, as shown later in this document. Note: The Results worksheet in the CRA risk tool can be unlocked to view the conversion of the impact scores in the form of VLOOKUP tables, using the scoring method described above. 3.1 Characteristic Scenarios Each worksheet can be populated with a representative list of hazard scenarios/vignettes for CBRNE events, with blank spaces provided to allow insertion of hazard-specific information. To begin, risk analysts enter the scenario code/id if available, and a general description of the characteristic scenario. The description should include the material/agent, target and effects or consequences, along with other relevant details. Generally speaking, a five year timeframe is considered for all events in the CRA process. There is no limit to the number of rows that can be added in the tool. However, the user is advised to ensure the Plots worksheet incorporates the data range where scenarios were incorporated (see Figure 3). (2) (3) 1 This is not meant to imply an algebraic addition; rather that vulnerability is plotted as a combination of impact and RTF. 4 DRDC-RDDC-2017-D001

11 Figure 3: Example worksheet from CRA Tool. Once the hazard scenarios have been inputted into the tool, it is necessary to evaluate them with respect to vulnerability and likelihood. 3.2 Rating Impacts In the tool, the impact for a hazard scenario is divided into four sub-categories, with each scored from 0 to 9: Dead Injured Magnitude of Response Economy To evaluate the impacts, choose the rating that best describes impacts using the drop-down menu. Once all of the ratings for each sub-category are entered into the CRA tool, an overall value is returned (see Figure 4). DRDC-RDDC-2017-D001 5

12 Figure 4: Screenshot of dropdown menu. In order to ensure a consistent scoring method, the impact ratings are defined in terms of an order of magnitude of the severity associated with a loss for each sub-category. This implies a constant ratio for the increase in severity of impacts from one rating to the next. The impact grid (Figure A.1) uses a logarithmic scale, with half scores for ratio increments of the square root of 10, used to increase the resolution of the assessment. 3.3 Rating RTF The CRA tool permits users to assess RFT. The RTF of each hazard scenario is assessed against five sub-criteria, consisting of the following scores from 0 to 9: Material Equipment Access to Target Technical Expertise Access to Critical Information As with the impact grid, the RTF grid (Figure A.2) follows a similar logarithmic scale. However, estimating the RTF of the hazard scenarios is slightly different, as the estimates must take into account whether mounting an attack is even technically feasible. The overall feasibility score in the CRA tool (i.e., the RTF Matrix Rating column) for a given hazard scenario is based on the concept of the highest hurdle, whereby each sub-category can be seen as a critical link on the path to a successful attack scenario. The final RTF score is determined by selecting the lowest rating across all sub-categories, thus reflecting the highest hurdle or weakest link constraint on the 6 DRDC-RDDC-2017-D001

13 feasibility of a given hazard scenario. In other words, the lower the feasibility, the higher the hurdle. This rating technique is consistent with other federal risk assessment methodology guidelines [3]. Once users are finished recording their ratings for RTF, the lowest value is entered into the RTF Matrix Rating column in the worksheet (see Figure 5). Figure 5: Screenshot of the RTF ratings note, the final score is derived by selecting the lowest sub-category rating. 3.4 Confidence Intervals The CRA tool allows users to assess the uncertainties associated with the hazard scenarios during risk assessment workshops. The degree of confidence in the ratings produced may vary, depending on the availability of data, relevant experience and existing body of knowledge. Uncertainties are estimated for the impact and RTF categories as a whole, using descriptive language ladders ranging from very high to very low. Users capture their ratings for confidence in the corresponding drop-down menus, with the results being recorded Template. The definitions for each of the confidence intervals is located in the worksheets along the bottom of the CRA tool, along with the degrees of uncertainty (e.g., ranging from % confidence) and associated values. Capturing the confidence level associated with each rating produced provides an essential piece of information that helps complete the risk assessment picture. Figure 6 presents definitions for confidence levels for each rating: DRDC-RDDC-2017-D001 7

14 Figure 6: Screen shot of confidence levels. Capturing confidence estimates as an essential part of the risk assessment process, allows users and stakeholders to visualize and interpret the results. For instance, very low confidence in the rating of a particular hazard scenario produces error bars in the Plot worksheet that is intended to convey caution in the interpretation of the hazard scenarios. Very high error bars indicate a degree of uncertainty with respect to RTF ratings, while wide error bars suggest impact uncertainty. Note: In order to view the error bars on the Plot worksheet, it is necessary to unlock the columns in the Results worksheet. 8 DRDC-RDDC-2017-D001

15 4 Risk Plots In the CRA tool, risk plots are produced by bringing forward the results of the ratings for impact and RTF for each hazard scenario. This information is then used to determine overall vulnerability and (when combined with intelligence judgement) risk ratings. For example, a hazard scenario is determined to have an impact score of 1.4 and a RTF Score of 2. The CRA tool automatically calculates the scores for impact and RTF in the following plot (see Figure 7). Figure 7: Example of Vulnerability Scores (with Error Bars) displayed within the CRA Tool. The overall risk rating for the hazard scenarios is also displayed in the Plots worksheet, adjacent to the vulnerability graph. It is calculated by combining the final vulnerability rating (impact + RTF) with the likelihood (i.e., intelligence judgement) rating. 2 Vulnerability and intelligence judgement data are plotted in the exact same way, to show which hazard scenarios posed the highest risk. Hazard scenario risk ratings can be compared against one another in a relative fashion to identify priorities threats/hazards. The CRA decision support tool allows users to quickly visualize results, compare the distribution of the vulnerabilities and risks in a given scenario set, and establish acceptable or unacceptable risk levels. Note: the labels for each of the hazard scenarios in the Plot worksheet are produced manually. 2 Note: this paper excludes a discussion of the rating factors and criteria associated with intelligence judgement, which will be documented in a separate report. DRDC-RDDC-2017-D001 9

16 5 Conclusion With the aid of the CRA tool, the impact and RTF ratings for multiple hazard scenarios were captured by the CBRNE Communities of Practice (CoP) during stakeholder engagement workshops. The CRA tool was an explicit part of the CRA process that allowed hazard scenarios to be captured in advance. It facilitated the capture of SME consensus ratings, and allowed for the results to be automatically tabulated and graphically presented back to key stakeholders, which extended the benefits of using software tools for the CRA process. The methodology and rating tool/techniques can be suitably modified to address non-malicious threats/hazard risk assessments. If needed, additional rating factors can be added to contribute to refining the overall approach, depending on information requirements and purpose of the assessment. 10 DRDC-RDDC-2017-D001

17 References [1] Waller, D. (2016). Proposal to modify the Consolidated Risk Assessment (CRA), Defence Research and Development Canada, Scientific Report, DRDC-RDDC-2016-R133. p. 8. [2] Lin, X. G. and Jarrett, R. (2009). A Practical Approach to Quantitative Risk Assessment, Division of Mathematical and Information Sciences, Canberra & Melbourne, Risk Conference, Wellington, [3] Public Safety. All Hazards Risk Assessment Methodology Guidelines , (Online): (Access date: 01 Nov 2016). DRDC-RDDC-2017-D001 11

18 This page intentionally left blank. 12 DRDC-RDDC-2017-D001

19 Annex A Rating Criteria Figure A.1: Impact Grid. DRDC-RDDC-2017-D001 13

20 Figure A.2: RTF Grid. 14 DRDC-RDDC-2017-D001

21 DOCUMENT CONTROL DATA (Security markings for the title, abstract and indexing annotation must be entered when the document is Classified or Designated) 1. ORIGINATOR (The name and address of the organization preparing the document. Organizations for whom the document was prepared, e.g., Centre sponsoring a contractor s report, or tasking agency, are entered in Section 8.) 2a. SECURITY MARKING (Overall security marking of the document including special supplemental markings if applicable.) DRDC Centre for Security Science Defence Research and Development Canada 222 Nepean St., 11th Floor Ottawa, Ontario K1A 0K2 Canada UNCLASSIFIED 2b. CONTROLLED GOODS (NON-CONTROLLED GOODS) DMC A REVIEW: GCEC DECEMBER TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.) The Chemical, Biological, Radiological/Nuclear Explosive Consolidated Risk Assessment Rating Tool Guide 4. AUTHORS (last name, followed by initials ranks, titles, etc., not to be used) Friesen, S. 5. DATE OF PUBLICATION (Month and year of publication of document.) January a. NO. OF PAGES (Total containing information, including Annexes, Appendices, etc.) 18 6b. NO. OF REFS (Total cited in document.) 3 7. DESCRIPTIVE NOTES (The category of the document, e.g., technical report, technical note or memorandum. If appropriate, enter the type of report, e.g., interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.) Reference Document 8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development include address.) DRDC Centre for Security Science Defence Research and Development Canada 222 Nepean St., 11th Floor Ottawa, Ontario K1A 0K2 Canada 9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.) 9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.) 10a. ORIGINATOR S DOCUMENT NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.) 10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.) DRDC-RDDC-2017-D DOCUMENT AVAILABILITY (Any limitations on further dissemination of the document, other than those imposed by security classification.) Unlimited 12. DOCUMENT ANNOUNCEMENT (Any limitation to the bibliographic announcement of this document. This will normally correspond to the Document Availability (11). However, where further distribution (beyond the audience specified in (11) is possible, a wider announcement audience may be selected.)) Unlimited

22 13. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual.) The aim of this document is to provide a basic user s guide of the Chemical, Biological, Radiological/Nuclear Explosive (CBRNE) Consolidated Risk Assessment (CRA) process and rating tool. Defence Research and Development Canada s Centre for Security Science (DRDC CSS) developed the CRA rating tool to aid risk analysts in evaluating the risks posed by a broad range of CBRNE hazard scenarios. Additionally, the background to the CRA tool and scoring approach is included for reference purposes, so that end users can understand how risk ratings are generated and calculated. The CRA tool is a Microsoft Excel file and is available from DRDC CSS L objectif du présent document est d offrir un guide d utilisation de base pour l outil de traitement et de cotation de l évaluation consolidée des risques (ÉCR) relatifs aux agents chimiques, biologiques, radiologiques, nucléaires et explosifs (CBRNE). (RDDC CSS) a élaboré l outil de cotation de l ÉCR pour aider les analystes à évaluer les risques courus dans divers scénarios mettant en jeu des agents CBRNE. Le document comprend également, à titre de référence, des renseignements généraux sur l outil ÉCR et l approche de cotation afin que les utilisateurs finaux puissent mieux comprendre comment générer et calculer les cotes de risques. L outil ÉCR est un fichier Microsoft Excel disponible auprès de RDDC CSS. 14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g., Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.) Chemical, Biological, Radiological/Nuclear Explosive (CBRNE); Risk Assessment; Methodology; Rating Tool.