The vignette, task, requirement, and option (VITRO) analyses approach to operational concept development

Transcription

1 CAN UNCLASSIFIED The vignette, task, requirement, and otion (VITRO) analyses aroach to oerational concet develoment atrick W. Dooley, Yvan Gauthier DRDC Centre for Oerational Research and Analysis Journal of Alied Oerational Research Vol. 10, No. 1, Date of ublication from Ext ublisher: January 2018 Defence Research and Develoment Canada External Literature () DRDC-RDDC January 2018 CAN UNCLASSIFIED

2 CAN UNCLASSIFIED IMORTANT INFORMATIVE STATEMENTS Disclaimer: This document is not ublished by the Editorial Office of Defence Research and Develoment Canada, an agency of the Deartment of National Defence of Canada, but is to be catalogued in the Canadian Defence Information System (CANDIS), the national reository for Defence S&T documents. Her Majesty the Queen in Right of Canada (Deartment of National Defence) makes no reresentations or warranties, exressed or imlied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, comleteness, currency or usefulness of any information, roduct, rocess or material included in this document. Nothing in this document should be interreted as an endorsement for the secific use of any tool, technique or rocess examined in it. Any reliance on, or use of, any information, roduct, rocess or material included in this document is at the sole risk of the erson so using it or relying on it. Canada does not assume any liability in resect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, roduct, rocess or material included in this document. This document was reviewed for Controlled Goods by Defence Research and Develoment Canada (DRDC) using the Schedule to the Defence roduction Act. Temlate in use: (2012) CR EL1 Advanced Temlate_EN _28-V04_WW.dotm Her Majesty the Queen in Right of Canada (Deartment of National Defence), 2018 Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2018 CAN UNCLASSIFIED

3 The vignette, task, requirement, and otion (VITRO) analyses aroach to oerational concet develoment atrick W. Dooley 1, and Yvan Gauthier 1 1 Centre for Oerational Research & Analysis, Defence Research and Develoment Canada, National Defence Headquarters, Ottawa, ON, Canada Keywords: Analysis Biological Caability Chemical Disaster resonse Force structure Homeland security Military oerations National defence Nuclear lanning Radiological Risk assessment Abstract We describe a systematic, generic, and transarent lanning aroach to oerational concet develoment by defence and security organizations. The aroach is based on sequential Vignette, Task, Requirement, and Otion (VITRO) analyses. The vignette analysis yields likelihood estimates for otential events or contingencies of interest, whereas the task analysis gauges robabilities for tasks that a military or security force may be requested to erform in resonse to such events. Next, the requirement analysis roduces estimates of what the force would need to erform such tasks adequately, across the vignette set. Finally, the otion analysis aggregates the three foregoing analyses results, to assess otential force ackages erformance versus a range of rosective tasks. The VITRO aroach was devised to aggregate rigorously thousands of inuts and yield results that are meaningful for military decision makers and readily enable them to weigh ossible trade-offs between resourcing otions and consequent force ackage caabilities. We originally used the VITRO aroach to suort the develoment of a concet of oerations for domestic Chemical, Biological, Radiological, or Nuclear (CBRN) resonse for the Canadian Armed Forces. This aers aims to describe the aroach in a concise and ublicly accessible manner, to enable its broader dissemination amongst OR ractitioners and alication to other lanning areas (military or civil). We use a domestic CBRN event resonse context to frame this aer s notional examles. ublished online 05 January 2018 by ORLab Analytics Inc. All rights reserved. Introduction Defence and security organizations must effectively anticiate and reare for the environment they in which will oerate. Doing so, however, can be challenging. Effective lanning requires sound rojections concerning the sectrum of otential missions and tasks the organization may have to erform, their robabilities of occurrence, oerational conditions, resource requirements, and many other factors. Such rojections can be difficult to formulate and aly. For instance, future oerational conditions may be highly uncertain, historically rare or unrecedented. The requisite lanning information, if it exists, may be diverse and may reside outside the organization. It may also be subjective or ambiguously communicated during lanning, thereby inhibiting a common understanding among stakeholders. An additional difficulty is that detailed lanning information generally needs some aggregation before being resented to senior-level decision makers, but this aggregation is not always done logically and transarently. This aer reresents the first ublicly accessible descrition of the Vignette, Task, Requirement, and Otion (VITRO) analyses aroach to the develoment of military concets of oerations. In essence, the aroach addresses the four following fundamental lanning questions. 1. For what tyes of scenarios or events should a military or security force lan? 2. For such scenarios or events, what tasks would the military or security force likely have to erform? 3. To erform such tasks adequately, what would the military or security force need? 4. Which force ackages would erform well, given the sectrum of ossible scenarios or events? The aroach was originally develoed to inform the rearation of a Canadian Armed Forces (CAF) concet of oerations (Canadian Joint Oerations Command, 2013) outlining how the CAF would suort civil authorities resonse to a chemical, biological, radiological, or nuclear (CBRN) event. In Canada, armed forces should not, tyically, be the first resonders to Corresondence: atrick Dooley, Centre for Oerational Research & Analysis, Defence Research and Develoment Canada, National Defence Headquarters, 101 Colonel By Dr, Ottawa, ON, K1A 0K2, Canada atrick.dooley2@forces.gc.ca Coyright Her Majesty the Queen in Right of Canada, as reresented by the Minister of National Defence (2018).

4 26 CBRN events. However, the CAF may be asked to lay a suorting role to, and only at the request of, federal, rovincial/ territorial or municial lead civil authorities and reare contingency lans accordingly (Deartment of National Defence, 2010). A CBRN event is meant here to encomass deliberate and accidental releases of CBRN agents, as well as releases of toxic industrial (i.e., hazardous) materials, a definition consistent with North Atlantic Treaty Organization terminology (NATO Standardization Agency, 2006). The VITRO aroach was devised and alied to aggregate rigorously thousands of inuts to a multifaceted real-world decision roblem and yield results that are meaningful for military decision makers and stakeholders. Due to large differences in military forces sizes, comositions, mandates, doctrines and legislation, the CAF s lanning aroach for resonding to CBRN events and the resulting force ackages are quite different than those used in the United States (see Heffelfinger, Tuckett & Ryan, 2013) and elsewhere. Furthermore, when the CAF initiated a review of its concet of oerations for CBRN event resonse in 2011, there was no oerational research (OR) aroach known to the authors that was directly alicable to augment the CAF s traditional lanning aroach, which was essentially based on existing doctrine and military exert judgment. Although over 100 OR aers had been ublished on disaster oerations management since the 1980s (see review by Altay & Green, 2006), only 24% of that research focused on the resonse hase. The majority of the literature focused on risk mitigation and rearedness, esecially in the CBRN realm (see for examle research by Linkov et al. (2012), Leung & Verga (2007), and Kollek (2003)). Moreover, although there was no shortage of aers describing otimization and simulation models for analyzing secific, tactical asects of disaster resonse such as transortation (e.g., Barbarosoǧlu & Arda, 2004) or the re-ositioning of emergency sulies (e.g., Rawls & Turnquist, 2010), there was no obvious method for suorting CBRN event resonse lanning from oerational and strategic ersectives that would satisfy the different requirements of military lanners and senior decision makers. For all these reasons, the VITRO analyses aroach was develoed and documented initially in an internal Canadian government reort by Dooley & Gauthier (2013a). The first results of the aroach were used by the CAF for CBRN event resonse lanning but are not ublicly available (Dooley & Gauthier (2013b)). The aim of this aer is to describe the VITRO analyses aroach in a concise, clear and ublicly accessible manner, to enable its broader dissemination amongst OR ractitioners and alication to other lanning areas (military or civil). The aroach is illustrated here in its original context: domestic CBRN event resonse lanning. Using notional unclassified data throughout, we show how the aroach addresses the four questions listed earlier, via four distinct analyses that we summarize as follows. 1. Vignette analysis: the identification of otential domestic CBRN events (vignettes) and the estimation of their relative robabilities of occurrence. 2. Task analysis: the identification of military tasks associated with a otential domestic CBRN event resonse and the estimation of the robabilities that the CAF would be requested by civil authorities to erform them. 3. Requirement analysis: the estimation of quantities and training levels of ersonnel required for adequate task erformance, when requested, within each vignette. 4. Otion analysis: the comarison of hyothetical force ackages, based on the ga robabilities associated with each otential task. This aer s structure reflects these analyses and the next four sections describe the vignette, task, requirement, and otion analysis methods in turn. For each, an overview of the method is rovided, along with a means of visualizing the results and a discussion of the method s advantages and limitations. Thereafter, the concluding section describes attributes of the overall VITRO analyses aroach and how it was successfully alied to the develoment of a concet for domestic CBRN event resonse. Vignette analysis The vignette analysis involves the construction of a sufficiently comrehensive set of vignettes, whose robabilities of occurrence relative to the entire set are estimated subsequently. Vignette set The quality of results obtained from the VITRO aroach deends strongly on the quality of the vignette set analysed. The set should be well thought out and sufficiently detailed. The number of vignettes considered should be large enough to san the relevant range of ossibilities, but limited enough to make their evaluation feasible.

5 For demonstration uroses, we will use here a notional set of 12 CBRN-related vignettes consisting of two chemical, four biological, two radiological, one nuclear, and three hazardous material vignettes. Each vignette would tyically include information such as the nature of the CBRN agent released, the tye of release (e.g., overt or covert, accidental or deliberate), the means of dissemination used, and the target(s) involved. It would also describe the magnitudes of consequences from the release in terms of fatalities, ersons ill or injured, ersons hositalized, ersons contaminated, ersons evacuated, et cetera. Such vignettes (or scenarios) are commonly used for CBRN resonse lanning uroses, including the US Deartment of Homeland Security (2006) and ublic Safety Canada (2012). We do not include the actual vignette descritions here since they are not required to exlain the aroach. Our notional set of vignettes is deicted schematically in Figures 1 and 2. The Venn diagram of Figure 1 resents the set s vignettes groued by agent class. The figure s double-headed arrows reresent airwise comarisons that were made between and within agent classes, which we exlain in the next section. Figure 2 is a hierarchical reresentation, in which otential CBRN events (tomost level) are decomosed first by agent class (middle tier), then by vignettes secific to each agent class (lowest level). Such a hierarchical diagram hels illustrate the data aggregation stes and conditional robability calculations. 27 Figure 1. Venn diagram of the notional vignette set deicting agent classes and airwise comarisons made during the analysis.

6 28 Figure 2. Hierarchical reresentation of notional domestic CBRN vignettes. Estimation of relative vignette robabilities Vignette robabilities may be estimated in either absolute terms (e.g., vignette C1 has an x% chance of occurring during the next 10 years ) or relative ones (e.g., vignette C1 has x% more/less chance of occurring than vignette C2 ). Absolute estimates can be difficult to roduce since our knowledge of the current threat environment is imerfect and that of a future one even more so. Absolute estimates can also be highly contentious. For concet develoment uroses, relative robability estimates generally suffice and require less information. Assuming that a CBRN event will occur (and will be roughly similar to one of the set s vignettes), one only needs to determine how likely each vignette is in comarison to the others. Still, estimating vignettes relative robabilities remains a challenging endeavour. For events that are rare or unrecedented, historical data may have limited utility. Consequently, robability estimates must often be elicited during workshos, whose articiants make subjective and biased judgements based on an imerfect knowledge of the current and future threat environments. Such elicitation of risk estimates resents multile itfalls, to which much consideration has been devoted (Wiedlea, 2008). An obvious rerequisite is the availability of articiants with suitably dee and ertinent knowledge. However, articiants biases and ways of reasoning are also imortant (Gilovich, Griffin & Kahneman, 2002). Moreover, the manner in which information is elicited from them can also affect the estimates obtained (Leung & Verga, 2007). Our aroach aims to strike a favourable balance between such considerations, given revailing time and resource constraints. We facilitated a worksho during which estimates were elicited from subject-matter exerts (CBRN advisors, intelligence officers, military lanners) on a consensus basis, and catured them in a urose-built sreadsheet tool to derive the robability estimates. To inform their deliberations, articiants also had access to historical data on CBRN events that have occurred worldwide. The facilitation involved focusing the discussion, challenging articiants assumtions and reasoning, seeking consensus, ensuring that estimates were stated unambiguously, and alying the airwise comarison method described next. airwise comarison method To obtain estimates of the vignettes relative robabilities (or weights), we used an eigenvector-based weight assignment method (Saaty, 1998) as in the Analytic Hierarchy rocess (AH) (Saaty, 1980), which decomoses decision roblems into a hierarchy of sub-roblems, as in Figure 2. airwise comarisons are arguably easier and more accurate than several comarisons done simultaneously, as was observed by Thurstone (1927). Considering only two vignettes at once also hels to focus grou discussions, leads articiants to consider each vignette more thoroughly, and makes it easier to achieve consensus on the estimates.

7 We first consider the round of comarisons between all ossible airs of agent classes. As Figure 1 illustrates, ten such comarisons can be made. These comarisons are listed individually as rows in Table 1, along with notional values for each. On a consensus basis, worksho articiants selected the agent class that they believed is more likely to be associated with a domestic CBRN event. They then estimated the factor by which that would be so, relative to the less likely agent class ( relative likelihood column). Let the relative robability of a domestic CBRN event involving an agent of class a be reresented by a conditional robability of the form a. Given such a CBRN event, the relative likelihood of a chemical warfare agent release is C, the relative robability of a biological warfare agent dissemination is B, etc. Since these quantities reresent a comlete set of relative robabilities,. Table 1. Notional data for airwise comarisons of agent class likelihoods. 29 Agent Classes Comared More Likely Agent Class Relative Likelihood Imlied Relationshis C & B B 40 C & R R 60 C & N N 20 C & H H 40 B & R R 2 B C N C R C H C C 40 ; R B B C 60 ; R C 20 ; N C 40 ; H B 2 ; B N B & N B 1 1 B & H H 2 H B N B R B 2 ; R N R & N R 1 1 R & H R 3 N & H H 2 R H H N N R H H 3 ; R N 2 ; Analogous airwise comarison rounds were conducted for vignettes secific to each agent class. Given a domestic event involving agent class a, we reresent the relative likelihood of the i th vignette ertaining to that agent class by a conditional robability of the form ai. For instance, C1 denotes the relative robability of the first vignette within the subset of vignettes involving chemical warfare agents, C1. Assuming that any CBRN event could be aroximated by a vignette in the set, the relative robabilities satisfy the agent class-secific normalization relationshis. airwise estimates are often internally inconsistent. In Table 1, for examle, the sixth estimate indicates that events involving either biological or nuclear agents are equally likely, i.e.,. Similarly, the eighth estimate suggests that B N H

8 30 radiological events and nuclear events have equal likelihoods, i.e.,, which imlies that. Yet, this contradicts the fifth estimate, which states that. Such internal inconsistency is a otential drawback of airwise comarisons (Bana e Costa & Vansnick, 2008) but it is an avoidable one, as we will describe later. Aggregation of airwise robability estimates We aggregated airwise comarison data in a series of stes following the hierarchical arrangement of notional vignettes in Figure 2, using an eigenvector method. The first ste involves aggregating the airwise comarison data for agent classes (Table 1), to yield the relative robabilities of each agent class involved in the event. These can be reresented by the vector V A :. Next, we construct a comarison matrix for the agent classes M A (Equation (2)), which we oulate based on the imlied relationshis listed in Table 1. In the matrix, each agent class is associated with both a row and a column. Each matrix element corresonds to how many times more likely its row s agent class is to be involved in a domestic CBRN event than its column s agent class. Note that because of the lack of internal consistency in Table 1 s agent class estimates, the comarison matrix M A is not transitive. (1) (2) To obtain the relative robabilities of each agent class being involved in a domestic event, we solve the eigenvalue equation where λ Amax reresents the rincial eigenvalue. Its range is, where is the number of agent classes. Had the matrix M A been internally consistent, the limiting case would have resulted. Since M A is inconsistent, λ Amax will exceed n A, so the difference can be used to gauge the inconsistency of the agent class likelihood estimates in Table 1. The most commonly used measure of matrix inconsistency was develoed by Saaty (1980), who defined the consistency ratio CR for a generic comarison matrix of order n having rincial eigenvalue λ max as follows: where CI is known as the consistency index and CV denotes Saaty s comarative value, which deends imlicitly on the comarison matrix order (Saaty, 1977). During elicitation workshos, we quantitatively assessed the degree of inconsistency of each set of airwise estimates, in real time. When the value of CR exceeded 0.1, a threshold value set by Saaty (1980, 125) and routinely used by AH ractitioners, we brought a subset of inconsistent estimates to the articiants attention. Following additional discussion, articiants revised one or more of their estimates, to imrove the set s consistency. (3) (4)

9 For examle, by solving Equation (3), the rincial eigenvalue Amax The agent class comarison matrix M A (Equation (2)) is of order n A 5 and so has a corresonding comarative value of CV A 1. 11(Saaty, 1980, 21). Together, these yield a consistency ratio of CR A (Equation (4)), which is in the generally acceted range. If we solve Equation (3), for examle by using the ower method (von Mises & ollaczek-geiringer, 1929), we obtain the normalized rincial eigenvector V A, which aroximates the relative robability for each agent class: V A C B R N H The same eigenvector aroach was used to estimate the robabilities of vignettes relative to their resective agent classes (i.e., for each subset of vignettes in the lowest tier of Figure 2). The relative likelihood for each vignette relative to the entire set, which we will exress as ai, is obtained by taking the roduct of the aroriate conditional robabilities. To better illustrate this, we resent intermediate results ertaining to the notional vignette set in Table 2. The overall likelihood for each vignette ai is calculated by taking the roduct of the associated agent class robability a and the vignette s relative robability within that agent class, i.e., ai a ai ai 31 (5) (6) Table 2. Relative robabilities of each agent class, each vignette with resect to its agent class, and each vignette with resect to the notional set. Agent Class Agent Class robability Vignette Within Agent Class Vignette robabilities Within Notional Set Chemical Biological Radiological C B R Nuclear N1 000 Hazardous Material N H C1 C C C2 0. C C B1 B B B2 B B B3 B B B4 B B R1 R R R2 R R N N H1 H H H2 H H H3 0. H H Figure 3 shows the results in grahical form. It grous vignettes by agent class and shows their robabilities relative to the entire notional set on a logarithmic scale. In ractice, we were able to calculate such robabilities in real time using a simle sreadsheet tool, which allowed us to udate articiants during elicitation workshos. The tool enabled articiants to consider the aggregated vignette robabilities holistically and confirmed that the lotted vignette robabilities were reasonably consistent with their collective exectations.

10 32 Dooley and Gauthier (2018) Figure 3. Relative robabilities of vignettes within the notional set, groued by agent class, i.e., chemical (C), biological (B), radiological (R), nuclear (N), and hazardous material (H). CAF lanners valued the reduced comlexity, enhanced focus, and structured aroach engendered by the use of airwise comarisons for estimating vignette likelihoods. They also areciated the rigorous and transarent means of aggregating airwise estimates, as well as the relative simlicity of the logarithmic lot used to deict and validate the results. The method has limitations, however. For examle, it requires a well-founded and reresentative vignette set at the outset. In our case, this involved a significant amount of research. Another limitation is that, although airwise estimates were elicited systematically from worksho articiants, their deliberations were relatively unstructured and lacked anonymity. Finally, although the airwise estimates degree of internal consistency was monitored and controlled, their uncertainties were not estimated or roagated. Consequently, the notional robabilities derived from such estimates Table 2 are stated without associated uncertainties. A method analogous to that described by Zahir (1991) for assessing uncertainties in airwise estimates can address this roblem, but would significantly increase the elicitation time and would comlicate the overall aroach. Task analysis In the task analysis, the likelihoods of being requested to erform articular tasks in the context of each vignette are estimated. Given such estimates and a sufficiently reresentative vignette set, the overall likelihood of having to erform a certain task is calculated. Task set Our analysis covered a set of 54 otential tasks that the CAF might be requested to erform, with resect to a domestic CBRN event. The set s tasks were selected from a Canadian variant (Centre for Security Sciences, 2012) of a document roduced by the US Deartment of Homeland Security (2007). The set was assumed to be sufficiently comrehensive for analytical uroses, yet suitably concise for use in a worksho setting.

11 We divided the task set based on the four temoral hases set out in Canada s Federal olicy for Emergency Management (ublic Safety Canada, 2016). The revention and rearedness hases resectively recede a domestic CBRN event, whereas the resective resonse and recovery hases jointly comrise its aftermath. Since the concet develoment focused on managing the consequences of a CBRN event, resonse-hase tasks redominated. They include various reconnaissance tasks, decontamination tasks, monitoring tasks, rotection tasks, and others. These ost-event tasks were subdivided, based on the three satial zones in which they would be conducted: a hot zone (in which contaminants are resent), a cold zone (which is free of such contaminants), and a warm zone (which reresents a transition area between the hot and cold zones, where decontamination and related activities would be erformed). Using such zones was concetually helful, as it facilitated worksho articiants discussions. 33 Estimation of task erformance request robabilities The likelihood that a force would be requested to erform a articular task is situation-deendent. Such likelihood estimates were elicited from subject-matter exerts during a series of workshos. Again, we used a sreadsheet tool to cature elicited data throughout the analysis. We rojected the tool s interface onto a screen, to inform and foster articiants discussions. An analogue of the data inut interface for the notional tasks and vignettes is resented in Figure 4. There, each row corresonds to a notional vignette and each column ertains to a notional task. During workshos, articiants considered each otential task in turn. For each task, they first oenly discussed and then estimated (on a consensus basis) the robability of the CAF being requested to conduct the task within the context of each vignette individually. Each likelihood estimate was entered into the interface, at the intersection of the alicable vignette s row and task s column. Estimates were made quantitatively using a colour-coded, seven-interval scale (Figure 4) with extrema of 0% (i.e., never or not alicable ) and 100% (i.e., always ). Though an interval scale is less recise than oint estimates, its use facilitated the attainment of consensus concerning 648 (i.e., 54 otential tasks 12 vignettes) likelihood estimates during the time-constrained workshos. Figure 4. Notional task erformance request robabilities by vignette Aggregation of task erformance request robabilities We accounted for the large variation in relative vignette likelihoods (Figure 3; Table 2) and the strong situational deendence of task erformance request robabilities (Figure 4) by combining the two data sets. In so doing, we obtain the vignetteweighted robabilities of receiving erformance requests for articular tasks, given a domestic CBRN event.

12 34 Dooley and Gauthier (2018) We denote each of the 648 task erformance request likelihoods resented in Figure 4 using the notation. Here, Tj corresonds to a articular task and ai ertains to the ith vignette involving agent class a, within which the robability of a erformance request was estimated. The overall robability of the CAF being requested to erform a articular task Tj in relation to a domestic CBRN event by Tj is readily calculated as a weighted sum: (7). For all tasks within the notional set, vignette-weighted erformance request robabilities Tj in relation to a domestic CBRN event are lotted in Figure 5. Tasks are groued by temoral hase and sorted in descending order of erformance request likelihood. Figure 5 s vignette-weighted task erformance request robabilities also san a broad range. For examle, whereas notional requests for the erformance of task T30 are exected after nearly three-quarters of domestic CBRN events, erformance requests for T38 are only anticiated in ~2% of such cases. For most tasks, the notional likelihood of a erformance request in relation to a domestic CBRN event is in the 10%-40% range. Figure 5. erformance request robabilities for tasks within the notional set, in relation to a domestic CBRN event.

13 The task analysis method and its results were embraced by CAF lanners and decision makers alike. The method s systematic elicitation and aggregation of several hundred subjective judgements encomassing a diversity of situations imosed valuable structure on a comlex lanning roblem. Even in isolation (without regard to vignette likelihoods), CAF lanners considered the task analysis to be valuable because it offered an exansive framework for considering the diverse circumstances in which the CAF may be requested to erform various tasks. CAF lanners valued the real-world analogue of Figure 4 because it was simle, concise, and fostered discussions. They also valued the real-world analogue of Figure 5, which clearly and concisely resented which tasks are the most likely to be requested from the CAF. There are limitations to such a task analysis. First, the 648 consensus estimates of vignette-secific task erformance request likelihoods were obtained in a manner which lacked anonymity. In rincile, alternative elicitation means could have been used to achieve anonymity of oinion, but their alication would have been infeasible given the large number of estimates required and the workshos time-limited nature. Similarly, uncertainties in the elicited inuts were not estimated. Though such uncertainties would have been analytically valuable, the time available to conduct workshos would have been insufficient to elicit and achieve consensus regarding so many uncertainties. 35 Requirement analysis The requirement analysis involves estimating requirements associated with the tasks that the military force may be requested to erform. For illustration uroses, we will focus on how ersonnel and training requirements can be established in a CBRN resonse context, but the aroach can also be alied to equiment or other tyes of requirements. Training levels and task set To address ersonnel requirements, we secified a set of six individual training levels, as defined in Table 3. Table 3. Definitions of individual training levels used during requirement and otion analyses. Training Level Short Form Descrition General Duty IS1-trained IS2-trained IS3-trained CBRN Secialist Non-CBRN Secialist GD IS1 IS2 IS3 CS NCS Military ersonnel who have not received any CBRN defence training Military ersonnel who have received CBRN defence training to a basic individual standard (IS), to enable their survival in a CBRN environment. Military ersonnel who have received IS1 training lus additional instruction, to enable them to oerate in a CBRN environment Military ersonnel who have received IS2 training lus further instruction, to enable them to erform secialized tasks in a CBRN environment Military ersonnel who have received additional, advanced CBRN defence training from the CAF Fire and CBRN Academy or foreign equivalent Military or civilian exerts in fields other than CBRN defence, e.g., logisticians, medical ersonnel, etc. Based on the vignette set and task set reviously described, a team of CAF lanners estimated the number of ersonnel that would be required, at each training level, to erform each task adequately, within the context of each vignette. We defined adequate task erformance as the minimum accetable level in each situation. Admittedly, adequate levels of task erformance can vary by agent tye, agent quantity, agent quality, dissemination means, event location, event scale, event duration, civilian resonse caabilities and caacities, etc. We sought to account for such diversity (at least artly) by estimating ersonnel requirements on a vignette-by-vignette basis.

14 36 Dooley and Gauthier (2018) Requirements estimation To estimate requirements, CAF lanners entered their requirements estimates into six tabs of another sreadsheet tool. Each tab was associated with a articular training level and contained a matrix analogous to that deicted in Figure 6, which resents fictitious general duty ersonnel requirements for the notional vignette and task. Five matrices with a similar format were roduced for the other training levels. CAF lanners considered each task in turn when estimating requirements. For a given task Tj, they entered estimated quantities of ersonnel rtj, required at each training level for adequate erformance within each vignette ai on the ( C1) corresonding tabs. For examle, using this notation, r T 4, GD denotes the estimated quantity of general duty ersonnel required to erform task T4 adequately within chemical vignette C1. To avoid unnecessary work, the lanners did not estimate ersonnel requirements for task-vignette airs for which the likelihood of receiving a erformance request was estimated to be zero during the task analysis (i.e., where 0 in Figure 4). Such cells are shaded grey in Figures 4 and 6. Tj Figure 6. Notional quantities of general duty ersonnel required to erform each task adequately within each vignette. Though relatively straightforward, the requirement analysis method advanced the CAF s concet develoment effort in many ways. The method simlifies the estimation of requirements by forcing lanners to consider the ersonnel requirements for each task-vignette air individually. Furthermore, the estimates detail and structured resentation facilitate external review. Again, this analysis is limited in some ways. For instance, all requirements estimates were subjective and based on the rofessional oinions of the CAF lanners involved. Some estimates could otentially be refined through exercises or exerimentation, but the amount of effort required to validate inuts emirically would almost certainly be rohibitive, and some degree of subjectivity in estimating such requirements is resumably inevitable. Otion analysis The otion analysis serves to evaluate resonse caabilities associated with various otential resource commitments. As an examle, we quantify otential trade-offs between commitments of various quantities of ersonnel, their levels of individual training, and their corresonding caability gas in terms of CBRN resonse.

15 37 Force ackages Force ackages are the otions analyzed here. Each force ackage secifies quantities of ersonnel at articular training levels, e.g., 200 IS1-trained lus 100 IS2-trained ersonnel. We reduced the infinite number of otential force ackages to a more easily tractable quantity by identifying thresholds that occurred frequently in the estimated ersonnel requirements at the general duty, IS1, IS2, and IS3 training levels and limited force ackages to combinations of those ersonnel quantities. We also limited force ackage comositions to include either no secialists whatsoever or quantities of CBRN and non-cbrn secialists that would suffice for any otential task. This restriction greatly reduces the number of otential force ackages, given the diversity of secialist tyes. Let q f, denote the quantity of ersonnel of individual training level associated with a given force ackage f. We define ten notional force ackages labelled A through J, with comositions as described in Table 4. To highlight training-related differences, force ackages A-F each consist of 500 non-secialist ersonnel, whose aggregate level of training increases from left to right in Table 4. Force ackages G-J each consist of twice as many (i.e., 1000) non-secialist ersonnel, whose training level also increases from left to right in the table. We assessed each force ackage twice, assuming different quantities of secialists in each instance. That is, we assumed (a) that quantities of secialists sufficient to erform any task were included initially, then, (b) that no secialists were included, during the second assessment. This collective, all or nothing aroach to secialists enables us to highlight succinctly their imortance in resonding to CBRN events. Table 4. ersonnel quantities by training level comrising notional force ackages A-J. Secialist quantities denoted by were collectively assumed to be either zero or sufficient for erforming any task, deending on the assessment conducted. Training Level Force ackage A B C D E F G H I J General Duty IS IS IS CBRN Secialist Non-CBRN Secialist Calculation of ga robabilities As a basis for otion comarisons, we first calculate the robability that a given force ackage would be unable to fulfill a articular task erformance request adequately, given a CBRN event. To obtain such task-secific ga robabilities for each force ackage, we must first calculate and aggregate the intermediate quantities resented in the following subsections. Ga robabilities by task-vignette air and training level We begin by considering, in turn, each task Tj within the context of each vignette ai. Our aim is to determine whether the ersonnel requirement rtj, at a given training level exceeds the available quantity of qualified ersonnel Q f, within a articular force ackage f. If so, a erformance ga will exist, and we will assign a training level-secific ga robability value of unity for the force ackage s task-vignette air. Otherwise, the ersonnel requirement can be met, so we will assign a training level-secific ga robability value of zero. In formal terms, the training level-secific ga robability for a given training level can be exressed in terms of the quantity of available qualified ersonnel Q as: f,

16 38 Dooley and Gauthier (2018) 1 if rtj, Q f,, 0 otherwise. Q When determining values of f,, we note that some of the training levels described in Table 3 are rerequisites for others. For examle, the general duty, IS1, and IS2 training levels are rerequisites for IS3-level training. So, an available IS3-trained erson can be emloyed to fulfill an IS2-, IS1-, or general duty-level requirement. Thus, when determining training level-secific ga robabilities, the available quantity of qualified ersonnel Q f, within a force ackage f able to fulfill a -level ersonnel requirement includes both (a) ersonnel who have been trained to level as well as (b) more highly trained ersonnel who have not been allocated in fulfillment of another ersonnel requirement. Accordingly, the available quantity of qualified non-secialist ersonnel Q f, able to satisfy a -level ersonnel requirement can be exressed as: Q q q f, q f, 1 f, f, q f, 1 rtj, 1 if 1 if 1 1, 0. We can aly Equations (8) and (9) iteratively, to obtain training level-secific ga robabilities for the IS3, IS2, IS1, and general duty levels. For secialists, we define the analogous training level-secific ga robabilities CS (for CBRN secialists) and NCS (for non-cbrn secialists). As discussed before, we assumed that force ackages included either (a) sufficient quantities of both secialist tyes to erform all requested tasks adequately or (b) no secialists of either kind. In the first case, the associated training level-secific ga robabilities for a given force ackage f were always zero, i.e., ( ) ( ) 0 for any task Tj erformed in any vignette ai. In the no-secialists case, 0,, ai f Tj CS NCS,, ai f Tj CS NCS CS only when secialists were not required to erform a task Tj within vignette ai. Otherwise, 1 when CBRN secialists were required and NCS 1 when non-cbrn secialists were necessary. In ractice, the determination of training level-secific ga robabilities is relatively straightforward. As an examle, consider force ackage B (Table 4), which consists of 200 IS1-trained and 300 general-duty ersonnel (i.e., q B, IS1 200, q B, GD 300 ) lus sufficient quantities of both CBRN and non-cbrn secialists. Next, let force ackage B be requested to erform task T30 in the context of vignette B2. For that task-vignette air, the quantity of ersonnel required at each training level is listed in Table 5. Table 5. Summary of notional values used to illustrate the determination of training level-secific ga robabilities. (8) (9) Training Level, Quantity of ersonnel in Force ackage Quantity of ersonnel Required Force ackage ersonnel Allocated Ga robability by Training Level GD q 300 r 400 B, GD IS1 200 B, IS1 IS2 0 T 30, GD GD: 300 ersonnel IS1: 100 ersonnel B, T 30, GD q r 0 None required 0 B, IS 2 IS3 0 CS NCS T 30, IS1 B, T 30, IS1 q r 0 None required 0 B, IS 3 T 30, IS 2 B, T 30, IS 2 q r 25 IS3: None available 1 T 30, IS 3 q B, CS sufficient T 30, CS 0 q B, NCS sufficient T 30, NCS 0 B, T 30, IS 3 r None required 0 B, T 30, CS r None required 0 B, T 30, NCS Since the force ackage does not contain any IS3-trained members, it cannot satisfy the IS3 training level requirement of r T 30, IS 3 25 ersonnel and the IS3 training level-secific ga robability is B, T 30, IS 3 1. Since no IS2-trained or IS1- trained ersonnel are required, the corresonding training level-secific ga robabilities are B, T 30, IS 2 0 and 0. Since there is no ersonnel requirement at the IS1 training level, all of the force ackage s 200 IS1-trained B, T 30, IS1 0

17 39 members remain available for allocation. The task-vignette air s general-duty requirement is r T 30, GD 400 ersonnel. Since this requirement can be met by allocating all of the force ackage s 300 general-duty ersonnel lus 100 of its unassigned IS1-trained members, the associated general-duty ga robability is B, T 30, GD 0. For CBRN secialists and non-cbrn secialists alike, the training level-secific ga robabilities are zero since no secialists of either tye are required. Such rocedure must be reeated to determine training level-secific ga robabilities for a force ackage f being requested to erform a task Tj in the context of vignette ai. Ga robabilities by task-vignette air We now wish to determine the likelihood that a articular force ackage f will be unable to erform a requested task Tj adequately in the context of vignette ai. For convenience, we will refer to this likelihood as the force ackage s ga robability for a given task-vignette air and denote it by g f, Tj. In concetual terms, adequate task erformance can only occur if a force ackage s comosition satisfies the task-vignette air s ersonnel requirements at every training level. Conversely, if any of the training-level secific ersonnel requirements cannot be met by the force ackage, then a erformance ga will result. Thus, a force ackage s ga robability for a given task-vignette air g f, Tj can take on values of either zero or one. It can be exressed formally in either of the two following manners: g or g f, Tj f, Tj if otherwise 1 for any. To illustrate this first aggregation ste, we consider the tabulated values for the examle resented in the revious section (Table 5). In that examle, force ackage B does not meet the IS3-level ersonnel requirement for task T30 in vignette B2. Consequently, the associated IS3 training level-secific ga robability is equal to one (i.e., B, T 30, IS 3 1; Table 5). Since the tabulated training level-secific ga robabilities are not all zero-valued (i.e., since B, T 30, 0 is not true for every training level ), force ackage B would be unable to erform task T30 adequately in vignette B2. This erformance ga is denoted by assigning a ga robability for the given task-vignette air of g B, T Force ackage-secific ga robabilities for task-vignette airs g f, Tj are valuable to lanners, since they illustrate how a force ackage s estimated ability to erform a given task adequately varies, based on the oerational context (i.e., by vignette). However, to gauge a force ackage s ability to erform a given task with resect to the entire vignette set (i.e., in the general case), a further aggregation ste is required. (10) Ga robabilities by task Given a domestic CBRN event, we can derive the vignette-weighted ga robability G f, Tj (i.e., that a force ackage f would be unable to erform a requested task Tj adequately). We generalize the revious result (which ertains to a single vignette) by summing over all vignettes in the set. In so doing, we obtain each force ackage s vignette-weighted ga robabilities G f, Tj, which aly to an arbitrary domestic CBRN event. Each of these reresents the exected likelihood that a force ackage f would be unable to erform adequately a requested task Tj in the context of an arbitrary domestic CBRN event. They form the basis of the otions analysis for lanners and decision makers. The vignette-weighted ga robabilities are given by: G f, Tj ai Tj g f, Tj (11) ai

18 40 Dooley and Gauthier (2018) We now demonstrate how such ga robabilities are calculated, via an examle which emloys the notional data resented earlier. We illustrate the calculation of G B,T14, i.e., the robability that force ackage B would be unable to erform task T14 adequately, given a domestic CBRN event. For our chosen case, Equation (11) becomes: GB, T14 ai T14g B, T14 (12) ai For task T14, the corresonding vignette-secific task erformance request robabilities T14 (Figure 4) are non-zero for only four vignettes, i.e., for ai C 1, B1, B3, R1. For these vignettes, we have tabulated the corresonding task erformance request robabilities by vignette T14 along with the corresonding vignette robabilities ai (Table 2; Figure 3) and task-secific ga robabilities g in Table 6. B, T14 Table 6. Notional data required to calculate the task-secific ga robability G, assuming that sufficient secialists are available. B,T14 Vignette Vignette robability Mean Task erformance Request robability Task-Secific Ga robability ai C1 C ( C1) 0. 5 ( C1) g 0 ai B1 B g 1 ai B3 0. B g 1 ai R1 R g 0 T14 ( B1) T14 ( B3) T14 ( R1) T14 T14 ( B1) T14 ( B3) T14 ( R1) T14 Figure 7. Notional vignette-weighted ga robabilities for force ackages A-J, assuming sufficient secialists (left) or no secialists (right).

19 Substituting the values into Equation (12), we obtain G B, T Thus, given a domestic CBRN event, there is a ~1% likelihood that force ackage B would be requested to erform task T14 but unable to conduct it adequately. During an otion analysis, the number of vignette-weighted ga robabilities G f, Tj under consideration can be large. For examle, our notional data set involves ten force ackages and 54 tasks, for a total of 540 vignette-weighted ga robabilities. Hence, effective means to visualize and facilitate the interretation of the ga robabilities are imortant. While assisting CAF lanners, we used a reresentation analogous to that shown on the left ortion of Figure 7, where each cell s colour denotes the vignette-weighted ga robability G f, Tj of the associated force ackage-task combination. Using such a chart, we can exlore trade-offs between quantities of ersonnel at various training levels and vignetteweighted ga robabilities for various tasks. As the quantities and/or degrees of training of ersonnel within a force ackage increase(s), vignette-weighted ga robabilities generally decrease, but not linearly. For instance, ackages E and F both include a total of 500 IS-trained ersonnel, but the enhanced training of 200 ersonnel in force ackage F affords significant reductions in vignette-weighted ga robabilities. In a similar fashion, we can exlore the notional imlications of excluding secialists from force ackages A through J. This causes vignette-weighted ga robabilities to increase significantly for many tasks (right ortion of Figure 7). By aggregating the results of the vignette, task, and requirement analyses in a systematic manner, the otion analysis readily enables comarisons between multile force ackages. In articular, reresentations such as Figure 7 facilitate lanning discussions and briefings to decision makers on the trade-offs between resource commitments, training costs, and caabilities. The notional otion analysis shown here is limited in some ways. For examle, it considers only quantities and training rofiles of ersonnel. Equiment and other resources could be easily added to force ackage comositions, but considering too many resources otions can lead to a combinatorial exlosion of the number of force ackages. As in the revious analyses, we omitted uncertainties. However, had uncertainty values been roduced for the three revious VITRO analyses, uncertainties for the otion analysis results could easily have been calculated. 41 Concluding remarks We have described the VITRO analyses aroach to oerational concet develoment in a concise, generic way that makes it alicable to a variety of lanning areas. We illustrated it in a ublicly accessible manner by using notional data on CBRN event resonse. When used to assist CAF lanners, many attributes of the aroach contributed to its successful alication. First, the early articulation of four key lanning questions associated with each ortion of the aroach allowed us to frame the analytical objectives in simler terms and reduced the erceived comlexity of the lanning roblem. Such focused design yielded clear objectives for each data collection worksho and simlified articiants contributions. Second, the results of each analysis could be briefed as they became available, which heled to demonstrate consistent rogress to articiants and stakeholders, thereby increasing their interest in the larger analytical effort. Since each analysis ertained to a secific lanning question, the results could be resented to decision makers in a relatively simle, linear narrative form. For decision makers, this facilitated comrehension and engendered confidence regarding the overarching analytical effort. Third, the quantitative nature of inuts and results made results much less ambiguous than the use of urely qualitative scales (e.g., low, medium, high ), which can be interreted differently from erson to erson (and from situation to situation). Fourth, the aroach enabled systematic and transarent aggregation of inuts and results. For examle, in this aer we aggregated 4557 notional inuts to obtain 54 vignette-weighted ga robabilities for each force ackage. Finally, the means of visualization used for each analysis enabled us to communicate results to senior decision makers succinctly and effectively, without overly reducing the scoe of the information resented. Because they are modular, VITRO analyses can be alied in isolation to address secific toics or lanning roblems. For examle, the VITRO aroach has already been successfully alied to multile lanning issues of the CAF, including CBRN resonse lanning (Dooley & Gauthier, 2013b) and Canadian Army modernization rojects related to indirect fire (elletier, 2015) and ga crossing (Bassindale & elletier, 2016). Some asects of the aroach could be extended or modified, as necessary. For instance, ossible extensions include the use of alternative data elicitation aroaches and the quantification of uncertainties.