C/S/E/L :2008. On Analytical Rigor A Study of How Professional Intelligence Analysts Assess Rigor. innovations at the intersection of people,

Transcription

1 C/S/E/L :2008 innovations at the intersection of people, technology, and work. On Analytical Rigor A Study of How Professional Intelligence Analysts Assess Rigor Daniel J. Zelik The Ohio State University Thursday, September 4, 2008

2 When is Analysis Rigorous? A better understanding of rigor can be leveraged against the ever-present risk of being too shallow in information analysis... Research Questions how does insight into the analysis process influence judgement of rigor? what cues do analysts use in making assessments of rigor? A Study of the Professional Intelligence Analyst these research questions are explored through a study of rigor in professional intelligence analysis, which provided the context for the study of rigor in information analysis more broadly.

3 Methodology Study overview... Participant Profile nine professional intelligence analysts Median = 20 years, Mean = 21 years, SD = 10 years varied strategic analyst demographics Varied in intelligence backgrounds (e.g., economics, linguistics, aeronautical engineering, etc.), current principle duties (e.g., senior analyst, linguist, analyst supervisor, etc.), and gender (6 male, 3 female) Procedure Overview the design of this study was based on the Elicitation by Critique (EBC) methodology. In the EBC approach, participants share their expertise by critiquing the processes of other domain practitioners, rather than by directly verbalizing their individual experiences. (Miller, Patterson, & Woods, 2006) due to changes from the original method, the design of this study was a modified EBC methodology, embedded in a scenario walkthrough.

4 Differences Between Analysis Processes Process Document Analysis Process 1 Analysis Process 2 Query Summary 4 query targets 3 query targets Documents Read (in order of reading) 26 documents 7 documents Where Documents Stored in Folders 1 folder, 6 sub-folders 4 folders Key Documents (relied on heavily) 4 key documents 3 key documents Hypotheses Considered Obstacles to LNG 5 hypotheses, 10 sub-hypotheses 3 hypotheses, 2 sub-hypotheses Collaborations with Others During Process 6 collaborations 1 collaboration Research Note-Sheet Note-sheet Not used

5 Supervisor s Dilemma A generic situation wherein a supervisor decides if the output product of an analyst is acceptably rigorous or if more resources must be invested in the analysis process before sending it forward. Supervisor s Perspective: When is Analysis Sufficient? is the analysis ready to send forward? If not, what else needs to be known? given what is not known, what is the best way to invest limited analytical resources to reduce this uncertainty and improve the analysis product? Supervisor s Dilemma Adapted for the Study as a senior analyst, you might be asked to decide if junior analysts reports are ready to be passed on to a policy-maker.

6 Liquified Natural Gas (LNG) Scenario The scenario reflects the debate in the U.S. energy sector as to whether the importation of LNG is an appropriate approach to cope with increasing natural gas consumption, in light of the projected near-term depletion of the domestic supply. Facets of the LNG Scenario (as a Cognitive Case) (1) energy policy (replanning) (2) terminal siting (planning) (3) safety issues (analysis) (4) security concerns (reanalysis)

7 Data Analysis The primary mechanism for eliciting comparable feedback across study participants was the prompt. The data collected during the sessions was analyzed by organizing participant feedback relative to four prompts designed into the study. Discussion Prompts two discussion prompts, comparison of briefing reports and comparison of process documents, were part of the study critique cycles in the product critique cycle and the processes critique cycle, respectively. These prompts served to elicit responses from participants about the cues they used to infer rigor relative to the various documents they interacted with throughout the study. Decision Prompts two decision prompts, rigor assessment and Supervisor's Dilemma, spanned across multiple phases of the study and were designed to capture how process insight influenced perceptions of rigor relative to analysis process and product, respectively.

8 Research Findings This research identified three general findings on rigor in information analysis... First, it found that process insight influenced judgments of rigor. This highlighted an important distinction between perceived rigor, based on cues inferred from an analytic product, and effective rigor, based on insight into an analytic process indicating that they often may not be aligned. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. While many of the same cues were used to make inferences about analytic rigor, there were substantial differences among participants as to which cues were perceived as being most relevant and in how judgments of rigor were drawn from these cues. Third, the results suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

9 Research Findings This research identified three general findings on rigor in information analysis... First, it found that process insight influenced judgments of rigor. This highlighted an important distinction between perceived rigor, based on cues inferred from an analytic product, and effective rigor, based on insight into an analytic process indicating that they often may not be aligned. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. While many of the same cues were used to make inferences about analytic rigor, there were substantial differences among participants as to which cues were perceived as being most relevant and in how judgments of rigor were drawn from these cues. Third, the results suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

10 Research Findings This research identified three general findings on rigor in information analysis... First, it found that process insight influenced judgments of rigor. This highlighted an important distinction between perceived rigor, based on cues inferred from an analytic product, and effective rigor, based on insight into an analytic process indicating that they often may not be aligned. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. While many of the same cues were used to make inferences about analytic rigor, there were substantial differences among participants as to which cues were perceived as being most relevant and in how judgments of rigor were drawn from these cues. Third, the results suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

11 Research Findings This research identified three general findings on rigor in information analysis... First, it found that process insight influenced judgments of rigor. This highlighted an important distinction between perceived rigor, based on cues inferred from an analytic product, and effective rigor, based on insight into an analytic process indicating that they often may not be aligned. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. While many of the same cues were used to make inferences about analytic rigor, there were substantial differences among participants as to which cues were perceived as being most relevant and in how judgments of rigor were drawn from these cues. Third, the results suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.

12 Decision Prompts Rigor Assessment and Supervisor s Dilemma Prompts Rigor Assessment Supervisor s Dilemma ID Order Before After Before After P0 A(1) A(2) A(1) A(2) A(2) P1 A(1) A(1) A(2) A(1) - P2 A(1) A(2) A(1) - - P3 A(2) A(1) A(1) - - P4 A(1) A(1) A(1) - - P5 A(2) A(1) A(1) - - P6 A(2) A(1) A(1) - - P7 A(2) A(1) A(2) A(1) - P8 A(1) A(2) A(1) - -

13 Comparison of Reports Tone of Reports Probe Responses Type Critique AP1 Briefing Report AP2 Briefing Report Analyst Stance Writing Style Content Selection author is against LNG importation author is in favor of LNG importation excessive focus on terrorism issue excessive focus on environmental issues focus on economic issues P0 P2 P0 P1 P2 P3 P4 P6 P7 P0 P2 P3 P4 P0 P6 P8 author is biased / had an agenda coming in P0 P2 P7 P2 P5 not technical enough / used seemingly non-technical language / used quote words informal writing style / language heavy on conjecture / opinionated / marketing style / editorial / opinion piece P3 P4 P5 sensationalistic / newspaper story / alarmist P3 P4 P3 P5 not as well written / looked like a draft / first pass at research / distracting style looked like cutting and pasting of different papers / picked out snippets / scattered report quantifies what was and was not done in analysis did not answer the task question / answered wrong question has passages that do not help policy maker / got off topic / includes irrelevant info P3 P3 P2 P8 P2 P5 P7 P2 P0 P4 P6 offers solutions / proposes policy P1 P2 P1 P3 P4 P5 P1 P2 P3 P5 P3 P7 P7 P4 P6 P8 P4 P6

14 Comparison of Process Process Document Rankings Document Tier Mean Range Key Documents First Hypotheses Considered Collaborations Second Research Note-Sheet Documents Read Query Summary Where Documents Stored Third

15 Rigor in Intelligence Analysis In contrast with the traditional perspective, the revised definition of rigor that emerged from the study frames the concept as the composite of multiple process dimensions... Traditional Perspective on Rigor in strategic analysis, there is a perspective among many in the intelligence community that achieving rigor in the analysis process depends, most centrally, on the application of a formal method for weighing and comparing of hypotheses commonly termed the Analysis of Competing Hypotheses (ACH). Revised Perspective on Rigor as a multi-attribute metric, analytic rigor is the composite of independent components of the analysis process which, when aggregated, reveal a composite assessment of analytic rigor. Specifically, the findings of the study led to the identification of eight critical attributes of analysis processes that contribute to assessments of rigor.

16 Attributes of Rigorous Analysis Attribute Hypothesis Exploration Information Search Information Validation Description Hypothesis Exploration describes the extent to which multiple hypotheses were considered in explaining data. Information Search relates to the depth and breadth of the search process used in collecting data. Information Validation details the level at which information sources are corroborated and cross-validated. Stance Analysis Stance Analysis is the evaluation of data with the goal of identifying the stance or perspective of the source and placing it into a broader context of understanding. Sensitivity Analysis Sensitivity Analysis considers the extent to which the analyst considers and understands the assumptions and limitations of their analysis. Specialist Collaboration Specialist Collaboration describes the degree to which an analyst incorporates the perspectives of domain experts into their assessments. Information Synthesis Information Synthesis refers to how far beyond simply collecting and listing data an analyst went in their process. Explanation Critique Explanation Critique is a different form of collaboration that captures how many different perspectives were incorporated in examining the primary hypotheses.

17 Rigor Attribute Model Hypothesis Exploration HIGH HIGH Information Search MODERATE MODERATE Explanation Critiquing LOW LOW Information Validation Information Synthesis Specialist Collaboration 2007 Zelik, Patterson, Woods Stance Analysis Sensitivity Analysis

18 Participatory Exchange Model REPLANNING 2007 D. Zelik, E. S. Patterson, & D. D. Woods ANALYSIS ANALYSIS REPLANNING BRIEFING ANALYST INFORMING ANALYSIS BRIEFING TASKING Analyst Supervisor Decision Maker AUDIENCE PARTICIPANTS BRIEFING INTERACTION CYCLE CONVERGING PERSPECTIVE

19 Process Insight Interface

20 Conclusions Understanding rigor offers direction for avoiding the trap of shallow analysis, by revealing opportunities to amplify the ability of both analyst and decision maker to more authentically judge the rigor of analysis products. Methodological Contributions LNG Scenario, Supervisor s Dilemma, EBC Methodology Validation & Refinement Research Findings process insight initiated change in perceptions of rigor. professional analysts tended to use similar cues in making judgments of rigor; however, the ways in which cues were transformed into assessments of rigor were more varied. a revised definition of rigor, reframing it as a multi-attribute, emergent measure of sufficiency rather than as a measure of process deviation.

21 C/S/E/L :2008 innovations at the intersection of people, technology, and work. Contact Information The Ohio State University 210 Baker Systems Engineering Building 1971 Neil Avenue Columbus, Ohio Daniel J. Zelik