What Makes a Good Cyber Operator?

Transcription

1 What Makes a Good Cyber Operator? Today s Presenter: Robert Dora, Research Engineer Assured Information Security, Inc. (AIS) September 28, 2016 Moderator: Steve Warzala swarzala@quanterion.com

2 What makes a good cyber operator? Analyzing critical skills and abilities for cyber operators and analysts. Cyber Security and Information Systems Information Analysis Center (CSIAC) Robert Dora dorar@ainfosec.com Assured Information Security, Inc. September 28, 2016

3 Presentation Overview Presentation Goal: Discuss skills, abilities, and traits (SATs) that are critical to cyber operations Describe preliminary research performed at AIS to investigate the differences in such SATs between non-technical, inexperienced, and experienced cyber security engineers Discuss appropriate methods for measuring these traits in cyber operators. Conclude with application of such research 3

4 Metrics Cognitive Factors/Abilities Measure innate traits, such as speed of processing information. Domain-specific Skills Reverse Engineering, Vulnerability Assessment Behavior Empathy, Motivation Past Research Cognitive abilities / factors / tests French, Ekstrom, & Price, 1963 Ekstrom, French, Harman, & Dermen, 1976 John B. Caroll, 1993 BAT System Pilot Selection: Basic Attributes Tests (BAT) System Carretta, 1987 Cyber Operators: S&T of Cyber Operators Jabbour,

5 Cognitive Skills Cognitive Factors/Abilities Identified by Ekstrom et al. Flexibility of Closure Speed of Closure Verbal Closure Associational Fluency Expressional Fluency Figural Fluency Ideational Fluency Word Fluency Induction Integrative Processes Associative Memory Memory Span Visual Memory Number Facility Perceptual Speed General Reasoning Logical Reasoning Spatial Orientation Spatial Scanning Verbal Comprehension Visualization Figural Flexibility Flexibility of Use Length Estimation Associative (Rote) Memory Mechanical Knowledge Memory Span Number Facility Originality Semantic Redefinition Syllogistic Reasoning Sensitivity to Problems Verbal Comprehension Semantic Spontaneous Flexibility 5

6 Cyber Operator What is a cyber operator? Broad Term Forensic Analysts Vulnerability Assessment Professionals / Red Teams Reverse Engineers Security Analysts OCO Operators CND Operators Engineers and Programmers? Are these factors applicable to all operators? Which factors overlap? Which are specific to certain areas 6

7 Our Experiment Send out a survey to AIS employees to measure a variety of skills with shortened surveys Survey Technical Skills Induction Letter Sets Test Integrative Processes Calendar Test Visual Memory Building Memory Test Flexibility of Closure Hidden Figures Test Empathy Quotient Deduction Logical Deduction Test Coding/Debugging 7

8 Results Divided Users into 3 Categories Non-Technical Inexperienced (Technical) Experienced (Technical) Analysis of Results Identified Significant Differences in: Induction: Inexperienced vs Experienced Integrative Processes: Non-Technical vs Technical; Non-Technical vs Inexperienced Visual Memory: Inexperienced vs Experienced Difference not significant, but further investigation is warranted: Induction: Non-Technical vs Experienced Integrative Processes: Non-Technical vs Experienced 8

9 Results (cont.) Research Challenges Collecting data on a sufficient number of users Measuring so many traits makes the survey lengthy, limiting both the number of users and the amount of data that can be collected for each trait Some surveys have specific time limits, requiring administration Some traits cannot be readily measured through surveys 9

10 Key Traits What is Induction and how do we measure it? What is Induction and how do we measure it? Induction is reasoning ability How well can you form and test hypotheses Measured with the Letter Sets test in which a subject must rapidly identify a common rule (or set of rules) from given inputs and identifier outliers Induction is essentially a measure of innate research ability Performing research, identifying anomalies, and detecting outliers is critical for security analysts, particularly forensic investigators and security officers working with large amounts of data Experienced operators likely perform more research and have spent more time identifying anomalies and detecting outliers 10

11 Key Traits Induction Letter Sets Tests Induction Letter Sets Tests Find a rule that makes four of the five sets of letters alike. Select the letter set that does not follow this rule. Example 1: QPPQ HGHH TTTU DDDE MLMM QPPQ is the answer is the answer because each other set has 3 of the same letters. Example 2: BCDE FGHI JKLM PRST VWXY PRST is the answer because each other set is contiguous, whereas PRST skips from P to R. 11

12 Technical Updates Induction Letter Sets Tests (Cont.) Find a rule that makes four of the five options alike. Select the option that does not follow this rule: A B C D E

13 Technical Updates Induction Letter Sets Tests (Cont.) Find a rule that makes four of the five options alike. Select the option that does not follow this rule: A B C D E C Each of the other options is a valid IP address 13

14 Technical Updates Induction Letter Sets Tests (Cont.) Find a rule that makes four of the five options alike. Select the option that does not follow this rule: A B C D E

15 Technical Updates Induction Letter Sets Tests (Cont.) Find a rule that makes four of the five options alike. Select the option that does not follow this rule: A B C D E Which is correct? D is not a valid IP address, but E fails to follow the same numerical convention as the others: A.B.C.D, where B=A+17, C=B+17, and D=C+17 15

16 Technical Updates Several studies were conducted with chess experts and novices to study memory Chess experts performed significantly better than novices or amateurs at remembering realistic board states (de Groot, 1946) The experts performed either no better or insignificantly better than amateurs and novices at remembering random positions where pieces were in unrealistic or invalid positions (Chase & Simon, 1973) As with chess experts, operators are subject to similar biases looking for specific patterns may fail to detect others 16

17 Key Traits What are Integrative Processes and how do we measure them? What are Integrative Processes and how do we measure them? Integrative Processes indicates how well an individual can follow multiple rules simultaneously This combines a variety of traits including short-term memory, perceptual speed, and flexibility of closure Measured with the Calendar test in which a subject must read a set of rules and search through visual stimuli to quickly identify an answer that satisfies all rules Typically, all operators are searching for more than a single stimulus, so the ability to multi-task and search through data with numerous rules in mind Differences were found for all groups, indicating that multi-tasking is a critical skill and may serve as a key indicator of a good operator 17

18 Key Traits Integrative Processes Calendar Test Integrative Processes Calendar Test This is a test of your accuracy in following directions. Each direction will ask you to find a date on a calendar. A circled number is a holiday and Saturdays and Sundays are weekend days. All days except holidays and weekends are work days Example 1: What is the third Tuesday of the Month? a. 15th b. 17th c. 22nd d. 24th e. Not given Example 2: What is the third working day after the holiday? a. 13th b. 14th c. 15th d. 16th e. Not given 18

19 Technical Updates Integrative Processes Calendar Test (Cont.) Examine the following log files and determine the appropriate person to contact based on the following rules: A. If malware is detected with a severity of 5 or higher, contact John Doe (Malware Department). B. If more than 6 instances of malware are attributed to a single user in a one-week period, contact John Doe and Barbara Ross (Policy Violations). C. If the external firewall is disabled, contact Richard Smith (Intrusion Detection). D. If a user transfers over 60 GB of data to an external device at one time, contact Carl Johnson (Data Exfiltration). E. If a user transfers over 2 GB of data to an external device for 5 consecutive days, contact Carl Johnson. 19

20 Key Traits Other Relevant Traits Other Relevant Traits Visual Memory Representation of short-term memory for visual stimuli Measured using the Building Memory test in which subjects must remember the locations of buildings and then identify their location on a blank map Short-term memory is critical component for any analysis and forensic analysts, among other operators, typically operate on visual stimuli Perceptual Speed Measure of how quickly a subject can search through visual stimuli Measured using the Finding A s test in which subjects must search through a list of words and identify those that contain the letter A Similar to Visual Memory, but focuses on how quickly an analyst can perform the task, which is likely a difference between inexperienced and experienced operators 20

21 Application Future Research Needs Future Research Needs Collect a larger more representative dataset that includes a sufficient number of operators from each group (e.g., forensics, red teams) Update surveys to gear towards technical Focus on traits that have shown significance: 1. Induction 2. Integrative Processes 3. Visual Memory 4. Perceptual Speed Expand to look beyond traits to include more situational data that has grown in recent years, including: (1) Vigilance, (2) Trust / Suspicion / Distrust, and (3) Cognitive Workload There is a major difference between a good operator and an experienced operator 21

22 Application Operator Selection Operator Selection Selection will require focus on fairly static (i.e., not situational) values, such as traits and technical abilities Determine which traits are necessary and which ones can be trained An operator can learn more about how network protocols work, however, increasing perceptual speed is unlikely to change Instinct and process may prove more valuable, but are difficult to quantify Determine if technical versions of tests are more valuable than traditional versions What is the end goal? Identify a battery of tests (similar to BAT system) to help with selection of a variety of cyber operators It will be critical to enumerate the different types of operators, as tests may vary based on the type of operator. 22

23 Application Training Training Can some factors can be trained to enhance new and existing operators? Traits (e.g., induction, integrative processes, visual memory, perceptual speed) are largely invariant, but may be somewhat improved Must determine which traits can be improved and to what degree Alternatively, can this help operators detect anomalies they may normally miss? Selective Attention Test: What is the end goal? Identify weaknesses in operators Create a training regiment to help operators and analysts improve certain abilities Reduce common errors 23

24 What makes a good cyber operator? Questions? Robert Dora dorar@ainfosec.com Assured Information Security, Inc x595 24