On Spotting the Terrorist in a Crowd

Transcription

1 On Spotting the Terrorist in a Crowd 2-3 March, 2018 Edwards Bayesian Research Conference CSU, Fullerton Richard S. John Dept. of Psychology University of Southern California Center for Risk and Economic Analysis of Terrorism Events

2 Individual Risk Assessment Any process involving the systematic gathering and interpretation of information pertaining to an individual in order to predict the likelihood that the individual will engage in the behavior of concern in the future.

3 Risk Assessment Approaches: Three Approaches to Risk Assessment Unaided Professional Judgment Structured Professional Judgment Actuarial approaches Key Distinction is the extent to which the risk categorization is expressed quantitatively

4 Predictor Variables Four Classes of Predictor Variables Racial-Physical Psychopathological Socioeconomic Behavioral Detection

5 Current Risk Assessment Practice Assessment of known terrorists for sentencing and rehabilitation Violent Extremist Risk Assessment Extremist Risk Guidance 22+

6 Prevention of Violent Radicalization Examples Vulnerability Assessment Framework (UK) Emotions & Cognitions, Intentions, Capabilities Observable Indicators of Possible Radicalization (EU) 28 consultants, 21 indicators in 5 thematic areas (Identity, Ingroup-Outgroup Differentiation, Pro-violence social interactions, Change in persona, & Association.) Identifying Vulnerable People (IVP; UK) Literature to ID red-flag behaviors Membership in radical groups, contact with extremists, adv military training, overseas combat experience Cultural/religious isolation, Isolation from family, risk-taking behaviors, isolated peer group, hate rhetoric, political activism

7 Current Practice Based on Risk Assessment Approaches in Other Domains Predicting Criminal Violent Recidivism Sentencing Parole Decision Making Mental Health Danger to Self or Others? Suicide Assessment

8 State of the Art in Risk Assessment for Criminal and Mental Health Domains Actuarial (algorithmic) assessment tools far outperform Clinical Judgment Diagnosticity of actuarial tools is relatively low Scurich and John (2012), Jurimetrics Journal, demonstrated one popular risk assessment tool to predict violent recidivism should be used to deny parole for ONLY those in (a very small) high-risk group, given conservative estimates of base-rates and costs of false positives and false negatives

9 Illusions of Efficacy Over Confidence Faith in Clinical Judgment Faith in Analytics Hindsight Can always find post-hoc indicators Seem obvious Scapegoating incompetence Representativeness Heuristic Use of Similarity to Known Offenders Probability not equal to Similarity

10 5 Problems Associated with Individual Risk Assessment Lack of Clarity in Defining Risk to Assess Poor Diagnosticity of Predictors Off-limits (Protected) Predictors Low Base-Rates Asymmetry in Costs of False-Positives and False-Negatives

11 Signal Detection Theory

12 Signal Detection Theory (SDT) Diagnosticity represented by d and AUC Neither depend on Base-Rate context nor on Value tradeoff between Error Types Optimal threshold (Beta) depends on: Base-Rate (Priors) Relative Costs of False Positives and False Negatives

13

14 Optimal Beta =(Ucr Ufp)/(Uh Ufn) * p(~attacker)/p(attacker) Set Ucr = Uh = U(correct classifications) to 0 =Ufa/Ufn * p(~attacker)/p(attacker) Ufn many orders of magnitude > Ufp How many FP tolerated to reduce FN by 1? 10^5? 10^6? P(~attacker) many orders of magnitude > p(attacker) 100M to 100? 100M to 1000? 10^6? 10^5? Optimal Beta = 1.0 realistic

15 p(false Positive) = p(false Negative) Error Rates vs. d for Beta = Error Rates vs. d' Beta = d-prime

16 Legal Example d Values Polygraphs (Swets, 1996) 2.7 (Raskin and Honts (2000) Eyewitness Facial ID 0.8 Meta Analysis Maximum (Shapiro and Penrod,1986)

17 Medical Example d Values Cervical Cancer testing 1.6 (Experts) 1.8 (Algorithmic) Prostate Cancer PSA testing 2.0

18 Other Example d Values Weather forecasting 1.5 Rain in Chicago 1.7 Min Temp near Albuquerque 1.0 Tornados Fog forecasts near Canberra airport Job Success for Navy personnel selection (Armed Forces Qualification Test)

19 FALSE POSITIVE RATE 1.0 FALSE POSITIVE RATE BY D-PRIME AND FALSE NEGATIVE RATE p(fn)=10^-1 p(fn)=10^ D-PRIME

20 Can we achieve adequate d values to identify attackers? New Indicators? Intrusiveness Constitutionality Political Will Big Data Analytics? Few Attacks --- Small N problem Context Dependency Dynamic

21 Can we achieve adequate d values to identify attackers? Negative Impacts of FP identification? Willingness to tolerate False Positives? Equal Protection Probable Cause Expectations for False Negatives? Even FN rate of 0.01 leads to missing10 attackers in 1000

22 Adaptive Adversaries Adversaries will learn predictive algorithm and adapt Hide Identity Steal Identity from Others Recruit Low Risk Attackers Possible Deterrence Value of Risk Assessment Attacker Beliefs also vulnerable to heuristics of overconfidence in prediction tools Delay Target Substitution Reduced Attack Severity

23 On Spotting the Terrorist in a Crowd 2-3 March, 2018 Edwards Bayesian Research Conference CSU, Fullerton Richard S. John Dept. of Psychology University of Southern California Center for Risk and Economic Analysis of Terrorism Events