07. Security Warnings; Designing and Analyzing Quantitative Studies. Blase Ur and Mainack Mondal April 16 th, 2018 CMSC / 33210

Transcription

1 07. Security Warnings; Designing and Analyzing Quantitative Studies Blase Ur and Mainack Mondal April 16 th, 2018 CMSC /

2 Amazon Mechanical Turk 2 2

3 Security Warnings 3

4 4

5 Users swat away warning dialogs How can we get users to pay attention? 5

6 NEAT and SPRUCE (from Microsoft) Rob Reeder, Ellen Cram Kowalczyk, and Adam Shostack. Poster: Helping engineers design NEAT security warnings. SOUPS NEAT 4 questions to ask when you design a security or privacy UX SPRUCE 6 elements to include in a security or privacy UX Good advice, but sometimes it may be better to keep it short and simple rather than include all 6 elements 6

7 7

8 8

9 Alice in Warningland 9

10 Old Warning (IE 6) 10

11 Slightly Newer Warning (IE 7) 11

12 Newer Warning (Firefox) 12

13 Newer Warning (Firefox): Step 2 13

14 Newer Warning (Chrome) 14

15 Alice in Warningland takeaways Field study: correlation or causation? Is clicking through an SSL warnings always wrong? Technically skilled users (e.g., Linux users) ignored warnings more often Comparison with lab studies Prior lab study using eye-tracking software 15

16 More takeaways Passive warnings vs. interstitial warnings Consent and ethics Sampling bias Dealing with noisy data Certificate pinning HSTS (HTTP Strict Transport Security) Prevents protocol downgrade attacks 16

17 How do you know when you are actually at risk? 17

18 Some hazards are ALWAYS dangerous 18

19 Some hazards are context dependent 19

20 Computer security dialogs are context-dependent Security warning dialogs more like warnings on wine than warnings on poison Software developers place burden of assessing risk on users 20

21 A good warning helps users determine whether they are at risk Stops users from doing something dangerous in risky context Doesn t interfere with non-risky contexts Need to test warnings in both contexts 21

22 Non-risky context Encounter self-signed certificate (familiar experience for developers) 22

23 Prior study idea: Create risky context Put users in situation where they have something they care about at risk Come to our lab and check bank account balance online Make users think they are actually at risk Use web proxy to do man-in-the-middle attack 23

24 Revised plan for former study Remove root certificate from browser Web site certificates can t be verified Visits to secure sites will trigger warnings 24

25 Lab study challenges Participants may feel safe They may think they have to do everything we tell them Their priority may be to finish study fast and get paid 25

26 Security-decision UI study How can we focus users attention on key information they need to make informed decisions? C. Bravo-Lillo, L.F. Cranor, J. Downs, S. Komanduri, R.W. Reeder, S. Schechter, and M. Sleeper. Your Attention Please: Designing security-decision UIs to make genuine risks harder to ignore. SOUPS

27 Can you spot the suspicious software? benign suspicious 27

28 Key question: Do you trust publisher? Name of publisher is critical information in trust decision 28

29 How can we get users to notice suspicious publishers? Use attractors to draw attention to publisher name Force delay before users can install Force interaction before users can install Force users to read publisher name 29

30 ANSI standard warning colors 30

31 Obstruct install button until user types publisher name 31

32 Do any of these work? Do attractors and other techniques prevent suspicious installs without preventing benign installs? How much do attractors delay benign installs? 32

33 Methodology requirements Massive, inexpensive, quick Remote observation/recording of behavior Participants should feel safety/risk and behave as they would in real life But should not actually be at increased risk through participation in experiment 33

34 Use Mturk game ruse Ruse previously developed for study of whether users would fall for fake OS password dialogs Operating System Framed in Case of Mistaken Identity: Measuring the success of web-based spoofing attacks on OS password-entry (ACM CCS 2012) 34

35 Online games evaluation survey 35 35

36 Assigned game #1: Mars Buggy Online Attention: The website whose URL appears above is external to this study. Our researchers do not control its contents 36 36

37 37 37

38 Were you able to play the game? Yes No (you will be assigned another game to evaluate) Please enter a one-sentence description of the game you played Have you ever played this game before? Do you think this game is fun? 38 38

39 Was there any other aspect of the game you thought could have been improved? 39 39

40 Assigned game #2: Tom and Jerry Refrigerator Raid Game 40 40

41 41 41

42 42 42

43 Assigned game #3: Colliderix Level Pack 43 43

44 44 44

45 Benign condition: 45 45

46 Suspicious condition: 46 46

47 Participant decision design Workers in Amazon's Mechanical Turk aim to: Complete the tasks they accept (otherwise, don't earn money) Minimize the time and effort in each task (each accepted task has an opportunity cost) Our message to participants: You may skip a game. If you do, we will assign you another The decision was designed to gamble time/money for security: Install Take small risk, play the game, finish sooner Not install Not take any risks, not play the game, waste time 47

48 Results are encouraging 2,227 participants encountered dialogs Benign scenario Installation not prevented But some approaches slowed people down Suspicious scenario Our new dialogs reduced installations Swipe, type, and delay were particularly effective 48

49 Debrief is crucial for ethics! Explain what the actual purpose of the study was 49

50 Statistics! The main idea and building blocks Major tests you ll see Non-independent data 50

51 Important Note In some cases in discussing stats, we will intentionally be imprecise (and sometimes not technically accurate) about certain concepts. We are trying to give you some intuition for these concepts without extensive formal background. 51

52 BUILDING BLOCKS 52

53 Statistics In general: analyzing and interpreting data Statistical hypothesis testing: is it unlikely the data would look like this unless there is actually a difference in real life? Statistical correlations: are these things related? 53

54 What kind of data do you have? Quantitative Discrete Continuous Categorical Nominal (no order) Ordinal (ordered) 54

55 Exploratory Data Analysis (EDA) Shape Center Mean Median Mode Left-skew Right-skew 55

56 EDA Continued Spread Standard Deviation Variance Interquartile range 56

57 Hypothesis testing Causation (X causes Y) vs. correlation (X is related to Y) Develop a hypothesis Assign to conditions (include a control) Terminology: Condition = Treatment H 0 (null hypothesis): there is no effect H A or H 1 (alternative hypothesis): there is an effect 57

58 Hypothesis testing variables Independent variables: the thing(s) you assign / vary Dependent variables: the thing(s) you measure for evidence of an effect Co-variates: other aspects of a participant that might explain some of the effect (e.g., age, technical expertise, etc.) 58

59 P values and statistics Much of hypothesis testing involves calculating an appropriate statistic p value: probability of observing an effect at least as extreme as observed assuming the null hypothesis is true (i.e., no effect) α (alpha): cutoff for rejecting H 0 Treat this as a binary decision Often α =.05 in usable security 59

60 Is testing for significance enough? No! Consider: Effect size (magnitude of the effect of the manipulation) Power (long-term probability of rejecting H 0 if there really is a difference) Type 1 error: wrongly reject H 0 even if there is no effect (α) Type 2 error: wrongly fail to reject H 0 even if there is an effect (β) 60

61 Type I Errors Type I error (false positive) You would expect this to happen 5% of the time if α = 0.05 What happens if you conduct a lot of statistical tests in one experiment? 61

62 Contrasts 62

63 Contrasts If we determine that the variables are dependent, we may compare conditions Planned vs. unplanned contrasts You have a limited number of planned contrasts (depending on the DF) for which you don t need to correct p values. Bonferroni correction (multiply p values by the number of tests) is the easiest to calculate but most conservative 63

64 Type II Errors Type II error (false negative) There is actually a difference, but you didn t see evidence of a difference Statistical power is the probability of rejecting the null hypothesis if you should You could do a power analysis, but this requires that you estimate the effect size 64

65 PICKING THE RIGHT TEST 65

66 Not all tests are created equal Different types of dependent and independent variables? Different tests! Different data distributions? Different assumptions Different tests!! Parametric vs non-parametric 66

67 Dependent Variable Which tests are we learning about today? Focusing on parametric tests! Independent Variable Categorical Quantitative Categorical Chi-Squared Test Fisher s Exact Test Logistic Regression Quantitative t-test ANOVA Correlation Linear Regression 67

68 DV: CATEGORICAL IV: CATEGORICAL 68

69 (Pearson s) Chi-squared (χ 2 ) Test Examples: Does the gender (male, female) of the unicorn correlate with a unicorn s favorite color? Does the type of food it eats correlate to its privacy concerns? H 0 : Variable X factors are equally distributed across variable Y factors (independence) (Not covered today) Goodness of fit: Does the distribution we observed differ from a theoretical distribution? 69

70 Contingency tables Rows are one variable, columns the other χ 2 = , df = 14, p = 1.767e-14 70

71 Chi-squared (χ 2 ) Notes Use χ 2 if you are testing if one categorical variable (usually the assigned condition or a demographic factor) impacts another categorical variable If you have fewer than 5 data points in a single cell, use Fisher s Exact Test Do not use χ 2 if you are testing quantitative outcomes! 71

72 What are Likert-scale data? Some people treat it as continuous (meh!) Other people treat it as ordinal (ok!) You can use Mann-Whitney U / Kruskal-Wallis (non-parametric) A simple way to compare the data is to bin (group) the data into binary agree and not agree categories (ok!) You can use χ 2 (parametric) 72

73 DV: CATEGORICAL IV: QUANTITATIVE 73

74 Choosing a numerical test Do your data follow a normal (Gaussian) distribution? (You can calculate this!) Image from If so, use parametric tests. If not, use non-parametric tests Does the data set have equal variance? Are your data independent? If not, repeated-measures, mixed models, etc. 74

75 Independence Why might your data not be independent? Non-independent sample (bad!) The inherent design of the experiment (ok!) If you have two data points of unicorns race completion times (before and after some treatment), can you actually do a single test that assumes independence to compare conditions? 75

76 Numerical data Are values bigger in one group? Normal, continuous data (compare mean): H 0 : There are no differences in the means. 2 conditions: t-test 3+ conditions: ANOVA Non-normal data / ordinal data: H 0 : No group tends to have larger values. 2 conditions: Mann-Whitney U (AKA Wilcoxon ranksum test) 3+ conditions: Kruskal-Wallis 76

77 DV: QUANTITATIVE 77

78 Correlation Usually less good: Pearson correlation Requires that both variables be normally distributed Only looks for a linear relationship Often preferred: Spearman s rank correlation coefficient (Spearman s ρ) Evaluates a relationship s monotonicity always going in the same direction or staying the same 78

79 Correlation DOES NOT imply causation 79

80 Regressions What is the relationship among variables? Generally one outcome (dependent variable) Often multiple factors (independent variables) The type of regression you perform depends on the outcome Binary outcome: logistic regression Ordinal outcome: ordinal / ordered regression Continuous outcome: linear regression 80

81 Interactions in a regression Normally, outcome = ax 1 + bx 2 + c + Interactions account for situations when two variables are not simply additive. Instead, their interaction impacts the outcome e.g., Maybe silver unicorns, and only silver unicorns, get a much larger benefit from eating pop-tarts before a race Outcome = ax 1 + bx 2 + c + d(x 1 x 2 ) + 81

82 Example (logistic) regression Outcome: completed unicorn race (or not) Independent variables: Age Number of prior races Diet: hay or pop-tarts (Indicator variables for color categories) Etc. 82

83 WHAT IF THERE IS NO INDEPENDENCE? 83

84 Non-independence Repeated measures (multiple measurements of the same thing) e.g., before and after measurements of a unicorn s time to finish a race Paired t-test (two samples per participant, two groups) Repeated measures ANOVA (more general) 84

85 Non-independence For regressions, use a mixed model Random effects based on hierarchy/group Case 1: Many measurements of each unicorn Case 2: The unicorns have some other relationship. e.g., there are 100 unicorns each trained by one of 5 trainers. The identity of the trainer might impact a whole class of unicorns performance. 85

86 What if you have lots of questions? If we ask 40 privacy questions on a Likert scale, how do we analyze this survey? One technique is to compute a privacy score by adding their responses Make sure the scales are the same (e.g., don t add agreement with privacy is dumb and privacy is smart reverse the scale) You should verify that responses to the questions are correlated! 86

87 What if you have lots of questions? Another option: factor analysis, which evaluates the latent (underlying) factors You specify N, a number of factors Puts the questions into N groups based on their relationships You should examine factor loadings (how well each latent factor correlates with a question) Generally, you want questions to load primarily onto a single factor to be confident 87

88 Picking a test est%20flow%20chart.pdf stics.html DS/Page% %20- %20Choosing%20the%20correct%20statistical%20test% 20made%20easy.pdf 88

89 What can we conclude statistically X varies in a way that s related to Y As the age of a unicorn increases, its max speed decreases Pearson s correlation / Spearman s correlation Assignment to X impacts Y (category) Unicorns randomly assigned to eat vegan food (as opposed to non-vegan food) are more likely to be rated as successful (as opposed to unsuccessful) χ 2, Fisher s exact test 89

90 What can we conclude statistically Assignment to X impacts Y (numerical) Unicorns randomly assigned to eat vegan food (as opposed to non-vegan food) are more likely to take a shorter time to run a race ANOVA, Kruskal-Wallis, etc. Lots of factors impact Y (category) Logistic regression Lots of factors impact Y (numerical) Regression 90