Comparing multiple proportions

Transcription

1 Comparing multiple proportions February 24, 2017 psych10.stanford.edu

2 Announcements / Action Items Practice and assessment problem sets will be posted today, might be after 5 PM Reminder of OH switch today

3 Last time Some ways to integrate previous material We can make better predictions about individual observations when considering the variable group, and we can quantify the improvement in these predictions using r 2 We can remove variance related to individual differences by using paired designs, but we must adjust our analysis approach to account for the fact that observations in each pair are not independent (single mean or single proportion)

4 This time Our final cycle with frequency (count) data and proportions Categorical variable: responses or are categories or groups ( levels ) Goal: handle response variables with any number of categories and grouping variables with any number of groups with a single statistic (why not compare each possible pair? we ll discuss on Monday) binary variable(s) any categorical variable(s) one variable (a response variable) z-test for a single proportion (1 x 2 table) chi-square test for goodness-of-fit (1 x any # table) two variables (one grouping, one response) z-test for a difference in proportions (2 x 2 table) chi-square test for independence (any # by any # table)

5 This time How can we test whether observed proportions are consistent with expected proportions? How can we use proportions to test whether two variables are associated?

6 This time How can we test whether observed proportions are consistent with expected proportions? How can we use proportions to test whether two variables are associated?

7 Previously converted non-binary variables to binary variables rock-paper-scissors rock paper scissors 1/3 1/3 1/3 rock-paper-scissors not scissors scissors 2/3 1/3 roll reported /6 1/6 1/6 1/6 1/6 1/6 roll reported not 6 6 5/6 1/6 section called upon 1st 2nd 3rd 4th 1/4 1/4 1/4 1/4 section called upon 1st not 1st 1/4 3/4

8 Today assess fit of all categories to hypothesized distribution rock-paper-scissors rock paper scissors 1/3 1/3 1/3 rock-paper-scissors not scissors scissors 2/3 1/3 roll reported /6 1/6 1/6 1/6 1/6 1/6 roll reported not 6 6 5/6 1/6 section called upon 1st 2nd 3rd 4th 1/4 1/4 1/4 1/4 section called upon 1st not 1st 1/4 3/4

9 Chi square: goodness-of-fit Goal: use observed sample proportions to test hypotheses about unobserved population proportions looking at a single variable General approach: Generate a null hypothesis (H 0 ) about population proportions Compute frequencies that we would expect if H 0 was true usually the hardest step Summarize the discrepancy between these with a single statistic (Χ 2 ) Use the distribution of all of the statistics that we could have observed if the null hypothesis was true to determine whether this statistic would be unlikely if the null hypothesis was true

10 Academic dishonesty Are cases of academic dishonesty evenly distributed across departments?

11 Academic dishonesty A simplified scenario. A university has four, equally sized departments: computer science, biology, sociology, and art history. They handle 100 honor code cases, as below. observed frequencies, Oi CS Bio Soc Art Total A null hypothesis: students in each department are equally likely to be included in these cases expected proportions CS Bio Soc Art Total

12 Academic dishonesty A simplified scenario. A university has four, equally sized departments: computer science, biology, sociology, and art history. They handle 100 honor code cases, as below. observed frequencies, Oi CS Bio Soc Art Total A null hypothesis: students in each department are equally likely to be included in these cases expected frequencies, Ei (multiply expected proportion by n) CS Bio Soc Art Total 0.25 * 100 = * 100 = * 100 = * 100 = 25 1 * n = 100

13 Academic dishonesty Do we expect the observed frequencies to perfectly match the expected frequencies if the null hypothesis is true? How likely is it to find a discrepancy this extreme or more if our null hypothesis is true? summarize discrepancy with a single statistic, Χ 2 observed frequencies, Oi CS Bio Soc Art Total expected frequencies, Ei CS Bio Soc Art Total 0.25 * 100 = * 100 = * 100 = * 100 = 25 1 * n = 100

14 Academic dishonesty 2 = X (O i E i ) 2 observed frequencies, Oi CS Bio Soc Art Total expected frequencies, Ei CS Bio Soc Art Total 0.25 * 100 = 25 E i numerator: discrepancy, squared why do we square it? denominator: divided by expected frequency (normalized) less surprised to be off by 4 if you expect 1000 than if you expect 8 sum terms from each level (each category) 0.25 * 100 = * 100 = * 100 = 25 1 * n = 100

15 Academic dishonesty 2 = X (O i E i ) 2 2 = observed frequencies, Oi CS Bio Soc Art Total expected frequencies, Ei CS Bio Soc Art Total 0.25 * 100 = 25 E i (35 25) (30 25) * 100 = 25 + (20 25) = = = * 100 = 25 + (15 25) * 100 = 25 1 * n = 100

16 Academic dishonesty A (slightly less) simplified scenario. A university has four departments: computer science, biology, sociology, and art history, which include 30%, 30%, 20% and 20% of the students, respectively. They handle 100 honor code cases, as below. observed frequencies, Oi CS Bio Soc Art Total A null hypothesis: students in each department are equally likely to be included in these cases expected frequencies, Ei (multiply expected proportion by n) CS Bio Soc Art Total 0.30 * 100 = * 100 = * 100 = * 100 = 20 1 * n = 100

17 Academic dishonesty 2 = X (O i E i ) 2 2 = E i (35 30) (30 30) = observed frequencies, Oi (20 20) (15 20)2 20 = = 2.08 CS Bio Soc Art Total expected frequencies, Ei CS Bio Soc Art Total 0.30 * 100 = * 100 = * 100 = * 100 = 20 1 * n = 100

18 We must take the base rate into account when we specify what we expect to see As a consumer of statistics, beware of raw counts!

19 The Χ 2 distribution(s) Is Χ 2 = 10 unlikely? Is Χ 2 = 2.08 unlikely? When some assumptions are met, the potential Χ 2 statistics we could observe if the null hypothesis is true are described by a Χ 2 distribution ( a sampling distribution of Χ 2 statistics) Like the family of t-distributions, we have a family of Χ 2 distributions, specified by degrees of freedom (df) Here, df specifies how many cell counts are free to vary For a goodness-of-fit test, df = # of categories - 1 observed frequencies, Oi CS Bio Soc Art Total

20 The Χ 2 distribution(s) Is Χ 2 = 10 unlikely? Is Χ 2 = 2.08 unlikely? 2 = X (O i E i ) 2 E i Think about the distribution of Χ 2 values that we could observe What shape do you expect? (Hint: can Χ 2 be negative?) positively skewed How does df affect central tendency? as df increases, central tendency increases How does df affect variability? as df increases, variability increases How does df affect shape? as df increases, the shape becomes less skewed Are we interested in unusually low Χ 2, unusually high Χ 2, or both? only unusually high values, unusually low values are consistent with H 0 (imagine X 2 = 0, what would this mean?) only interested in the upper tail, which includes deviations in any direction

21 The Χ 2 distribution(s) Is Χ 2 = 10 unlikely? Is Χ 2 = 2.08 unlikely? 2 = X (O i E i ) 2 E i Think about the distribution of Χ 2 values that we could observe What shape do you expect? (Hint: can Χ 2 be negative?) positively skewed How does df affect central tendency? as df increases, central tendency increases How does df affect variability? as df increases, variability increases How does df affect shape? as df increases, the shape becomes less skewed Are we interested in unusually low Χ 2, unusually high Χ 2, or both? only unusually high values, unusually low values are consistent with H 0 (imagine X 2 = 0, what would this mean?) only interested in the upper tail, which includes deviations in any direction

22 The Χ 2 distribution(s) Is Χ 2 = 10 unlikely? Is Χ 2 = 2.08 unlikely? Is Χ 2 = 10 unlikely? p(x 2 10) = Χ 2

23 The Χ 2 distribution(s) Is Χ 2 = 10 unlikely? Is Χ 2 = 2.08 unlikely? Is Χ 2 = 2.08 unlikely? p(x ) = Χ 2

24 Hypothesis test in simplified scenario Two hypotheses: H0: students in each department are equally likely to be included in honor code cases the proportions of students involved across departments are.(25,.25,.25,.25) for CS, biology, sociology, and art history HA: the students in each department are not equally likely to be included in honor code cases it is not the case that the proportions of students involved across departments are (.25,.25,.25,.25) for CS, biology, sociology, and art history (this is not the same as saying that none of these proportions is.25)

25 Hypothesis test in simplified scenario The probability of observing frequencies as or more extreme as our sample frequencies if the null hypothesis was true is.019 if α =.05, reject the null hypothesis and infer that the true proportions are not (.25,.25,.25,.25) We still do not know exactly which proportions are significantly different from what we expect can perform follow-up tests why the proportions are not what we expect this is observational this is a measure of people who are caught and reported why might this differ across departments?

26 Benford s law danielfisher/2014/05/30/the-simplemathematical-law-that-financialfraudsters-cant-beat/#56886f5b4612

27 Benford s law Class survey collected the first digit of your previous address Does the distribution of digits follow Benford s law? (Does it deviate from the law beyond what we would expect by random fluctuations?). H 0 : the distribution of first digits follows Benford s law H A : the distribution of first digits does not follow Benford s law What could it mean if we rejected the null hypothesis? inaccuracy in reporting? something non-random about Stanford student addresses? Type I Error? something else? a small cautionary note about this example, it is best to have at least 10 observations per cell

28 Benford s law Benford s law states that first digits follow a distribution of: total expected probabilities multiply by n Ei Our class data of first digits of previous addresses: total Oi n = 141 (Oi - Ei) 2 / Ei Χ 2 = 11.71

29 Benford s law Benford s law states that first digits follow a distribution of: total expected probabilities multiply by n Ei Our class data of first digits of previous addresses: total Oi n = 141 (Oi - Ei) 2 / Ei Χ 2 = 11.71

30 Benford s law Benford s law states that first digits follow a distribution of: total expected probabilities multiply by n Ei Our class data of first digits of previous addresses: total Oi n = 141 (Oi - Ei) 2 / Ei Χ 2 = pchisq(11.71, df = 8, lower.tail = FALSE) p =.16

31 Benford s law Class survey collected the first digit of your previous address Does the distribution of digits follow Benford s law? (Does it deviate from the law beyond what we would expect by random fluctuations?). H 0 : the distribution of first digits follows Benford s law H A : the distribution of first digits does not follow Benford s law Fail to reject the null hypothesis, conclude that we do not have evidence that the distribution of first digits is inconsistent with Benford s law

32 The role of sample size law of large numbers ask, where is n in this calculation? 2 = X (O i E i ) 2 E i = X (nˆp i n i ) 2 n i = X n 2 (ˆp i i ) 2 n i = X n(ˆp i i ) 2 i = n X (ˆp i i ) 2 i = sample size * effect size

33 Interim summary We ve extended the ideas behind asking whether an observed distribution of a binary variable is consistent with a hypothesized distribution to asking whether an observed distribution of any categorical variable is consistent with a hypothesized distribution! binary variable(s) any categorical variable(s) one variable (a response variable) z-test for a single proportion (1 x 2 table) chi-square test for goodness-of-fit (1 x any # table) two variables (one grouping, one response) z-test for a difference in proportions (2 x 2 table) chi-square test for independence (any # by any # table)

34 This time How can we test whether observed proportions are consistent with expected proportions? How can we use proportions to test whether two variables are associated?

35 Previously analyzed independence of 2 x 2 table, comparing difference in observed proportions to difference we would expect by random chance group 1 group 2 group 3 response A response B response C

36 Today analyzed independence of any size table, comparing difference in observed proportions to difference we would expect by random chance group 1 group 2 group 3 response A response B response C

37 Chi-square: independence Goal: use observed sample proportions to test hypotheses about unobserved population proportions specifically, are two variables associated or are they independent? General approach: Null hypothesis (H 0 ) is that our variables are independent (no association) between them we expect population proportions to follow certain rules due to this independence Compute frequencies that we would expect if H 0 was true usually the hardest step Summarize the discrepancy between these with a single statistic (Χ 2 ) Use the distribution of all of the statistics that we could have observed if the null hypothesis was true to determine whether this statistic would be unlikely if the null hypothesis was true

38 Hand washing and city hand washing study: a 2005 study observed hand-washing behavior in public restrooms in four major cities Observed Atlanta Chicago NY SF Total Washed Did not wash Total

39 Marginal probabilities ignoring city probability of washing, 5194 / 6336 =.82 probability of not washing, 1142 / 6336 =.18 *caution of self-report, in a parallel telephone survey 91% of people reported washing ignoring washing probability of Atlanta, 1588 / 6336 =.25 probability of Chicago, 1509 / 6336 =.24 probability of New York, 1503 / 6336 =.24 probability of San Francisco, 1736 / 6336 =.27 Observed Atlanta Chicago NY SF Total Washed Did not wash Total

40 Conditional probabilities p(washing Atlanta) = 1175 / 1588 =.74 p(washing Chicago) = 1329 / 1509 =.88 p(washing NY) = 1169 / 1503 =.78 p(washing SF) = 1521 / 1736 =.88 comparing distributions all cities have a mode of washing Chicago and SF have lower variability (higher relative freq. at the mode) Observed Atlanta Chicago NY SF Total Washed Did not wash Total

41 Independence of city and washing two hypotheses: H0: there is no association between city and washing HA: there is an association between city and washing Observed Atlanta Chicago NY SF Total Washed Did not wash Total

42 Reminder: independence of A and B If A and B are independent then p(a) * p(b) = p(a and B) p(a B) = p(a and B) / p(b) = p(a) * p(b) / p(b) = p(a) p(a not B) = p(a and not B) / p(not B) = p(a) * p(not B) / p(not B) = p(a) Here, independent means there is no relationship between the grouping variable and the response variable

43 Independence of city and washing what frequencies would we expect to see if city and washing were independent? p(a and B) = p(a) * P(B) consider frequency of (Atlanta and washed) = n * p(atlanta and washed) = n * p(atlanta) * p(washed) = n * (# Atlanta / n) * (# washed / n) = (# Atlanta * # washed) / n = (5194 * 1588) / 6336 = Atlanta Chicago NY SF Total Washed 5194 Did not wash 1142 Total

44 Independence of city and washing what frequencies would we expect to see if city and washing were independent? p(a and B) = p(a) * P(B) more generally, frequency of (A and B) = (# A * # B) / n = (# row * # column) / n Atlanta Chicago NY SF Total Washed 5194 Did not wash 1142 Total

45 Independence of city and washing what frequencies would we expect to see if city and washing were independent? p(a and B) = p(a) * P(B) more generally, frequency of (A and B) = (# A * # B) / n = (# row * # column) / n Expected Atlanta Chicago NY SF Total Washed 5194 * 1588 / * 1509 / * 1503 / * 1736 / Did not wash 1142 * 1588 / * 1509 / * 1503 / * 1736 / Total

46 Independence of city and washing what frequencies would we expect to see if city and washing were independent? p(a and B) = p(a) * P(B) more generally, frequency of (A and B) = (# A * # B) / n = (# row * # column) / n Expected Atlanta Chicago NY SF Total Washed Did not wash Total

47 Independence of city and washing Χ 2 = Ʃ((O i -E i ) 2 /E i ) = ( ) 2 / ( ) 2 / Observed Atlanta Chicago NY SF Total Washed Did not wash Expected Atlanta Chicago NY SF Total Washed Did not wash Total Total

48 Independence of city and washing Χ 2 = Ʃ((O i -E i ) 2 /E i ) = ( ) 2 / ( ) 2 / Χ 2 = Ʃ((Oi-Ei) 2 /Ei) = ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / =

49 Degrees of freedom how many cells are free to vary? df = (# rows - 1) * (# columns - 1) = (2-1) * (4-1) = 1 * 3 = 3 > pchisq(161.74, df = 3, lower.tail = FALSE) [1] e-35 Observed Atlanta Chicago NY SF Total Washed constraint 5194 Did not wash constraint constraint constraint constraint 1142 Total

50 Independence of city and washing two hypotheses: H0: there is no association between city and washing HA: there is an association between city and washing but we don t know which cities have different proportions from each other inference or estimation with pairwise proportion differences Observed Atlanta Chicago NY SF Total Washed Did not wash Total

51 Placebic information A researcher attempts to cut someone in line for the photocopier to make copies of 5 pages with three strategies (adapted from Langer et al., 1978) (1) no info may I use the Xerox machine (2) real info may I use the Xerox machine, because I m in a rush (3) placebic info may I use the Xerox machine, because I need to make copies is form of request associated with rate of compliance? Observed none real placebic Total Yes No Total

52 Marginal probabilities ignoring form of request probability of yes, 227 / 292 =.78 probability of no, 65 / 292 =.22 ignoring compliance probability of none, 90 / 292 =.31 probability of real, 100 / 292 =.34 probability of placebic, 102 / 292 =.35 Observed none real placebic Total Yes No Total

53 Conditional probabilities p(yes none) = 50 / 90 =.56 p(yes real) = 90 / 100 =.90 p(yes placebic) = 87 / 102 =.85 comparing distributions all request types have a mode of yes no information has greatest variability (lowest relative freq. at the mode) Observed none real placebic Total Yes No Total

54 Independence of request and compliance two hypotheses: H 0 : there is no association between request type and compliance H A : there is an association between request type and compliance Observed none real placebic Total Yes No Total

55 Independence of request and compliance what frequencies would we expect to see if request type and compliance were independent? p(a and B) = p(a) * P(B) more generally, frequency of (A and B) = (# A * # B) / n = (# row * # column) / n none real placebic Total Yes 227 No 65 Total

56 Independence of request and compliance what frequencies would we expect to see if request type and compliance were independent? p(a and B) = p(a) * P(B) more generally, frequency of (A and B) = (# A * # B) / n = (# row * # column) / n Expected none real placebic Total Yes 227 * 90 / 292 = * 100 / 292 = * 102 / 292 = No 65 * 90 / 292 = * 100 / 292 = * 102 / 292 = Total

57 Independence of request and compliance Χ 2 = Ʃ((O i -E i ) 2 /E i ) = ( ) 2 / ( ) 2 / Observed none real placebic Total Yes No Total Expected none real placebic Total Yes No Total

58 Independence of request and compliance Χ 2 = Ʃ((O i -E i ) 2 /E i ) = ( ) 2 / ( ) 2 / Χ 2 = Ʃ((Oi-Ei) 2 /Ei) = ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / ( ) 2 / = 37.66

59 Independence of request and compliance how many cells are free to vary? df = (# rows - 1) * (# columns - 1) = (2-1) * (3-1) = 1 * 2 = 2 > pchisq(37.66, df = 2, lower.tail = FALSE) [1] e-09 Observed none real placebic Total Yes No Total

60 Independence of request and compliance two hypotheses: H 0 : there is no association between request type and compliance H A : there is an association between request type and compliance but we don t know (for sure) which request types have different proportions from each other inference or estimation on pairwise proportion differences Observed none real placebic Total Yes No Total

61 Interim summary We ve extended the ideas behind asking whether there is an association between two binary variables to asking whether there is an association between any categorical variables! binary variable(s) any categorical variable(s) one variable (a response variable) z-test for a single proportion (1 x 2 table) chi-square test for goodness-of-fit (1 x any # table) two variables (one grouping, one response) z-test for a difference in proportions (2 x 2 table) chi-square test for independence (any # by any # table)

62 If you re curious A Χ 2 distribution with k degrees of freedom describes the the distribution of k independent observations that come from normal distributions with a mean of 0 and a standard deviation of 1 (a z-distribution) that are squared and summed The distribution of SS = Ʃ(x - μ) 2 can be linked to a Χ 2 distribution (after some maneuvering) The t-distribution and F-distribution (coming up) are derived in part by using the Χ 2 distribution

63 Recap If we have a single categorical variable, we can use a Χ 2 test for goodness-of-fit to compare the proportions of responses in each category to a hypothesized model If we have multiple categorical variables, we can use a Χ 2 test for independence to ask whether the two variables are associated (not independent) or not associated (independent)

64 Questions