Reasoning about probabilities (cont.); Correlational studies of differences between means

Transcription

1 Reasoning about probabilities (cont.); Correlational studies of differences between means Phil 12: Logic and Decision Making Fall 2010 UC San Diego 10/29/2010

2 Review You have found a correlation in a sample How likely is it that there is a correlation in that direction in the actual population? At what level is the correlation statistically significant? p <.05: less than 1 chance in 20 that the result is due to chance factors in drawing the sample p <.01: less than 1 chance in 100 that the result is due to chance factors in drawing the sample Choose significance level (p < some value) according to how important it is not to be wrong Statistical significance and importance are very different things

3 Review Humans are notoriously bad at determining whether two variables are correlated We see correlations that don t exist We fail to see correlations that do exist Some surprising results are simply sampling phenomena, so don t require special explanation

4 Clicker question 1 Last month you took the SAT/GRE and scored 750 out of a possible 800 on the quantitative part. For kicks, you decide to take the test again with different questions but of same difficulty assume no learning or practice effect from first test What score should you/we predict for you on the second test? A. 750 B. above 750 C. below 750

5 Limits to regression analysis: Regression to the mean Last month you took the SAT/GRE and scored 750 out of a possible 800 on the quantitative part. For kicks, you decide to take the test again with different questions but of same difficulty assume no learning or practice effect from first test What score should you/we predict for you on the second test? The surprising answer is that the person is more likely to score below 750 than above 750; the best guess is that the person would score about 725.

6 Regression to the mean Phenomenon discovered by Francis Galton, half cousin of Charles Darwin Developed a regression analysis of height between human children and their parents Found that "It appeared from these experiments that the offspring did not tend to resemble their parents in size, but always to be more mediocre than theyto be smaller than the parents, if the parents were large; to be larger than the parents, if the parents were small."

7 A way to understand regression to the mean A given test is really a sampling from a distribution Assume that there is a large number, say 1,000 forms of a test and that you take all 1,000 tests there are no learning, practice, or fatigue effects Scores will be distributed: Frequency Score

8 A way to understand regression to the mean 2 Identify the mean of this distribution as the true score Differences in the scores on these tests are due to chance factors: guessing knowing more of the answers on some tests than on others Frequency True score

9 A way to understand regression to the mean 3 How could a first score of 750 have arisen: It reflected the true score (all chance factors balanced out) Your true score was <750 and you scored above it due to chance factors pushing you up Your true score was >750 and you only scored 750 due to chance factors dragging you down Which is more likely? There are very few people with "true" scores above 750 (roughly 6 in 1,000) There are many more people with true scores between 700 and 750 (roughly 17 in 1,000). Thus, it is more likely that you are from the latter group

10 Regression to the mean For any situation where there is some random variability affecting the outcome: a variable that is extreme (high or low) on its first measurement will tend to be closer to the mean of the distribution on a later measurement Due to probability of selecting samples with particular values true score = mean of sampling distribution of scores for extreme high (low) scores, more likely their true score is less different from the mean than when they obtained a very high (low) score

11 More regression examples Sales representative who had a spectacular or horrible year The sophomore slump : Almost 9/10 rookies of the year perform worse in their second year than in their rookie year Of 58 Cy Young Award winners, 52 had fewer victories the next year and 50 had higher earnedrun average Hitters who hit 30 home runs before midseason hit fewer thereafter, and those who hit 30 in the second half hit fewer before midseason

12 Regression and punishment Makes it seem like punishment works: When someone is doing particularly poorly (for them), chastising them seems to result in better performance But praising someone does not seem to work: When someone is doing particularly well (for them), praise is usually followed by poorer results Both can just be instance of regression to the mean! Nature operates in such a way that we often feel punished for rewarding others and rewarded for punishing them (David Myers, Intuition, p. 148).

13 Watch out for pseudo explanations A program proposes to help those who score at the very bottom end of a standardized test For example, intervenes with those scoring less than 300 on the SAT After the intervention, the individuals are tested again A larger proportion of this group exhibits improved scores than decreased scores The program claims success. But it may have contributed nothing! The results might totally be due to regression

14 Clicker question 2 What if I flipped a coin 4 times and got the following results: H T T T What do you think the result of the next flip will be? A. Heads B. Tails

15 Do streaks require explanation? THTTTHHTTT HTTTTHTHHH HHHTTHTHTT THHHHHTTTT HTTTTTTTTH THHHTTHTHH TTHTHTHTHH HHHHHHHHTT HHHHHTHHHT THHTTTHHTT Streaks are expected! For 2 flips, P(2H or 2T) = 2/4 = 0.5 For 3 flips, P(3H or 3T) = 2/8 = 0.25 For 4 flips, P(4H or 4T) = 2/16 = For 5 flips, P(5H or 5T) = 2/32 = To assume a streak must end to even out the distribution of outcomes is to commit the gambler s fallacy

16 Hot hand? If someone just hit three shots in a row, is it a good idea to pass to them? What if they had missed three in a row? Philadelphia 76ers' game data from the season (using all shots from the field) success on next shot Three Straight Hits.46 Two Straight Hits.50 One Hit.51 One Miss.54 Two Straight Misses.53 Three Straight Misses.56 Source: Gilovich, Vallone, and Tversky (1985, Cognitive Psychology, Table 1)

17 Clicker question 3 Frank was drawn at random from a group of 30 lawyers and 70 engineers. He spends most of his free time hanging around his country club. At the bar he often talks about his regrets at having tried to follow in his esteemed father s footsteps. The long hours he spent slaving in school could have been better spent learning to be less quarrelsome in his relationships with other people. Is Frank a lawyer or an engineer? A. Frank is a lawyer B. Frank is an engineer

18 What to base predictions on? In trying to make predictions, we very often ignore the most important variable for making a prediction With regard to whether Frank is a lawyer or engineer: Frank was drawn at random from a group of 30 lawyers and 70 engineers. He spends most of his free time hanging around his country club. At the bar he often talks about his regrets at having tried to follow in his esteemed father s footsteps. The long hours he spent slaving in school could have been better spent learning to be less quarrelsome in his relationships with other people.

19 In a city in which two cab companies, Blue and Green, operate, a taxicab was involved in a hit and run accident one night 85% of the cabs in the city are Green, 15% Blue A witness identified the cab as Blue The Court tested the ability of the witness to identify cab colors under appropriate visibility conditions He/she made the correct identification 80% of the time What is the probability that the cab involved was Blue? Confident enough to convict? A. Yes B. No Clicker question 4

20 What to base legal decisions on? Blue Said Blue Said Green 12 3 Totals 15 85% Green 15% Blue Green Totals witness 80% accurate Of the times he/she said it was blue, how often was it blue? 12/29 or 41% Is <50% accuracy good enough to convict?

21 Base rates Base rates are often the best predictor It matters greatly whether the population was 70/30 lawyers or 70/30 engineers It matters greatly that 85% of the cabs were Green This trumps the witness s 80% accuracy! But humans almost universally ignore base rates i.e., commit the base rate fallacyif there is anything else on which to base the decision Police, lawyers, scientists, doctors...

22 Clicker question 5 Fred is a 40 something college professor active in social causes and very concerned about the environment. Which is more likely? A. Fred is a Republican B. Fred is a Republican who recycles his trash

23 Dealing with probabilities rationally Fred is a 40 something college professor active in social causes and very concerned about the environment. How likely is it that: Fred is a Republican? Fred recycles his trash? Fred is a Republican who recycles his trash? Fred is a Republican and recycles his trash cannot be more likely than the other two!

24 Combining probabilities Two events are independent if the probability of one does not depend on the probability of the other You determine the joint probability for independent events by multiplying the probabilities of each event Since probabilities have values in the range 0 to 1, the product will never be larger than the probability of the more likely event to think so is to commit the conjunction fallacy If the events are not independent, one has to adjust the formula, but multiplication is still the central operation and the above point still holds

25 Combining probabilities Reference class: US citizens is a Republican recycles their trash

26 Summary: Mistakes in reasoning about probabilities Some surprising results are simply sampling phenomena, and so don t require special explanation Regression to the mean Streaks in samples Sampling phenomena involve probabilities Humans commonly make mistakes when reasoning about probabilities Gambler s fallacy Base rate fallacy Conjunction fallacy