Soci708 Statistics for Sociologists

Transcription

1 Soci708 Statistics for Sociologists Module 3 Producing Data 1 François Nielsen University of North Carolina Chapel Hill Fall Adapted in part from slides for course taught by John Fox (McMaster University) and Robert Andersen (University of Toronto) 1 / 59

2 Producing Data Main Sections Introduction Design of Experiments Sampling Design Toward Statistical Inference 2 / 59

3 Anecdotal Evidence Anecdotal evidence is based on haphazardly selected individual cases, which often come to our attention because they are striking in some way. These cases need not be representative of any larger group of cases. Examples: Your hairdresser said she knows three children in town who live near a power line and who have been diagnosed with leukemia. At a congressional hearing a woman testified that her son was diagnosed with autism six months after being vaccinated. Anecdotal evidence does not establish an association between variables, let alone that one is cause of the other. 3 / 59

4 Available Data Available data are data that were produced in the past for some other purposes but that may help answer a present question. Examples: Historical records of the numbers of young men and women graduating from high school (kept since 1870) reveal that until about the 1970s, women in the U.S. graduated at a higher rate than men. The National Center for Health Statistics keeps records on the causes of death by age groups and gender. Principal causes of death of year old are accidents, homicide, and suicide; overall death rate is three times higher for men. A Belgian demographer uses parish records of the 1600s to study trends in age at marriage, fertility and infant mortality. 4 / 59

5 Sample Surveys and Experiments Sample Surveys In a sample survey a small subset of a larger population is systematically selected to provide information on the larger population. The General Social Survey has sampled 1,500 3,000 adults in the U.S. every other year since Opinion polls sample about 1,000 likely voters. Employment surveys such as the Current Population Survey (CPS) in the U.S. sample about 60,000 households. (Why so many?) 5 / 59

6 Sample Surveys and Experiments Observational Study vs. Experiment In an observational study we observe individuals and measure variables of interest but do not attempt to influence the response. In an experiment we deliberately impose some treatment on individuals and we observe their response. Example: Observation or experiment? A study of effect of cell phone use on risk of brain cancer matched 469 patients with brain cancer with cancer free individuals of same sex, age and race, and measured cell phone use. Researchers concluded that use of cell phone is not associated with brain cancer. 6 / 59

7 Sample Surveys and Experiments Problem of Observational Studies Study of 1364 infants followed up to grade 6 found that children who had been in daycare from infancy to age 4.5 found that time spent in daycare was associated with: getting along less well with others greater assertiveness more disobedience greater aggressiveness But: Effect of daycare is confounded with other characteristics of families using daycare. (Which?) 7 / 59

8 Introduction Terminology of Experimental Designs The individuals on which the experiment is done are the experimental units. When the units are humans, they are called subjects or participants. A specific experimental condition applied to the units is called a treatment. The explanatory variables in an experiment are often called factors. In multi-factorial experiments, each treatment is formed by combining a specific level (value) of each of the factors. 8 / 59

9 Comparative Experiments What is an Experiment? Key features of experiments are manipulation and control. More specifically, a change in the explanatory variable is imposed on subjects i.e., there is an intervention. Classical experiments meet the three criteria for causation: 1. An empirical association is established. 2. The cause precedes the effect in time. 3. Rival explanations can be eliminated (i.e., no confounding variables) Three types of experiments found in the Social Sciences: Natural experiments, field experiments, classical experiments. 9 / 59

10 Comparative Experiments Natural Experiments In a natural experiment a change in the environment occurs naturally. Measures of the response variable are taken before and after the event. Occasions are rare, but can be instructive. Examples: Homicide rates of U.S. states are compared before and after passage of gun control legislation. Rates of birth defects are compared before and after the marketing (at different times) of Thalidomide in British counties. Problem: no control group to compare to. 10 / 59

11 Field Experiments In a field experiment a change in environment is introduced in a natural setting. An example is the gastric freezing experiment in which patients ingesting a deflated balloon later filled with refrigerated liquid experienced relief from pain. Problems: No control group to compare to we don t know what would happen if the intervention had not taken place. Also sensitive to placebo effect: Spontaneous recovery from medical condition over time. Benefit from any treatment. Uncontrolled experiments in medicine and social sciences (as opposed to physical sciences) often biased by subject selection and placebo effects. Later controlled experiment showed gastric freezing was no better than placebo. 11 / 59

12 Randomized Comparative Experiments Structure of a Randomized Comparative Experiment A randomized comparative experiment is the strongest method for showing causation because it easily meets the requirements for causation. Steps: 1. Randomization of subjects into groups: Experimental group, and control group. 2. Pre-test: Measure response variable. 3. Treatment: Set value(s) of explanatory variable(s). 4. Post-test: Re-measure response variable. 5. Test for change in the response variable. 12 / 59

13 Randomized Comparative Experiments Random Assignment Most experiments have multiple groups. When all experimental units are allocated at random among all treatments, the design is called completely randomized. Subjects are placed into the experimental or control groups using a true random process: Each case has an equal chance of ending up in any group e.g., subjects names are put into a hat and pulled out randomly. It is a purely a mechanical process the researcher has no control over placement. Random assignment to groups may be done manually with a table of random digits, or with software (next slide). 13 / 59

14 Randomized Comparative Experiments Example of Random Group Assignment. * in Stata. * want 16 groups of 6 subjects. clear. set obs 96. generate lab=_n. generate random=runiform(). list in 1/ lab random sort random. list in 1/ lab random * first group includes subjects 57, 12, 73, / 59

15 Randomized Comparative Experiments Example of Random Group Assignment (Cont d). * in Stata * continue from previous slide lab random seq group. * now want to identify all * 16 groups of 6 subjects generate seq=_n generate group=ceil(seq/6) list in 1/ / 59

16 Randomized Comparative Experiments Overall Structure of a Classical Experimental Design Randomization Pretest Treatment Posttest Experimental Group Treatment Random Allocation Compare Differences in DV Control Group No treatment 16 / 59

17 Randomized Comparative Experiments E.g., Impact of TV violence on attitudes 1. Select a sample of children 2. Randomly divide the children into experimental and control groups 3. Measure their attitudes at the start 4. Show the children TV programs One group gets violent content, the other nonviolent programs 5. Re-test attitudes and examine for changes 6. Changes in attitude, even if temporary, can probably be attributed to the program (i.e., violent TV > violent attitudes) 17 / 59

18 Cautions About Experimentation Other considerations Ethics: Often necessary to deceive people Population: Must clearly define the eligible population Use of a Placebo: Prevents people from knowing which group they are in (avoids Hawthorne effect) Measurement: Standardized outcome measures (y) of known reliability and validity Double blind: Measurement by staff who do not know who is in which group 18 / 59

19 Cautions About Experimentation E.g., New teaching method and grades We want to improve grades in methods courses 1. We decide extra tutorial consulting with tutors may make a difference 2. On a test beforehand both the experiment and control group average 60% 3. I give extra teaching tutorial classes to my class, another instructor does not 4. We measure performance again: my class has an average of 55%, the other class has an average of 65% Can we conclude that the added tutorials had a detrimental effect? 19 / 59

20 Cautions About Experimentation Internal Validity: Was the treatment the true cause of a change in the dependent variable? 1. Unexpected causes An unexpected event occurs during the experiment that could affect the DV e.g., New computers make the course easier 2. Selection Bias Groups of subjects are not equivalent and there could be pre-existing differences among them with regard to the dependent variable e.g., Perhaps students chose to take the course from one instructor over the other 20 / 59

21 Cautions About Experimentation Internal Validity: Was the treatment the true cause of a change in the dependent variable? 3. Participant Attrition People drop out of the experiment Those who leave may differ from those remaining with regard to the dependent variable e.g., Maybe those who drop out do not like tutorials 4. Instrumentation & Testing Pre- and post measurements may not be comparable e.g., The two tests measure different things 21 / 59

22 Cautions About Experimentation Internal Validity: Was the treatment the true cause of a change in the dependent variable? 5. Diffusion of Treatment Contamination Subjects in the different groups communicate with each other e.g., control group works harder or control instructor feels sorry for control group so gives extra office hours to compensate 6. Experimenter Expectancy Researcher may want a result, and unintentionally relay this message to subjects. Subjects then want to please the researcher. Double-blind experiments prevent this 22 / 59

23 Cautions About Experimentation External Validity: The ability to generalize results to outside of the experimental conditions 1. Experimental Realism Can t always reproduce natural setting e.g., Can an experimental study on tactical voting taking place when there is not an election to tell us how people will really vote? 2. Participant Reactivity Subjects behave differently simply because they know they are being watched e.g., Hawthorne Effect 23 / 59

24 Matched Pairs Designs Alternative to completely randomized design can sometimes be more precise. One such alternative is the matched pairs design, in which subjects are matched in pairs with members chosen with the same age, sex, socio-economic status, and other variables thought to affect the response variable. BUT, in matching researcher must know which variables are important. One variation imposes both treatments on the same subject, so subjects serve as their own control. E.g., study of braking time compared the same subjects driving with and without using the cell phone. Randomization was used to decide which treatment (using cell ohone or not) came first for the subject. 24 / 59

25 Block Designs A block is a group of experimental units known in advance to be similar in some way expected to affect the response to treatments. In a block design the random assignment of units to treatments is carried out separately within each block. E.g., blocking by sex in a cancer experiment: 25 / 59

26 Science of Sampling While the individual man is an absolute puzzle, in the aggregate he becomes a mathematical certainty. You can never foretell what any one man will do, but you can say with precision what an average number will be up to. Individuals vary, but averages remain constant. Sir Arthur Conan Doyle 2 Source: Worcester, R British Public Opinion. London: Basil Blackwell, p Author of the Sherlock Holmes series 26 / 59

27 Adolphe Quetelet ( ) The Concept of l homme moyen (Average Man or Person) Born in Ghent, died in Brussels. Essai de physique sociale (1835) Invents notion of average person central to social statistics. Invents body-mass index (BMI): weight in kg BMI = (height in m) 2 Normal BMI range: Corresponds with Florence Nightingale ( ) 27 / 59

28 What is Sampling? Process of selecting a small number of cases (sample) in a way that it will accurately represent a larger number of cases (population) To provide useful descriptions of a population, the sample must contain essentially the same variations as the population If all members of society were identical in all respects, we would not need to sample Two Major Types of Samples: Non Probability Probability or Random 28 / 59

29 General Sampling Terms (1) Population: This is the group of study elements about which we want to make generalizations Finite population: E.g., all eligible voters in Canada Infinite population: Resulting from a process: e.g., computer chips made by a certain assembly line; members of species mus musculus Sampling Frame: List of cases in the population. Should include all elements once and only once: No duplications or omissions. Sampling Pool: List of numbers to choose from in random digit dialing Sampling Element: Each case that is being sampled from the population 29 / 59

30 General Sampling Terms (2) Sample: the part of the populaitn that we actually examine in order to gather information. Sampling Ratio: Percentage of population that is in the sample Sampling ratio = sample/population, e.g., sample size = 1000 population = 1, 000, 000 Sampling Ratio = 1000/1, 000, 000 = or 0.1% Population Parameter: True value of a feature (e.g. percentage, mean) in the whole population Statistic: Value of the feature in the sample data. Statistics are often used to estimate an unknown population parameter 30 / 59

31 Validity & Sampling Bias External validity: The degree to which the conclusions of a study would hold for other persons in other places and at other times. Does our sample represent the population? Sampling Bias: Those selected for the sample are not typical or representative of the population. Two types of sampling bias: Noncoverage: Some groups in the population are systematically left out of the process of choosing the sample E.g., homeless people with no telephone Nonresponse: Associated with survey research when an individual chosen for the sample canšt be reached or refuses to cooperate E.g., Current Population Survey (CPS) has low nonresponse rate (3% 4%); polls by opinion polling firms may be as high as 50% 60% 31 / 59

32 Non-probability Samples Haphazard samples (convenience samples; voluntary response samples) No plan usually not representative Quota samples Match proportion of selected groups to population Acceptable in exploratory research Purposive samples (judgmental samples) Acceptable for difficult to locate, special populations (e.g., homeless people) Snowball samples (network samples) Used in special situations when it is difficult to obtain a list of the population, but people know one another 32 / 59

33 Development of Scientific Sampling (1) Early Modern Polling ( ) Using convenience samples Literary Digest correctly predicted all US elections from 1920 to 1932 Literary Digest gained great prestige Disaster in 1936 election (Predicted Landon over FDR) 2,000,000 of 10,000,000 questionnaires returned Biased sampling frame based on LD subscription list, car & telephone ownership Excluded poor & low response rate 33 / 59

34 Development of Scientific Sampling (2) Era of Quota Sampling ( ) Gallup used quota sampling in 1936, correctly predicting FDR s win Quotas on gender, urban/rural, education, race Other firms began using quotas Disaster in 1948 election wrongly predicted Dewey victory; Truman won huge victory Quotas were not representative of population (they were based on 1940s census data which under-represented urban population) Stopped polling too soon 34 / 59

35 Simple Random Samples A probability or random sample (in general) is one chosen by chance, so that each possible sample has a known probability of being chosen Typically this condition is relaxed somewhat to mean: Each case has a known probability of being selected Outcomes are predictable in the long run over many cases Selection of cases is mechanical and thus rules out bias or influence by the researcher in the selection process Random sampling forms the basis of inferential statistics, i.e. deriving conclusions on a population on the basis of a sample from that population 35 / 59

36 Types of Random Sampling (1) Simple Random Samples (SRS) A simple random sample (SRS) is a probability sample chosen in such a way that each possible sample has the same probability of being chosen Informally, one in which each case has an equal chance of being selected SRS requires a good sampling frame It must be possible to reach all cases in the population to do it properly Seldom done in practice in social research (especially with respect to survey research) Often cannot get a population list It is not usually the most efficient method 36 / 59

37 Principle of Simple Random Sampling 37 / 59

38 Types of Random Sampling (2) Systematic Samples Short-cut form of random sampling (results often nearly identical to SRS) 1. Obtain a list of the population. 2. Create a sampling interval Sampling interval = Population size Sample size 3. Count cases and select every kth case, where k is the size of the interval Cannot be used when there is a pattern in the cases Important to begin with a random start, rather than with the first case 38 / 59

39 Stratified Samples Stratified Sampling Stratified, Random Sample Divide people or cases into homogeneous groups Select a random sample from each group Add the samples together to create a complete sample of the population Stratified, Systematic Sample Divide people or cases into homogeneous groups Put the groups together a continuous list Using a random start, select a systematic sample from the list 39 / 59

40 Stratified Samples 40 / 59

41 Multistage Samples Multistage Cluster Samples Used when cases are geographically distant or when population cannot be easily listed Steps: 1. Draw a sample from a collection of cases (clusters) e.g., select a sample of high schools in the U.S. 2. Sample individual cases from the clusters e.g., select samples of 36 students in 10th and 12th grades Probability Proportionate to Size (PPS) Used when clusters are of greatly differing sizes Each cluster is given a chance of selection that is proportionate to its size Caution: Multi-stage cluster samples are prone to high sampling error. Errors are compounded at each stage 41 / 59

42 Multistage Samples Example (1) Goal: A national election study. Want a sample of Problem: No complete population list Solution: Multi-Stage Cluster Sample 1. Start with a list of all parliamentary constituencies 2. Randomly select 30 constituencies 3. Randomly select 10 polling stations within each selected constituency 4. Randomly select 10 people from each selected polling station area 5. Add all the clusters together (N=3000) 42 / 59

43 Multistage Samples Example (2) Goal: Study the attitudes of Catholic women in England. Want a sample of Problem: No population list Solution: Multi-Stage Cluster Sample 1. Start with a list of all Catholic churches 2. Randomly select 10 geographic regions 3. Randomly select 10 churches from each region 4. Randomly select 10 women from congregation lists 5. Add all the clusters together (N=1000) 43 / 59

44 Cautions About Sample Surveys TBA 44 / 59

45 Determining Sample Size How much sampling error are you willing to accept? Sample size needed given desired confidence level & margin of error Confidence level 95% 90% Margin of error 5% 3% 5% 3% Population size Required sample size , , , , ,000, , / 59

46 Review of the Sampling Process 1. Decide on the population that you want to study 2. Determine the appropriate sampling method Probability samples best (SRS best, Cluster samples for national surveys) Nonprobability samples used in special circumstances (exploratory research, hard to reach populations) 3. Obtain a sampling frame 4. Pick your sample from the sampling frame Over-sampling? Stratification? 5. Perform statistical analyses, post-weighting the sample if necessary 46 / 59

47 Sampling Variability Parameter & Sample From before (repeat): Population Parameter: True value of a feature (e.g. percentage, mean) in the whole population usually unknown Statistic: Value of the feature in the sample data. Statistics are often used to estimate an unknown population parameter Sampling variability: the value of a statistic varies in repeated random samples The one central idea of statistical inference: to see how trustworthy a procedure is, ask what would happen if we repeated it many times 47 / 59

48 Sampling Distributions Sampling Variability Suppose that the proportion of people who agree that The President is doing a good job with the economy is p =.6 If we take a sample of 2500 we might find ˆp =.609 If we take another sample of 2500 we might have ˆp = This is sampling variability If we were to sample many times with n = 2500 the resulting distribution of the values of ˆp is called the sampling distribution of ˆp We could describe this distribution with a graph (e.g., a histogram) or with numbers (mean, spread,... ) The idea of the sampling distribution is the most important idea of statistics 48 / 59

49 Sampling Distributions Computer Simulation of the Sampling Distribution The sampling distribution of a statistic is the distribution of values taken by the statistic in all possible samples of the same size from the same population One can approximate the sampling distribution of ˆp by simulating the drawing of many samples from a population where p =.6 The next two slides demonstrate the sampling distributions of ˆp in 1000 SRS for p =.6 and n = 100 and n = 2500 The three slides after that do the same thing in 500 SRS for p =.3 and n = 10, n = 100, and n = / 59

50 Sampling Distributions Sampling Distribution for p =.6, n = / 59

51 Sampling Distributions Sampling Distribution for p =.6, n = / 59

52 Sampling Distributions Sampling Distribution for p =.3, n = 10 (500 Samples) Density Histogram of survey10 > survey10<-rbinom(500,10,0.3)/10 > survey10[1:5] [1] > summary(survey10) Min. 1st Qu. Median Mean 3rd Qu. Max > hist(survey10, probability=true) > lines(density(survey10), col="red", lwd=3) survey10 52 / 59

53 Sampling Distributions Sampling Distribution for p =.3, n = 100 (500 Samples) Density Histogram of survey100 > survey100<-rbinom(500,100,0.3)/100 > survey100[1:5] [1] > summary(survey100) Min. 1st Qu. Median Mean 3rd Qu. Max > hist(survey100, probability=true) > lines(density(survey100), col="red", lwd=3) survey / 59

54 Sampling Distributions Sampling Distribution for p =.3, n = 2500 (500 Samples) Density Histogram of survey2500 > survey2500<-rbinom(500,2500,0.3)/2500 > survey2500[1:5] [1] > summary(survey2500) Min. 1st Qu. Median Mean 3rd Qu. Max > hist(survey2500, probability=true) > lines(density(survey2500), col="red", lwd=3) survey / 59

55 Bias and Variability Properties of the Sampling Distribution: Shape, Center & Spread 1. Shape: histogram looks normal; one can confirm this impression with a normal quantile plot 2. Center: center of the distributions of ˆp tends to be close to p mean is.29 for n = 10,.3013 for n = 100 and.3002 for n = 2500 In technical terms one says that the ˆp has no bias as an estimator of p 3. Spread: values of ˆp tend to be less spread out, the larger n IQR is =.20 for n = 10, =.06 for n = 100 and =.0124 for n = / 59

56 Bias and Variability Properties of the Sampling Distribution: Bias & Variability Bias: concerns the center of the sampling distribution. A statistic used to estimate a parameter is unbiased if the mean of its sampling distribution is equal to the true value of the parameter being estimated Variability of a statistic described by the spread of its sampling distribution: this spread is determined by the sampling design and the sample size n statistics from larger probability samples have smaller spreads Controlling bias & variability: To reduce bias: use random sampling statistic computed from a SRS is unbiased, neither consistently overestimating or underestimating the population parameter To reduce variability: use a larger sample statistic computed from a larger sample has smaller spread 56 / 59

57 Bias and Variability Bias & Variability Illustrated in the Context of Target Shooting (IPS6e p. 218) 57 / 59

58 Sampling From Large Populations Population Size and Sample Variability Population size does not matter p / 59

59 Why Randomize? The idea of sampling distribution applies to all kinds of statistics used as estimators of population parameters proportion, mean, variance, regression coefficients, etc. The next step if to develop principles of probability theory, and from them develop a mathematical description of the sampling distribution of a statistic The standard deviation of the sampling distribution of the statistic is called the standard error of the statistic The standard error is then used to construct confidence intervals for the parameter tests of hypothesis on the value of the parameter Constructing confidence intervals & tests of hypothesis is called statistical inference And that s all there is to it! 59 / 59