What we will do today Five Experimental Module 3 and Module 4 David Meredith Department of Mathematics San Francisco State University September 24, 2008 Five Experimental 1 Five 2 Experimental Terminology Terminology Five Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population. The average height of the sample should be close to the average height of the population. The division of the sample in Republicans and Democrats should reflect the division within the population. Every sample is selected according to some method or plan. Five Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936 Sent 2.3M straw ballots to addresses from phone books and auto registrations Survey: Landon 55%, Roosevelt 41% Actual election: Roosevelt 61%, Landon 37% Two errors: bad sampling frame and response bias. A sample is biased if it is not representative of the population, particularly if it was selected in a way that almost guaranteed that it would not be representative.
Volunteer samples Convenience samples Five Experimental A volunteer sample is a sample made up of volunteers from the population Example: I want to know how you like working in teams, so I ask for volunteers to come to my office to talk with me about their experience. The population is this class. The sample people who choose to come to my office is likely to be highly biased, with only people who really like teamwork or really hate it coming to see me. Example: people who post comments to news articles are not necessarily representative of the population of all readers. Five Experimental A convenience sample is a sample selected according to a plan marked more by its ease of execution than its likelihood of selecting a representative sample. Example: you want to know the ratio of men to women among people who actually attend class at SF State, so you count the men and women in your classes. The population is all students. The sample is students in your classes. The sample is likely to be biased, since the classes you take are biased by your major and not likely to be broadly representative of the campus as a whole. Biased sampling frame Systematic sampling Five Experimental A biased sampling frame draws the sample from a subpopulation that may be be representative of the entire population. A valid sampling method applied to a biased sampling frame probably will not produce a representative sample. Example: you want to know the political opinions of Californians aged 18-30, so you survey a sample of SF State students. The population is all Californians aged 18-30. The sampling frame is SF State students. Five Experimental A systematic sample draws a sample from a population according to fixed rule Example: sample SF State students by taking every student whose student number ends in 99. This will be 1% of the student population. You might get a representative sample, but generally statisticians think that random samples are more likely to be unbiased than systematic samples.
Random sampling Simple random sampling (SRS) Five Five Experimental A random sample draws a sample from a population at random. Example: put all the student numbers on slips of paper, mix them up and draw 100. Better example: let a computer select 100 students at random (computer programmers know how to do this). Experimental A simple random sample is a random sample where everyone in the population has an equal chance of being selected. This is usually practical only for small populations.. Cluster sampling Stratified sampling Five Experimental A cluster sample chooses a sample by choosing small groups at random instead of individuals. Example: sample students at SF State by choosing a few classes at random and sampling everyone in those classes. Five Experimental A stratified sample divides the population into subgroups and chooses a sample from each subgroup. Everyone in the population falls into one of the subgroups, so in principle no one is excluded from the sample. The sampling frame is the entire population. Stratified sampling is used to assure that all important subgroups within the population are represented in the sample. Example: divide SF State students by ethnicity and take a sample from each ethnic group.
Multistage sampling Three types of Five Experimental A multistage sample first selects some large subgroups at random, then selects the sample from these. There may be more than two stages. This may be the only way to sample a large population like the US population, particularly if extensive personal interviews are required. Five Experimental Experimental Example: Choose six cities at random and 100 people at random from each city. Five What is an observational study Five Advantages of observational Experimental observe people behaving naturally Three general types Researcher observes people who don t know they are under observation. (Ethical issues?) Researcher asks people to keep a diary or record of their behavior Researcher asks people to recall their behavior or decisions Experimental Subjects behave naturally ecological validity. Some behaviors are too dangerous to investigate experimentally.
Five Disadvantages of observational Five What is an experimental study Experimental Difficulty of controlling all variables. Example: does tutoring help students? An observational study might ask if tutored students do better than untutored students, but there might be some lurking variables (men in general are less likely to use tutoring than women, working students may be less like to use tutoring than unemployed students). Consequent difficulty of identifying cause and effect. Experimental Experimental subject people to carefully designed environments and treatments like drug tests or psychological behaviorial. Usually there are at least two treatment groups for comparison Sometimes one treatment group is the control group to which no treatment or just a fake treatment (placebo) is given. Five Refinements to experimental Five Advantages of experimental Experimental If the subjects don t know which treatment group they belong to, then the study is blind If the research staff dealing with the subjects also don t know which treatment the subjects are receiving, the study is double blind. Experimental Lurking variables can be controlled, so cause and effect is easier to establish. Drug testing must be done with experiments Some behavior involve situations that would never arise naturally (Solomon Asch experiments, 1950 s)
Five Experimental Disadvantages of experimental Lack of ecological validity, especially for behavioral. People who know they are being observed and manipulated may not behave naturally. Hawthorne effect: workers improve when put under observation Five Experimental Survey questions good and bad Open vs. closed questions open questions get more honest answers closed questions easier to analyze and tabulate closed questions should include other response to avoid forcing a choice among undesirable alternatives Avoid one-sided response scales Just last week I got a survey with the choices: 1 (extremely dissatisfied) 1 (strongly disagree) 2 2 3 3 4 (satisfied) 4 (agree) 5 5 6 6 7 (extremely satisfied) 7 (strongly agree) The middle term is not half-way between the extremes. Survey questions good and bad Advantages of surveys Five Experimental Always avoid leading questions: Since Sarah Palin is such a great hunter, do you think she would support wise policies on gun ownership? (Push polls) Confusing or complicated questions, especially those that ask more than one question, are bad Do you favor placing a high or low maximum on the level of federal funds assistance to leveraged hedge funds participating in national insurance programs, and do you favor or oppose increasing or decreasing regulation of such funds? Five Experimental Cheap Can explore a wide variety of issues
Disadvantages of surveys Five Experimental People misremember People lie