Experimental Math 124: Module 3 and Module 4 David Meredith Department of Mathematics San Francisco State University September 24, 2009
What we will do today Experimental 1
What we will do today Experimental 1 2 Experimental
Terminology Population: individuals to be studied. Experimental
Terminology Population: individuals to be studied. Sample: a subset supposed to represent the population. Experimental
Terminology Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population.
Terminology Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population. The average height of the sample should be close to the average height of the population.
Terminology Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population. The average height of the sample should be close to the average height of the population. The division of the sample in Republicans and Democrats should reflect the division within the population.
Terminology Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population. The average height of the sample should be close to the average height of the population. The division of the sample in Republicans and Democrats should reflect the division within the population. Every sample is selected according to some method or plan.
Terminology Experimental Population: individuals to be studied. Sample: a subset supposed to represent the population. Ideally, whatever is true about the sample should be true about the population. The average height of the sample should be close to the average height of the population. The division of the sample in Republicans and Democrats should reflect the division within the population. Every sample is selected according to some method or plan. A sample is biased if it is not representative of the population, particularly if it was selected in a way that almost guaranteed that it would not be representative.
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample.
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936 Sent 2.3M straw ballots to addresses from phone books and auto registrations
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936 Sent 2.3M straw ballots to addresses from phone books and auto registrations Survey: Landon 55%, Roosevelt 41%
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936 Sent 2.3M straw ballots to addresses from phone books and auto registrations Survey: Landon 55%, Roosevelt 41% Actual election: Roosevelt 61%, Landon 37%
Terminology Experimental frame: the individuals that might be selected for the sample. A bad sampling frame can lead to a biased sample. Famous bad sampling frame: Literary Digest poll in 1936 Sent 2.3M straw ballots to addresses from phone books and auto registrations Survey: Landon 55%, Roosevelt 41% Actual election: Roosevelt 61%, Landon 37% Two errors: bad sampling frame and response bias.
Volunteer samples A volunteer sample is a sample made up of volunteers from the population Experimental
Volunteer samples Experimental A volunteer sample is a sample made up of volunteers from the population Example: I want to know how you like working in teams, so I ask for volunteers to come to my office to talk with me about their experience. The population is this class.
Volunteer samples Experimental A volunteer sample is a sample made up of volunteers from the population Example: I want to know how you like working in teams, so I ask for volunteers to come to my office to talk with me about their experience. The population is this class. The sample people who choose to come to my office is likely to be highly biased, with only people who really like teamwork or really hate it coming to see me.
Volunteer samples Experimental A volunteer sample is a sample made up of volunteers from the population Example: I want to know how you like working in teams, so I ask for volunteers to come to my office to talk with me about their experience. The population is this class. The sample people who choose to come to my office is likely to be highly biased, with only people who really like teamwork or really hate it coming to see me. Example: people who post comments to news articles are not necessarily representative of the population of all readers.
Convenience samples Experimental A convenience sample is a sample selected according to a plan marked more by its ease of execution than its likelihood of selecting a representative sample.
Convenience samples Experimental A convenience sample is a sample selected according to a plan marked more by its ease of execution than its likelihood of selecting a representative sample. Example: you want to know the ratio of men to women among people who actually attend class at SF State, so you count the men and women in your classes.
Convenience samples Experimental A convenience sample is a sample selected according to a plan marked more by its ease of execution than its likelihood of selecting a representative sample. Example: you want to know the ratio of men to women among people who actually attend class at SF State, so you count the men and women in your classes. The population is all students. The sample is students in your classes.
Convenience samples Experimental A convenience sample is a sample selected according to a plan marked more by its ease of execution than its likelihood of selecting a representative sample. Example: you want to know the ratio of men to women among people who actually attend class at SF State, so you count the men and women in your classes. The population is all students. The sample is students in your classes. The sample is likely to be biased, since the classes you take are biased by your major and not likely to be broadly representative of the campus as a whole.
Biased sampling frame Experimental A biased sampling frame draws the sample from a subpopulation that may not be representative of the entire population.
Biased sampling frame Experimental A biased sampling frame draws the sample from a subpopulation that may not be representative of the entire population. A valid sampling method applied to a biased sampling frame probably will not produce a representative sample.
Biased sampling frame Experimental A biased sampling frame draws the sample from a subpopulation that may not be representative of the entire population. A valid sampling method applied to a biased sampling frame probably will not produce a representative sample. Example: you want to know the political opinions of Californians aged 18-30, so you survey a sample of SF State students.
Biased sampling frame Experimental A biased sampling frame draws the sample from a subpopulation that may not be representative of the entire population. A valid sampling method applied to a biased sampling frame probably will not produce a representative sample. Example: you want to know the political opinions of Californians aged 18-30, so you survey a sample of SF State students. The population is all Californians aged 18-30. The sampling frame is SF State students.
Systematic sampling Experimental A systematic sample draws a sample from a population according to fixed rule
Systematic sampling Experimental A systematic sample draws a sample from a population according to fixed rule Example: sample SF State students by taking every student whose student number ends in 99. This will be 1% of the student population.
Systematic sampling Experimental A systematic sample draws a sample from a population according to fixed rule Example: sample SF State students by taking every student whose student number ends in 99. This will be 1% of the student population. You might get a representative sample, but generally statisticians think that random samples are more likely to be unbiased than systematic samples.
Random sampling Experimental A random sample draws a sample from a population at random.
Random sampling Experimental A random sample draws a sample from a population at random. Example: put all the student numbers on slips of paper, mix them up and draw 100.
Random sampling Experimental A random sample draws a sample from a population at random. Example: put all the student numbers on slips of paper, mix them up and draw 100. Better example: let a computer select 100 students at random (computer programmers know how to do this).
Simple random sampling (SRS) Experimental A simple random sample is a random sample where everyone in the population has an equal chance of being selected.
Simple random sampling (SRS) Experimental A simple random sample is a random sample where everyone in the population has an equal chance of being selected. This is usually practical only for small populations..
Cluster sampling Experimental A cluster sample chooses a sample by choosing small groups at random instead of individuals.
Cluster sampling Experimental A cluster sample chooses a sample by choosing small groups at random instead of individuals. Example: sample students at SF State by choosing a few classes at random and sampling everyone in those classes.
Stratified sampling A stratified sample divides the population into subgroups and chooses a sample from each subgroup. Experimental
Stratified sampling Experimental A stratified sample divides the population into subgroups and chooses a sample from each subgroup. Everyone in the population falls into one of the subgroups, so in principle no one is excluded from the sample. The sampling frame is the entire population.
Stratified sampling Experimental A stratified sample divides the population into subgroups and chooses a sample from each subgroup. Everyone in the population falls into one of the subgroups, so in principle no one is excluded from the sample. The sampling frame is the entire population. Stratified sampling is used to assure that all important subgroups within the population are represented in the sample.
Stratified sampling Experimental A stratified sample divides the population into subgroups and chooses a sample from each subgroup. Everyone in the population falls into one of the subgroups, so in principle no one is excluded from the sample. The sampling frame is the entire population. Stratified sampling is used to assure that all important subgroups within the population are represented in the sample. Example: divide SF State students by ethnicity and take a sample from each ethnic group.
Multistage sampling Experimental A multistage sample first selects some large subgroups at random, then selects the sample from these. There may be more than two stages.
Multistage sampling Experimental A multistage sample first selects some large subgroups at random, then selects the sample from these. There may be more than two stages. This may be the only way to sample a large population like the US population, particularly if extensive personal interviews are required.
Multistage sampling Experimental A multistage sample first selects some large subgroups at random, then selects the sample from these. There may be more than two stages. This may be the only way to sample a large population like the US population, particularly if extensive personal interviews are required. Example: Choose six cities at random and 100 people at random from each city.
Experimental
Three types of studies Experimental studies
Three types of studies Experimental studies Experimental studies
Three types of studies Experimental studies Experimental studies
What is an observational study Experimental studies observe people behaving naturally
What is an observational study Experimental studies observe people behaving naturally Three general types
What is an observational study Experimental studies observe people behaving naturally Three general types Researcher observes people who don t know they are under observation. (Ethical issues?)
What is an observational study Experimental studies observe people behaving naturally Three general types Researcher observes people who don t know they are under observation. (Ethical issues?) Researcher asks people to keep a diary or record of their behavior
What is an observational study Experimental studies observe people behaving naturally Three general types Researcher observes people who don t know they are under observation. (Ethical issues?) Researcher asks people to keep a diary or record of their behavior Researcher asks people to recall their behavior or decisions
Advantages of observational studies Experimental Subjects behave naturally ecological validity.
Advantages of observational studies Experimental Subjects behave naturally ecological validity. Some behaviors are too dangerous to investigate experimentally.
Disadvantages of observational studies Experimental Difficulty of controlling all variables.
Disadvantages of observational studies Experimental Difficulty of controlling all variables. Example: does tutoring help students? An observational study might ask if tutored students do better than untutored students, but there might be some lurking variables (men in general are less likely to use tutoring than women, working students may be less like to use tutoring than full-time students).
Disadvantages of observational studies Experimental Difficulty of controlling all variables. Example: does tutoring help students? An observational study might ask if tutored students do better than untutored students, but there might be some lurking variables (men in general are less likely to use tutoring than women, working students may be less like to use tutoring than full-time students). Consequent difficulty of identifying cause and effect.
What is an experimental study Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Experimental
What is an experimental study Experimental Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Usually there are at least two treatment groups for comparison
What is an experimental study Experimental Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Usually there are at least two treatment groups for comparison Sometimes one treatment group is the control group to which no treatment or just a fake treatment (placebo) is given.
What is an experimental study Experimental Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Usually there are at least two treatment groups for comparison Sometimes one treatment group is the control group to which no treatment or just a fake treatment (placebo) is given. The division of subjects into groups must be random, even if the subjects overall are chosen by some other method.
What is an experimental study Experimental Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Usually there are at least two treatment groups for comparison Sometimes one treatment group is the control group to which no treatment or just a fake treatment (placebo) is given. The division of subjects into groups must be random, even if the subjects overall are chosen by some other method. Often the sample overall is a convenience sample: all patients in a clinic All students in a psychology class
What is an experimental study Experimental Experimental studies subject people to carefully designed environments and treatments like drug tests or psychological behaviorial studies. Usually there are at least two treatment groups for comparison Sometimes one treatment group is the control group to which no treatment or just a fake treatment (placebo) is given. The division of subjects into groups must be random, even if the subjects overall are chosen by some other method. Often the sample overall is a convenience sample: all patients in a clinic All students in a psychology class But the division into treatment groups should be random
Refinements to experimental studies Experimental If the subjects don t know which treatment group they belong to, then the study is blind
Refinements to experimental studies Experimental If the subjects don t know which treatment group they belong to, then the study is blind If the research staff dealing with the subjects also don t know which treatment the subjects are receiving, the study is double blind.
Refinements to experimental studies Experimental If the subjects don t know which treatment group they belong to, then the study is blind If the research staff dealing with the subjects also don t know which treatment the subjects are receiving, the study is double blind. I recently saw the phrase triple blind, which meant that the statisticians analyzing the results didn t know which group was the treatment group and which was the control group.
Advantages of experimental studies Experimental Lurking variables can be controlled, so cause and effect is easier to establish.
Advantages of experimental studies Experimental Lurking variables can be controlled, so cause and effect is easier to establish. Drug testing must be done with experiments
Advantages of experimental studies Experimental Lurking variables can be controlled, so cause and effect is easier to establish. Drug testing must be done with experiments Some behavior studies involve situations that would never arise naturally (Solomon Asch experiments, 1950 s)
Disadvantages of experimental studies Experimental Lack of ecological validity, especially for behavioral studies.
Disadvantages of experimental studies Experimental Lack of ecological validity, especially for behavioral studies. People who know they are being observed and manipulated may not behave naturally.
Disadvantages of experimental studies Experimental Lack of ecological validity, especially for behavioral studies. People who know they are being observed and manipulated may not behave naturally. Hawthorne effect: workers improve when put under observation
Survey questions good and bad Open vs. closed questions Experimental
Survey questions good and bad Open vs. closed questions open questions get more honest answers Experimental
Survey questions good and bad Open vs. closed questions open questions get more honest answers closed questions easier to analyze and tabulate Experimental
Survey questions good and bad Experimental Open vs. closed questions open questions get more honest answers closed questions easier to analyze and tabulate closed questions should include other response to avoid forcing a choice among undesirable alternatives
Survey questions good and bad Experimental Open vs. closed questions open questions get more honest answers closed questions easier to analyze and tabulate closed questions should include other response to avoid forcing a choice among undesirable alternatives Avoid one-sided response scales In 2008 I got a survey with the choices: 1 (extremely dissatisfied) 1 (strongly disagree) 2 2 3 3 4 (satisfied) 4 (agree) 5 5 6 6 7 (extremely satisfied) 7 (strongly agree) The middle term is not half-way between the extremes.
Survey questions good and bad Always avoid leading questions: Do you support Gov. Schwarzenegger s plan to release dangerous criminals that threaten our children? Experimental
Survey questions good and bad Experimental Always avoid leading questions: Do you support Gov. Schwarzenegger s plan to release dangerous criminals that threaten our children? A push poll is one conducted not to gather information but to put ideas into voters heads: We re conducting a poll about voter awareness. Were you aware the Gavin Newsom, a Democratic candidate for Governor, is a radical supporter of gay rights?
Survey questions good and bad Experimental Always avoid leading questions: Do you support Gov. Schwarzenegger s plan to release dangerous criminals that threaten our children? A push poll is one conducted not to gather information but to put ideas into voters heads: We re conducting a poll about voter awareness. Were you aware the Gavin Newsom, a Democratic candidate for Governor, is a radical supporter of gay rights? Confusing or complicated questions, especially those that ask more than one question, are bad
Survey questions good and bad Experimental Always avoid leading questions: Do you support Gov. Schwarzenegger s plan to release dangerous criminals that threaten our children? A push poll is one conducted not to gather information but to put ideas into voters heads: We re conducting a poll about voter awareness. Were you aware the Gavin Newsom, a Democratic candidate for Governor, is a radical supporter of gay rights? Confusing or complicated questions, especially those that ask more than one question, are bad Do you favor placing a high or low maximum on the level of federal funds assistance to leveraged hedge funds participating in national insurance programs, and do you favor or oppose increasing or decreasing regulation of such funds?
Survey questions asked differently How a question is asked can make a big difference in the answers Experimental
Experimental Survey questions asked differently How a question is asked can make a big difference in the answers During the financial crisis of 2008, people were asked about rescuing large banks (Washington Post online, 9/24/08).
Experimental Survey questions asked differently How a question is asked can make a big difference in the answers During the financial crisis of 2008, people were asked about rescuing large banks (Washington Post online, 9/24/08). The Times/Bloomberg poll asked respondents whether they believed it was the government s responsibility to bail out private companies with taxpayers dollars. A majority said no.
Experimental Survey questions asked differently How a question is asked can make a big difference in the answers During the financial crisis of 2008, people were asked about rescuing large banks (Washington Post online, 9/24/08). The Times/Bloomberg poll asked respondents whether they believed it was the government s responsibility to bail out private companies with taxpayers dollars. A majority said no. The Pew poll asked respondents if investing billions to try to keep financial institutions and markets secure was the right thing to do. A majority said yes."
Advantages of surveys Experimental Cheap
Advantages of surveys Experimental Cheap Can explore a wide variety of issues
Disadvantages of surveys It is hard to construct good questions Experimental
Disadvantages of surveys Experimental It is hard to construct good questions People misremember
Disadvantages of surveys Experimental It is hard to construct good questions People misremember People lie
Disadvantages of surveys Experimental It is hard to construct good questions People misremember People lie Example: find out how often people go to church.
Disadvantages of surveys Experimental It is hard to construct good questions People misremember People lie Example: find out how often people go to church. Don t ask: How many weeks did you go to church last year? or (worse) Do you attend church regularly?
Disadvantages of surveys Experimental It is hard to construct good questions People misremember People lie Example: find out how often people go to church. Don t ask: How many weeks did you go to church last year? or (worse) Do you attend church regularly? Ask: Did you go to church last Sunday? or (more eucumenical) Did you attend a religious service within the past seven days?