Polling. Polling. History of Polling

Transcription

1 Polling Readings: Herbert Asher, Polling and the Public Polling Polling and Surveys are synonymous. George Gallup began using the term poll in his earliest surveys to connect his questions to the election and add a degree of importance. History of Polling The Harrisburg Pennsylvanian reported a straw poll in In 1904 the New York Herald conducted street-corner polls. The Literary Digest began their polling in

2 Campaigns Media Government Marketing Courts Academics Who uses polling? Understanding Polls To evaluate public opinion polls we have to ask: Who is questioned? How are they asked? What questions are they asked? Who is asked? Drawing a sample 2

3 Who is questioned? A Sample A subset of the population we use to make inferences about the population in general Why do we use samples? Types of Samples Nonprobability Samples/Straw Polls Probability or Random Samples Nonprobability Samples Straw Polls People may self-select or be chosen by some non-rigorous method. You can not estimate the likelihood someone will be included 3

4 Nonprobability polls Typical people Purposive Volunteer subjects/self selection Haphazard sampling Quota Snowball sample The Perils of Straw Polls Shere Hite vs.. Dear Abbey Hite distributed 100,000 survey and found that 70% of women married 5 or more years had affairs. Abby received 200,000 responses to a column and found 15% of wives were unfaithful The Literary Digest Poll The Literary Digest Poll In 1936 Literary Digest sent out 10,000,000 ballots and got back 2,376,523 and predicted a 57% to 43% victory for Alf Landon FDR won with 62.5% of the vote 4

5 George Gallup Gallup predicted more accurately that FDR would get 56% of the public. Probability Samples Random samples Lets us judge how likely what we observe is not the product of chance. Judging a fair coin Let s s say that you suspect a coin is loaded or biased. How would you know when to be suspicious? If you flipped a coin 100 times and it came up heads 51 times would you be suspicious? What if it came up head 95 times? 5

6 Understanding Samples What happens if you flip a coin 100 times? Sampling Error Probabilities and Standard Deviations Shaded area = Shaded area = Shaded area = 0.997! +! 2 +2! % chance of falling 95% chance of falling between! and + between! 2 and % chance of falling between! 3 and + 3 6

7 Sampling Error Arises from representing a population with a random sample. Not a mistake on the part of the pollster, it is caused by the nature of random samples. Can not be calculated for a non-probability sample. Remember: : A random sample is a calculated risk. The Advantage of Random Samples Randomness insures that the only source of error/bias in the sample is sampling error -- which we can calculate Margin of Error or Confidence Interval Every poll has a confidence interval. Accurately reported it will say something like ± 3% with a 95% confidence level. 7

8 Confidence Interval A poll shows a candidate having the support of 47% of respondents with ± 3% This means the candidate s s support is--most likely--between 50% and 44%. Confidence Interval Most surveys use a 95% confidence interval or a 5% alpha level. This tells us that in 95% of the polls the percentage found in the sample will fall within the confidence interval around the actual percentage that exists in the population Tradeoffs in the confidence interval Level of certainty Do you want to be 90% or 95% certain? Precision of results Is ±3% accurate enough or more precise than you need Sample size The bigger the random sample, the better 8

9 Sample Size and Confidence Interval Gallup Tracking Poll of Likely Voters Gallup Tracking Poll of Likely Voters 9

10 Simple Random Sample Respondents are drawn at random from the individuals in a population Sub-samples The margin of error for the sample only applies to the sample overall. Stratified Samples What if you want to know something about the characteristics of sub-units? For example, what if you want to compare GW Bush s s rating in Texas to the nation? 10

11 Stratified Samples Divides the population into smaller geographic chunks and randomly surveys within each chunk. Cluster or Multistage Sampling Wrong population sampled Targeting the wrong people How do you predict voting? 11

12 Random samples of what? Sampling Frame A listing of the target population from which a sample is drawn Usually not possible to assemble Sampling Lingo Population (Universe) All items of interest Sample Portion chosen from population Parameter Characteristic of a population Statistic (Descriptive) Characteristic of a sample What is the difference between a sample and a population? Parameter and Statistic An Example The mean height of all 10th graders is 60 inches. Parameter The mean height of Mrs. Jones tenth graders is 60 inches. Statistic Population Sample 12

13 Declining Response Rates Polling s Dirty Little Secret What happens when people don t t want to answer pollster s s questions? Are people who do not respond different from those who do? Declining Response Rates Today most polling firms find that they can not reach about 20% of their sample-- even after making 6-7 attempts to reach them. Some firms use specially trained interviewers to contact reluctant respondents Non-coverage errors Phone surveys About 30% of phone numbers are unlisted 7% of voters in 2004 were cell phone only users About 5.5% of homes do not have a phone (25% of households with income under $15,000 a year) Answering machines and caller id are used to screen calls 5% of phone interviews are not finished because the respondent hangs up 13

14 Bias in non-responses In one experiment, after 1 attempt to reach respondents, 38% of the sample was working outside the home After 4 tries the percentage had risen to 58% After 7 tries the percentage had risen to 63.5% Phone-type Bias CPOs favored Kerry over Bush 54% to 45% Weighted Results Sometimes in order to correct for nonresponse errors that lead some groups to be under-represented in a poll 14

15 How are they asked? By mail, in person or by phone? Reaching Respondents By mail or May be slow since respondents may respond at their leisure Survey is self administered and may not be filled out by correct person In person Presents risks and interviewer bias Travel time is extensive Phone Surveys Most common, but hard to reach some people and interviews must be short 15

16 RDD Random Digit Dialer Used to pick numbers/respondents Proactive dialing has the PC (or central equipment) dial the number, but an interviewer is waiting on the line. Predictive has central dialing equipment dial and listens. Then interviewers are switched on the line after someone answers CATI Computer Assisted Telephone Interviewing helps speed administering the survey and analyzing results. Caller-id The use of caller-id (now in about 25% of homes) allows some to dodge calls from a unavailable number. Changing to a university s s name from unavailable can raise response rates by more than 3%. 16

17 Cell Phones The Federal Telephone Consumer Protection Act (TCPA) probits: ALL calls made to a cellular phone, without the prior consent of the person called IF the call is made using an automatic telephone dialing system (defined as equipment which has the capacity to store or produce telephone numbers to be called using a random or sequential number generator and to dial such numbers) or an artificial or prerecorded voice AND IF the party is charged for the call Interviewer Bias When the response is altered as a reaction to the individual asking the question. Interviewer Bias- Gender The decision to have an abortion is a private matter that should be left to the woman to decide without government question. Respondent Interviewer Female Male Female 84% 77% Male 64% 70% 17

18 Interviewer Bias- Race Respondents will shape their responses based on the race (or perceived race) of the person asking the questions. Perceptions of race As part of the National Black Politics survey of 1,204 black respondents researchers asked respondent believed the race of the interviewer was. Seventy six of the respondents correctly identified their interviewer as black (only African American interviewers were used) while 14% thought the interviewer was white and 10% guessed another race. While 3 of the 25 interviews were guessed to be white or some other race by a majority of the respondents, none of the interviewer sounded black to all the respondents. Perceptions of race Perceived race of interviewer Black White Black elected officials can best represent 64% 49% the interests of the black community American society is fair to everyone 14% 31% The American legal system is 84% 72% unfair to blacks The police are too much like just 52% 42% another gang to stop gang violence. 18

19 Perceived race of interviewers Black White Black elected officials can best represent the interests of the black community 64% 49% American society is fair to everyone 14% 31% The American legal system is unfair to blacks 84% 72% The police are too much like just another gang to stop gang violence. 52% 42% What is asked? Questionnaire Construction Types of Questions Open-ended Closed-ended 19

20 Open Ended Questions Lets respondents use their own words to answer questions What do you like/dislike about the Republican Party? What do you do you consider the most important problem facing the nation today? But, hard to summarize or code Closed ended Questions Respondents choose from a list of answers selected by the survey researcher Do you approve or disapprove of the way Bill Clinton is handling his job as president? Respondents are like cats, if you give them a box, they will go in it. 7-point scales The seven point scale comes from the psychological conclusion that individuals have difficulty reliably making distinctions between more than seven options. In phone surveys it is generally regarded as having no more than five response categories. 20

21 7-point scales Recently there has been a lot of talk about women s s rights. Some people feel that women should have an equal role with men in running business, industry, and government. Others feel that women s s places is in the home Women and men should have an equal role A woman s place is in the home Agree Disagree Respondents are asked to agree or disagree with a series of statements Agree Disagree Should marijuana be legalized? Strongly agree Strongly disagree

22 Feeling thermometer 100 Very warm and very favorable feeling 85 Quite warm or quire favorable feeling. 70 Fairly warm or 50 No Feeling at all 40 Slightly cold or slightly unfavorable feeling 30 Fairly cold or fairly unfavorable feeling 15 Quite cold or 0 Very cold or very unfavorable. Branching Rather than having a long question can have a series of short questions Branching One Question Do you consider yourself a strong Democrat, a weak Democrat, an independent leaning Democratic, an independent, an independent leaning Republican, a weak Republican or a strong Republican? Two Branched Questions Do you consider yourself a Democrat, Republican, and Independent or what? Do you consider yourself a strong or weak? 22

23 Branching Do you think the U.S. should or should not make it easier for illegal immigrants to become citizens of the United States? (Gallup Aug 24-26, 2001) Should 28% Should not 67% No opinion 5% Which of the following proposals would you prefer for illegal immigrants currently living in the United States: they should be granted general amnesty, (or) they should be allowed to stay in the U.S. only if they have worked and paid taxes for a certain length of time]? [Based on those who say it should be easier for immigrants to become citizens; ±6 PCT. PTS.] Granted general amnesty 20% Should stay if they paid taxes 73% Combining branched questions Easier, prefer general amnesty 6% Easier, prefer citizenship for those who have worked/paid taxes 20% Easier, no preference 2% Not easier 67% No opinion 5% Batteries of Questions Sometimes a series of questions can be used to get at broad issues. Worker satisfaction 23

24 Worker Satisfaction For each, please tell me how satisfied or dissatisfied you are with your current job in this regard. First, are you completely satisfied, somewhat satisfied, somewhat dissatisfied, or completely dissatisfied with? [Gallup Organization, August 16-19, 2001] What messes up question? Loaded Questions Push Polls Uncertainty and Ambiguity Loaded Questions Questions designed to return a particular response. 24

25 Loaded Questions A A recent study by a psychology professor at a leading university concluded that the amount of violence children see on television has an effect on their likelihood of being aggressive and committing crimes. From what you have seen or heard about this subject, do you agree Loaded Questions Which segment of the television viewing audience do you think will suffer more if federal funding for PBS was cut back sharply or eliminated--children or adults? Loaded Questions Do you agree or disagree with the statement When (Bernhard) Goetz said in his confession that he used dum- dum bullets, that he was sorry he didn t t gouge out the eyes of the four he shot, and that if he could have reloaded his gun fast enough, he would have taken out after them, he looked more like a Death Wish gunman out stalking to kill criminals, not an innocent victim trying to defend himself Harris poll 25

26 Push Polls Polls that provide respondents with bits of information to see what impact they have on citizens choices Abuse of Push Polls When information is provided that is intended to influence-rather than measure-respondents opinions Thousands of homes are called. Push Polls As you may know, in 1974, Jerry Springer, who had gotten married 6 months earlier, was arrested on a morals charge with 3 women in a hotel room. He also used a bad check to pay for the women s s services, and subsequently resigned as mayor of his city. Does this make you much more likely, somewhat more likely, somewhat less likely, or much less likely to support Jerry Springer for governor this year? 26

27 Push Polls If you knew that Pat Buchanan had said women were less equipped psychologically for the working world, would you be more likely to vote for him, less likely, or does it make no difference in your vote? Nonattitudes Nonattitudes Sometimes called Doorstep responses When respondents are unsure about the facts or the question being asked they may offer up answers that are not reliable. 27

28 Uncertainty When you ask citizens about something they know little or nothing about what does their response mean? If you do a 20-minute poll on environmental issues, you ve engaged most people in a longer conversation than they ve ever had about the environment in their lives. Uncertainty Do you favor or oppose the Public Affairs Act of 1975? 33% of respondents expressed a preference Do you favor or oppose the Public Affairs Act of 1975 or don t t you have an opinion? Percentage expressing a preference dropped to 2%. Ambiguity When the question creates uncertainty Does it seem possible or does it seem impossible to you that the Nazi extermination of Jews never happened? [1992 Roper Poll] 22% of respondents said that it seemed possible that the Holocaust never occurred Another 12% answered don t t know. 28

29 Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened? Only 1% expressed doubt Ambiguity How many public officials do you think are a little bit dishonest-most, some, a few, or none? Types of Question Problems 29

30 Context Bias Or Multiple Stimuli Some questions have more than one question or stimulus to which respondents may react. Women in Combat Context Bias President Clinton said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree? 36% of Democrats and 16% of Republicans agreed The Republican Congress said that the 1975 Public Affairs Act should be repealed. Do you agree or disagree? 36% of Republicans and 19% of Democrats agreed Context Bias The U.S. has 36,000 troops and airmen in South Korea. If North Korea invaded South Korea, we have a firm commitment to defend South Korea with our own military forces. If South Korea were invaded by Communist North Korea, would you favor or oppose the U.S. using troops, air power and naval power to defend South Korea? [43%] As things stand now, do you feel the war in Korea has been worth fighting, or not? [27%] 30

31 Context Bias Sometimes you hear it said that there should be a law to ensure that a certain number of federal contracts go to minority contractors. Do you favor or oppose such a law? Favor 43% Oppose 57% The Congress of the United States-- both the House of Representatives and the Senate--have passed a law to ensure that a number of federal contracts go to minority contractors. Do favor or oppose such a law? Favor 57% Oppose 43% Social Desirability Bias People don t t want to admit to unpopular beliefs or actions. Just ask them about Racial attitudes Sexual behavior Shere Hite Consistency Bias People want to appear consistent Moderation Bias n People want to avoid appearing extreme 31

32 Agree-ability bias Response acquiescence People prefer to agree rather than disagree. Respondents to consistency agree on agree/disagree questions while not seriously reading the questions. No Middle Point If people who feel neutral forcing them in one direction or another draws out something random or unrelated to the question Unbalanced Questions Having more arguments or responses on one side than another. 32

33 Unbalanced Questions Do you agree or disagree with this statement: Any able-bodied person can find a job and make ends meet? Agree 65% No opinion 10% Disagree 25% Some people feel that any able-bodied person can find a job and make ends meet. Others feel that there are times when it is hard to get along and some able-bodied people may not be able to find work. Who do you agree with most? Can make ends meet 43% No opinion 18% Sometimes hard to get along 39% Framing Effects The effect of previous content of interview may have on specific questions. Questions about personal finances may encourage pocketbook voting Response Choice Order When questions are read aloud to respondents they tend to choose among the last few options When questions are read by the respondent they tend to choose among the early options 33

34 Other Types of Surveys Panel Studies Polls in which the same voters are questioned repeatedly over time. Designed to study change in opinions Problems with Panel Studies People become sensitized by questions and may end up behaving differently than other citizens. 34

35 Exit Polls Surveys taken as voters leave the polling place In recent years conducted by Voter News Service (VNS) in a cooperative effort with numerous news outlets Exit Polls Short interviews conducted with people as they leave polling places Usually large samples In ,000 voters from 1,500 precincts were interviewed to insure samples of most states Focus Groups Most often 7 to 10 people (even though they may range from 4 to 12) who do not know each other. Brought together to generate an informal discussion that will reveal their perceptions in a friendly, casual atmosphere 35

36 Advantages of Focus Groups Avoids the Passive Respondent Ability to explore the reasons behind opinions Simulates normal interaction Cost Focus Groups in Politics Shape surveys Probe ideas Test advertising/themes Disadvantage of Focus Groups Not a representative sample 36

37 On-line polling Problems with polls Voluntary No population list Not anonymous Computer users younger, whiter, more male, better educated and wealthier than population Web Polls/Panels Some researchers are using on-line surveys that contact the same respondents on a repeated basis to create electronic panels. 37

38 Internet Polls People magazine s s website poll that named Howard Stern's sidekick "Hank the Angry, Drunken Dwarf" as one of the 50 most beautiful people in the world Electronic ballot stuffing 38