Appendix Reassessing Public Support for a Female President. 1. Sampling Methodology

Transcription

1 Appendix Reassessing Public Support for a Female President The purpose of this appendix is to provide additional information about seven topics that could not be fully addressed in the manuscript. 1. Sampling Methodology Our data were collected via an online survey experiment that was fielded on March 10-15, The sample of voting-eligible adults was drawn by Survey Sampling International (SSI). SSI samples have been used in a variety of well-regarded political science survey studies in recent years (e.g., Berinsky, Margolis, and Sances 2014; Healy and Lenz 2014; Iyengar and Westwood 2015; Kam 2012; Malhotra, Margalit, and Mo 2013). To reduce sampling error and improve the population representativeness of our sample, we instructed SSI to use a stratified sampling method that used the following four stratifications: the region of residence (Northeast, Midwest, South, and West), sex (male and female), race/ethnicity (white, black, Hispanic, Asian, and others), and age groups. We collected our samples from the SSI s panel so that a sampling fraction in each of the strata becomes proportional to the one of the U.S. census population. To be more specific, in selecting our samples, we first asked demographic screening questions to 3,152 people in the SSI panel, and then invited 1,733 people among them to take part in our study based on this sampling method. A total of 1,578 respondents completed our survey. We randomly assigned these respondents to either a control (the shorter list) or treatment group (the longer list including the sensitive item). In this appendix, we provide more details about our survey experiment and its results. Table A1 displays how our sample compares to the Census. Each cell reports the deviation between the Census and our sample for a particular group. An example is black females aged 65 and over in the Midwest where the share our sample differs from the Census by just -.10 percentage points. In most cases the deviations are extremely small, almost always below one percentage point, and are always within the margin of error. 1

2 Table A1. Deviations of Key Demographic Characteristics from U.S. Census Region Age Hispanic Excluding Hispanic White Black Asian Other Male Female Male Female Male Female Male Female Male Female Northeast over Midwest over South over West over As a result of the panel selection process that guarantees a sample to be representative of the population on key variables, sampling weights are unlikely to be consequential. Nonetheless, we have generated post-stratification weights using Hainmueller s (2012) approach for balancing our sample against the U.S. Census. Table A2 shows the results after applying the weights. There are some small differences, but the pattern observed in these results are similar to the ones without any weights. 2

3 Table A2. Opposition to a Female President with and without Weights Demographic Control Treatment Difference Difference without Condition Condition with weights weights All respondents * 12.6 ** (0.04) (0.05) (6.3) (5.5) Male ** 25.7 *** (0.06) (0.07) (9.4) (8.3) Female (0.06) (0.06) (8.5) (7.3) No BA Degree *** 18.3 *** (0.06) (0.07) (8.7) (7.5) BA or above (0.07) (0.06) (8.9) (8.2) years old ** 28.4 ** (0.11) (0.12) (16.2) (12.7) years old *** 20.2 *** (0.05) (0.06) (7.8) (8.5) years old (0.07) (0.06) (9.5) (10.0) 66 years old or above (0.18) (0.24) (29.0) (18.1) Lower and lower middle class (0.08) (0.08) (11.6) (9.7) Middle class (0.06) (0.06) (8.5) (7.5) Upper middle and upper class ** (0.12) (0.11) (16.5) (14.7) South (0.08) (0.07) (10.9) (9.3) Nonsouth ** 19.4 *** (0.05) (0.06) (7.8) (6.9) White * (0.05) (0.05) (7.6) (6.6) Black (0.13) (0.14) (19.1) (17.2) Hispanic *** 41.1 *** (0.10) (0.15) (17.7) (15.5) Others (0.17) (0.16) (23.1) (20.6) Democrats (0.07) (0.06) (9.3) (8.1) Republicans ** 27.4 *** (0.08) (0.10) (12.4) (10.8) Independents (0.08) (0.10) (12.7) (10.9) Note: ***p <.01, **p <. 05, *p <.10. Entries in the first two columns are mean number of items with standard errors in parentheses. 3

4 2. Question Wording The Streb, Burrell, Frederick, and Genovese (SBFG) experiment was carried out over the telephone as a part of an Opinion Research Corporation CARAVAN poll. In contrast, we conducted our experiment online. To accommodate the change in survey mode, we slightly modified the question wording. The introductory script was altered modestly, but, importantly, we employed the identical statements for both the sensitive and non-sensitive items. SBFG wording: Now I am going to read you four/five things that sometimes make people angry or upset. After I read all four/five, just tell me HOW MANY of them upset you. I don t want to know which ones, just HOW MANY. (Respondents answer the number.) [Statements were rotated and read by an interviewer] 1. The way gasoline prices keep going up 2. Professional athletes getting million dollar-plus salaries 3. Requiring seat belts to be used when driving 4. Large corporations polluting the environment 5. A woman serving as president Our wording: The following statements sometimes make people angry or upset. Please carefully read all of the statements and indicate HOW MANY of them upset you. You do not need to indicate which ones. (Respondents choose from options with a number.) [Statements are shown in the randomized order] 1. The way gasoline prices keep going up 2. Professional athletes getting million dollar-plus salaries 3. Requiring seat belts to be used when driving 4. Large corporations polluting the environment 5. A woman serving as president 4

5 3. Potential Design Effects The Imai and Blair models for multivariate analysis of list experiments have now been used in several published studies by scholars in a variety of subfields (Ahlquist, Mayer, and Jackman 2014; Guerguiev and Malesky 2012; Meng, Pan, and Yang forthcoming). de Jonge and Nickerson s (2014) investigation provided support for the applicability of the Imai and Blair model in the field. Our implementation of the list experiment paradigm mimics the SBFG design. Their design has been lauded for having an effective design for yielding useful inferences that minimize floor and ceiling effects (Glynn 2013). As additional evidence that the SBFG design mitigates these concerns, Figure A1 shows the observed percentages of respondents that selected various numbers of responses as upsetting. The results show that responses are normally distributed and that the extreme cases have relatively few responses in both control and treatment conditions. This suggests that the risk of observing ceiling and floor effects in our experiment is relatively low. Figure A1: Percentage of Respondents Selecting a Number of Items Control Treatment Blair and Imai (2012) proposed a test to detect the presence of the design effect in which the addition of the sensitive item would change the sum of affirmative answers to the control items. The absence of such an effect is a key assumption underlying a list experiment. Our test result shows that the Bonferroni-corrected p-value is.87. Thus, we cannot reject the null hypothesis that there is no design effect. 5

6 4. Balance Test There is disagreement in the scholarly literature about whether to assess covariate balance and what to do if such an assessment suggests imperfect randomization. While Mutz and Pemantle (2015) recommend against a test, in the spirit of transparency we follow the practice recommended by Gerber et al. (2014) to provide basic information about randomization. First, Table 1 in the paper reports the mean number of selected items and standard errors by treatment and control condition for each of the subgroups in our analysis. Second, to assess the effectiveness of the randomization, we conducted a simple balance test. We estimated the effects of various covariates on the likelihood of being in the treatment condition using a logistic regression analysis. The results appear in Table A2. Among the array of variables in the model, only South has a coefficient that is significantly different from zero (p =.05). The chi-squared statistic for testing the joint significance of all of the variables yields a (p =.09). It thus appears that respondents living in the South were more likely to be assigned to the treatment group by chance. To the degree that this modest imbalance in one variable taints the simple means reported in Table 1, the imbalance is addressed more directly in the regression analysis that appears in Figure 1 (and Table A2 below). By explicitly holding the South variable (and all other variables) constant, we are even more confident about other observed experimental effects within other groups. 6

7 Table A3. Logit Model Test of Balance in Randomization Male.038 (.106) South.257* (.110) White.101 (.134) Black (.194) Education.077 (.043) Social Class.028 (.067) Democrat.155 (.130) Republican (.137) Age (.004) Constant (.299) Log Likelihood Chi-squared N 1,485 Note: Dependent variable is indicator for whether the respondent was assigned to the treatment condition. *p <.05, ** p <. 01, one-tailed test. 5. Multivariate Regression Results Table A3 shows the estimated coefficients and their standard errors from the fitted maximum likelihood model. We employ the constrained version of the estimator developed by Blair and Imai (2012), assuming that the answer to the sensitive item and the answers to the control items are independent. We employ the following variables in the regression model. They are also used to categorize our respondents into subgroups in the difference-in-means table. 7

8 Male: Dichotomous variable (male = 1; female = 0) Education: Categorical variable (8 grades or less = 1; 9-12 grades ( high school ) and no diploma/equivalency = 2: 12 grades and diploma or equivalency = 3; 12 grades and diploma or equivalency plus non-academic training = 4; Some college and no degree or junior/community college level degree (AA degree) = 5; BA level degree = 6; Advanced degree including LLB = 7) Age: Numerical variable (age in years) Social class: Categorical variable (lower class = 1; lower middle class = 2; middle class = 3; upper middle class = 4; upper class = 5) South: Dichotomous variable (South = 1; non-south = 0) White: Dichotomous variable (white = 1; non-white = 0) Black: Dichotomous variable (black = 1; non-black = 0) Democrat: Dichotomous variable (Democrat = 1; non-democrat = 0) Republican: Dichotomous variable (Republican = 1; non-republican = 0) Table A4. Multivariate Regression Results Sensitive items Control items Est. S.E. Est. S.E. Male Education Age Social class South White Black Democrat Republican Intercept Note: The outcome variable is whether or not A woman serving as president will make respondents upset. 6. Controlling for the Hillary Effect Following the suggestion made by SBFG, we test for whether attitudes toward Hillary Clinton underlie views about a female president by re-estimating the multivariate model after including a measure of the degree to which respondents like or dislike Clinton. We draw on the 8

9 following item, which is contained in a battery of questions about various political figures in our survey. The distribution of respondents on their favorability toward Hillary Clinton is shown below the question. What do you think about each of the following politicians? Please rate it on a scale from 0 to 10, where 0 means you strongly dislike that politician and 10 means that you strongly like that politician. If you have not heard of, or you feel you do not know enough about the politician, just choose don t know. Figure A2. Favorability toward Hillary Clinton (strongly dislike) (neutral) (strongly like) DK NA Note: Bars indicate percentages of respondents in each category. We begin by replicating the results in the table above after including the Clinton variable. The results appear in Table A4. We find that attitudes toward Clinton do not lead respondents to select a larger number of items in the treatment condition compared to the control condition. In addition, effects for the other variables are substantively similar to those in the model without the Clinton variable. 9

10 Table A5. Multivariate Regression Results Controlling for Hillary Effect Sensitive items Control items Est. S.E. Est. S.E. Male Education Age Social class South White Black Democrat Republican Clinton Intercept Note: The outcome variable is whether or not A woman serving as president will make respondents upset. Based on these models, in Figure A3 we display the estimated proportions of respondents upsetting a woman president by various social and demographic groups. This mimics Figure 1 in the paper but with the addition of the Hillary Clinton variable. For estimation purposes, we set the Clinton variable into three levels: strongly dislike (zero), neutral (five), and strongly like (ten). The figure shows that opposition to a female president does not vary across three levels of like toward Hillary Clinton. In addition, the other variables show similar results as in the original estimation. We do not believe that the Clinton question primed or contaminated the experiment. The Clinton item was in a battery asking respondents about their favorability toward nine political figures including Donald Trump, Jeb Bush, and Carly Fiorina. The experiment occurred much later in the survey and was separated by a number of unrelated questions. 10

11 Figure A3. Multivariate Estimates of Opposition to a Female President Controlling for Hillary Effect Note: Dots represent estimated proportions of respondents upset by a female president and lines are 95% confidence intervals from the regression model in Table A3. 7. Observational Survey Results SBFG reported data from national surveys asking respondents directly about their willingness to vote for a female presidential candidate if she was qualified and nominated by the respondent s party. Their list experiment challenged the truthfulness of those responses. We suggest that alternative observational questions that are less direct might provide more accurate responses. Since the publication of SBFG s study, CNN has asked on three occasions whether 11

12 respondents think America is ready for a woman president or not. Figure A4 displays the results of opinion polls asking about a female president that were conducted between 1945 through 2016 by Gallup, General Social Survey (GSS), and CNN. The Gallup and GSS results up through 2005 were reported by SBFG. We have extended the time series and added responses from the alternative different question wording from CNN. As shown in this figure, the support for female presidential candidates among the American public is steadily increasing since It appears that the CNN results are slightly lower than the other two opinion polls. This is at least partly because the CNN poll asks if Americans are ready for a woman president, instead of asking for respondents personal willingness. Figure A4. Willingness to Support a Female Presidential Candidate ( ) If your party nominated a generally well-qualified person for president who happened to be a woman, would you vote for that person? (Gallup) If your party nominated a woman for President, would you vote for her if she were qualified for the job? (GSS) Do you think America is ready for a woman president or not? (CNN) 12

13 References Ahlquist, John S., Kenneth R. Mayer, and Simon Jackman Alien Abduction and Voter Impersonation in the 2012 U.S. General Election: Evidence from a Survey List Experiment. Election Law Journal 13: Berinsky, Adam H., Michele F. Margolis, and Michael W. Sances Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention to Self-Administered Surveys. American Journal of Political Science 58: de Jonge, Chad P. Kiewiet, and David W. Nickerson Artificial Inflation or Deflation? Assessing the Item Count Technique in Comparative Surveys. Political Behavior 36: Gerber, Alan, Kevin Arceneaux, Cheryl Boudreau, Conor Dowling, Sunshine Hillygus, Thomas Palfrey, Daniel R. Biggers, and David J. Hendry Reporting Guidelines for Experimental Research: A Report from the Experimental Research Section Standards Committee. Journal of Experimental Political Science 1: Glynn, Adam N What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment. Public Opinion Quarterly 77: Gueorguiev, Dimitar, and Edmund Malesky Foreign Investment and Bribery: A Firm-Level Analysis of Corruption in Vietnam. Journal of Asian Economics 23: Hainmueller, Jens Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies. Political Analysis 20: Healy, Andrew, and Gabriel S. Lenz Substituting the End for the Whole: Why Voters Respond Primarily to the Election-Year Economy. American Journal of Political Science 58: Iyengar, Shanto, and Sean J. Westwood Fear and Loathing across Party Lines: New Evidence on Group Polarization. American Journal of Political Science 59: Kam, Cindy D Risk Attitudes and Political Participation. American Journal of Political Science 56: Malhotra, Neil, Yotam Margalit, and Cecilia Hyunjung Mo Economic Explanations for Opposition to Immigration: Distinguishing between Prevalence and Conditional Impact. American Journal of Political Science 57: Meng, Tianguang, Jennifer Pan, and Ping Yang. Forthcoming. Conditional Receptivity to Citizen Participation: Evidence from a Survey Experiment in China. Comparative Political Studies. 13

14 Mutz, Diana, and Robin Pemantle Standards for Experimental Research: Encouraging a Better Understanding of Experimental Methods. Journal of Experimental Political Science 2: