Pre-analysis plan: Voter evaluations of candidate socioeconomic characteristics

Transcription

1 Pre-analysis plan: Voter evaluations of candidate socioeconomic characteristics Sophie Hill August 31, Introduction In most democracies, elected officials are overwhelming drawn from higher income, more educated, and more middle class backgrounds. Perhaps surprisingly, recent experimental work has found limited evidence of voter bias against working-class candidates. However, these studies have typically used a conjoint or vignette design with at most one possible working-class occupation, limiting generalizability. The first goal of this study is to build on this prior literature with an experimental design that includes a much richer set of occupations that reflect the diversity of post-industrial working-class jobs, to allow for more robust inferences about the effect of candidate class. Secondly, this study draws on prior literature in political science and social psychology to investigate whether voters use a candidate s socioeconomic characteristics as a heuristic for two important valence qualities: competence and character. A large body of research in social psychology suggests that these two dimensions (typically referred to by psychologists as competence and warmth) are fundamental to social cognition (Fiske, Cuddy, and Glick, 2007). Competence relates to intelligence, talent, skill, confidence, and experience; character (or warmth ) relates to honesty, approachability, integrity, helpfulness, Harvard University, Government Department. sophie_hill@g.harvard.edu

2 Sophie Hill 2 and trustworthiness. Political scientists have long recognized that these two dimensions also structure the way that voters perceive political leaders. While early studies typically proposed other dimensions in addition to these (Kinder, 1986; Miller, Wattenberg, and Malanchuk, 1986; Pancer, Brown, and Barr, 1999), more recent work has explicitly adopted the two-dimensional competence-character framework (Bittner, 2011; Costa and Ferreira da Silva, 2015). Importantly, these dimensions can be thought of as valence qualities: voters share a common preference for competent and trustworthy politicians over incompetent and untrustworthy politicians (D. Stokes, 1992; D. E. Stokes, 1963). However, the literature in social psychology finds that high and low status socioeconomic groups typically correspond to ambivalent stereotypes, which are (relatively) high along one dimension and (relatively) low on the other. For example, groups such as professionals and the rich tend to be regarded as high in competence and (relatively) low in warmth, while blue collar workers and the poor tend to be regarded as low in competence and (relatively) high in warmth (Cuddy et al., 2009; Fiske et al., 2002). Thus, if voters use a candidate s socioeconomic status (SES) as a heuristic in this way, they face a trade-off: do they prefer the competence-advantaged high-ses candidate or the character-advantaged low-ses candidate? To examine this trade-off, I combine a conjoint candidate choice design with a priming experiment, in which respondents are either primed to think about the competence-related aspects of a politician s job (such as understanding complex policy issues) or the character-related aspects (such as representing their constituents). The goals of this study can be summarized with two questions: How do voters evaluate candidate socio-economic characteristics? And do these evaluations vary when respondents are primed to think about different dimensions of candidate valence (specifically: competence and character)? This study will focus on the United States, which was chosen as a hard case for the theory, since class has historically been less politically salient in 2

3 Sophie Hill 3 the U.S. compared to other high-income democracies (Gerteis and Savage, 1998; Lipset, 1990). However, the electoral dynamics on which this theory is based apply to a broad range of democracies. Thus, this study aims to contribute to a growing literature in comparative politics that examines why elected office-holders in most democracies are disproportionately drawn from the well-off. 2 Experimental design This experimental design employs two types of randomization: The first type is the randomization of candidate profiles used in the conjoint task. This allows us to isolate the effects of a candidate s socioeconomic characterististics specifically, education and occupation on their overall favorability by estimating average marginal component effects (AMCEs). The second type is a between-subjects design in which respondents are randomly assigned to receive either a competence-related prime or a character-related prime before completing the conjoint task. This allows us to investigate whether priming different valence qualities causally affects the way that candidate socio-economic characteristics are evaluated. Specifically, we can compute the conditional AMCEs based on treatment assignment and explicitly test for differences in effect sizes. The survey has four sections: 1. Occupation coding 2. Demographics 3. Priming manipulation 4. Candidate choice task 2.1 Occupation coding The occupation-coding section asks respondents to classify the eight occupations used in the candidate profiles into the associated social class. The purpose of this module is to 3

4 Sophie Hill 4 gauge whether respondents are able to accurately code these occupations. This is important if we want to be able to interpret the effects of a candidate s occupation in terms of class. Clearly, there is some ambiguity about the correct answers in this context, since there are multiple typologies of class and ways to map occupation onto class. Nevertheless, the occupations used here were deliberately chosen so as to be familiar to most respondents, and relatively straightforward to categorize based on skill-level. When presenting results in the main analysis that relate to the effect of a candidate s occupation, I will also show how these effects are moderated by the respondent s ability to classify the occupations correctly. One potential concern is that this module constitutes a treatment in itself. Since all the occupations listed in this module are possible candidate occupations in the conjoint task, class may become more salient to respondents than it otherwise would have been. To test for and measure the size of any possible spillover effects, I will randomly assign respondents to complete the occupation coding module either at the start or end of the survey. This random assignment is orthogonal to the main priming treatment assignment. I will report the results of the occupation coding module in two ways. First, I will report the percentage of respondents who classified each occupation into the correct social class as defined in Table 1. Second, I will report the percentage of respondents who correctly rank the four pairs (i.e. a less stringest test). To do this, I will assign a numeric value ( 1 for unskilled working-class, 2 for semi-skilled working-class, 3 for middle-class, 4 for upper-class). For each of the four pairs of occupations, I will create an average class score based on the respondent s answers, and check whether the scores 4

5 Sophie Hill 5 are monotonically ascending across the four pairs, i.e.: µ W CUS µ W CSS µ MC µ UC. Table 1: List of occupations used in candidate profiles, with reference information Occupation Median pay Entry-level education Social class Retail salesperson $23,370 No formal requirement Working-class (unskilled) Farm laborer $23,730 No formal requirement Working-class (unskilled) Office clerk $31,500 High school diploma Working-class (semi-skilled) Carpenter $45,170 High school diploma Working-class (semi-skilled) Medical lab tech $51,770 2-year/4-year degree Middle class High school teacher $59,170 4-year degree Middle class Business executive $104,700 4-year degree Upper class Lawyer $119,250 Postgraduate degree Upper class Notes: Median pay in 2017 and typical entry-level education as listed in the Bureau of Labor Statistics Occupational Outlook Handbook 2.2 Demographics The following demographic and political variables will be measured pre-treatment. Each of these variables is included for the purposes of testing for treatment effect heterogeneity (as specified in the hypotheses section below), or for generating hypotheses for future research. While respondents will be pre-screened based on party identification and education (this is how the stratified sample is collected), these items are also included here to reduce measurement error since and it is unknown how often a respondent s pre-screening information is updated. Demographic covariates: Gender Age Household income Household size Employment status 5

6 Sophie Hill 6 Education Race/ethnicity Party ID Political ideology Union membership Subjective class identity 2.3 Priming manipulation Respondents are randomly assigned to receive either a competence-related prime or a character-related prime, with equal probability. The priming text is embedded in the instructions for the conjoint task, as follows: Instructions We want you to imagine that these candidates are running for the U.S. House of Representatives. [Competence prime] Members of the U.S. House of Representatives have to make important decisions and must be able to understand complex issues, like economic policy and national security. [Character prime] Members of the U.S. House of Representatives have to represent their constituents and must be able to understand the problems faced by ordinary Americans. After seeing this screen, respondents proceed to the conjoint task. Since there are a relatively high number of tasks (10 pairs of profiles for the forced choice task), priming effects may decay as respondents proceed. To examine this, I will use the order in which profiles appear (in the first five pairs or the second five pairs) as a moderator variable. 6

7 Sophie Hill Candidate choice task In this section, respondents are shown 10 pairs of randomly-generated candidate profiles. 1 This is a forced choice design, in which respondents are asked to pick the profile of the candidate they would be most likely to vote for. This produces a profile-level dataset, where the outcome variable either takes the value 1 (profile picked) or 0 (profile not picked). The paired forced choice design has been shown to perform well in approximating real-world voting behavior (Hainmueller, Hangartner, and Yamamoto, 2015). The profiles were generated using the Conjoint Survey Design Tool provided by Strezhnev et al. (2013). Table 2 displays the eight profile attributes and their possible values. Figure 1 shows an example of a pair of candidate profiles as they appear to the survey participants. The order of the attributes is randomized but kept constant for each respondent to aid comprehension. The main quantity of interest is the average marginal component effect (AMCE), which represents the average change in probability that a profile is chosen when it includes a given attribute value compared to a researcher-chosen baseline value. If no profile restrictions are imposed, then the AMCE can be recovered by regressing the binary outcome on indicator variables for each level except the omitted baseline category. However, in this context some of the implied counterfactuals such as a lawyer with only a high school diploma would not be meaningful. To account for these cases, a series of pairwise restrictions is imposed on (i) occupation and education; and (ii) party and ideology. Table 3 displays the eight pairwise restrictions imposed. The occupation/education restrictions are imposed because some of the educational levels would not be compatible with the standard minimum entry requirements for that occupation. The party/ideology restrictions are imposed for a more subtle reason: the link between party and ideology 1 A further 2 pairs of profiles are shown after these, where respondents are asked to rate each candidate on competence and trustworthiness. These results will not be part of the main analysis but are included to confirm some of the underlying assumptions of the theory being tested. 7

8 Sophie Hill 8 Table 2: Conjoint profile attributes Attribute Levels Gender Male*, Female Race/ethnicity White*, African American, Hispanic, Asian American Age 34*, 39, 43, 48, 54, 61, 68 Years in office 0*, 1, 2, 4, 6, 8 Party Republican*, Democrat Ideology Very liberal, Liberal, Moderate*, Conservative, Very conservative Education High school*, Community college, State university, Ivy League university Previous occupation Farm laborer, Retail salesperson, Carpenter, Office clerk, Teacher, Medical lab tech, Lawyer*, Business executive Notes: * denotes the baseline categories for initial presentation of results. pairwise restrictions. Profiles are subject to certain Figure 1: Conjoint design: example of profile pair 8

9 Sophie Hill 9 is so strong that respondents will likely interpret the label conservative relative to the partisan label. This means that the counterfactuals are not well-defined. To avoid this ambiguity, restrictions are imposed. The only ideological value that candidates of both parties can share is moderate. Table 3: Restricted attribute combinations Attribute 1 Value 1 Attribute 2 Value 2 Party Democrat Ideology Conservative Party Democrat Ideology Very conservative Party Republican Ideology Liberal Party Republican Ideology Very liberal Occupation Teacher Education High school Occupation Medical lab tech Education High school Occupation Lawyer Education High school Occupation Lawyer Education Community college Notes: These attribute combinations are prohibited. The AMCE can still be estimated as a weighted average of the appropriate coefficients from a linear regression including the indicator variables and interaction terms for any values with randomization restrictions. This can be implemented easily with the R package cjoint (Strezhnev et al., 2014). Other quantities of interest are the average component interaction effect (ACIE), which represents how the causal effect of one attribute varies depending on what value another attribute is held at, and the conditional AMCE, which represents the AMCE conditional on a respondent-level covariate. We can compute the conditional AMCE for demographic covariates, such as the respondent s party ID, as well as by treatment assignment (whether the respondent received the competence-prime or the character-prime). The hypotheses to be tested are listed below in Table 6. All results will be presented 9

10 Sophie Hill 10 graphically, with point estimates displayed along with 95% confidence intervals. Standard errors will be clustered at the respondent-level to account for the correlation of outcomes within pairs (by construction) and the possibility of correlation in respondent-specific error terms (Hainmueller, Hopkins, and Yamamoto, 2014). When computing conditional AMCEs, I will present the AMCEs estimated according to the conditioning variable, as well as the difference between them. In order to ensure that the number of comparisons is manageable, for the interactions I will collapse the eight occupation values into four classes (defined as in Table 1), and I will collapse the four education values into a binary indicator for university degree ( State university or Ivy league university ) and below ( High school or Community college ). For all tests, I will show how the effects are moderated by respondent attention (measured by an attention check question). For tests relating to candidate occupation, I will also show how the effects are moderated by the respondent s ability to classify the occupations into social classes (measured by the occupation coding module). For tests relating to the priming manipulation, I will show how the effects are moderated by the position of the profiles, since the effects may decay over the course of 10 tasks. In addition to these analyses, I will also perform standard diagnostic checks to test for lefthand/righthand profile preferences, profile order effects, attribute order effects, and carryover effects (Hainmueller, Hopkins, and Yamamoto, 2014). I will also show how AMCEs are moderated by the number of atypical profiles seen by a respondent. This refers to attribute combinations that are meaningful and well-defined (and thus not excluded by design) but are empirically rare. It is important to check that respondents are not distracted by these unusual profiles. A profile is defined as atypical if it contains one or more of the attribute combinations listed in Table 4. 10

11 Sophie Hill 11 Table 4: Atypical attribute combinations Attribute 1 Value 1 Attribute 2 Value 2 Occupation Teacher Education Community college Occupation Farm laborer Education Ivy league university Occupation Farm laborer Education State university Occupation Retail salesperson Education Ivy league university Occupation Retail salesperson Education State university Occupation Carpenter Education Ivy league university Occupation Carpenter Education State university Occupation Office clerk Education Ivy league university Occupation Office clerk Education State university Occupation Carpenter Gender Female Race/ethnicity African American Party Republican Race/ethnicity Hispanic Party Republican Race/ethnicity Asian American Party Republican Years in office 8 Age 34 Years in office 8 Age 39 Years in office 6 Age 34 Notes: These attribute combinations are allowed in the randomization but are coded as atypical. 11

12 Sophie Hill 12 3 Data collection A convenience sample of U.S. adults will be recruited via the online survey platform Prolific. The key advantage of Prolific over similar platforms such as Amazon s Mechanical Turk is that it allows researchers to prescreen respondents according to a range of sociodemographic variables. This makes it possible to construct a stratified sample that will correct for some of the known observable differences between online convenience samples and representative samples. For this study, I will stratify on partisan identification and educational attainment. These variables were chosen because (i) U.S. online convenience samples typically skew far more liberal and more educated than a nationally representative sample; (ii) these variables are also important in this study as conditioning variables (thus, for example, it is important to recruit enough Republican respondents to have statistical power to compute AMCEs conditional on respondent party ID). Participants will be recruited according to the proportions shown in Table 5, which were derived from the nationally representative samples collected by the Cooperative Congressional Election Study (Ansolabehere and Schaffner, 2016). Party ID is measured along a three-point scale, and educational attainment is collapsed into a binary variable for a bachelor s degree or above. Table 5: Stratified sample: cell proportions from CCES (2016) Democrat Republican Independent No BA 26% 22% 25% BA or above 11% 7% 9% Simulations indicate that a sample size of n = 1, 200 respondents (where each respondent see 10 profile pairs) is sufficient to detect the smallest priming effect sizes that are judged to be substantively interesting (τ = 0.03 on a 0 1 scale) with 80% power. 12

13 Sophie Hill 13 Figure 2 shows how power varies with sample size, under certain assumptions about the parameter distributions. Figure 2: Simulation results I will aim to recruit participants according to the proportions displayed in Table 5. According to the demographic breakdowns of registered survey-takers on Prolific, it will be feasible to construct this stratified sample. However, it is unknown exactly how many of these registered users are active and available to participant in a study at a given time. Thus, in order to fully specify the data collection strategy, I will adhere to the following rules: If an insufficient number of participants in any one of the six cells has been recruited after 48 hours, I will top-up the remainder by relaxing the constraint on educational attainment. (For example: if an insufficient number of Republicans without bachelor s degrees has been recruited, then I will allow Republicans with any educational attainment to fill up that quota.) If after a further 24 hours the number is still insufficient, I will top-up the remainder from the Prolific respondent pool without either constraint. 2 2 Though all respondents must still meet the eligibility criteria: to be resident in the United States and aged over

14 Sophie Hill 14 Respondents will be paid $1.50 for completing the survey, which is approximately 10 minutes long (i.e. a $9/hr rate). 4 Data processing In the survey, respondents are required to enter their Prolific ID number, which makes it possible to approve their compensation. While the platform should prevent any participant from taking the survey twice, I will check the collected ID s and discard data from any duplicates (that is, only the first response will be included). Respondents are able to skip any question (except the informed consent question on the first screen), and so it is possible for there to be missing values on respondent covariates or on the conjoint task outcomes. Missing values on covariates that are required as conditioning variables (e.g. in computing a conditional AMCE) will be multiply imputed. Missing values on the conjoint task will be dropped from the analysis. I will also regress a dummy variable for missing outcome values on the priming treatment condition, to check for differential rates of attrition/skipping with α = For any aspect of data processing that is not otherwise specified in this pre-analysis plan, I will default to the standard operating procedure given in Lin, Green, and Coppock (2016). 5 Hypotheses This study has two parts: the first is a standard conjoint candidate choice design and the second is a combination of that conjoint design with a between-subjects priming manipulation. Table 6 displays the individual hypotheses, divided into two corresponding groups. For this first set of hypotheses, respondents will be pooled across both treatment conditions. Hypotheses H1 H4 are based on prior findings that voters prefer middle-class and skilled working-class candidates over both unskilled working-class and upper-class 14

15 Sophie Hill 15 candidates (Wüest and Pontusson, 2018). In contrast, H5 and H6 posit that education and experience have more straightforward linear effects, where higher levels of education (as well as more prestigious educational institutions) and higher levels of political experience tend to be preferred. H7 and H8 state the obvious expectation that voters prefer candidates who share their partisan identification and their political ideology. 3 Hypotheses H9 H11 state that upper-class candidates are viewed more negatively by respondents with lower vs. higher socioeconomic status (by subjective class-identification, household income, and education). In contrast, hypothese H12 H14 state that biases against unskilled working-class candidates are shared across respondents of different socioeconomic statuses. Both sets of expectations are consistent with the findings from Wüest and Pontusson (2018). Hypotheses H15 H16 state that political experience crowds out the signals derived from socioeconomic characteristics. Thus, occupation and education are expected to matter less for candidates with more political experience, since their tenure in office provides a more direct signal about their domain-specific qualifications. The second set of hypotheses relate to the priming manipulation, where respondents were encouraged to think about either the competence-related aspects of a politician s job (such as understanding complex policy issues) or the character-related aspects of a politician s job (such as representing constitutents). To investigate whether this priming manipulation affected the way that the candidate profiles were evaluated, we can compute conditional AMCEs, where the conditioning variable is the treatment assignment itself. Each of these hypotheses is based on the theory that priming competence (character) will lead respondents to favor attribute values associated with competence (character) more heavily. As described above, high-status socioeconomic groups are typically seen as high in competence and (relatively) low in warmth/character, whereas lower-status 3 The respondent-level party ID variable will be coded to include leaners, since many of those who identify as independent do consistently vote for one party. Candidate and respondent ideology will be collapsed down to a 3-point liberal/moderate/conservative scale. 15

16 Sophie Hill 16 socioeconomic groups are typically seen as high in warmth/character and (relatively) low in competence. Thus, H17 H19 state that the competence prime, as compared to the character prime, will increase the advantage (or decrease the disadvantage) associated with being from an upper-class occupation and having higher levels of education. I will test for heterogenous treatment effects, but these analyses will be considered exploratory, since I do not have firm theoretical expectations here. All of these hypotheses relate to the forced-choice outcomes (for which each respondent completes 10 paired profile tasks). In addition, respondents see a further two pairs of profiles. For the first, they are asked to rate both candidates in terms of trustworthiness, which measures the character/warmth dimension in language that will be more familiar to respondents. For the second, they are asked to rate both candidates in terms of competence. These tasks are included to complement the main analysis by directly examining whether the proposed mechanisms (perceptions or competence and warmth) apply. In particular, I will test the claims that upper class occupations and higher levels of education contribute to higher competence ratings, and that middle-class and skilled working-class occupations and lower levels of education contribute to higher trustworthiness ratings. 6 IRB approval This study was approved by the Harvard University Institutional Review Board (protocol number: IRB ) on August 24,

17 Sophie Hill 17 Table 6: Summary of hypotheses # Attributes Quantity of interest Expectation Sample pooled across treatment conditions H1 Occupation AMCE MC > UC occupations H2 Occupation AMCE MC > WC US occupations H3 Occupation AMCE WC SS > UC occupations H4 Occupation AMCE WC SS > WC US occupations H5 Education AMCE Higher > lower education H6 Experience AMCE Higher > lower levels of political experience H7 Party Conditional AMCE Co-partisan > out-partisan candidates H8 Ideology Conditional AMCE More proximate > less proximate candidates H9 Occupation Conditional AMCE UC vs. MC candidates < by WC vs. MC respondents H10 Occupation Conditional AMCE UC vs. MC candidates < by low- vs. high-income respondents H11 Occupation Conditional AMCE UC vs. MC candidates < by less vs. more educ respondents H12 Occupation Conditional AMCE WC US vs. MC candidates by WC vs. MC respondents H13 Occupation Conditional AMCE WC US vs. MC candidates by low- vs. high-income respondents H14 Occupation Conditional AMCE WC US vs. MC candidates by less vs. more educ respondents H15 Occ x Experience ACIE Effect of occupation < for high vs. low experience H16 Educ x Experience ACIE Effect of education < for high vs. low experience Priming treatment effects H17 Occupation Conditional AMCE UC occupation > with competence vs. character prime H18 Education Conditional AMCE High education > with competence vs. character prime H19 Experience Conditional AMCE High education > with competence vs. character prime Notes: WC US, WC SS, MC, and UC stand for working-class (unskilled), working-class (semi-skilled), middle-class, and upper-class, respectively. X > Y is used as a shorthand for the effect of X is more positive on candidate favorability than the effect of Y. 17

18 Sophie Hill 18 References Ansolabehere, Stephen and Brian Schaffner (2016). COOPERATIVE CONGRESSIONAL ELECTION STUDY, 2016: COMMON CONTENT.[Computer File] Release 2: August 4, Cambridge, MA: Harvard University [producer]. Bittner, Amanda (2011). Platform or personality?: the role of party leaders in elections. Oxford University Press. Costa, Patrício and Frederico Ferreira da Silva (2015). The impact of voter evaluations of leaders traits on voting behaviour: Evidence from seven European Countries. West European Politics 38.6, pp Cuddy, Amy JC et al. (2009). Stereotype content model across cultures: Towards universal similarities and some differences. British Journal of Social Psychology 48.1, pp Fiske, Susan T, Amy JC Cuddy, and Peter Glick (2007). Universal dimensions of social cognition: Warmth and competence. Trends in cognitive sciences 11.2, pp Fiske, Susan T et al. (2002). A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology 82.6, p Gerteis, Joseph and Mike Savage (1998). The salience of class in Britain and America: a comparative analysis. British Journal of Sociology, pp Hainmueller, Jens, Dominik Hangartner, and Teppei Yamamoto (2015). Validating vignette and conjoint survey experiments against real-world behavior. Proceedings of the National Academy of Sciences 112.8, pp Hainmueller, Jens, Daniel J Hopkins, and Teppei Yamamoto (2014). Causal inference in conjoint analysis: Understanding multidimensional choices via stated preference experiments. Political Analysis 22.1, pp Kinder, Donald R (1986). Presidential character revisited. Political cognition 19, p

19 Sophie Hill 19 Lin, Winston, Donald P. Green, and Alexander Coppock (2016). Standard operating procedures for Don Green s lab at Columbia. Version Lipset, Seymour Martin (1990). Continental Divide: The Values and Institutions of the United States and Canada. Vol. 59. Psychology Press. Miller, Arthur H, Martin P Wattenberg, and Oksana Malanchuk (1986). Schematic assessments of presidential candidates. American Political Science Review 80.2, pp Pancer, S Mark, Steven D Brown, and Cathy Widdis Barr (1999). Forming impressions of political leaders: A cross-national comparison. Political Psychology 20.2, pp Stokes, Donald (1992). Valence politics. Electoral politics, pp Stokes, Donald E (1963). Spatial models of party competition. American political science review 57.02, pp issn: Strezhnev, Anton et al. (2013). Conjoint Survey Design Tool: Software Manual. Strezhnev, Anton et al. (2014). cjoint: AMCE estimator for conjoint experiments. R package version 1.3. Wüest, Reto and Jonas Pontusson (2018). Descriptive Misrepresentation by Social Class: Do Voter Preferences Matter? 19