A Bayesian Measurement Model of Political Support for Endorsement Experiments, with Application to the Militant Groups in Pakistan Kosuke Imai Princeton University Joint work with Will Bullock and Jacob Shapiro July 12, 2010 Kosuke Imai (Princeton) Endorsement Experiments Renmin University 1 / 23
Motivation I Measuring support for political actors (e.g., candidates, parties) Standard survey questions: Likert scale, feeling thermometer, etc. Sensitive questions: nonresponse, social desirability bias, safety concerns Existing survey techniques: Randomized response technique Item count technique Endorsement experiments: Ask respondents to rate their support for a set of policies endorsed by randomly assigned political actors Kosuke Imai (Princeton) Endorsement Experiments Renmin University 2 / 23
Motivation II Item response theory: Originates in the educational testing literature Ideal point estimation in US Congress Scaling justices (via votes), newspapers (via editorials), etc. A Bayesian measurement model for endorsement experiments: Measuring support and issue ownership on the ideal points scale Hierarchical modeling for efficient partial pooling Individual level covariates and poststratification Kosuke Imai (Princeton) Endorsement Experiments Renmin University 3 / 23
Endorsement Experiments Goals: 1 Measure the strength of support and issue ownership 2 Reduce nonresponse rate 3 Minimize social desirability bias 4 Address safety concerns Experimental design: 1 Select policy questions 2 Randomly divide sample into control and treatment groups 3 Across respondents and questions, randomly assign political actors for endorsement (no endorsement for the control group) 4 Compare support level for each policy endorsed by different actors Kosuke Imai (Princeton) Endorsement Experiments Renmin University 4 / 23
The Pakistani Survey Experiment 6,000 person urban-rural sample Four different groups: Pakistani militants fighting in Kashmir (a.k.a. Kashmiri tanzeem) Militants fighting in Afghanistan (a.k.a. Afghan Taliban) Al-Qa ida Firqavarana Tanzeems (a.k.a. sectarian militias) Four policies: WHO plan to provide universal polio vaccination across Pakistan Curriculum reform for religious schools Reform of FCR to make Tribal areas equal to rest of the country Peace jirgas to resolve disputes over Afghan border (Durand Line) Response rate; over 90% Kosuke Imai (Princeton) Endorsement Experiments Renmin University 5 / 23
Distribution of Responses Punjab Sindh NWFP Firqavarana Tanzeems Al Qaida Afghan Taliban Pakistani militant groups in Kashmir Control Group Firqavarana Tanzeems Al Qaida Afghan Taliban Pakistani militant groups in Kashmir Control Group Firqavarana Tanzeems Al Qaida Afghan Taliban Pakistani militant groups in Kashmir Control Group Polio Vaccinations Curriculum Reform FCR Reforms Durand Line Balochistan Firqavarana Tanzeems Al Qaida Afghan Taliban Pakistani militant groups in Kashmir Control Group 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Not At All A Little A Moderate Amount Kosuke Imai (Princeton) Endorsement Experiments Renmin University 6 / 23 A Lot A Great Deal
Endorsement Experiments Framework Data from an endorsement experiment: N respondents J policy questions K political actors Y ij {0, 1}: response of respondent i to policy question j T ij {0, 1,..., K }: political actor randomly assigned to endorse policy j for respondent i Y ij (t): potential response given the endorsement by actor t Covariates measured prior to the treatment Kosuke Imai (Princeton) Endorsement Experiments Renmin University 7 / 23
The Proposed Model Quadratic random utility model: U i (ζ j1, k) = (x i + s ijk ) ζ j1 2 + η ij, U i (ζ j0, k) = (x i + s ijk ) ζ j0 2 + ν ij, where x i is the ideal point and s ijk is the support level The statistical model (item response theory): Pr(Y ij = 1 T ij = k) = Pr(Y ij (k) = 1) = Pr(U i (ζ j1, k) > U i (ζ j0, k)) Hierarchical modeling: = Pr(α j + β j (x i + s ijk ) > ɛ ij ) x i s ijk λ jk indep. indep. N (Z i δ, σ 2 x) N (Z i λ jk, ω 2 jk ) i.i.d. N (θ k, Φ k ) Noninformative hyper prior on (α j, β j, δ, θ k, ω 2 jk, Φ k) Kosuke Imai (Princeton) Endorsement Experiments Renmin University 8 / 23
Quantities of Interest and Model Fitting Average support level for each militant group k τ jk (Z i ) = Zi λ jk for each policy j κ k (Z i ) = Zi θ k averaging over all policies Standardize them by dividing the (posterior) standard deviation of ideal points Issue ownership: variation of average support for each group across policies Bayesian Markov chain Monte Carlo algorithm Multiple chains to monitor convergence Implementation via JAGS (Plummer) Kosuke Imai (Princeton) Endorsement Experiments Renmin University 9 / 23
Model for the Division Level Support Ordered response with an intercept α jl varying across divisions The model specification: x i s ijk δ division[i] λ k,division[i] indep. N (δ division[i], 1) indep. N (λ k,division[i], ω 2 k ) indep. N (µ province[i], σ 2 province[i] ) indep. N (θ k,province[i], Φ k,province[i] ) Averaging over policies Partial pooling across divisions within each province Kosuke Imai (Princeton) Endorsement Experiments Renmin University 10 / 23
Estimated Division Level Support Standardized Level of Support 1.0 0.5 0.0 0.5 1.0 Pakistani militant groups in Kashmir Militants fighting in Afghanistan Al Qaida Firqavarana Tanzeems Bahawalpur n=118 Dera Ghazi Khan n=0 Faisalabad n=313 Gujranwala n=403 Lahore n=579 Multan n=495 Rawalpindi n=208 Sargodha n=131 Hyderabad n=203 Karachi n=473 Larkana n=311 Mirpurkhas n=0 Sukkur n=293 Bannu n=0 Dera Ismail Khan n=84 Hazara n=287 Kohat n=50 Malakand n=0 Mardan n=215 Peshawar n=288 Kalat n=103 Makran n=0 Nasirabad n=210 Quetta n=320 Sibi n=67 Punjab Sindh NWFP Balochistan Zhob n=61 Kosuke Imai (Princeton) Endorsement Experiments Renmin University 11 / 23
Model with Individual Covariates Ordered response with an intercept α jl varying across divisions The model specification: x i s ijk δ division[i] λ k,division[i] indep. indep. N (δ division[i] + Z i δ Z, 1) N (λ k,division[i] + Z i λ Z k, ω2 k ) indep. N (µ province[i], σ 2 province[i] ) indep. N (θ k,province[i], Φ k,province[i] ) Expands upon the division level model to include individual level covariates: gender, urban/rural, education, income Individual level covariate effects after accounting for differences across divisions Poststratification on these covariates using the census Kosuke Imai (Princeton) Endorsement Experiments Renmin University 12 / 23
Estimated Effects of Individual Covariates Pakistani militant groups in Kashmir Militants fighting in Afghanistan Standardized Level of Support 0.2 0.1 0.0 0.1 0.2 Female Rural Income Education Standardized Level of Support 0.2 0.1 0.0 0.1 0.2 Female Rural Income Education Al Qaida Firqavarana Tanzeems Standardized Level of Support 0.2 0.1 0.0 0.1 0.2 Female Rural Income Education Standardized Level of Support 0.2 0.1 0.0 0.1 0.2 Female Rural Income Education Demographics play a small role in explaining support for groups Kosuke Imai (Princeton) Endorsement Experiments Renmin University 13 / 23
Regional Clustering of the Support for Al-Qaida Kosuke Imai (Princeton) Endorsement Experiments Renmin University 14 / 23
Correlation between Support and Violence Pakistani militant groups in Kashmir Militants fighting in Afghanistan 400 400 Number of Incidents 300 200 100 0 Number of Incidents 300 200 100 0 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.4 0.3 0.2 0.1 0.0 0.1 0.2 Division Level Estimated Support Division Level Estimated Support Al Qaida Firqavarana Tanzeems 400 400 Number of Incidents 300 200 100 0 Number of Incidents 300 200 100 0 0.4 0.3 0.2 0.1 0.0 0.1 0.2 Division Level Estimated Support 0.4 0.3 0.2 0.1 0.0 0.1 0.2 Division Level Estimated Support Kosuke Imai (Princeton) Endorsement Experiments Renmin University 15 / 23
Simulation Studies 1 Based on the Pakistani Data Same 2 models plus province-level issue ownership model Top-level parameters held constant across simulations Sample sizes and distribution same as before Ideal points, endorsements and responses follow IRT models 2 Varying sample sizes Model for division-level estimates with no covariates Model for province-level estimates with no covariates but support varying across policies N = 1000, 1500, 2000 Again, top-level parameters held constant across simulations while ideal points, endorsements and responses follow IRT models 100 simulations under each scenario (3 chains, 60000 iterations) Frequentist evaluation of Bayesian estimators Kosuke Imai (Princeton) Endorsement Experiments Renmin University 16 / 23
Monte Carlo Evidence based on the Pakistani Data The Issue Province Model The Division Model Density Density 0 10 20 30 40 0 10 20 30 40 Bias 0.05 0.00 0.05 Density Density 0 10 20 30 40 0 10 20 30 40 Coverage Rate of the 90% Confidence Intervals 0.80 0.85 0.90 0.95 1.00 Proportion Statistically Significant Proportion Statistically Significant 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Statistical Power α level = 0.1 0.0 0.2 0.4 0.6 0.8 1.0 Effect Size 0.05 0.00 0.05 0.80 0.85 0.90 0.95 1.00 0.0 0.2 0.4 0.6 0.8 1.0 Effect Size The Division Model With Individual Covariates Density 0 10 20 30 40 Density 0 10 20 30 40 Proportion Statistically Significant 0.0 0.2 0.4 0.6 0.8 1.0 0.05 0.00 0.05 0.80 0.85 0.90 0.95 1.00 0.0 0.2 0.4 0.6 0.8 1.0 Kosuke Imai (Princeton) Endorsement Experiments Renmin University 17 / 23 Effect Size
Monte Carlo Evidence with Varying Sample Size The Division Model Bias 0.2 0.1 0.0 0.1 0.2 Bias Coverage Rate 1.05 1.00 0.95 0.90 0.85 0.80 0.75 Coverage Rate of the 90% Confidence Intervals Proportion Statistically Significant 1.0 0.8 0.6 0.4 0.2 0.0 Statistical Power α level = 0.10 N=2000 N=1500 N=1000 N=1000 N=1500 N=2000 N=1000 N=1500 N=2000 0.0 0.2 0.4 0.6 0.8 1.0 Effect Size The Issue Province Model Bias 0.2 0.1 0.0 0.1 0.2 Coverage Rate 1.05 1.00 0.95 0.90 0.85 0.80 0.75 Proportion Statistically Significant 1.0 0.8 0.6 0.4 0.2 0.0 N=2000 N=1500 N=1000 N=1000 N=1500 N=2000 N=1000 N=1500 N=2000 0.0 0.2 0.4 0.6 0.8 1.0 Effect Size The Division Model With Individual Covariates Bias 0.2 0.1 0.0 0.1 0.2 Coverage Rate 1.05 1.00 0.95 0.90 0.85 0.80 0.75 Proportion Statistically Significant 1.0 0.8 0.6 0.4 0.2 0.0 N=2000 N=1500 N=1000 N=1000 N=1500 N=2000 N=1000 N=1500 N=2000 0.0 0.2 0.4 0.6 0.8 1.0 Kosuke Imai (Princeton) Endorsement Experiments Renmin University 18 / 23 Effect Size
Concluding Remarks Endorsement experiment as an alternative to the randomized response technique when studying sensitive questions A hierarchical Bayesian measurement model; partial pooling across policies and regions Empirical findings: Substantial within-province variation in support for militancy Small across-group variation in support Conditional on division effects, covariates matter relatively little The politics of militancy are intensely local Simulation evidence: Model and estimation procedure have good frequentist properties Similar studies with smaller sample sizes are also feasible Kosuke Imai (Princeton) Endorsement Experiments Renmin University 19 / 23
Further Research on Sensitive Survey Questions Item count technique as another alternative to the randomized response technique Also known as list experiment and unmatched count technique Use aggregation to protect privacy Example: The 1991 National Race and Politics Survey The script for the randomized control group Now I m going to read you three things that sometimes make people angry or upset. After I read all three, just tell me HOW MANY of them upset you. (I don t want to know which ones, just how many.) (1) the federal government increasing the tax on gasoline; (2) professional athletes getting million-dollar-plus salaries; (3) large corporations polluting the environment. Kosuke Imai (Princeton) Endorsement Experiments Renmin University 20 / 23
Further Research on Sensitive Survey Questions Item count technique as another alternative to the randomized response technique Also known as list experiment and unmatched count technique Use aggregation to protect privacy Example: The 1991 National Race and Politics Survey The script for the randomized treatment group Now I m going to read you four things that sometimes make people angry or upset. After I read all four, just tell me HOW MANY of them upset you. (I don t want to know which ones, just how many.) (1) the federal government increasing the tax on gasoline; (2) professional athletes getting million-dollar-plus salaries; (3) large corporations polluting the environment. (4) a black family moving next door to you. Kosuke Imai (Princeton) Endorsement Experiments Renmin University 21 / 23
New Methodology for Item Count Technique Easy for researchers to implement Easy for respondents to understand More widely applicable than endorsement experiments Need to carefully choose non-sensitive items Aggregation = loss of efficiency Need for multivariate analysis but not possible with existing methods Two new regression estimators Two-step nonlinear least squares Maximum likelihood estimators with EM algorithm Bayesian hierarchical modeling Development of an R package Application in Afganistan; support for Taliban Kosuke Imai (Princeton) Endorsement Experiments Renmin University 22 / 23
Project Reference Bullock, Will, Kosuke Imai, and Jacob Shapiro. Measuring Political Support and Issue Ownership Using Endorsement Experiments, with Application to Militant Groups in Pakistan. Imai, Kosuke. Statistical Inference for the Item Count Technique. Available at http://imai.princeton.edu/projects/sensitive.html Kosuke Imai (Princeton) Endorsement Experiments Renmin University 23 / 23