Instructors: Patrick Brandt Skyler Cranmer and Jong Hee Park

Transcription

1 Advanced Bayesian Models for the Social Sciences ICPSR Second Session, Instructors: Patrick Brandt Skyler Cranmer and Jong Hee Park TA: TBD. Date and Time: July 19 to August 13, 1 3 PM. Office Hours: TBD. Description and Schedule: This course covers the theoretical and applied foundations of Bayesian statistical analysis at a level that goes beyond the introductory course at ICPSR. Therefore knowledge of basic Bayesian statistics (such as that obtained from the Introduction to Applied Bayesian Modeling for the Social Sciences workshop) is assumed. The course will consist of four modules. First, we will discuss model checking, model assessment, and model comparison, with an emphasis on computational approaches. Second, the course will cover Bayesian stochastic simulation (Markov chain Monte Carlo) in depth with an orientation towards deriving important properties of the Gibbs sampler and the Metropolis Hastings algorithms. Extensions and hybrids will be discussed. The third module will focus on Bayesian item response theory (IRT) models, looking at theoretical foundations as well as practical issues such as identification and specification of hierarchies. The fourth week introduces the Bayesian approach to modeling time series data. This includes basic forms as well as recent developments such as Bayesian vector autoregression methods. Throughout the workshop, estimation with modern programming software (R, C, C++, and WinBUGS) will be emphasized. Week I: Bayesian Model Checking, Assessment and Comparison. Skyler Cranmer (University of North Carolina) The first week has three components: assessing model quality, comparing models in a Bayesian context, and standard statistical computing tools that are useful for Bayesian analysis. The emphasis is on in depth technical understanding of the mathematical statistics that justify and govern the use of these tools. Monday: Quick Review of Bayesian Inference 1. This is not intended to be (nor will it be) a substitute for an introductory Bayes course. Rather it will be a refresher to make sure we re all on the same page. 2. Essential Reading: Gill (2007) Chapters 1 4 or equivalent

2 Tuesday: The Bayesian Prior 1. Bayesian Shrinkage 2. (Many) Types of Priors 3. Essential Reading: Gill (2007) Chapter 5 Wednesday: Assessing Model Quality 1. Global Sensitivity Analysis 2. Local Sensitivity Analysis 3. Global Robustness 4. Local Robustness 5. Comparing Data to the Posterior Predictive Distribution 6. Essential Reading: Gill (2007) Chapter 6 Thursday: Model Comparison 1. Posterior Probability Comparison 2. Cross Validation 3. Bayes Factors 4. AIC, BIC, DIC 5. Software Issues 6. Essential Reading: Gill (2007) Chapter 7 Friday: Introduction to Monte Carlo Integration 1. Rejection Sampling 2. Classical Numerical Integration 3. Importance Sampling 4. Mode finding and the EM Algorithm 5. Essential Reading: Gill (2007) Chapter 8 Optional Additional Reading (for the week): a. Carlin, B. P. and Chib, S. (1995). ``Bayesian Model Choice via Markov Chain Monte Carlo Methods.'' Journal of the Royal Statistical Society, Series B 57, b. Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). ``Maximum Likelihood from Incomplete Data via the EM Algorithm.'' Journal of the Royal Statistical Society, Series B 39, c. Kennedy, W. J. and Gentle, J. E. (1980). Statistical Computing. New York: Marcel Dekker. d. Metropolis, N. and Ulam, S. (1949). ``The Monte Carlo Method.'' Journal of the American Statistical Association 44, e. Mooney, C. Z. (1997). Monte Carlo Simulation. Thousand Oaks, CA: Sage. f. Rubin, D. B. (1987). ``A Noniterative Sampling/Importance Resampling Alternative to the Data Augmentation Algorithm for

3 Creating a Few Imputations When Fractions of Missing Information Are Modest: the SIR Algorithm.'' Discussion of Tanner & Wong (1987). Journal of the American Statistical Society 82, Week II: Markov Chain Monte Carlo. Skyler Cranmer (University of North Carolina) This week we continue our focus on computational techniques. We will expand on the idea of Monte Carlo integration introduced last week and then discuss Markov chaings, Markov Chain Monte Carlo, MCMC algorithms (esp. Metropolis Hastings and Gibbs Sampling) and conclude by discussing convergence diagnostics. Monday: Markov Chains 1. What are Markov Chains? 2. Some Simple Examples 3. Marginal Distributions 4. Properties of Markov Chains 5. The Ergodic Theorem 6. Essential Reading Tuesday: Gibbs Sampling 1. The Gibbs Sampler 2. Software Topic: Bayesian Analysis with MCMCpack Wednesday: Metropolis Hastings 1. The Metropolis Hastings Algorithm 2. The Hit and Run Algorithm 3. Software Topic: Bayesian Analysis with WinBUGS Thursday: Convergence Diagnostics 1. Trace Plots 2. Running mean plots 3. Density/HPD plots 4. The Geweke Diagnostic 5. The Gelman and Rubin Diagnostic 6. The Raftery and Lewis Diagnostic 7. The Heidelberger and Welch Diagnostic Friday: Convergence Diagnostics (cont) 1. Finish whatever we did not cover Thursday (~ 1/3) 2. Software topic: Using the CODA and BOA packages in R

4 Optional Additional Reading (for the week): a. Casella, G. and George, E. I. (1992). ``Explaining the Gibbs Sampler.'' The American Statistician 46, b. Gelfand, A. E. and Smith, A. F. M. (1990). ``Sampling Based c. Approaches to Calculating Marginal Densities.'' Journal of the American Statistical Association 85: d. Geman, S. and Geman, D. (1984). ``Stochastic Relaxation, Gibbs Distributions and the Bayesian Restoration of Images.'' IEEE Transactions on Pattern Analysis and Machine Intelligence 6, e. Geyer, C. J. (1992). ``Practical Markov Chain Monte Carlo.'' Statistical Science 7, f. Hastings, W. K. (1970). ``Monte Carlo Sampling Methods Using Markov Chains and Their Applications.'' Biometrika 57, g. Jackman, S. (2000). ``Estimation and Inference via Bayesian Simulation: An Introduction to Markov Chain Monte Carlo.'' American Journal of Political Science 44, h. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller E. ``Equation of State Calculations by Fast Computing Machine.'' Journal of Chemical Physics 21, i. Peskun, P. H. (1973). ``Optimum Monte Carlo Sampling Using Markov Chains.'' Biometrika 60, j. Tierney, L. (1994). ``Markov Chains for Exploring Posterior Distributions.'' Annals of Statistics 22, k. Cowles, M. K., Roberts, G. O., and Rosenthal, J. S. (1999). ``Possible Biases Induced by MCMC Convergence Diagnostics.'' Journal of Statistical Computation and Simulation 64, l. Gelfand, A. E. and Sahu, S. K. (1994). ``On Markov Chain Monte Carlo Acceleration.'' Journal of Computational and Graphical Statistics 3, m. Gelman, A., Rubin, D. B. (1992). ``Inference from Iterative Simulation Using Multiple Sequences.'' Statistical Science 7, n. Geyer, C. J. (1992). ``Practical Markov Chain Monte Carlo.'' Statistical Science 7, o. Zellner, A. and Min, C K. (1995). ``Gibbs Sampler ConvergenceCriteria.'' Journal of the American Statistical Association 90,

5 WEEK III: Bayesian Methods for Ideal Point Estimation. Jong Hee Park (University of Chicago). This module covers theoretical foundations and Bayesian estimations of item response theory (IRT) models. We start from the history of roll call data analysis in political science and social sciences and discuss the connection between spatial voting models and item response theory models. We learn Bayesian implementation of item response theory models using Markov chain Monte Carlo methods. Then, we review important extensions of IRT models such as the issue of multidimensionality, dynamic ideal points, and hierarchical modeling. Students are expected to have basic understandings of Bayesian statistics and working knowledge of R programming. Monday: History of Roll Call Data Analysis James Enelow and Melvin Hinich The Spatial Theory of Voting. New York: Cambridge University Press. Keith Poole and Howard Rosenthal A Spatial Model of Legislative Roll Call Analysis," American Journal of Political Science, Keith Poole Spatial Models of Parliamentary Voting. New York: Cambridge University Press. James J. Heckman and James M. Snyder Jr Linear Probability Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators," The RAND Journal of Economics, 28: Keith Poole, 2000, Non parametric unfolding of binary choice data" Political Analysis 8: Tuesday: Binary Choice Models and IRT Models Daniel McFadden Conditional Logit Analysis of Qualitative Choice Behavior." in P. Zarembka (ed.), Frontiers of Econometrics. New York: Academic Press. J. Scott Long Ch. 3. Binary Outcomes: The Linear Probability, Probit, and Logit Models" in Regression Models for Categorical and Limited Dependent Variables (Advanced Quantitative Techniques in the Social Sciences). Sage Publications. George Rasch Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: The University of Chicago Press. Yoshio Takane and Jan de Leeuw, On the relationship between item response theory and factor analysis of discretized variables," Psychometrika, 52(3):

6 Jim Albert and Siddhartha Chib Bayesian Analysis of Binary and Polychotomous Response Data" Journal of the American Statistical Association, 88: Wednesday: Bayesian Estimation of IRT Models in Political Science Chapter 6. Johnson, Valen, and James Albert Ordinal Data Modeling. New York: Springer. Simon Jackman, Multidimensional Analysis of Roll Call Data via Bayesian Simulation: Identification, Estimation, Inference, and Model Checking." Political Analysis. 9: Joshua Clinton, Simon Jackman, and Doug Rivers, 2004, The Statistical Analysis of Roll Call Voting: A Unified Approach," American Political Science Review 98: Joseph Bafumi, Andrew Gelman, David Park, and Noah Kaplan. 2005, Practical Issues in Implementing and Understanding Baysian Ideal Point Estimation." Political Analysis 13: John Londregan Estimating Legislators Preferred Points, Political Analysis 8: Thursday: Extensions, Applications, and Critiques Michael Bailey and Kelly H. Chang, 2001, Comparing Presidents, Senators, and Justices: Interinstitutional Preference Estimation" Journal of Law, Economics, and Organization 17: Andrew Martin and Kevin Quinn Dynamic Ideal Point Estimation via Markov Chain Monte Carlo for the US Supreme Court, Political Analysis 10: Johsua Clinton and Adam Meirowitz Testing Accounts of Legislative Strategic Voting: The Compromise of 1790," American Journal of Political Science 48(4): Daniel E. Ho and Kevin M. Quinn, 2008, Measuring Explicit Political Positions of Media," Quarterly Journal of Political Science, 3: Michael Peress, 2009, Small Chamber Ideal Point Estimation." Political Analysis 17: Jong Hee Park, Analysis of Preference Changes using Bayesian Changepoint Item Response Theory Model, To appear in Steve Brooks, Andrew Gelman, Galin Jones and Xiao Li Meng, eds., Handbook of Markov Chain Monte Carlo, Chapman & Hall/CRC Press.

7 Friday: Software Implementations Example BUGS and R codes. Week IV: Bayesian Time Series. Patrick Brandt (University of Texas, Dallas). This week's material is divided into five parts corresponding roughly to the presentations on each of the five days. We begin with a brief review of frequentist approaches to analyzing multiple time series. Topics covered include Granger causality, innovation accounting and unit root testing. The role that asymptotic theory plays in these and other aspects of the frequentist approach is stressed (e.g., in the construction of error bands for impulse responses in (S)VAR models and in knife-edge type tests for non-stationarity). We then turn, in parts two and three, to Bayesian time series analysis. We discuss time series priors and how elicitation and elucidation are used to construct these priors. Part three studies the Sims-Zha prior and analyzes in more depth such things as the way error bands are constructed for impulse responses. We illustrate the application of this prior and of likelihood shaped error bands. This includes a presentation of a recent piece on forecasting conflict and cooperation in the Levant. Problems of computation and of model evaluation are studied in part four. A topic from the frontiers of Bayesian time series analysis is presented on the last day of the week--markov-switching. Frequentist approaches to studying Markov switching are reviewed. Then some new developments in the Bayesian approach to analyzing switching are presented. The application on this last day is an application of a Markov-switching BVAR for forecasting. Students will be given some experience using two software packages: RATS and Brandt's MSBVAR. Details of the week follow. Monday: Review. The frequentist approach to analyzing multiple time series 1. Principles/basic time series concepts D 2. Granger causality, impulse responses, DFEVS (and relationships to univariate analogs). 3. Topics a. Vector autoregressions b. Error correction models c. Structural VARs 4. Fitting frequentist models in STATA 5. Required Reading: a. Brandt, Patrick T and John T. Williams Multiple Time Series Models Thousand Oaks: Sage Press. b. Freeman John R., John T Williams, and Tse-min Lin "Vector Autoregression and the Study of Politics" American Journal of Political Science, Vol. 33, No. 4 (Nov. 1989), pp Reference Reading:

8 a. Stock, J. and M. Watson "Vector Autoregressions". The Journal of Economic Perspectives, Vol. 15, No. 4 (Autumn, 2001), pp b. Sims, C.A "Money, Income, and Causality" The American Economic Review, Vol. 62, No. 4 (Sep., 1972), pp c. Sims, C.A "Macroeconomics and Reality". Econometrica, Vol. 48, No. 1 (Jan., 1980), pp d. Sims, C.A., J. Stock, and M. Watson "Inference in Linear Time Series Models with some Unit Roots" Econometrica, Vol. 58, No. 1 (Jan., 1990), pp e. DeBoef, S. and J. Granato "Near-Integrated Data and the Analysis of Political Relationships." American Journal of Political Science, Vol. 41, No. 2 (Apr., 1997), pp Tuesday: Bayesian time series analysis, Part One 1. Principles D 2. Time series priors, elicited and elucidated 3. VAR v. BVAR example 4. Fitting BVARs in RATS and MSBVAR 5. Required Reading: a. Brandt, Patrick T. and John R. Freeman "Advances in Bayesian Time Series Modeling and the Study of Politics: Theory Testing, Forecasting, and Policy Analysis." Political Analysis 14(1):1-36. b. Robertson, J. and E. Tallman "Vector Autoregressions: Forecasting and Reality." Economic Review (Federal Reserve Bank of Atlanta). First Quarter 1999/Volume 84, Number1. c. Zha, T. "A Dynamic Multivariate Model for Use in Formulating Policy". Economic Review (Federal Reserve Bank of Atlanta) First Quarter 1998/Volume 83, Number 1. d. McGinnis, Michael D. and John T. Williams "Change and Stability in Superpower Rivalry" The American Political Science Review, Vol. 83, No. 4 (Dec., 1989), pp Reference Reading: a. Robertson, J. and E. Tallman "Improving Forecasts of the Federal Funds Rate in a Policy Model" March Federal Reserve Bank of Atlanta Working Paper, b. Sims, C.A. and H. Uhlig "Understanding Unit Rooters: A Helicopter Tour". Econometrica, Vol. 59, No. 6 (Nov., 1991), pp

9 c. Uhlig, H "What Macroeconomists Should Know about Unit Roots: A Bayesian Perspective" Econometric Theory, Vol. 10, No. 3/4, Symposium Double Issue: Bayes Methods and Unit Roots (Aug. - Oct., 1994), pp d. Doan, Thomas and Robert Litterman and Christopher Sims "Forecasting and Conditional Projection Using Realistic Prior Distributions. Econometric Reviews. 3: e. Kadane, J.B., N.H. Chan, and L. Wolfson "Priors for unit root models." Journal of Econometrics. 75: f. Litterman, R "A Statistical Approach to Economic Forecasting" Journal of Business & Economic Statistics, Vol. 4, No. 1 (Jan., 1986), pp g. Litterman, R "Forecasting with Bayesian Vector Autoregressions: Five Years of Experience" Journal of Business & Economic Statistics, Vol. 4, No. 1 (Jan., 1986), pp Stable URL: Wednesday: Bayesian time series analysis, Part Two 1. Error bands for impulse response functions, The concept of Bayesian shape error bands 2. Testing theories with Bayesian SVAR models 3. Forecasting with Bayesian VAR models 4. Introduction to the package MSBVAR 5. Required readings a. Brandt, Patrick T. and John R. Freeman "Advances in Bayesian Time Series Modeling and the Study of Politics: Theory Testing, Forecasting, and Policy Analysis." Political Analysis 14(1):1-36. b. Brandt, Patrick T., Michael P. Colaresi and John R. Freeman "The Dynamics of Reciprocity, Accountability and Credibility" Journal of Conflict Resolution 52(3): c. Sattler, Thomas, John R. Freeman and Patrick T. Brandt "Popular Sovereignty and the Room to Maneuver: A Search for a Causal Chain" Comparative Political Studies. df (Get this version, since the published one is incorrect!) d. Sattler, Thomas, Patrick T. Brandt and John R. Freeman Democratic Accountability in Open Economies. Quarterly Journal of Political Science. 5(1): Reference papers a. Sims, C.A. and T. Zha "Bayesian Methods for Dynamic Multivariate Models". International Economic Review, Vol. 39, No. 4, Symposium on Forecasting and Empirical Methods in Macroeconomics and Finance (Nov., 1998), pp

10 b. Sims, C.A. and T. Zha "Error Bands for Impulse Responses". Econometrica, Vol. 67, No. 5 (Sep., 1999), pp c. Waggoner, D. and T. Zha "Conditional Forecasts in Dynamic Multivariate Models". The Review of Economics and Statistics, Vol. 81, No. 4 (Nov., 1999), pp Thursday: Practical problems in using Bayesian time series models 1. Computational challenges 2. Formulating priors 3. Assessing model fit 4. More on MSBVAR 5. Required readings: a. Brandt, Patrick T. and John R. Freeman "Modeling Macro Political Dynamics". Political Analysis. 17(2): b. Waggoner, Daniel F. and Tao A. Zha "Likelihood Preserving Normalization in Multiple Equation Models". Journal of Econometrics. 114: c. Waggoner, Daniel F. and Tao A. Zha "A Gibbs sampler for structural vector autoregressions". Journal of Economic Dynamics & Control. 28: d. Notes on MSBVAR, R Package for B-(S)VAR models. 6. Reference papers: a. Hamilton, James, Daniel Waggoner and Tao Zha, "Normalization in Econometrics". Econometric Reviews, 2007, 26(2-4): Friday: Frontiers of research in Bayesian time series analysis --The concept of switching 1. Frequentist approaches, Testing for structural change with frequentist methods, Hamilton's switching model 2. Application: the political economy of exchange rates 3. The MSBVAR model 4. Applications 5. Required readings: a. Hamilton, James "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle". Econometrica. 57(2): b. Freeman, John R., Jude C. Hays and Helmut Stix "Democracy and Markets: The Case of Exchange Rates" American Journal of Political Science 44(3): c. Hays, Jude C., John R. Freeman and Hans Nesseth "Exchange Rate Volatility and Democratization in Emerging Market Countries" International Studies Quarterly 47(2):

11 d. Brandt, Patrick Empirical Evidence for and tests of conflict phases in international and regional conflicts. Annual Meeting of the Midwest Political Science Association. 6. Reference papers: a. Sims, C.A., D. Waggoner, and T. Zha METHODS FOR INFERENCE IN LARGE MULTIPLE-EQUATION MARKOV- SWITCHING MODELS. Journal of Econometrics 146(2): b. Kim, Chang-Jin and Charles R. Nelson State-Space Models with Regime Switching. Cambridge: MIT Press c. Krolzig, Hans-Martin Markov-Switching Vector Autoregressions: Modelling, Statistical Inference, and Application to Business Cycle Analysis. Berlin: Springer. d. Kim, C. J. Piger, and and R. Startz "Estimation of Markov regime-switching regression models with endogenous switching." Journal of Econometrics. 143: