Part 1: Modelling and Estimation. Maximum Likelihood Estimation. A nonparametric regression smoother. Social Science and Parametric Models

Transcription

1 Part 1: Modelling and Estimation Maximum Likelihood Estimation Charles H. Franklin What is a model How do we estimate its parameters? What are the properties of the estimator? franklin@polisci.wisc.edu University of Wisconsin Madison Lecture 1 Parametric Models and History Last Modified: June 13, 2005 Maximum Likelihood Estimation p.1/44 Maximum Likelihood Estimation p.2/44 Social Science and Parametric Models A nonparametric regression smoother The goal of social science is parsimonious explanation of social phenomena Parsimonious because we can never explain every detail Explanation because we want more than mere description Compare with the nonparametric approach: Bush minus Gore Percent Presidential Horserace Polls with a lowess nonparametric fit Mar May Jul Sep Nov Date Maximum Likelihood Estimation p.3/44 Maximum Likelihood Estimation p.4/44

2 How d he do that? (in R) Digression on coding dogma ## Create Horserace lowess plot attach(horserace) postscript(file="horserace.eps", onefile=false, hor=false, wid=6,he=6) plot(enddate,lead,type="n", main="2000 Presidential Horserace Polls \n with a lowess nonparametric fit", xlab="date", ylab="bush minus Gore Percent") points(enddate,jitter(lead),pch=1,cex=.6, col="red") lines(lowess(lead enddate,f=1/10),col="blue") abline(0,0) dev.off() # Open an empty plot: type="n" suppresses points and lines but # scales axes correctly for the x and y variables. # The "\n" in the main title is the line-break command to split # the title across two lines. # Plot the points, adding a little noise to reduce overprinting of # data, use plot character 1 (open circle), set the size # to.6 of normal and color the points red # Add a line at zero to make it clear who is ahead # (0,0) specifies a zero intercept and zero slope, # hence a flat line at y=zero. ALWAYS comment your code. Worry more about saying too little rather than saying too much in comments. Use indentation to visually clarify blocks of code, such as multiple lines for one command or multiple commands that produce one logical step. Plan for your code to be run from source files rather than interactively. Interactive analysis is great for exploration but is terrible for analysis on which your reputation depends. So create command source files with all your analysis, and create final plots that go to files rather than the screen. Maximum Likelihood Estimation p.5/44 Maximum Likelihood Estimation p.6/44 Nonparametric virtues Nonparametric vices 2000 Presidential Horserace Polls with a lowess nonparametric fit 2000 Presidential Horserace Polls with a lowess nonparametric fit Bush minus Gore Percent Bush minus Gore Percent Mar May Jul Sep Nov Date Mar May Jul Sep Nov Date Very flexible, can fit any pattern of data Makes minimal (virtually no) assumptions about data Can reveal unexpected patterns and departures from linear assumptions Too flexible. Sensitive to overfitting Without parameters there is no simple interpretation of effects Hard to incorporate substantive theory and tests Maximum Likelihood Estimation p.7/44 Maximum Likelihood Estimation p.8/44

3 A nonparametric future? What is a parametric model? A great deal of research on modern nonparametric methods is going on. Lots of new developments. But for social scientists, perhaps not the wave of the future. The reason is that parametric models can do a lot for us. We begin with the specification of a specific distribution describing the behavior under study. Specification requires theoretical understanding. Specification also requires making assumptions explicit. While this places a considerable burden on our theory, it forces us to confront the limits of our knowledge and helps avoid making implicit and unwarranted assumptions. Specification should make our assumptions clear to all, including ourselves! Maximum Likelihood Estimation p.9/44 Maximum Likelihood Estimation p.10/44 Specification Examples of specific models We are concerned with the estimation of parametric models of the form: y i f(θ,x i ) where θ is a vector of parameters x i is a vector of exogenous characteristics of the ith observation. The specific functional form, f, provides an almost unlimited choice of specific models. Poisson: Binomial: Normal: y i e λ λ y i y i! ( ) N y i π y i (1 π) N y i y i y i ( 1 e 1 2 2πσ 2 ) (y i µ) 2 σ 2 Maximum Likelihood Estimation p.11/44 Maximum Likelihood Estimation p.12/44

4 How did we get where we are? Methods of The history of estimation is about 250 years old. Bayesian methods is both the oldest in theory and the most recent in practice. Least squares is the oldest practical method, a product of the turn of the 19th century. Maximum likelihood is a child of the early 20th Century. Let s see who our ancestors are. Adrien Marie Legendre ( ) first published the method in 1805 a clear explanation and a worked example in his Nouvelles Méthodes pour la détermination des orbites des Comètes. Maximum Likelihood Estimation p.13/44 Maximum Likelihood Estimation p.14/44 Carl Friedrich Gauss ( ) immediately claimed in correspondence, and in print in 1809, that he had discovered the method and had been using it since The principle which I have used since 1794, that the sum of squares must be minimized for the best representation of several quantities which cannot all be represented exactly, is also used in Legendre s work and is most thoroughly developed. Gauss to W. Olbers, 30 July Maximum Likelihood Estimation p.15/44 Maximum Likelihood Estimation p.16/44

5 ... our principle, which we have made use of since the year 1795, has lately been published by Legendre... Gaussin Theoria Motus Corporum Coelestium (Theory of the Motion of the Heavenly Bodies), Maximum Likelihood Estimation p.17/44 I will therefore not conceal from you, Sir, that I felt some regret to see that in citing my memoir p. 221 you say principium nostrum quo jam inde ab anno 1795 usi sumus etc. There is no discovery that one cannot claim for oneself by saying that one had found the same thing some years previously; but if one does not supply the evidence by citing the place where one has published it, this assertion becomes pointless and serves only to do a disservice to the true author ofthediscovery....youhavetreasuresenough of your own, Sir, to have no need to envy anyone. Legendre to Gauss, 31 May Maximum Likelihood Estimation p.18/44 Perhaps you will find an opportunity sometime, to attest publicly that I already stated the essential ideas to you at our first personal meeting in I find among my papers that in June 1798, when the method was one which I had long applied, I first saw Laplace s method and indicated its incompatibility with the principles of the calculus of probability in a short diary-notebook about my mathematical occupations. Gauss to W. Olbers, 24 January The 1798 diary entry reads in its entirety: # Calculus probabilitatis contra La Place defensus. Gott. Jun. 17. The # indicates Gauss attached some importance to the entry though less than to others where there is multiple vertical and horizontal scoring (Plackett, 1972, p240). Maximum Likelihood Estimation p.19/44 Maximum Likelihood Estimation p.20/44

6 How can Mr. Gauss have dared to tell you that the greater part of your theorems were known to him and that he discovered them as early as 1808?...But thisisthesame man who, in 1801, wished to attribute to himself the discovery of the law of reciprocity published in 1785 and who wanted to appropriate in 1809 the method of least squares published in Other examples will be found in other places, but a man of honour should refrain from imitating them. Legendre to Jacobi, 30 November innoeventwillIdiscussthispassage,wherethemethod [Gauss s use of least squares] was publicly indicated for the first time, also that I do not wish one of my friends to do it with my assent. This would amount to recognizing that my announcement that I had used this method many times since 1794 is in need of justification, and with that I shall never agree. When Obers attested, that I communicated the whole method to him in 1802, this was certainly well meant; but if he had asked me beforehand, I would have disapproved it strongly. Gauss to Schumacher, 3 December Maximum Likelihood Estimation p.21/44 Maximum Likelihood Estimation p.22/44 Legendre has the first and last word: it is his phrase that still lives: méthode des moindres quarrés Maximum Likelihood Estimation p.23/44 Maximum Likelihood Estimation p.24/44

7 Estimation: Method of Moments Plackett, R. L Studies in the History of Probability and Statistics. XXIX. The discovery of the method of least squares. Biometrika, 59: Seal, H. L Studies in the History of Probability and Statistics. XV. The historical devlopment of the Gauss linear model. Biometrika 54:1-24. Stigler, Stephen M Gauss and the Invention of Least Squares. The Annals of Statistics 9: Not as interesting a story Chebyshev ( ) first used the method of moments in a proof of the central limit theorem published in , and Markov ( ) extended and improved the work in Markov was Chebyshev s student. Maximum Likelihood Estimation p.25/44 Maximum Likelihood Estimation p.26/44 Estimation: Method of Moments Estimation: Method of Moments In the 20th century the method of moments was championed by Karl Pearson ( ), the dominant figure of British statistics, especially from 1911 until Less often used today, though the Generalized Method of Moments approach is an important recent development in econometrics. (See Greene, Section 4.7, pp for a full development of the approach.) Maximum Likelihood Estimation p.27/44 Moment estimators are generally consistent but R. A. Fisher showed that moment estimators can be very inefficient. This was one among many battles between Pearson and Fisher that continued until the very end of Pearson s life in The issue of Biometrika that announces Pearson s death also contains an article by him attacking Fisher s claim that maximum likelihood is more efficient. Maximum Likelihood Estimation p.28/44

8 Estimation: Method of Moments Estimation: Maximum Likelihood Professor Fisher...should state, now that the Method of Likelihood has taken hold of so many of the younger generation of mathematical statisticians, wherein he conceives ittogive better results... Hemustdefinewhathemeans by best, before he can prove that the principle of Likelihood provides it. Karl Pearson, 1936 (Biometrika 28:34-59). R. A. Fisher ( ) was the greatest statistician of the first half of the 20th century. He developed the method of maximum likelihood, pioneered experimental design, developed many of what are now standard statistical tests (the F-test for example) and Analysis of Variance. Maximum Likelihood Estimation p.29/44 Maximum Likelihood Estimation p.30/44 Estimation: Maximum Likelihood Estimation: Maximum Likelihood Fisher first published the method of maximum likelihood in 1912 (as a third year undergraduate) but unfortunately used the term inverse probability to describe it, a phrase implying a Bayesian approach (which likelihood is not.) Fisher clarified the usage in a 1915 paper and first coined the term likelihood in The full development of the method appears in articles of 1922 and About which much more later! Beyond statistics, Fisher was also one of the creators of modern genetics, a field which still considers his The Genetical Theory of Natural Selection (1930) a foundational work. Maximum Likelihood Estimation p.31/44 Maximum Likelihood Estimation p.32/44

9 Estimation: Maximum Likelihood The likelihood is proportional to the probability of observing the data, treating the parameters of the distribution as variables and the data as fixed (and assuming independent observations). L(θ Y) = kp(y θ) N p(y i θ) i=1 Estimation: Maximum Likelihood The maximum likelihood estimate is that value of the parameter θ for which the likelihood of the observed sample is a maximum. Alternatively, the ML estimate is the mode of the likelihood function. The ML estimator turns out to have several useful properties, as we shall see. Maximum Likelihood Estimation p.33/44 Maximum Likelihood Estimation p.34/44 Fisher Seminal Articles Fisher, R. A On an Absolute Criterion for Fitting Frequency Curves. Messenger of Mathematics 41: Fisher, R. A Frequency Distribution of the Values from the Correlation Coefficient in Samples from an Indefinitely Large Population. Biometrika 10: Fisher, R. A On the Probable Error of a Coefficient of Correlation Deduced from a Small Sample. Metron 1:3-32. Fisher, R. A On the Mathematical Foundations of Theoretical Statistics. Philosophical Transactions of the Royal Society of London, A 222: Fisher, R. A Theory of Statistical Estimation, Proceedings of the Cambridge Philosophical Society 22: ReviewsofFisher slifeandwork Savage, Leonard J On Rereading R. A. Fisher. Annals of Statistics 4: Bartlett, M. S R. A. Fisher and the Last Fifty Years of Statistical Methodology. Journal of the American Statistical Association 60: Aldrich, John R. A. Fisher and the Making of Maximum Likelihood Statistical Science, 12: Edwards, A. W. F What Did Fisher Mean by Inverse Probability in ? Statistical Science, 12: Box, Joan Fisher R. A. Fisher: The Life of a Scientist. New York: John Wiley and Sons. Many others. See the Extended Syllabus for more. Maximum Likelihood Estimation p.35/44 Maximum Likelihood Estimation p.36/44

10 Estimation: Bayes The fundamental problem of inference Given that y p(y, θ) how can we make inferences about the value of θ? This is the reverse of the probability problem: given θ what can we say about the distribution of y. Sometimes called inverse probability, we seek the distribution p(θ y), the distribution of the unknown parameter conditional on the observed data. Thomas Bayes ( ) published one article on statistics, but it was a good one. Published posthumously in 1763 An Essay towards Solving a Problem in the Doctrine of Chances provides the first derivation of what we now call a posterior distribution. Maximum Likelihood Estimation p.37/44 Maximum Likelihood Estimation p.38/44 The Bayesian Model Basic Bayes The basic result: A joint distribution is the product of a marginal and a conditional. and also p(θ, y) = p(θ)p(y θ) p(θ, y) = p(y)p(θ y) We seek this last term: p(θ y) so using these two basic identities: p(θ, y) p(θ y) = p(y) = p(θ)p(y θ) p(y) This fundamental result: is often written as p(θ y) = p(θ)p(y θ) p(y) p(θ y) p(θ)p(y θ) The posterior is proportional to the prior times the likelihood So you can t do Bayes without likelihood! Maximum Likelihood Estimation p.39/44 Maximum Likelihood Estimation p.40/44

11 Excuse me, is that p(θ)? How can we write p(θ)? Isn t θ just a fixed population parameter? Even if it isn t how do you know what p(θ) is? Different p(θ) give different inferences, so the same data can produce different conclusions. This is the point at which frequentists leave the room and Bayesians go to Spain for a conference. The Posterior The posterior is proportional to the prior times the likelihood. The stronger the data the more the likelihood dominates the prior in the posterior. The weaker the prior the more the likelihood dominates. Over repeated experiments, even different priors will eventually converge to a common posterior. Non-informative priors allow the likelihood to completely dominate the posterior (at the loss of any prior information!) Maximum Likelihood Estimation p.41/44 Maximum Likelihood Estimation p.42/44 Bayesian History The End of History Stigler, Stephen M Who Discovered Bayes s Theorem? The American Statistician, 37: Stigler, Stephen M Thomas Bayes s Bayesian Inference. Journal of the Royal Statistical Society. Series A, 145: Zabell, Sandy R. A. Fisher on the History of Inverse Probability. Statistical Science, 4: Efron, Bradly Why Isn t Everyone a Bayesian? (with commentary). The American Statistician, 40:1-11. Gill, Jeff Bayesian Methods: A Social and Behavioral Sciences Approach Boca Ratom: Chapman & Hall/CRC. Least Squares and Method of Moments fit data but avoid saying much (anything!) about how the data are generated. Likelihood and Bayes begin with the process that generates the data and base estimation on this process (and in Bayes, on prior uncertainty). In this Likelihood and Bayes are more demanding of our theory but are more rewarding in their ability to illuminate the theory based on the data. So how do we specify these models? Come back tomorrow and find out! Maximum Likelihood Estimation p.43/44 Maximum Likelihood Estimation p.44/44