Introductory Statistical Inference with the Likelihood Function

Transcription

1 Introductory Statistical Inference with the Likelihood Function

2

3 Charles A. Rohde Introductory Statistical Inference with the Likelihood Function 123

4 Charles A. Rohde Bloomberg School of Health Johns Hopkins University Baltimore, MD, USA ISBN ISBN (ebook) DOI / Springer Cham Heidelberg New York Dordrecht London Library of Congress Control Number: Springer International Publishing Switzerland 2014 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (

5 To the memory of Henry Laurence (Curly) Lucas Advisor and Mentor and my family Savilla and RD David Brooks Kitchel II Denise Kitchel David Brooks Kitchel III Parker Kitchel

6

7 Preface This book is a result of many years of teaching introductory statistical theory at the Johns Hopkins University Department of Biostatistics. It is designed for advanced undergraduates or master s level students. The approach used is to introduce students to statistical theory without the use of too many advanced mathematical concepts which often inhibit the understanding of the basic philosophical foundations of statistics. In particular, attention is paid to the continuing debate on the foundations of statistics and the reasons why statistics works using any one of the major philosophical approaches to the subject. Most standard texts pay little or no attention to the contrasts between schools of statistical thought and how they related to each other. In particular I emphasize the Law of Likelihood as a way to connect various approaches. Students must be made aware of the fact that there are no agreed upon methods for solving all of the problems to which modern statistics is asked to find solutions. I suspect that many of the current ad hoc procedures currently used will continue to be used for decades to come. A student can, however, be equipped at a modest level of mathematics with tools necessary to understand the myriad of statistical methods now available. The explosion of statistical packages in the last twenty years makes it possible for almost any one to perform analyses deemed intractable just a few years ago. This book is about some of the basic principles of statistics needed to criticize and understand the methods for analyzing complex data sets. I have included a short chapter on finite population sampling since I believe that every statistician should have some knowledge of the subject and since it forms the basis for much of what we know about contemporary society. It also clearly illustrates the need for some understanding of the foundations of statistics. I also included a section in the appendix on interpretations of probability, a subject which is often omitted in statistics texts. The remainder of the appendix consists of material on probability and some mathematical concepts which I find convenient to have in one place for reference. A word on regularity conditions is in order. It often happens that a result is true provided some conditions are assumed. These are called regularity conditions. vii

8 viii Preface In this book I am cavalier about these since being precise provides an additional layer of mathematics and often obscures the statistical concepts. In many cases it is easier to prove results from scratch rather than verify a list of regularity conditions. I have tried to cite references for the many examples in the text. Unfortunately, as is often the case when lecture notes are developed into a book, some sources have been forgotten. I apologize in advance for these omissions and will correct them on the website as I become aware of them. The text is deliberately thin on exercises. Many more (with solutions) will appear on the website. I thank the many students who have suffered through versions of these notes. I thank Richard Royall for many discussions and advice on statistics in general and likelihood in particular. Finally, I thank my wife, Savilla, for putting up with me. Baltimore, Maryland Charles A. Rohde July 2014

9 Contents 1 Introduction Introductory Example Likelihood Approach Bayesian Approach Frequentist Approach Comments on the Example Choice of k and α What Is Statistics? General Setup Scope of Statistical Inference The Statistical Approach The Setup Approaches to Statistical Inference Types of Statistical Inference Statistics and Combinants Statistics and Sampling Distributions Combinants Frequentist Inference Bayesian Inference Likelihood Inference Exercises Estimation Frequentist Concepts Bias, Unbiasedness, and Standard Errors Consistency Mean Square Error Asymptotic Distributions Efficiency Equivariance ix

10 x Contents 3.2 Bayesian and Likelihood Paradigms Exercises Interval Estimation The Problem Frequentist Approach Importance of the Long Run Application to the Normal Distribution Pivots Likelihood Intervals Bayesian Approach Objective Bayes Comparing the Intervals Interval Estimation Example Exercises Hypothesis Testing Law of Likelihood Neyman-Pearson Theory Introduction Neyman-Pearson Lemma Using the Neyman-Pearson Lemma Uniformly Most Powerful Tests A Complication Comments and Examples A Different Criterion Inductive Behavior Still Another Criterion p-values Duality of Confidence Intervals and Tests Composite Hypotheses One-Way Analysis of Variance The Multiple Testing Problem Exercises Standard Practice of Statistics Introduction Frequentist Statistical Procedures Estimation Confidence Intervals Hypothesis Testing Significance Tests Maximum Likelihood: Basic Results Basic Properties Consistency of Maximum Likelihood General Results on the Score Function... 71

11 Contents xi 7.4 General Maximum Likelihood Cramer-Rao Inequality Summary Properties of Maximum Likelihood Multiparameter Case Maximum Likelihood in the Multivariate Normal Multinomial Linear Models Introduction Basic Results The Fitted Values and the Residuals The Basic Regression Model Adding Covariates Interpretation of Regression Coefficients Added Sum of Squares Identity of Regression Coefficients Likelihood and Bayesian Results Interpretation of the Coefficients Factors as Covariates Exercises Other Estimation Methods Estimation Using Empirical Distributions Empirical Distribution Functions Statistical Functionals Linear Statistical Functionals Quantiles Confidence Intervals for Quantiles Method of Moments Technical Details of the Method of Moments Application to the Normal Distribution Estimating Functions General Linear Model Maximum Likelihood Method of Moments Generalized Linear Models Quasi-Likelihood Generalized Estimating Equations Generalized Method of Moments The Bootstrap Basic Ideas Simulation Background Variance Estimation Using the Bootstrap Confidence Intervals Using the Bootstrap Normal Interval Pivotal Interval

12 xii Contents Percentile Interval Parametric Version Dangers of the Bootstrap The Number of Possible Bootstrap Samples Decision Theory Introduction Actions, Losses, and Risks Admissibility Bayes Risk and Bayes Rules Examples of Bayes Rules Stein s Result Exercises Sufficiency Families of Distributions Introduction to Sufficiency Rationale for Sufficiency Factorization Criterion Sketch of Proof of the Factorization Criterion Properties of Sufficient Statistics Minimal Sufficient Statistics Importance of Sufficient Statistics in Inference Frequentist Statistics Bayesian Inference Likelihood Inference Alternative Proof of Factorization Theorem Exercises Conditionality Ancillarity Problems with Conditioning Statistical Principles Introduction Birnbaum s Formulation Framework and Notation Mathematical Equivalence Irrelevant Noise Likelihood Principle Equivalence of Likelihood and Irrelevant Noise Plus Mathematical Equivalence Sufficiency, Conditionality, and Likelihood Principles Fundamental Result Stopping Rules Comments Jeffreys/Lindley Paradox

13 Contents xiii Randomization Permutation or Randomization Tests Bayesian Inference Frequentist vs Bayesian The Bayesian Model for Inference Why Bayesian? Exchangeability Stable Estimation Bayesian Consistency Relation to Maximum Likelihood Priors Different Types of Priors Vague Priors Bayesian Statistics: Computation Computation Monte Carlo Integration Importance Sampling Markov Chain Monte Carlo The Gibbs Sampler Software Bayesian Inference: Miscellaneous Bayesian Updating Bayesian Prediction Stopping Rules in Bayesian Inference Nuisance Parameters Summing Up Pure Likelihood Methods Introduction Misleading Statistical Evidence Weak Statistical Evidence Sample Size Birnbaum s Confidence Concept Combining Evidence Exercises Pure Likelihood Methods and Nuisance Parameters Nuisance Parameters Introduction Neyman-Scott Problem Elimination Methods Evidence in the Presence of Nuisance Parameters Orthogonal Parameters Orthogonal Reparametrizations Varieties of Likelihood

14 xiv Contents 18.5 Information Loss Marginal Likelihood Conditional Likelihood Estimated Likelihoods Profile Likelihood Introduction Misleading Evidence Using Profile Likelihoods General Linear Model Likelihood Functions Using Profile Likelihoods Profile Likelihoods for Unknown Variance Computation of Profile Likelihoods Summary Other Inference Methods and Concepts Fiducial Probability and Inference Good s Example Edward s Example Confidence Distributions Bootstrap Connections Likelihood Connections Confidence Curves P-Values Again Sampling Distribution of P-Values Severe Testing Cornfield on Testing and Confidence Intervals Finite Population Sampling Introduction Populations and Samples Principal Types of Sampling Methods Simple Random Sampling Horvitz Thompson Estimator Basu s Elephant An Unmentioned Assumption Prediction Approach Proof of the Prediction Result Stratified Sampling Basic Results Cluster Sampling Practical Considerations Sampling Frame Problems Nonresponse Sampling Errors Non-sampling Errors Role of Finite Population Sampling in Modern Statistics

15 Contents xv 21 Appendix: Probability and Mathematical Concepts Probability Models Definitions Properties of Probability Continuity Properties of Probability Measures Conditional Probability Properties of Conditional Probability Finite and Denumerable Sample Spaces Independence Random Variables and Probability Distributions Measurable Functions Random Variables:Definitions Distribution Functions Discrete Random Variables Continuous Random Variables Functions of Random Variables Random Vectors Definitions Discrete and Continuous Random Vectors Marginal Distributions The Multinomial Distribution Independence of Random Variables Conditional Distributions Functions of a Random Vector The Multivariate Normal Distribution Expected Values Expected Value of a Random Variable Distributions and Expected Values Moments Properties and Results on Expected Values and Moments Other Functionals of a Distribution Probability Generating Functions Moment-Generating Functions Cumulant Generating Functions and Cumulants Expected Values of Functions of Random Vectors Conditional Expectations and Variances What Is Probability? Frequency Interpretation Belief Interpretations Rational Belief Interpretation Countable Additivity Assumption Lindley s Wheel Probability via Expectations Final Comments

16 xvi Contents 21.7 Introduction to Stochastic Processes Introduction Types of Stochastic Processes Markov Processes Markov Chains Convergence of Sequences of Random Variables Introduction Basic Techniques Convergence in Probability Convergence in Distribution Extension to Vectors Results on Convergence in Probability and in Distribution The Delta Method Sets Definitions Properties of Set Operations Indicator Functions Counting and Combinatorics σ-algebras and Borel Sets σ-algebras Borel σ-algebras Properties and Examples of Borel Sets Measurable Functions Borel Sets in n Dimensions Background: Limits, o, and O Notation Notation Limits of Sequences liminf and limsup Sequences of Events o, O Definitions and Results Results and Special Cases Extension to Functions Extension to Vectors Extension to Vector-Valued Functions References Index