Applied Linear Regression

Transcription

1 Applied Linear Regression

2

3 Applied Linear Regression Third Edition SANFORD WEISBERG University of Minnesota School of Statistics Minneapolis, Minnesota A JOHN WILEY & SONS, INC., PUBLICATION

4 Copyright 2005 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, , fax , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) , fax (201) Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the U.S. at , outside the U.S. at or fax Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic format. Library of Congress Cataloging-in-Publication Data: Weisberg, Sanford, 1947 Applied linear regression / Sanford Weisberg. 3rd ed. p. cm. (Wiley series in probability and statistics) Includes bibliographical references and index. ISBN (acid-free paper) 1. Regression analysis. I. Title. II. Series. QA278.2.W dc Printed in the United States of America

5 To Carol, Stephanie and to the memory of my parents

6 Contents Preface xiii 1 Scatterplots and Regression Scatterplots, Mean Functions, Variance Functions, Summary Graph, Tools for Looking at Scatterplots, Size, Transformations, Smoothers for the Mean Function, Scatterplot Matrices, 15 Problems, 17 2 Simple Linear Regression Ordinary Least Squares Estimation, Least Squares Criterion, Estimating σ 2, Properties of Least Squares Estimates, Estimated Variances, Comparing Models: The Analysis of Variance, The F -Test for Regression, Interpreting p-values, Power of Tests, The Coefficient of Determination, R 2, Confidence Intervals and Tests, The Intercept, Slope, 33 vii

7 viii CONTENTS Prediction, Fitted Values, The Residuals, 36 Problems, 38 3 Multiple Regression Adding a Term to a Simple Linear Regression Model, Explaining Variability, Added-Variable Plots, The Multiple Linear Regression Model, Terms and Predictors, Ordinary Least Squares, Data and Matrix Notation, Variance-Covariance Matrix of e, Ordinary Least Squares Estimators, Properties of the Estimates, Simple Regression in Matrix Terms, The Analysis of Variance, The Coefficient of Determination, Hypotheses Concerning One of the Terms, Relationship to the t-statistic, t-tests and Added-Variable Plots, Other Tests of Hypotheses, Sequential Analysis of Variance Tables, Predictions and Fitted Values, 65 Problems, 65 4 Drawing Conclusions Understanding Parameter Estimates, Rate of Change, Signs of Estimates, Interpretation Depends on Other Terms in the Mean Function, Rank Deficient and Over-Parameterized Mean Functions, Tests, Dropping Terms, Logarithms, Experimentation Versus Observation, 77

8 CONTENTS ix 4.3 Sampling from a Normal Population, More on R 2, Simple Linear Regression and R 2, Multiple Linear Regression, Regression through the Origin, Missing Data, Missing at Random, Alternatives, Computationally Intensive Methods, Regression Inference without Normality, Nonlinear Functions of Parameters, Predictors Measured with Error, 90 Problems, 92 5 Weights, Lack of Fit, and More Weighted Least Squares, Applications of Weighted Least Squares, Additional Comments, Testing for Lack of Fit, Variance Known, Testing for Lack of Fit, Variance Unknown, General F Testing, Non-null Distributions, Additional Comments, Joint Confidence Regions, 108 Problems, Polynomials and Factors Polynomial Regression, Polynomials with Several Predictors, Using the Delta Method to Estimate a Minimum or a Maximum, Fractional Polynomials, Factors, No Other Predictors, Adding a Predictor: Comparing Regression Lines, Additional Comments, Many Factors, Partial One-Dimensional Mean Functions, Random Coefficient Models, 134 Problems, 137

9 x CONTENTS 7 Transformations Transformations and Scatterplots, Power Transformations, Transforming Only the Predictor Variable, Transforming the Response Only, The Box and Cox Method, Transformations and Scatterplot Matrices, The 1D Estimation Result and Linearly Related Predictors, Automatic Choice of Transformation of Predictors, Transforming the Response, Transformations of Nonpositive Variables, 160 Problems, Regression Diagnostics: Residuals The Residuals, Difference Between ê and e, The Hat Matrix, Residuals and the Hat Matrix with Weights, The Residuals When the Model Is Correct, The Residuals When the Model Is Not Correct, Fuel Consumption Data, Testing for Curvature, Nonconstant Variance, Variance Stabilizing Transformations, A Diagnostic for Nonconstant Variance, Additional Comments, Graphs for Model Assessment, Checking Mean Functions, Checking Variance Functions, 189 Problems, Outliers and Influence Outliers, An Outlier Test, Weighted Least Squares, Significance Levels for the Outlier Test, Additional Comments, Influence of Cases, Cook s Distance, 198

10 CONTENTS xi Magnitude of D i, Computing D i, Other Measures of Influence, Normality Assumption, 204 Problems, Variable Selection The Active Terms, Collinearity, Collinearity and Variances, Variable Selection, Information Criteria, Computationally Intensive Criteria, Using Subject-Matter Knowledge, Computational Methods, Subset Selection Overstates Significance, Windmills, Six Mean Functions, A Computationally Intensive Approach, 228 Problems, Nonlinear Regression Estimation for Nonlinear Mean Functions, Inference Assuming Large Samples, Bootstrap Inference, References, 248 Problems, Logistic Regression Binomial Regression, Mean Functions for Binomial Regression, Fitting Logistic Regression, One-Predictor Example, Many Terms, Deviance, Goodness-of-Fit Tests, Binomial Random Variables, Maximum Likelihood Estimation, The Log-Likelihood for Logistic Regression, 264

11 xii CONTENTS 12.4 Generalized Linear Models, 265 Problems, 266 Appendix 270 A.1 Web Site, 270 A.2 Means and Variances of Random Variables, 270 A.2.1 E Notation, 270 A.2.2 Var Notation, 271 A.2.3 Cov Notation, 271 A.2.4 Conditional Moments, 272 A.3 Least Squares for Simple Regression, 273 A.4 Means and Variances of Least Squares Estimates, 273 A.5 Estimating E(Y X) Using a Smoother, 275 A.6 A Brief Introduction to Matrices and Vectors, 278 A.6.1 Addition and Subtraction, 279 A.6.2 Multiplication by a Scalar, 280 A.6.3 Matrix Multiplication, 280 A.6.4 Transpose of a Matrix, 281 A.6.5 Inverse of a Matrix, 281 A.6.6 Orthogonality, 282 A.6.7 Linear Dependence and Rank of a Matrix, 283 A.7 Random Vectors, 283 A.8 Least Squares Using Matrices, 284 A.8.1 Properties of Estimates, 285 A.8.2 The Residual Sum of Squares, 285 A.8.3 Estimate of Variance, 286 A.9 The QR Factorization, 286 A.10 Maximum Likelihood Estimates, 287 A.11 The Box-Cox Method for Transformations, 289 A.11.1 Univariate Case, 289 A.11.2 Multivariate Case, 290 A.12 Case Deletion in Linear Regression, 291 References 293 Author Index 301 Subject Index 305

12 Preface Regression analysis answers questions about the dependence of a response variable on one or more predictors, including prediction of future values of a response, discovering which predictors are important, and estimating the impact of changing a predictor or a treatment on the value of the response. At the publication of the second edition of this book about 20 years ago, regression analysis using least squares was essentially the only methodology available to analysts interested in questions like these. Cheap, widely available high-speed computing has changed the rules for examining these questions. Modern competitors include nonparametric regression, neural networks, support vector machines, andtree-based methods, among others. A new field of computer science, called machine learning, adds diversity, and confusion, to the mix. With the availability of software, using a neural network or any of these other methods seems to be just as easy as using linear regression. So, a reasonable question to ask is: Who needs a revised book on linear regression using ordinary least squares when all these other newer and, presumably, better methods exist? This question has several answers. First, most other modern regression modeling methods are really just elaborations or modifications of linear regression modeling. To understand, as opposed to use, neural networks or the support vector machine is nearly impossible without a good understanding of linear regression methodology. Second, linear regression methodology is relatively transparent, as will be seen throughout this book. We can draw graphs that will generally allow us to see relationships between variables and decide whether the models we are using make any sense. Many of the more modern methods are much like a black box in which data are stuffed in at one end and answers pop out at the other, without much hope for the nonexpert to understand what is going on inside the box. Third, if you know how to do something in linear regression, the same methodology with only minor adjustments will usually carry over to other regression-type problems for which least squares is not appropriate. For example, the methodology for comparing response curves for different values of a treatment variable when the response is continuous is studied in Chapter 6 of this book. Analogous methodology can be used when the response is a possibly censored survival time, even though the method of fitting needs to be appropriate for the censored response and not least squares. The methodology of Chapter 6 is useful both in its xiii

13 xiv PREFACE own right when applied to linear regression problems and as a set of core ideas that can be applied in other settings. Probably the most important reason to learn about linear regression and least squares estimation is that even with all the new alternatives most analyses of data continue to be based on this older paradigm. And why is this? The primary reason is that it works: least squares regression provides good, and useful, answers to many problems. Pick up the journals in any area where data are commonly used for prediction or estimation and the dominant method used will be linear regression with least squares estimation. What s New in this Edition Many of the examples and homework data sets from the second edition have been kept, although some have been updated. The fuel consumption data, for example, now uses 2001 values rather than 1974 values. Most of the derivations are the same as in the second edition, although the order of presentation is somewhat different. To keep the length of the book nearly unchanged, methods that failed to gain general usage have been deleted, as have the separate chapters on prediction and missing data. These latter two topics have been integrated into the remaining text. The continuing theme of the second edition was the need for diagnostic methods, in which fitted models are analyzed for deficiencies, through analysis of residuals and influence. This emphasis was unusual when the second edition was published and important quantities like Studentized residuals and Cook s distance were not readily available in the commercial software of the time. Times have changed, and so has the emphasis of this book. This edition stresses graphical methods including looking at data both before and after fitting models. This is reflected immediately in the new Chapter 1, which introduces the key idea of looking at data with scatterplots and the somewhat less universal tool of scatterplot matrices. Most analyses and homework problems start with drawing graphs. We tailor analyses to correspond to what we see in the graphs, and this additional step can make modeling easier and fitted models reflect the data more closely. Remarkably, this also lessens the need for diagnostic methods. The emphasis on graphs leads to several additional methods and procedures that were not included in the second edition. The use of smoothers to help summarize a scatterplot is introduced early, although only a little of the theory of smoothing is presented (in Appendix A.5). Transformations of predictors and the response are stressed, and relatively unfamiliar methods based both on smoothing and on generalization of the Box Cox method are presented in Chapter 7. Another new topic included in the book is computationally intensive methods and simulation. The key example of this is the bootstrap, in Section 4.6, which can be used to make inferences about fitted models in small samples. A somewhat different computationally intensive method is used in an example in Chapter 10, which is a completely rewritten chapter on variable selection. The book concludes with two expanded chapters on nonlinear and logistic regression, both of which are generalizations of the linear regression model. I have

14 PREFACE xv included these chapters to provide instructors and students with enough information for basic usage of these models and to take advantage of the intuition gained about them from an in-depth study of the linear regression model. Each of these can be treated at book-length, and appropriate references are given. Mathematical Level The mathematical level of this book is roughly the same as the level of the second edition. Matrix representation of data is used, particularly in the derivation of the methodology in Chapters 2 4. Derivations are less frequent in later chapters, and so the necessary mathematics is less. Calculus is generally not required, except for an occasional use of a derivative, for the discussion of the delta method, Section 6.1.2, and for a few topics in the Appendix. The discussions requiring calculus can be skipped without much loss. Computing and Computer Packages Like the second edition, only passing mention is made in the book to computer packages. To help the reader make a connection between the text and a computer package for doing the computations, we provide several web companions for Applied Linear Regression that discuss how to use standard statistical packages for linear regression analysis. The packages covered include JMP, SAS, SPSS, R, and S-plus; others may be included after publication of the book. In addition, all the data files discussed in the book are also on the website. The web address for this material is Some readers may prefer to have a book that integrates the text more closely with a computer package, and for this purpose, I can recommend R. D. Cook and S. Weisberg (1999), Applied Regression Including Computing and Graphics, also published by John Wiley. This book includes a very user-friendly, free computer package called Arc that does everything that is described in that book and also nearly everything in Applied Linear Regression. Teaching with this Book The first ten chapters of the book should provide adequate material for a one-quarter course on linear regression. For a semester-length course, the last two chapters can be added. A teacher s manual, primarily giving solutions to all the homework problems, can be obtained from the publisher by instructors. Acknowledgments I am grateful to several people who generously shared their data for inclusion in this book; they are cited where their data appears. Charles Anderson and Don Pereira suggested several of the examples. Keija Shan, Katherine St. Clair, and Gary Oehlert helped with the website and its content. Brian Sell helped with the

15 xvi PREFACE examples and with many administrative chores. Several others helped with earlier editions: Christopher Bingham, Morton Brown, Cathy Campbell, Dennis Cook, Stephen Fienberg, James Frane, Seymour Geisser, John Hartigan, David Hinkley, Alan Izenman, Soren Johansen, Kenneth Koehler, David Lane, Michael Lavine, Kinley Larntz, John Rice, Donald Rubin, Joe Shih, Pete Stewart, Stephen Stigler, Douglas Tiffany, Carol Weisberg, and Howard Weisberg. Sanford Weisberg St. Paul, Minnesota April 13, 2004