Linear Regression Analysis

Transcription

1

2

3 Linear Regression Analysis

4 WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Peter Bloomfield, Noel A. C. Cressie, Nicholas I. Fisher, Iain M. Johnstone, J. B. Kadane, Louise M. Ryan, David W. Scott, Adrian F. M. Smith, JozefL. Teugels Editors Emeriti: Vic Barnett, J. Stuart Hunter, David G. Kendall A complete list of the titles in this series appears at the end of this volume.

5 Linear Regression Analysis Second Edition GEORGE A. F. SEBER ALAN J. LEE Department of Statistics University of Auckland Auckland, New Zealand iwiley- INTERSCIENCE A JOHN WILEY & SONS PUBLICATION

6 Copyright 2003 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) , fax (201) , permreq@wiley.com. Limit of Liability /Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representation or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the U.S. at , outside the U.S. at or fax Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic format. Library of Congress Cataloging-in-Publication Data Is Available ISBN

7 Contents Preface xv 1 Vectors of Random Variables Notation Statistical Models Linear Regression Models Expectation and Covariance Operators 5 Exercises la Mean and Variance of Quadratic Forms 9 Exercises lb Moment Generating Functions and Independence 13 Exercises lc 15 Miscellaneous Exercises Multivariate Normal Distribution Density Function 17 Exercises 2a Moment Generating Functions 20 Exercises 2b Statistical Independence 24

8 vi CONTENTS Exercises 2c Distribution of Quadratic Forms 27 Exercises 2d 31 Miscellaneous Exercises 2 31 Linear Regression: Estimation and Distribution Theory Least Squares Estimation 35 Exercises 3a Properties of Least Squares Estimates 42 Exercises 3b Unbiased Estimation of a 1 44 Exercises 3c Distribution Theory 47 Exercises 3d Maximum Likelihood Estimation Orthogonal Columns in the Regression Matrix 51 Exercises 3e Introducing Further Explanatory Variables General Theory One Extra Variable 57 Exercises 3f Estimation with Linear Restrictions Method of Lagrange Multipliers Method of Orthogonal Projections 61 Exercises 3g Design Matrix of Less Than Full Rank Least Squares Estimation 62 Exercises 3h Estimable Functions 64 Exercises 3i Introducing Further Explanatory Variables Introducing Linear Restrictions 65 Exercises 3j Generalized Least Squares 66 Exercises 3k Centering and Scaling the Explanatory Variables Centering Scaling 71

9 CONTENTS vii Exercises Bayesian Estimation 73 Exercises 3m Robust Regression M-Estimates Estimates Based on Robust Location and Scale Measures Measuring Robustness Other Robust Estimates 88 Exercises 3n 93 Miscellaneous Exercises 3 93 Hypothesis Testing Introduction Likelihood Ratio Test F-Test Motivation Derivation 99 Exercises 4a Some Examples The Straight Line 107 Exercises 4b Multiple Correlation Coefficient 110 Exercises 4c Canonical Form for H 113 Exercises 4d Goodness-of-Fit Test F-Test and Projection Matrices 116 Miscellaneous Exercises Confidence Intervals and Regions Simultaneous Interval Estimation Simultaneous Inferences Comparison of Methods Confidence Regions Hypothesis Testing and Confidence Intervals Confidence Bands for the Regression Surface Confidence Intervals Confidence Bands 129

10 viii CONTENTS 5.3 Prediction Intervals and Bands for the Response Prediction Intervals Simultaneous Prediction Bands Enlarging the Regression Matrix 135 Miscellaneous Exercises Straight-Line Regression The Straight Line Confidence Intervals for the Slope and Intercept Confidence Interval for the i-intercept Prediction Intervals and Bands Prediction Intervals for the Response Inverse Prediction (Calibration) 145 Exercises 6a Straight Line through the Origin Weighted Least Squares for the Straight Line Known Weights Unknown Weights 151 Exercises 6b Comparing Straight Lines General Model Use of Dummy Explanatory Variables 156 Exercises 6c Two-Phase Linear Regression Local Linear Regression 162 Miscellaneous Exercises Polynomial Regression Polynomials in One Variable Problem of Ill-Conditioning Using Orthogonal Polynomials Controlled Calibration Piecewise Polynomial Fitting Unsatisfactory Fit Spline Functions Smoothing Splines Polynomial Regression in Several Variables Response Surfaces 180

11 CONTENTS ix Multidimensional Smoothing 184 Miscellaneous Exercises Analysis of Variance Introduction One-Way Classification General Theory Confidence Intervals Underlying Assumptions 195 Exercises 8a Two-Way Classification (Unbalanced) Representation as a Regression Model Hypothesis Testing Procedures for Testing the Hypotheses Confidence Intervals 204 Exercises 8b Two-Way Classification (Balanced) 206 Exercises 8c Two-Way Classification (One Observation per Mean) Underlying Assumptions Higher-Way Classifications with Equal Numbers per Mean Definition of Interactions Hypothesis Testing Missing Observations 220 Exercises 8d Designs with Simple Block Structure Analysis of Covariance 222 Exercises 8e 224 Miscellaneous Exercises Departures from Underlying Assumptions Introduction Bias Bias Due to Underfitting Bias Due to Overfitting 230 Exercises 9a Incorrect Variance Matrix 231 Exercises 9b 232

12 x CONTENTS 9.4 Effect of Outliers Robustness of the F-Test to Nonnormality Effect of the Regressor Variables Quadratically Balanced F-Tests 236 Exercises 9c Effect of Random Explanatory Variables Random Explanatory Variables Measured without Error Fixed Explanatory Variables Measured with Error Round-off Errors Some Working Rules Random Explanatory Variables Measured with Error Controlled Variables Model Collinearity Effect on the Variances of the Estimated Coefficients Variance Inflation Factors Variances and Eigenvalues Perturbation Theory Collinearity and Prediction 261 Exercises 9d 261 Miscellaneous Exercises Departures from Assumptions: Diagnosis and Remedies Introduction Residuals and Hat Matrix Diagonals 266 Exercises 10a Dealing with Curvature Visualizing Regression Surfaces Transforming to Remove Curvature Adding and Deleting Variables 277 Exercises 10b Nonconstant Variance and Serial Correlation Detecting Nonconstant Variance Estimating Variance Functions Transforming to Equalize Variances Serial Correlation and the Durbin-Watson Test 292 Exercises 10c Departures from Normality Normal Plotting 295

13 CONTENTS xi Transforming the Response Transforming Both Sides 299 Exercises lod Detecting and Dealing with Outliers Types of Outliers Identifying High-Leverage Points Leave-One-Out Case Diagnostics Test for Outliers Other Methods 311 Exercises 10e Diagnosing Collinearity Drawbacks of Centering Detection of Points Influencing Collinearity Remedies for Collinearity 320 Exercises lof 326 Miscellaneous Exercises Computational Algorithms for Fitting a Regression Introduction Basic Methods Direct Solution of the Normal Equations Calculation of the Matrix X'X Solving the Normal Equations 331 Exercises 11a QR Decomposition Calculation of Regression Quantities Algorithms for the QR and WU Decompositions 341 Exercises lib Singular Value Decomposition Regression Calculations Using the SVD Computing the SVD Weighted Least Squares Adding and Deleting Cases and Variables Updating Formulas Connection with the Sweep Operator Adding and Deleting Cases and Variables Using QR Centering the Data Comparing Methods 365

14 xii CONTENTS Resources Efficiency Accuracy Two Examples Summary 373 Exercises lie Rank-Deficient Case Modifying the QR Decomposition Solving the Least Squares Problem Calculating Rank in the Presence of Round-off Error Using the Singular Value Decomposition Computing the Hat Matrix Diagonals Using the Cholesky Factorization Using the Thin QR Decomposition Calculating Test Statistics Robust Regression Calculations Algorithms for Li Regression Algorithms for M- and GM-Estimation Elemental Regressions Algorithms for High-Breakdown Methods 385 Exercises lid 388 Miscellaneous Exercises Prediction and Model Selection Introduction Why Select? 393 Exercises 12a Choosing the Best Subset Goodness-of-Fit Criteria Criteria Based on Prediction Error Estimating Distributional Discrepancies Approximating Posterior Probabilities 410 Exercises 12b Stepwise Methods Forward Selection Backward Elimination Stepwise Regression 418 Exercises 12c 420

15 CONTENTS xiii 12.5 Shrinkage Methods Stein Shrinkage Ridge Regression Garrote and Lasso Estimates 425 Exercises 12d Bayesian Methods Predictive Densities Bayesian Prediction Bayesian Model Averaging 433 Exercises 12e Effect of Model Selection on Inference Conditional and Unconditional Distributions Bias Conditional Means and Variances Estimating Coefficients Using Conditional Likelihood Other Effects of Model Selection 438 Exercises 12f Computational Considerations Methods for All Possible Subsets Generating the Best Regressions All Possible Regressions Using QR Decompositions 446 Exercises 12g Comparison of Methods Identifying the Correct Subset Using Prediction Error as a Criterion 448 Exercises 12h 456 Miscellaneous Exercises Appendix A Some Matrix Algebra 457 A.l Trace and Eigenvalues 457 A.2 Rank 458 A.3 Positive-Semidefinite Matrices 460 A.4 Positive-Definite Matrices 461 A.5 Permutation Matrices 464 A.6 Idempotent Matrices 464 A.7 Eigenvalue Applications 465 A.8 Vector Differentiation 466 A.9 Patterned Matrices 466