Performance of Median and Least Squares Regression for Slightly Skewed Data

Transcription

1 World Academy of Science, Engineering and Technology 9 Performance of Median and Least Squares Regression for Slightly Skewed Data Carolina Bancayrin - Baguio Abstract This paper presents the concept of quantile regression which is a nonparametric procedure used for prediction. This method is more robust compared to the Ordinary Least Squares (OLS) for asymmetric data. Specifically, in this study the procedure for Median Regression was emphasized and the attempt to compare its efficiency to that of Least Squares Method was done for slightly skewed data. The study utilized a subset of the data on the study entitled The Profile of Mindanao State University-Iligan Institute of Technology Students and Its Effects to their Academic Performance with total sample size of 677 students coming from different colleges in the Institute for the year -. The grade point average (gpa) is the dependent variable and their academic load, entrance exam scores ((SASE), age and number of hours for studying are the independent variables. It was found out that values of the standard errors of the estimates of the regression coefficients are relatively smaller for some predictors using the median regression compared to those coefficients obtained using the least squares method. In order to validate the performance of the median regression in terms of asymptotic efficiency, a bootstrapped median regression with, replicates was employed to compute the estimates of the regression coefficients. It was shown that the standard errors of the estimates of the median and bootstrapped regression are comparable while the values of the pseudo R are the same. This result confirms the asymptotic efficiency of the median regression for relative large sample sizes of the pseudo replicates for the data generated by bootstrapping. Keywords Quantile regression, robust, asymmetric distribution, nonparametric procedure, ordinary least squares, least trimmed squares. I. INTRODUCTION T has been observed that most of the studies in various I fields deal with the investigation of causal relationships of response on some realistic phenomena and predictor variables. Most often, linear regression analysis using least squares is used by researchers without the knowledge of a robust regression. This method is being driven with several assumptions like normally distributed errors, independence of observations, linearity, and absence of outliers. It was proven by so many studies that ordinary least squares is not robust compared to other methods such as the Median Regression, Least Trimmed Squares, and the Least Median Squares (Yaffee, R.A., ). The Least Absolute Deviation (LAD) was proven to be better than the Ordinary Least Squares (OLS) as revealed in Carolina Bancayrin - Baguio, PhD, is with MSU-Iligan Institute of Technology, Iligan City, 9, Philippines ( cbbaguio@yahoo.com). many studies which resulted into the Median Regression for the th quantile or in general for any quantile, the so called Quantile Regression. This method is classified under the nonparametric statistics since it is not assumed that data follow certain kind of distribution. Quantile Regression was never popular several decades ago and is not being taught in the basic Statistics course due to its inherent difficulty in the computation. Since the LAD function is not differentiable unlike the OLS, the Linear programming method was employed to solve the optimization problem instead of using calculus. The computation of which is greatly facilitated nowadays since the advent of better solution methods and advances in computing. The curse of multidimensionality or parsimony which arise if there are so many variables and data points was already addressed by some advanced algorithms other than the simplex method of Linear programming, the so called Interior Point Algorithm of Linear Programming. II. OBJECTIVES OF THE STUDY The main objective of this study is to investigate the potential of median regression for slightly skewed data. Specifically, it attempted to: ) To discuss the procedure for computing the estimates of Median Regression ) To compare Median Regression with Ordinary Least Squares (OLS) through the magnitudes of the standard errors of the regression coefficient estimates III. THEORY AND CONCEPTS A. Median Regression Median Regression estimates the median of the dependent variable, conditional on the values of the independent variable whereas the Least-Squares Regression estimates the mean of the dependent variable. The basic difference of the two methods is that the median regression finds the regression plane that minimizes the sum of the absolute residuals rather than the sum of the squared residuals for the Ordinary Least Squares. B. Bootstrapping Bootstrapping is a way of testing the reliability of the data set. It is the creation of pseudoreplicate datasets by resampling and a statistical method for obtaining an estimate of standard error. Moreover, bootstrapping allows one to assess whether the distribution of characters has been influenced by stochastic 6

2 World Academy of Science, Engineering and Technology 9 effects. C. Quantile Function Let Y be any real valued random variable characterized by its distribution function as, The τ th quantile of Y for any < τ <, on the other hand, can be written as: The quantile function in () provides a complete characterization of the random variable just like the distribution function. Using this function the median can be written as Q(/). The quantiles defined above can be formulated as the solution to a simple optimization problem. For any < τ <, define the piecewise linear function, ρ τ (v) = v(τ-i(v<)). Minimizing the expectation of ρ τ (Y θ) with respect to θ, yields solutions in which the smallest is the Q(τ) defined in (). The sample analog q(τ) of Q(τ), based on a random sample, (y,y,...,y n ) of Y s, is called the τ th sample quantile, which can be found by solving () Equation in () yields a natural generalization to the regression context. The linear conditional quantile function can be estimated by solving, for X ε R K and β ε R K : D. Computation of Quantile Regression Estimates The package Quantreg procedure using the R computer programming language solves Linear programs using the simplex algorithm of Barrodale and Roberts (97).The algorithm solves the linear program by two stages. The first stage picks the X or X as pivotal columns. The second stage interchanges the columns in I and I as basis or nonbasis columns. The optimal solution can be obtained by executing the two stages interactively. Only the main data matrix is stored in the current memory because of the special structure of the matrix A. This special version of the Simplex Algorithm for Median Regression can be extended to quantile regression for any given quantile, even for the entire quantile process (Koenker and d Orey 99). It was found out that this procedure reduces greatly the computing time required by a general simplex algorithm, and suitable for data sets with less than, observations and variables. The software called STATA version 9. can also solve the estimates of the coefficients of Median Regression as well as the Bootstrapped Median Regression. () () E. Hypothesis Testing Using the Dual Quantile estimation can be represented as a linear program. The confidence intervals of the coefficient estimates can be constructed by the solution of dual problem in quantile regression. It involves the rank statistics due to the dual problem of quantile regression. The advantage of rank test is that the nuisance parameter estimation can be avoided and the result is robust.the testing of the hypothesis for linearity of the quantiles in quantile regression can be done by the Rank inverse test. Gutenbrunner and Jureckova (99) showed that the solutions of the dual problem which is formulated for computing regression quantiles generalize the duality of ranks and quantiles to linear models. The dual solution called regression rank score process establishes the link between the linear rank statistics and regression quantiles. The procedure emanates from the classical theory of rank tests which can be extended to the test of the hypothesis IV. METHODS A subset of the data on the study entitled The Profile of Mindanao State University-Iligan Institute of Technology Students and its Effect to their Academic Performance with total sample size of 677 students coming from different colleges in the Institute for the year - was utilized in the study in order to illustrate the performance of Median Regression. The grade point average (gpa) is the dependent variable and their academic load, entrance exam scores (SASE), age and number of hours for studying are the independent variable. The software called STATA version9. licensed to Mindanao State University - Iligan Institute of Technology, Iligan City Philippines was used for the computation. V. EMPIRICAL RESULTS TABLE I DESCRIPTIVE STATISTICS OF THE DEPENDENT AND INDEPENDENT VARIABLES Variables Skewness Mean s.d. / cv Grade Point.86.. /.87% Average Entrance Exam /.8% Academic Load / 6.% Age / 9.7% Study hours /.% s.e. = standard error cv = coefficient of variation It is shown in Table I that the skewness values of the variables are relatively small and different from zero which implies that the data are slightly skewed. The percentage of variation is more than % for most of the variables except the ages of the students. This means that these variables are quite dispersed from the mean except for the ages of the 7

3 World Academy of Science, Engineering and Technology 9 students. Figs. to give the histograms and the Q-Q plots of the different variables. These charts confirm that the data departs from normality and are slightly skewed academic load.. Mean = Std. Dev. = 6.76 Fig. Histogram of the Students Academic Load Mean =. Std. Dev. = gpa Fig. Histogram for the Students Grade point Average(gpa) Normal Q-Q Plot of gpa Normal Q-Q Plot of academic load Fig. 6 Q-Q Plot of Students Academic Load Fig. QQ Plot of the Students Grade Point Average (gpa) Histogram for Students' age Mean = 9.6 Std. Dev. =.79 AGE Mean = 98.6 Std. Dev. = admission test result (raw score) Fig. Histogram of the Students Admission Test (SASE) Normal Q-Q Plot of admission test result (raw score) Fig. 7 Histogram of the Students Age Normal Q-Q Plot of age Fig. Q-Q Plot of the Students Admission Test (SASE) Fig. 8 Q-Q Plot of Students Age 8

4 World Academy of Science, Engineering and Technology 9 Mean =.88 Std. Dev. = time spent for studying per day Fig. 9 Histogram for Students Study Hours Normal Q-Q Plot of time spent for studying per day TABLE III COMPARATIVE PERFORMANCE OF MEDIAN AND BOOTSTRAPPED MEDIAN LEAST SQUARES RESPONSE MEDIAN VARIABLE coeff. s.e. coeff. s.e. (gpa) Predictors Admission Test -.7**. -.7**. -6. < <. Academic Load -.8**. -.7** Age.6**..6**...<...<. Study Hours -.6**. -.6** Constant.9**.8.9**. No. of Students N=677 Coeff.=coefficient s.e. = standard error ** highly significant at. level 6.8.<. 6.9.<. Pseudo R =.6 R =.6 Fig. Q-Q Plot of Students time spent for study per Day TABLE II COMPARATIVE PERFORMANCE OF MEDIAN AND LEAST SQUARES LEAST SQUARES RESPONSE MEDIAN VARIABLE coeff. s.e. coeff. s.e. (gpa) Predictors Admission Test -.7**. -.6**. -6. <. -..<. Academic Load -.8**. -.7** Age.6**..6**...<...<. Study Hours -.6**. -.6** Constant.9**.8.97**.98 No. of Students N=677 Coeff.=coefficient s.e. = standard error ** highly significant at. level 6.8.<. 6.6.<. Pseudo R =.6 R =.6 In order to compare the performance of the Median and Least Squares Regression for slightly skewed regression data that is normality assumption is waived for all the variables, Table II is constructed. It can be observed from the results of both methods that the significant linear predictors for the students grade point average are the admission test scores, academic load, age, and time spent for studying in hours. It is apparent from this result that the standard errors in the regression coefficient for both methods are the same for some variables like admission test score, academic load, study hours but there is a slight difference on the standard errors of the coefficients for the age and constant. However, the value of the R for the Least Squares method is larger than the pseudo R value of the Median Regression. But the violation of the assumption of normality on the values of the data can be waived for the median regression but not for the least squares regression. Hence, median regression is the appropriate procedure and leads to accurate statistical results. Table III gives the comparison of the regression estimates of the Median and Bootstrapped Median Regression with, pseudo replicates of the data. It can be observed that the standard errors of the estimates are just slightly different and so with the values of the significance levels. The values of the pseudo R for the goodness of fit for the two methods are equal. These findings reveal that the Median Regression gives asymptotic efficient estimates for the coefficients of the regression model. Hence, it can be said that Median Regression is the robust and asymptotically efficient procedure to address the problem of lightly skewed data which violates the assumption of normality required for regression data. VI. CONCLUSION From the findings of this study the following inferences can be drawn: 9

5 World Academy of Science, Engineering and Technology 9 a. The Median Regression is relatively better than the Least Squares Regression for slightly skewed data basing from the magnitudes of the standard errors of the regression coefficients. b. The Median Regression gives asymptotic efficient estimates for the coefficients of the regression model. VII. FUTURE DIRECTION There are still other robust methods in dealing with slightly skewed data mentioned in the literature. Hence, it is hereby recommended to compare the findings of this study to the results from other robust methods. Other resampling plans which reduce the biasedness in the standard errors of the regression estimates are also endorsed to be explored. REFERENCES [] Barrodale,I. and Roberts,F.D.K. (97). An Improved Algorithm for Discrete l Linear Approximation, SIAM J. Nmer. Anal.,, [] Gutenbrunner, C. and Jureckova, J. (99). Regression rank scores and regression quantiles, Annals Of Statistics (), -. [] Koenker, R and d Orey,V. (99), Computing Regression Quantiles, Applied Statistics,, -. [] Koenker, R. And Bassett,G.W. (978). Regression Quantiles,,Econometrica, 6, -. [] Yaffee, R. A. (). Robust Regression Analysis: Some Popular Statistical package Options, Statistics, Social Science and mapping Group, Academic Computing Services, Information Technology Services.