A Note on the Finite Sample Properties of Ridge Estimator

 

Madhulika Dube^1*, Isha^2**, Vinod Kumar^3**

{¹Professor and HOD, ^{2, 3}Research Scholars}, Department of Statistics, M. D. University, Rohtak, Hariyana, INDIA.

Corresponding Addresses:

 ^**[email protected], ^***[email protected]

 

Abstract: The article studies the finite sample properties of generalized ridge regression estimator using a different approach. A comparative study of relative bias and relative efficiency of the estimator with respect to the ordinary least square have been made empirically. The results have also been compared with the existing results and are found to be quite different from those already existing in literature.

Keywords: Ordinary Least Squares, Generalized Ridge Regression (GRR), Relative Bias, Relative MSE, Relative Efficiency, Finite Sample Properties.

 

Introduction

In linear regression models, Ridge Regression is perhaps the most widely used technique in the presence of multicollinearity. Proposed by Hoerl and Kennard [3], [4] the Ridge Regression Estimator is characterized by a scalar, the choice of which is subjective requiring the judgment of the analyst. However, working with the canonical form of the regression model Hoerl and Kennard [3] defined general ridge regression estimator suggesting an initial choice of the characterizing scalar. Extensive work has been carried out since then, a good account of which is available in Vinod and Ullah [7] and [6]. Working with the initial choice of the characterizing scalar, Dwivedi et al. [1] worked out the first two moments of individual coefficients of GRR estimator assuming error distribution to be normal. Hemmerle and Carey[2] also worked out exact properties of two different forms of GRR estimators but demonstrated that the one suggested by Hoerl and Kennard [3] performs better in terms of relative bias and relative risk. It will not be out of context to mention that the estimators perform differently when sample size is small and it is more so in the presence of multicollinearity as negative effects of multicollinearity are magnified in smaller samples. Owing to this, assuming error distribution to be normal the paper attempts to assess the finite sample behavior of generalized ridge regression. For this purpose the relative bias, relative mean squared error and the relative efficiency of the estimator in comparison to OLS have been evaluated numerically and compared with the existing results. Interestingly, the expressions of relative bias and relative risk are found to be different from the existing results obtained by Dwivedi et al. [1]. The following section describes the estimator and its properties and empirically enumerates the results. A brief outline of proof of the theorem is provided in the end.

The Estimator and its Properties

Consider the canonical form of the linear regression model

                                                                                                                                           (2.1)

where y is an vector of  observation on the dependent variables, X is an  full column rank matrix of  observation of  explanatory variables, and  vector of unknown regression coefficients . The elements of disturbance vector u are assumed to be i.i.d each following normal distribution with mean zero and variance σ² so that

                                                                                          (2.2)

Following [3], we can write

This canonical reduction can be obtained by using singular value decomposition of the  matrix X (see, [7], p5-6). Using it the general ridge regression estimator is given by

                                                                                                                               (2.3) 

and K is a diagonal matrix with nonnegative elements k₁,k₂,…k_p as the characterizing scalar, and

 

is the ordinary least square estimator of . Clearly, GRR is a biased estimator, with bias vector

                                                                                                                          (2.4)

 

and

   

                                                                                                                                        (2.5)

as  and K are assumed to be diagonal matrices,  is also diagonal and

                                                                                                                (2.6)   

Provided  ‘s are non-stochastic.

Now minimizing the expression (2.6) term by term i.e. minimizing the diagonal elements of the mean squared error matrix of (2.5) with respect to  yields

                                                                                                                   (2.7)     

[3] Suggested to start with

                                                                                                                                         (2.8)    

where  is the i^th element of the ordinary least squares estimator b of  and

                                                                                                                      (2.9)    

is an unbiased estimator of  where

Using (2.8) in (2.3) leads to an adaptive estimator of as

                                                                                                                                  (2.10)    

where the i ^th element of is given by

                                                                                                                                        (2.11)     

For finite sample sizes, the following theorem gives the first and second moments of

Theorem: Assuming normality of errors the first and second moments of of (2.11) are given by

 

where  is the non-centrality parameter. Using (2.12) and (2.13) we can compute the bias and mean squared error of  .

Proof: see Appendix.

Using these, it is easy to compute the relative bias and relative mean squared error using

                                                                                                                              (2.14)   

and

                                                                                                                       (2.15)   

Respectively. The efficiency of the OLS relative to GRR estimator is obtained from

                                                                                                                               (2.16)   

 

The values of  and have been tabulated for a few selected vales of  and .These results are provided in Tables 2.1, 2.2, 2.3 respectively and have been graphed for selected values of  and.

The expressions (2.12) and (2.13) are clearly different from those obtained by Dwivedi et al. [1] and therefore a substantial difference is observed numerically. Unlike the results obtained by Dwivedi et al. [1], the magnitude of relative bias is found to be decreasing function of the non-centrality parameter and an increasing function of degrees of freedom, so long as  and . However, for  the magnitude of relative bias is found to be increasing. Interestingly, as  increases the relative bias tends to -1, justification of this easily comes from the fact that the Ridge regression estimator shrinks the parameter vector towards zero. The relative MSE and relative efficiency are also observed to be decreasing for specific values of  and Hence, the finite sample properties of the Ridge regression estimator are not only heavily dependent upon the non-centrality parameter but on the degrees of freedom as well. It is also pertinent to mention that ambiguity in the numerical computations in relative bias, relative MSE and relative efficiency are found in the paper by Dwivedi et al. [1] when  which are evident in the respective tables.

Table 2.1: Relative Bias for specific values of non-centrality parameter and degrees of freedom

 

E0 –Results by our approach

E1-Results by Dwivedi et al. [1].

Table 2.2: Relative MSE for specific values of non-centrality parameter and degrees of freedom

E1-Results by Dwivedi et al. [1].

 

    

Graph: Showing relative bias and relative MSE of GRR Estimator (Figure 1 and Figure2)

Table 2.3: Relative Efficiency for specific values of non-centrality parameter and degrees of freedom

 

E0 –Results by our approach

E1-Results by Dwivedi et al. [1].

Appendix        

In order to find the expression (2.12) of the theorem, let us define

                                                                                                                   (A.1)

where x_i (i=1,2…p) is the i^th column vector of X. Since is the OLS estimator of  following

N(), therefore, the distribution of  where  .

Next, the distribution of is  with  degrees of freedom and is independent of the distribution of. Using, we can write

                                                                                                                                  (A.2)

                                                                                                                        (A.3)    

Following [5] we can write the above equation as

 We notice that the value of integral is zero for odd values of j because then the power of z is odd. Dropping such terms we have

                                                                                 (A.7)

Substituting the above value of (A.7) and using it in (A.6) we get the first raw moment of as given in the theorem.

Proceeding in the same way the second and higher raw moments of  can be obtained.

 References

1. Dwivedi, T. D. Srivastava, V. K. Hall, R. L., “Finite Sample Properties of Ridge Estimators”. Technometrics, Vol. 22, No. 2. Pp205-212, 1980.
2. Hemmerle, W. J. and Carey, M.B., "Some Properties of Generalized Ridge estimators" Communications in Statistics: Simulation and Computation. 12:3, 239-253, 1983.
3. Hoerl, A. E. and Kennard, R. W., “Ridge Regression Biased Estimation for Non-Orthogonal Problems”. Technometrics, 12, 55-67, 1970a.
4. Hoerl, A. E. and Kennard, R. W. “Ridge Regression Applications to Non-Orthogonal Problems”. Technometrics, 12, 69-82, 1970b.
5. Judge, G.G. and Bock, M.E., the Statistical Implications of Pre-Test and Stein-Rule Estimators in Econometrics. North Holland Publishing Company, Amsterdam, 1978.
6. Marvin, H.J. Gruber, Regression Estimators: A Comparative Study. The John Hopkins University Press, 2010.
7. Vinod, H.D. and Ullah, A., Recent Advance in Regression Analysis. Marcel Dekker, New York, 1981.

 

 

 

Copyrights � statperson consultancy www.statperson.com  2013. All Rights Reserved.

Developer Details