Re: transformation of regressors to remove collinearity

On Mar 11, 9:05 am, vontres...@xxxxxx wrote:
On Mar 10, 8:34 pm, papu <prac...@xxxxxxxxx> wrote:





Hi All,

In regression, is there any way we can transform the predictors to
remove the collinearity?

There is this article in wikipedia which talks about geometric
interpretation of correlation.http://en.wikipedia.org/wiki/Correlation#Geometric_Interpretation_of_...

Basically here they say that, if we look at the predictors as vectors
then the correlation coefficient can be treated as the cosine of the
angle between the two predictors.

Lets say we define our dimensions first take projections of predictors
(vectors) on to those dimension. Then the correlation between those
projections will be zero because they lie on the dimension
(correlation between dimensions is zero). If we make use of these
projections we can get a regression equation without collinearity.

Well that is the notion I have and I am not able to find any material
on that. Can somebody point me to information on this?

Or if this is a pretty obvious fact already considered in regression,
please let me know how.

-Papu

Papu,
  Collinearity is another way of saying that two or more predictors
are redundant. You really only need one since the others provide the
same information. There are lots of ways to deal with this problem,
but I will share a simple way that works for small numbers of
predictor variables.

Fit a regression of each pair of predictor variables with each other.
Let the response (independent variable) be one predictor and the
"predictors" be another set of the predictor variables. If there is
high correllation between two predictor variables, then they are
supplying redundant information and one of them can be deleted. This
is the basic idea of all of the methods for removing collinearity, but
it delves into obsure words like singular valued decomposition or
principal components analysis - which are harder to interpret.

Also, note that a near constant variable will be collinear with the
intercept, so near constant variables can also be ignored.

Once you have removed all of the redundant predictors, you should be
able to fit your regression model without collinearity.

Mark- Hide quoted text -

- Show quoted text -

Thanks mark..Actually because of model governance rules we are
required to use a certain minimum number of variables. The VIFs need
to be less than 3 and correlation coeffcient should be less than 0.7.
I have some predictors that have correlation coefficient 0.75 or
so..they are not highly correlated but not desirable either..Hence I
thought why not separate the correlated and uncorrelated parts between
the predictors using geometric decomposition or projections and use
those transformation as predictors. This way the collinearity is zero.
But the question remains how to decompose these predictors and will
the predictive power of the transformed predictors remain the same as
a group.

.

Relevant Pages

  • Re: transformation of regressors to remove collinearity
    ... Then the correlation between those ... projections we can get a regression equation without collinearity. ... It seems a *little* bit fruitful if all the useful Predictors ...
    (sci.stat.math)
  • Re: transformation of regressors to remove collinearity
    ... In regression, is there any way we can transform the predictors to ... if we look at the predictors as vectors ... projections we can get a regression equation without collinearity. ... Fit a regression of each pair of predictor variables with each other. ...
    (sci.stat.math)
  • Re: Principal Component Analysis
    ... That set can be all the raw variables or the set of summative scales based of the factor analysis. ... PCA and "Correlation among predictors". ... prediction with no fear of loosing information, ...
    (sci.stat.consult)
  • Re: Questions about square errors
    ... Take a look at the 10X10 correlation coefficient matrix and the ... multicollinearities. ... least squares and/or multiple regression. ... Your model may have several unnecessary predictors. ...
    (sci.stat.math)
  • Re: Principal Component Analysis
    ... When you have 30 input variables, and they are correlated, ... predictors are correlated and I am fully in agreement with his ... Interpretation assumes cause-and-effect relationship. ... repeated so many times: Correlation is not causation. ...
    (sci.stat.consult)