Re: transformation of regressors to remove collinearity
- From: Richard Ulrich <Rich.Ulrich@xxxxxxxxxxx>
- Date: Tue, 11 Mar 2008 18:55:09 -0400
- I'm going to repeat some of what has been posted
already, and (hopefully) add to it.
On Tue, 11 Mar 2008 07:08:13 -0700 (PDT), papu <prachar@xxxxxxxxx>
wrote:
On Mar 11, 9:05 am, vontres...@xxxxxx wrote:
On Mar 10, 8:34 pm, papu <prac...@xxxxxxxxx> wrote:
Hi All,
In regression, is there any way we can transform the predictors to
remove the collinearity?
There is this article in wikipedia which talks about geometric
interpretation of correlation.http://en.wikipedia.org/wiki/Correlation#Geometric_Interpretation_of_...
Basically here they say that, if we look at the predictors as vectors
then the correlation coefficient can be treated as the cosine of the
angle between the two predictors.
Lets say we define our dimensions first take projections of predictors
(vectors) on to those dimension. Then the correlation between those
projections will be zero because they lie on the dimension
(correlation between dimensions is zero). If we make use of these
projections we can get a regression equation without collinearity.
By the way, the way this is done is what others have
called Regression on Principal Components, which was
implemented in some of the earliest regression packages.
It has not been greatly popular, and for good reasons.
It seems a *little* bit fruitful if all the useful Predictors
are defined by the largest two or three Components, and
you don't mind excluding the other components.
- The big problems arise when "small" components are
useful in prediction. Is this the result of capitalizing on
chance? Is the small-eigen-value component going to be
robust, that is, reliable enough to be useful?
- That still leaves the mess of interpreting or using the
component scores, which have "loadings" of every variable.
What is more useful, for some data, is the use of Principal
Components or Comnon-Factor Analysis to derive a few
composite scores -- which might be scored-up as (say) a
"Depression" total, "Anxiety", etc. These will still be somewhat
correlated, but there will be far fewer of them to describe
as separate "hypotheses", compared to using every Item.
Well that is the notion I have and I am not able to find any material
on that. Can somebody point me to information on this?
Or if this is a pretty obvious fact already considered in regression,
please let me know how.
-Papu
Papu,
Collinearity is another way of saying that two or more predictors
are redundant. You really only need one since the others provide the
same information. There are lots of ways to deal with this problem,
but I will share a simple way that works for small numbers of
predictor variables.
Fit a regression of each pair of predictor variables with each other.
Let the response (independent variable) be one predictor and the
"predictors" be another set of the predictor variables. If there is
high correllation between two predictor variables, then they are
supplying redundant information and one of them can be deleted. This
- Delete one; or replace two with a composite of them both.
is the basic idea of all of the methods for removing collinearity, but
it delves into obsure words like singular valued decomposition or
principal components analysis - which are harder to interpret.
Also, note that a near constant variable will be collinear with the
intercept, so near constant variables can also be ignored.
This is a good recommendation to make a "sensible" reduction
of variables - "near constant" might be because a sample has
been restricted (narrow range, as in Age, weight, etc.), or because
(say) only two or three cases exist on one side of a dichotomy.
However, it is not always immediately evident -- When a person
may be running a fever, the exact body temperature might
look like a near-constant variable, but it *might* be an important
predictor.
Once you have removed all of the redundant predictors, you should be
able to fit your regression model without collinearity.
Mark- Hide quoted text -
- Show quoted text -
Thanks mark..Actually because of model governance rules we are
required to use a certain minimum number of variables. The VIFs need
"model governance rules" is something that throws me off.
If the first (largest) Principal Component score is used, a-priori,
it has weights for every variable. Is this "using one variable", or
more? What if a factor analysis leads you to form a composite
of 4 particular variables? - Does that get counted as 1 or 4?
- I could ask someone to use "4 or more predictors" in
order to get a more robust prediction, assuming that I know
this much about my problem; and using a factor score
would count that way. But if I was assigning homework, I
suppose I might want to see some *discussion* of several
variables in the regression equation.
to be less than 3 and correlation coeffcient should be less than 0.7.
I have some predictors that have correlation coefficient 0.75 or
so..they are not highly correlated but not desirable either..Hence I
thought why not separate the correlated and uncorrelated parts between
the predictors using geometric decomposition or projections and use
those transformation as predictors. This way the collinearity is zero.
But the question remains how to decompose these predictors and will
the predictive power of the transformed predictors remain the same as
a group.
When you do "stepwise" selection or elimination of variables
while letting the computer program make the choices,
you get into a situation that has enormous potential for
capitalizing on chance, thus producing worthless equations.
The empirical solution for this requires a very large sample,
since it is done by large-scale cross-validation. Even so,
small changes in the "Population" of interest can invalidate
the predictions. (This is used by some Wall street stock-pickers,
and the change-of-Population is why their best validated equations
only work for a year or so.)
If a human helps make the choices in reducing variables, using
knowledge of the content and measurement issues, then
there is a better chance for obtaining a robust solution.
--
Rich Ulrich
http://www.pitt.edu/~wpilib/index.html
.
- References:
- transformation of regressors to remove collinearity
- From: papu
- Re: transformation of regressors to remove collinearity
- From: vontressms
- Re: transformation of regressors to remove collinearity
- From: papu
- transformation of regressors to remove collinearity
- Prev by Date: Sum of gamma distributed random variables
- Next by Date: Re: Sum of gamma distributed random variables
- Previous by thread: Re: transformation of regressors to remove collinearity
- Next by thread: Re: transformation of regressors to remove collinearity
- Index(es):
Relevant Pages
|
