Re: A basic question on Canonical Correlation Analysis


ziman137 wrote:
> Dear Bob,
>
> Please see my comments below. I am trying not to state things in my
> humble opinion (imho), instead state what I knew as facts. However,
> since I am not a statistician, I may have missed important points, if
> so, please correct me - I would really appreciate if you could also
> point me to specific references (I will do so as much as I can). Thanks
> in advance!
>
Thank you for your very friendly excessively humble preamble. I
shall try to reciprocate, at least with respect to the friendliness
of this discussion.

>
> Reef Fish wrote:
> > Jerry Dallal wrote:
> > > Actually, it's baaaaaaaaaaaack! Maybe.
> >
> > I don't think so, and I hope not; at least not among statisticians.
> >
>
> Comments: I apologize if I have not clearly stated what I meant for
> Reduced-rank Regression, and I really did not mean Principal-Component
> Regression (PCR). From what you stated below, it seems that I have not
> explained clearly enough - forgive my bad English. However, here I do
> refer to reduced-rank regression as CCA-based multivariate regression.
> BTW, these are not my innovative ideas. Specific references were back
> to early mid 70's:

I wasn't aware of YOUR meaning of "Reduced-rank Regression", since
THIS is the first post of yours I've ever read. I was commenting on
Jerry Dallal's use of that term. I know Jerry well, and know that
he was trained as a statistician at Yale and he is presently holding
an academic position as a statistics professor in a reputable
university. Thus, I was merely GUESSING that he may have
used that for terms that sounded similar and had appeared in the
statistical literature, but not in any of the graduate level
multivariate
statistical analysis textbooks from which I've taught for well over
20 years.

>
> Izeman, A. 1975, Reduced-rank regression for multivariate linear model,
> Journal of Multivariate Analysis 5: 248-264
>
> van de Merwe, A. and Zidek, J. 1980, Multivariate regression analysis
> and canonical variates. The Canadian Journal of Statistics 8: 27-39
>
> The much newer refs (smoothing shrinkage of canonical variates), e.g.,
>
> Brieman, L. and Friedman, J. (1997) Predicting multivariate responses
> in multiple linear regression, J. Roy. Statist. Soc. B. 59: 3-37.

I have not read any of the above, nor do I have access now to any
of them. But only judging by the titles of the papers, there does not
seem to have any direct connection of multiple or multivariate
regression (and especially prediction) to the "classical" definitions
of the problem of Canonical Correlation Analysis.

>
> Point is: Reduced-rank regression that I was/am talking about is NOT
> PCR.

I wasn't referring to anything YOU talked about.
While those may not be about PCR, they appear not to be CCA either
because CCA (just like PCA and Factor analysis) are concerned with
the multivariate structures rather than "predictions" as in regression.

>
> > It may be back in some areas in the Social Sciences where most
> > abuses of statistics occur in the first place, such as using
> > correlation as a proof of causation. But that's a Dead Horse
> > I don't care to revive or exhume.
> >
>
> Comments: correlation having nothing to do with causation is well known
> to everyone, and it's not relevant to the topic at all.

If you've been in the sci.stat.math/edu groups, you wouldn't have
made your statement about correlation and causation. It is relevant
to THIS group only to the extent I was using it as an example of
the ABUSE of statistical methods, by others in sci.stat.math/edu,
in my response to Jerry Dallal.
>
> >
> > > I think CCA underlies what's
> > > being called reduced rank regression.
> >
> > I heard the terms "principal components" and "factor analysis"
> > in several of the discussions. But those are NOT, and are not
> > related to, the definition and practice of Canonical Correlations!
> >
>
> Comments: you are absolutely right that "PCA" and "factor analysis" are
> not "CCA", which perfectly explains why "PCR" has fundamentally
> different ideas from CCA-based Reduced-rank regression. Hereafter, I
> simply refer to CCA-based reduced-rank regression as reduced-rank
> regression (I suppose it is so among general statistical community,
> imho, sorry - my opinion).

I am glad to see you clarify your point and your position on the
subject,
and confirmed my suspicion (as stated to Jerry Dallal on the CCA
subject) about PCA, PCR, and Factor Analysis as fundamentally
different ideas from CCA.
>
> >
> > Your term "reduced rank regression" may be the same or
> > similar to "principal components regression" -- which is a
> > REGRESSION method (not related to Canonical Correlations)
> > that has been used (abused) for years, both by statisticians
> > and social scientists.

It COULD be called "reduced rank" -- as I said, I was merely
speculating on what Jerry Dallal might have meant by HIS use
of that term.
> >
>
> Comments: No! "reduced rank regression", as I commented above and
> somehow explained by your own statement, is NOT PCR, instead it is
> CCA-based. Please carefully look into the references I pointed above
> before you decide not to agree - I'm trying to state facts here,
> instead of my opinions, and my deep apology if I made it confusing.

Having seen some of the similar/related methods I mentioned,
which some writers have used for regression, I have absolutely
no interest in any "reduced rank regression" whether it is related
to CCA or not.

The simplest and most FACTUAL way of expressing my own
understanding of the subject of regression is that I think any of
the shrinkage regression methods (Ridge, Stein's, etc.) are
as unjustified and ridiculous as any other shrinkage or reduced
rank methods. No one interested in applying statistics would
take Stein's well-known results seriously.

The only way to use a "regression" method well is to understand
what is entailed in the DEFINITIONS, assumptions, and what
one CAN or CANNOT do or infer from those principles. Any
thing that tries to justify deviations that are unjustibiable
(PCR, Ridge, etc., etc.) are at best mis-applications, and
at worst statistical Quackery.

I have given many multiple regression analysis and linear
models references in my posts in sci.stat.math in many threads
of discussions held earlier this year.

>
> > Hadi and I showed, in a nutshell (again :-)) , that there is
> > absolutely no redeeming value in the use of PCR (principal
> > Components Regression), but in a published article with
> > a title in a softer tone:
> >
> > " Some Cautionary Notes on the Use of Principal Components
> > Regression".
> >
> > http://www.amstat.org/publications/tas/index.cfm?fuseaction=hadi1998
> >
>
> Comments: I carefully read the whole abstract of your paper, which
> should have some nice points - since I tried to get the full electronic
> paper and was not successful, I could not look into details.

I am not even aware of a full electronic version of the paper, unless
it's in the ASA journal subscription option. But if you send me your
mailing address (to my Large_Nassau_Grouper@xxxxxxxxx address)
I'll be glad to snail mail you a hard copy of that paper.

Else, you might inquire it from Hadi's Cornell email address -- which
he still has access, though he has been back to Egypt in recent
years. He was the one who put up the abstract and related
materiial in webpages.

> The
> possibility indeed exists such that the least principal component
> contributes everything - consider that for a set of multiple-variable
> inputs, find its least principal component, then project each sample
> into this component and construct a perfect linear correlation with
> some fixed slope to generate the single-output response. That's worst
> senario you mentioned in your paper. However, the least principal
> component must correspond to the first (and the only) canonical
> variate, and it gives us the first canonical variate coefficients.

While your last statement about PCA and CCA may be true under
certain circumstances, it definitely does not follow from the
definitions of PCA and CCA given in statistical textbooks on
Multivariate Statistical Analysis (of any of the well-known authors).

> This
> is a perfect example to show PCR is different from CCA-based
> regression. (The proof is trivial, since the perfect correlation
> indicates 1.0 and it must be the highest and the only canonical
> correlation)
>
> > The hidden subtitle of the paper is, "Don't EVER use PCR --
> > you ALWAYS do better without the use of those PC's".
> >
>
> Comments: this statement is plainly wrong.

The statement is a direct consequence of the results in my joint
paper with Hadi, and the dozens of references we've both studied
in the literature which use PCA regression methods. In fact, we
had found published papers given covariance matrices for PCA
that had negative eigenvalues!

> The logic is quite simple.
> Consider my example I constructed in the last comment: now, instead of
> projecting each sample into its least principal component, we can
> project each of samples into the first principal component and, again,
> construct a perfect correlation with fixed slope to generate the mean
> output response. Besides the mean output response, try generating a
> random noise for each sample along the direction of the least principal
> component. The final response is the add-up of mean output (perfect
> response to the 1st PC) and the noises (random response to the least
> PC). In this case, the 1st-rank-cut PCR does the perfect job to
> predict the mean response. To nullify the statement "you ALWAYS do
> better without the use of those PC's", one counter-example like this
> will do. Sure infinite counter-examples are there to nullify this
> statement.

Your example, which I do not fully follow, suffers from the rather
obvious misunderstanding of yours about "multiple regression" in
statistical analysis. It is NOT concerned with any PC nor the
construction of any perfect correlation, other than the FACT
that a perfect "multiple R" in a multiple correlation corresponds
to a perfect simple correlation between the observed Y and the
fitted/predicted Y in the resulting regression.


> > For some reason, your post remind me of an old saying,
> >
> > "Fools rush in where angels fear to tread".
> >
> > -- Bob.
>
> Nice quote, Bob, very nice. I'll make a note of that.
>
> Gary

Again, I was saying that about what Jerry's post reminded me.
I am not going to try to second guess what hidden meaning your
line may have.

-- Bob.

.