Re: ANOVA on composite scores & interaction

Here is a comment on the comment, and then a comment
on the problem.

On Thu, 26 Apr 2007 00:58:23 GMT, "MathCraft Consulting"
<mathcraft@xxxxxxxxxxxxx> wrote:

Without familiarity with your particular data, my initial reaction is to
ask, "Why didn't you just apply MANOVA, instead of doing a PCA for ANOVA?"
Any overall effects can be followed up by tests of individual measures or
significant linear combinations of measures (based on the characteristic
roots and vectors or E^(-1)xH, which you should be able to specify as output
from the procedure.

Well, there is the loss of power, which can be considerable.

Using the overall test from MANOVA is not necessarily a
good option, compared to doing an overall test on a few
composite scores -- especially if the composite scores are
intelligible. Also, using the MANOVA leaves the same
criticism intact, if the overall test is a main effect, and follow-ups
are interactions.


MathCraft Consulting
Dayton, Ohio

<itjohnstone@xxxxxxxxx> wrote in message
news:1177539316.525226.52270@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I have a statistical question for which I would very much appreciate
receiving some help.

I have an experimental dataset in which two factors were independently
manipulated, and I recorded a bunch of measures. The measures are
correlated to varying degrees, so as a data reduction approach (at the
bequest of reviewers and editors to reduce multiple comparison-related
Type I error) I used PCA to generate 3 composite measures. I then
analysed each composite measure using a 2-way univariate ANOVA.

For one of the composite measures the ANOVA yielded a main effect of
one of the factors (call it A), but not the other factor (call it B),
nor any interaction of AxB. Fairly straighforward. But - when I look
at each of the original measures that loaded on the composite measure,
I see that most of them show an AxB interaction, and no main effect of
A.

My tendency in this case is to stick close to the data, and report the
individual measure interactions. But I have received criticism that
this is "cheating" because my composite score only shows a main effect
of factor A, and by even looking at the single measure tests I have
undone my data reduction solution. But I feel that reporting just a
main effect would be misleading, and that since I only examined the
individual measure ANOVAs after finding an effect on the composite
measure, that this is justified.

Any thoughts or suggestions? Or even better - a reference to an
article or book that deals with this situation. I would really
appreciate any advice.

I share the bias, which other people have expressed, that
an "interaction" very often expresses some flaw in the expression
or understanding of the data.

Reading the description above, I immediately suspect that
the individual measures have serious problems in scaling --
such as, displaying basement and ceiling effects, and the
consequential heterogeneity of variance at extremes. In that
case, the items could show interactions (as reported) and the
sum could show a main effect (as reported).

What is the nature of the interaction? Of course, this is
much easier to describe if the design is 2x2 instead of
something bigger.

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: ANOVA on composite scores & interaction
    ... composite scores are intelligible," seems like a big one. ... so as a data reduction approach (at the ... My tendency in this case is to stick close to the data, and report the ...
    (sci.stat.math)
  • ANOVA on composite scores & interaction
    ... receiving some help. ... so as a data reduction approach (at the ... Type I error) I used PCA to generate 3 composite measures. ... analysed each composite measure using a 2-way univariate ANOVA. ...
    (sci.stat.math)