Re: gpower: Does post hoc power analysis make sense?


Nina H. wrote:
Is there a text like this on the "Box-M-Test" as well? I am still
wondering about the exact hypotheses it tests, despite many hours
searching the net. As I understand it, the Box-Test evaluates
homogeneity of variance in multivariate approaches, that is,
homogeneity across all levels of each factor...? Or does it test
"compound symmetry", after all?

Most of the criticisms of Mauchley's tests are general ones and can be
extended to any significance test of assumptions. AIUI the Box M test
is also very vulnerable to departures from normality though it isn't a
test I've ever used and I'm not really a fan of MANOVA anyway. In amny
(most?) cases MANOVA is followed up by univariate tests which make the
MANOVA tests redundant.

Actually, GPower does multivariate analyses
(http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/reference/reference_manual_09.html),
but only *post hoc*. This is what I am doing, even though I am not sure
whether it makes much sense.

Yes - it is a while since I looked. I'm unsure how useful they are
because of the complexity of the power issue - especially when
comparing MANOVA and ANOVA approaches (though Karl Wuensch has some
good pages on this). Working out power for a priori contrasts is a
win-win situation because the contrast is a more powerful test of a
more interesting hypothesis and because it simplifies power
calculations.

That is, I now followed your advise and estimated power for paired
sample tests of the main hypotheses. I report these values in the text
(along with F, df, p and effect sizes) for effects which failed to
reach significance. Does this make sense?

OK. I didn't realize this was a retrospective power analysis. I'm
really not keen on that kind of analysis (there is a link in my
sphericity piece to a paper by me that mentions this - Baguley, 2004
and gives several references). Power calcualtions make most sense as
part of the planning process but can be problematic if conducted post
hoc. The crucial problem is where the effect size estimate is from. If
the effect size estimate is from the sample you are estimating power
for then the resulting power will at best be redeundant and at worse
misleading. This is because the power is more-or-less a monotonic
function of the observed p. So significant effects will produce high
estiamtes of power and non-significant effects low estiamtes of power.
If you want to get meaningful post hoc power estimates you need an
effect size estimate from some other source. This is possible, for
example if you had a model that predicted a certian effect size, or
wanted to know the probability to replicate an effect from a previous
studiy you could use those effect sizes to estimate post hoc power.

Thom

.