Re: power analysis: within-subjects, 2x2, random factors: How ?
From: Richard Ulrich (Rich.Ulrich_at_comcast.net)
Date: 06/28/04
- Next message: Vince: "monte carlo and variance"
- Previous message: Richard Ulrich: "Re: Survival Analysis"
- In reply to: Martin O'Hare: "power analysis: within-subjects, 2x2, random factors: How ?"
- Next in thread: Martin O'Hare: "Re: power analysis: within-subjects, 2x2, random factors: How ?"
- Reply: Martin O'Hare: "Re: power analysis: within-subjects, 2x2, random factors: How ?"
- Messages sorted by: [ date ] [ thread ]
Date: Mon, 28 Jun 2004 01:31:28 -0400
On Mon, 28 Jun 2004 00:38:35 +0100, "Martin O'Hare"
<intersity@hotmail.com> wrote:
> Dear all,
>
> One of the reviewers of an article I submitted to a journal asked about some
> results of an experiment I reported as not reliable whether the design had
> enough power to detect such a difference, if one existed, and asked for a
> power test. I'm quite confused because I've read (and been convinced) that
> post-hoc power analysis is bogus. On the other hand it seems that I have to
> do it (proposed interval analysis but the editorial board didn't get it/like
> it). I am not statistician and my knowledge of statistics is limited to the
> basics. I'm not sure how to do a power analysis with the design I have.
Okay, I haven't see their criticism. And I haven't see what
you have actually provided them, in multiple pages and tables.
I've got guesses.
One: They were confused by whatever you did, and aiming you
at 'power' was a best-guess for what might help clarify your
procedure, because you should have been averaging scores
instead of using 100 items. I say that because I certainly
am confused. And I know that I have tendered "suggestions"
as a reviewer, without wanting to impose that requirement on
the authors -- I try to show what it was that *thought* they
were aiming at, or how to reach it.
Two: You don't say that you average the 100 items into cells,
or any other way. That bothers me, because it suggests to me
that you could be doing the wrong tests. "Power" might be
the issue, if you are testing with the irrelevant variability
between "items" as the error term.
Three: When you put scores into the framework for an item
analysis, you have to lay out the numbers in a way that you
can see what the 'effect size' is that you have on hand, and
what effect size was needed for achieving statistical significance.
When you look at your data that way, does it make *sense*?
Are you merely achieving effects that are small, and everyone
would agree? Or do they look big enough to be useful,
but (for whatever reason) they don't test out?
But it seems to me, my criticism might be hastily stripped down
to match theirs, "asking for a power test" - even though I agree
that "post-hoc power analysis is bogus" is a pretty fair statement.
I'm suggesting that "there is something wrong with the
presentation" and it is probably related to power; but giving
them a formal power analysis is not the end of it. The start
of it is what matters, where you show that the outcome is
'small' in everyone's terms. Or you fix the analyses by
averaging scores to get a better criterion, or by testing
simple contrasts against their proper errors.
>
> I had about ~25 subjects (same number to other studies in the field
> examining similar stuff), all of them examined 100 items, DV was continuous,
> items were balanced across two binary 2-level IVs. Here are some more
> details:
>
> Factor 1 and Factor 2 were crossed in a 2x2 within-subjects design. The two
> within-subjects factors were Factor 1 (level 1 or level 2) and Factor 2
> (level 1 or level 2) and the analysis was carried out by-subjects (F1) and
> by-items (F2). For the by-subjects analysis Factor 1 and Factor 2 were
> treated as fixed factors whereas subjects was the random factor. For the
> by-items analysis, items nested within the compound type and familiarity
> conditions were the random factor.
> All terms and possible interactions between them were included in the model.
Items nested? Is that crucial?
All terms and interactions? I see an important 2x2 design,
with an important interaction. The between-subject variation
in a contrast is the useful error term for the within-subject effect
- just like a paired t-test.
Does a simple test get anything?
> General Linear Model's regression was used for the analysis of variance of
> means.
>
> What I should do ? I looked high and low for software and formulae. I found
> formulae for different designs but I find it difficult to adapt them to my
> problem and do not want to do something wrong or based on "intuition".
>
> Any help will be greatly appreciated !
Repeated measures is tough to describe power analyses for.
The key is to simplify. What is the contrast? What is the
error term? The power analysis for a contrast of two groups
is most intelligible as the paired t-test: Which is: the one-sample
t-test on the difference in means. For that, you want the
difference, and you want the standard deviation of the
difference, across subjects; you might *estimate* that
from the raw standard-deviation between subjects, for
one set of 50 items (if I've reconstructed the experiment right),
and the average correlation between sets of 50 items.
The ANOVA analysis that I described earlier is the basis for
the power analysis, so you want to get these means, even
if you have justification in your head for later testing
something about the items.
Hope this helps. If I missed the design too far, try again.
-- Rich Ulrich, wpilib@pitt.edu http://www.pitt.edu/~wpilib/index.html
- Next message: Vince: "monte carlo and variance"
- Previous message: Richard Ulrich: "Re: Survival Analysis"
- In reply to: Martin O'Hare: "power analysis: within-subjects, 2x2, random factors: How ?"
- Next in thread: Martin O'Hare: "Re: power analysis: within-subjects, 2x2, random factors: How ?"
- Reply: Martin O'Hare: "Re: power analysis: within-subjects, 2x2, random factors: How ?"
- Messages sorted by: [ date ] [ thread ]
Relevant Pages
|
|
