Re: Bootstrapping for confidence interval of effect size

Hi all,

Thanks for your reply Rich. I take your point about how you can't
compare a confidence interval to a non-constant mean. However, I have
seen it written in many places that comparing a confidence interval to
a constant is equivalent to a hypothesis test. So it seems to me that
logically there ought to be some way to compare two confidence
distributions (which is what you get with a bootstrap).

For example, you could compare every point (Ai) in bootstrap
distribution A with every point (Bi) in bootstrap distribution B, and
count how often Ai > Bi. That would give your your confidence that A
is greater than B, right? It seems to me that it is exactly the same
logic as comparing against a fixed constant.

Your point about how the control groups are not equal doesn't apply -
I think you missed a point in my original point - I am comparing the
effect sizes of the two comparisons, not the means. There may be
something wrong with that aspect of what I did but it isn't what you
said! ;)

By the way, I have always in the past transformed my data too, but
this data is just untranformable - you would see what I mean if you
saw it.

Cheers!

Ben

On Jul 19, 4:38 am, Richard Ulrich <Rich.Ulr...@xxxxxxxxxxx> wrote:
On Wed, 18 Jul 2007 16:02:38 -0000, amorphia <spam.onto...@xxxxxxxxx>
wrote:





Hi all,

I have a question about whether I have used bootstrapping correctly.
For two different ages I have a treatment group and a control group.
The data is not at all normally distributed so I have done a bootstrap
two-sample test to show that for the older individuals there is a
significant effect of treatment, and a second such test for the
younger ones which shows no significant effect. I am happy that I have
done this correctly. The next bit I am less sure of. I have calculated
a bootstrapped 99% confidence interval for the effect size (Cohen's d)
for the older individuals, by repeatedly resampling the treatment and
control groups and calculating the effect size for each pair of
treatment and control resamples. The effect size of the non
significant trend for the younger individuals lies outside this 99%
confidence interval for the older age group. I therefore conclude that
it is unlikely that an effect of equivalent size exists in the
population of younger individuals which I have failed to detect.

Does that make sense? If not, is there a better way to do this?

Well, it "makes sense" in the sense that it is a common mistake.

You do not *demonstrate* a difference by looking at one
non-constant mean in comparison to another confidence limit
of a mean; it does not matter which group you use for the CI.

It is not even proper to infer your conclusion from the
observation (if it is so) that the two Confidence Limits do
not overlap. It might be fairly safe, but it is not proper inference.

Your question is whether the *differences* are different, so
that difference is what a CI should be placed on.

By the way, when groups are not randomly chosen -- your groups,
Young versus Old, are certainly not randomly chosen -- the additional
problem exists in that the two Control groups are assumed to
be equal, or else the test is questionable. Your first obligation
is to confirm that the Control groups are equal, before comparing
their improvement-scores. -- For instance, if both groups *end up*
with the same score, it is some artifact (not, "treatment") that makes
groups different for the Controls.

I
think perhaps I should do it the other way round, i.e. calculate the
confidence interval for the younger group and show it does not contain
the effect size for the older group? Essentially what I am trying to
do is show that the non-significant effect in the younger groups is
unlikely to be due to lack of power.

Same at the start, different at the end. Yes, it is useful to
point to the actual scores.

I would have done a post-hoc
power test, except the data isn't normally distributed and anyway lots
of people seem to say post-hoc power tests are a Bad Thing (I'm not
sure why, and I see it done in the literature, but anyway).

I prefer to transform my scores to get a reasonable
scaling. Post-hoc tests add no information, and are
misleading when the residual d.f. is small.

--
Rich Ulrich, wpi...@xxxxxxxxxxxx://www.pitt.edu/~wpilib/index.html- Hide quoted text -

- Show quoted text -


.

Relevant Pages

  • Re: Bootstrapping for confidence interval of effect size
    ... I have a question about whether I have used bootstrapping correctly. ... For two different ages I have a treatment group and a control group. ... younger ones which shows no significant effect. ... problem exists in that the two Control groups are assumed to ...
    (sci.stat.edu)
  • Re: First digital TV switch date set
    ... prepared to compare it with the treatment of prisoners by the British. ... Japanese military did back then is completely unacceptable and has no ... which is being fuelled by the ignoring Japans actions during WWII ... Comparing the activities of soldiers who were punished for any activities to ...
    (uk.tech.broadcast)
  • Re: One or Two tails: II ?
    ... Put in terms of comparing a new treatment to ... standard, anyone who insists on a one- tailed test is saying the new treatment can't do worse ... superior to the medical care they would receive, ...
    (sci.stat.math)
  • Re: Method of alternation of lenses. Whether it is applied in America?
    ... We usually estimate efficacy by measuring the average change in a treated ... group and comparing it to the average change in another group not receiving ... When we compare treated and untreated groups, we find treatment with lenses ...
    (sci.med.vision)
  • Re: I found the holy grail that explains audiophile beliefs
    ... scales, superposition, etc) comparing the two locations with treatment. ... We aimed to improve only the bass frequencies. ... I do have a graph already posted that adds to the discussion. ...
    (rec.audio.pro)
Quantcast