Re: Bootstrapping for confidence interval of effect size

On Mon, 30 Jul 2007 10:05:58 -0000, amorphia <spam.ontoast@xxxxxxxxx>
wrote:

Hi all,

Thanks for your reply Rich. I take your point about how you can't
compare a confidence interval to a non-constant mean. However, I have
seen it written in many places that comparing a confidence interval to
a constant is equivalent to a hypothesis test.

- Right. You can create a CI by inverting the test.

So it seems to me that
logically there ought to be some way to compare two confidence
distributions (which is what you get with a bootstrap).

Well, yes, but not efficiently, and only if they are independent.

For example, you could compare every point (Ai) in bootstrap
distribution A with every point (Bi) in bootstrap distribution B, and
count how often Ai > Bi. That would give your your confidence that A
is greater than B, right? It seems to me that it is exactly the same
logic as comparing against a fixed constant.

Do it that way. That gives a better limit. Complications that
arise are most likely with highly unequal Ns and variances,
and I really don't know what is assured for that sort of case.



Your point about how the control groups are not equal doesn't apply -
I think you missed a point in my original point - I am comparing the
effect sizes of the two comparisons, not the means. There may be
something wrong with that aspect of what I did but it isn't what you
said! ;)

Sorry, but I still think you are missing a basic point of experimental
design. There are three potential criteria for what group ends
up better -- which has the better final score, which has the better
change score, or which has the better *regressed* change score.
These are not assured to be in agreement, if the groups are not
comparable at their starting scores,

These especially can arise when there are scaling problems,
such as 'basement' and 'ceiling' effects. It is nice to assure
the audience that such problems won't effect the analysis or
conclusions.

There are separate problems that can arise in deciding what is
the appropriate regression for the change, in special cases.



By the way, I have always in the past transformed my data too, but
this data is just untranformable - you would see what I mean if you
saw it.
[snip, previous]

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: TECH: Capcom BBB lights "pulsing"
    ... I can assure you that this problem has nothing to do ... with power supply loading. ... but I have a feeling I'm going to have to compare ... in cases where there are multiple grounds with differing potentials. ...
    (rec.games.pinball)
  • Re: TECH: Capcom BBB lights "pulsing"
    ... I can assure you that this problem has nothing to do ... with power supply loading. ... Cliffy: I appreciate the offer. ... but I have a feeling I'm going to have to compare ...
    (rec.games.pinball)
  • Re: TECH: Capcom BBB lights "pulsing"
    ... I can assure you that this problem has nothing to do ... with power supply loading. ... You don't by chance have a scope, ... but I have a feeling I'm going to have to compare ...
    (rec.games.pinball)
  • Web Screen Scraping Tools
    ... Are there any other tools out there like SWEA: ... I just like to compare the options out there... ... I'm trying build a rich ... client which will scrape data and interact with a website. ...
    (microsoft.public.dotnet.framework.aspnet)
  • Re: Which scope would you rather use? (Deepsky)
    ... Rich wrote: ... I had both scopes at the same time and was able to compare ...
    (sci.astro.amateur)