Re: Paper recommendation



Jerry Dallal wrote:
> clemenr@xxxxxxxxxx wrote:
> > Let's assume that someone is trying to find some improved version of
> > some medicine. If they create a variant of the medicine, then standard
> > double-blind testing with both treatment (variant) and control
> > (original version) groups is performed. The difference in performance
> > of both the original and variant medicine is then used in a one-sided
> > hypothesis test where the null hypothesis is that the original medicine
> > works the same or better than the variant, and the alternate hypothesis
> > is that the variant medicine works better, and let's say that a
> > significance level of 0.05 is used in deciding whether to reject the
> > null hypothesis. That is, bar garbling, fairly standard I believe.
>
> No!
> http://www.tufts.edu/~gdallal/onesided.htm

Ahhh... after reading your page I must say I now agree with the
criticisms of the one-sided tests. I didn't see using a one-sided test
as being an assumption that the variant medicine can't do worse than
the original, but as meaning that we're uninterested if it is
significantly worse. But, as your page points out, using a one-sided
test makes it much easier to conclude that there has been an
improvement.

Also, I did make the example a medical one, and again (the cholesterol
example) I note your page's point that it's also informative if (as an
example) a medicine intended to reduce cholesterol actually increases
it, this is still interesting and informative and shouldn't be ignored.

> > But, what I'd like to see is a good example where some researcher has
> > actually considered their previous failures when evaluating a later
> > experiment. I'm not concerned what branch of science the paper is from,
> > as long as I have a chance of understanding it.
> >
> > Any recommendations?
>
> Not for a paper. However, investigators are not stupid. No one in his
> right mind undertakes a drug study in the belief that the new drug will
> fail. Many would say that information from the previous failures went
> into the design of the new trial.
>
> BTW, 100 failures in a row would be remarkable in and of itself.

Hmmm.... My example was a medical example, but I'm more used to
computer science. As you point out, in medicine people will be learning
from the failures, and therefore the probability of future success will
rise. But, in certain areas in computer science, creating variations of
(say) a genetic algorithm and experimenting with it can be quite quick
and easy, and people may be making changes without really understanding
them or learning from failures. I don't mean to criticise your
response, as you were responding to my original example which was a
medical one. And of course it would be rather unrealistic to expect
your response to fit with things I was thinking of but didn't mention
:-)

So, in one way of thinking, my using a medical example could be seen as
a mistake. But, I've learnt from your reply and your page so maybe it's
been a useful mistake for me to make.

Cheers,

Ross-c

.