Re: Two nit-picks re definition of p-value (Was: goodness of fit ?)


Kevin E. Thorpe wrote:
Reef Fish wrote:
Kevin E. Thorpe wrote:
<SNIP>

Cutting to the point of controversy about the inclusion or exclusion of

"2" in the application of "more extreme" when "2" is actually observed,
in the N-P definition of a p-value.

The old saw, "don't fix it if it ain't broke" clearly apply to the
EXCLUSION of X = 2 from the p-value definition unambigously
defined under the N-P approach, instead of fixing it (when it ain't
broke) as some textbook writers tried to do. The Agresti approach
is completely OFF THE WALL -- it has NO justifiable support of
his p+ and p- and other witchcraft brew cooked by him, unjustified
by all THREE approaches -- Bayssian, N-P, or Fisher.

It would appear that most statisticians I have encountered
would compute a p-value, particularly in the discrete case,
in the manner of Fisher even when using NP hypothesis
testing. This will naturally be a conservative test in terms
of alpha whereas excluding the observed value will increase
the type I error

It COULD be used as a conservative test -- or it is more LIKELY to
be an error, as most people do make errors about the p-value even
in the continuous case!

Secondly, if one wants to be more conservative than setting alpha
as the level of the test, it would be MUCH MORE easy and DIRECT
to simply lower the level of alpha, to whatever extent of conservatism
desired.

probability since, in general you don't have
an exactly level alpha test for the discrete case.

That is a very WEAK excuse, especially in the use of p-value for
making the test decision! The use of p-value does NOT presume
any value of alpha to use! One simply looks at the p-value and
decide (on the basis of whatever alpha desired) whether to accept
or reject.

A perfectly straightforward procedure, IRRELEVANT to the issue
of whether the Test Statistic can actually ATTAIN any particular
alpha value.


In my
opinion, the "best" practice may well depend on the
consequences of a type I error. In my work with primarily
clinical trials, you REALLY do not want to conclude one
treatment is better when it is not, so I would err on the
conservative side.

Then set your alpha < 0.05 if that's the one you usually use.
That's why other folks perfer to use alpha = 0.01.

In short, I consider your preceding paragraph a limp EXCUSE rather
than a rationale in defense of the misuse of "more extreme" to include
that value which is NOT "more extremely", and then use the convenient
"conservatism" as the corollary excuse.

If you want to be more conservative, just choose a SMALLER value
of alpha for Probability of Type I error, or Pr(rejecting Ho when Ho is
true).

All other things aside, that is the most LOGICAL thing one can do,
without doing all the song-and-dance to justify "2" is MORE EXTREME
than "2".

Simplicity and the principle of parsimony is usually, and in this
particular
case, preferable to the ad hoc fix, expecially when the ad hoc fix has
no
theoretical justification. Fisher didn't have to justify his use
because
he doesn't talk about p-values. But for those who DO use p-values,
I have yet to see any valid argument that "2" should be considered
"more extreme than 2" regardless of the hypotheses or the alpha value
chosen for a test.

-- Reef Fish Bob.


--
Kevin E. Thorpe
Assistant Professor, Department of Public Health Sciences
Faculty of Medicine, University of Toronto

<SNIP REST>

.

Relevant Pages
Loading