Re: Splitting samples to minimize false positives?
- From: adiamond@xxxxxxxxxxxxxx
- Date: Mon, 31 Dec 2007 11:59:32 -0800 (PST)
The way I think about is this:
There's really two possibilities (Ho, Ha)
1) The sample was taken from the expected distribution
2) It wasn't.
In the first case, a low pval is simply a happenstance/false
positive. If you cut the sample size in half (or whatever), the
nominal failure rate will be the same. Take the example I gave in the
original post. I compared samples taken from a random distribution to
a random distribution and failed 5% of the time regardless of whether
I used the whole sample(s) or cut them in half. After all, the
standard error takes the sample size into account.
So, by cutting those samples in half (again as demonstrated by the
example), creating two tests for everyone I had, a failure requires
two happenstances (FP) and even though the full sample failed, the
probability of both samples failing simultaneously are much smaller.
(Actually, this is an odd item because once you know the full sample
failed (i.e. was Ha), then the probability that samples obtained from
that sample will fail also. Even so, my thousands of tests seem to
confirm that the failure rate dropped drastically).
The other side is the second case: Let's say that I drew my samples
from a uniform distribution and compared it to a normal distribution.
Regardless of whether I cut my samples in half or not, even for a very
small sample, the normal test will fail, i.e. yielding Ha, just about
always.
However, as I think you pointed out, it is true that by cutting down
the sample that there will be an increase in False Negatives but
that's not tied to the nominal rate and for reasonable sample sizes
that rate will be small (right?)
Obviously, if I took the samples from a distribution that was much
more similar to normal (say just a small difference in means) then
disambiguating FP from TP is more difficult and anding the hypothesis
results together will likely drastically increase the FN rate. This
is your point about "tiny differences"
So,let's take a database (db) example where I'm comparing, say, the
number of blanks, across all records, in the "name" field in the old
versus the new database. I want to know if the fraction of blank
names has changed between the old and new database (lots of blank
names indicating a data entry problem).
1) Say that the new db doesn't have a data entry issue. Given a 5%
significance, a proportion test will fail yield a 5% false positive
rate.
Given a sufficient sample size, if I divide the samples in half and
demand that both fail simultaneously to truly believe the negative
result my false positive rate will be a fraction of what it was.
2) Now lets say the new db DOES have a data entry issue. The sample
size would have to be pretty small for it to fail so demanding that
both half sized samples fail to believe that the dbs are different
won't hurt much. (which is to say that the FN rate due to employing
this technique will be higher by a bit but I won't get a nominal 5% FP
rate).
Maybe the issue here is that my problems have a built in asymmetry
coupled with the fact that the statistical tests have no concepts
about the distribution of the class of data that corresponds to Ha
(perhaps this is frequentist vs. Bayesian issue?). If the databases
(e.g. just the name field w/r to blanks) are the same than halving the
sample size and 'anding" them together will drastically lower the FP
rate and, for this possibility, there is no such thing as a FN.
On the other hand, if the databases aren't the same, the screw up will
be large (that's an assumption) and so the proportions will be very
different. So given sufficient sample sizes (e.g. splitting the
sample in halvs), the proportion test will just about never succeed
(I.e. low FNs) and so ANDing them won't change much (because the
results of both half samples will invariably be 1).
The bottom line is that in such a situation I seem to be reducing my
FP rate for free.
I think the questions gets more interesting if the data entry screw up
can be minor. Then you have the problem where a failed proportion
test could be due to the nominal happenstance or a slight difference
in proportions.
Even so, as you've written, you can take the results of the two tests
and combine their probabilities vs. just anding there Hypothesis
results.
.
- References:
- Splitting samples to minimize false positives?
- From: adiamond
- Re: Splitting samples to minimize false positives?
- From: Richard Ulrich
- Splitting samples to minimize false positives?
- Prev by Date: Re: Simple probability question (volume vs. probability)
- Previous by thread: Re: Splitting samples to minimize false positives?
- Next by thread: Question on independence in multi-way tables
- Index(es):
Relevant Pages
|
|
