Re: (hyper)sensitivity of goodness-of-fit tests

On 20 Nov 2006 07:15:59 -0800, "Old Mac User"
<chendrixstats@xxxxxxxxx> wrote:

Why would you go to so much trouble to numb the test so that it fails
to show significance?

- The obvious next step seems to me to be to *quantify* the
amount of deviation of the fit. The Ns are huge, so the tests
are big, but does it *matter* to the OP? What is the purpose of
the fit?


The data you posted indicate a strong departure from an exponential
distribution. Just a simple plot of the data reveals this, even without
applying a Chi-sq test to verify it. IMHO this departure is bad enough
to disallow using an exponential approximation. Why not consider
another distribution (gamma, perhaps) and move on with it? It would be
much faster to do that than to waste most of the data just to numb the
Chi-sq test. OMU

I agree with OMU, "fit something else" -- *if* the sample size was
wisely chosen to detect a bad 'fit' that matters. But if the sample
is big because data just happens to be there... what matters next
is the OP's purpose, and how much the fit (and non-fit) matters.
Does it matter that the observed 'tail' is much fatter than
predicted?

There are trends in the deviations of the fit. The best clue about
the nature of a distribution is often in the question, "How was it
generated?" Does a reason suggest itself? For the purpose of the
OP, unmentioned so far, it *might* be enough to describe the fit,
and describe the deviations.


[rest of post included without additional comments.]




morfysster@xxxxxxxxx wrote:
Thank you very much for your responses.

What if I took random subsets of the observed data, and conducted the
goodness-of-fit tests using these smaller subsets and then used the
average of the p-values corresponding to these tests? Would such an
approach be valid?


On Nov 17, 4:16 am, "Reef Fish" <large_nassua_grou...@xxxxxxxxx> wrote:
morfyss...@xxxxxxxxx wrote:
I have a large amount of empirical data consisting of interarrival
times that I believe are exponentially distributed. Looking at the
quantile-quantile plot between the empirical and theoretical/fitted
distribution, I see an almost perfect linear relationship.

However, if I conduct a chi-squaretest, or kolmogorow-smirnovtest,
the tests strongly reject the hypothesis that the interarrival times
are exponentially distributed. It seems that these tests are "too
sensitive" for large data sets.Chi-squared and the Kolmogorov-Smirnov TESTS are arguably the
two WORST tests ofGoodness of Fit, for more reasons than I can
state in a 100 page treatease on the subject. :-)

What you have observed is the seemingly paradoxical FACT that
those tests are insensitive to departures and violations that can be
seen easily in small samples (via a Q-Q plot say), but are ultra-
sensitive in very large samples -- but this latter DEFECT is inherent
in ALL Neyman-Pearson type of hypothesis testing -- so Kolmogorov
Smirnov, Ron Barcardi, and other tests are only to be blamed IN
PART -- Neyman Pearson (you can include FIsher as well) are part
of the guilty gang!

It is analogous to the fact that if youtestHo: mu = 0 against its two
tail alternative, you'll know FOR SURE that Ho will be rejected if
your
sample size is large enough -- perhaps your several million will do.

It's all imbedded into FOLLY of non-Bayesian statistics in general,
and in the methodogy of Hypothesis Testing in particular.

-- Reef Fish Bob.

(In our case, we have over 1.5 million
interarrival data points.) Are there standard techniques for applying
goodness-of-fittests to large pools of empirical data to make them
less senstive to arguably insignificant and minor discrepencies between
real-world data and a theoretical distribution? Thanks very much for
any help.

(Below I give an example of what happens with a chi-squaretest.

For example:

interval length empirical expected chi-square
summand

0-2 927,256 911,012.88 289.61
2-4 397,959 413,849.31 610.13
4-6 181,593 188,000.92 218.41
6-8 85,075 85,403.90 1.27
8-10 40,289 38,796.76 57.40
10-12 18,914 17,624.35 94.37
12-14 9,443 8,006.28 257.82
14-16 4,510 3,637.04 209.52
16-18 2,185 1,652.21 171.81
18-20 1,037 750.56 109.32
20 1,098 624.78 358.43

totals: 1669359 1669359.00 2378.07

Looking at a chi-square table, with 10 degrees of freedom from 11 bins,
we reject the null hypothesis that the interarrival lengths follow an
exponential distribution.)

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: (hyper)sensitivity of goodness-of-fit tests
    ... amount of deviation of the fit. ... consider another distribution and move on with it? ... There are trends in the deviations of the fit. ... designed to look for departures from an exponential distribution .. ...
    (sci.stat.math)
  • Re: (hyper)sensitivity of goodness-of-fit tests
    ... amount of deviation of the fit. ... The data you posted indicate a strong departure from an exponential ... consider another distribution and move on with it? ... There are trends in the deviations of the fit. ...
    (sci.stat.math)
  • Re: Probit analysis
    ... In a log-likelihood ratio test, you would fit the probit model ... compared with a chi-square distribution. ...
    (sci.stat.math)
  • Re: Goodness of fitting of a distribution
    ... plot that points out that the best distribution that fit my data is a ... linear combination of a weibull and a normal distribution. ... I don't need to read your Berkeley Symposium to know that the K-S ... it is the large number of bins which reduces the ...
    (sci.stat.math)
  • Re: Goodness of fitting of a distribution
    ... plot that points out that the best distribution that fit my data is a ... linear combination of a weibull and a normal distribution. ... It is the chi-squared test with many classes which has ... it is the large number of bins which reduces the ...
    (sci.stat.math)