Re: (hyper)sensitivity of goodness-of-fit tests


morfysster@xxxxxxxxx wrote:
I have a large amount of empirical data consisting of interarrival
times that I believe are exponentially distributed. Looking at the
quantile-quantile plot between the empirical and theoretical/fitted
distribution, I see an almost perfect linear relationship.

However, if I conduct a chi-square test, or kolmogorow-smirnov test,
the tests strongly reject the hypothesis that the interarrival times
are exponentially distributed. It seems that these tests are "too
sensitive" for large data sets.

Chi-squared and the Kolmogorov-Smirnov TESTS are arguably the
two WORST tests of Goodness of Fit, for more reasons than I can
state in a 100 page treatease on the subject. :-)

What you have observed is the seemingly paradoxical FACT that
those tests are insensitive to departures and violations that can be
seen easily in small samples (via a Q-Q plot say), but are ultra-
sensitive in very large samples -- but this latter DEFECT is inherent
in ALL Neyman-Pearson type of hypothesis testing -- so Kolmogorov
Smirnov, Ron Barcardi, and other tests are only to be blamed IN
PART -- Neyman Pearson (you can include FIsher as well) are part
of the guilty gang!

It is analogous to the fact that if you test Ho: mu = 0 against its two
tail alternative, you'll know FOR SURE that Ho will be rejected if
your
sample size is large enough -- perhaps your several million will do.

It's all imbedded into FOLLY of non-Bayesian statistics in general,
and in the methodogy of Hypothesis Testing in particular.

-- Reef Fish Bob.




(In our case, we have over 1.5 million
interarrival data points.) Are there standard techniques for applying
goodness-of-fit tests to large pools of empirical data to make them
less senstive to arguably insignificant and minor discrepencies between
real-world data and a theoretical distribution? Thanks very much for
any help.

(Below I give an example of what happens with a chi-square test.

For example:

interval length empirical expected chi-square
summand

0-2 927,256 911,012.88 289.61
2-4 397,959 413,849.31 610.13
4-6 181,593 188,000.92 218.41
6-8 85,075 85,403.90 1.27
8-10 40,289 38,796.76 57.40
10-12 18,914 17,624.35 94.37
12-14 9,443 8,006.28 257.82
14-16 4,510 3,637.04 209.52
16-18 2,185 1,652.21 171.81
18-20 1,037 750.56 109.32
20 1,098 624.78 358.43

totals: 1669359 1669359.00 2378.07

Looking at a chi-square table, with 10 degrees of freedom from 11 bins,
we reject the null hypothesis that the interarrival lengths follow an
exponential distribution.)

.

Relevant Pages

  • Re: sampling from a circular distribution
    ... The distribution of this sample is even less uniform. ... > standard chi-square test for significance since you know the observed ... > and expected values for each pixel value. ...
    (sci.stat.math)
  • Re: clutter modeling techniques
    ... > I am a newbie in stats and mostly work in digital electronics. ... Clutter modeling is the term used in radar/sonar ... > options that can give best results, should the distribution is known. ... chi-square test and KS test ...
    (sci.stat.math)
  • (hyper)sensitivity of goodness-of-fit tests
    ... I have a large amount of empirical data consisting of interarrival ... distribution, I see an almost perfect linear relationship. ... However, if I conduct a chi-square test, or kolmogorow-smirnov test, ... real-world data and a theoretical distribution? ...
    (sci.stat.math)
  • Probability of getting relative frequency (or probability) distribution
    ... There is one distribution of relative frequencies which I treat as default ... I am thinking of Chi-square test, and p-level, but I am not sure in both, my reasoning, and the test itself. ... Is it possible to have the probability of getting a distribution from the default one. ...
    (sci.stat.math)
  • Re: (hyper)sensitivity of goodness-of-fit tests
    ... What if I took random subsets of the observed data, ... goodness-of-fit tests using these smaller subsets and then used the ... distribution, I see an almost perfect linear relationship. ... in ALL Neyman-Pearson type of hypothesis testing -- so Kolmogorov ...
    (sci.stat.math)
Loading