paired test result interpretation

jkkuchar_at_mit.edu
Date: 03/18/05 

Date: 18 Mar 2005 07:22:23 -0800

First, thanks for the comments on my earlier question regarding the
variance of a ratio of sums.

Now I have a different issue to resolve - I'd like to determine whether
there is a significant difference between two conditions, A and B,
after running a large set of paired tests.

Here's an example: I ran a randomly-generated set of 150,000 vehicle
simulations with a certain type of collision avoidance system (A). I
then ran the identical set of 150,000 cases with a different system
(B). Each of the 150,000 runs resulted in a non-zero probability of
collision. Only 113/150,000 simulations resulted in a difference in
collision probability between A and B; the rest had absolutely no
difference in collision probability. So, there is a very large set of
data with 0 difference, and a relatively small set of data with values
of positive or negative difference. The non-zero differences are skewed
and do not appear to be normally-distributed.

The mean difference between collision probabilities for A and B
suggests that A is slightly better than B. A paired T-test, however,
shows no significant difference. A sign test shows a significant
difference (p<0.001) and suggests B is better than A. A Wilcoxon
Matched-Pairs Signed-Ranks Test also shows a significant difference
(p<0.001) and suggests B is better than A.

Am I justified in concluding that B is signficantly better than A, even
though the mean difference actually suggests A is better than B? Seems
to be a battle between the mean and median as a measure of performance.

Is there a different test I should be using?

- Jim