On paired samples spontaneously arising in computer-simulated experiments



In my opinion an experiment should be designed so as to yield paired
samples (i.e.: before-treatment and after-treatment measurements) only
in the case one wants to understand whether the treatment results in
some effect or not.
Such experiment biases things in such a way so as to reveal even the
least effect of the treatment. That is, the experiment des not
simulate the reality, but it tries only to detect an effect, whatever
little it may be. Then the experiment that yields paired samples is
not good in order to reveal whether the treatment is worth giving;
that is, whether the population of after-treatment measurements is
significantly different from the one generated by before-treatment
measurements.
In my computer-simulated experiments, I have obtained spontaneously
two paired samples (= before- and after-treatment measurements), even
if I had no intention to test for understanding whether the treatment
had any effect. I was perfectly aware the treatment was somehow
effective!
Instead, I made computer-simulations in order to understand whether
the treatment was worth giving, since the implementation of the
treatment was not free of charge.
To be more precise: the object under test was a network whose
configuration was changing randomly. The treatment consisted in the
implementation of a deterministic criterion for assigning transmission
frequencies, instead of letting the frequencies be randomly assigned.
The treatment avoids the transmission frequency interference resulting
from random assignment of near frequencies to adjacent transmission
paths. This explains why it was obvious to me that the treatment had
some effect in any case.
I judged the network performance by means of a figure of noise that
accounted for transmission frequencies interference. The cost of the
treatment consisted in larger energy consumption due to the
implementation of the deterministic criterion for assigning
transmission frequencies. The consumption of energy was a critical
factor for the network design since network nodes (= elements) were
battery operated and the batteries can not be easily replaced.
Then I decided that I would have implemented the treatment if - and
only if - the population of the noise figures generated by the
treatment would have been significantly different from the one
resulting from the random transmission frequency assignment.
For this reason, I have used the 2 sample Kolmogorov-Smirnov test as
if the two samples were independent (actually the two samples were not
independent since they were obtained as repeated measurements, that
is, before- and after-treatment measurements).
The 2 sample Kolmogorov-Smirnov test resulted not significant, so that
I could not reject the null hypothesis that the samples came from the
same population. Then I did not implement the treatment.
I would like to know if my reasoning is correct, or rather if I should
have used a 1 sampled test applied to the differences between the two
samples.
.



Relevant Pages