Re: Statistics to test data pairs

From: Richard Ulrich (Rich.Ulrich_at_comcast.net)
Date: 02/21/05 

Date: Mon, 21 Feb 2005 15:45:46 -0500

On Sat, 19 Feb 2005 00:16:16 +1100, "CW" <CW@nospam.com> wrote:

> I haven't used statistics for many years, so please bear with me if my
> questions sound rudimentary.
>
> Here is the scenario:
>
> I am responsible for testing a new billing application. I am thinking of
> testing using the following methodology.
>
> For each billing type, I would generate billing result from this new
> application (and as well as from the old billing application the new system
> is replacing).
>
> As the old and the new system are supposed to have identical inputs, the
> results are supposed to be the same. Of course, in real life, it will not be
> the case. The differences could be due to (1) error in the old billing
> application; (2) error in the new billing application and (3) error in both
> systems.

This is hard to imagine. You expect the computer-part of a
billing system to generate so many errors that you need to
compare them statistically?
 - I think I've seen such a lousy system, but not where it
mattered.

>
> For each billing type, there are hundreds to thousands of result (grouped by
> customer and period).
>
> Say, I could generate 1000 pair of data (i.e., for 1000 customers) from
> both billing systems.

There are so many errors, that you expect dozens to show
up within 1000 bills-generated? AND you want to count them?

This is not how computer systems are validated, I think,
though my personal experience is limited.

You should have a file of previous errors that were fixed -
the new system should not re-create old problems.

You should have a computerized system of comparing
New vs. Old so that there is not an artificial limit of (say)
1000 pairs of data, simply because you can't review that many.
>
> Out of them, x pairs would be identical between the old and the new system,
> and 1000 - x would be different.
>
> I am thinking of picking a sub-sample, say N, out 1000 - x pairs for my
> testing purpose.
>
> What sort of statistical methodology I can employ to say, after having
> compared N pairs and finding the new billing system to be correct each and
> every time, I am 99% confident that the new billing system has generated
> correct data.

The "Mythical Man-Month" is a good book on software
engineering, or so I've been told. 

-- 
Rich Ulrich, wpilib@Pitt.edu
http://www.pitt.edu/~wpilib/index.html

Relevant Pages

  • RE: Comparison Query
    ... You would need a query showing all the companies table with right join to ... show all compaines and only billing records that match. ... customers did NOT use our services the week before. ... to structure a query to show or compare from week to week who give us ...
    (microsoft.public.access.queries)
  • Statistics to test data pairs
    ... I haven't used statistics for many years, so please bear with me if my ... I am responsible for testing a new billing application. ... As the old and the new system are supposed to have identical inputs, ... correct data. ...
    (sci.stat.edu)