Re: Statistical Analyses of Non-Static Group Question

On Aug 12, 4:06 pm, Doug Morse <mo...@xxxxxxxx> wrote:
Just for completeness, I suppose I should also mentioned the statistics
approaches that I do think might be relevant. After getting through your
last post, the thought of some sort of time-series linear regression came to
mind. Time series regressions are pretty common in business, finance, and
economics. There are many different kinds of time-series regression models.
For your situation, I suspect that one of the moving average models might be
most applicable (i.e., ARMA, ARIMA). This link provides a sound overview,
but is fairly technical:

http://en.wikipedia.org/wiki/Autoregressive_moving_average_model

These models are intermediate to advanced, so you really have to know what
you are doing to do them "right". You have to make sure you've met all the
assumptions of the model, such as variable independence, univariate or
multivariate normality (or other distributional assumptions), variance
assumptions, etc., as well as check for model validity after it's been run,
such as no systematicity in the residuals, look for possible collinearity
issues, etc.

As I mentioned before, though, I'm wondering if just re-conceptualizing your
data and perhaps re-aggregating in a different way might do the trick. When
it comes to these advanced statistical models, the question really becomes:
Do you really need such a scientific and rigorous treatment of your data?

Doug



On Sun, 12 Aug 2007 04:12:40 +0000 (UTC), Doug Morse <mo...@xxxxxxxx> wrote:

I think that's it for questions. It sounds like what you have in place is
working well for you -- up to the limits of the approach, and it seems
you've done a fine job identifying those limits. I'm not sure that
"statistics" per se is will be the fix to those limits, but it may might
play some role.- Hide quoted text -

- Show quoted text -

Doug,

Your analysis, to say the least, is spot on. In the testing program
there are 20 some odd projects representing 26 or so different models
of product that in all case has the same basic functionality. Over
time the product has been enhanced with different feature sets and
capabilities. To date none of the products have been retired though
some are close. The products can be roughly grouped into three
categories: active, stable, and dormant; at least this is how we refer
to them. Active products receive frequent updates which may consist of
fixes or new features. Stable product updates are less frequent and
mostly consist of maintenance updates, although on occasion features
may be back-fitted from the newer models and implemented in the older.
Dormant products receive the least attention; these are the products
that are near the end of their life-cycle.

The program itself has grown from what I would call an ad hoc or
unmanaged state to something that is better managed. Some of the
changes are due to the toolset used to manage the program; other
changes are due to the definition, implementation, standardization,
and enforcement of a fair set of guidelines and policies (something
I'm proud to have been a part of).

The size of each project is different and varies from a low of 4 to a
high of well over 250 participants. The older projects have the fewest
members while the newest has the most. As complexity in the products
increase, so do the number of participants in the project.
Participants may participate in a maximum of two projects and cannot
test a newer product until they have proven themselves on an older
model (this is a fairly new policy).

Initial attempts of gauging performance/participation were by counting
the number of problem reports. While this works well for active
projects it's less effective and even useless for the less active
projects. Other forms of feedback were introduced so that the
participants in the least active projects would have something to
submit for each evaluation cycle (even if it was just to report that
they haven't seen a problem).

It's been 18 months or so since the new feedback forms were
introduced. At the time the forms were introduced I reset the rating
system so the introduction of the forms marks the beginning as far as
data collection is concerned.

The analogy of "data bundles or streams" is very insightful. As I
mentioned in my previous post we weight each form of feedback based on
its importance to the program. I would say there are two distinct
streams, the error reporting stream and the status stream. The error
reporting stream actually serves two purposes in that it is also
considered part and parcel with the status stream.

I'd like to state for the record that the goal of this exercise is not
to determine who the better tester is. The goal is to determine how
each group of participants is performing. Once we can determine a
trend we can then perhaps make additional changes as 'see' what the
effect is. The reason for such interest in the group performance is
that for any given project, regardless of its state, the number of
underachievers far outweighs the number of achievers. An achiever is
defined as someone who, by our rating system, is rated average or
better. A typical ratio is somewhere between 50/50 and 70/30
underachiever to achiever.

We have done some analysis and as far as I'm concerned the results are
indeterminate. And the reason for that, I believe, is because there is
so many variables to consider for each sample period such as the
number of updates released, the nature of the updates (fixes, feature
addition, feature removal, or a combination), the duration that each
update is available for testing (before being superseded by the next
update), the number of participants, the quality or experience of the
participants, and participant turnover (churn). All of these factors
play a part in driving what I will call the rate of return and I think
they all need to be considered when attempting to compare the results
of one sample period against another.

One thing I haven't mentioned which is probably very important is that
as new products are introduced and projects are manned for testing the
new products, the best (proven) testers are siphoned off from the
older projects leaving the older projects stripped of quality/
experienced testers and often undermanned. I don't condone this
practice, but then I am but a cog in much larger machine and don't
have much say in the matter.

I think I've answered your 3 main questions. Here's a summarized
version:

1. There really isn't a typical size, it depends on the project (which
is analogous to the product it represents), and varies from 4 to 250+.
If by inactive you mean less active then the ratio is as stated above.

2. Yes we have done some analysis and I don't trust the results due to
a combination of factors.

3. No it's not one large group but a collection of groups. Each group
is receiving the same experience for the given product they are
testing. In some cases participants are members in multiple projects
and since some projects are under different management you might say
these participants are receiving a different experience even though
the same rules for reporting apply to all projects.

You've given me a great deal to think about. It's great to have a
fresh perspective on an issue that I have been thinking on for some
time. It's quite possible that I'm considering (or think I should be
considering) to many variables.

I will certainly take a look at time-series regression and moving
average models and what application they may have.

Thank you so much for your time and effort,

Gary

.

Relevant Pages

  • Re: Cross thread question .....
    ... I have an app that reads data params from a stream and updates ... the main thread that created the controls. ... Your stream thread just reads results and updates the "neutral zone" ...
    (microsoft.public.dotnet.languages.csharp)
  • Going to abercrombie.com site streaming radio
    ... ISS Defender and it hasn't affected the stream before and I have not made any updates for a month. ... Last known updates from SYSTEM RESTORE is Installed WMP KB837272 and Windows Update V4. ...
    (microsoft.public.windowsmedia.player)
  • Re: Re: NetBSD disk backup over network
    ... it spake thus: ... You can easily save the stream of updates as a redo log (well, ...
    (freebsd-hackers)
  • Re: More DPreview Dribble
    ... site - and still not one single mention of the excellent Silkypix software, despite a stream of updates that have reinforced Silky's position as *the* number 1 RAW converter currently available. ... Which all goes to prove the truth of old Confucian saying; "He who does not stuff freebies up Askey's arse will not be mentioned on DP Review" ...
    (rec.photo.digital.slr-systems)
  • Re: ext3 __log_wait_for_space: no transactions
    ... Any updates on whether the patch that I've posted has solved this regression? ... More majordomo info at http://vger.kernel.org/majordomo-info.html ...
    (Linux-Kernel)
Loading